I believe that “the right size” is relative. Everyone’s preferences are different, and penetration isn’t the only thing that makes sex pleasurable... So my opinion is that there’s no size that’s too small for fun and pleasure.
As far as being able to achieve penetration, there are situations/bodytypes/penis size where penetration wouldn’t work very well, so I suppose if you’re talking strictly “p in v” or “p in a”, then yes, there are penises that will be too small/incompatible for that specific experience...
I once had a lover with a micro penis and we still had great sex without his penis inside me. Maybe if he was with a tiny spinner body type of person, penetration *might* have been able to happen.. But who knows. The only way I’d classify him as too small is strictly in his ability to enter my pussy with it. Otherwise, I wouldn’t say his dick is too small, as a general statement.