The latter, yes. To make a very blunt but (to me) accurate comparison:

Imagine every work lunch involves people slapping your eyeballs with sausages every ten seconds. It's fine. You are prepared for it. You have years of practice not letting it visibly bother you. But every once in a while you are not 100% up for it and ask people to "please, sorry, just take a break with the... fucking sausages for five minutes?" and you have completely and utterly fucked up socially.

Just don't let the sausage eye-slapping bother you.

It is wrong for you to be annoyed by it. Your feelings are wrong.

Other people don't even notice and you must be like that too.

This in all situations for years and years and the end result is a weird kind of toughening. You get really good at not caring that you sit uncomfortably, are too cold, too hot, are scared, bump your elbow, are exhausted, etc. Then you can get annoyed by people around you letting minor inconveniences stop them while simultaneously calling you sensitive.