Part of exercising your liberty comes down to not caring what everyone else thinks. Hopefully, you are able to do this in a way that doesn't come off as rude to observers, but sometimes it's not possible.
You should care what some people think. It's probably not healthy to not care what anyone thinks. You'd be a psychopath or something if that were the case. But most people "out there" aren't really worth coddling.
These people will try to bully you into doing or saying what they want. It's going to probably seem rude to them when you refuse. But how would it feel to you if you caved in? How would it look to others who see you comply with something you (and they) know isn't right? Which is worse?
I usually try to find a way to refuse to go along that isn't confrontational. Often I simply ignore them-- they aren't worth the effort. Sometimes you're not left that option. Sometimes to refuse is to be confrontational. In that case, it helps if you don't get too wrapped up in the opinions of those who are trying to bully you.
If I cared too much about what others think-- when they do not have my best interests at heart-- my liberty would be destroyed. One person at a time, they'll destroy everyone's liberty. Don't let them. Stop them at your liberty, and stop them if you see them targeting someone else. Refusing to be bullied is not the same as imposing your will on them.
