It's interesting to me how many times I find it necessary to remind people that not allowing myself to be controlled is not the same as me controlling the person who wants to control me.
Refusing to be used by you doesn't violate your rights. You have no right to use others.
If you have a mutual understanding that you'll do some specific thing for someone, that's not what I'm talking about when I say "being used". That's just being helpful or useful, voluntarily of your own free will. Consent is the key.
As much as I hate mowing and yard work, I do it for my parents to be helpful. I don't resent it or feel like I am being used. They've done things for me, too.
Being used can create resentment and devalue you. It's akin to slavery. Being useful can make you feel good, even if it's not particularly enjoyable in and of itself.
Government employees don't get this distinction (almost by definition), but many freelancers also don't get it. Governing others is just another way of using others; of controlling them in ways no one has a right to do. It's not exclusive to bureaucrats and other political criminals.
Refusing to be controlled, in this case, can be a valid form of self-defense. The control freaks won't like it.