The "Zero Aggression Principle"
"No human being has the right - under any circumstances - to initiate force against another human being, nor to threaten or delegate its initiation."
This version of The Zero Aggression Principle, formulated by L. Neil Smith, is generally (but not universally) agreed to be the core principle of libertarian philosophy. Personally, I DO think this is the foundation of libertarianism. This is how you show your respect for the self-ownership of those around you. If you follow this principle, you may not be a perfect person, but you would probably be a pretty good neighbor. You would definitely be a good example of true libertarianism.
The ZAP has the same message as The Golden Rule and most other guides for dealing ethically with others; each culture has its own way of saying basically the same thing. I have heard the argument that "initiating force" can be defined any way the person wishes to define it. I do not believe this. Even small children understand the concept of "he started it!" Someone calling you a nasty name has not initiated force; someone pointing a gun at you has. Only someone physically attacking you or making a credible threat against you has initiated force. I don't see that it is a difficult concept to grasp. Once force has been initiated, you have the right to counter that force with defensive actions, including force.
You may have a moral obligation to use an appropriate amount of force. In other words, if someone shoves you, you can't justify beating that person to death with a statuette of Gandhi. In most common situations, you would be smart to simply walk away. This is not always an option. You might need to point a gun at that person and warn them to leave or be shot. At that point, they have a choice to escalate the situation, or leave.
Some people may claim that this is "Utopian" but I know it works, for real, in everyday life. I have never run across a situation where it failed to provide the proper perspective in dealing with others. If you don't want to accept it, you can sit around and formulate all sorts of "what if" scenarios that you will probably never face. That just shows me that you have a desire to keep open the option of attacking someone you don't like, even if they have not attacked you first. That is a sign that you may not be a trustworthy, or nice, person.