My recent shock at running into an openly (though self-deluded) racist blogger was an eye-opener. After getting involved in libertarianism and anarchism I had forgotten that such people existed. Everyone has a "right to be wrong"about how they think. That is not the issue. The issue is when you begin advocating actions and trying to spread your hatred based on those wrong ideas. Racism is wrong. Treating someone as a second-class person because of their race is wrong. Giving anyone "special perks" based on their race is wrong. Making an issue of race at all is ridiculous; just as it is to make an issue of sex or sexual orientation , or which side of an imaginary national border you were born on.
Based as it is on absolute self-ownership and non-aggression, libertarianism can not act in racist ways without violating its own core values. You can't be fighting for freedom if you are making an issue of someone's race. Authoritarianism, however, based as it is on a small group controlling the rest of society, thrives on racism. It helps to keep people angry, resentful, suspicious, and hateful if you want to be able to pass endless laws against the "them" of either side. Therefore they endlessly fan the flames of racism, seeing it everywhere except in their own behaviour. Don't empower authoritarians by falling for this trick. Judge people on their actions. That is what counts, not silly things like how much melanin is in the outer layers of their skin. If you want to hate me, hate me for the choices I make and the ideas I hold, not my genetic makeup.
Everyone has the right to associate with whoever they want for any reason. It is not within governmental authority to control this. Personal responsibility and Darwinian selection will be much more effective in the long run, anyway. Don't expect others to treat you fairly or rise to defend you if you are hateful to any particular genetic group. If there are people out there calling themselves "libertarians" while advocating racist views, they are doing harm to the cause of liberty. They can not really be libertarians. If this is you, please do us all a favor and admit you are an authoritarian. Liberty has no place for your sickness.
