OPINION: When Senator George Brandis said that we have a right to be a bigot, he was in one sense right. He did not (I assume) mean that it is right to be a bigot, but that in liberal society we ought all to have freedom of conscience - that is, we have a right to think whatever we like no matter how horrible it is. So Bill Shorten's and many others' attacks on him involved either misunderstanding or deliberately misinterpreting his comments in the Senate on Monday night.

Where Brandis is wrong, or at least unjustifiably selective, is on what follows from this right. In his attempt to repeal section 18C of the Racial Discrimination Act, Brandis is using this ''right to be a bigot'' to allow people who have bigoted views to express them in speech. This is the common justification of free speech: freedom of conscience is meaningless without being able to act on it in some way.

So far so good. The problem, and I'm certainly not the first person to say this, is that while we think conscience should be entirely free, actions that we take on behalf of our conscience are an entirely different matter. These actions affect other people, and thus their interests need also to be considered before giving carte blanche to free speech.

Of course, and here Brandis is also right, mere offence is not normally sufficient to limit one's free speech. But it is here that Brandis is on some very shaky ground and appears to have been particularly selective in reading from his well-known library of political theory. There are two types of arguments that resist Brandis' insistence that if you are a bigot you have a right to say whatever you think in the public sphere.

To understand racist slurs and insults as causing ''mere offence'' is to deeply and tragically misunderstand the nature of racism. Racism further marginalises the most vulnerable in our community, and undermines any attempts to achieve social equality. They may well be offended and insulted, but there is a deeper harm here.

The second argument is entirely separate. While much of the first argument relies on a particular understanding of what a good society is, and on a particular notion of harm, this second argument simply points out the inconsistencies in arguing for free speech in one domain but not others.

If, as Brandis and Tim Wilson (following JS Mill) believe, free speech is important in a society for the discovery of truth and for the intellectual and moral development of its citizens, then why pick on section 18C? Unlike the United States, speech in Australia is very limited. Even if I conscientiously believe (without sufficient evidence) that a particular individual or corporation is, for example, polluting a waterway, I'm not free to simply say it publicly. And if I decide to publish a doctored picture of somebody having sex with an animal, I'm going to need a very good lawyer.

There are countless examples of how our speech is limited in Australia. I'm not saying that there is something wrong with these restrictions. But if expressing our conscience is so important in a free society, why is it only bigots who have this right? Surely, if we want a more vigorous intellectual climate and to uncover and maintain truth, then allowing all of us to regularly speak our mind in public would be a good thing, and it would seem there are much more important laws that Brandis should be looking at repealing.

But wait. What would happen if we removed all these protections? People would risk having their lives disrupted, their reputations besmirched, their businesses fail, all because of somebody's right to express whatever wacky or ill-founded view they might hold.

Of course, in this world of free speech you can talk back. But once you've been labelled a paedophile, a wife beater or a philanderer there is the small problem of ''the more one doth protest…'' And all this is assuming that you have the power (and resources) for your voice to be heard.

Balancing free speech is a difficult task, and I do not envy legislators on this matter. And, to be honest, I'm still not sure where I stand. But something seems more than a bit fishy about focusing all our free speech efforts on eradicating the quite weak protection that exists for some of the most vulnerable people in our society, and leaving the rest of us, the rich and powerful, fully protected.

Dr Peter Balint is a lecturer in politics at UNSW Canberra. 

This opinion piece was first published in The Age.