What I've come to appreciate is that freedom of association is a prerequisite of freedom of speech.

You can't meaningfully exercise freedom of speech when shouted down by a mob or drowned out by a megaphone. The power to form associations which exclude those who won't give you a chance to be heard is fundamental.