It'd be weird and very abusable if free speech ever implied the right to a venue.

If they couldn't stop people talking at their venue what's to stop someone completely sabotaging their agenda?