Though I personally feel this would be perfectly acceptable, the speaker might come off sounding like a jerk to some people.
I have the same problem when teaching class, and I usually say, "That's a good question, but talk to me about it after class." Switch "class" to "the talk," and you can probably get the same effect.
A designated asshole is nearly mandatory, especially for mixed technical audiences and issues where someone can have an [opinion on the bike shed](http://en.wikipedia.org/wiki/Parkinsons_Law_of_Triviality).
Most people can't fill this role because they don't like conflict or have a reputation to protect (real or imaginary) but conferences should always assign a moderator to play the jerk.
I suggest the moderator "take the bullet" and make a statement similar to this blog and cut out questions that don't follow the rules.