This line of thinking is almost always a less-than-useful simplification.
Should punching Richard Spencer be punishable by the law?
Yes.
Is punching Richard Spencer "moral"?
No.
"Should" you punch Richard Spencer?
No.
Is punching Richard Spencer "understandable"?
... this is grey. However, if you have a sufficient ideological motivation in a certain direction, the argument essentially makes itself: Violence! It's bad. If you're not motivated in the opposite direction, it's the easiest thing in the world to stop there. If you're motivated enough, you can tack on some extra easy hops in logic: "You're advocating violence! You're evil!"
Yet if somebody drives a car into a group of fellow citizens whom you know or identify with, killing one of them, and a news outlet puts out a story calling the dead a whore, you'll quickly find yourself willing to meditate on the scenario more thoroughly.
No? I'm not sure why you're advocating violence. If we start allowing violence against bad people, where do we stop?