Then all you get is people asking why they were downvoted. IMO downvoting should include a comment if one doesn't already exist sufficiently explaining your reason for downvoting.

That doesn't mean it needs to turn into a discussion if they defend themselves, but I think some indication of why is almost always warranted.