This is generally true, but policy is hard to define and individual cases are easy to have an opinion about. Specifically, what you're describing is sometimes literally impossible, because society is zero-sum. When situations between people are at odds due to no direct fault of either party, it's unreasonable to expect people to cede their own interests to those of others. Of course there's a limit - you shouldn't oppose nearby housing if it would help people in need and wouldn't hurt you in any appreciable way, obviously. Point is, it's subjective.

Also, I would suggest closely analyzing your thoughts and feelings when you run into a situation where you are broadly considering some category of people "morons or evil". Such categories obviously exist, but when you see them in the real world, the picture is often more complex than you might be willing to admit at first.