"Because we're familiar with humans. Most humans don't want to hurt/kill other humans, nor do they have the power to do that on a large scale."
Well, large scale mass murder is hardly unknown in human history. The best one can say is that most citizens in times of genocide were just following orders. So most people being supposedly nice isn't really that relevant to the problem of neutral AI that does what ill-intentioned humans tell it.
But I would offer the counter-point that we avoid this problem because it shades to all the problems we see and fail to deal with right now. The advance of technology isn't serving many people now and an order where a few people control more and more powerful things seems to continue this.
Basically, it's easier to talk about the elephant that might lumber into our living room sometime later than it is to talk about the elephant that's already crushed the couch.
Well, large scale mass murder is hardly unknown in human history. The best one can say is that most citizens in times of genocide were just following orders. So most people being supposedly nice isn't really that relevant to the problem of neutral AI that does what ill-intentioned humans tell it.
But I would offer the counter-point that we avoid this problem because it shades to all the problems we see and fail to deal with right now. The advance of technology isn't serving many people now and an order where a few people control more and more powerful things seems to continue this.
Basically, it's easier to talk about the elephant that might lumber into our living room sometime later than it is to talk about the elephant that's already crushed the couch.