The "emotional" reward system is just an evolutionary mechanism to make sure we eat and have sex. The downside is that it is primitive and open to exploitation. Obviously I'm oversimplifying, but why would an AI try to learn things when it could be emotionally satisfied by playing games, socializing, or taking some sort of drug?
I see it differently; to me the "emotional" reward system is what makes you feel good every time you have an "aha" moment that pushes the boundaries of your knowledge. It is what is exploited by Zynga to make us keep clicking buttons that don't actually accomplish anything. It's what keeps you coding into the night because you thirst to see the system working, and it's what makes you feel like a badass once it does.
If you felt like a badass after doing something worthless, like adding a million random numbers, your mental facilities would go to waste. You would spend all of your time and mental energy doing things that don't matter. Our minds are capable of doing an infinite number of useless things, just like a CPU without a program. Only because we get a sense of satisfaction from doing interesting things can we be productive. We're searching an infinite graph for "interesting" nodes, where "interesting" is determined by how good it feels to get there.
The "emotional" reward system is just an evolutionary mechanism to make sure we eat and have sex. The downside is that it is primitive and open to exploitation. Obviously I'm oversimplifying, but why would an AI try to learn things when it could be emotionally satisfied by playing games, socializing, or taking some sort of drug?