I couldn't agree more, I can't imagine how much more I could have picked up at that young age if I'd just had, say, a tty, instead of endlessly playing with window themes, notification sounds and so on.
Another thing that's important though, you need to be able to make the computer do fun things pretty easily, otherwise you'll (as a kid) loose interest.
Hmm... as a kid, I had an 8086 with DOS, GWBASIC and a white manual for both. Nothing magical came out of it :) , my best creation was a "Copa America" text-based game, full of GOTOs.
I still program in (Visual) Basic (6 and .NET) 20 years later :P though I try not to, and I swore when I finished university that I wouldn't do it again - when my boss told me to, I caved pretty fast :)
Another thing that's important though, you need to be able to make the computer do fun things pretty easily, otherwise you'll (as a kid) loose interest.