I've always assumed part of it is more relative. Meaning: When I'm 5, 10 years old seems like a lifetime away... because it is. Going from 5 to 6 is, relatively speaking, the same as going from 50 to 60. I assume my brain has so much to look back on as I'm older to compare against that 1 year just "feels" like less now.

With that said, I'm sure there's a million explanations. And all of them are probably, at least somewhat, true.