Working with your mind in a stress free environment will make you old. Mining coal, crawling around in asbestos, being pushed by a never ending competition or sitting still doing nothing won't.

So there are upsides and down sides to work/retirement thing. The best you can do is academia or a retirement spent learning new things.