I can't help but think of how simple things like injuries and aging affect everyone and how driving practically solves these problems.

You want to ride, go ahead. He wants to drive, that's fine. Neither one is more right or wrong than the other. Stop acting like driving is morally wrong (if you're not, many are).