I've listened to audio books in my car for years now, and the worst thing that happens is that I'll fail to listen to the audio for a while if anything important happens while driving.

It has never, ever happened the other way.