I'm a person who thinks very deeply about stuff and that often includes latching on to something that was either said or I thought of during a conversation and then zoning out and thinking about that (just for context).

When I was younger I was single and almost never had passengers in my car. I rarely talked on my phone (before we were forced to be hands free) but occasionally I would and I'd say overall did not find having it in my hand a particular distraction (honestly what would that have to do with anything).

Anyway, I remember when "bans" came into effect and put my phone on bluetooth. I was talking to my mom on the speakerphone in the car, and suddenly realized I'd gone through a red light, and realized I'd better hang up and drive.

Tldr, for me anyway it's being on the phone that's distracting because it, as mentioned above, gets me thinking about other stuff, and I actually find talking to the air more confusing than either phone to ear or a person in the car. Though I routinely tell my wife to stop talking when I'm driving in traffic (I don't mean this as a joke) because I find it too distracting to be involved in conversation when I need to think about driving. Obviously I'm a shitty multitasker but I suspect this is true to some extent for most people.