For me, you nailed it. Maybe how I feel on this will change over time, yet at the moment (and since the movie Her), I feel a deep unsettling, creeped out, disgusted feeling at hearing a computer pretend to be a human. I also have never used Siri or Alexa. At least with those, they sound robotic and not like a human. I watched a video of an interview with an AI Reed Hastings and had a similar creeped out feeling. It's almost as if I want a human to be a human and a computer to be a computer. I wonder if I would feel the same way if a dog started speaking to me in English and sounded like my deceased grandmother or a woman who I found very attractive. Or how I'd feel if this tech was used in videogames or something where I don't think it's real life. I don't really know how to put it into words, maybe just uncanny valley.
Yea, gives that con artist vibe. "I'm sorry, I can't help you with that." But you're not sorry, you don't feel guilt. I think in the video it even asked "how are you feeling" and it replied, which creeped me out. The computer is not feeling. Maybe if it said, "my battery is a bit warm right now I should turn on my fan" or "I worry that my battery will die" then I'd trust it more. Give me computer emotions, not human emotions.
What creeps me out is that this is clearly being done deliberately. They know the computer is not feeling. But they very much want to present it as if it is.
From a tech standpoint, I admire its ability to replicate tone and such on the fly. I just don't know how it'll do from a user experience standpoint. Many stories of fascinating tech achievements that morphed a lot to be digestible by us humans.
"All the doors in this spacecraft have a cheerful and sunny disposition. It is their pleasure to open for you and their satisfaction to close again with the knowledge of a job well done"!
