I think if I understand the article correctly it sounds like Google might also be reading the messages so it can respond for you. Regardless I think the other thing people might not be happy about is Gemini can still interact with apps regardless of if you have app activity turned on or off, as quoted from the linked email in the article:
What's changing
Gemini will soon be able to help you use Phone, Messages, WhatsApp, and Utilities on your phone, whether your Gemini Apps Activity is on or off
Google's own documentation explicitly states it cannot read your messages or notifications. You can ask it to compose a message for you or start a call, though.
