One of the costs of saying no to meetings is that going to other people's (useless) meetings is a super low effort way to say "I value our working relationship." Not going often explicitly sends the opposite message.

Sometimes there is a whole set of rituals used to "prove" you actually care about the group, and the rituals only ever happen in meetings, and you cannot change them without bothering a lot of people.