No. The gravity of the Earth doesn't do any work on the probe. It's a coupling mechanism. The work that is done is by Earth's orbital energy around the Sun. That is the energy that is less after the spacecraft encounters the Earth than before. The Earth is "dragging" the spacecraft along with it around the Sun. Not trebucheting it.
The gravity of the Earth absolutely does net work done on the probe. That’s an unambiguous mathematical statement. It’s just that this work is frame dependent. It sounds like you want it to frame-independent and to be given by the value in the center-of-mass frame. But it’s not.
Respectfully, I have a background in aerospace engineering. This is the number one popular misconception about gravity assists/slingshot maneuvres.