> Wouldn't Earth's gravity take away when departing just as much as given when arriving?

Only in the Earth’s center-of-mass frame. In the solar system rest frame, the probe leaves with a different speed than it entered with.

> However, the probe's direction could change based on how close it passes Earth.

Both the magnitude and the direction of the velocity vector change.