This premise is incorrect: SSH doesn't need to be an suid binary because it's already running as root, and then SSH creates a new environment for the user, exactly like sudo does, but with all the added complexity and overhead (and surface) of privileged network access.
To be clear, I love SSH and we even run a userify instance to distribute keys, but juts comparatively the surface area of the ssh daemon alone is greater than sudo alone.
(however, even with the extra complexity, you might trust the history of portable OpenSSH more than sudo, and that's a good, but different, conversation to have also.)
But the area under control by the invoking user is data over one socket vs the whole calling environment e.g. environment vars, local files. Surely that counts for something.
To be clear, I love SSH and we even run a userify instance to distribute keys, but juts comparatively the surface area of the ssh daemon alone is greater than sudo alone.
(however, even with the extra complexity, you might trust the history of portable OpenSSH more than sudo, and that's a good, but different, conversation to have also.)