This is close, but I think it's more than an instinct: it was a philosophical challenge. Math is part of a larger project to formalize thought, and "we have a bunch of tools for a bunch of different things that work in different ways" is a lot less metaphysically/intuitively satisfying than "we have a single cohesive system of formalization that can be applied to any system of quantities and qualities found in human experience, all based on the same solid foundation."
The present work has two main objects.
One of these, the proof that all pure mathematics deals exclusively with concepts definable in terms of a very small number of fundamental concepts, and that all its propositions are deducible from a very small number of fundamental logical principles... will be established by strict symbolic reasoning...
The other object of this work... is the explanation of the fundamental concepts which mathematics accepts as indefinable. This is a purely philosophical task.
Well yes, but it takes a certain kind of instinct to take on that philosophical challenge, and again to guide your particular angles of attack on it. I took the question to be, within the framework of formalized math, how do people arrive at particular abstractions like groups? How do you pick the axioms and rules?
This challenge/goal was best expressed by Bertrand Russell and Alfred Whitehead, I think: https://plato.stanford.edu/entries/principia-mathematica/