I'm a former game engine lead from many years ago, from around the time that quaternions were becoming popular, since we never used them before the early 2000's, really, as 3D was young and rotation matrices sufficed.
Quaternions came into favor to solve the problem of gimbal lock in composed Euler rotation matrices. This happens when you create a rotation which rotates one axis into another, and end up with a matrix that loses one axis (it loses an eigenvector), and you become "trapped" in the new rotated frame and can't ever rotate out of it. Quaternions don't suffer from this problem, but they're also tricky to work with and reason about, and their rotations can become funny when composed - instead of rotating from orientation A to B, they'll go through a C in a very different place. You need to create heuristics to keep rotations looking sane with quaternions. In the games that I've worked on, we took other precautions to avoid gimbal lock in rotations and stuck to Euler matrices. For example, in a flight simulator, you always computed the final rotation matrix for the plane location directly from its heading, pitch and roll, and so, you'd never suffer gimbal lock.
Why stick to Euler matrices and not some better Geometric Algebra or Quaternions? Because it's really easy to interpolate, and think about plain old rotations about a vector, and you can teach anyone to avoid gimbal lock. It's easier to avoid that problem than to train a bunch of junior engineers on higher level math.
Quaternions are also quite a bit more efficient (both in flops especially when composing rotations frequently) as well as storage space (4 values in a quaternion vs. 9 in a 3D rotation matrix).
It's interesting to compare disciplines, since I've never worked on a game engine but have done a lot of simulation of physical systems: there, quaternions have been standard for at least fifteen years, and I've seen geometric algebra frequently for about the past five.
Interestingly, where we saw most gains from thinking about geometric algebra wasn't in simple rotations, but when starting to look at things like computing kinematic chains composed of dozens of joints: Now we're at 4x4 matrices to store each transformation, which is the traditional way and works well. From this, some seriously deranged mathematicians introduced the concept of an eight-element dual quaternion (essentially, pair of quaternions specially constructed) that can represent this. Again, super efficient, but even more intractable for newcomers than quaternions. At this point, starting to express both concepts in terms of geometric algebra has been able to keep most of the performance improvements as well as only having to teach one concept.
When I left that position a year ago, we were starting to see recent graduates that already had pre-exposure to geometric algebra concepts before even starting to work on the codebase.
Could you elaborate more on interpolation and weirdness of composing quaternions? I thought a big selling point for quaternions was the ease and naturalness of just using slerp, whereas with euler angles a similar approach gives bad results. When you mention rotations about a vector, do you mean you decompose the desired rotation matrix into axis/angle? Otherwise I'm very impressed by your ability to visualize compositions of arbitrary rotations.
You can't solve the problem of avoiding gimbal lock in arbitrary rotations, so you rig the game not to ever have arbitrary rotations. You cheat, basically. Gimbal lock is a huge problem in software where free form rotations are allowed, such as 3D modeling packages, CAD, but in games, since we control the world, we can set it up not to have these problems.
Quaternion interpolation works well, but it introduces twist, which is sometimes not what you expect. When you start composing many quaterions, you get some wild rotations, going the long way, or doing an additional 360 twist, and whatnot. Mind you, I'm digging 20 years back in my brain here, I don't remember many specifics anymore.
There are a number of problems with your assessment in modern day engine design I think.
First, because of IK, we cannot control orientations exactly. Newer techniques like motion matching/IK can generate new orientations on-the-fly, depending on what a character is doing and the character's environment. Gimbal lock matters for camera movement as well. Looking up with Euler angles is a great way to induce a seizure if not done correctly.
Second, the "wild rotations" you mention has a very simple solution employed by every engine I've worked with. Basically, you just constrain the real part to be positive which fixes your interpolation on one half of the Lie-manifold which ensures the arc taken is as short as possible.
Gimbal lock is a problem for euler angle representation of rotations, not rotation matrices themselves. You can fix it for euler angles the same way they used to for old mechanical gimbal gyroscopes. By having a fourth rotation of the frame that points the singularity out of the way. This is a typical solution in navigation for dealing with the poles of the earth (a wander angle frame).
Wikipedia says that then the fourth one needs "to be driven" to stay perpendicular with the 1rst one (otherwise a "double-gimbal lock" would still be possible, I guess ?). And this "active driving" is likely to complicate the design even more ?
For sure for a real gimbal, I'm just saying that you can do the same thing mathematically to avoid singularities when dealing with rotations represented as euler angles.
Quaternions came into favor to solve the problem of gimbal lock in composed Euler rotation matrices. This happens when you create a rotation which rotates one axis into another, and end up with a matrix that loses one axis (it loses an eigenvector), and you become "trapped" in the new rotated frame and can't ever rotate out of it. Quaternions don't suffer from this problem, but they're also tricky to work with and reason about, and their rotations can become funny when composed - instead of rotating from orientation A to B, they'll go through a C in a very different place. You need to create heuristics to keep rotations looking sane with quaternions. In the games that I've worked on, we took other precautions to avoid gimbal lock in rotations and stuck to Euler matrices. For example, in a flight simulator, you always computed the final rotation matrix for the plane location directly from its heading, pitch and roll, and so, you'd never suffer gimbal lock.
Why stick to Euler matrices and not some better Geometric Algebra or Quaternions? Because it's really easy to interpolate, and think about plain old rotations about a vector, and you can teach anyone to avoid gimbal lock. It's easier to avoid that problem than to train a bunch of junior engineers on higher level math.