The problem with 'something like SDL, but 3D' very quickly turns into a full blown engine. There's just such a combinatorial explosion of different ways to do things in 3D compared to 2D that 3D 'game engine' is either limiting or complicated.

OpenGL was designed as a way to more or less do that and it turned complicated fast.