I'm currently building a test framework for my graphics engine, and have run into the exact problems described in this post. A graphics engine, by design, must be able to do millions of different things that are created by the interaction between relatively few methods, none of which can go wrong at any time. One of my new features is probably buggy, but attempting to brute force test border conditions requires tens of thousands of tests because of all the interacting elements, any two of which could interact in just the wrong way to blow something up. It gets even more ridiculous when you have precision bugs, where only certain numbers in certain ranges in certain cases will explode. Testing that using inputs is impossible.
This occurs so often and with such regularity I am now convinced that everything I write is riddled with bugs that I will probably never find without beta-testing in real-world scenarios, plus many more bugs I will simply never find because they never come up.
A much better design would be a test platform that analyzes the assembly structure to pinpoint actual edge cases in the code itself, which could then be used as a guide for finding bugs instead of relying on hundreds of thousands of test cases.
This was featured recently in Bryan O'Sullivan's "Running a startup on Haskell" presentation: "[QuickCheck] generates random data, then tells me if all properties that I expect to be true of my functions and data hold. QuickCheck is shockingly more effective at finding bugs than unit tests."
There are ports of QuickCheck to tons of other languages...
This gets complicated in a graphics engine where the only place you can tell something has gone wrong is after the image shows up in the wrong place. It would help a lot for the majority of cases, though.
Depending on the language you're using, there might be some tools to help you.
For example, MS recently released a library for Design by Contract (aka Contract Programming) in .NET 4.0. It allows you to specify constraints on your methods, such as things you expect to be true before the method is called, after, and between all method calls in a class. The library is capable of static verification, but it's partly an experimental feature and you need to pay for expensive version of VS. (Runtime checking is available for free.)
But, here is the cool part: MS also released an automatic test generator called PEX. It can do exactly what you've described - go into your code and automatically find edge cases, and generate tests that cover each of them. And it's free.
So, you can write contracts, run PEX, and if something goes wrong, you will see which inputs generated exceptions.
D also has DbC functionality. I don't think it runs any static verification on it yet, but you can use it to detect abnormalities during functional testing.
DbC doesn't need to be dealt with at every level of the call chain if that's what you mean.
It may seem a bit verbose, but it usually expresses logic that would be in your program or in your tests anyway. Difference is, you will be doing it in a pretty terse and declarative manner. IMO, DbC is one of the coolest features of .NET 4.0.
If you have interactions between relatively few methods, then unit-testing those interactions shouldn't be too much of a problem. You wouldn't test all possible inputs, but focus on making sure that you have a test-case for each input that needs to be handled differently. Bugs will still creep through, but they will be substantially fewer.
If you really do need tens of thousands of tests because you have so much interaction with unique states going on in your engine then you will probably never ship a product with any reasonable quality.
And that floating point operations lacks precision is just the way things are, and you need to understand how to write stable code despite it if you want a stable graphics engine.
This occurs so often and with such regularity I am now convinced that everything I write is riddled with bugs that I will probably never find without beta-testing in real-world scenarios, plus many more bugs I will simply never find because they never come up.
A much better design would be a test platform that analyzes the assembly structure to pinpoint actual edge cases in the code itself, which could then be used as a guide for finding bugs instead of relying on hundreds of thousands of test cases.