Just sensor data to replay the scenario as fully as possible should be sufficient. Whatever it looks like to humans, there's clearly something in there that can be difficult for AI systems and so everyone should be using it as a regression test.