This was pretty painful. Their backups were too old, so it was necessary to do InnoDB data recovery rather than a straight-forward restore from backup.

Since InnoDB table spaces never shrink, 80G of their truncated data was still all available in a single monolithic ibdata file. An InnoDB recovery tool named page_parser read their 80G ibdata file and spits out a maze of 16k InnoDB page files organized by an arbitrary index id.

There are two internal InnoDB meta tables called SYS_INDEXES and SYS_TABLES which can give you a mapping from table name to PK index id. Unfortunately after the mass TRUNCATE all the tables got new index mappings, so it became a bit of table hide-and-seek.

The InnoDB recovery tools lack a certain polish and maturity. You need to create a C header file for each table you want recover from the pile of 16k page files. You end up having to build a separate version of the constraints_parser binary for each table. There were bugs with the output of negative numbers, unicode handling, VARCHAR types with >= 127 characters, and some edge cases where InnoDB chooses to store CHAR(15) types as VARCHARs internally. Aleksander at Percona really saved the day, he was able to find and fix these bugs pretty quickly.

I remember that magic moment when I finally was able to successfully checksum a huge block of the recovered data against the too-old-to-be-useful backup.

"I love the smell of CRC32 in the morning. It smells like... victory."