Probably one of my harder ones was finding a memory leak in one of my servers. Didn't happen often, never in development and only leaked a fraction of a byte per request. Turns out it was a bug in the logger, where if on rolling the log and it couldn't open a new file, the logger would just queue up messages. Was a real pain to track down as we normally had plenty of disk for logs.
The fraction of a byte was amortorized over the number of requests (e.g. the mean bytes lost per total requests), which is what made it puzzling and hard to track down. It only happened under specific conditions that were not evident running under valgrind.
