I recall seeing a graph of a stock index in which all but the 10 (e.g.) best days within a very long time span (~years) were removed - to mock up a trader who removed his money at a few unlucky times. You ended up losing huge.
The lesson was exactly what you're saying -- you also have to count that one huge loss. You can't fence it off as an exception and remove it from the accounting.
Yep, I tried to find the graph but failed. It would be interesting to see again. I have a professional interest in probability and stochastic processes, so it stuck in my mind.