I'm not so sure I agree. I think it is saying "if you have to think hard, and can only come up with one counterexample, there is pretty much a rule"

It isn't for things like provable facts, it is for things like "only men win best director oscars". If you gotta dig to come up with a couple counterexamples the statement is pretty much right.

This is how I've always understood it a well. Additionally, often the nature of the given exception shows what great lengths one must go to in order to break the rule.

Example: "I never run into traffic on my way home. Except for that one time when the bridge collapsed."

In this case, the "exception" is not an exception to the rule. It's an exception of the object to which the rule applies.

Let's say we have a rule that is said to be valid for all mammals. To prove the rule ("make it more likely" to be strict) we should check it for exceptional mammals like the whale (a huge mammal that lives in the water). When the rule applies for these cases, the rule is more likely to be true cause people that make rules tend to overlook exceptional/extreme cases.

Perhaps it means that it is impossible to create a rule that applies to a category without exception, real life is much more nuanced than that. So if you think you have such a rule, that is always valid, as people often do, we know it's a bad rule. A rule only becomes a good rule once the exception is found because that means it is a rule that has a hope of succeeding in an ever evolving real life.

I think Nassim Taleb's Black Swan book gives examples of the exception proving the rule a few times. Unfortunately I don't have the book to hand to provide a direct quote.

prove used to mean test. Hence, that's the exception which tests the rule.

in fact, that meaning survives in the phrase "proving ground"

This is a masterful parody of sophistry.

There are no even prime numbers, except for 2.

And 2 is very odd.