I've been coding for 50 years. When I write code, I think it is great work. About five years later, I realize it was crap. This is true of all the code I write.
So, about five years later is the right time for refactoring.
P.S. It takes about five years to forget what you thought you were doing with that code, and see the reality of what you wrote.