I'm not sure what approach you've described here, but it isn't TDD. In the case of adding new features to existing code, as you are continually running tests you will know straight away which you have broken. At this point you would fix them so you get all green again before continuing. In this way you incrementally modify the codebase. Remember unit tests are quite simple 'Arrange, Act, Assert' code pieces, so refactoring them is not a time sink.
Also some refactorings are easier with tests, some are harder.
The kind @viraptor mentiones is the kind that spans more than one compoment. For example when you decide that a certain piece of logic was in the wrong place.
The kind of refactoring that becomes easier is when you don't need to change the (public) API of a component.
Take for example the bowling kata. If you want to support spares and strikes and you need extra bookkeeping, that's the easy kind of refactor where your tests will help you.
But if so far you have written your tests to support a single player and now you want to support two players who play frame by frame... Now you can throw away all the tests that affect more than the very first frame. (yes in the case of the bowling kata, you can design with multiple players in mind, but that's a lot harder in the real world when those requirements are not known yet)
