Adding tests while refactoring in test driven development

Let's say that I am refactoring some classes that have unit tests already written. Let's assume that the test coverage is reasonable in the sense that it covers most use cases.

While refactoring I change some implementations. Move some variables, add/remove a few, abstract out things into some functions etc. The api of the classes and it's function has remained the same though.

Should I be adding tests when I am refactoring these classes? Or should I add a new test for every bit of refactoring? This is what I usually do when I am building up code rather than refactoring.

PS: Apologies if this is really vague.