Martin Fowler writes in his blog under RefactoringMalapropism, [the term] "refactoring is often used when it's not appropriate"
Surprisingly soon after the introduction of the practice, there was widespread and gratuitous misuse of the term at a former employer. I was one of the first users of the term and Fowler's book there, and I began to discuss and demonstrate the techniques to my fellow programmers.
For some reason, a few managers got the idea that doing refactoring was extra work. This kind of work, they believed, reduced productivity and was not useful as part of maintenance programming.
This mistaken understanding spread widely, at least in the dysfunctional management groups. At one point a project manager misused it in a directive outlining the goals for the maintenance of one system. Among other things, this directive said that anything beyond the specific changes requested was a "refactoring effort" and therefore out of scope.
Sigh.
After that, "refactoring" started to be synonymous with any change.