Summary
A large body of unit tests as a regression test system is important to many organizations. What do you do when your framework of choice dies or is neglected? When is it time to move on and how do you budget for abandoning or rewriting tests? Have you ever thought about being caught by this situation?
Advertisement
After several years in other languages, I've currently got two projects in C or C++ with a unit test requirement. One code migration and major refactoring presents an opportunity to add tests. The other project needs updating from Visual Studio.Net 2003 and already has a couple of hundred tests which I expect to keep running.
This is the kind of situation many people will encounter if there is a need to maintain a legacy project, especially a mature one which has worked with little maintenance for several years.
My first step was to go and get the current version of cppunit but with some dismay found that the projects included were last used with Visual Studio v6, don't build in VS.Net 2008 and the project admin has gone AWOL - emails bounce (Update: this bad impression is courtesy of a bug with sourceforge mail handling - I've since been contacted and told that webmail through the contact form works and the project is, just barely, alive).
Whilst there is a range of alternatives and debate over which C++ framework to use, due to my familiarity with its patterns and amount of legacy code with tests I am hesitant to migrate.
Thinking about the issue of migration made me wonder - do people ever plan for their testing framework going away?
Hopefully, the bulk of the test bodies would remain the same if you had to migrate, possibly with changes to the assert macros. However, I expect that the registration of tests, establishment of fixtures and combining tests into suite would all change significantly. These seem to be the areas of most dispute in how to design test frameworks :-)
Would you migrate to a new framework, or would you assume the burden of maintaining the old test framework as well as your code?
Our system also has many tests written using cppunit, but also many other types of tests (for example, regression tests, domain specific tests...). I don't see a problem with using multiple testing frameworks; the existing tests aren't broken, aren't inhibiting the development and refactoring of the main source code, but still provide value.
If cppunit no longer does what you need, and another framework does, just use both; by their very nature unit tests don't need to work together. So long as your build script is capable of launching both types (and interpreting the results), that's good enough.
In the specific case of cppunit, we want to move on from Visual Studio.Net 2003 and the original version no longer compiles.
This raises an interesting issue about the robustness of a framework over time. If you have a GUI, that's probably the fastest-decaying and most vulnerable part of a test framework. So, whilst I value being able to pick and choose subsets of tests when working in a Test-Driven style, for robust testing it makes sense to have the simplest possible non-GUI framework.
Bringing things back to a Unix-style pipeline architecture allows a build script to run tests and interpret results and mix tests written in a variety of languages, provided they can be executed and generate textual output. Whether you then wrap that in a GUI or use a simple build script can be made a separate decision. One corollary for cppunit is that it makes sense to have a simple command-line build even for Windows with the GUI test-runners decoupled so they can evolve separately.
I see projects get bitten by this occasionally. Their unit tests are tightly connected to the particular framework, their code expects a certain build tool or even version control tool. It's one of the arguments against building using an IDE, since not everyone uses the same one.
I just bear it in mind when using *any* tool these days and check both import and export options when evaluating.
Cppunit is already quite portable (windows / posix, with the core functionality available in command line programs), in part due to its simplicity (which, as you say, increases robustness). When weighing the likely low cost of fixing / porting cppunit against the known large cost of porting to a new framework, or the loss in 'features' by cutting the unit tests, I fall on the side of maintaining the existing test framework.
This is a specific instance that we've dealt with recently, made easier for us by the fact that we've already made some custom modifications to cppunit. It is unfortunate however that the upstream project seems to be unmaintained.
I agree in the general sense though, as even a compiler version change can be seen as a change in platform; sometimes incompatible features have to be cut. But planning for the testing framework itself going away; no, not a situation we've dealt with.
I think that for many organizations, the best thing to do with something as simple as a UT framework is to fork internally. Sure, you don't get updates from the outside, but at least you've taken responsibility and you aren't held hostage by fear of the original developers abandoning it.
Previous Topic
