Summary
Think of your tests at too course a granularity level, and you may lose key benefits of test-driven development, according to Andrew Glover's recent IBM developerWorks article. Distinguishing between system, component, and unit tests, Glover outlines a strategy that lets JUnit run just the tests you need.
Over a project's lifetime, developers often accumulate thousands of unit tests, making it unrealistic to run them all at every checkin, let alone at every build. If those tests are not carefully categorized, according to Glover, developers naturally opt to wait with testing until they complete a longer coding task, eschewing many benefits of a test early, test often development style.
At the heart of the problem is that not all unit tests are true unit tests, says Glover. He distinguishes between the following kinds of tests, all of which might be written with JUnit:
Unit tests [verify] an object or multiple objects in isolation. Unit tests don't deal with databases, file systems, or anything else that would tend to make tests run longer; consequently, unit tests can be written from day one... Unit tests are usually easy to code because they are not reliant on architectural dependencies, and they also run quickly. On the downside, individual unit tests offer somewhat limited code coverage. The value of unit tests (which is considerable) is that they enable developers to guarantee object reliability at the lowest possible level.
Component tests verify multiple objects interacting together, but they breach the notion of isolation... They often deal with databases, file systems, network elements... Component tests take longer to write because they're more involved than unit tests. On the other hand, they achieve more code coverage than unit tests do because of their wide reach. They also take longer to run, so running a lot of them together can increase your overall test time dramatically.
System tests verify a software application end-to-end... They introduce a high degree of architectural complexity: the entire application must be running for a system test to take place... System tests are challenging to author and also take a good deal of time to actually execute. On the other hand, they offer quite a bit ... in terms of architectural code coverage.
Glover claims that making these distinctions helps decide when to run each category of tests:
It can be helpful to categorize tests by type. Doing so enables more precise language and can enable your team to run different test types at different frequencies. Categorization is key to avoiding the dreaded three-hour build that runs all "unit tests."
Long test runs, for instance, often indicate a large number of component tests that exercise multiple application layers:
When the testing aspect of a build is prolonged, you can generally predict that a large suite of component tests is involved. Because these tests take longer to run than true unit tests, you may find that you can't run them all the time. Accordingly, it makes sense in a [continuous integration] environment to run the tests at least hourly. You should always run these tests on a local developer's box before checking in any code, too.
In the concluding segment of the article, Glover outlines a technique to separate the different types of tests into JUnit test suites, and then set up Ant tasks to run only a subset of tests in a test run.
As an editorial note, we must mention that this type of categorization is one of the great benefits of TestNG. (See the Artima Leading-Edge Java interview with TestNG creator Cedric Beust, Next-Generation Testing with TestNG.)
How do you categorize tests in your projects? Do you find JUnit's ability to create test suites adequate for test categorization?
Previous Topic
