This post originated from an RSS feed registered with Agile Buzz
by James Robertson.
Original Post: Code Coverage
Feed Title: Michael Lucas-Smith
Feed URL: http://www.michaellucassmith.com/site.atom
Feed Description: Smalltalk and my misinterpretations of life
During an update meeting of where WithStyle is at for a potential customer of ours, the CEO of that company threw us a question which we didn't have an answer for. We told him we had 400 passing tests, 4 failing tests and 0 erroring tests. His question, what is the coverage of those tests?
This is a very valid question to ask. You could have 10,000 tests which are completely meaningless to an application - how much application logic do they actually test.
In steps the Zork-Analysis package which can be loaded from Public Store. This package is really neat. It's designed for VW5i which means its browser navigates by namespaces and categories, but that's okay, since categories are not completely defunct yet (they will be in 7.3).
You select your code in the Zork browser and tell it to install method wrappers. Once it has done this, you run your code. It monitors all calls to your code and records the types of the objects being passed in, types of the objects in the instance variables, the type of the object calling it and the type of the object it returns from the method.
Once you've finished, the Zork browser lets you browse all of this informatio in a hypertext kind of fashion. You view the method and click on a variable or selector and it tells you everything it has learnt.
Once you're done, you uninstall the method wrappers and all is back to normal (and back to 100% speed).
But! The zork browser doesn't end there. While you have your information collected, it also tells you coverage statistics. What bits of your code were run and what percentage of that code ran. What is really interesting is when you look at a method and half the method never ran. This is an indication of two things. (a) dead code that will never run, (b) not enough coverage of your code to make that execution path execute.
So lets bring this all together. You wrap up all your code, then run all your tests, and Zork can tell you how much of your code was run - thus, telling you the coverage of your application your tests are giving you. If you wanted to take it further, you could then record all your type information and produce some sort of javadoc type equivalent. We don't do that.
We just record the percentage of code that was executed. Now here's the cincher.. we've added this to our nightly releaser program, which means every night we get an update on how much of our code base we've covered in our tests!
I just sent an email off to the CEO to inform him that our tests cover 71% of our application. That's quite a bit by anyones standards (except Sam of Pollock fame, who probably has 99% coverage :))
Previous Topic
