The Story of an Open Source Endeavor

---

This week Matt Gerrans, Frank Sommers, and I released Artima SuiteRunner, a free open source testing toolkit for Java. You can use this tool with JUnit to run existing JUnit test suites, or standalone to create unit and conformance tests for Java APIs. JUnit, originally written by Erich Gamma and Kent Beck, is very popular among Java developers and especially the extreme programming (XP) community. Like most Java developers, when we needed to create tests we first turned to JUnit. But as we started using JUnit in our own projects, we encountered some limitations. We created a new tool, Artima SuiteRunner, that better addressed our testing needs. In this article, we explain why JUnit didn't fit the bill, and how Artima SuiteRunner was created.

Creating a Conformance Test Kit for the ServiceUI API

As leader of the Jini community's ServiceUI project, I wrote a specification and implemented an API that defines a standard way to attach user interfaces to Jini services. Within the Jini community, the ServiceUI API quickly became a de facto standard, but I wanted to make it official. To propose the ServiceUI API as an official Jini community standard through the Jini Decision Process, however, I needed a conformance test kit (CTK). To help me build the ServiceUI CTK, I turned to JUnit.

My initial experience with JUnit was both pleasant and frustrating. I quickly learned how to create a test case and write test methods, but found it difficult to run a test suite. In fact, I spent three hours sitting with my laptop at a coffee shop in Ft. Collins, Colorado, trying hard not to pull my hair out in frustration. But once I set up JUnit to run my test suite, I was quite happy. I liked the simplicity of creating JUnit test cases and adding test methods. I liked the red or green status bar of JUnit's graphical runner. Except for one special signature test, I proceeded contentedly to write the entire ServiceUI CTK with JUnit.

Before long I had only one missing piece remaining in my ServiceUI CTK: a signature test. A signature test verifies the artifacts of an API's signature, such as name, type, parameters, access level, and other modifiers. It does not verify the API's semantics. Although not necessary for unit testing, a signature test is a critical part of any conformance test. A signature test ensures that each type, field, and method required by an API specification is actually present in an implementation.

Terence Parr, creator of ANTLR, once said, "Why write something in five days that you can spend five years automating?" When it was time to write the signature test portion of the ServiceUI CTK, I decided rather than write it by hand, I would write a generic tool to automatically generate signature tests. I figured I would design this tool as an add-on to JUnit, and release it open source, primarily for the benefit of the Jini community.

So I wrote a tool to generate signature tests, and then attempted to make it work with JUnit. Once again, my hair was in danger of being pulled, but this time it was worse. My previous difficulty was primarily in figuring out how to use JUnit as an application. Now I was trying to figure out how to use JUnit as an API. At this point I was 9000 feet above sea level, in Crested Butte, Colorado. Perhaps the thin air affected my cognitive abilities, but at one point I just threw up my hands and said to myself, "Why is this so hard? It would be easier to rewrite JUnit than figure it out."

Just some friendly advice: if you ever find yourself saying it would be easier to rewrite something rather than figure it out, slap yourself. Chances are you are wrong. I certainly was. Creating SuiteRunner was a huge amount of work. Despite my frustrations with JUnit, it would have been orders of magnitude easier to decipher JUnit's source code than to create a brand new testing toolkit.

Nevertheless, I was fairly far along in the process of creating Artima SuiteRunner before I realized this, and along the way two of my friends, Matt Gerrans and Frank Sommers, became involved in the project. So despite many moments of doubt, including one decision to cancel the project entirely, ultimately Matt, Frank, and I decided to finish what we had started.

A Little Help from My Friends

Initially I got some help from Matt Gerrans, who wanted a Java project to work on to improve his Java skills. Although a veteran programmer, Matt had spent most of the second half of the nineties programming in languages such as C++ and Python. Over many months, Matt and I, in between rounds of foosball, drew circles and arrows on white boards and hacked away at code. The design and implementation of Artima SuiteRunner evolved through much discussion and many iterations.

Frank Sommers became involved with Artima SuiteRunner project after running into limitations using JUnit to test network services. Frank and a group of students from the University of Southern California had built a set of Jini services that formed the infrastructure of a cluster-computing framework. They needed to test not only each Jini service implementation, but also the system as a whole, under a variety of conditions over a sustained period of time. They started to write JUnit test cases, but soon ran into problems.

Their problems with JUnit started even before their initial test would execute. In fact, that was exactly the problem: the first tests would not execute. Jini's infrastructure relies on Java's ability to load classes into a running Java virtual machine (JVM) from any network location. Dynamic class loading alleviates the need to pre-install service implementation code at every Jini client. But Jini's mobile code requires that class loading at a Jini client conform to the delegation-based class loading mechanism specified in the JVM 1.2 specifications. Unfortunately, JUnit's graphical runner did not follow that class loading model, and worked in a way that prevented class loading from the network URLs annotated into their Jini stubs. That made testing Jini services with JUnit's graphical runner impossible. Class loading worked correctly in JUnit's text runner, but that meant having to forgo the graphical UI.

As soon as Frank and his team overcame their initial hurdle by using JUnit's text runner, they encountered a second obstacle. Since each test result represented a sample of the overall system reliability, they wanted to collect many such samples in a format amenable to statistical analysis. They found JUnit's reporting facility simplistic for their purposes. To achieve their testing objective, they faced a choice: extend JUnit's reporting capability, or look for another tool more suitable for their testing requirements.

When Frank Sommers heard about the Artima SuiteRunner project, he tried it and immediately sent Matt and me joyous emails. Even though we didn't know about Frank's problems at the time, our Reporter notion was perfect for Frank's needs. So Frank joined the project and contributed much design insight, code, and time.

Adding Value to JUnit

JUnit is the de facto standard open-source unit test toolkit for Java. Countless organizations have written countless JUnit test cases. We realized most developers would never use Artima SuiteRunner if it meant throwing out their investment in JUnit test cases, so we needed to make sure Artima SuiteRunner added value on top of JUnit. We wanted to find ways to help JUnit users get more out of their existing JUnit test suites.

One difference between JUnit and Artima SuiteRunner is that in JUnit, the notion of results reporting is linked to the notion of running test suites. For example, to generate test results as text, you use JUnit's text runner. To generate graphical test results, you use JUnit's Swing runner. By contrast, Artima SuiteRunner separates the notion of running from that of reporting. In Artima SuiteRunner, we provide one runner that can be configured with multiple reporters. Each reporter presents customized test results: for example, a text reporter presents results as text, a graphical reporter presents results graphically.

Because JUnit links the concepts of running and reporting in the notion of a runner, we made Artima SuiteRunner a JUnit runner. You can use Artima SuiteRunner to run and report results of JUnit tests, just like you can use JUnit's text or Swing runners. The three main advantages of using SuiteRunner to run your JUnit tests are reporters, runpaths, and recipe files:

• A reporter collects and presents test results in a highly customizable way to the user. Examples are text, graphics, Web pages, database output, CSV files, XML, and email alerts.
• A runpath lets you load classes for your tests from anywhere with an easy-to-configure list of filenames, directory paths, and/or URLs.
• A recipe file captures and saves in a file the run properties of a particular suite of tests for easy reuse.

Artima SuiteRunner can help developers create unit and conformance tests, whether or not they have an existing investment in JUnit test cases.

Rethinking JUnit

Originally, I intended to create a tool that generated signature tests for conformance test kits using JUnit. Frank Sommers originally intended to test the reliability of network services and the distributed system they composed using JUnit. Both of us ran into trouble. The beauty of open source is that when a tool doesn't quite fit your needs, you have the option to morph the code into a new form that better solves your problem. Open source projects are occasionally forked, and you can consider Artima SuiteRunner to be a fork of JUnit. But it is a design fork not a code fork, because we didn't reuse any JUnit code. We rethought and reworked JUnit's ideas, and wrote Artima SuiteRunner's code from scratch.

Download Artima SuiteRunner Now (It's Free!)

Coming Soon to Artima.com: Refactoring Tales

In the coming weeks and months, I will publish a series of "Refactoring Tales" articles on Artima.com, each of which focuses narrowly on one aspect of the JUnit and Artima SuiteRunner APIs. Each Refactoring Tales article will in effect be a highly focused design review of one aspect of both JUnit and Artima SuiteRunner. Through these articles I hope to stimulate discussion about API design in the Artima Forums. If you would like to be notified each week of new articles at Artima.com, including the Refactoring Tales series, please subscribe to the Artima Newsletter.

Resources

JUnit is available at:
http://www.junit.org

Getting Started with Artima SuiteRunner, How to Run the Simple Example Included in the Distribution:
http://www.artima.com/suiterunner/start.html

Artima SuiteRunner Tutorial, Building Conformance and Unit Tests with Artima SuiteRunner:
http://www.artima.com/suiterunner/tutorial.html

Runnning JUnit Tests with Artima SuiteRunner, how to use Artima SuiteRunner as a JUnit runner to run your existing JUnit test suites:
http://www.artima.com/suiterunner/junit.html

Create an XML Reporter for Your Unit Tests, how to create a customer reporter for Artima SuiteRunner that formats unit test results in XML:
http://www.artima.com/suiterunner/xmlreporter.html

Artima SuiteRunner home page:
http://www.artima.com/suiterunner/index.html

Artima SuiteRunner download page (You must log onto Artima.com to download the release):
http://www.artima.com/suiterunner/download.jsp

The SuiteRunner Forum:
http://www.artima.com/forums/forum.jsp?forum=61

ANTLR, a translater generator from Terence Parr and friends:
http://www.antlr.org

Talk back!

Have an opinion? Readers have already posted 12 comments about this article. Why not add yours?

About the authors

Matt Gerrans began his professional life as an electronic engineer but quickly saw the light and switched to software development. He now has more than 12 years professional software development under his belt, including work in C++, Java, Python, and yes, even JavaScript.

Bill Venners is president of Artima Software, Inc. and editor-in-chief of Artima.com. He is author of the book, Inside the Java Virtual Machine, a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Bill has been active in the Jini Community since its inception. He led the Jini Community's ServiceUI project that produced the ServiceUI API. The ServiceUI became the de facto standard way to associate user interfaces to Jini services, and was the first Jini community standard approved via the Jini Decision Process. Bill also serves as an elected member of the Jini Community's initial Technical Oversight Committee (TOC), and in this role helped to define the governance process for the community. He currently devotes most of his energy to building Artima.com into an ever more useful resource for developers.

Frank Sommers is an editor with Artima Developer. He is also founder and president of Autospaces, Inc., a company providing collaboration and workflow tools in the financial services industry.