Artima.com has published an article by Bill Venners, Matt Gerrans, and Frank Sommers in which they explain why JUnit didn't solve their particular testing problems, and how they created Artima SuiteRunner.---xj40dkcfea73---Artima.com has published an article by Bill Venners, Matt Gerrans, and Frank Sommers in which they explain why JUnit didn't solve their particular testing problems, and how they created Artima SuiteRunner.

http://www.artima.com/suiterunner/why.html

Here's an excerpt:

Because JUnit links the concepts of running and reporting in the notion of a runner, we made Artima SuiteRunner a JUnit runner. You can use Artima SuiteRunner to run and report results of JUnit tests, just like you can use JUnit's text or Swing runners. The three main advantages of using SuiteRunner to run your JUnit tests are reporters, runpaths, and recipe files:

* A reporter collects and presents test results in a highly customizable way to the user. Examples are text, graphics, Web pages, database output, CSV files, XML, and email alerts.

* A runpath lets you load classes for your tests from anywhere with an easy- to-configure list of filenames, directory paths, and/or URLs.

* A recipe file captures and saves in a file the run properties of a particular suite of tests for easy reuse.

What do you think of the authors' comments?

The story writes:
"I spent three hours sitting with my laptop at a coffee shop in Ft. Collins, Colorado, trying hard not to pull my hair out in frustration."

To poorly, but most cogently, paraphrase Hemingway: no great writing was ever done in a coffee shop.

Let's not forget that.

Were you in a coffee shop when you wrote this? ;-)

Why do people continually feel the need to rewrite rather than extend?

Recently I investigated migrating from optimizeit to an open source code profiler, quite a number exist, each replicating the basics, none providing anything extra. If you look at the time lines for each generally one started when another finished. Rather than extending what was already developed, for no apparant reason a rewrite was considered the most appropriate.


I wonder is this another example of rewrite rather than extension?

One of the controversial but frequent requests of JUnit is an easy way to do one-time setup. One-time setup being a setUp method that runs only once for N tests. The gurus say it is bad. I think it is practical. What are your opinions?

Also, does SuiteRunner have built-in support for one-time setup?

> Why do people continually feel the need to rewrite rather
> than extend?
>
> Recently I investigated migrating from optimizeit to an
> open source code profiler, quite a number exist, each
> replicating the basics, none providing anything extra. If
> you look at the time lines for each generally one started
> when another finished. Rather than extending what was
> already developed, for no apparant reason a rewrite was
> considered the most appropriate.
>
>
> I wonder is this another example of rewrite rather than
> extension?

Sometimes, refactoring is necessary when a system can no longer accommodate certain types of extensions without breaking down. In the case of writing signature tests or unit testing Jini services, extending JUnit may have bloated JUnit to the detriment of developers that don't need those extensions.

But your question cuts to the heart of a very interesting design problem. Most of the basic building blocks of any software have, in fact, multiple implementations. For instance, there are dozens of filesystems (ext2fs, NFS, NTFS, etc), relational database implementations, or authentication frameworks. One could argue that, in the case of DBMSs, for instance, one would never have to write another DBMS, but could instead just extend MySQL or Postgres (well, that's 2 already). But would that be the right approach in every case?

I think extension of a system is effective only up to a certain level of system complexity. Where that level is, how much extension a system tolerates without being overburdened, depends on the flexibility of that system's design. In general, the simpler the inital design, the more extension a system allows.

But some systems, by their very nature, must be complex: For instance, a relational database, or even a testing framework, must have a level of complexity in order to deliver its benefits. Depending on where that level is, such systems tolerate only so much complexity before being overburdened. I think JUnit may have reached that level, at least with respect to the extensions we needed to make to it.

I'm a big fan of simplicity. But high levels of complexity are sometimes what differentiates a system in a positive way - it's what makes it unique. But that also makes those systems hard to extend.

I think that's not unique to software. For instance, the other day I was listening to a CD by a young singer (Vienna Teng, I believe). Her songs were nice, but were extremely simple. Their simplicity might tire one after listening to her songs for 50 minutes. As a musician, I could think of dozens of ways to extend her songs, to make them more interesting. On the other hand, when I listen to the Beatles (or Beethoven), I cannot come up with many ways to "extend" their music: They reached a level of complexity - perfection might be a better word - that does not tolerate much fiddling with their design (compositions).

To answer your question of why people sometimes feel the need to rewrite rather than extend, continuing the musical analogy, when Beethoven died some composers tried to extend his style. But most of us have never heard of them. The reason is that their attempts didn't work: Their style ("design" or "architecture") lost its coherence. Others, however, instead of extending his style, went on to rethink - refactor - the symphony, the opera, the song. They were the Schuberts, the Schumanns, the Wagners, or the Beatles.

I think similar principles apply even to political systems. For instance, the American revolutionaries didn't want to extend the British imperial system, but instead went on to "refactor" the social contract between the citizens and their government. So a revolution (refactoring) was necessary, because the old system couldn't accommodate the requirements (extensions) of the new design.

The beauty of open source is that when you reach the inflection point of a system's design complexity, you can take its main ideas and build something new upon those concepts. It may not always be worth doing, but it almost always leads to interesting new ideas. In the case of SuiteRunner, one such idea is coming up with a cleaner API to interface with. That might or might not be the "perfect" design (most likely, not)... So others may end up refactoring SuiteRunner to suite (no pun intended) their needs....

Your Suite object can do this one-time setup during its construction. Then all the tests in that Suite use that configuration.

I use jUnit w/ ant (junit and junitreport tasks) and am quite happy and satisfy of it. For now, junit fully supply what i need to get my testing work (well and quickly) done ... I'll have a look at Suite Runner when there's need (for me) to do API testing :-)
Anyway, thanks to Bill et al for this package ... it will certainly help somebody in his testing work ... and will prevent him to pass a sleepless night with a laptop and a mountain of coffee cup ... :-)


/hermann

["May the source be with you"]

You're ending your article with "Coming Soon to Artima.com: Refactoring Tales", where can I find those articles?

Thanks,

Jean Lazarou

We have a test suite of ~1500 tests. For a test suite of this size, simple JUnit just doesn't cut it. We need to be able to run parts of the test suite, generate test cases automatically baed on files in the file system, etc etc.

I'm not particularly impressed with JUnit. The design is nice if you follow the intended use, but it's not very flexible.

> You're ending your article with "Coming Soon to
> Artima.com: Refactoring Tales", where can I find those
> articles?
>
Sorry, we haven't written them yet. Development on SuiteRunner halted a few months back when I decided to add the Weblogs and Buzz/News features to Artima.com. We have a few more features planned for SuiteRunner before releasing it 1.0. There will be articles to accompany those new features, and after that, we will do the Refactoring Tales articles.

The new features we plan to implement are:

- ability to test private methods like any other method
- making Report serializable
- adding user-defined properties to the recipe
- passing context into tests

> We have a test suite of ~1500 tests. For a test suite of
> this size, simple JUnit just doesn't cut it. We need to be
> able to run parts of the test suite, generate test cases
> automatically baed on files in the file system, etc etc.
>
I believe you can do both of those with JUnit, though you will have to roll your own mechanisms, but it shouldn't be too difficult. There may be some JUnit extension that can help you with the automatic test case generation. We decided to not support an automatic Suite generation directly in SuiteRunner itself, because there are many ways to do it. But I did plan to write an article that includes an example of building a Suite that way. Haven't done it yet, though.

> I believe you can do both of those with JUnit, though you
> will have to roll your own mechanisms, but it shouldn't be
> too difficult.

We did do it with JUnit, and we did roll our own mechanisms, and it wasn't too hard, but getting the GUI to understand those mechanisms requires quite a bit of extra work. So while we could do it, I wouldn't say that JUnit supported us in doing it.

> We decided to not support an automatic Suite
> generation directly in SuiteRunner itself, because there
> are many ways to do it.

I don't think you need to actually provide support for that, since as you say there are just too many ways to do that, but what should be provided is an abstraction that lets people slot in their own generators, so that higher-level code can deal with this case correctly. (Whether you do that or not I have no idea.)