Create an XML Reporter for Your Unit Tests

by Bill Venners

February 24, 2003

Summary

In this tutorial, Bill Venners shows you how to create a custom reporter for Artima SuiteRunner that formats unit test results in XML.

Artima SuiteRunner is a free open source testing toolkit for Java released under the Open Software License. You can use this tool with JUnit to run existing JUnit test suites, or standalone to create unit and conformance tests for Java APIs.

One advantage of using Artima SuiteRunner to run your JUnit tests is Artima SuiteRunner's reporter architecture. A reporter is an object that collects test results and presents them in some way to users. Artima SuiteRunner includes several build-in configurable reporters that can write to the standard error and output streams, files, and a graphical user interface. But Artima SuiteRunner also supports custom reporters. If you want to present results of tests in a different way, such as HTML, email, database, or log files, you can create your own custom reporter that presents results in those ways. This tutorial will show you how to create a custom reporter, using as an example a custom reporter named com.artima.examples.reporter.xml.ex1.XMLReporter (XMLReporter) that formats test results in XML.

Class XMLReporter was added to the Artima SuiteRunner distribution zip file in version 1.0beta5. If you have a release prior to 1.0beta5, please download the latest version of Artima SuiteRunner. Once you unzip the distribution ZIP file, you'll find the source code for XMLReporter in the suiterunner-[release]/example/com/artima/examples/reporter/xml/ex1 directory. You can also view the complete listing in HTML. Because XMLReporter.java is released under the Open Software License, you can use it as a template when creating your own custom Reporter.

`Reporter`'s Event Handler Methods

To create a custom reporter, you create a class that implements interface org.suiterunner.Reporter (Reporter). Interface Reporter declares 13 methods: a setConfiguration method, a dispose method, and 11 event handler methods. In your custom Reporter, you must of course implement all 13 of these methods.

As tests run, the 11 event handler methods are notified of events such as test starting, test succeeded, and test failed. The event handler methods determine how your Reporter will present test results to the user. For example, if you want to present test results as HTML, you write event handler methods that produce HTML. Or, if you want to store test results in a database, you write event handler methods that insert records into that database. In this article, I will show you an XMLReporter whose event handler methods write XML to the standard output.

Figure 1 shows all 11 event handler methods and their meanings.

Figure 1. Reporter's Event Handler Methods

org.suiterunner
Reporter

public interface Reporter
Interface implemented by classes whose instances collect the results of a running suite of tests and presents those results in some way to the user.

Methods

public void infoProvided(Report report)
Provides information that is not appropriate to report via any other Reporter method.

public void runAborted(Report report)
Indicates a runner encountered an error while attempting to run a suite of tests.

public void runCompleted()
Indicates a runner has completed running a suite of tests.

public void runStarting(int testCount)
Indicates a runner is about run a suite of tests.

public void runStopped()
Indicates a runner has stopped running a suite of tests prior to completion, likely because of a stop request.

public void suiteAborted(Report report)
Indicates the execution of a suite of tests has aborted, likely because of an error, prior to completion.

public void suiteCompleted(Report report)
Indicates a suite of tests has completed executing.

public void suiteStarting(Report report)
Indicates a suite of tests is about to start executing.

public void testFailed(Report report)
Indicates a suite (or other entity) has completed running a test that failed.

public void testStarting(Report report)
Indicates a suite (or other entity) is about to start a test.

public void testSucceeded(Report report)
Indicates a suite (or other entity) has completed running a test that succeeded.

`Reporter` Configuration and Disposal

In addition to the 11 event handler methods, interface Reporter declares two other methods, setConfiguration and dispose, shown in Figure 2. You must of course also implement these two methods in your custom Reporter.

Figure 2. Reporter's Configuration and Disposal Methods

org.suiterunner
Reporter

public interface Reporter
Interface implemented by classes whose instances collect the results of a running suite of tests and presents those results in some way to the user.

Methods

public void dispose()
Release any non-memory finite resources, such as file handles, held by this Reporter.

public void setConfiguration(java.util.Set configs)
Configures this Reporter with a specified Set of configuration Characters.

The dispose method is called at the end of the Reporter's life, before it is discarded for garbage collection. If your custom reporter holds finite non-memory resources, such as file handles, database connections, or sockets, you should release these in your dispose method.

The setConfiguration method takes a Set of configuration constants that indicate which of the 11 possible events should actually be reported to the user. For example, a Reporter can be configured to report test failure events, but not test starting or test succeeded events. The valid contents of the Set passed to setConfiguration are defined as configuration Character constants in interface Reporter, as shown in Figure 3.

Figure 3. Reporter's Configuration Character Constants

org.suiterunner
Reporter

public interface Reporter
Interface implemented by classes whose instances collect the results of a running suite of tests and presents those results in some way to the user.

Fields

public static final Character PRESENT_INFO_PROVIDED
Configuration Character that indicates infoProvided method invocations should be presented to the user.

public static final Character PRESENT_RUN_ABORTED
Configuration Character that indicates runAborted method invocations should be presented to the user.

public static final Character PRESENT_RUN_COMPLETED
Configuration Character that indicates runCompleted method invocations should be presented to the user.

public static final Character PRESENT_RUN_STARTING
Configuration Character that indicates runStarting method invocations should be presented to the user.

public static final Character PRESENT_RUN_STOPPED
Configuration Character that indicates runComleted method invocations should be presented to the user.

public static final Character PRESENT_SUITE_ABORTED
Configuration Character that indicates suiteAborted method invocations should be presented to the user.

public static final Character PRESENT_SUITE_COMPLETED
Configuration Character that indicates suiteCompleted method invocations should be presented to the user.

public static final Character PRESENT_SUITE_STARTING
Configuration Character that indicates suiteStarting method invocations should be presented to the user.

public static final Character PRESENT_TEST_FAILED
Configuration Character that indicates testFailed method invocations should be presented to the user.

public static final Character PRESENT_TEST_STARTING
Configuration Character that indicates testStarting method invocations should be presented to the user.

public static final Character PRESENT_TEST_SUCCEEDED
Configuration Character that indicates testSucceeded method invocations should be presented to the user.

Create a No-Arg Constructor for your `Reporter`

Before a run, org.suiterunner.Runner (Runner) instantiates each Reporter via the Reporter's no-arg constructor. Therefore, when you create a custom Reporter, you must give it a public no-arg constructor.

In the XMLReporter's no-arg constructor, I initialize the three private variables of the class: pw, configuration, and validConfigChars. pw is a buffered PrintWriter that wraps System.out. The event handler methods use pw to print XML to the standard output. validConfigChars is a convenience Set of all 11 valid configuration Character constants (see Figure 3). configuration is a Set that holds the current configuration of the XMLReporter.

Here are XMLReporter's instance variables:

public class XMLReporter implements Reporter {

    // A PrintWriter that wraps the standard output
    private PrintWriter pw;

    // A Set that holds this Reporter's current configuration
    private Set configuration;

    // A Set that contains all valid configuration characters, which
    // are defined in interface Reporter.
    private Set validConfigChars;

Every Reporter has a default configuration. A default configuration is a set of configuration Character constants defined by the Reporter's designer. When you create a custom Reporter, therefore, you must decide what its default configuration will be.

For XMLReporter, I defined the default configuration to include all 11 configuration Character constants. In other words, by default, an XMLReporter will report every event handler method invocation in its XML output stream. In the constructor, therefore, I initialize the configuration instance variable to a copy of the validConfigChars Set, which contains all 11 configuration Character constants.

If you use XMLReporter.java as a template when creating your own custom Reporter, you can most likely reuse the configuration and validConfigChars variables as is. Unless you want to print test results in some form to the standard output, however, you will likely want to replace pw with something else, such as a FileOutputStream, a database connection, a socket, a log, a reference to a GUI component -- whatever you will be sending your results.

Here's XMLReporter's no-arg constructor:

    /**
     * Construct an <code>XMLReporter</code>, which writes test results
     * in XML to the standard output stream. The <code>XMLReporter</code> is
     * created with a default configuration that includes all valid configuration
     * characters.
     */
    public XMLReporter() {

        pw = new PrintWriter(new OutputStreamWriter(new BufferedOutputStream(System.out)));

        // Build a set that contains all valid configuration characters
        Set validSet = new HashSet();
        validSet.add(Reporter.PRESENT_INFO_PROVIDED);
        validSet.add(Reporter.PRESENT_RUN_ABORTED);
        validSet.add(Reporter.PRESENT_RUN_COMPLETED);
        validSet.add(Reporter.PRESENT_RUN_STARTING);
        validSet.add(Reporter.PRESENT_RUN_STOPPED);
        validSet.add(Reporter.PRESENT_SUITE_ABORTED);
        validSet.add(Reporter.PRESENT_SUITE_COMPLETED);
        validSet.add(Reporter.PRESENT_SUITE_STARTING);
        validSet.add(Reporter.PRESENT_TEST_FAILED);
        validSet.add(Reporter.PRESENT_TEST_STARTING);
        validSet.add(Reporter.PRESENT_TEST_SUCCEEDED);

        validConfigChars = Collections.unmodifiableSet(validSet);

        // Initialize configuration to validConfigChars, because the default
        // configuration for this Reporter is defined to be everything. (See
        // the JavaDoc comment for the entire class.)
        configuration = new HashSet(validConfigChars);
    }

Implement the `setConfiguration` Method

In addition to initializing your custom Reporter to its default configuration in the no-arg constructor, you must also allow the configuration to be changed via the setConfiguration method. setConfiguration accepts a single parameter, configs: a Set of configuration Character constants that define the Reporter's new configuration. An empty configs Set indicates the Reporter should return to its default configuration.

In XMLReporter's setConfiguration method, I start by checking for null and invalid input, as required by setConfiguration's method contract defined in interface Reporter. Next, I check to see if the passed Set is empty. An empty Set indicates I should reset the XMLReporter back to its default configuration. Therefore, if configs.size() is equal to 0, I assign to configuration a copy of the validConfigChars Set, which I decided would be the default. Otherwise I assign to the configuration instance variable a copy of the passed Set.

If you use XMLReporter.java as a template when creating your own custom Reporter, you can most likely reuse the setConfiguration method without any changes. Here's XMLReporter's setConfiguration method:

    /**
     * Configures this <code>XMLReporter</code>. If the specified <code>configuration</code>
     * set is zero size, the <code>XMLReporter</code> will be configured to its
     * default configuration. (The default configuration is described in the main
     * comment for class <code>XMLReporter</code>.)
     *
     * @param configuration set of <code>Config</code> objects that indicate the new
     *     configuration for this <code>XMLReporter</code>
     *
     * @exception NullPointerException if <code>configuration</code> reference is <code>null</code>
     * @exception IllegalArgumentException if <code>configuration</code> set contains any objects
     *    whose class isn't <code>org.suiterunner.Config</code>
     */
    public synchronized void setConfiguration(Set configs) {

        if (configs == null) {
            throw new NullPointerException("Parameter configs is null.");
        }

        for (Iterator it = configs.iterator(); it.hasNext();) {

            Object o = it.next();

            if (!(o instanceof Character)) {

                throw new IllegalArgumentException("Passed object is not a Character.");
            }

            if (!validConfigChars.contains(o)) {

                throw new IllegalArgumentException("Passed object is not a valid configuration Character.");
            }
        }

        if (configs.size() == 0) {
            // If passsed Set is empty, reset this Reporter to its default
            // configuration, which is to report everything.
            this.configuration = new HashSet(validConfigChars);
        }
        else {
            this.configuration = new HashSet(configs);
        }
    }

Implement the `dispose` Method

With the no-arg constructor and setConfiguration methods finished, you may wish to turn your attention to the end of your custom Reporter's lifetime and implement dispose. If your custom Reporter holds onto finite non-memory resources, such as file handles or database connections, you should release them in the dispose method. If your custom Reporter does not hold onto any resources, your dispose method should do nothing (but you still have to implement it, of course).

Because XMLReporter simply writes to the standard output stream, it holds no finite non-memory resources to release in dispose. As a result, XMLReporter's dispose method does nothing.

Here's XMLReporter's do-nothing dispose method:

    /**
     * Does nothing, because this object holds no finite non-memory resources.
     */
    public void dispose() {
    }

Implement the `runStarting` Method

Now that the no-arg constructor, setConfiguration, and dispose methods are behind you, you need only implement the 11 event handler methods. The custom way you implement these methods determines the custom way your Reporter will present test results to the user. In XMLReporter, the event handler methods write XML to the standard output.

A good place to start implementing is the runStarting method, the first method Runner invokes for each test run. In XMLReporter, all event handler methods have the same basic structure. Each method checks the configuration to see if it is supposed to report the event. If so, it prints some XML to the standard output. For example, the runStarting method checks to make sure the PRESENT_RUN_STARTING configuration Character is contained in the current configuration. If so, it writes an XML header <?xml version="1.0"?> and the element <run> to the standard output.

In XMLReporter, I ignore runStarting's testCount parameter, which indicates the number of tests expected in the run. In your custom Reporters, you may wish to present this information to the user. I do make sure that testCount is non-negative, which is required by runStarting method contract as defined in interface Reporter.

Here's XMLReporter's runStarting method:

    /**
     * Prints information indicating that a run with an expected <code>testCount</code>
     * number of tests is starting, if
     * the current configuration includes <code>Reporter.PRESENT_RUN_STARTING</code>.
     * If <code>Reporter.PRESENT_RUN_STARTING</code> is not included in the the
     * current configuration, this method prints nothing.
     *
     * @param testCount the number of tests expected during this run.
     *
     * @exception IllegalArgumentException if <code>testCount</code> is less than zero.
     */
    public void runStarting(int testCount) {

        if (testCount < 0) {
            throw new IllegalArgumentException();
        }

        if (configuration.contains(Reporter.PRESENT_RUN_STARTING)) {

            pw.println("<?xml version=\"1.0\"?>");
            pw.println("<run>");
        }
    }

Implement the `runCompleted` Method

The final method invoked by Runner during a test run is either runCompleted, runStopped or runAborted. You may wish to implement these methods next in your custom Reporter. When a run completes normally, Runner invokes runCompleted.

In XMLReporter's runCompleted method, I need to print out a closing </run> element to match the opening <run> element printed out in runStarting. First, I check to make sure the PRESENT_RUN_COMPLETED configuration Character is contained in the current configuration. If so, I write the element </run> to the standard output. I flush pw's buffers to make sure that at the end of the run, the complete XML document shows up at the standard output.

Here's XMLReporter's runCompleted method:

    /**
     * Prints information indicating a run has completed, if
     * the current configuration includes <code>Reporter.PRESENT_RUN_COMPLETED</code>.
     * If <code>Reporter.PRESENT_RUN_COMPLETED</code> is not included in the the
     * current configuration, this method prints nothing.
     */
    public synchronized void runCompleted() {

        if (configuration.contains(Reporter.PRESENT_RUN_COMPLETED)) {

            pw.println("</run>");
            pw.flush();
        }
    }

Implement the `runStopped` Method

If a run is stopped—for example, if the user presses the stop button on the GUI—Runner invokes runStopped instead of runCompleted. In your custom Reporter, you may wish to implement the runStopped method next. In XMLReporter, the runStopped method looks much like the runCompleted method, except that I also print out an extra <runStopped/> element before the </run> element. I print the extra <runStopped/> so that users can tell by looking at the XML document that the run stopped rather than completed normally.

Here's XMLReporter's runStopped method:

    /**
     * Prints information indicating a runner has stopped running a suite of tests prior to completion, if
     * the current configuration includes <code>Reporter.PRESENT_RUN_STOPPED</code>.
     * If <code>Reporter.PRESENT_RUN_STOPPED</code> is not included in the the
     * current configuration, this method prints nothing.
     */
    public synchronized void runStopped() {

        if (configuration.contains(Reporter.PRESENT_RUN_STOPPED)) {

            String stringToReport = "<runStopped/>\n";
            stringToReport += "</run>";

            pw.println(stringToReport);
            pw.flush();
        }
    }

Implement the `runAborted` Method

The runAborted method is a bit more complex than runCompleted or runStopped, because runAborted takes an org.suiterunner.Report (Report). A Report is a bundle of information about the event being reported, including a name, message, source, thread, date, and optional Throwable. You can think of a Report as an event object, like MouseEvent.

In XMLReporter's runAborted method, I print information about the Report between open <runAborted> and close </runAborted> elements, then I print the </run> element. Because all the remaining event handler methods will also need to print information about the Report, I created a private helper method called printReport that prints out information about the passed Report.

Here's the runAborted method:

    /**
     * Prints information indicating a run has aborted prior to completion, if
     * the current configuration includes <code>Reporter.PRESENT_RUN_ABORTED</code>.
     * If <code>Reporter.PRESENT_RUN_ABORTED</code> is not included in the the
     * current configuration, this method prints nothing.
     *
     * @param report a <code>Report</code> that encapsulates the suite aborted event to report.
     * @exception NullPointerException if <code>report</code> reference is <code>null</code>
     */
    public synchronized void runAborted(Report report) {

        if (report == null) {
            throw new NullPointerException("Parameter report is null.");
        }

        if (configuration.contains(Reporter.PRESENT_RUN_ABORTED)) {

            String stringToReport = "<runAborted>\n";
            stringToReport += printReport(report);
            stringToReport += "</runAborted>\n";
            stringToReport += "</run>";

            pw.println(stringToReport);
            pw.flush();
        }
    }

The `printReport` Helper Method

In printReport, I print out five elements containing data from the Report: name, message, date, source, and thread. If a Throwable is contained in the Report, I also print out a throwable element. To ensure I print out valid XML for the character data delimited by these elements, I process the raw Strings via a private insertEntities helper method, which replaces certain characters with their XML entity equivalents.

Figure 4 shows the public interface of Report.

Figure 4. Report's Public Interface

org.suiterunner
Report

public class Report
Class used to send reports to a reporter.

Constructors

public Report(Object source, String name, String message)
Constructs a new Report with specified source, name, and message.

public Report(Object source, String name, String message, Throwable throwable)
Constructs a new Report with specified source, name, message, and throwable.

public Report(Object source, String name, String message, Rerunnable rerunnable)
Constructs a new Report with specified source, name, message, and rerunnable.

public Report(Object source, String name, String message, Throwable throwable, Rerunnable rerunnable)
Constructs a new Report with specified source, name, message, throwable, and rerunnable.

public Report(Object source, String name, String message, Throwable throwable, Rerunnable rerunnable, Thread thread, java.util.Date date)

Constructs a new Report with specified source, name, message, throwable, rerunnable, thread, and date.

Methods

public java.util.Date getDate()
Get the Date embedded in this Report.

public String getMessage()
Get a String message.

public String getName()
Get a String name of the entity about which this Report was generated.

public Rerunnable getRerunnable()
Get a Rerunnable that can be used to rerun the test or other entity reported about by this Report, or null if the test or other entity cannot be rerun.

public Object getSource()
Get the object that generated this report.

public Thread getThread()
Get the Thread about whose activity this Report was generated.

public Throwable getThrowable()
Get a Throwable that indicated the condition reported by this Report.

Here's the printReport method:

    /*
     * Print a Report to the standard output.
     */
    private String printReport(Report report) {

        if (report == null) {
            throw new NullPointerException("Parameter report is null.");
        }

        String name = insertEntities(report.getName());
        String message = insertEntities(report.getMessage());
        String dateString = insertEntities(report.getDate().toString());
        String sourceName = insertEntities(report.getSource().toString());
        String threadName = insertEntities((report.getThread()).getName());
        Throwable throwable = report.getThrowable();

        String reportString = "    <name>" + name + "</name>\n";
        reportString += "    <message>" + message + "</message>\n";
        reportString += "    <date>" + dateString + "</date>\n";
        reportString += "    <source>" + sourceName + "</source>\n";
        reportString += "    <thread>" + threadName + "</thread>\n";
        if (throwable != null) {
            CharArrayWriter caw = new CharArrayWriter();
            PrintWriter capw = new PrintWriter(caw);
            throwable.printStackTrace(capw);
            capw.close();
            String stackTrace = caw.toString();
            reportString += "    <throwable>\n";
            reportString += "        " + insertEntities(stackTrace);
            reportString += "    </throwable>\n";
        }
        return reportString;
    }

The `insertEntities` Helper Method

In the insertEntities helper method, I simply subsitute XML entities for any &, <, >, ", or ' characters in the passed raw String.

Here's the insertEntities method:

    /*
     * Replace &, <, >, ", and ', in passed raw String with their
     * XML entity representation.
     */
    private String insertEntities(String raw) {

        if (raw == null) {
            throw new NullPointerException("Parameter raw is null.");
        }

        StringBuffer buf = new StringBuffer();

        for (int i = 0; i < raw.length(); ++i) {
            char c = raw.charAt(i);

            if (c == '&') {
                buf.append("&amp;");
            }
            else if (c == '<') {
                buf.append("&lt;");
            }
            else if (c == '>') {
                buf.append("&gt;");
            }
            else if (c == '"') {
                buf.append("&quot;");
            }
            else if (c == '\'') {
                buf.append("&apos;");
            }
            else {
                buf.append(c);
            }
        }

        return buf.toString();
    }

Completing Your Custom `Reporter`

To finish your custom Reporter, you need only finish implementing the remaining event handler methods. I implemented XMLReporter's remaining event handler methods in much the same way as runAborted. In XMLReporter's testFailed method, for example, I print information about the Report between open <testFailed> and close </testFailed> elements.

Here's XMLReporter's testFailed method:

    /**
     * Prints information extracted from the specified <code>Report</code>
     * about a test that failed, if
     * the current configuration includes <code>Reporter.PRESENT_TEST_FAILED</code>.
     * If <code>Reporter.PRESENT_TEST_FAILED</code> is not included in the the
     * current configuration, this method prints nothing.
     *
     * @param report a <code>Report</code> that encapsulates the test failed event to report.
     *
     * @exception NullPointerException if <code>report</code> reference is <code>null</code>
     */
    public synchronized void testFailed(Report report) {

        if (report == null) {
            throw new NullPointerException("Parameter report is null.");
        }

        if (configuration.contains(Reporter.PRESENT_TEST_FAILED)) {

            String stringToReport = "<testFailed>\n";
            stringToReport += printReport(report);
            stringToReport += "</testFailed>";

            pw.println(stringToReport);
        }
    }

As mentioned previously, you can view XMLReporter's complete listing in HTML, or look at the actual file in the suiterunner-[release]/example/com/artima/examples/reporter/xml/ex1 directory the Artima SuiteRunner distribution ZIP file, version 1.0beta5 or later.

Figure 5 shows the public interface of XMLReporter.

Figure 5. XMLReporter's Public Interface

com.artima.examples.reporter.xml.ex1
XMLReporter

public class XMLReporter
A Reporter that formats test results as XML and prints to the standard output stream.

Constructors

public XMLReporter()
Construct an XMLReporter, which writes test results in XML to the standard output stream.

Methods

public void dispose()
Does nothing, because this object holds no finite non-memory resources.

public synchronized void infoProvided(org.suiterunner.Report report)
Prints information extracted from the specified Report, if the current configuration includes Reporter.PRESENT_INFO_PROVIDED.

public synchronized void runAborted(org.suiterunner.Report report)
Prints information indicating a run has aborted prior to completion, if the current configuration includes Reporter.PRESENT_RUN_ABORTED.

public synchronized void runCompleted()
Prints information indicating a run has completed, if the current configuration includes Reporter.PRESENT_RUN_COMPLETED.

public void runStarting(int testCount)
Prints information indicating that a run with an expected testCount number of tests is starting, if the current configuration includes Reporter.PRESENT_RUN_STARTING.

public synchronized void runStopped()
Prints information indicating a runner has stopped running a suite of tests prior to completion, if the current configuration includes Reporter.PRESENT_RUN_STOPPED.

public synchronized void setConfiguration(java.util.Set configs)
Configures this XMLReporter.

public synchronized void suiteAborted(org.suiterunner.Report report)
Prints information indicating the execution of a suite of tests has aborted prior to completion, if the current configuration includes Reporter.PRESENT_SUITE_ABORTED.

public synchronized void suiteCompleted(org.suiterunner.Report report)
Prints information indicating a suite of tests has completed executing, if the current configuration includes Reporter.PRESENT_SUITE_COMPLETED.

public void suiteStarting(org.suiterunner.Report report)
Prints information indicating a suite of tests is about to start executing, if the current configuration includes Reporter.PRESENT_SUITE_STARTING.

public synchronized void testFailed(org.suiterunner.Report report)
Prints information extracted from the specified Report about a test that failed, if the current configuration includes Reporter.PRESENT_TEST_FAILED.

public synchronized void testStarting(org.suiterunner.Report report)
Prints information extracted from the specified Report about a test about to be run, if the current configuration includes Reporter.PRESENT_TEST_STARTING.

public synchronized void testSucceeded(org.suiterunner.Report report)
Prints information extracted from the specified Report about a test that succeeded, if the current configuration includes Reporter.PRESENT_TEST_SUCCEEDED.

Take `XMLReporter` for a Spin

To use a custom Reporter, you must add it to the org.suiterunner.Reporters property of the recipe. To make it easy to try XMLReporter, I added a new recipe file, xmlreporter.srj, to the Artima SuiteRunner distribution ZIP file in version 1.0beta5. The xmlreporter.srj recipe file indicates that XMLReporter should be used to collect results during the running test. If you have a release prior to 1.0beta5, please download the latest version of Artima SuiteRunner. Once you unzip the distribution ZIP file, you'll find xmlreporter.srj in the suiterunner-[release] directory. Here are the contents of xmlreporter.srj:

org.suiterunner.Suites=-s com.artima.examples.account.ex6test.AccountSuite
org.suiterunner.Runpath=-p "example"
org.suiterunner.Reporters=-r com.artima.examples.reporter.xml.ex1.XMLReporter

In xmlreporter.srj:

The value of org.suiterunner.Runpath (-p "example") specifies a runpath with a single directory, example.
The value of org.suiterunner.Suites (-s com.artima.examples.account.ex6test.AccountSuite) indicates Artima SuiteRunner should load the specified class, a subclass of org.suiterunner.Suite, and invoke its execute method.
The value of org.suiterunner.Reporters (-r com.artima.examples.reporter.xml.ex1.XMLReporter) indicates that Artima SuiteRunner should load the specified custom XMLReporter from the runpath, instantiate the XMLReporter via its public no-arg constructor, and notify the XMLReporter of test events as the test runs.

When invoked via the previous command that specifies xmlreporter.srj as a command line parameter, Artima SuiteRunner will:

Create a URLClassLoader that can load classes from the example directory, the directory specified via the recipe file's org.suiterunner.Runpath property.
Via the URLClassLoader, load class com.artima.examples.account.ex6test.AccountSuite, the class specified via the recipe file's org.suiterunner.Suites property.
Discover that com.artima.examples.account.ex6test.AccountSuite class is a subclass of org.suiterunner.Suite.
Instantiatecom.artima.examples.account.ex6test.AccountSuite.
Instantiate an XMLReporter, the reporter specified via the recipe file's org.suiterunner.Reporters property, which prints test results formatted in XML to the standard output.
Place the XMLReporter inside a dispatch Reporter, which will dispatch all Reporter method invocations to the XMLReporter.
Run the suite of tests by invoking execute on the com.artima.examples.account.ex6test.AccountSuite instance, passing in the dispatch reporter that contains the XMLReporter. Test results will, therefore, be reported to the XMLReporter as the test runs.

To see the XMLReporter in action, run the following command from the directory in which you unzipped the Artima SuiteRunner distribution ZIP file:

java -jar suiterunner-[release].jar xmlreporter.srj

You should see this at the standard output, the results of the test expressed in XML:

<?xml version="1.0"?>
<run>
<suiteStarting>
    <name>AccountSuite</name>
    <message>The execute method of a subsuite is about to be invoked.</message>
    <date>Sat Feb 22 22:29:16 PST 2003</date>
    <source>Thread[Thread-0,5,main]</source>
    <thread>Thread-0</thread>
</suiteStarting>
<testStarting>
    <name>AccountSuite.testConstructor()</name>
    <message>com.artima.examples.account.ex6test.AccountSuite</message>
    <date>Sat Feb 22 22:29:16 PST 2003</date>
    <source>com.artima.examples.account.ex6test.AccountSuite@7ccad</source>
    <thread>Thread-0</thread>
</testStarting>
<testSucceeded>
    <name>AccountSuite.testConstructor()</name>
    <message>com.artima.examples.account.ex6test.AccountSuite</message>
    <date>Sat Feb 22 22:29:16 PST 2003</date>
    <source>com.artima.examples.account.ex6test.AccountSuite@7ccad</source>
    <thread>Thread-0</thread>
</testSucceeded>
<testStarting>
    <name>AccountSuite.testDeposit()</name>
    <message>com.artima.examples.account.ex6test.AccountSuite</message>
    <date>Sat Feb 22 22:29:16 PST 2003</date>
    <source>com.artima.examples.account.ex6test.AccountSuite@7ccad</source>
    <thread>Thread-0</thread>
</testStarting>
<testFailed>
    <name>AccountSuite.testDeposit()</name>
    <message>Account.deposit() didn't deposit 20 correctly. Resulting balance should
        have been 20, but was 19.</message>
    <date>Sat Feb 22 22:29:16 PST 2003</date>
    <source>com.artima.examples.account.ex6test.AccountSuite@7ccad</source>
    <thread>Thread-0</thread>
    <throwable>
        org.suiterunner.TestFailedException: Account.deposit() didn't deposit 20 correctly.
            Resulting balance should have been 20, but was 19.
        at org.suiterunner.Suite.verify(Suite.java:779)
        at com.artima.examples.account.ex6test.AccountSuite.testDeposit(AccountSuite.java:31)
        at java.lang.reflect.Method.invoke(Native Method)
        at org.suiterunner.Suite.executeTestMethod(Suite.java:458)
        at org.suiterunner.Suite.executeTestMethods(Suite.java:352)
        at org.suiterunner.Suite.execute(Suite.java:266)
        at org.suiterunner.SuiteRunnerThread.run(SuiteRunnerThread.java:128)
    </throwable>
</testFailed>
<testStarting>
    <name>AccountSuite.testWithdraw()</name>
    <message>com.artima.examples.account.ex6test.AccountSuite</message>
    <date>Sat Feb 22 22:29:16 PST 2003</date>
    <source>com.artima.examples.account.ex6test.AccountSuite@7ccad</source>
    <thread>Thread-0</thread>
</testStarting>
<testFailed>
    <name>AccountSuite.testWithdraw()</name>
    <message>Account.withdraw() didn't withdraw 10 correctly. Remaining balance should have been
        10, but was 9.</message>
    <date>Sat Feb 22 22:29:16 PST 2003</date>
    <source>com.artima.examples.account.ex6test.AccountSuite@7ccad</source>
    <thread>Thread-0</thread>
    <throwable>
        org.suiterunner.TestFailedException: Account.withdraw() didn't withdraw 10 correctly. Remaining
            balance should have been 10, but was 9.
        at org.suiterunner.Suite.verify(Suite.java:779)
        at com.artima.examples.account.ex6test.AccountSuite.testWithdraw(AccountSuite.java:68)
        at java.lang.reflect.Method.invoke(Native Method)
        at org.suiterunner.Suite.executeTestMethod(Suite.java:458)
        at org.suiterunner.Suite.executeTestMethods(Suite.java:352)
        at org.suiterunner.Suite.execute(Suite.java:266)
        at org.suiterunner.SuiteRunnerThread.run(SuiteRunnerThread.java:128)
    </throwable>
</testFailed>
<suiteCompleted>
    <name>AccountSuite</name>
    <message>The execute method of a subsuite returned normally.</message>
    <date>Sat Feb 22 22:29:16 PST 2003</date>
    <source>Thread[Thread-0,5,main]</source>
    <thread>Thread-0</thread>
</suiteCompleted>
</run>

For more information about how to add your own custom Reporter to a recipe, please see Getting Started with Artima SuiteRunner, Running JUnit Tests with Artima SuiteRunner, or Artima SuiteRunner Tutorial,

Customizing `Report`s

Now that you know how to make a custom Reporter, keep in mind that you can also make custom Report classes. If you want to report information that is not included in class Report, you can create a subclass of Report that holds that information and a custom Reporter that knows about and presents that extra information. You can then pass instances of the custom Report class to the Reporter right from your test methods by declaring your test methods to accept a Reporter. An article describing this in detail with an example will be forthcoming soon on Artima.com.

Get Help in the SuiteRunner Forum

For help with Artima SuiteRunner, please post to the SuiteRunner Forum.

Resources

JUnit is available at:
http://www.junit.org

Why We Refactored JUnit
http://www.artima.com/suiterunner/why.html

Artima SuiteRunner Tutorial, Building Conformance and Unit Tests with Artima SuiteRunner:
http://www.artima.com/suiterunner/tutorial.html

Getting Started with Artima SuiteRunner, How to Run the Simple Example Included in the Distribution:
http://www.artima.com/suiterunner/start.html

Runnning JUnit Tests with Artima SuiteRunner, how to use Artima SuiteRunner as a JUnit runner to run your existing JUnit test suites:
http://www.artima.com/suiterunner/junit.html

Artima SuiteRunner home page:
http://www.artima.com/suiterunner/index.html

Artima SuiteRunner download page (You must log onto Artima.com to download the release):
http://www.artima.com/suiterunner/download.jsp

The SuiteRunner Forum:
http://www.artima.com/forums/forum.jsp?forum=61

Talk back!

Have an opinion? Be the first to post a comment about this article.

About the author

Bill Venners is president of Artima Software, Inc. and editor-in-chief of Artima.com. He is author of the book, Inside the Java Virtual Machine, a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Bill has been active in the Jini Community since its inception. He led the Jini Community's ServiceUI project that produced the ServiceUI API. The ServiceUI became the de facto standard way to associate user interfaces to Jini services, and was the first Jini community standard approved via the Jini Decision Process. Bill also serves as an elected member of the Jini Community's initial Technical Oversight Committee (TOC), and in this role helped to define the governance process for the community. He currently devotes most of his energy to building Artima.com into an ever more useful resource for developers.

Reporter's Event Handler Methods

Reporter Configuration and Disposal

Create a No-Arg Constructor for your Reporter

Implement the setConfiguration Method

Implement the dispose Method

Implement the runStarting Method

Implement the runCompleted Method

Implement the runStopped Method

Implement the runAborted Method

The printReport Helper Method

The insertEntities Helper Method

Completing Your Custom Reporter

Take XMLReporter for a Spin

Customizing Reports

Get Help in the SuiteRunner Forum

Resources

Talk back!

About the author

`Reporter`'s Event Handler Methods

`Reporter` Configuration and Disposal

Create a No-Arg Constructor for your `Reporter`

Implement the `setConfiguration` Method

Implement the `dispose` Method

Implement the `runStarting` Method

Implement the `runCompleted` Method

Implement the `runStopped` Method

Implement the `runAborted` Method

The `printReport` Helper Method

The `insertEntities` Helper Method

Completing Your Custom `Reporter`

Take `XMLReporter` for a Spin

Customizing `Report`s