Angle Brackets and Curly Braces
Introducing the ScalaTest Shell: a DSL for the Scala Interpreter
by Bill Venners
May 6, 2011

Summary
The ScalaTest Shell is a new DSL coming in ScalaTest 1.5 that aims to make ScalaTest easier to use from the Scala interpreter. This post gives preview.

ScalaTest, like Scala, is designed to grow with the demands of its users. The main way that ScalaTest does that is by defining clearly specified extension points, such as the lifecycle methods in trait Suite: run, runNestedSuites, nestedSuites, runTests, testNames, tags, runTest, and withFixture. These methods are designed to be overridden by users to allow them to more easily build custom solutions for their own specific needs.

But Scala is "scalable" not only by allowing its users to morph it into a language suitable for specific tasks, but also because it is useful for small tasks as well as large ones. One area in which I had always wanted to make ScalaTest more useful was inside the Scala interpreter (also called "the REPL," for Read-Evaluate-Print-Loop), a tool used primarily to do "small tasks." In the early days of ScalaTest, I wasn't sure how people would want to use a test framework in the Scala interpreter, so I did nothing. I then waited to see what users tried to do with ScalaTest in the Scala interpreter, and what they said they wanted.

I finally got a spark of inspiration a few months ago from a post to the JUnit mailing list by Joakim Ohlrogge. Joakim had done some experimenting to see how he could make it easier to use JUnit from the Scala interpreter, which he released as poju4s. This user feedback pointed me in a design direction, and the ScalaTest Shell was born.

The main command of the ScalaTest shell is run, which you can use to run a suite of tests. The shell also provides several commands for configuring a call to run:

color - display results in color (green for success; red for failure; yellow for warning; blue for statistics)
nocolor - display results without color
durations - display durations of (i.e., how long it took to run) tests and suites
nodurations - do not display durations of tests and suites
shortstacks - display short (i.e., truncated to show just the most useful portion) stack traces for all exceptions
fullstacks - display full stack trackes for all exceptions
nostacks - display no stack trace for StackDepth exceptions and a short stack trace for non-StackDepth exceptions
stats - display statistics before and after the run, such as expected test count before the run and tests succeeded, failed, pending, etc., counts after the run
nostats do not display statistics before or after the run

The default configuration is color, nodurations, nostacks, and nostats.

All of these commands are methods of class org.scalatest.Shell. Each configuration command is a method that produces another Shell instance with every configuration parameter the same except for the one you've asked to change. For example, when you invoke durations, you'll get back a Shell instance that has every parameter configured the same way, except with durations enabled. When you invoke run on that, you get a run with durations enabled and every other configuration parameter at its default value.

Two other useful "commands" to know about, though not technically part of the shell, are the apply factory methods in the Suites and Specs singleton objects. These allow you to easily create composite suites out of nested suites, which you can then pass to run. This will be demonstrated later in this documentation.

Using the ScalaTest shell

The package object of the org.scalatest package extends Shell with all its parameters set to their default values. A good way to use the ScalaTest shell, therefore, is to import the members of package org.scalatest:

scala> import org.scalatest._
import org.scalatest._

One thing importing org.scalatest._ allows you to do is access any of ScalaTest's classes and traits by shorter names, for example:

scala> class ArithmeticSuite extends FunSuite with matchers.ShouldMatchers {
     |   test("addition works") { 
     |     1 + 1 should equal (2)
     |   } 
     |   ignore("subtraction works") {
     |     1 - 1 should equal (0)
     |   }
     |   test("multiplication works") {
     |     1 * 1 should equal (2) 
     |   }
     |   test("division works") (pending)
     | } 
defined class ArithmeticSuite

But importing org.scalatest._ also brings into scope the commands of the Shell, so you can, for example, invoke run without qualification:

scala> run(new ArithmeticSuite)
ArithmeticSuite:
- addition works
- subtraction works !!! IGNORED !!!
- multiplication works *** FAILED ***
  1 did not equal 2 (<console>:16)
- division works (pending)

Configuring a single run

To configure a single run, you can prefix run by one or more configuration commands, separated by dots. For example, to enable durations during a single run, you would write:

scala> durations.run(new ArithmeticSuite)
ArithmeticSuite:
- addition works (102 milliseconds)
- subtraction works !!! IGNORED !!!
- multiplication works *** FAILED *** (36 milliseconds)
  1 did not equal 2 (<console>:16)
- division works (pending)

To enable statistics during a single run, you would write:

scala> stats.run(new ArithmeticSuite)
Run starting. Expected test count is: 3
ArithmeticSuite:
- addition works
- subtraction works !!! IGNORED !!!
- multiplication works *** FAILED ***
  1 did not equal 2 (<console>:16)
- division works (pending)
Run completed in 386 milliseconds.
Total number of tests run: 2
Suites: completed 1, aborted 0
Tests: succeeded 1, failed 1, ignored 1, pending 1
*** 1 TEST FAILED ***

And to enable both durations and statistics during a single run, you could write:

scala> durations.stats.run(new ArithmeticSuite)
Run starting. Expected test count is: 3
ArithmeticSuite:
- addition works (102 milliseconds)
- subtraction works !!! IGNORED !!!
- multiplication works *** FAILED (36 milliseconds)***
  1 did not equal 2 (<console>:16)
- division works (pending)
Run completed in 386 milliseconds.
Total number of tests run: 2
Suites: completed 1, aborted 0
Tests: succeeded 1, failed 1, ignored 1, pending 1
*** 1 TEST FAILED ***

The order doesn't matter when you are chaining multiple configuration commands. You'll get the same result whether you write durations.stats.run or stats.durations.run.

To disable color, use nocolor:

scala> nocolor.run(new ArithmeticSuite)
ArithmeticSuite:
- addition works
- subtraction works !!! IGNORED !!!
- multiplication works *** FAILED ***
  1 did not equal 2 (<console>:16)
- division works (pending)

To enable short stack traces during a single run, use shortstacks:

scala> shortstacks.run(new ArithmeticSuite)
ArithmeticSuite:
- addition works (101 milliseconds)
- subtraction works !!! IGNORED !!!
- multiplication works *** FAILED *** (33 milliseconds)
  1 did not equal 2 (<console>:16)
  org.scalatest.TestFailedException:
  ...
  at line2$object$$iw$$iw$$iw$$iw$ArithmeticSuite$$anonfun$3.apply$mcV$sp(<console>:16)
  at line2$object$$iw$$iw$$iw$$iw$ArithmeticSuite$$anonfun$3.apply(<console>:16)
  at line2$object$$iw$$iw$$iw$$iw$ArithmeticSuite$$anonfun$3.apply(<console>:16)
  at org.scalatest.FunSuite$$anon$1.apply(FunSuite.scala:992)
  at org.scalatest.Suite$class.withFixture(Suite.scala:1661)
  at line2$object$$iw$$iw$$iw$$iw$ArithmeticSuite.withFixture(<console>:8)
  at org.scalatest.FunSuite$class.invokeWithFixture$1(FunSuite.scala:989)
  ...
- division works (pending)

Changing the default configuration

If you want to change the default for multiple runs, you can import the members of your favorite Shell configuration. For example, if you always like to run with durations and statistics enabled, you could write:

scala> import stats.durations._
import stats.durations._

Now anytime you simply call run, statistics and durations will be enabled:

scala> run(new ArithmeticSuite)
Run starting. Expected test count is: 3
ArithmeticSuite:
- addition works (9 milliseconds)
- subtraction works !!! IGNORED !!!
- multiplication works *** FAILED *** (10 milliseconds)
  1 did not equal 2 (<console>:18)
- division works (pending)
Run completed in 56 milliseconds.
Total number of tests run: 2
Suites: completed 1, aborted 0
Tests: succeeded 1, failed 1, ignored 1, pending 1
*** 1 TEST FAILED ***

Running multiple suites

If you want to run multiple suites, you can use the factory methods in either the Suites or Specs singleton objects. If you wrap a comma-separated list of suite instances inside Suites(...), for example, you'll get a suite instance that contains no tests, but whose nested suites includes the suite instances you placed between the parentheses. You can place Suites inside Suites to any level of depth, creating a tree of suites to pass to run. Here's a (contrived) example in which ArithmeticSuite is executed four times:

scala> run(Suites(new ArithmeticSuite, new ArithmeticSuite, Suites(new ArithmeticSuite, new ArithmeticSuite)))
Run starting. Expected test count is: 12
Suites:
ArithmeticSuite:
- addition works (0 milliseconds)
- subtraction works !!! IGNORED !!!
- multiplication works *** FAILED *** (1 millisecond)
  1 did not equal 2 (<console>:16)
- division works (pending)
ArithmeticSuite:
- addition works (1 millisecond)
- subtraction works !!! IGNORED !!!
- multiplication works *** FAILED *** (0 milliseconds)
  1 did not equal 2 (<console>:16)
- division works (pending)
Suites:
ArithmeticSuite:
- addition works (0 milliseconds)
- subtraction works !!! IGNORED !!!
- multiplication works *** FAILED *** (0 milliseconds)
  1 did not equal 2 (<console>:16)
- division works (pending)
ArithmeticSuite:
- addition works (0 milliseconds)
- subtraction works !!! IGNORED !!!
- multiplication works *** FAILED *** (0 milliseconds)
  1 did not equal 2 (<console>:16)
- division works (pending)
Run completed in 144 milliseconds.
Total number of tests run: 8
Suites: completed 6, aborted 0
Tests: succeeded 4, failed 4, ignored 4, pending 4
*** 4 TESTS FAILED ***

Running a single test

The run command also allows you to specify the name of a test to run and/or a config map. You can run a particular test in a suite, for example, by specifying the test name after the suite instance in your call to run, like this:

scala> run(new ArithmeticSuite, "addition works")
ArithmeticSuite:
- addition works

Give it a try

I released an updated ScalaTest-1.5-SNAPSHOT today that includes the new ScalaTest shell. The snapshot works with Scala 2.8. You can download the snapshot release via the scala-tools.org Maven repository with:

group id: org.scalatest
artifact id: scalatest_2.8.1
version: 1.5-SNAPSHOT

Or you can just grab the jar file from:

http://www.scala-tools.org/repo-snapshots/org/scalatest/scalatest_2.8.1/1.5-SNAPSHOT/scalatest_2.8.1-1.5-SNAPSHOT.jar

I put the Scaladoc for this snapshot release up here:

http://www.artima.com/docs-scalatest-1.5-SNAPSHOT-5-May-2011/

These enhancements will be released as part of ScalaTest 1.5 within the next few weeks. I'm posting this preview now because I want to get feedback in general on the API and find if there are any bugs to fix or any code-breakages. (I expect no source code to break with any of this enhancement, so let me know if you have a problem.) So please give it a try and either post feedback to the discussion forum for this blog post, or email the scalatest-users mailing list.

Talk Back!

Have an opinion? Readers have already posted 12 comments about this weblog entry. Why not add yours?

RSS Feed

If you'd like to be notified whenever Bill Venners adds a new entry to his weblog, subscribe to his RSS feed.

Digg |

del.icio.us |

About the Blogger

Bill Venners is president of Artima, Inc., publisher of Artima Developer (www.artima.com). He is author of the book, Inside the Java Virtual Machine, a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Active in the Jini Community since its inception, Bill led the Jini Community's ServiceUI project, whose ServiceUI API became the de facto standard way to associate user interfaces to Jini services. Bill is also the lead developer and designer of ScalaTest, an open source testing tool for Scala and Java developers, and coauthor with Martin Odersky and Lex Spoon of the book, Programming in Scala.


	Web Artima.com