Summary
The 2.0.M6 milestone release of ScalaTest is coming soon. It is a major release with many new features. In this post, I give a preview of the new features, and ask for feedback.

---

Advertisement

I wanted to give a sneak preview of ScalaTest 2.0.M6, which we will release for both Scala 2.10 and 2.9, and get some feedback. I was hoping to get M6 out before ScalaDays, but we didn't quite make it. Since we missed that self-imposed deadline we will give it a bit more time to finish baking. If you want to try some of the new features described here, you can use the 2.0.M6-SNAP21 release that we made yesterday. I would like to get feedback so please post comments, suggestions, constructive criticism, etc., to either the discussion forum here or the scalatest-users mailing list.

In the remainder of this article I'll give highlights, with links to the 2.0.M6-SNAP21 Scaladoc documentation you can go to for more detail. 2.0.M6 will be a very major release with a lot of changes and additions, but as usual we have tried very hard to make it as easy as possible to migrate existing code.

ScalaUtils

The first big change is ScalaTest 2.0.M6 will include a package named org.scalautils. This package is focused on aspects of equality, normalization, and tolerance that may make sense to use in your production code as well as your tests. The org.scalautils package is included in the ScalaTest jar, but is also available separately in a ScalaUtils jar. The idea is you can add a dependency on ScalaUtils in your production classpath, so you can use it in your production code without having ScalaTest on the classpath. I think people should never include ScalaTest in production code, because when it doesn't like something it tends to throw a TestFailedException. ScalaUtils will always be released alongside ScalaTest, with version numbers that match exactly. So when ScalaTest 2.0.M6 is released, for example, ScalaUtils 2.0.M6 will also be released.

Equality

ScalaUtils 2.0.M6 allows you to customize what equality means for a type T by defining an Equality[T] typeclass. This can be used with the === operator in production code and assertions and the "should equal" and the new-in-2.0.M6 "should ===" matcher syntax in tests.

ScalaUtils defines a DefaultEquality that will be used by default for any type T for which an implicit Equality[T] is not otherwise available. The default equality will first call the deep method on an array on either the left or right side (or both sides), then use == to determine equality. This default is compatible with ScalaTest's existing === operator and equal and be matchers, so existing test code will continue to work as before.

Equality constraints

ScalaUtils 2.0.M6 also provides a way to get a compile-time type error if two types being compared with === don't adhere to a tunable EqualityConstraint. By default you can compare any type with any type (which means all existing uses of ScalaTest's === continue to compile and work as before). But you can use traits such as TypeCheckedTripleEquals, ConversionCheckedTripleEquals, TraversableEqualityConstraints to get a compile error when comparing, for example, an Option[String] with a String for equality. (Equality and EqualityConstraint are inspired in part by the Equal typeclass in scalaz.)

Tolerance

ScalaUtils 2.0.M6 also includes a Tolerance trait that facilitates checking tolerances of numeric types with a +- symbolic operator. The plusOrMinus operator used with matchers previously in ScalaTest has been deprecated in 2.0 in favor of the +- symbol, which is also now more broadly usable. In addition to:

result should be (2.0 +- 0.1)

You can now also use the syntax with the equal matcher:

result should equal (2.0 +- 0.1)

Or just with plain-old ===, in an assertion in tests or just as part of a Boolean expression in production code:

// In tests:
assert(result === (3.0 +- 0.2))

// In production code:
if (result === (2.0 +- 0.1)) "close enough" else "sorry"

Normalization

ScalaUtils 2.0.M6 also contains a Normalization trait that allows you to define strategies for normalizing types. The NormalizingEquality trait allows you to define an Equality[T] in terms of a normalization of that type T by first normalizing the left and, if also a T, the right hand side, then using a specified "after-normalization Equality" to compare the normalized values.

Explicitly

ScalaUtils 2.0.M6 also includes an Explicitly DSL for defining an implicit Equality parameter explicitly. If you want to customize equality for a type in general, you would usually want to place an implicit Equality[T] for that type in scope (or in T's companion object). That implicit equality definition will then be picked up and used when that type is compared for equality with the equal matcher in tests and with === in both tests and production code. If you just want to use a custom equality for a single comparison, however, you may prefer to pass it explicitly. For example, if you have an implicit Equality[String] in scope, you can force a comparison to use the default equality with this syntax:

// In production code:
if ((result === "hello")(decided by defaultEquality)) true else false

// In tests:
result should equal ("hello") (decided by defaultEquality)

The explicitly DSL also provides support for specifying a one-off equality that is based on a normalization. For example, ScalaUtils has a StringNormalizations trait that provides trimmed and lowerCased methods that return Normalization[String]s that normalize by trimming and lower-casing, respectively. If you bring those into scope by mixing in or importing the members of StringNormalizations, you could use the explicitly DSL like this:

// In production code:
if ((result === "hello")(after being lowerCased)) true else false

// In tests:
result should equal ("hello") (after being lowerCased and trimmed)

Inspectors

ScalaTest 2.0.M6 includes an Inspectors trait that allows you to make assertions about (to inspect) collections, including deeply nested collections:

• forAll - succeeds if the assertion holds true for every element
• forAtLeast - succeeds if the assertion holds true for at least the specified number of elements
• forAtMost - succeeds if the assertion holds true for at most the specified number of elements
• forBetween - succeeds if the assertion holds true for between the specified minimum and maximum number of elements, inclusive
• forEvery - same as forAll, but lists all failing elements if it fails (whereas forAll just reports the first failing element)
• forExactly - succeeds if the assertion holds true for exactly the specified number of elements

We worked hard to craft error messages that will help you understand what went wrong when an inspection against a nested collection fails. For example, given this nested collection:

val xss =
  List(
    List(1, 2, 3),
    List(1, 2, 3),
    List(1, 2, 3)
   )

And this inspection:

forAll (xss) { xs =>
  forAll (xs) { y => y should be < 2 }
}

Your error message will be:

TestFailedException: forAll failed, because:
  at index 0, forAll failed, because:
    at index 1, 2 was not less than 2 (<console>:20)
  in List(1, 2, 3) (<console>:20)
in List(List(1, 2, 3), List(1, 2, 3), List(1, 2, 3))
     at ...

Inspector shorthands

If you are using Matchers, you can use "inspector shorthands" for inspecting top-level (i.e., non-nested) collections:

• all - succeeds if the assertion holds true for every element
• atLeast - succeeds if the assertion holds true for at least the specified number of elements
• atMost - succeeds if the assertion holds true for at most the specified number of elements
• between - succeeds if the assertion holds true for between the specified minimum and maximum number of elements, inclusive
• every - same as forAll, but lists all failing elements if it fails (whereas forAll just reports the first failing element)
• exactly - succeeds if the assertion holds true for exactly the specified number of elements

Here are some examples:

all (xs) should be > 0
atMost (2, xs) should be >= 4
atLeast (3, xs) should be < 5
between (2, 3, xs) should (be > 1 and be < 5)
exactly (2, xs) should be <= 2
every (xs) should be < 10

org.scalatest.Matchers

Speaking of matchers, in ScalaTest 2.0.M6 ShouldMatchers and MustMatchers, both members of package org.scalatest.matchers have been deprecated in favor of Matchers, which resides in package org.scalatest. For folks using ShouldMatchers, getting rid of the deprecation warning should be as simple as replacing org.scalatest.matchers.ShouldMatchers with org.scalatest.Matchers. For folks using MustMatchers, however, it will unfortnately be slightly more trouble, because the new Matchers trait only supports should not must. So you will need to search and replace your uses of must with should. MustMatchers and must will continue to work for a good long deprecation period, but eventually it will be removed to make way for must possibly coming back later to serve a different purpose. Apologies for this rather large deprecation.

Contain matchers

The contain matcher syntax has been greatly enhanced in 2.0.M6, in part inspired by similar syntax in specs2. In addition to simple assertions about containership, like:

xs should contain ("hi")

You can also make assertions such as:

xs should contain oneOf (1, 2, 3)
xs should contain noneOf (1, 2, 3)
xs should contain atLeastOneOf (1, 2, 3)
xs should contain allOf (1, 2, 3)
xs should contain only (1, 2, 3)
xs should contain inOrderOnly (1, 2, 3)
xs should contain inOrder (1, 2, 3)
xs should contain theSameElementsAs List(1, 2, 3)
xs should contain theSameElementsInOrderAs List(1, 2, 3)

In the new contain syntax, "containership" is determined by Equality, so you can customize how containership is determined for a type T by via an implicit Equality[T] or the explicitly DSL. Here's an example:

(result should contain oneOf ("hi", "he", "ho")) (after being trimmed)

Implicit conversions => implicit parameters

We are currently in the middle of a major refactor of how matchers are implemented. Since ScalaTest 1.0 matchers have been implemented primarily with implicit conversions, and as of 2.0.M6 we have started migrating to implicit parameters instead. This will make it simpler to enable the syntax on your own types, because you will be able to enable syntax by defining the typeclass "enabler" for that syntax for your own type.

For example, the contain (<value>) syntax is enabled for type T by an implicit Containing[T]. ScalaTest 2.0.M6-SNAP21 provides implementations for GenTraversable, java.util.Collection, java.util.Map, Array, Option, and String in the Containing companion object. These enable the contain syntax to be used on more types than in previous versions of ScalaTest, such as strings and options:

Some("hi") should contain ("hi")
"hello" should contain ('e')

Despite the major reorganization of matchers implementation, existing matchers user code should continue to work fine in all but very rare cases.

Outcomes

One breaking change in 2.0.M6 is that the withFixture life-cycle methods, and the test functions passed to them, now return an Outcome. Because they previously returned Unit, this will likely require users to insert an = before the open curly brace, or in some cases, :Outcome =. That will most often be all that's required, but if you were catching exceptions in a withFixture implementation, you will likely need to change your catch clauses to a corresponding pattern match on Outcome types.

And more...

There's still more in 2.0.M6, but this post is probably long enough already. Here's a list of links to Scaladoc for new features you may find useful:

• Checkpoints - accumulate and report multiple assertions in a single test
• Informing - trait that contains the info method, which was previously provided via an implicit that has been removed in 2.0.M6
• Documenting - trait that contains the markup method, which was previously provided via an implicit that has been removed in 2.0.M6
• Sequential - combinator for nested suites that you want executed completely sequentially
• Stepwise - combinator for nested suites that you want executed one after another, but with parallel execution still enabled for the individual nested suites
• TryValues - for making assertions when you expect a Try to be a Success (2.10 only)
• ScalaFutures - write tests using Scala futures (2.10 only)
• org.scalatest.tags - new tags CPU, Disk, Network, and Slow
• FlatSpecLike, FunSpecLike, etc. - all style traits have been refactored into the original name as a class and the trait with the name with Like appended for speedy compiles
• Framework - an implementation of the new Framework API, which will make testing with sbt much more awesome in sbt 0.13

I'm at ScalaDays this week, so if you're also at the conference one other way to provide feedback is to find me. Otherwise please use the discussion forum attached to this blog (link below), or post to the scalatest-users mailing list.

Talk Back!

Have an opinion? Be the first to post a comment about this weblog entry.

RSS Feed

If you'd like to be notified whenever Bill Venners adds a new entry to his weblog, subscribe to his RSS feed.

Digg | del.icio.us | Reddit

About the Blogger

Bill Venners is president of Artima, Inc., publisher of Artima Developer (www.artima.com). He is author of the book, Inside the Java Virtual Machine, a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Active in the Jini Community since its inception, Bill led the Jini Community's ServiceUI project, whose ServiceUI API became the de facto standard way to associate user interfaces to Jini services. Bill is also the lead developer and designer of ScalaTest, an open source testing tool for Scala and Java developers, and coauthor with Martin Odersky and Lex Spoon of the book, Programming in Scala.