Summary
The 3.0.0 release of ScalaTest is coming soon. It is a major release with significant new features. In this post, I give a preview of the new features, and ask for feedback.

---

Advertisement

This week I released ScalaTest and Scalactic 3.0.0-M10 for Scala 2.10 and 2.11 and Scala.js 0.9.5. If you have time please try this release out on your projects and post feedback to scalatest-users@googlegroups.com or the ScalaTest gitter channel. Here's the dependency for an sbt build:

libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0-M10" % "test"

The main new feature of ScalaTest 3.0 is Scala.js support. We support all aspects of ScalaTest that can reasonably work on Scala.js. A few features are missing, such as the have and be matchers that take symbols--for example, bird should have ('color(blue))--and the Spec trait, which use reflection. These continue to work on the JVM as before. Other than a few such features that don't make sense or aren't possible on Scala.js, all of ScalaTest is available for testing Scala.js applications.

ScalaTest is the most full-featured test framework available on Scala.js. Its ability to run in both worlds means you test code that you run both on the JVM and on JavaScript with tests that you run both on the JVM and JavaScript.

Another major new feature in ScalaTest 3.0, which was in great part motivated by Scala.js, is async testing styles. ScalaTest 3.0 will offer an async flavor of each style trait in which tests are functions registered at construction time. For example in addition to FunSuite, there's now an AsyncFunSuite. In addition to FeatureSpec, there's now an AsyncFeatureSpec, and so on for the other test-as-function styles. The difference between FunSuite and AsyncFunSuite is that in FunSuite, the result type of a test is "Assertion", whereas in AsyncFunSuite, it is Future[Assertion].

Assertion is a type alias, new in 3.0, for org.scalatest.Succeeded.type, the type of the Succeeded singleton. Succeeded was added in ScalaTest 2.0 to represent a successful test Outcome. In 2.0 ScalaTest's withFixture method signature was modified to take a test function with result type Outcome as its parameter and to return an Outcome itself. This allows users to execute shared code at the beginning and/or end of a test, execute test bodies multiple times, or change the outcome of tests under certain condition--all without needing to catch exceptions. In 3.0, all the assertions and matcher expressions have been changed to return the Succeeded singleton instead of the Unit value singleton, which was the result in ScalaTest 1.x and 2.x. They still throw the same exceptions when a problem is detected, but when they succeed they return a different singleton object. I expect this change break no existing code in practice, because in earlier versions of ScalaTest the result type of tests was Unit or Any, essentially, and so any result of an assertion or matcher expression was normally ignored. Here's an example from the REPL using 3.0.0-M10:

scala> val x = 1
x: Int = 1

scala> assert(x > 0)
res0: org.scalatest.Assertion = Succeeded

scala> assert(x == 0)
org.scalatest.exceptions.TestFailedException: 1 did not equal 0
  at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:507)
  at org.scalatest.Matchers$.newAssertionFailedException(Matchers.scala:7743)
  at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:472)
  ... 43 elided

scala> x should be > 0
res2: org.scalatest.Assertion = Succeeded

scala> x shouldEqual 0
org.scalatest.exceptions.TestFailedException: 1 did not equal 0
  at org.scalatest.MatchersHelper$.newTestFailedException(MatchersHelper.scala:148)
  at org.scalatest.MatchersHelper$.indicateFailure(MatchersHelper.scala:358)
  at org.scalatest.Matchers$AnyShouldWrapper.shouldEqual(Matchers.scala:6792)
  ... 43 elided

You write traditional (synchronous) tests just like you always did, and all existing code should compile and run exactly as before, but in async styles you must return from your tests a Future[Assertion]. The purpose of async styles is to allow you to write tests in a non-blocking manner. On the JVM this enables you to write tests consistently with production code in reactive frameworks like Akka and Play. On Scala.js it enables testing asynchronous code that would not otherwise be possible to test using ScalaTest blocking constructs (like ScalaFutures), because in JavaScript you *can't* block.

In an async test you work in futures. Given, for example, a Future[Int] that has been returned by some code under test, you can map an assertion onto it like this:

scala> import scala.concurrent.Future
import scala.concurrent.Future

scala> import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.ExecutionContext.Implicits.global

scala> val fut = Future { 1 }
fut: scala.concurrent.Future[Int] = scala.concurrent.impl.Promise$DefaultPromise@52c293d0

scala> fut map { x => x should be > 0 }
res4: scala.concurrent.Future[org.scalatest.Assertion] = scala.concurrent.impl.Promise$DefaultPromise@1ca1e995

This value of type Future[Assertion] you can return to ScalaTest by leaving it as the result of your async test. When that future completes, the test will either succeed or fail. This future assertion would result in a successful test:

scala> res4.value
res5: Option[scala.util.Try[org.scalatest.Assertion]] = Some(Success(Succeeded))

Here's one that would result in a failed test:

scala> fut map { x => x shouldEqual 0 }
res6: scala.concurrent.Future[org.scalatest.Assertion] = scala.concurrent.impl.Promise$DefaultPromise@18daf92c

scala> res6.value
res7: Option[scala.util.Try[org.scalatest.Assertion]] =
        Some(Failure(org.scalatest.exceptions.TestFailedException: 1 did not equal 0))

You can include non-async tests that result in Assertion in an async style alongside your tests that result in Future[Assertion]. The final expression of such tests will be converted to a Future[Assertion] that has already succeeded. At run time, the test will either throw an exception or return that successful Future[Assertion].

The traditional styles like FunSuite and FeatureSpec also have changed the test result type. In ScalaTest 1.x and 2.x the result type of tests was Unit. In 3.0 it will be Assertion. However, we have a trait named Compatibility that is currently in effect by default that provides an implicit conversion from Any to Assertion. This will keep all old code working by providing a kind of value discarding similar to what the compiler does for Unit. The Compatibility "registration policy" offers convenience if you're using 3rd party tools that provide assertion syntax the returns the Unit value, such ask Akka's testkit of any of several Java mocking frameworks. If you prefer more type safety, you can mix in trait Safety, which will hide that Any => Assertion implicit. Any type errors you get you can fix by ending your test with Succeeded, or what I will recommend, simply succeed. The "succeed" construct is a new one in ScalaTest 3.0 that returns the Succeeded singleton.

scala> succeed
res8: org.scalatest.Succeeded.type = Succeeded

One issue I would like to hear user's opinions on is whether Compatibility or Safety should be the default. My feeling is that safety should be the default in async styles, because no prior code exists that would break, and I fear Compatibility might be error prone when futures are involved. If safety (AsyncSafety actually) is the default in async styles, then I am thinking Safety should be the default in traditional styles for consistency. That would break a lot of code, so I'd do that through a long deprecation cycle. But still it would require everyone to either mix in Compatibility into their base classes, if they have base classes, or actually fix the type errors one way or another. To give you one example of the kind of type error you might encounter, ScalaTest's intercept construct, if it appears last in a test body, would give a type error because its result type is Throwable, not Assertion, because it returns the caught exception if the assertion succeeds.

scala> intercept[IndexOutOfBoundsException] { "hi".charAt(-1) }
res9: IndexOutOfBoundsException = java.lang.StringIndexOutOfBoundsException: String index out of range: -1

You could fix this type error by placing "succeed" after intercept, but we added a new construct to Assertions in ScalaTest 3.0 called assertThrows, which has the same behavior as intercept except it returns the Succeeded singleton. So you could instead replace intercept with assertThrows:

scala> assertThrows[IndexOutOfBoundsException] { "hi".charAt(-1) }
res10: org.scalatest.Assertion = Succeeded

That could end up being quite being quite a few changes to get rid of deprecation warnings, so please give your opinions on whether Compatibility or Safety should be the default.

The BeforeAndAfter, BeforeAndAfterEach, and BeforeAndAfterEachTestData traits do work with async styles, but you will need to synchronize access to shared mutable state between "before" and "after" code and test code. So I will recommend people override withAsyncFixture by default instead. The withAsyncFixture method takes a test function that returns a Future[Outcome] and itself returns Future[Outcome]. You can execute "beginning-of-test" code before invoking the test function, then map "end-of-test" code onto the returned Future[Outcome], or just register the end-of-test code as a call back to be executed when the future completes.

Although tests in async styles run assertions in futures, by default tests will execute one at a time in order. The second test won't start until the first test completes. The third test won't start until the second test completes, and so on. If you want the async tests to run in parallel, you mix in ParallelTestExecution to the test class, just as with traditional styles.

ScalaTest 3.0.0-M10 has a handful of deprecations compared to the current, 2.x release, but should only have two breaking changes. First, one of the overloaded "thread" methods in Conductor would no longer overload after changing the result type of tests to Assertion. It has been renamed to threadNamed in 3.0. I will likely release a 2.2.x version of ScalaTest with that deprecated prior to the 3.0 final release. Also, if someone had been invoking the whenCompleted method of Status, that now takes a Try[Boolean] => Unit instead of just a Boolean => Unit. Otherwise no existing ScalaTest 2.x code should break (so long as you clear all deprecation warnings before upgrading). Please let me know if you see a breakage when trying 3.0.0-M10 in your projects.

Thanks and happy testing.

Talk Back!

Have an opinion? Readers have already posted 6 comments about this weblog entry. Why not add yours?

RSS Feed

If you'd like to be notified whenever Bill Venners adds a new entry to his weblog, subscribe to his RSS feed.

Digg | del.icio.us | Reddit

About the Blogger

Bill Venners is president of Artima, Inc., publisher of Artima Developer (www.artima.com). He is author of the book, Inside the Java Virtual Machine, a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Active in the Jini Community since its inception, Bill led the Jini Community's ServiceUI project, whose ServiceUI API became the de facto standard way to associate user interfaces to Jini services. Bill is also the lead developer and designer of ScalaTest, an open source testing tool for Scala and Java developers, and coauthor with Martin Odersky and Lex Spoon of the book, Programming in Scala.