Today I released a revised ScalaTest 1.5 snapshot that contains a new API for table- and generator-driven property checks. The snapshot works with Scala 2.8. You can download the snapshot release via the scala-tools.org Maven repository with:

• group id: org.scalatest
• artifact id: scalatest
• version: 1.5-SNAPSHOT

Or you can just grab the jar file from:

http://www.scala-tools.org/repo-snapshots/org/scalatest/scalatest/1.5-SNAPSHOT/scalatest-1.5-SNAPSHOT.jar

I put the Scaladoc for this snapshot release up here:

http://www.artima.com/docs-scalatest-1.5-SNAPSHOT-3-Apr-2011/

The action is in the org.scalatest.prop package, which as of ScalaTest 1.3 contains one trait, Checkers, and one exception type. Trait Checkers facilitates property checks in a native ScalaCheck style. I have enhanced this trait by dropping the Suite self type so it can be mixed into non-Suites, added a companion object that mixes in the trait so you can import the members of Checkers as an alternative to mixing in the trait, and enhanced the way you can configure the property checks. If you used Checkers in the past, you'll need to recompile your tests, but otherwise make no changes. The enhancements to Checkers are compatible with existing source code.

In addition to Checkers, the org.scalatest.prop package now also sports a PropertyChecks trait (and companion object), which simply extends two other traits that have been added to prop: TableDrivenPropertyChecks and GeneratorDrivenPropertyChecks.

Checkers versus GeneratorDrivenPropertyChecks

GeneratorDrivenPropertyChecks is an alternative to Checkers. Both are essentially front-ends to ScalaCheck, and both require that ScalaCheck's jar file be on the class path. The difference is Checkers facilitates the traditional ScalaCheck style of writing properties, whereas GeneratorDrivenPropertyChecks facilitates a ScalaTest style of writing properties that takes advantage of ScalaTest's assertions and matchers. To see the difference, consider an example written in each style. With Checkers you'd usually write a property function like this:

check { (n: Int) =>
  n > 1 ==> n / 2 > 0
}

This is the native ScalaCheck style: you write a function that takes some data, then use that data in a boolean expression, returning the result. If the property is only valid for a subset of the full range of values of the types being passed (for example, in this case the expression only makes sense for integers greater than one), you would use the implication operator ==>. The code, n > 1 ==> in front of n / 2 > 0 essentially means that whenever n > 1, n / 2 should be greater than 0. In the ScalaTest property style you use the word whenever instead of ==> and either an assertion or matcher expression instead of a boolean expression:

forAll { (n: Int) =>
  whenever (n > 1) { n / 2 should be > 0 }
}

The two main goals I have for providing a ScalaTest style of writing properties are consistency and clarity. By allowing people to write properties similarly to how they write regular unit test, I'm hoping it will make property-based tests easier to write and read, and thereby make property-based testing more accessible to the many programmers who have never seen it before. That's the consistency.

Clarity is a general goal I have when I make design decisions for ScalaTest. When someone looks at some ScalaTest code written by someone else, to the extent possible I want them to be able to understand the intent of the programmer without having to look anything up in the ScalaTest documentation. I want it to be plainly obvious. The ==> operator does not pass that test, but whenever does. Also I think readers may find it hard to parse properties where => is near ==> because they look so similar. Moreover, things can get harder to read with the traditional ScalaCheck style when properties get more complex. Here's a Checkers example inspired by code in the ScalaCheck documentation describing ScalaCheck labels:

def myMagicFunction(n: Int, m: Int) = n + m

check { (m: Int, n: Int) =>
  val res = myMagicFunction(n, m)
  (res >= m)    :| "result > #1" &&
  (res >= n)    :| "result > #2" &&
  (res < m + n) :| "result not sum"
}

Like ==>, the :| operator doesn't pass the test of not needing to look things up in the documentation. What :| (and its cousin, |:) does is attach a string label to a part of the boolean expression. That way if a complex property fails, you can more easily figure out which part of the expression caused the failure. When you check the above labeled property it will fail with arguments (0, 0) and indicate the label of the failing part was "result not sum".

By contrast, the equivalent property in the ScalaTest style might look like this:

forAll { (m: Int, n: Int) =>
  val res = myMagicFunction(n, m)
  res should be >= m
  res should be >= n
  res should be < m + n
}

I think this style is easier to both write and read. When it fails, it tells you the failing arguments were (0, 0), gives you a failure message that reads, "0 was not less than 0," and provides the line number of the failing expression, which is res should be < m + n. Alternatively, you could write it this way:

forAll { (m: Int, n: Int) =>                  
  val res = myMagicFunction(n, m)
  res should (be >= m and be >= n and be < m + n)
}

This form would also fail indicating the args were (0, 0), giving the line number of the offending matcher expression and failure message that reads, "0 was greater than or equal to 0, and 0 was greater than or equal to 0, but 0 was not less than 0." So either way you write this property in ScalaTest style, it is pretty easy to figure out what went wrong, and you figure it out the same way you would for regular (non-property-based) tests.

Trait GeneratorDrivenPropertyChecks provides overloaded forAll methods for property functions with 1 to 6 arguments (the arities supported by ScalaCheck). It also provides overloaded forAll variants that take custom generators and/or string argument names. For information and examples of these variants, see the documentation starting with the Supplying argument names section. Trait GeneratorDrivenPropertyChecks also allows you to configure all property checks in an entire suite with an implicit parameter, and to override those "default" values on a case-by-case basis at individual forAll call sites. For information on configuration, see the Property check configuration section.

Table-driven property checks

In a table-driven property check, the data used to check the property is supplied not by generators but by a table. A table is a sequence of tuples. Each tuple represents one row of data. The first member of each tuple represents the first column, the second member represents the second column, and so on. All tuples in a table must have the same arity, which can be from 1 to 22 (though in the case of 1, it isn't a tuple, but simply an object). So you can define tables with 1 to 22 columns of data, and any number of rows. Tables also require that you specify string names for each column.

The org.scalatest.prop package includes a class for each table arity: TableFor1 is a table with 1 column, TableFor2 is a table with 2 columns, and so on, all the way up to TableFor22. Usually you'd use the factory method in object Table (a member of trait Tables, which is a supertrait of TableDrivenPropertyChecks) to create one. Here's an example:

val examples =
  Table(
    ("a", "b"),
    (1,   2),
    (2,   3),
    (3,   -4),
    (4,   5)
   )

Given this table, you could check a property against it with:

forAll (examples) { (a, b) => a should be < b }

This will fail, giving an error message with information to help you figure out what went wrong, such as:

Message: 3 was not less than -4
Location: SourceFile.scala:13
Occurred at table row 2 (zero based, not counting headings), which had values (
  a = 3,
  b = -4
)

A table is also a Seq of its tuple type, so you can do anything with its tuples that you can do with the elements of a Seq. For example, the org.scalatest.FailureOf trait offers a failureOf method that executes a function passed by-name and returns None if the function returns normally, or, if the function throws an exception, a Some wrapping that exception. Using this you can write:

import org.scalatest.FailureOf._

for ( (a, b) <- examples) yield failureOf { a should be < b }

This will give you a Seq[Option[Throwable]], with the values:

Seq(None, None, Some(org.scalatest.TestFailedException: 3 was not less than -4), None)

This way you can get the result of executing a property on each row of data. The exception for failed evaluations contains all the information such as row number of failed data, argument values, etc.

Some use cases for table-driven property checks that are described in the documentation are testing stateful functions, testing mutable objects, and testing invalid argument combinations. You can follow the links to gain more insight into how you might use table-driven property checks.

Give it a try

I showed some of the syntax of these property checks enhancements at the end of my Devoxx talk, Get Higher with ScalaTest, mentioning it hadn't been released yet. You can watch this talk to get the bigger picture of how this fits in with the larger goals of ScalaTest's design. These enhancements will be released as part of ScalaTest 1.5 in the next couple of months. I'm posting this preview now because I want to get feedback in general on the API and find if there are any bugs to fix or any code-breakages. (No source code should break with any of these enhancements, so let me know if you have a problem.) So please give it a try and either post feedback to the discussion for for this blog post, or email the scalatest-users mailing list.

Advertisement Next Scala training course: Applied and Advanced Scala: May 23rd - 27th 2011 at the Four Points Sheraton, Ann Arbor, MI. Register