Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

The `PropSpec` trait

PropSpec allows you to create a suite of property-based tests. It works the same as a FunSuite, but instead of test you write property, and instead of testsFor you write propertiesFor.

PropSpec supports both ScalaCheck and ScalaTest property styles (as well as any other property style you may wish to use it with.) If you want to write properties in the ScalaCheck style, mix Checkers into your PropSpec. If you want to write them in the ScalaTest style, mix in PropertyChecks. Here's an example that uses both generator- and table-driven property checks:

  import org.scalatest.PropSpec
  import org.scalatest.prop.PropertyChecks
  import org.scalatest.matchers.ShouldMatchers

  class FractionSpec extends PropSpec with PropertyChecks with ShouldMatchers {
  
    property("Fraction constructor normalizes numerator and denominator") {
  
      forAll { (n: Int, d: Int) =>
        whenever (d != 0 && d != Integer.MIN_VALUE
            && n != Integer.MIN_VALUE) {
  
          val f = new Fraction(n, d)
  
          if (n < 0 && d < 0 || n > 0 && d > 0)
            f.numer should be > 0
          else if (n != 0)
            f.numer should be < 0
          else
            f.numer should be === 0
  
          f.denom should be > 0
        }
      }
    }
  
    property("Fraction constructor throws IAE on bad data.") {
  
      val invalidCombos =
        Table(
          ("n",               "d"),
          (Integer.MIN_VALUE, Integer.MIN_VALUE),
          (1,                 Integer.MIN_VALUE),
          (Integer.MIN_VALUE, 1),
          (Integer.MIN_VALUE, 0),
          (1,                 0)
        )
  
      forAll (invalidCombos) { (n: Int, d: Int) =>
        evaluating {
          new Fraction(n, d)
        } should produce [IllegalArgumentException]
      }
    }
  }

For more information, see the Scaladoc documentation for PropSpec and my earlier post, ScalaTest Property Checks Preview. Trait PropSpec is the last piece of ScalaTest's new support for property-based testing described in that previous post.

Note: Trait PropSpec is in part inspired by class org.scalacheck.Properties, designed by Rickard Nilsson for the ScalaCheck test framework.

The `FreeSpec` trait

The other new style trait introduced in today's snapshot release is FreeSpec. Whereas ScalaTest's other specification-style traits facilitate writing text with certain grammatical structures using words like "when," "should," and "can," FreeSpec allows you to structure the text of your specification however you wish. You write a test in a FreeSpec with a string followed by in and a block of test code, just as you do in WordSpec and FlatSpec:

  "should pop values in last-in-first-out order" in {
    // ...
  }

You can surround tests with description clauses composed of a string, a dash character (-), and a block. Here's an example:

  "A Stack" - {
    "should pop values in last-in-first-out order" in {
      // ...
    }
  }

You can nest description clauses inside description clauses to any number of levels. Here's an example:

  import org.scalatest.FreeSpec

  class StackSpec extends FreeSpec {
    "A Stack" - {
      "whenever it is empty" - {
        "certainly ought to" - {
          "be empty" in {
            // ...
          }
          "complain on peek" in {
            // ...
          }
          "complain on pop" in {
            // ...
          }
        }
      }
      "but when full, by contrast, must" - {
        "be full" in {
          // ...
        }
        "complain on push" in {
          // ...
        }
      }
    }
  }

When run in the interpreter, you'd see:

scala> (new StackSpec).execute()
StackSpec:
A Stack
  whenever it is empty
    certainly ought to
    - be empty
    - complain on peek
    - complain on pop
  but when full, by contrast, must
  - be full
  - complain on push

Another use case for FreeSpec is writing specification-style test suites in a language other than English, as demonstrated here:

  import org.scalatest.FreeSpec
  
  class ComputerRoomRulesSpec extends FreeSpec {
    "Achtung!" - {
      "Alle touristen und non-technischen lookenpeepers!" - {
        "Das machine is nicht fuer fingerpoken und mittengrabben." in {
          // ...
        }
        "Is easy" - {
          "schnappen der springenwerk" in {
            // ...
          }
          "blowenfusen" in {
            // ...
          }
          "und poppencorken mit spitzen sparken." in {
            // ...
          }
        }
        "Das machine is diggen by experten only." in {
          // ...
        }
        "Is nicht fuer gerwerken by das dummkopfen." in {
          // ...
        }
        "Das rubbernecken sightseeren keepen das cottenpicken hands in das pockets." in {
          // ...
        }
        "Relaxen und watchen das blinkenlights." in {
          // ...
        }
      }
    }
  }

Running the above ComputerRoomRulesSpec in the Scala interpreter would give you:

scala> (new ComputerRoomRulesSpec).execute()
ComputerRoomRulesSpec:
Achtung!
  Alle touristen und non-technischen lookenpeepers!
  - Das machine is nicht fuer fingerpoken und mittengrabben.
    Is easy
    - schnappen der springenwerk
    - blowenfusen
    - und poppencorken mit spitzen sparken.
  - Das machine is diggen by experten only.
  - Is nicht fuer gerwerken by das dummkopfen.
  - Das rubbernecken sightseeren keepen das cottenpicken hands in das pockets.
  - Relaxen und watchen das blinkenlights.

The FreeSpec concept first saw light of day way back in ScalaTest 0.9.4 under the name SpecDasher. I released it already deprecated with a warning that I would remove it in 0.9.5, which I did. I only released it because I'd shown it in Programming in Scala, which had already gone to the printer. I wanted all the code in the book to work for the version of ScalaTest mentioned in the book. I removed it in 0.9.5 because I wasn't convinced I had figured out the best way to do it, and even if I figured that out, I only planned to add it if users actually convinced me it would be useful to them.

Eric Torreborre's specs framework always had a >> operator that provided a similar nesting ability to FreeSpec's dash character, but in the specs case >> was an alias for in, which meant tests (called "examples" in specs) were being nested not specification text (and you still could put a should on top). The specsy project by Esko Luontola came closer to the concept, and in fact Esko wrote a blog post about the troubles with pre-defined words, Choice of Words in Testing Frameworks. Over time I did get input from users that they would find value in this kind of trait, for example, in this scalatest-users discussion started by Sukant Hajra. And I got the opposite feedback, in a way, of seeing someone use trait Spec and completely ignoring the guiding structure. So now FreeSpec is here.

Formatting and indentation

Another user feedback that I got over time is that users really liked the formatted output of the BDD-style (Behavior-Driven-Development-style) traits, and wanted to see the same thing in the TDD-style (Test-Driven-Development-style) traits. I did things that way originally because making the output of running a test suite a more useful artifact was a push from the BDD folks. But the users gave me this feedback, so now even when you run a Suite, FunSuite, JUnitSuite, JUnit3Suite, or a TestNGSuite, you get nicely formatted output. For example, given this FunSuite:

  import org.scalatest.FunSuite
  
  class MySuite extends FunSuite {
  
    test("addition") {
      val sum = 1 + 1
      assert(sum === 2)
      assert(sum + 2 === 4)
    }
  
    test("subtraction") {
      val diff = 4 - 1
      assert(diff === 3)
      assert(diff - 2 === 1)
    }
  }

Running from the interpreter in ScalaTest 1.3 gives you:

scala> (new MySuite).execute()
Test Starting - MySuite: addition
Test Succeeded - MySuite: addition
Test Starting - MySuite: subtraction
Test Succeeded - MySuite: subtraction

But as of the latest ScalaTest 1.5 snapshot release will give you:

scala> (new MySuite).execute()
MySuite:
- addition
- subtraction

Another bit of user feedback was that people preferred to see the nested levels in the test class echoed in indentation levels in the output. ScalaTest has supported arbitrary levels of indentation since 1.0 in its event hierarchy, but didn't fire indentation of more than one level from any ScalaTest trait, not even from Spec, which allowed arbitrarily deep levels of nesting. My thought was that most often people wouldn't actually nest more deeply than two levels, and I felt that it was more readable to flatten two levels to one. Also I observated Ruby's RSpec tool, which inspired Spec's describe/it syntax, at the time flattened everything to one level. I wasn't sure which way to go, so I followed RSpec initially. Meanwhile Eric Torreborre indented specs output to match the input, and I observed that led some specs users to nest text more deeply, and even RSpec now indents its output this way. The ScalaTest users I asked about this for the most part indicated they would like the indented output. So now given this Spec:

  import org.scalatest.Spec
  
  class MySpec extends Spec {
    describe("A Stack") {
      describe("(when empty)") {
        it("should be empty") (pending)
        it("should complain on peek") (pending)
        it("should complain on pop") (pending)
      }
      describe("(when full)") {
        it("should be full") (pending)
        it("should complain on a push") (pending)
      }
    }
  }

Instead of getting the output you get with ScalaTest 1.3:

scala> (new MySpec).execute()
A Stack (when empty) 
- should be empty (pending)
- should complain on peek (pending)
- should complain on pop (pending)
A Stack (when full)
- should be full (pending)
- should complain on a push (pending)

As of the latest ScalaTest 1.5 snapshot, you will get:

scala> (new MySpec).execute()
MySpec:
A Stack 
  (when empty) 
  - should be empty (pending)
  - should complain on peek (pending)
  - should complain on pop (pending)
  (when full)
  - should be full (pending)
  - should complain on a push (pending)

For two levels of indentation, it isn't necessarily an improvement in readability, but at more levels it is more clearly a readability win. For example, given this WordSpec:

  import org.scalatest.WordSpec

  class ScalaTestGUISpec extends WordSpec {
  
    def theUser = afterWord("the user")
    def display = afterWord("display")
    def is = afterWord("is")
  
    "The ScalaTest GUI" when theUser {
      "clicks on an event report in the list box" should display {
        "a blue background in the clicked-on row in the list box" in {}
        "the details for the event in the details area" in {}
        "a rerun button" that is {
          "enabled if the clicked-on event is rerunnable" in {}
          "disabled if the clicked-on event is not rerunnable" in {}
        }
      }
    }
  }

Instead of the ScalaTest 1.3 output of:

scala> (new ScalaTestGUISpec).execute()
The ScalaTest GUI (when the user clicks on an event report in the list box)
- should display a blue background in the clicked-on row in the list box
- should display the details for the event in the details area
- should display a rerun button that is enabled if the clicked-on event is rerunnable
- should display a rerun button that is disabled if the clicked-on event is not rerunnable

As of the latest ScalaTest 1.5 snapshot, you'll get:

scala> (new ScalaTestGUISpec).execute()
ScalaTestGUISpec:
The ScalaTest GUI 
  when the user clicks on an event report in the list box 
    should display 
    - a blue background in the clicked-on row in the list box
    - the details for the event in the details area
      a rerun button that is 
      - enabled if the clicked-on event is rerunnable
      - disabled if the clicked-on event is not rerunnable

Give it a try

These enhancements will be released as part of ScalaTest 1.5 within the next few weeks. I'm posting this preview now because I want to get feedback in general on the API and find if there are any bugs to fix or any code-breakages. (I expect no source code to break with any of these enhancements, so let me know if you have a problem.) So please give it a try and either post feedback to the discussion for for this blog post, or email the scalatest-users mailing list.

Next Scala training course:

Applied and Advanced Scala: August 8th - 12th, 2011

in San Francisco, CA. Find out more

Previous Topic

Next Topic


	Web Artima.com

The PropSpec trait

The FreeSpec trait

Formatting and indentation

Give it a try

The `PropSpec` trait

The `FreeSpec` trait