Summary
Tonight I released a new version of ScalaTest, which contains a few changes to the way in which you express assertions and a Suite discovery feature. This release contains all the features that I plan to have in version 1.0, so it is ready for a few months of beta testing.
Advertisement
In the latest version of ScalaTest, I adjusted the default way to do assertion-based testing, and completed a Suite discovery mechanism for Runner.
In the discussion about the previous release of ScalaTest, and on other forums and mailing lists, the difficulty of remembering which parameter (to methods like JUnit's assertEquals) was expected and which actual was brought up. In the previous version, the === operator was defined such that actual was on the left, and expected on the right. This turned out to be the opposite order from JUnit, which I didn't like. Michael Feathers said that because === was a palindrome, it made him expect its semantics would be symmetric, but that the "Expected 2, but got 1" reporting made it asymmetric. Elizabeth Wiethoff pointed out that Python's message didn't care about the order of parameters, and didn't mention expected or actual values. It would just say, 2 != 1. Elizabeth also expressed her preference for the Python approach.
This sent me back to the drawing board. In the current release, === is symmetric. Here's an example:
val a = 1
val b = 2
assert(a === b)
The failure report for the above code will simply say that "1 did not equal 2". I also created an expect construct that looks like this:
val a = 4
val b = 3
expect(2) {
a - b
}
In this example you indicate that you expect the value 2 to be the result of the code in the curly braces. The failure message from this code will read, "Expected 2, but got 1". In the expect construct, it is clear which value is expected. The expect construct works very well as expressions get longer, whereas the assert is most readable when the expressions are shorter. The expect construct also looks very much like the intercept construct, which checks that expected exception type is thrown. Here is a test method from ScalaTest's own tests that shows interceptions and expectations doing their thing:
Here are some assertions from a different test method:
def testSimpleNameForTest() {
val s = new SuiteFriend(new Suite {})
assert(s.simpleNameForTest("testThis") === "testThis")
assert(s.simpleNameForTest("testThis(Reporter)") === "testThis")
assert(s.simpleNameForTest("test(Reporter)") === "test")
assert(s.simpleNameForTest("test") === "test")
}
The other major change in this release is that I completed the Suite discovery feature. Suite discovery was implemented in SuiteRunner (the Java testing tool from which ScalaTest sprung), but back then we did discovery when you pushed a button in the GUI. I ended up never using that feature, and long ago realized it should be on the command line. I never added command-line support to SuiteRunner, but just did so to ScalaTest. You can now request Suites to be discovered on the runpath with the -m and -w command line arguments to Runner.
ScalaTest is now "feature complete" for the 1.0 release. I want to release it 1.0 around JavaOne week. Prior to that, I'm hoping to get feedback from real users to fix bugs and potentially make other breaking changes like I did this week. I also plan to work to integrate with ScalaCheck, JUnit, TestNG, and JUnit 4, and make a FuncSuite in which tests are represented by function values. If ScalaTest is up for all that, I figure it's ready for anything, and I'll release it 1.0. So please download ScalaTest and give it a try. Post feedback either in the forum topic for this blog post or in the ScalaTest Forum.
Keep up the good work; Scala is getting to be an ever more compelling place to be. It would be great if someone with the time and knowledge could start a living document (e.g. wiki) about the testing tools available for Scala, including ScalaTest and ScalaCheck.
I looks like he's using a BDD style, in which he has fine-grained methods and name that express a little specification. His names don't start with "test". I think the best way to do that in ScalaTest would be to use the functional Suite I plan on adding. I don't know what to call it. FunctionalSuite is too verbose. FuncSuite is kind of cute because it sounds like Funk. But it also sounds a bit close to another 4-letter word that starts with f. I'm considering calling it FuncSuite because each test is represented as a function value. Anyway, the code would look something like this:
gen = new Generator super.runTest(testName, reporter, stopper, properties) }
test("expect new generator to be on") { assert(gen.isOn) assert(!gen.isOff) }
test("expect turned off generator is off") { gen.turnOff assert(!gen.isOn) assert(gen.isOff) }
test("when the generator is turned off make sure connection is notified") { // given val mockPowerSource = createMockWithConnectionTo // when gen.turnOff // then verify(Array(mockPowerSource)) }
test("when the generator is turned off make sure all connections are notified") { // given val powerSourcesToBeNotified: List[PowerSource] = List( createMockWithConnectionTo, createMockWithConnectionTo, createMockWithConnectionTo )
// when gen.turnOff
// then powerSourcesToBeNotified.foreach(verify(Array(_))) }
test("last power source in chain should be notified when first in chain is turnedoff") {
//given val first = new Generator val secondToLast = new Wire
This won't compile given that FuncSuite doesn't exist yet. But what it seems would be a pain is writing such long method names in camel case. This would allow those "specifications" to be expressed more naturally in a String.
The way it would work is that "test" here is a method call, and you're passing in two things. You pass a String, which is the test name, such as "expect new generator to be on", and a function, which shows up between the curly braces. So:
test("the test name") { // the test code as a function object assert(1 === 2) }
This method call would be made by the constructor of the GeneratorTest, because any raw code you place in between the curly braces of a class go into the constructor. The method "test" would be defined in FuncSuite. What "test" would do is register the function for execution later, when execute is called on FuncSuite.
The way the fixture is reinitialized is by overriding runTest is ugly, but in way it is purposely so. This approach uses mutable state (reinitializing the gen var) which while not evil is something I'd prefer people shy away from. What I'd rather do is come up with some way for people to pass a fixture into each test function. Scala's type system is probably rich enough to do that in some way, but I haven't had time to think about it. But if so you could avoid overriding runTest and get rid of the gen var.
> I don't know what > to call it. FunctionalSuite is too verbose. FuncSuite is > kind of cute because it sounds like Funk. But it also > sounds a bit close to another 4-letter word that starts > with f. I'm considering calling it FuncSuite because each > test is represented as a function value. > Hey, if this Suite were primarily used to facilitate Behavior Driven Development (BDD), then maybe it could be called BSuite. That's kind of cute too, though not very obvious. If it is about writing "specifications", then it could be called SpecSuite, and instead of "test" it could be called "spec" or "specify".
specify("expect new generator to be on") { assert(gen.isOn) assert(!gen.isOff) }
FYI, Eric Torreborre is working on a Scala BDD framework called Specs, here:
> Hey, if this Suite were primarily used to facilitate > Behavior Driven Development (BDD), then maybe it could be > called BSuite. That's kind of cute too, though not very > obvious. If it is about writing "specifications", then it > could be called SpecSuite, and instead of "test" it could > be called "spec" or "specify". > I woke up this morning with the name: FunSuite. It could potentially have both "test" and "specify" methods so people can use whichever they prefer. I figure it could also have an "ignore" method so that to ignore a test or spec, you change "test"/"specify" to "ignore". It could also take a variable length list of case classes of some kind to specify groups. Whereas you can use annotation on test methods in Suites to place a test into groups, as in;
@SlowAsMolasses def testNetworkTimeout() { // do something that takes a while }
You'd have to put a spec say in a group like this:
specify("nework should timeout after 5 minutes", SlowAsMolasses) { // do something that takes a while }
But if you wanted to turn this spec off for a while, you'd replace "specify" with "ignore":
ignore("nework should timeout after 5 minutes", SlowAsMolasses) { // do something that takes a while }
Which I think is more readable than adding an ignore group at the end.
The functional way to do the fixture would be to make a control abstraction, like withFixture. For example, Josh Cough's tests that require a generator could be done like this:
def withFixture(f: (Generator) => ()): Unit { val gen = new Generator f(gen) }
test("expect new generator to be on") { withFixture { (gen) => { assert(gen.isOn) assert(!gen.isOff) } } }
test("expect turned off generator is off") { withFixture { (gen) => { gen.turnOff assert(!gen.isOn) assert(gen.isOff) } } }
There are no vars in site here. No mutable data, except that gen is mutable. Potentially FunSuite itself could be parameterized with the type of the fixture, and then execute could automatically pass invoke each test through the withFixture method. So instead of making a setUp and tearDown method, you'd make a withFixture method. That way you wouldn't have to call withFixture explicitly from your test functions.
gen = new Generator super.runTest(testName, reporter, stopper, properties) }
Is all your setup happening here? This seems like a bit of clutter.
Is it possible to match all of TestNG's annotations? Here are two quick examples:
@BeforeMethod becomes:
beforeTest("create generator"){ gen = new Generator }
@AfterSuite becomes:
afterSuite( "final teardown" ){ whatever... }
I haven't though everything yet so I'm not sure how you're doing discovery, but I think things would look a lot cleaner if you didn't have to override runTest, and used an approach like this.
Also, I blogged a month or so ago that I think we should have a pluggable xUnit IDE plugin. I noticed ScalaTest has its own little UI, specs has nothing, rehersal and scunit also have nothing...TestNG has everything, which is why I wanted to use it on Scala to begin with.
If we had a pluggable plugin (yeah, you read that right) we could all just plug in our particular framework into the IDE plugin and we wouldn't wouldn't all have to write our own each time we create a new framework.
Is anyone else in agreement with me on this? Or am I just nuts? Maybe its too difficult, I don't know, I'm not a UI guy at all, but I do get the feeling that lack of IDE support is going to hold all these new frameworks back for some time.
Cedric, you have the best support so far, do you think it would be possible to pull something out that everyone could use?
> Bill, > > You're right, the camel casing is a pain. I like the way > you have it much better. > > I have a question about this method: > > override protected def runTest(testName: String, > g, reporter: Reporter, > stopper: Stopper, properties: Map[String, Any]) { > > gen = new Generator > super.runTest(testName, reporter, stopper, > per, properties) > } > > Is all your setup happening here? This seems like a bit of > clutter. > The clutter is really in the number of arguments to the method and he super call. The reason I didn't support an @BeforeMethod or setUp method or something like that is I think it would lead people in the direction of reassigning variables. That's what you do in your test class, and I did in my "refactor" of it in a previous post here. I'd rather the framework lead people in the direction of using vals everywhere. I'll make a post after this one showing a couple ways to do that.
> Is it possible to match all of TestNG's annotations? Here > are two quick examples: > > @BeforeMethod becomes: > > beforeTest("create generator"){ gen = new Generator } > > @AfterSuite becomes: > > afterSuite( "final teardown" ){ whatever... } > Yes, I decided against that in Suite, for the reason I described above. I am planning a TestNGSuite, in which you can write a TestNG test class like you've done but use ScalaTest's assertion and property check stuff inside the test methods, and run it with either TestNG or ScalaTest. I also plan one for JUnit and JUnit 3. All three of those do something like setUp, @BeforeTest, etc. So people who want to go that route will be able to do so.
> > I haven't though everything yet so I'm not sure how you're > doing discovery, but I think things would look a lot > cleaner if you didn't have to override runTest, and used > an approach like this. > I'll show one way here. What you could do is make a control abstraction, a method named withGenerator or withFixture. You would define it like this:
def withFixture(f: (Generator) => Unit) { val gen = new Generator f(gen) }
This defines a method that takes a function, which has as its single parameter is a Generator, and returns Unit (like void in Java). What the method does is create the new fixture object, the new Generator, then passes it to the function. (Remember the function itself takes a Generator as its single parameter. Because the f parameter is "unnamed", you can call it like this:
withFixture { (gen) => { // some code } }
So it looks like a control abstraction. And all you need to do then is do that inside your test methods that actually need the fixture, like this:
test("expect new generator to be on") { withFixture { (gen) => { assert(gen.isOn) assert(!gen.isOff) } } }
What happens here is that the code of your test gets bundled up into a little function value that gets passed to the withFixture method, which instantiates a new Generator and passes that to your code as it invokes it. I'll follow up with a post that will show you how to get rid of the explicit withFixture call.
In the previous post I showed this approach to doing fixture without reassigning variables:
> test("expect new generator to be on") { > withFixture { > (gen) => { > assert(gen.isOn) > assert(!gen.isOff) > } > } > }
What I have in mind is to make it such that you can say this instead:
testWithFixture("expect new generator to be on") { (gen) => { assert(gen.isOn) assert(!gen.isOff) } }
I haven't written this much yet, though I have sketched it out in code. (By the way, Scala is quite handy for sketching out ideas in code.) I have a FunSuite that does the test("test name here") { /* test code here */ } thing. Then I have a FunSuite1 that takes a type parameter, which is the type of a single fixture object. If you have two fixture objects, you'd use FunSuite2, etc. So what you'd really do is parameterize it with the type of the fixture, as in:
class Generator(val max: Int)
class GenSuite extends FunSuite1[Generator] {
// Here you implement an abstract withFixture method inherited from FunSuite1
def withFixture(f: (Generator) => Unit) { val gen = new Generator(5) f(gen) }
// FunSuite1 extends FunSuite, so you can call plain old test still
test("this is a test") { assert(1 === 1) }
test("this is only a test") { assert(1 === 2) }
// If you call testWithFixture, though, the function passed will need to have // one argument of type Generator, the type with which you parameterized // FunSuite1 above.
testWithFixture("this is a test with fixture") { (gen) => { assert(gen.max === 5) } }
testWithFixture("this is only a test with fixture") { (gen) => { assert(gen.max === 2) } } }
For reference, here's a bit of FunSuite1:
abstract class FunSuite1[F] extends FunSuite {
protected def withFixture(f: F => Unit) // this is abstract // ... }
The thing to note here is that the type parameter to FunSuite1 is used as the parameter type to the function passed to withFixture. FunSuite2 would look like this:
abstract class FunSuite2[F, G] extends FunSuite {
protected def withFixture(f: (F, G) => Unit) // this is abstract // ... }
And so on.
Now I figure this will look quite foreign to most people coming from Java, which is one big reason I don't even bring this up in the documentation for Suite, which is the default. Although a tad cluttery, people can figure out how to override runTest. It is quite easy for Java folks to understand I think. If someone prefers @BeforeClass, then they can use TestNGSuite or JUnit4Suite. Those are perfectly usable tools and I will encourage their use in Scala. The basic Suite is really similar to JUnit 3, which according to Cedric most people are still using, so I think ScalaTest will be easy to get into.
As people learn more about Scala, some of them may prefer to adopt a more functional style, and that's what FunSuite and FunSuiteN will be for.
By the way, one other reason the FunSuite approach is not the default in Suite is that Suite is a trait, but FunSuite must be a class because it must maintain state. Because Suite is doing reflection looking for methods starting with "test", it can just look the up each time and need not keep track of them. But the little test function values that you register with test("...") {} in FunSuite must be stored. Thus FunSuite must be a class.
> Is it possible to match all of TestNG's annotations? Here > are two quick examples: > > @BeforeMethod becomes: > > beforeTest("create generator"){ gen = new Generator } > > @AfterSuite becomes: > > afterSuite( "final teardown" ){ whatever... } > On the other hand, I'll bet a lot of people coming from Java would be more comfortable with setUp and tearDown-like methods. So maybe I should also have an ImpSuite (for "imperative suite"), that offers those methods. If you want them, you'd just extend ImpSuite instead of Suite. I'd probably use the method names I used for SuiteRunner, which were setUpFixture, tearDownFixture, setUpSuite, tearDownSuite.
One other way I remembered this morning that you can do a fixture without a var is by simply making a createFixture method and calling it at the beginning of each test that needs it. For Generator it is a bit overkill, because you just need one object each time. You could create it like this:
test("expect new generator to be on") { val gen = new Generator assert(gen.isOn) assert(!gen.isOff) }
test("expect turned off generator is off") { val gen = new Generator gen.turnOff assert(!gen.isOn) assert(gen.isOff) }
If you have, say, three objects in your fixture, you'd make a createFixture method. Here's how it looks in the interpreter:
def createFixture = (new Generator, new Radiator, new SparkPlug)
test("expect new generator to be on") { val (gen, rad, sp) = createFixture assert(gen.isOn) assert(!gen.isOff) }
test("expect turned off generator is off") { val (gen, rad, sp) = createFixture gen.turnOff assert(!gen.isOn) assert(gen.isOff) }
You'd just call createFixture at the top of each method that needs it, and initialize as many variables you need in one swoop. Remember also that you only need to do this for objects that mutate. A lot of times Scala will lead folks to use immutable objects, which you can just define as instance variables in the Suite class. There would be no need to make a createFixture method for the List and Map I used in the interpreter for example, because those are immutable. You can safely reuse them in 1000 test methods if you want. Just define them as instance variables:
private val myList = List(1, 2, 3) private val myMap = Map(1 -> "hi", 2 -> "ho")
Anyway, the createFixture approach has a bit of code duplication, but not too painful as it is just one line per test method. It is also explicit which test functions (in this case, but this also works for test methods in the basic Suite trait) are using a fixture and which ones not. And you could have multiple createFixture methods also, if different test methods need different fixtures. I think this approach is easy for imperative-minded folks to grok as well. There is a need for documentation of all this stuff. As things progress I plan to publish lots of articles and a book about how to use it. In that documentation I'll recommend a preferred style so the many choices hopefully won't bewilder people.
I could have easily done this in my test. I probably should have, but for some reason I was on a kick to remove all the duplication. Its probably more readable your way, given that its only one line per test method, and you don't have to look around for where the gen actually came from. Its declared in one place, instantiated in another, and used all over.
You say for just generator its overkill, but I think the overkill is probably worth it.
> I could have easily done this in my test. I probably > should have, but for some reason I was on a kick to remove > all the duplication. Its probably more readable your way, > given that its only one line per test method, and you > don't have to look around for where the gen actually came > from. Its declared in one place, instantiated in another, > and used all over. > > You say for just generator its overkill, but I think the > overkill is probably worth it. > OK. Maybe the createFixture approach should be the recommended one, even for just one fixture object. It does look explicit and easy to read, and would certainly be easy to understand. And it avoids the use of a var, which the more cluttery approach of overriding runTest does not.
Previous Topic
