What I Like About The Generated Tests

For one thing, I like the fact that, in addition to testing for the basic values (e.g. 10000 and 20000), JUnit Factory applied boundary value analysis (a best practice in testing) and created test cases just above and below the boundary values (e.g. 9999.999 and 10000.001). I should have those values in my tests.

In addition to boundary value testing, JUF applied a classic testing heuristic and used 0.0 as an input value. I like this because: 1) using zero for a numerical input value is a always a good testing idea and, 2) because it got me thinking and helped me realize that the existing code will do the right thing with 0.0, but it will also gladly accept a negative number for totSales and will return a negative commission. Now, call me paranoid or over-protective, but this method begs for some input checking.

By looking at the tests, I also realize that with this code I have a real problem with fractional cents both in the input and the output. The double data type is not ideal for representing dollars. I knew this all along, but seeing code like the following,

assertEquals("result", 2000.0003000000002, result, 1.0E-6);

in the generated tests made the problem more real and the need for a proper solution more urgent

How The Generated Tests Could Be Even Better

I like the fact that the tests used 0.0 as an input. But why didn’t JUF use some negative values? I would have also liked to see some very large number; this would have helped me to think about putting a reality-check upper bound on the input – before we accidentally pay a commission of several million dollars.

Why doesn’t JUnit Factory generate the tests I just described? It’s not a technical problem. For us it’s very easy to add a heuristic to generate additional tests with negative and/or large input values for the totSales parameter. The answer is that we are trying to find the proper balance between bare minimum and overkill in the number and types of tests JUnit Factory generates. This is one of the aspects that makes it experimental, and there are other default behaviors that are up for debate. A few of the many things we are trying to decide are:

• Should we make assertions on private fields? Some people believe that having to assert on private fields is an indication that there’s something wrong with your design. Others believe that testing trumps encapsulation.
• If a method has an object parameter, should we always generate a test using a null value? Some think that testing some methods with null is a waste of time – at best: “A null will never make this far”. Others have seen too many unexpected NullPointerExceptions percolate up to the end-user, and believe that having such a test might help developers think more carefully about their null handling behavior.
• If we can’t construct an object, should we automatically mock it? How far do we take mocking? Some believe that proper unit tests should make extensive use of mocks. Others believe that mocks are a weapon of last resort since they can hide serious problems between collaborating classes.

All good questions with strong arguments and proponents for both sides. Why don’t you give JUnit Factory a try yourself, with your own sample code, and let us know what you think. The simple web-based demo (http://www.junitfactory.com/demo/) gives you an opportunity to rate the generated tests (i.e. 1 to 5 stars) and also provide free-form text feedback. For the full JUnit Factory experience, you should download the Eclipse plug-in.

The way I see it, the best way to use automatically generated characterization tests is to consider them a starting point. These tests get my testing juices flowing and they make me think of cases I might have otherwise ignored. But, ultimately, I believe in taking control. Keep the test cases I like (and possibly edit them) and add my own. If some generated don’t make sense or don’t apply, I simply delete them from the set.

In this case, I combined my original tests with the generated tests and added a few of my own to characterize behavior for negative and very large input values. Below is the result:

public void testCalculateCommissionDue1() throws Throwable {
    double result = SalesUtil.calculateCommissionDue(-10000.00);
    assertEquals("result", -2000.00, result, 0.01);
}

public void testCalculateCommissionDue2() throws Throwable {
    double result = SalesUtil.calculateCommissionDue(-0.01);
    assertEquals("result", 0.0, result, 0.01);
}

public void testCalculateCommissionDue3() throws Throwable {
    double result = SalesUtil.calculateCommissionDue(0.0);
    assertEquals("result", 0.0, result, 0.01);
}

public void testCalculateCommissionDue4() throws Throwable {
    double result = SalesUtil.calculateCommissionDue(1000.00);
    assertEquals("result", 200.00, result, 0.01);
}

public void testCalculateCommissionDue5() throws Throwable {
    double result = SalesUtil.calculateCommissionDue(9999.99);
    assertEquals("result", 2000.00, result, 0.01);
}

public void testCalculateCommissionDue6() throws Throwable {
    double result = SalesUtil.calculateCommissionDue(10000.0);
    assertEquals("result", 2000.00, result, 0.01);
}

public void testCalculateCommissionDue7() throws Throwable {
    double result = SalesUtil.calculateCommissionDue(10000.01);
    assertEquals("result", 2000.00, result, 0.01);
}

public void testCalculateCommissionDue8() throws Throwable {
    double result = SalesUtil.calculateCommissionDue(19999.99);
    assertEquals("result", 5000.00, result, 0.01);
}

public void testCalculateCommissionDue9() throws Throwable {
    double result = SalesUtil.calculateCommissionDue(20000.0);
    assertEquals("result", 5000.00, result, 0.01);
}

public void testCalculateCommissionDue10() throws Throwable {
    double result = SalesUtil.calculateCommissionDue(20000.01);
    assertEquals("result", 5000.00, result, 0.01);
}

public void testCalculateCommissionDue11() throws Throwable {
    double result = SalesUtil.calculateCommissionDue(999999.99);
    assertEquals("result", 886999.99, result, 0.01);
}

Too many tests? Too few? Just right?

What about the pesky fractional cents problem? Is it acceptable to check the commission to the nearest cent?

Going forward, should we create a DollarAmount class instead of using a double type? Should we throw an exception for negative values or unrealistically large values?

As you can see tests, even those that are automatically generated, really help you think about current and potential problems. Sometimes they raise a lot of questions – good questions.

That’s it for this installment. I hope you found this detour into JUnit Factory interesting and that it motivated you to experiment with it yourself.

Also, if you have followed this series on characterization tests so far, please let me know what you think of it and what you’d like me to cover next.