This post originated from an RSS feed registered with Ruby Buzz by Ryan Davis. Original Post: Functional Test Matrix Feed Title: Polishing Ruby Feed URL: http://blog.zenspider.com/index.rdf Feed Description: Musings on Ruby and the Ruby Community... Latest Ruby Buzz Posts Latest Ruby Buzz Posts by Ryan Davis Latest Posts From Polishing Ruby

This is an idea I've been tossing around for a while. Akin to Ward's FIT, how do you make testing complex specifications with many edge cases clearer?

I have something similar to the idea below that I'm slowly evolving towards this idea. I'm not entirely sold on it, but it does seem to cut down on the amount of code I have to write and how I have to think about it. What do you think? Suggestions?

The Idea:

This is supposed to get us thinking about the various dimensions our testing should address. If there are states orthogonal to each other (eg. readable vs unreadable, logged in vs not logged in) each of those states should comprise a dimension in the matrix. By addressing it this way, we should be able to minimize the amount of setup/teardown code and get full coverage across our actions for all these edge cases and as a result have extremely clear tests.

Example Test Matrix Specification:

setups          :edge1, edge2, :edge3, ...
matrix :action1,  :new,  :err,   :err, ...
matrix :action2,  :del,  :err,    :na, ...
matrix ...

setups:

I envision the setups being a code that combines the different dimensions of edge case state.

Something for a CMS might look like: [df][uga][rRwW] where:

• [df] for dir/file
• and the rest is in the style of symbolic args to chmod.
• lowercase X == Xable, uppercase X == unXable, where X is read/write

matrix:

:new/:err/:del are just examples, they should have semantic info attached to them.

Edge cases specific to an action that fall outside the matrix are regular tests.

How it Fits Together:

Setups stores off the edge cases. Matrix creates a test method for every applicable (read: a non :na result) in the (rough) form of:

def test_#{action}_#{setup}
  matrix_setup_configuration #{setup}.split(//) # global setup
  matrix_setup_#{action} #{setup}, #{expected}  # action setup + execution
  matrix_test_#{expected}, #{setup}             # expected verification
end