Test First Design is an integral component of Extreme Programming and generally leads to designs that are simpler and more error-free than otherwise. Although it works well in most cases, I've come across cases where Test First Design encounters some difficulties.

The first difficulty of Test First Design is one I call "Even I don't know the answer". Test First Design starts with the assumption that you write the tests first. To write the test, you have to write the asserts that check that the answer is correct. The problem is, what if you don't know the answer? It may sound strange, but sometimes, you don't know the answer until your program gives it to you.

I once tried to develop a ray tracing program similar to POVRay using test first design. The end result of a ray tracer is an image. Without having a ray tracing program in the first place, it's impossible to specify the image which the ray tracer is supposed to produce (except, of course, in trivial cases like scenes without a lightsource). In this case, you can do lower-level testing on the individual steps in creating the image, but even then, the only way to verify the result is to mimic the calculations that the code performs. Doing this manually is sometimes close to impossible.

Sometimes, the answers you expect are actually the wrong answers. I recently wrote a Tic Tac Toe game that had an unbeatable strategy. One testcase I wrote was to calculate the response for 'O' to the board:

X O _
_ X _
_ _ _   
 

The obvious answer is to play the bottom right corner. My program, however, was recommending the top left corner. Why? If X plays the bottom left corner, it can force a win regardless what O plays. My program calculated that any play would result in losing the game. It just picked one losing scenario that ended the game sooner rather than later. Perhaps the strategy could be improved to prolong the game or to hope that the other player makes a mistake, but the answer was valid for the algorithm I implemented. Unexpected, perhaps, but valid.

I once implemented a physics program (ElastoLab) that could simulate gravity, springs, air resistance and rockets. Once again, the only way to determine the answers to the simulation are to manually perform the same calculations that the code under test has to perform. In this case, the calculations for adaptive runge kutta on a 3D system are prohibitive to do without a computer. The program under test can do it, but that hasn't been written yet, so what can you do?

Sometimes, you have to resort to the following strategy:

1. Write the test with a dummy result
2. Watch the test fail
3. Write the code
4. Watch the test fail again
5. Verify visually that the result appears correct
6. Capture the result of the code into the test
7. Run the test and watch it pass

It's not true test first design, but it's about all you can really do in some cases.