Summary
What can you learn from testing? When you look beyond the red and the green, the fail and the pass, you can learn a lot more about the nature of the code and the nature of the problem domain. And there is a lot to learn — software development is called knowledge work for a reason.

---

Advertisement

Software development is often described as knowledge work. This label is invariably used as a shorthand for "work that doesn't involve getting your hands dirty", "jobs your parents never had and you struggle to explain to them" or, without any apparent irony over the use of keyboards, "the digital economy". But step back from these simple substitutions and an obvious yet deeper truth becomes visible: knowledge work is about knowledge. It's about knowing stuff and, most often, also about how you deal with stuff you don't know.

By far and away the most popular response to ignorance is to ignore it. Sound good? Feel free to stop reading! You must have stumbled across this article by mistake.

OK, you're still here. In that case I'll assume curiosity on your part. Which is a good thing: curiosity is key to how we address a lack of knowledge. Questions are the agents of curiosity. Even without answers, questions can help us to increase and refine our knowledge, to learn.

Software testing is a form of learning. A set of tests can be considered a set of questions. The most obvious question a test poses is "Does the code pass?" to which there are two simple answers: yes or no. A test allows us to move from belief to knowledge — for example, to move from merely believing something works to knowing that, in a particular context, it does. Even limiting the scope of testing to just this question and these two answers reveals a rich little set of possible outcomes:

• It passes, which is what we expected.
• It passes and we are surprised, as this is not what we expected.
• It fails, which is what we expected.
• It fails and we are surprised and disappointed, as this is not what we had hoped or expected.

The reasons for failure (or success) run even deeper: a test might fail because the test is at fault, not the code; a test might pass because both the test and the code are at fault and in sympathy; and so on.

And that's just what we can learn from a simple red or green! Of course can is not will — having the opportunity is not the same as taking it — but the first part of any feedback-based process is generating the feedback; what you do with it becomes the next challenge.

But more learning opportunities are available from testing: in the formulation rather than the execution of tests.

What does it mean if a test is hard to write? In particular, a unit test. Difficulty in writing unit tests is a principal demotivator among programmers trying to get into unit testing, causing many to get out of it. "We tried unit testing, but it took too much effort" is a common lament. The more experienced practitioner will typically rejoin with "if your code is difficult to test, it means your code is messy", a comeback that is not necessarily unreasonable but also not necessarily correct. At best it is a provocative oversimplification intended to make you reflect. At worst it may prevent you from learning about your assumptions and the nature of your work.

The difficulty in being able to write a test can be boiled down to the two broad themes of complexity and ignorance, each manifested in a couple of different ways:

• The essential complexity of the problem being solved. Such complexity is intrinsic to the problem domain and is not the fault of the software architecture, the code quality or, indeed, any of those who have worked on the code, past or present. If you're looking for someone to blame try the problem domain or the customer. Perhaps the customer would prefer a "Hello, World" app instead of software to control the rail network? That would be one way to eliminate the essential complexity, but probably not what the customer needed. The reality is that some things just are harder to code and test no matter how you write the code or the tests. When complexity was being handed out, not all applications and domains of interest were created equal.

• The accidental complexity of the problem being solved. Such complexity is an artefact of the way the software has been developed, external to the nature of the problem. It is non-essential and, in principle, avoidable. This is the realm of technical debt, baroque frameworks and rococo architectures. It's where speculative generality has come home to roost, copy-and-paste code has blossomed and coupling has weighed anchor. This is where the observation "if your code is difficult to test, it means your code is messy" may often apply.

• Uncertainty over what the code should actually do. One of the most common responses to the question of why one should test one's code is "to show that it works". This offers all the illusion of being a satisfactory answer without any of the substance. It leaves us with a lingering, killer question: What exactly do we mean by it works? What is it actually supposed to do? In short, what's the spec? Tests may be difficult to write because we don't actually know exactly what we want. We may understand the gist, but not the detail. We may apparently be able to write the code — having followed the gist we may have elaborated something in code that followed and meandered along that gist — but writing a test throws our ignorance into the sharp relief of cold light. Or perhaps not: being unaware of our own ignorance is a common unawareness and ignorance, a difficult-to-see blindspot. This kind of difficulty in testing is an open invitation to talk to someone else, to clarify and find out more.

• Lack of testing know-how. Programmers may be unaware of or unfamiliar with the techniques necessary to make a particular test easier, or even possible. If programmers don't know about test doubles (mocks, stubs, etc.), what are the odds they will reinvent and rediscover this approach? Without help, would a typical programmer know how to test outputs whose sequence is non-deterministic or collections whose ordering is unspecified? What about testing the output of an image processing algorithm, such as lossy compression or edge detection? The difficulty of testing may reflect testing skills we need to acquire or techniques we need to improvise.

Of course, each of these aspects is not entirely independent from the others: a challenging domain may often require novel techniques for testing; uncertainty over the functionality may reflect a complex or, by being ill-defined, seemingly complex domain; and so on. Nonetheless, by recognising these four aspects we can learn more about our assumptions, our knowledge, our ignorance and the nature of our work than either writing off tests as too hard to be worthwhile or assuming that messy code is necessarily the root cause.

Testing? It's an education.

Talk Back!

Have an opinion? Readers have already posted 6 comments about this weblog entry. Why not add yours?

RSS Feed

If you'd like to be notified whenever Kevlin Henney adds a new entry to his weblog, subscribe to his RSS feed.

Digg | del.icio.us | Reddit

About the Blogger

Kevlin is an independent consultant and trainer based in the UK. His development interests are in patterns, programming, practice and process. He has been a columnist for various magazines and web sites, including Better Software, The Register, Application Development Advisor, Java Report and the C/C++ Users Journal. Kevlin is co-author of A Pattern Language for Distributed Computing and On Patterns and Pattern Languages, two volumes in the Pattern-Oriented Software Architecture series. He is also editor of the 97 Things Every Programmer Should Know site and book.