• Roy Osherov's 'blog;
• Sadek Drobi's 'blog;
• Martin Jul's 'blog.

This 'blog was inspired by Grig Gheorghiu's 'blog on on Agile Testing that agonized over the question of whether to do TDD or acceptance testing. My initial reply, on that site, provides the foundation on which the following post builds.

The Wrong Question

We seem to be looking for the right checklist that will give us quality. The checklist at hand has TDD and acceptance testing on it and we're asked to check zero, one, or two boxes.

I think it's the wrong question.

The objective is to deliver what the customer wanted while optimizing quality; the broader goal is to create value. Quality suffers if you misunderstand what the customer wanted, or if the code has internal interactions that are difficult to foresee, or if local code violates array boundaries or uses untethered pointers. It also suffers if the interface makes it possible for the user to commit errors, or if it takes too many keystrokes to enter a small amount of common information. Testing covers only a fraction of these and does so in the least efficient way known to our industry.

For some reason, we have invented and are following religions, or perhaps even cults, intent on ceremonies and rituals rather than thinking or inspection. We sometimes even throw in platitudes that are to pass for "values," but there is little talk about the kind of value that an end user would care about. We now even need to make it explicit that there are methodologies that encourage "inspect and adapt," having long ago forgotten that the word "software" contains the word "soft." One role of religion is to remind us what is important in matters of identity and relationship which, curiously enough, both figure prominently in software development. After my talk at JavaZone Trygve Reenskaug came up to me and said, "You just gave a talk where you highlighted the importance of the user interface and of domain knowledge. And you had to tell them that!"

Discussions in this area inspire passion. Why? If I were to attack 40-hour week people would just roll their eyes. The sober and sobering evaluations of On-Site Customer haven't led to a revolt. Why TDD? Gertrud Bjørnvig and I were talking about this the other day and agree that it could be because TDD is the one aspect of Agile that is personal. People have tied their Agile success to their introduction of TDD practices into their work. It's the one thing they can do as individuals rather than as an enterprise. Pair programming comes close, and if it weren't for the fact that it works at least half the time, any research that criticised it would likely be received with equal emotional content. It becomes a religious issue.

Religious Movement

Religion can be grounded in ceremonies and rituals that long have lost any tie to value, if they ever had such ties at all. The industry is at an uncomfortable juncture where we cannot and should not take some things for granted, and yet we do. We are told that you have to do TDD to be a professional (in another great Trygve moment and in response to this passionate plea, Trygve and I gleefully agreed that we must not be professionals) and yet are given no reason to believe this, no substantiation, and no evidence. Just believe.

(Of course, one problem with religions is that there are so many of them. In our recent explorations of TDD we have found that it means so many things to so many different people that it's no surprise that there is resentment and confusion. We simply aren't communicating that. Watch for Roy Osherove to pick up that thread in his 'blog.)

These days, all things testing seem to be moving to the forefront of these religious debates. The 'blog I pointed to above that led me to write this one found itself in the middle of this debate: Are you of the test-last religion or the TDD religion? The answer (credible, but unsupported by data) ended being a universal platitude of: "Do both." Why are we focusing so much on testing, or even on TDD in particular, which even Kent Beck defines as a state-of-mind kind of thing that extends to concepts such as spending a week iterating a paper design and then test-driving the code? (Yes, he uses exactly those words.)

Raising the Bar

I can tell you where my current experience--including what I've garnered from a broad range of clients and from academic publications alike--points. Focus on quality assurance instead of testing. That means using Use Cases (which are an Agile way of engaging customers—in the sense of your full constituency, not just the sacrificial victim the company sends as the On-Site Customer) instead of XP-style user stories (so you understand feature interactions up-front), doing pair programming or desk-check code inspections, doing design by contract, driving the process with usability experts and doing good usability testing, and a host of other things.

On one hand we should be encouraged to use every weapon in our arsenal; on the other hand, we'll never get done if we try to do everything. It then becomes a matter of cost effectiveness. Integration and system testing have long been demonstrated to be the least efficient way of finding bugs. Recent studies (Siniaalto and Abrahamsson) of TDD show that it may have no benefits over traditional test-last development and that in some cases has deteriorated the code and that it has other alarming (their word) effects. The one that worries me the most is that it deteriorates the architecture. And acceptance testing is orders of magnitude less efficient than good old-fashioned code inspections, extremely expensive, and comes too late to keep the design clean.

Jef Raskin often said that the "interface is the program." If we're going to do testing, that's where the focus should lie. The interface is our link to the end user. Agile is about engaging that user; TDD, about engaging the code. TDD is a methodology. TDD is about tools and processes over people and interactions. Woops. Agile pundits go so far as to marginalize the end-user-facing side of software (read the first and foremost point of the articulated Philosophy. Perhaps Philosophy is supposed to be more objective than religion).

Whither Systems Engineering?

Sorting these things out requires a system view. It used to be called "systems engineering," but the New Age Agile movements consistently move away from such practices. It shows in the quality of our interfaces, in the incredible code bloat we have today relative to two decades ago, and in the fact that we keep getting stuck in practices that look good from a simple analysis, but which fall apart when you look at them closer. Pair programming has been getting mixed results (summarization by Abrahamsson); TDD is under attack (North, who calls TDD "a series of blind alleys"; Siniaalto and Abrahamsson, who call the findings about TDD "alarming"; Bjørnvig, Coplien and Harrison, who suggest practices that can help); on-site customer has also been at least partially discredited (Martin and Noble).

The answer for industry, I think, is to focus on Quality and to find the techniques that work best and are the most cost-effective for your project. Just answering "both" to the "TDD-or-acceptance" question is not only at the wrong scope, it is missing the whole point.

The Amazing Shrinking Academic Collective Mind

It's about thinking, not about checklists. I think that one root of the problem lies in modern education, which has increased its focus on technique and reduced its focus on thinking. I also think that because of a hot job market more and more people are getting into the industry before having had enough exposure to academia to properly develop their thinking tools. Fewer and fewer job applicants have Masters' degrees; fewer and fewer students in Denmark wait to finish even their undergraduate before succumbing to the lure of money and their chance to make a mark on the world. Also, fewer and fewer programmers understand software history. Not knowing design-by-contract or Clean Room, they never know to even ask the question of whether TDD can be as effective as they can in exactly the same improvements to code.

There was a time when good schools afforded their graduates at least the basics of systems engineering thinking. The notion of a "liberal education" used to be valued, even for computer scientists. I haven't heard these words in a good long while. At North Central College, where I was recently a full-time Associate Professor, management's most-discussed idea of progress was to move to a Java curriculum. I think their idea of thinking skills reduced largely to a course in first-order logic. Students in first-year programming classes taught by my colleagues have stronger memories of being corrected on their commenting conventions and on exactly when to use --v and when to use v-- than they recall anything about logical design.

...and a place for industry?

Such grounding, of course, makes it difficult for industry to rise to the challenge I posed above: to focus on Quality and on the thought process rather than on techniques. But industry itself is much to blame for the state of academia. Middle-tier academics pride themselves not on academic excellence (which is impossible to measure, in spite of anyone who would claim to the contrary) but on their placement of graduates. As someone who has been both on the hiring end and on the "brain-factory" side, I can tell you that placement has much to do with unwritten contracts between industry and academica. North Central made a big deal about its relationship with local industry, and it appealed annually to local companies to ask what skill sets would make new hires employable. That's where Java curricula come from (or C++ curricula, or C# curricula). It almost makes one take one's hat off in salute to those institutions who base their instruction on Scheme.

I don't think there is much we can do to help the poor ambassadors that industry sends to academia to ask schools to do their bidding. I do think that as academics, we can educate the ambassadors. We can also hold our ground. That requires that time-honored but fading academic value of courage. It takes more courage to be a good educator these days than perhaps it did in the 1960s, when anything that could be labeled as technologically intellectual passed for good academics. It is probably as difficult as it was for Newton who, in the face of predominate views in a broad and powerful religion, had difficulty prevailing with new insights on the working of the universe. In fighting the broad misunderstandings that industry has chosen to adopt from selected interpretations of Agile, SOA, and programming language buzzwords, we face no less a religion, and no less threat to our cozy industry-financed academic positions, than he did.

I have sympathy with those who comment (as Grig does in his 'blog) that this is not an academic problem but a problem of industry. I simply think it's too late at that point, and I don't know of any good solutions when things have come that far. One solution lies in industrial training programs. Many of those are in the hands of consultants brought on board to teach a given tool or testing technique, and the likelihood that such engagements will lead to a culture of thinking are small. Too few organizations have license to freely explore alternative techniques and to truly evolve. There are places where inspect-and-adapt doesn't work; it is such places where paradigm shifts have their place. Just as many interesting advances in biology took place not through the classical Darwinian model but through discontinuities, so we need platforms that enable practitioners to drop what they are doing and to completely re-direct their efforts. That happens infrequently enough in industrial projects for me to not make it my first focus.