Summary
Cédric Beust, creator of TestNG, responds to Alberto Savoia's recently proposed Testivus Manifestivus, warning us that extensive developer testing does not obliterate the need for a debugger.
Advertisement
Earlier this week, Alberto Savoia proposed in his Artima blog a manifesto for developer testing "for the rest of us," the Testivus Manifestivus. One of the points in this manifesto stated that:
Testing beats debugging
In explaining this point, Savoia noted that:
If you like debugging, developer testing may not be for you. Most people who start practicing developer testing find that they spend much less time using the debugger. We believe that time invested writing and running tests is better than time spent debugging.
Savoia then pointed to a blog post by Sue Spielman from a few years ago, where Spielman essentially writes a farewell letter to her debugger, Dear John'er...I Mean Debugger:
Test driven development has reduced the need, or requirement, for me to be in the debugger. While maybe not eliminating the need for you altogether, it recently dawned on me that I rarely have to fire up the debugger at all. After all, if you are writing test cases before you are writing code there's nothing to debug. As you start writing the code, the test cases will validate your code for you. By doing frequent, short iterations and running your tests after each and every change, you find many bugs immediately.
... a new vanguard of smug programmers come out of the woodwork with their superior attitude and resurrect the old mantra that only bad developers use debuggers...
Ever since I've become serious at testing, my usage of debuggers has increased, if only because I write more code now than before (I didn't used to write any tests), and this extra code needs to be debugged, like any other. Or do these people assume that just because the code that you write is called "tests", it's suddenly become bug-free?
Beust, however, notes that his use of a debugger changed from hunting down bugs to that of inspecting and verifying code:
But here is my real secret: most of the time, I don't use a debugger to debug. I use it to verify that my code works as I think it does...
That's right: I launch my debugger even before a bug has manifested itself. Even before my code is working at all!
I launch it to inspect all the variables, verify my assumptions, stare at my code for what it really is, not for what my biased view tells me it is...
While test-driven development has become increasingly accepted as a best practice in recent years, debuggers in the major IDEs have also progressed in capability and ease of use.
Do you feel that practicing test-driven development, or just developer-driven testing, reduces your need for a debugger? How often do you reach for your debugger, and for what tasks?
It's hard not to take this into ad hominem territory, but from the posts and blog entries I've read of Cédric's on testing, I'm convinced that he never really learned how to test well nor how to write highly-testable code, which is the main explanation for the existence of his utterly unnecessary TestNG. A few of his JUnit complaints I shared for a a brief time until I got better at testing, so I intimately understand his early reactions and difficulties.
The fact is that unit testing, more than any other other of its blessings, when done with skill, helps skilled programmers write vastly better code. That frequently means different granularity than untestable code tends to exhibit. So these assumptions that Cédric "verifies" in the debugger are the same sorts of things that very good tests over very good code also verify, but unit tests verify these things upon every build. And code so testable is, in practice, more enjoyable to use and easier to adapt to new requirements.
I apologize for this foul tone, but I don't know how to say it more sensitively, and I think that resolution of this issue is deeply important for our field.
That makes it sound like a very fragile and/or difficult process and one with a built-in get-out clause when it doesn't work.(i.e. If it doesn't work it's because you lack programming skills).
In addition, it sounds like "test-first" programming is only of benefit to skilled programmers. Which begs the question(s); What should unskilled programmers do and at what point could they be considered sufficiently skilled to benefit from "test-first" programming?
I totally agree with Cédric. In fact, I'm sure in a near future IDE's wil allow you editing, compiling and debugging (and even testing) at the same time on the fly. Now there is some limiting hotswaping allowed during debugging, but for this convergence to be practical some more advance dynamic techniques for Java are needed, I guess.
Except for when I am stumped by a new (to me) language, I haven't used a debugger for years but it's not because of unti testing. I use logging statements. The reasons are 1. I've had too many experiences where a bug will not occur in the debugger or something will blow up that should not. A huge waste of time. 2. Debuggers don't help you to understand why things are crashing in production but not in the development or QA environment.
But I do agree with his main point about testing. I think there's a false sense of security around unit testing as I've been yammering on about in the other thread.
> I totally agree with Cédric. In fact, I'm sure in a near > future IDE's wil allow you editing, compiling and > debugging (and even testing) at the same time on the fly. > Now there is some limiting hotswaping allowed during > debugging, but for this convergence to be practical some > more advance dynamic techniques for Java are needed, I > guess.
Tests aren't (necessarily) bug free. Cedric: "just because the code that you write is called 'tests', it's suddenly become bug-free?" Though I use logging (and temporary System.out's) more than debugging, this is a good argument for using a debugger (especially since I don't use logging in tests themselves).
The more interesting one, to me, is using the debugger as a sort of "live test". Cedric: "I also use the debugger to modify variables and try to trip my code ..." When I first read Tim Vernum's characterization of a debugger as a "runtime analyser" (in the Testivus thread), I wasn't sure how to work that into practice, but I think Cedric nails it on the head here.
I agree with Cedric. Tool-worship, no matter what it is, just leads to blindness. If your tool of choice is a unit test, that's fine, but when you turn unit tests into a religion and go off on a holy crusade, then it's not fine at all.
Imagine this... You can do anything that a saw does with a chisel. Should you stop using saws? Is a chisel better than a saw? It's all nonsense.
Debugger is a great tool. There are many ways to use it. Debugger is not a single purpose tool. Debugger is a multi-purpose tool.
Let's say a bug has occurred and your unit tests and your logging didn't catch it. What's the best tool to find the bug quickly? Add more logging? Where? Add more tests? Where? What kind? A debugger may just be the fastest way to find that bug. Once the bug has been found, you can write your unit test.
> Let's say a bug has occurred and your unit tests and your > logging didn't catch it.
Just to be clear, logging doesn't 'catch bugs'. It reports what the code is doing as it runs.
> What's the best tool to find the > bug quickly? Add more logging? Where?
You'll have log statements before things go wrong and log statements after things go wrong. If you don't see where it went wrong, you add statements in the middle. It's no different than determining where to place breakpoints. In my experience it's usually a lot faster than the equivalent analysis in a debugger. The real advantage is that logging is permanent.
I find debugging useful when I am new to a tool but after I become familiar with the tool I find it usually takes more time. There are clearly some advantages but I rarely need them.
The better tests I have, the less time I spend debugging. It doesn't matter how I'm debugging: breakpoints and stepping through the debugger, adding and studying log output, or just looking at the source code trying to find the bug. The important thing is that the tests let me find the bugs easier and faster. The bugs become shallow, more like compilation errors.
I don't understand Cedric's point. Of course the tests are also code that contains bugs. But how does he write more code now than he did before he used tests? If his rate of coding has increased, I would assume it's because he spends less time debugging.
The point I wanted to make is that Cedric makes a straw man argument. The manifesto said that one benefit of testing is less time spent debugging. Cedric argues that even with testing the debugger is still useful and he uses it even more. But he says that most of the time he doesn't use the debugger for debugging, just for looking at running code (not looking for a bug). So his post doesn't support its rather sensational title, "More testing doesn't mean less debugging," unless you take "debugging" to mean "anything you use the debugger for."
Previous Topic
