Summary
Developers need to take more responsibility for testing their code. But the majority of developers are not willing, nor ready, nor able to jump on the bandwagon of the more extreme and demanding developer testing movements such as Test Driven Development. Testivus is a proposed developer testing movement "for the rest of us".
Advertisement
In a recent blog, I shared my belief that when it comes to testing their own code developers fall into three basic categories based on how susceptible they are to becoming test-infected. T1 developers take to testing like ducks to water. T2 developers need some time and encouragement but eventually get it, and testing becomes a normal part of their development work-flow. T3 developers, on the other hand, seem to have an immunity to test infection. I also shared a personal observation that T3 developers appear to outnumber T1 and T2 developers combined.
The post triggered a bunch of interesting comments and follow-on discussions. One sentence in one of the comments, however, was particularly striking to me. Mike Kaufman wrote:
“Maybe we need a ‘testing for the rest of us’ movement’”
His comment got my attention for two reasons.
First, as a fan of the Seinfeld TV show, the sentence “testing for the rest of us”, reminded me of one hilarious episode where one of the characters, fed up with the dogma, commercialism, pressures, etc., associate with the holiday season, creates an alternative holiday: Festivus, the festival for the rest of us. The idea of Festivus must have struck a chord with people because, since the show aired, it acquired a life of its own. A growing number of people now celebrate Festivus (at my company, Agitar, our holiday picnic is called Festivus and is modeled after it). There are books and websites dedicated to the subject and some entrepreneurs sell Festivus items – including the famous Festivus aluminum pole (the alternative to the Christmas tree).
Second, and more importantly, I have seen enough evidence that the motivation, practices, tools, and expectations that work well for T1 developers and, with some effort and persistence, can be embraced by T2s, are simply too much to swallow for T3. Most developers admit that they should play a more active role in testing, but they are not ready to take the big steps, or commit to the degree of change and amount of time that the more publicized developer testing practices, such as TDD, require. They need a simpler, more gradual, less extreme and less dogmatic approach. They need Testivus – the testing movement for the rest of us.
I told Mike Kaufman that we should take his idea of a testing movement for the rest of us to the next level but, unfortunately, he currently does not have the cycles to dedicate to it. However, after expressing some reservations about the term festivus (“Not sure the term ‘testivus’ sounds too smart” – and he’s probably right), he said he’d be happy if I carried the testing-for-the-rest-of-us torch, and would be impressed if we got something rolling.
It is in that spirit that I present to you the first draft of The Testivus Manifestivus, the testing manifesto for the rest of us.
The following Manifestivus is obviously rough and incomplete (feel free to propose changes and addition). Consider it a starting point for discussion.
The Testivus Manifestivus
(First Draft)
Less testing dogma, more testing karma
Any tests are better than no tests
Testing beats debugging
Test first, during, or after – whatever works best for you
If a method, technique, or tool, gives you more or better tests use it
Below are quick explanations of the current Manifestivus points:
Less testing dogma, more testing karma
Dogma can be informally described as authoritative principles, beliefs, often considered to be absolutely true. Testivus tries to keep testing dogma to a minimum. What works for some people sometimes, may not work for some other people, or even the same people at some other time.
Karma, on the other hand, can be informally explained as: “Do good things, and good things will happen to you.” We believe that writing and running tests is a good thing so if you write and run tests good things will happen to you … well perhaps just to your code.
We’d like to say that this is the central tenet of Testivus, but calling something a tenet would be too dogmatic.
Any tests are better than no tests
Self-explanatory and inspired by Martin Fowler, who once wrote “Imperfect tests, run frequently, are much better than perfect tests that are never written at all”.
Testing beats debugging
If you like debugging, developer testing may not be for you. Most people who start practicing developer testing find that they spend much less time using the debugger. We believe that time invested writing and running tests is better than time spent debugging.
Sue Spielman wrote an amusing and clever “Dear John” letter to her debugger after she started practicing developer testing:
Test first, during, or after – whatever works best for you
Test Driven Development is a very clever idea with a lot of merit and strong supporters, but we realize that it’s not everybody’s cup of tea. The earlier you test, the better it is, but even tests written after you think the code is done are better than no tests at all. Do what works for you.
If a technique, metric, or tool, gives you more or better tests use it
Even among T1s there is disagreement on the benefits of certain testing techniques (e.g. using mock objects and to what extents), metrics (e.g. using code coverage), and tools (e.g. automated test generation). Don’t get caught up in those arguments. If something works for you, and makes your testing easier, more efficient, or more fun, just use it.
There you go. What do you think? Does the idea of Testivus movement make sense to you? If so, what should our Manifestivus say, and what should we do to promote it?
There was a weblogs article here a long time ago, which I considered rather enlightening because it noted that what unit tests often encapsulate is knowledge of specific bugs, and how to make sure they don't make it into your software.
Of course restivus people are likely to point out that if those bugs were never in the software in the first place, there may be little benefit in writing a test case for it. And writing a test case can sometimes be hard work, especially if it requires complex initializations and or mock objects.
Yet, they still have to agree that when it comes to quality control, any tests are better than no tests.
Tests may have one possible negative impact on refactoring. The more tests you have, the more likely you have to refactor tests as well. Or does that just mean your testing is at the wrong level of granularity?
I like the point of "Less testing dogma, more testing karma". The point of unit testing is to have a good set of test cases (high-coverage, up-to-date) which make you always confident in your being able to make change to the code without afraid of unexpectedly breaking other areas. And thus, you can do whatever you like to reach that goal (e.g. not necessarily test-first, as some people claim to be the only way to implement TDD).
> I like the point of "Less testing dogma, more testing > karma". The point of unit testing is to have a good set > of test cases (high-coverage, up-to-date) which make you > always confident in your being able to make change to the > code without afraid of unexpectedly breaking other areas. > And thus, you can do whatever you like to reach that goal > l (e.g. not necessarily test-first, as some people claim > to be the only way to implement TDD).
I agree and I think one of the more effective ways to do this is to create regression test suites for projects as they go live. These generally will come from your testing during development. I think there is far too much time and effort spent testing 'units' in isolation. It works for certain types of code but it's impossible at a high-level of complexity. It's impossible to get 100% coverage for many applications and modules.
I'd also like to say that testing helps catch errors, it doesn't negate the need for well written code. Maybe this is obvious but they way people talk about testing, you'd think it was more important than the code being written.
Can we add to this list that a test for code that is never written or used is useless? I've seen a project fail partly because the developers spent a lot more time writing tests upfront and didn't leave enough time to write the code (and they were given an more than an extra year to finish on top of the 1 year scheduled.) Because they waited so long to start proving the design, it was too late when it became clear that it would never work.
I'm not saying that writing tests early isn't good, but I've seen a projects where designs are changed. If you write all the tests first, you might find that a lot of them are never used. Maybe people consider this a cost of doing business. It just seems inefficient to me.
> I'm not saying that writing tests early isn't good, but > I've seen a projects where designs are changed. If you > write all the tests first, you might find that a lot of > them are never used. Maybe people consider this a cost of > doing business. It just seems inefficient to me.
Hi James,
Thank you for your comments. Just one clarification, you don't write all the tests first.
Most developers who practice TDD/Test-First do it in very quick test-code-test-code cycles:
1) write a few lines of JUnit for code that's not yet implemented
2) run the tests - they fail
3) add/fix the code to make the new tests pass
4) run the tests again - if they pass, move on to write a test for the next bit of implementation
It's not atypical for a TDD developer to go through one of these cycles in a minute or less.
The bottom line is that with proper TDD you don't accumulate a bunch of failing tests. The body of tests grows along with the code.
It's not inefficient, just different - and some people find it works great for them.
>And writing a test case can sometimes be hard work, >especially if it requires complex initializations and or >mock objects.
No argument there. Writing tests can be a challenge. One of the people who commented on my previous post says he spends considerably more time writing tests than writing code. This is not uncommon, nor surprising. Testing is a combinatorial problem. I found that to get 90%+ code coverage you typically need 3-4 lines of JUnit for every line of Java.
Even then, I would argue that time spent testing is an investment and will end-up saving you time long term.
> Tests may have one possible negative impact on > refactoring. The more tests you have, the more likely you > have to refactor tests as well. Or does that just mean > your testing is at the wrong level of granularity?
A couple of comments on this point:
1) If you refactor taking advantage of IDE refactoring commands in many cases the tests will be automatically refactored along with the code.
2) More importantly, refactoring (or any code changes for that matter) without any tests is like working without a net. I would argue that having a trusty set of tests has a positive impact on refactoring and other code changes. Not having tests is what would keep me from touching the code for fear of breaking some major functionality and not know it until it's too late.
...
It's a fact that writing tests can be very time consuming and, for many developers, not fun. It's also a fact that, once written, maintaining test is also time consuming and even less fun. But you still need tests.
Testivus is not about avoiding tests, or minimizing the number of tests you have because they are a hassle to maintain. Quite the contrary.
Testivus says that developer testing is the right thing to do, but it also faces and accepts the reality that most developers (the T3s) simply don't like to write tests and/or don't know how to test properly and/or are not given enough time to include testing.
Some proponents of some developers testing movements (e.g. TDD) take the position that developers think that they don't like testing, think that they don't know how to test, and think that they don't have time to test. That once you show them the proper way to tackle testing, they'll see the light and change both their thinking and their action. And you know what? They've done an amazing job. XP and TDD have turned more developers to testing than I would have ever thought possible. I want to be very clear that Testivus is not about undermining or replacing movements like TDD. On the contrary, we want to continue and extend their work.
My point in the previous post is that the message of testing movements like TDD works great on T1 developers, can be made to work for T2 developers - with some time and effort, but it's looking pretty hopeless for test-infecting T3 developers (who happen to be the majority of developers).
Testivus is all about getting the T3 developers to start testing - at the time, in the way, and with the tools that work best for them.
> Most developers who practice TDD/Test-First do it in very > quick test-code-test-code cycles: > > 1) write a few lines of JUnit for code that's not yet > implemented > > 2) run the tests - they fail > > 3) add/fix the code to make the new tests pass > > 4) run the tests again - if they pass, move on to write a > test for the next bit of implementation
OK I can see how that would work but I just really have doubts that it tests anything truly meaningful. You can only write trivial tests in a matter of minutes.
To give you an example of what I mean, I once had to write a good bit of code that would take a set of groups and based on some arcane rules recursively combine the groups until certain conditions were met. There was a lot of testing required for this. There was no way to toss off a couple of tests in JUnit to verify this. I couldn't look at a complex set of groups and tell you what the result was. I had to walk through the steps to determine the correct result. I could create a bunch of tests to verify each step or a small set of steps but this is not the same as testing the whole process. This is what bothers me about TDD. A lot of proponents of it seem to be under the mistaken impression that perfect parts make perfect wholes.
Good ideas all around, but can we please kill the "more testing means less debugging" myth once and for all?
This is very damaging to the software industry.
Ever since I've become serious at testing, my usage of debuggers has increased, if only because I write more code now than before (I didn't used to write any tests), and this extra code needs to be debugged, like any other.
But here is my real secret: most of the time, I don't use a debugger to debug. I use it to verify that my code works as I think it does.
That's right: I launch my debugger even before a bug has manifested itself. Even before my code is working at all!
I launch it to inspect all the variables, verify my assumptions, stare at my code for what it really is, not for what my biased view tells me it is. I also use the debugger to modify variables and try to trip my code, cause errors that could or shouldn't happen and make sure it reacts accordingly. Of course, eventually, I capture all of this in tests, but these approaches are complementary.
Don't throw away your debugger, or the quality of your code will suffer.
And by the way, Alberto, you tend to equate "testing" to "JUnit". There are alternatives, as you know :-)
> That's right: I launch my debugger even before a bug has > manifested itself. Even before my code is working at > all! > > I launch it to inspect all the variables, verify my > assumptions, stare at my code for what it really is, not > for what my biased view tells me it is. I also use the > debugger to modify variables and try to trip my code, > cause errors that could or shouldn't happen and make sure > it reacts accordingly. Of course, eventually, I capture > all of this in tests, but these approaches are > complementary.
I don't use debuggers but I sometimes use trace and debug level logging in much the same way. This especially for things that don't necessarily manifest themselves as obvious problems like releasing resources or clearing states. I have some code that ignores certain things. I wanted to make sure it was 1. actually receiving these things and 2. that it was ignoring them for the right reason. I could see from the code behavior that these items were not processed but that didn't guarantee the code was working. So I turned on the trace level debugging and could see that the item was parsed correctly and that my code was indeed explicitly ignoring them. This is one of the things that I find unit testing to be really bad at: verifying negative behavior.
Here is another area where extreme debugging has helped me: identifying two bugs canceling each other.
In such cases, your test will come out green but a careful inspection of the internal state of your method will typically reveal the misbehavior right away.
While we are applying Buddhist ideas to software development, how about Mindfulness. Once you complete a piece of code, ask yourself: "Why did I do it that way?". This is testing the software that runs in your head.
Cedric, I used to step through every LOC back in the win32 days, when the debugger was all you had. It's certainly a good form of code review. One trick there was to breakpoint every possible branch in your app, so that the debugger would kick in whenever a new branch got hit, and you could see which bits hadnt been explored.
but we've moved on now. test frameworks. coverage tools, a stack of vmware images each providing a different virtual system to test on. continuous integration servers to see that things still work. And, I no longer have to look at x86 assembly code and know that xor eax,eax is just a trick to set the eax register to zero.
move on. try a test framework. you know, like testng :)
I certainly didn't say I was single stepping through every single line of code, but there are often a few critical parts of what I submit that I want to double check.
This might also be due to the fact that for the past three years, every single line of code that I check in gets reviewed by one of my peers, a constraint that probably very few developers face in real life. This kind of scrutiny really makes me want to make sure that my code looks *and* works well before I submit, regardless of the tests I submit.
As for this TestNG thing, yes, I will definitely check it out, I heard a lot of good things about it.
> Good ideas all around, but can we please kill the "more > testing means less debugging" myth once and for all? >...
That's an interesting perspective and I can definitely see your point and the advantages of using a debugger that way. But since I haven't had anyone come to the defense of debugging - until now - I hadn't given much thought to it. I'd love to hear if other people feel the same way or use a debugger the in the same way and to the same extent that Cedric does. I would not have a problem removing (or modifying) the "less time debugging" idea from the Testivus manifest. Especially if, as you say, the time spent in the debugger eventually leads to reusable test.
The debugger usage mode I am not too fond of is to hack the code together without any tests and ship it to QA. When QA reports a bug (that would probably have been found right away with some unit testing) spend a few hours in the compiler tracking it down. Fixing the bug - perhaps. And sending the code back to QA. I don't like this approach because time invested in testing results in reusable assets, while time spent on the debugger in this way is typically spent chasing a particular problem and leaves no reusable artifacts.
> And by the way, Alberto, you tend to equate "testing" to > "JUnit". There are alternatives, as you know :-)
I know, but I don't want to use this forum to pitch Agitator :-) ... You are right, I am lazily using JUnit as convenient proxy for all the developer testing frameworks (including TestNG) and Java as a proxy for all languages. I'll try to be more explicit about that in future postings.
Good post Alberto ! Completly agree with you. However i would like to add my own angle to this
(I dont mean to imply that what I am saying is releveant to all developers- but unfortunately I have seen this to be true for many developers.)
1) Often developers carry an air of superirity over QA personel. They somehow feel testing is too much of a humbling task for them to do- They have greater tasks to finish.
2) Devlopers are often more concerned with completing the product rather than enusing that it works correctly. It seems to be someone else's task.
So my point in nutshell is that that the reason why different developers have different gene is partly because of their attitude
Flat View: This topic has 51 replies
on 4 pages
[
1234
|
»
]
Previous Topic
