Summary
Testing is tough for most programmers, but without some amount of developer testing delivering quality software is even tougher. We propose a new software specialization and position – Developer Testing Master – to help software organizations fill the horrifically expensive testing gap between development and QA.
Advertisement
Today, we take having a software quality assurance department armed with dedicated QA engineers and tools for granted. Even most software start-ups, when they get to half-a-dozen or so engineers, start thinking about their first dedicated QA hire. This wasn’t always so. I remember that when I joined Sun Microsystems in the early 80s, the company headcount was about three hundred people and there was no formal software QA department, or even a dedicated software QA person. I was told “If you can run ‘vi’ after building the system, then everything is probably working OK.” By the time I left Sun in 1998, they had a VP of Quality and hundreds of software QA engineers.
A cynic would probably explain the evolution and growth of software QA as a ploy for developers to pawn off testing responsibilities. But the reality, and my experience, is that having an organization and group of people whose main job and focus is software testing makes a lot of sense. In most fields, when the scope of work and body of knowledge grows past a certain point, specialists eventually emerge. In the beginning, almost all doctors were general practitioners; today most of them are specialists. In the Old West your dentist and barber might have been the same person – I guess the kind of chairs they used were close enough. It’s more likely, however, that in those Old West towns there weren’t enough people to keep both a barber and a dentist fully employed, nor many hairstyles to master, or cures to know about. But if there is enough work to keep busy, and enough domain knowledge to leverage, specialization makes a lot of sense. I’d rather have a surgeon operate on me than a general practitioner. And if my gray matter needed tending, I would not settle for any surgeon, I’d want someone even more specialized – a brain surgeon.
A QA organization is a must. The kind of black-box, functional, integration, testing that most QA organizations perform is necessary, but it is not sufficient. Waiting for QA to find all bugs is very inefficient – especially when it comes to unit level defects that could have been caught and fixed during development at a fraction of the cost.
I am a big believer in developer testing. The idea that developers should take responsibility for unit testing their own code makes a lot of sense. Thanks to eXtreme Programming and other Agile methodologies, and testing frameworks like JUnit, developer testing is finally catching on – but slowly, and it is also encountering a number adoption problems and technical difficulties that could keep it from becoming a mainstream practice.
In many cases, developers lack the drive or skills to adequately unit test their own code. Testing can be as technically challenging as development – and often more challenging. On top of that, unit testing takes time. It takes 3-4 lines of JUnit for every line of Java to get good code coverage, for example. And when most developers, or their managers, are pressed for time, features and schedules usually trump testing.
I believe that software development has gotten to the point where we need more specialization in testing. We need a new type of tester. One that will bridge the gap between development and QA. A champion that will leverage and maximize the developer testing effort. Let’s tentatively call this position Developer Testing Master (as in Build Master and Web Master) – feel free to suggest better names.
What would a Developer Testing Master (DTM) do? Here’s the draft for a hypothetical job spec:
This is a high-visibility, high-impact, senior software development position, reporting to the VP of Software Engineering at Company ABC. The successful candidate will help our software engineering organization bridge the gap between development and QA by providing developers with the infrastructure, tools, training, and support necessary to be successful at developer/unit testing.
Job Requirements:
[Should start with the organization’s standard requirements for non-junior software developers. Remember, this should not be an entry-level or junior position. This person has to be at least as good as other developers – ideally better than most of them since the DTM will often have to understand, clean-up, and make testable some of the un-testable code the other developers have created.]
Must have demonstrated passion for, and experience in, developer testing. Should be extremely familiar with continuous integration and testing tools and frameworks (CruiseControl, xUnit, code-coverage tools, etc.)
Must have leadership ability to evangelize, motivate, and train developers in the art and science of unit testing.
Job Responsibilities
Help to set-up a software development environment that enables continuous integration and testing. Environment must support automated hourly and nightly builds. The former must run smoke tests and the latter the full battery of developer tests. Might have to implement distributed testing framework to allow parallel test execution.
Analyze the existing code base and recommend and/or implement re-designs and refactorings to make the code base testable.
Extend and customize xUnit framework to standardize and simplify unit test writing for the other developers. Extend the “TestCase” class by creating “CompanyABCTestCase” and maintain the latter. Implement initial set of mocks, test fixtures, and other set-up methods to enable other developers to write true unit tests that don’t depend on external system resources.
Create and deliver basic unit testing training material to educate all developers in the art and science of unit testing.
Work with the team to decide on developer testing metrics and objectives. Create, maintain, and publish a developer-testing dashboard to track those objectives.
There it is. Any takers? Needless to say, the position should pay at least as much as a similar pure development position, ideally more (specialists usually charge more).
Note: To give credit where credit is due, this post (including the brain surgeon analogy) was inspired by the following David Rubinstein’s article in SD Times:
David concludes the article as follows: “Bring in the specialist. Just as you wouldn’t have your family doctor perform brain surgery, you should leave the test scripting and maintenance to a tester, freeing up the developers to do what they do best—innovate.”
While I like the idea of bringing in a specialist, I think that completely exonerating developers from testing, as David’s last sentence suggest, might be going to far. The job of the Developer Testing Master is to help get things started and keep them going, overcome the initial hurdles, make it easier for the developers to test their own code by providing the basic framework, and then hang around to provide additional help as needed. In other words: teach a man to fish, but stick around to help them clean the fish, cook it, and eat it.
I've done something like that for a while and besides qualities described above, I'd say one must have a passion not only for quality of code but also for specific product being developed. To work together with team, not ride around as firefighter.
> I’d rather have a surgeon operate on me than a > general practitioner. And if my gray matter needed > tending, I would not settle for any surgeon, I’d want > someone even more specialized – a brain surgeon.</p> > <p>A QA organization is a must. So you give your brain to some very innovative brain surgeon. Would you want to have a Brain Test Master to exercise your brain?
> <p>In many cases, developers lack the drive or skills to > adequately unit test their own code. I think this needs to be fixed at the college level, it should be a mandatory part of the curriculum.
> <p>I believe that software development has gotten to the > point where we need more specialization in testing. We > need a new type of tester. One that will bridge the gap > between development and QA. A champion that will leverage > and maximize the developer testing effort. While other industries have QA, I don't know if they have a separate people for testing. Are there are dentist or barber,or surgeon, or plumber testers? No. I think if you a specialist you must know how to test your product, and you must do it.
> "In many cases, developers lack the drive or skills to adequately unit test their own code."
Help them fix their inadequacy, or jettison them if they cannot or will not. Or don't hire them to begin with. Unit testing and writing unit-testable code are required skills for programmers, and anybody who cannot or will not acquire that skill is not worth having.
> While other industries have QA, I don't know if they have > a separate people for testing. Are there are dentist or > barber,or surgeon, or plumber testers? No.
The trouple with analoges is that they usually don't stand up to close scrutiny (much like oil paintings in a way).
Three of the professions above do indeed have testers and the practitioners are all tested first, before being released into the wild. Many barbers serve an informal 'apprentice' scheme (aka cheap labour) honing their skills under supervision before working independently.
I think I'd love to take on a job like that. I was just describing that kind of thing to a friend today, and he then noticed this forum post. My background is on the developer side (rather than test/qa) but I'm pretty sick and tired of working in groups that don't grok the ideas of self improvement, of which this kind of testing is part.
Of course, a lot of it would be working the cultural, social, economic and political angles, rather than just the technical. "We've managed to ship revisions of this product 4 times already, why do we need to slow ourselves down with all this overhead you're talking about?" Etc. I don't want to be pessimistic about the chances of adoption, but I suspect that more often than not, the institution will mostly be against really buying into such things.
Well, anybody hiring for this kind of thing - at a place that has good financial backing? :-)
Perhaps when people start choosing software and tools based on things like features and quality metrics things will change. I was looking around at JMS implementations and what I wanted was to see (in addition to the usual things people want to see) were testing metrics.
Which implementation has good code coverage? Unit tests? Regression tests? How often are they run? What is the break-down of remaining documented bugs (priority, severity) in each release? What code changed most often (e.g. the big method that does everything and so is breaking all the time)? When changes were made, how many things broke on average? If a bug was filed and fixed and resolved, was a test added to make sure that would be caught if it ever happened again? Are things checked in with a decent revision control system (i.e. one that does atomic changelist commits)? If it is open source, can I browse the history in a detailed fashion (e.g. the "blame" feature) and rapidly (the web server and script processing doesn't take bloody for ever to let me browse)? Are the specifications written down and automatically checked against the code? What static checking tools are used to Find Bugs?
So, to repeat: I want metrics! Were there any to find? Mostly, no, not really - certainly not to the degree I'd like. Actually it turns out that there are a few test suites for JMS (e.g. http://jmscts.sourceforge.net/) but it doesn't seem to be that they are adopted that well, and they are addressing overlapping but different concerns I think.
There's probably a chicken-and-egg problem with all this: nobody is going to invest time in it if there's no apparent pay-off. If the "competition" doesn't advertise or reveal their testing world-view, then why would you? It takes a lot of work to do all of the things I mentioned.
Does the presence or absence of a Developer Testing Master (and the developers he's trained) in an organisation have any effect on the desirability (or not) of test-first or test-driven design (TFD or TDD)?
"A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects."
Notice that he said "should". He also mentions "program a computer" and not "test a computer program" :-).
> Help them fix their inadequacy, or jettison them if they > cannot or will not. Or don't hire them to begin with. > Unit testing and writing unit-testable code are required > d skills for programmers, and anybody who cannot or will > not acquire that skill is not worth having.
Helping developers learn how to, practice, and appreciate the benefits of, unit testing has been my mission for the past several years. But it has been a tough fight (see my other posts on the topic.) The majority of developers are not easily test-infected.
Perhaps I was not clear enough in the original post: The job of a Developer Testing Master is not to relieve developers of their testing responsibility, quite the contrary. The DTM's main responsibility is to help developers get over the initial adoption hump and (re)training necessary to make them self-sufficient.
After the initial training and set-up, the DTM's job is to make sure that developers don't slip into their old "bad-habits" and provide help and advice on design, testing, and testability issues.
While I like Heinlein and blurted out the "specialization is for insects" quote several times myself, I believe that an organization with, say, 15 developers, 1 DTM, and 4 QA people, would be more effective at releasing well-tested code than one with, say, 15 developers and 5 QA people.
> Does the presence or absence of a Developer Testing Master > (and the developers he's trained) in an organisation have > any effect on the desirability (or not) of test-first or > test-driven design (TFD or TDD)?
A DTM should educate and show developers how to practice TDD but should not be too dogmatic or expect too much to soon. TDD is tough pill to swallow for most developers.
I'll take a page from my Testivus post and recommend that a DTM should focus on "Less testing dogma and more testing karma".
> >Well, anybody hiring for this kind of thing - at a > >place that has good financial backing? :-) > Hint: > Search and you will find.
If only I hadn't just started a new job after having just started a new job. Or, wait, maybe the bubble is back and it is actually just all about being upwardly mobile!? Never hurts to interview, I always say.
I started my career as a Developer, did that for a year and a half, and then got "sucked" into the Test / QA world, which I experienced for 2 years doing a range of things from White box / Unit testing, Black box Functional testing, Large scale daily productwide regression testing, Test Automation, Stress Testing, Test Driven Development, Agile Development, etc. Since Nov. 2006 I have been back into C++ / Systems type Development, and the time I spent in Test / QA has been extremely valuable and allows me to debug and diagnose problems much more quickly, get to the root cause of problems efficiently, write better code the first time around, etc.
I think as software systems grow in scale and complexity, and there is more interoperation between various versions of various disparate codebases running on systems rife with heterogenity in any real-world system, it will become more and more important for a seasoned developer to have acquired some experience in Test.
Therefore, I believe that all developers on a project should eventually become highly proficient at writing testcases, then automating unit tests, for any code they contribute to the codebase. Developers can no longer let their ego get in the way of fooling them into thinking their code is always perfect.
It may help to hire a seasoned QA / Test veteran to provide consultation regarding processes, best practices, conduct testcase reviews, suggest testcases after performing white-box analysis on the codebase, etc., so I could see the idea of a Developer Testing Master making sense in some contexts.
Anit, that sounds like very good experience. I guess maybe it would be good if Dev teams could be staffed from QA rather than there being a divide between them - I remember hearing Devs say words that showed how they thought doing anything other than writing features was, basically, beneath them. :-(
Flat View: This topic has 23 replies
on 2 pages
[
12
|
»
]
Previous Topic
