This post originated from an RSS feed registered with Agile Buzz
by Steven E. Newton.
Original Post: Cem Kaner at PNSQC 2003
Feed Title: Crater Moon Buzz
Feed URL: http://www.cmdev.com/buzz/blosxom.cgi?flav=rss
Feed Description: Views and experiences from the software world.
On October 14 I had the opportunity to hear Cem Kaner talk at the Pacifc Northwest Software Quality
Conference in Portland, Oregon. The title of his talk was How
Many Lightbulbs Does It Take To Change A Tester? These are my
notes and impressions from that talk.
Kaner's opening remarks focused on his views of where software testing
has been and is going. After 20 years of practices, he said, little
has changed, and the common practices have their problems. Compared to
the programming community, testers have changed very little in their
self-image.
Old attitude: Defensive, fearful, hierarchical, grasping for control, procedural. The goal was to "reduce testing to a set of steps even dummies can follow".
New attitude: The tester as technical investigator: Find stuff that other people don't know how to find. Often these are bugs, but not always. Generally the work is focused around the quality of the product.
One of the recommended practices (from Kaner's own work) is to
only do black-box testing. Kaner question that and asked why it was
considered good. What value is there in working without knowledge of
the underlying code? The answer is that it allows the test, instead of
learning about the product from the code, to learn about the product from
the stakeholders and see how well the code does that. Tester gathers
information the programmer didn't or couldn't consider. The more
recent discovery is that sometimes the best source of that information
is the code.
He then discussed the wisdom that black-box testing and external
testing is necessary because "programmers can't catch their own bugs".
Kaner drew a distinction between a private bug, one the programmer
makes, sees, and fixes before anyone else sees it. Rate: 15-150 per 100
statements, and public bugs: all bugs left in code when programmer passes
it to another person. Rate: 1-3 per 100 statements. In other words: 90% of
the bugs are private, so programmers are actually pretty good at finding
bugs. The insight here is that programmers find one kind of bugs, but
it needs someone else to test other things to find the remaining bugs.
In the next part of his talk, Kaner discussed the view that testers
and programmers should be kept independent. He brought up several areas
where close collaboration actually is more beneficial than separation.
Para-functional Testing What used to be called
Non-Functional testing, but Kaner discovered that too often
mean "disfunctional testing". For good testing of things like
security, usability, accessibility, supportability, localizability,
interoperability, installability, performance, scalability, and the
various other "ilities", testers must work with programmers because the
stakeholders don't have rules.
Preventative testing. The best kind of preventative
testing -- testing that prevents bugs before they happen, is Test-driven
Development, said Kaner. It implies tight collaboration between
the testing role and
1000
the programming role (sometimes so tight, they are
the same person) Fast, efficient feedback is possible, and fast is good
for learning. Communications cost is low.
Kaner's talk then covered the value of fast and efficient feedback
in software development. Traditional external testing is say, 10
calendar days or more, plus all the processing, cycle time, said Kaner.
In hours, cycle time for efficient companies, it's 4 hours for a simple
defect. Up to 20 hours for a less-efficient companies. Filing lots
of bugs is a form of denial of service attack to a programming team,
because the processing time of dealing with the bug reports leaves no
time to actually fix bugs.
The purpose of testing is to find bugs, right? Well, said Kaner,
it depends. In discussing some of the points raised by the paper, A
Manager's Guide to Evaluating Test Suites, Kaner examined how test
results can guide a project manager in determining the status of the
system. Do you want to find and eliminate the most bugs before a fixed
date, Kaner asked, or do you want to develop an overview of the state of
the system to allow the project manager to decide if it's ready to ship?
One area where the testing is not done to find bugs, but to protect the
company from risk of lawsuits -- do we test enough and well enough to
accept the risk that our system may fail (and kill someone) but know we
did the best we could?
Testers in Agile Teams
One of the facts of software development, said Kaner, is that pulling
features out late will not save money. In the tradeoff between features,
reliability, cost and time in a standard waterfall process, features
and cost are used up by the end, so the only thing left is reliability
vs. time. Thus testing is often compressed or cut. In evolutionary
development where we have a limited set of features that work, then
towards the end, tradeoff is do we have enough features to sell it or
do we need to spend more time adding features?
Test then code
Test-first development
Big Tests Up Front
One test, one feature, iterate
The lesson for testing is that test code and testers must be able to cope
with simple changes. This is where IEEE Std 829 Testing Practices is,
according to Kaner, an industry worst practice.
A Final Note
Outsourcing, said Kaner is a by-product of making the process rigid
and "fully documented". It makes software into a manufacturing process,
something that is easily outsourced. This is a fatally flawed approach
for most commercial software.
Previous Topic
