Weblogs Forum - Is Complete Test Coverage Desirable

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Weblogs Forum
Is Complete Test Coverage Desirable - or Even Attainable?

21 replies on 2 pages. Most recent reply: Feb 20, 2005 8:55 PM by Curt Sampson

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 21 replies on 2 pages [ « | 1 2 ]

Keith Ray

Posts: 658
Nickname: keithray
Registered: May, 2003

Re: Is Complete Test Coverage Desirable - or Even Attainable?

Posted: Feb 17, 2005 6:49 AM

At the BayXP meeting last night, two people told of how much statement coverage they had gotten using TDD and XP practices: 70% for one project, 97% for another project. Neither project was measuring test coverage during development.

If you're doing Java development, it seems that the Agitator product from http://www.agitar.com/ will be able to mostly-automatically create and run as many as 30,000 tests with little effort on the user's part.

Malte Finsterwalder

Posts: 20
Nickname: jufoda
Registered: Aug, 2003

Re: Is Complete Test Coverage Desirable - or Even Attainable?

Posted: Feb 17, 2005 8:03 AM

It is indeed a good idea to think about what you require from your environment. And I think it is a good idea to write some tests for that. But I'm not thinking about unit tests here, since they are useless in the final environment.
I'm thinking about a checkup-feature. The application could run a checkup of it's environment at startup or regularly or by request of the user, or... But I have seldom seen such a feature. I have seen a system that monitored the network and blocked the application when the network became unreachable.

I can think of a lot of things that would be sensible to routinely check at least at startup:
- Is the database reachable?
- Does it have a schema I can work with?
- Do I reach all the other systems I need?
- Are all configuration parameters set?
- Do they have meaningfull values?

A lot of this stuff could be embedded in frameworks.
e.g. An OR-Mapping-Tool like Hibernate for example has a definition of what is required from the DB schema. It should be possible to connect to the database and check, whether the schema fullfills all those requirements.

I'm also missing a framework to handle configurations. Everybody writes their own config file read/write code. Does anyone know of a simple framework for handling software configurations?

Greetings,
Malte

Merriodoc Brandybuck

Posts: 225
Nickname: brandybuck
Registered: Mar, 2003

Re: Is Complete Test Coverage Desirable - or Even Attainable?

Posted: Feb 17, 2005 8:33 AM

I think testing as much as you can is always desireable. Where I have an issue with unit tests generally is that the hard core XP/Unit test advocates always seem to say "If you had good unit tests..." every time somebody even mentions having any kind of bug, as if Unit Testing is some silver bullet that can find all bugs, fix all problems and likely help me lose 30 pounds.

"I got N% coverage with my unit tests." Great. You did all your development on Red Hat 9. What if I try running that on a Debian based distribution? You said that you support Linux, why won't it run? Or substitute your favorite versions of Windows or Office or whatever. They are not a panacea and I hate that they get portrayed as such by some people.

What Frank's article touches on is that there are system wide issues that need to be accounted for and things that you just can't unit test. If I'm writing an application for any mainstream operating system these days, except maybe OS X because it hasn't been through too many versions yet, I cannot possibly test every permutation of every possible environment including security updates, kernel updates, browser updates, etc. Some things you have to take on faith. That being said, I don't think you need to question whether complete test coverage is desirable. I think we all know it is not attainable, but that should not stop you from beating on your application as much as possible during the testing phase. All other things being equal (that pesky ROI acronym came up... :-), the more you know the better off you will be.

As far as making applications configurable goes, this helps in some ways and makes things harder in others. Generally there are less areas in which a configurable application can go wrong because the goal is to reduce the amount of code and logic in the code. Less code equals less chance for bugs. However, if you do not select good defaults and the majority of users have to tweak things, you can bet people will be putting in, accidentally of course, all kinds of wacky data which you will have to defensively code against. Granted that code isn't hard to write, but it is boring, which means that it doesn't get done a lot of times. That is its own sort of problems, although those issues are usually a lot less problematic than horrible nasty logic errors that cause your data base to go bye-bye or your machine to periodically and predictably crash.

In our projects we put a lot more weight on system and integration testing than we do unit testing. That is where all the interesting bugs have come up, anyway. At least that's been my experience. You can unit test Component A and Widget B to death and have them all work, but then you hook Component A to Widget B and in some cases where the date ends in 9 on a month starting with J during lunch in the Pacific time zone, the whold damn thing fails. If the program was responsibly logging its problems and told you the database was locked and it couldn't access it and you followed this trail and saw that this is when database maintenance was happening, that's useful. No amount of unit testing is going to tell you that.

Eat well. Exercise. Die anyway. There has to be something similar somewhere about unit testing, code reviews and crashing software...

Frank Sommers

Posts: 2642
Nickname: fsommers
Registered: Jan, 2002

Re: Is Complete Test Coverage Desirable - or Even Attainable?

Posted: Feb 17, 2005 8:55 AM

> I'm thinking about a checkup-feature. The application
> could run a checkup of it's environment at startup or
> regularly or by request of the user, or... But I have
> seldom seen such a feature. I have seen a system that
> monitored the network and blocked the application when the
> network became unreachable.
>
> I can think of a lot of things that would be sensible to
> routinely check at least at startup:
> - Is the database reachable?
> - Does it have a schema I can work with?
> - Do I reach all the other systems I need?
> - Are all configuration parameters set?
> - Do they have meaningfull values?
>
> A lot of this stuff could be embedded in frameworks.
> e.g. An OR-Mapping-Tool like Hibernate for example has a
> definition of what is required from the DB schema. It
> should be possible to connect to the database and check,
> whether the schema fullfills all those requirements.
>

This is interesting, because I've been doing this kind of stuff for product that we ship. So when it starts up, it goes through a set of rules, and verifies that those rules are satisfied. If a rule is not satisified, there some amount code there that tries to fix the problem, i.e., update the db schema, as for missing user account information, etc.

This, btw, brings up the interesting question of developer testing vs runtime testing. Runtime testing offers the possibility of automatic error correction. But, of course, only those system aspects can be corrected that runtime testing can test.

One way to do some runtime testing would be to ship your developer unit tests with the product, and periodically run those tests in the deployment environment. In commercial software, does anyone ship unit tests with an application?

Frank Sommers

Posts: 2642
Nickname: fsommers
Registered: Jan, 2002

Re: Is Complete Test Coverage Desirable - or Even Attainable?

Posted: Feb 17, 2005 9:05 AM

>
> What Frank's article touches on is that there are system
> wide issues that need to be accounted for and things that
> you just can't unit test. If I'm writing an application
> for any mainstream operating system these days, except
> maybe OS X because it hasn't been through too many
> versions yet, I cannot possibly test every permutation of
> every possible environment including security updates,
> kernel updates, browser updates, etc. Some things you have
> to take on faith.

I think it is possible to test the interaction of components only when components advertise and implement an interface that has clear semantics. If we know the semantics of an interface, we can encode those semantics in testing code, and then run that testing code in a given environment. The problem is that many subsystems just don't have standard interfaces with well-defined semantics. For instance, even databases don't have standard, unified error codes.

But the bigger point is whether it's feasible to test for such things. In the case of the database insert, for instance, it's almost always more effective to just assume that the operation works.

Andrew Dalke

Posts: 291
Nickname: dalke
Registered: Sep, 2003

reference to Gödel

Posted: Feb 18, 2005 11:26 PM

We must assume that the CPU works as advertised, that the file system behaves as intended, that the compiler produces correct code. Not only can we not test for those assumptions from within the system, we also cannot recover from situations where the axiomatic aspects of the system turn out to be invalid.

I disagree. The FDIV bug on the Pentium was discovered, as I recall, by getting different results from two algorithms that should agree. fsck does checks on the file system. I've written code to work around compiler errors.
If a disk goes flaky it can be detected (eg, via checksum comparisons) and perhaps worked around, as by switching to an alternate disk.

My disagreement though is more on the formalism you use. You use the language of logic and talk about testability as axioms. Computers are an approximation to a Turing machine. No machine has infinite memory and all are built on top of an analog/quantum world. The logical model doesn't include the possibility of the tape of one machine getting too large and knocking down another machine.

More relevant, your formalism does not include economics. Some things are testable ("axioms") but are aren't worthwhile to test. I could implement three distinct algorithms to cross-check floating point multiplication. I could use a file format that can recover from a sector going bad. But these cost time and money and with almost no benefit these days.

A cardinal aspect of a test plan, then, is to determine a system's axioms, or aspects (not in the AOP sense) that cannot be proven true or false from within the system. Apart from those system axioms, all other aspects of the system can, and should, be covered in the test plan.

While I would say that a test plan should bear in mind that many possibilities are testable but the skill is in knowing which should be tested and which can be ignored.

Curt Sampson

Posts: 21
Nickname: cjs
Registered: Mar, 2004

Re: Is Complete Test Coverage Desirable - or Even Attainable?

Posted: Feb 20, 2005 8:55 PM

I see two things here that look like they might be misperceptions to me.

1. When you're trying to test your code, transient failures and suchlike are irrelevant. A server becoming unavailable due to a network failure is not the fault of your code, and not something you can fix by changing your code. You instead need to do systems analysis, and write a system that will not fail in this way (if that's the requirement).

What you can test, if you want, is your code's reaction to that sort of failure. If printing out a stack trace and exiting is not the right behavior, test that your code doesn't do that.

2. I think you're making an artificial distinction between configuration and code that is better, from a testing point of view, not to be made. Whether you use Java code or XML code or an entry in a properties file to decide what database server to connect to, you are still writing down something and saving it in a file to change the behavior of your program. If you feel it needs testing, test it!

Your web server example bought to mind something that came up a few months ago in a web application I'm working on. Most of the code is contained in PHP files, but the correct working of the site also depends on certain rewrite rules in the Apache configuration file. In fact, given that these deal with some stuff related to session control when cookies are not available, these rewrite lines are absolutely criticial to the correct functioning of the site. So I don't treat them as "configuration," I treat them as code, and test them as code. These lines reside in just one file, and that same file is used when generating test or production httpd.conf files. That way I ensure that I'm testing and rolling out the same code. And I actually bring up a web server for my functional tests to make sure that those rewrite lines are working properly.

To answer your question about the desirability of complete test coverage, or more particularly about what to cover: let pain be your guide. If something causes you pain, and does so more than once, that's probably a signal that something wants automating.

If a config file was broken, and not in an obvious way, what can you change to lessen the chances that it will be broken next time you need to make one? Can your application check that necessary settings are set? Can you generate all or part of the config file from source checked into your repository, preferably tested source? If multicast needs to be enabled for something to work, can you write a test to fail if it's not enabled?

Testing is really just about getting really creative to solve these sorts of problems.

Flat View: This topic has 21 replies on 2 pages [ « | 1 2 ]

Previous Topic

Next Topic