Alex Blewitt
Posts: 44
Nickname: alblue
Registered: Apr, 2003
|
|
Re: Go Ahead. Break the Build
|
Posted: Jan 23, 2007 8:53 PM
|
|
In my experience, people who focus on tests breaking the build haven't worked on large complex systems or have such a trivial set of tests that they trivially pass all the time.
In the real world, a build system is supposed to generate deployable code. There are other things that it can do (generate documentation, pretty reports, run tests) but it's naive to assume that build=compile+test.
When you have code that needs to be deployed to remote systems in order to test -- for example, deploying stored prodedures into a database, uploading into an application server, or refreshing a QA database with known-good content -- then things can go wrong at that level. And just because there's some failure in an obscure piece of library code doesn't mean that the entire system is useless.
Yet the anal "I'm not even going to consider running a build until all tests pass" would prevent any kind of QA test like this. Indeed, sometimes there's just runtime errors at the system test level because of borked configuration, rather than code, or because a machine is down.
The idea might be fine for trivial unit tests, but it fails to scale up.
The logical extension of this anal approach is "I'm not going to let the build continue if there are any bugs in the system". After all, when you discover a bug, you write a test for it, right? And every time someone files a new bug, a new test should be written, right? Therefore the build should fail whilst there are bugs in the system. And that's not a sensible proposition.
The right way of building large systems is to separate the concerns of 'compile/package/deploy' and 'test'. Yes, having all tests pass is a good goal to aim for. But that's not practical all the time -- if, for example, Eclipse followed this we-are-stopping-everything-because-of-a-test-failure then it would be decidedly *less* agile than it is at the moment. You only have to look at the nightlies to see that tests do fail on a nightly basis, and that they consider the build 'failed' as a whole if test pass -- but they still generate and supply the deployable outputs. That's necessary, because if I'm writing (say) the JDT code, and someone else is (say) writing the Platform code, then I still need to be able to compile and run my JDT code against the current Platform, even if it has got known bugs or test failures.
Maven's approach of not even considering any further modules in the presence of a test failure tells you *nothing* about what it didn't get around to. Yes, there may be a domino effect whereby one test causes a raft of other test failures. However, in most cases, that won't be the case -- and just because someone else has a test failure shoulnd't mean you don't get to see the results of *your* test.
The other naive thing that is built into this assumption is that the build (and its tests) are quick, and can be re-run again. As the system grows larger, this becomes less and less true. It's not uncommon to have builds which take several hours to complete for really large systems; fail-on-build in these scenarios really doesn't work.
And let's face it -- the whole point of 'agility' is to do things that work, and learn to amend/adapt the process. It's not a prescribed you-must-do-this-or-you're-not-agile approach; it's about building workable software. Anyone who says "you must do this or you're not agile" has completely missed the point of agile development techniques -- the process is supposed to *adapt*, not be fixed in stone.
Fortunately, as systems grow, and people get more experienced, they find that this doesn't work all the time and so adapt their processes. Those that haven't been working on such systems for a while, tend to mandate "this is the way it must be done" without appreciating the downsides. The point is this: you're either agile, and adapt; or you don't.
Alex.
|
|
Previous Topic
