As I understand it the canonical form of XP coding is something like this:
Sync
Write Tests
Code
Build cleanly
Test cleanly
Merge
Build cleanly
Test cleanly
Commit
Often times a restructuring of code results in the modification of a large number of files. For example let's say a goodly portion of your code implements an interface or subclasses something called ReallyPervasiveAndUsefulSomethingOrOther and you've decided to rename it to the more finger friendly UseItOrDie. With a reasonable tool this is more or less at mechanical operation requiring little thinking. Being a good XP citizen you run the unit tests and, since this logically single operation has touched lots and lots of files you need to run the entire suite. Let's say this suite takes some real time because your application isn't some out of a book trivial something or other, it's a real product with lots of bits and pieces. Maybe you start the suite and go to lunch. Come back from lunch and, hey, everything passed. No surprise you didn't really change anything. Push the commit button.
Suddenly the lights dim. It's hot. Smokey. The smell of brimstone assaults your nose and your eyes are watering. You hear the unmistakable sounds of people, some cursing under their breaths, some quietly weeping. The frantic, panicy click-clack slapping of keys. Welcome to Continuos Integration Hell. If you haven't been here before you might be wondering how you got here. I mean, honestly, you followed the true path and doesn't that path lead to coding nirvana? Evidently that's only in the story books.
If the time it takes to run a set of tests exceeds the average time it takes for a commit to the tree to occur by someone not yourself, your commit can fail because, surprise, your large and pervasive change has touched a file or files that someone else was working on and has now checked in. This is the basis for Continuous Integration Hell. You merge/build/test/commit/fail again and again and again, trying to keep up with everyone on your team. There is no tasteful exit from this loop.
You can lock the tree. Doing this often is an excellent way of discovering who your real friends are. It is also an excellent way of discovering that you have fewer friends than you thought you did.
You can try to address the fundamental problem and make the tests run faster. There is a point where even this won't help. If you have 2000 tests with an average time of just 1 second it's still 1/2 an hour. And 2000 tests doesn't seem to be a particularly large test suite.
Partitioning the problem space is an excellent idea but has profound implications for the XP process. I won't address those here. Suffice to say that XP in the large looks to be a really dubious proposition.
You can not run all, or any, of the tests. This is what seems to happen most frequently and seems to be the most human path. Inevitably you break things. Eventually everyone breaks things. So when you break something nobody is too terribly bothered. It's unfortunate because, of course, sometimes some poor soul has pulled your changes into their tree. If they see the broken bits you just put in they may (usually will) spend some time determining it's not them. Best in this case to hope they broke things at some point in the past also, lest they decide to cause you harm. That's human too.
You can schedule your change for a time when nobody will be working on the tree. This is semantically equivalent to locking the tree with out the impact on those you formerly called your friends. Sleep or the illusion of friendship is often a hard choice to make.
For myself continuous integration is, far and away, better than isolated branches and the big bang merge towards the end of a project. You know more sooner, and your schedule is outrageously easier to manage. But sometimes it's just hell.
I think the goal of writing a good unit test harness is that running all the tests should be extremely quick.
If your tests require database connections, consider using Mock Objects. Or break out your data access into interfaces and create a very light weight file based data store. You're solved two issues here: your tests run w/out a database, and your code is a bit more flexible and testable.
The more and more I write my tests using Mock Objects, the quicker my tests become. Also, it minimizes the need for external resources which makes setup easier.
Tests can take a long time, but through careful creation, can execute quickly.
I think some form of locking is inevitable. Steps 6 through 9 (Merge, Build Cleanly, Test Cleanly, Commit) need to be protected from commits by other developers, otherwise someone is wasting their time.
In XP Explained, the integration machine is a very visible form of this locking. We've the same thing with "check-in icons" (a large foam hand in one case, a plush German Shepherd in another). Before entering step 6, get the icon, and put it back in the middle of the room after you've committed.
There are source control systems that try to make branches and merges easier, giving developers private streams to commit to, but this just defers the problem to the time when you need to merge developer streams. (To me, that sounds like you've traded a problem for an uglier one.)
Seth, the problem with mock objects is you aren't testing real code. This devalues the tests to a large degree imho. I'd rather have slower tests that are testing under more real conditions. It's in this interaction where most interesting problems occur.
Also, the only two options aren't big bang integration or continuous integration. It seems in XP everything is black or white. We have defined integration points for each feature and it seems to work fine.
Also also, you don't have to lock until you are really ready to commit. Sync and integrate then test. Fix things. Then sync, integrate, lock, test. There's a window where changes can come in the second round, but it is rare. Most problems are solved in the first phase and nobody was blocked on you.
I reckon this "CIH" problem doesn't occur if your version control system only allows exclusive checkouts. I thought exclusive checkouts were still pretty much the standard outside of CVS. But even with CVS, don't you do an update before a commit to see if you have any conflicts? I guess the problem is if someone added the new line
OldClassName newThing = new OldClassName();
when you had just changed the class name? If that's the case, wouldn't this be caught during the compile after the checkin/merge, before running unit tests? Or have I misapprehended the problem?
> If your tests require database connections, consider using > Mock Objects. Or break out your data access into > interfaces and create a very light weight file based data > store. You're solved two issues here: your tests run > w/out a database, and your code is a bit more flexible and > testable. > > The more and more I write my tests using Mock Objects, the > quicker my tests become. Also, it minimizes the need for > external resources which makes setup easier.
This seems like it may facilitate the unit-test simplicity and speed at the expense of veracity.
It seems to me that writing mock objects could be a lot of work, with questionable value in many cases. To simulate a database, would you have to write a mock object that parses SQL and implements the whole database interface with connections and cursors and the lot? Where do you draw the line?
I recently worked on a project where I was writing DVDs. I gave up on creating a mock object to simulate the DVD writing component because the biggest problem I had with that component was unexpected errors, hangs and crashes from a variety of software, firmware and hardware configurations. At each new discovery, I could continue to update the mock object to misbehave the way the real hardware and drivers did, but that didn't really seem to be very productive and would have to be revisited with every software, firmware, driver or hardware update. Additionally, when really writing a DVD, there are many interesting timing and system performance issues that arise which would be nearly impossible to predict and then pretty difficult to simulate with pure code.
This question received a dearth of responses on another similar thread, but I'll try again here: What is the unit testing solution to this kind of problem? As I've not heard any unit testing advocates concede that there are cases where unit testing doesn't apply, I gather there must be an answer.
Sometimes pervasive changes are necessary, but in general under continuous integration they would be unusual events. Before making a pervasive change, consider having a formal review of the changes, and make the change day part of the whole team's awareness, say right at the beginning of an iteration.
It's all about communications on an agile team. Reviewing only before big changes would be a sign that there is a need for more communications among the team. Similarly. a single person doing a pervasive change without the rest of the team having some forewarning that it is coming would be a sign that communications can be improved.
But you should still look at trying to cut the time it takes for the test suite to run. There are case studies out there where this has been done.
> Often times a restructuring of code results in the > modification of a large number of files. For example let's > say a goodly portion of your code implements an interface > or subclasses something called > ReallyPervasiveAndUsefulSomethingOrOther and you've > decided to rename it to the more finger friendly > UseItOrDie. With a reasonable tool this is more or less at > mechanical operation requiring little thinking. Being a > good XP citizen you run the unit tests and, since this > logically single operation has touched lots and lots of > files you need to run the entire suite. Let's say this > suite takes some real time because your application isn't > some out of a book trivial something or other, it's a real > product with lots of bits and pieces. Maybe you start the > suite and go to lunch. Come back from lunch and, hey, > everything passed. No surprise you didn't really change > anything. Push the commit button.
Hmmm, a couple of thoughts here. AFAIK one of the "laws" in the Refactoring book is: never make big refactorings at once. Of course the book was talking about complex refactorings, but I use this rule in any situation where I need to change classes that aren't in my immediate control. Also for mathematically proven behaviour preserving refactorings (yes there's math behind refactorings) I don't run the tests, because I believe in math ;) In this case is really simple.
-Step two: pick one class, update from CVS, change it to use UseItOrDie instead of ReallyPervasiveAndUsefulSomethingOrOther where the use is covariant, compile and commit.
-Step three: if the class from step two has a contravariant or invariant use of ReallyPervasiveAndUsefulSomethingOrOther, find its clients and make then use UseItOrDie instead. get the class back and finish the changes to UseItOrDie.
Step four: if you still have more classes using the old name go to step two, else you're done.
While you are making this refactoring you'll sure notice that so many classes depending directly on a single interface is poor factoring, so apply another forms of refactoring to reduce coupling and never get yourself in that place again.
I'm not a XPer, never could find a team that would use it (I like the practices), but I use TDD, YAGNI and refactoring all the time (pair-programming whenever I can) and I never had problems with CIH or had to do big refactorings. After finishing a new feature (story, use case, whatever) but before final commiting I always do design tests: dependency checks and CCN counts. If anything is highly coupled or overly complex I refactor and test until things are loosely coupled and simple. There are lot's of good free (either as in beer or as in speech) tools for that (e.g. JDepend, Compuware's Pasta, JavaNCSS).
People always build a strawman with XP practices, forgeting that all should be followed. If refactoring lot's of classes is hard: don't do it, YAGNI. If too many classes depend on a single class: don't do it in the first place because everything should be said once and only once. If your entire test suite takes hours to run, refactor it: reduce duplicated tests, factor out tests so you'll know, looking at dependency graphs, wether your change affected or not some of your tests. Changing XML generation code shouldn't affect the DBMS connector stuff, if it does there's something really wrong with your code. But I think the entire test suite should run at least once every day, after all the daily commits to ensure the consistency of the system.
> Seth, the problem with mock objects is you aren't testing > real code. This devalues the tests to a large degree > imho. I'd rather have slower tests that are testing under > more real conditions. It's in this interaction where most > interesting problems occur.
When you use mocks you are testing real code. You are testing whether it communicates correctly with another class. The salient point is that mocking is a pain so if you are serious about testing and keeping your test suite speedy, you ruthlessly simplify the protocol between classes. That that has its own beneficial design effects, including making problems less likely.
>When you use mocks you are testing real code. >You are testing whether it communicates correctly >with another class
For the same reason people say you don't need to test setters and getters, i don't really find a lot of problems with incorrect communication with other classes. For all the pros of the design part of TDD, i still want to find bugs and to find the "real" bugs you need to test the real code. If that takes time then that takes times.
> For the same reason people say you don't need to test > setters and getters, i don't really find a lot of > problems > with incorrect communication with other classes.
Good, so the unit tests which test the functionality of a class in isolation rather than in communication are more important.
> all the pros of the design part of TDD, i still want > to find bugs and to find the "real" bugs you need > to test the real code. If that takes time then that > takes times.
That's a sore point with me. When I concentrate on finding bugs, I often feel that I should be spending more time working in ways which prevent bugs. One way I do this is to write preventative tests. They show me mistakes I make during change when I make them.
>That's a sore point with me. When I concentrate on finding >bugs, I often feel that I should be spending more time >working in ways which prevent bugs. One way I do this is >to write preventative tests. They show me mistakes I make >during change when I make them.
It's fine for you to take corrective measures to balance your weaknesses, but don't generalize and make that the whole of the law.
But i am not sure where this preventing bugs came from. The idea is to test against what something actually does, not what your mock simulates. This has nothing to do with doing anything extra to prevent bugs, it has to do with wasting your time on work that doesn't work.
For example, if you make a file system encapsulation class that works across nfs, htfs, dos, ntfs, and a flash file system on multiple different operating systems, then testing against a mock is a waste of time. I can tell you from personal experience that these all have their own quirks. Adding an intermediate mock is sink, not a help. If the tests pass in one part of the test matrix then it's very likely the underlying logic is correct. If they pass it's usually do to some wierdness that i need to account for to make the class work in real life, and this is the goal.
Mocks let your unit tests run fast. For slow things like network connections to other servers, have a FEW tests that use the network, and have MORE tests that use a mock. How are you going to test network failures without bringing down the server all the time?
Automated Acceptance Tests, which are run less often, should use real objects "all the way down".
If you have io-stream classes for a bunch of difference protocols, each of those classes has unit tests. Elsewhere, a piece of code that uses an io-stream can use a 'fast' (MAYBE mock) io-stream in its unit tests, instead of the actual one that will be used in the real product (and the acceptance tests).
Any io-stream-related bugs discovered in the acceptance tests should result in new unit tests for the relavant io-streams classes and other relevant classes.
>Mocks let your unit tests run fast. For slow things >like network connections to other servers, have a >FEW tests that use the network, and have MORE tests >that use a mock. How are you going to test network > failures without bringing down the server all the time?
Then i am spending considerable time creating tests that skip about a gazillion failure interaction modes. That's time i would rather spend on finding real problems in real code and creating real code to solve them. Mostly because they are much easier to find, reproduce, and fix in the unit test environment. If i wanted to find these problems in other levels of test then i think you can do away with most unit testing period because you can use the same arguments.
I do use mocks, have for many many years, but that is a last resort for me rather the way to do everything.
In the network case i create a process on the same box and i can fail that in various realistic ways relatively simply. I can't test ip related failover problems, system tests have to do that, but i can do a lot and have done a lot.
This may sound stupid, but things are what they are. If you have a rule like tests must be fast, and apply that rule blindly, then you are not paying attention to what things are, you are just paying attention to the rule. This causes you to justify dropping testing in favor of test speed.
If your code passed a unit test using mocks but failed in a system test, you would get no sympathy for me. Your shit must work, i don't want to hear about the rest.
Subversion's commits are atomic, and will fail if someone else has changed a file you are going to check in since you last updated/merged. See http://subversion.tigris.org/
The manual page says the following about the state where the problem you mention could happen:
Locally changed, and out-of-date
The file has been changed both in the working directory, and in the repository. An svn commit of the file will fail with an "out-of-date" error.
So if you make changes and run your tests, your commit may reveal that one of your files is out-of-date, and your commit will fail. In this case, you need to update/merge the local source, compile, and test, and try to commit again.
Flat View: This topic has 40 replies
on 3 pages
[
123
|
»
]
Previous Topic
