Weblogs Forum - Continuous Integration Hell

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Weblogs Forum
Continuous Integration Hell

40 replies on 3 pages. Most recent reply: Oct 28, 2008 9:14 AM by tracy ragan

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 40 replies on 3 pages [ « | 1 2 3 ]

Matt Gerrans

Posts: 1153
Nickname: matt
Registered: Feb, 2002

Re: Continuous Integration Hell

Posted: Jan 30, 2004 10:52 AM

Reply

Advertisement

> Here are some real numbers. According to Junit I have 420
> tests. They executed in 5.88 seconds. That is, instead
> of the one second per test that was assumed I
> measured 14 milliseconds.
>
> 2,000 tests would take 28 seconds.

That's just meaningless anecdotal data and it makes no sense to extrapolate it. Obviously, the time it takes to run a unit test is intimately related to what the test does; in many cases it may be simple and quick, but in some cases it may necessarily be a more lengthy process.

Jon Strayer

Posts: 10
Nickname: jstrayer
Registered: Sep, 2003

Re: Continuous Integration Hell

Posted: Feb 3, 2004 11:10 AM

Reply

> That's just meaningless anecdotal data and it makes no
> sense to extrapolate it.

If you have better data then post it. It may not be much, but it's better than the once second per test assumed in the article.

> Obviously, the time it takes
> to run a unit test is intimately related to what the test
> does; in many cases it may be simple and quick, but in
> some cases it may necessarily be a more lengthy process.

A unit{/b] test that takes a second to execute is testing way too much.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Continuous Integration Hell

Posted: Feb 3, 2004 4:18 PM

Reply

>
> A unit test that takes a second to execute is
> testing way too much.

Here's an example. I have unit test that tests the RSS feed grabber that is used in News and Buzz on Artima that tests that the socket actually times out if the other side is too slow. If after 20 or 30 seconds -- I don't remember the exact cutoff -- have elapsed and the socket is still sucking down data, or more likely waiting for data to arrive, the socket should abort. That unit test sets up a socket to another socket that never hangs up, to test the timeout. So therefore, that one test takes 20 or 30 seconds.

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: Continuous Integration Hell

Posted: Feb 3, 2004 6:14 PM

Reply

> Here's an example. I have unit test that tests the RSS
> feed grabber that is used in News and Buzz on Artima that
> tests that the socket actually times out if the other side
> is too slow. If after 20 or 30 seconds -- I don't remember
> the exact cutoff -- have elapsed and the socket is still
> sucking down data, or more likely waiting for data to
> arrive, the socket should abort. That unit test sets up a
> socket to another socket that never hangs up, to test the
> timeout. So therefore, that one test takes 20 or 30
> seconds.

How about changing it to time out after one second? It would be a slight design change, parameterizing the class so it can be fast under test but patient during production, but it would allow you test the logic without taking as long.

Matt Gerrans

Posts: 1153
Nickname: matt
Registered: Feb, 2002

Re: Continuous Integration Hell

Posted: Feb 4, 2004 7:23 PM

Reply

Okay, how about if I'm unit testing my Mars lander commication module?

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: Continuous Integration Hell

Posted: Feb 4, 2004 7:47 PM

Reply

> Okay, how about if I'm unit testing my Mars lander
> commication module?

Same deal. Fake the timer or fake the socket. You are testing the logic not the sockets and timers. You can use higher level tests to test the integration of the pieces, but those tests aren't unit tests.

Jeremy Ferry

Posts: 4
Nickname: ferr0084
Registered: Nov, 2002

Re: Continuous Integration Hell

Posted: Feb 9, 2004 7:48 AM

Reply

> Same deal. Fake the timer or fake the socket. You are
> testing the logic not the sockets and timers. You can use
> higher level tests to test the integration of the pieces,
> but those tests aren't unit tests.

Exactly. It seems that there is some blurring of the lines between unit tests and functional or integration tests. My understanding is that a unit test is intended to test a concise unit of code not how well your many code units work together. Has unit testing become a Golden Hammer?

Matt Gerrans

Posts: 1153
Nickname: matt
Registered: Feb, 2002

Re: Continuous Integration Hell

Posted: Feb 9, 2004 3:32 PM

Reply

> My understanding is that a unit test is intended to test
> a concise unit of code not how well your many code
> units work together.

Well, I guess that would depend on the definition of concise unit. Is it smaller than a single function?

As far as faking goes, it may be useful, but I think there are two serious limitations it can come up against:

1) Setting up the fake part (mock objects, data and so on) may be an exorbitant amount of work.

2) It is a fake test. I'm not saying that this means it is worthless, mind you.

So, if you do have automated tests that involve hardware or networking and the concomitant time delays that might be involved, are those not unit tests? Would you call them "automated functional tests" or something like that?

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: Continuous Integration Hell

Posted: Feb 10, 2004 5:40 AM

Reply

> > My understanding is that a unit test is intended to
> test
> > a concise unit of code not how well your many
> code
> > units work together.
>
> Well, I guess that would depend on the definition of
> concise unit. Is it smaller than a single
> function?

It depends on what you want to test.

> As far as faking goes, it may be useful, but I think there
> are two serious limitations it can come up against:
>
> 1) Setting up the fake part (mock objects, data and so on)
> may be an exorbitant amount of work.

It's often less than you think. If you do it often, you get used to it and it goes faster.

> 2) It is a fake test. I'm not saying that this means it
> is worthless, mind you.

It is no less fake than any other test. When you write a test with mocks you are being explicit about what you are testing and what you are not testing. There's none of this wishy-washy "oh, yeah, I have tests." You can point at the particular pieces of code that are tested in the system.

> So, if you do have automated tests that involve hardware
> or networking and the concomitant time delays that might
> be involved, are those not unit tests? Would you call
> them "automated functional tests" or something like that?

Yes. There are many names for those sorts of tests.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Continuous Integration Hell

Posted: Feb 11, 2004 12:13 PM

Reply

> > Here's an example. I have unit test that tests the RSS
> > feed grabber that is used in News and Buzz on Artima
> that
> > tests that the socket actually times out if the other
> side
> > is too slow. If after 20 or 30 seconds -- I don't
> remember
> > the exact cutoff -- have elapsed and the socket is
> still
> > sucking down data, or more likely waiting for data to
> > arrive, the socket should abort. That unit test sets up
> a
> > socket to another socket that never hangs up, to test
> the
> > timeout. So therefore, that one test takes 20 or 30
> > seconds.
>
>
> How about changing it to time out after one second? It
> would be a slight design change, parameterizing the class
> so it can be fast under test but patient during
> production, but it would allow you test the logic without
> taking as long.

Well, I wasn't complaining about the wait. I was just giving an example of a reason to have a longer test. The full test only takes a minute or so to run, inluding the 20 or 30 seconds of this socket timeout test, and that gives me time to stare off into the distance and dream of ocean breezes and swaying palm trees. If the total time were unacceptable, I would likely parameterize the timeout just as you suggest, but then of course I would lose 20 seconds of valuable ocean breezes dreaming time.

tracy ragan

Posts: 1
Nickname: tracyragan
Registered: Oct, 2008

Re: Continuous Integration Hell

Posted: Oct 28, 2008 9:14 AM

Reply

Part of the challenge with continuous integration is "just how continuous" your integration is. The best situation for a continuous process is to prevent each developer from bringing down the entire code tree to their local sandbox. Instead, your process should allow them to bring down only the files that they are working on to the local sandbox and the remaining components come from a shared area that is a reflection of the latest changes checked into your SCM tool and compiled with your CI server.

I know this is a challenge as standard build scripts do not allow you to concatenate your directories using a "first found" method, and pass the correct files to the compilers. Meaning, look local first if you find it use it, if not look now at what is stored in SCM tool and use that. When you develop a build process that supports this level of file management, you are creating a continuous process that occurs automatically, all the time. Every developer immediately sees the changes committed by other developers without having to synchronize their local code. When your build can manage files in this way you start solving some of the CI "hell" you refer to. In other words, you find the problems as soon as your team mate checks in source code, instead of finding them later, when you commit your code and then realize you just coded down a rabbit trail that now needs to be integrated - no small task.

Great post topic. We at Openmake Software hear about this challenge all of the time from our prospects and love to explore solutions. After all CI builds is more than just executing a static build script and test at certain times of the day - it offers us the chance to really improve the way we work and coordinate our changes on a frequent basis to avoid the hell you refer to.

Flat View: This topic has 40 replies on 3 pages [ « | 1 2 3 ]

Previous Topic

Next Topic

Sponsored Links

Copyright © 1996-2019 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use