Summary
Software production encompasses the steps between developers checking code into a version control system and users enjoying the fruits of developers' work, according to Electric Cloud CEO Mike Maciag. Maciag thinks that production is currently the biggest bottleneck in software development, and believes his company has the answer in a new product.
Advertisement
A few weeks ago, John Ousterhaut spoke on these pages about the importance of recognizing the next big bottleneck in software development. Mike Maciag, CEO of Ousterhaut's company, Electric Cloud, believes he spotted the biggest current bottleneck: the steps software has to go through between developers checking code into the version control system and a final release—what Maciag calls software production.
Artima spoke with Maciag, and Electric Cloud product marketing manager Erin Curtis, about software production, and a new Electric Cloud product, Electric Commander, that aims to smooth passage through the production straights:
What happens after software leaves the source code control systems? How does it get built, how does it get packaged, how do you test it, and how do you deploy it?
While people use make or ant as build tools, they need tools past the build process. [For example], if you want software to build every time a developer checks source code into a source code management system, you'd typically use some custom scripting to do that. Or, to tell the system that when someone does a certain type of check-in, go and execute a parallel build across several platforms and, if you run into a problem with the build or the test suite, email Joe and Ed the log files so they can figure out what happened. And if you get a green build, move the code over to a system to do system-level testing, and ultimately put that [code] up on an FTP Web site.
That infrastructure is outside of make and ant. This is what we call production, and is done typically with home-grown tools.
The reason the production part is becoming the bottleneck today is that software is being integrated and produced much more frequently than has ever been before. Agile development causes production to be the next bottleneck, because it encourages more frequent builds and deployment.
Maciag believes tools and processes haven't caught up with the amount of software teams create, and with the frequency teams must release their work:
Software creation, design, definition, and coding [are steps] that had lots of investment in recent years, making those steps much more efficient, much faster, and [of] much higher quality. That [investment] has created an entire market around source control management, it created an entire IDE environment, it created visual development tools, and so on.
What has been largely ignored, is the software production side now. In ninety-nine percent of the environments today, all of that is done with scripting. People over time have cobbled together their own software production environments...
The traditional problems in the software production side have been that it's too slow, it's brittle, and is too hard to understand... People spend a lot of time writing tools rather than writing software to get good productivity out of developers. These [solutions] also don't lend themselves to reuse, and are hard to diagnose.
While some organizations have highly complex production requirements—to comply with regulations, for instance—other companies have numerous release processes that are individually less complex but, taken together, are hard to understand, according to Electric Cloud's Erin Curtis:
There are different levels of complexity [in software production]. Some [companies] have a few jobs that are massively complex. Other companies have just a lot of jobs and configurations. Each job may not necessarily be large, but there are a large number of them. Some companies have a large matrix of product features and combinations, and each combination has to run through a series of tests. And those tests must run frequently.
Decentralization of the production process [around home-grown departmental solutions] is a problem because there is no way to do reuse, no way to share common components, and there is very little you can do to share standards across an organization from a compliance or organizational standards point of view... Scalability is also a problem: If you have to run your tests frequently, for instance, you may get more performance out of parallelizing some test runs... but creating such a solution is not trivial.
Maciag believes he has the solution to the software production problem in Electric Commander, a highly configurable continuous integration system:
Electric Commander is a Web application [that] ties the entire deployment from build to deployment together. It lets you specify any task you'd like it to perform in the production process via either an Ajax-based Web user interface, or a command line. It integrates with Perforce and ClearCase, and support for other source control systems are in the works.
Projects in Electric Commander [serve as] an organizing mechanism, and under projects, are procedures. A job gets run as part of a procedure, and each procedure may have a series of jobs that are run. You can define all those tasks through the UI... Some of the jobs will execute in parallel, as agents sitting on each of the machines we manage listen to events. The Web-based application also allows you manage those agents.
According to Maciag, you can define custom tasks in Java, and tie that into the production manager software to execute when certain conditions are met, such as when source code is checked into the system, or when tests succeed or fail.
Maciag believes Electric Commander extends the concept of a continuous integration server because it provides more flexibility, and also provides more analytic feedback on the process so that teams can improve the effectiveness of their software production process.
To what degree is production a bottleneck in your projects? What tools and techniques do you use to release software to end-users?
I agree that getting software through the production stage can be a problem, and there are quantifiable risks with home-grown solutions. Those risks are largely the same as any build-vs-buy situation, though, and whether or not Electric Commander is the right solution for a given shop would have to be determined individually. It certainly seems "enterprisey" enough.
Based on my experience, a more fruitful solution might be to address the forces that result in the problem in the first place.
First, replace any programmers on the team whose "expertise" consists of knowing which wizards to run and what settings to choose in the favored IDE. Claiming experience in EJBs or Web Services on the bases of knowing how to make the development environment spit out the generated code borders on fraud. Along the same lines, a developer that claims a feature as "done" because the source that is checked in will build and run on that programmer's workstation under the IDE is also a candidate for replacement.
Second and most importantly, project managers and management in general that accept a half-baked definition of "doneness", such as outlined above, also deserve to be replaced. Until the functions are integrated into a deployed version of the software to the point were real users (or their proxies) can exercise and give feedback on a feature, there's no point in any project plan having it marked as done. Yet I've seen plenty of instances where scads of requirements are marked "done" long before they are in any shape to be put in front of users.
If there's a bottleneck in the steps between "checked in to source control" and "production-ready deployment", it's from bad practices, not the lack of tools or lack of centralized, standardized control over anything.
> If there's a bottleneck in the steps between "checked in > to source control" and "production-ready deployment", it's > from bad practices, not the lack of tools or lack of > centralized, standardized control over anything.
I agree completely with your definitions of the symptoms but not your solution, which I would paraphrase as "Sack everyone until you're only left with the ones who do things the way you want.". I suspect motivation and staff retention might become problems. In fact, it's nothing more than 'blame culture' imposed from the top.
To run a successful team you need to offer more "carrot" than "stick" and there's no sign of "carrot" in your solution (and no, not getting sacked is not a carrot).
One small "carrot" might include telling your staff what you expect of them (there's no mention of that in your solution other than sacking ones that don't do it). Defining procedures, offering training, mentoring, etc. There are a whole host of solutions (most of which come from the top) that are way superior to a 'hire & fire and hope the right people come along' style of (non-)management you propose.
Previous Topic
