Summary
Software engineering is a lot less like other kinds of engineering than most of us would like to think. There is an aspect of art to what we do, that is learned not in school but by finding a master and serving an apprenticeship.

It's been over a month since I last posted, a break explained by the intervention of JavaOne and the experience of being Slashdotted on my last installment. But July is here, so it is time to declare independence once again and post some thoughts.

During the last couple of weeks, I've been reading a number of articles by Paul Graham on the connection between programming and art and the similarities in different kinds of things that we call design. I like a lot of what Graham has to say (and agree with some of it as well), and while he sometimes takes a long time to say it (at least for a web publication), it is because his points are complex rather than because his writing is long-winded. I highly recommend the site, both for its look and its content, especially his articles on what it is to develop software.

Software development is an odd sort of activity. While we in the industry talk about being engineers, it is engineering of a rather odd sort. The body of established science that you need to know to become a software engineer is rather small, and there are a number of very good software developers who have never formally studied the field at all. I consider myself at least a competent software designer. But my own formal training was in philosophy (as it turns out is also the case with Paul Graham), and about a third of those with whom I went to graduate school are now designing software. And nearly all of us made the switch with little or no formal training in software engineering or computer science.

While it is less common now than it was when I started in the field (low those many years ago), there are a surprising number of excellent software developers and designers whose background is in subjects such as mathematics, or music, or anthropology, or a number of other disciplines that are not thought of as engineering. While I don't think that it is impossible, I would bet there are very few people who switched from, say, philosophy or music to, say, mechanical or electrical engineering without any additional formal education. This is certainly less true now than it was 20 or so years ago when I started; I doubt that I could find the positions now that I was able to find then that allowed me to make the switch. But the fact that it was possible until fairly recently says something, I think, about the nature of software production.

One possible explanation is that software engineering is a fairly new field. There isn't a large body of knowledge that one needs to master to become a software engineer; you can start doing reasonable (and useful) work while picking up that knowledge. The same may have been true in the past of other forms of engineering, but now those fields have matured to a point where there is a lot of accumulated knowledge that is best learned as part of a formal education. But I think that the real reason is somewhat different than the youth of the subject; I think the real reason is that software engineering isn't like other forms of engineering so much as it is like a craft or an art form, both activities that can be learned by apprenticeships rather than formal education.

The notion of an apprenticeship is rarely talked about by the human resources types (although the newest incarnation of the idea, that of mentoring, is becoming pretty respectable), but it is a pretty standard way of getting your real training in software development. Ask most any good software developer who he or she served an apprenticeship with, and they will start telling stories of the master craftsman from whom they learned their art. Most won't even find the question odd, or ask for an explanation. How else could we have learned what we do?

This is even (or perhaps especially) true of those who have had formal education in software engineering. I've often told university professors that people coming straight out of school are mostly useless for at least six months when they enter my groups, and that it takes them about a year to become really useful. This is true for those with master's degrees and with bachelors (don't even ask me about Ph.D.s). And it shouldn't be surprising...after all, the sort of software development that is done in a university setting is very different than the sort of development (and maintenance) done in an industrial setting. Learning the latter takes more than a course; it takes practice, and the accumulated practice of someone (or someones) who have done it long enough and successfully enough to have an idea of the craft involved.

Of course, another reason that there is no good way to teach a class that will train people in the aspects of design and software development that are learned during an apprenticeship is that there is no single method that is right. There are many ways of doing good software development and design, just as there are many ways of doing those things badly. Some good designers will work long and hard on a specification before any code is written, trying to figure out all of the structure of a system before changing that structure becomes difficult. Others will do lots of prototypes, iterating over the code until the structure of the overall design becomes clear. Others do some combination of these two approaches; I'm sure that there are others that do something entirely different. The designs that result from these different approaches are not better or worse than those that result from some other way of coming at the problem. But they are different.

All of this was studied by Fred Brooks, author of the classic Mythical Man Month and Turing Award winner; the results of his study led to his papers on the design of design. His conclusion, which surprises few good software engineers but dismays many in management, is that good design comes from good designers, not from some process. This may dismay many managers because it means that the engineers they employ are not interchangeable parts that can be insured to give good results by the application of the right process. But it is the truth, and they need to deal with it, as do those who want to take the individual out of design by having a more (supposedly) rigorous set of rules for software engineering.

A corollary of the Brooks conclusion that good design comes from good designers is that good designers receive their training from other good designers. This training happens during the apprenticeship, when attempts at design are reviewed and criticized, when code is read, or during conversations over the lunch table. It generally doesn't happen in a classroom (although the master craftsman may be a teacher, who does the training during reviews, readings, and lunches). And, much like working with a Zen master, it happens at different times and in different ways for different people. Someone who is a good master for one person learning the craft may be just wrong for another.

The point of all this, if there is one, is that those who want to reduce software engineering (by which I mean the designing and producing of high-quality software systems) to a set of rules and procedures that can be followed by anyone to get consistently good results simply don't understand the field. Yes, there are processes which can help in the production of good software (any group that doesn't have a good way of doing code reviews is fooling itself if it claims to be producing quality software). But while lack of such processes can insure the production of bad software, having such processes does not insure the production of good software. For that something extra is needed, which is the craft and aesthetics that result from training with a master in the art.

In this way, what we call software engineering is really more like architecture (real architecture that produces buildings) than it is like other kinds of engineering. There is an element of science (a building has to obey the laws of physics) but there is also a large element of art. Depending on what kind of software you produce, the mix of the two may differ some, but there is always a mix. Much of the science can be taught in schools, but the art can only be learned by practice and can be produced in many ways. Those who are looking for the one true way to do the art don't understand what it is we produce, or wish that what they understand can be replaced by something more "scientific." It can't, so they should just get over it.

And, I might add, it is a good thing that what we do requires both art and science. That is what makes it so much fun...

Talk Back!

Have an opinion? Readers have already posted 27 comments about this weblog entry. Why not add yours?

RSS Feed

If you'd like to be notified whenever Jim Waldo adds a new entry to his weblog, subscribe to his RSS feed.

Digg | del.icio.us | Reddit

About the Blogger

Jim Waldo is a Distinguished Engineer with Sun Microsystems, where he is the lead architect for Jini, a distributed programming system based on Java. Prior to Jini, Jim worked in JavaSoft and Sun Microsystems Laboratories, where he did research in the areas of object-oriented programming and systems, distributed computing, and user environments. Before joining Sun, Jim spent eight years at Apollo Computer and Hewlett Packard working in the areas of distributed object systems, user interfaces, class libraries, text and internationalization. While at HP, he led the design and development of the first Object Request Broker, and was instrumental in getting that technology incorporated into the first OMG CORBA specification.