A Conversation with Andy Hunt and Dave Thomas, Part V

---

Andy Hunt and Dave Thomas are the Pragmatic Programmers, recognized internationally as experts in the development of high-quality software. Their best-selling book of software best practices, The Pragmatic Programmer: From Journeyman to Master (Addison-Wesley, 1999), is filled with practical advice on a wide range of software development issues. They also authored Programming Ruby: A Pragmatic Programmer's Guide (Addison-Wesley, 2000), and helped to write the now famous Agile Manifesto.

In this interview, which is being published in ten weekly installments, Andy Hunt and Dave Thomas discuss many aspects of software development:

• In Part I. Don't Live with Broken Windows, they discuss the importance of software craftsmanship and the importance of staying on top of the small problems in your projects.
• In Part II. Orthogonality and the DRY Principle, they discuss the importance of keeping your system orthogonal, and the real meaning the DRY, or Don't Repeat Yourself, principle.
• In Part III. Good Enough Software, they discuss the myth of bug-free software, the importance of specifying level of quality as a system requirement, and the need for every team member to inject quality throughout the development cycle.
• In Part IV. Abstraction and Detail, they discuss an approach to design in which details are pulled out of the code and stored as metadata.
• In this installment, they discuss reversible design decisions, the cost of change curve, going beyond the requirements, and making systems configurable.

Reversibility and the Cost of Change Curve

Bill Venners: In your book, The Pragmatic Programmer, you suggest keeping in mind that design decisions are not necessarily final. You recommend organizing the system so design decisions are reversible. How do you balance reversibility with other concerns, such as speed of development, performance, clarity, simplicity? For example, say I decide today that I'll use an Oracle database. Do I use an Oracle-specific API to talk to the database, or a generic database API? Perhaps the Oracle-specific API is clearer or faster, but makes it harder to change databases later. Either way I'm taking a risk.

Andy Hunt:You're taking a risk either way, but I would say that approach is backwards. Instead of committing to the Oracle-specific API now and hoping it's faster, use the more general API first. If the general API is not fast enough, then make the conscious optimization decision to use the specific API for certain performance critical parts. Everybody going back to the K&R book has warned against premature optimization, and I think that's an example of it.

Dave Thomas: It also comes down to our old friend the cost of change curve. The cost of change curve basically says that the cost of making a change increases exponentially over time. There are various expressions of it. For example, the cost of fixing a bug after a system has been deployed is 1000 times more than fixing it when the system is being designed. But the general agreement is that the curve goes up non-linearly as time goes on.

The meter of the cost of change curve starts running when you make a decision. If you don't make a decision, then there's nothing to change, and the curve is still flat.

Andy Hunt: The world can change its mind as many times as it wants. If you haven't made a decision or committed yet, your cost is zero.

Dave Thomas: So rather than make a whole bunch of decisions up front and start the meter running, we try to defer each decision as long as we can. We end up with a lot of small cost of change curves, because each one hasn't had a chance to get up too high. Cumulatively, the effect of adding up those small curves is a lot less than having one curve that starts at zero that ramps up to infinity real quickly.

Why Are You Guessing?

Bill Venners: You say in your book, "Not sure how marketing wants to deploy the system? Think about it up front and you can support a stand-alone, client-server, or n-tier model just by changing a configuration file. We've written programs that do just that." What complexity does this add and how much more time does it take? And if marketing doesn't know how it wants to deploy, why are you guessing? How good are you in practice at predicting?

Andy Hunt: It's a multiple choice question. It's not like we have no idea how to deploy, just pick something and make it up. Marketing says they might deploy the system like A, B, or C, but they don't know yet. They aren't going to know until it's far too late in the game, therefore you have to be prepared for any of A, B, or C.

Dave Thomas: In fact, our previous client had exactly this happen to them. They knew they wanted a client-server application, but they weren't too sure how they wanted to present it to the user. For months they went back and forth between using an applet or application on the front end. They did this as they were writing the program. If instead they'd stopped to work out exactly what they were doing up front, they would have delivered the code four months later. Since they didn't know which they we're going to do, they in effect decided to code both an applet and an application. Because of the way Java works, it turned out coding for both added very little overhead.

Andy Hunt: There was basically just a thin wrapper for each of the environments, and from that point backwards everything was the same.

Dave Thomas: They could actually choose between applet and application by changing a couple configuration files.

Adaptable Beyond the Requirements

Bill Venners: You say in your book, "We like to write adaptable, dynamic systems using metadata to allow the character of applications to change at runtime." To what extent are you making things adaptable beyond the requirements?

Andy Hunt: That depends on what you mean by the requirements. If you're talking about the formal written-down requirements document, yeah, we're probably going beyond that. But we're not going beyond what we've gotten from talking with the user and from seeing the environment. You would never go beyond the requirements just on a whim, or just because you thought it was cool. There must obviously be some need. Maybe it's expressed directly by the client, maybe not. Maybe it's implicit in the environment. The client may not be aware that something is going to be a problem directly, but there may be other ways to tell it is going to be a problem. You know from the situation. It would be irresponsible, bordering on malpractice, just to put in extra adaptability where it's not needed.

Dave Thomas: I can give you an indirect example. We had a house built when we came to the states. We talked to the builder about what we wanted. Among other things, we told him we wanted a lot of storage space. The builder came up with some plans that looked wonderful, and started building. As they got to the sheetrock stage, he was walking around and noticed an unused alcove. He realized if he sheetrocked the inside of the framing of the alcove, he could actually form a new cupboard. He took some scrap sheetrocking and put on the inside of this alcove before sheetrocking over the alcove. The next time we saw him he said, "I've done sheetrocking inside, do you want me to cut a door through and make a cupboard out of this? We said, "Absolutely." So we got one extra cupboard.

By sheetrocking the frame at that stage, he saved us the hassle of saying, "Hey we could use that space. Rip off all that sheet rock you just put on the outside so we can sheetrock the inside." He'd gone ahead. It wasn't a stated requirement. It wasn't on the plan. But he knew what we wanted, and based on that he made a decision.

Andy Hunt: That's the key. He knew what they wanted. It's a question of intent, not necessarily a stated requirement. We know they want cupboard space. We know this is a big goal of theirs. We know this is their intent. We are going to try and meet that any way we can.

Introducing Bugs by Making Code Configurable

Bill Venners:You say in your book, "Details mess up our pristine code—especially if they change frequently. Every time we have to go in and change code to accommodate some change in business logic or the law, or in management's personal taste of the day, we run the risk of breaking the system—of introducing a new bug." Is a system never broken by changing a configuration parameter? Are bugs never introduced by making code generic and configurable?

Dave Thomas: Sure they are. But first, we find that if a system is configurable, you typically end up with less code. Because the pieces are plugged together, you get a higher reuse of individual functions. Second, because you've designed it with configuration in mind, you tend to get cleaner designs that by their nature tend not to have as many bugs. The pieces are better decoupled. They have cleaner interfaces. They have stronger functions.

Andy Hunt: And you have fewer assumptions. Right out of the gate as you write the code, you know that you don't know what this value is going to be. You're getting it from the parameter file. The value is not hard-coded staring you in the face, where you might make some assumption like, "I know this will fit in a short," or "I know this is unsigned." You've already consciously admitted doubt. "I don't know what this is going to be." You are less likely to make any sort of assumption about what's coming in from the configuration file. You're more likely to code defensively.

Dave Thomas: But in reality, you get a different class of bug when you make systems configurable. Typically that comes about when a configuration exercises code in a different sequence from the way you anticipated it being used. And every now and then that hits you. The nice thing, though, is it's a real easy bug to find and test, because it's all controlled from an external configuration file. You just feed in a different set of parameters to the configuration, and you expose the bug.

Complexity of Configuration

Bill Venners: Does the complexity of configuration matter? Isn't configuration a kind of user interface? If I have a million configuration parameters, isn't that quite a lot for a user to deal with?

Dave Thomas: You have to be careful to distinguish between deployment and user options. Many Windows applications, for example, have a preferences or options box, which lets you set the color of this and the font of that. That's different from deciding whether I use CORBA or RMI to talk between the front and back end. That's not likely to be in a drop down list box somewhere. So there's developer-oriented deployment configuration, which tends to be more low level. You don't expose that to the user. You don't have to build the fancy user interfaces for the deployment options that you do for the user options.

Building Frameworks

Bill Venners: In your book, you say "You may even be able to implement several different projects using the same app engine, but with different metadata." That makes alarm bells go off in my brain. I get nervous when people say, "Let's go build a framework for our app, instead of building the app."

Dave Thomas: Absolutely, I couldn't agree more. You want to avoid focusing on the framework. The framework supports the application you're writing. You start writing the pieces and realize you need a way to glue them together. You write whatever you need to glue them, but you don't take it any further than that. The mistake people make is saying, "OK, I'm going to write these components. That means I need a framework," and starting with the framework. I can guarantee that any project that starts by writing a framework will never finish writing the framework.

Andy Hunt: When you are working on framework code, if you find yourself saying, "OK, I'm going to add this feature to the framework just in case we might need it," don't. A framework is not the place you want to do something you might need. You want to be much stricter and say, "I need this framework feature now to support my application, therefore I'm going to add it to the framework."

Dave Thomas: Andy and I wrote a debit card switch as our first project together. We wrote it as a very decoupled hungry consumer model. We needed a way of having multiple processes talking back and forth with each other. We initially figured we'd have to write some kind of high falutin framework to do that, but we just ended up using Unix sockets. The whole communication mechanism implementing two way priority queue messages with flow control was probably about 150 lines of code. That very simple and trivial code was all the framework we needed. Had we started off saying, "We're designing this high performance distributed application with lots of processes. We need a framework to support it," that would have been an entire project in its own right.

Andy Hunt: We'd still be writing it.

Dave Thomas : We would still be writing it.

Without the code that uses a framework, you'll never know when to finish the framework, because you'll never know when you've done enough. You'll always be guessing. So start the other way around. Always write the code first, then plumb in the least amount of glue you need to make it all work.

Andy Hunt: Whenever you have to make a decision like that, you want to know, not guess. Dave was just saying, "How would you know when you're done building the framework?" By taking the approach Dave just suggested, you have a way to know exactly what you need. If you did it the other way around, you wouldn't know what you need. You would have to guess. So don't guess, know.

Next Week

Come back Monday, April 7 for Part VI of this conversation with Pragmatic Programmers Andy Hunt and Dave Thomas. If you'd like to receive a brief weekly email announcing new articles at Artima.com, please subscribe to the Artima Newsletter.

Resources

Andy Hunt and Dave Thomas are authors of The Pragmatic Programmer, which is available on Amazon.com at:
http://www.amazon.com/exec/obidos/ASIN/020161622X/

The Pragmatic Programmer's home page is here:
http://www.pragmaticprogrammer.com/

Dave Thomas was not the first person I've interviewed who mentioned the arcade game Whack-a-Mole. James Gosling also called upon the versatile Whack-a-Mole metaphor while pointing out that it is sometimes hard in engineering to know if you've solved a problem or moved it:
http://www.artima.com/intv/gosling34.html

The Agile Manifesto is here:
http://agilemanifesto.org/

Ward's Wiki, the first WikiWikiWeb, created by Ward Cunningham, is here:
http://c2.com/cgi/wiki?WelcomeVisitors

Extreme Programming: A Gentle Introduction:
http://www.extremeprogramming.org/

XProgramming.com: An Extreme Programming Resource:
http://www.xprogramming.com/

Talk back!

Have an opinion? Be the first to post a comment about this article.

About the author

Bill Venners is president of Artima Software, Inc. and editor-in-chief of Artima.com. He is author of the book, Inside the Java Virtual Machine, a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Bill has been active in the Jini Community since its inception. He led the Jini Community's ServiceUI project that produced the ServiceUI API. The ServiceUI became the de facto standard way to associate user interfaces to Jini services, and was the first Jini community standard approved via the Jini Decision Process. Bill also serves as an elected member of the Jini Community's initial Technical Oversight Committee (TOC), and in this role helped to define the governance process for the community. He currently devotes most of his energy to building Artima.com into an ever more useful resource for developers.