The Creative Side of Software Design

Often the hardest part about designing software is not figuring out how to organize the software, but figuring out what software to organize. Although you may wish you could simply define requirements and then implement them, significant aspects of the requirements are often discovered or at least better understood as the software itself is designed, implemented, and used. Figuring out what software to build is a creative process, one that feels a lot like exploration.

Exploring Physical Spaces

Last week Bruce Eckel and I gave a Design Patterns seminar in Prague, Czech Republic. The first morning after I arrived, with a little help from jet lag, I set off jogging at 5AM with a desire to explore new places in the neighborhood around my hotel. I had stayed at this communist-era hotel in previous years, and had jogged several times in the neighborhood, but that morning I headed down a new street to see what I might find. An hour before sunrise, the streets were peaceful. The only sounds were birds chirping and my feet slapping the sidewalk. I made my way through streets lined with red-roofed houses, careful to stop and look back at each corner, so I would recognize the turn on my way back. At one point, I happened upon some stairs that wound down into a dark woods.

Here I paused, because if you jog down, you have to eventually jog back up. But I decided to see where these stairs led. With care I began to descend the crumbling steps, which were damp and slippery and covered with matter that had fallen from the trees. After only a short distance, the stairs deposited me onto a slippery cement path. I followed this path as it wound its way down through the forest. After many twists and turns, I emerged from the woods at a small house. I passed under a trellis coated with hanging purple wisteria, and onto a narrow street. As I jogged down this street, I looked up to my left and saw a huge hillside covered with trees and mist. At the top of this hill, the steeple of a church jutted up above the trees. To my right a bit farther away was another steep hillside. I was in a valley that felt far from any city.

I began to hear the sound of water. Following the sound, I soon discovered a fast moving stream. I jogged along a thin path next to the stream and before long found myself at the base of a path heading back up through the woods.

Once again I paused. I wanted to explore this new path, but if I jogged uphill on this path, instead of turning around and going back up the way I had come down, I'd basically be lost. I'd have to find my way back to the hotel by intuition. A friend once told me that whenever he travels to new places, he takes walks and tries to purposely get lost. He said getting lost helps him get a feel for the place and often brings him a bit of adventure. In that spirit, I headed uphill through the woods on this new path. At the top of the hill I found myself once again in a neighborhood, but had no idea where I was. I jogged by intuition for a while, then I saw above the houses the steeple rising in the distance. I used the steeple as a guide, and soon found myself back at a corner I had etched into memory earlier that morning.

Exploring Design Spaces

When I interviewed Python creator Guido van Rossum last year, he mentioned he preferred programming problems where you can't see the solution clearly:

I always feel that the interesting programming jobs are the ones in which you don't know exactly where you'll end up. Implementing another spreadsheet is boring. There have been many spreadsheets. We know what the user interface should be like. We know what the right implementation techniques are. We know a whole bunch of things. Yes, you can write another spreadsheet. You can probably plan it very well, because you can do a feature-by-feature checklist of all the other known spreadsheets. I want this. I want that. I want to fix this problem with this particular one. I want to avoid that bug. That's easy, but it's not very interesting.

I like programming problems where you think, "There has to be something really interesting over there, but I can't see it clearly." All you can do is move one step over there, with a small bit of code, and start exploring to see it more clearly. And maybe it actually wasn't there, it was over here. Or it had a different shape than you thought initially. Maybe it wasn't interesting at all, and you didn't waste a lot of time.

Guido's comments made me reflect on some open source projects in which I'd been involved, and the extent to which exploration has been part of the design process in those projects. All three projects arose out of a feeling I had in 1996 that mobile code was the key to improving the user experience of network services. When I saw applets running in web pages, I thought the architecture seemed inside out. Instead of pasting applet objects inside of a web page documents, shouldn't people just be using network mobile objects, including some objects that represent documents? This feeling is what first attracted me to Jini, because Jini uses mobile proxy objects to deliver network services. But Jini was designed to let software, not users, interact with network services. So in 1999 I started the ServiceUI project with the goal of defining a standard way to attach user interfaces to Jini services.

ServiceUI, Place, and SuiteRunner

The ServiceUI project clearly involved exploring fresh territory. Figuring out how to associate user interfaces with service proxy objects hadn't been done before. It was not clear how to do it. Over the course of nine months in 1999 and 2000, over 40 members of the Jini Community collaborated on the design of the ServiceUI API, primarily via the project's mailing list. Although I released several iterations of ServiceUI API code along the way, the main way we explored this project's space was via discussion on the mailing list. About a year after I started the project, I released the specification and code for ServiceUI API Version 1.0. The ServiceUI API was accepted as a de facto standard in the Jini Community, and was recently approved as the first de jure Jini Community standard via the new Jini Community Decision Process (JDP).

After the ServiceUI API was released, I turned my open-source attention to how users would organize their relationships to network services delivered as objects. The original vision I had was of a network service that worked similarly to the Macintosh Finder. But instead of organizing files in directories, this organizer would organize services into places, where each place was itself a network mobile service represented by a proxy object. I started the Place project to investigate this new territory.

Although the Macintosh Finder certainly already existed, delivering a Finder-like service across a network was a very different problem. The vision for this project was harder to explain than that of the ServiceUI project, and fewer members of the Jini Community participated. But over the course of two years Berco Beute, Jakob Eg Larsen, Frank Sommers and I tried to figure it out. The nature of exploration in this project was primarily in-person discussion at rather infrequent face-to-face meetings. We also built and iteratively refactored a Place API—code that helped us find our way.

The Place project proved to be difficult path. In late 2001, I got to a point where I lost confidence in the API, because our exploration had uncovered as many issues as answers. In addition, I felt the Place API was too complicated, and I couldn't see how to make it any simpler.

About that time I needed to write a conformance test kit for the ServiceUI API, to prepare it for being proposed as a standard via the JDP. This effort spawned my third open-source project, Artima SuiteRunner, a JUnit-like testing toolkit. For the next year I put the Place project on hold and focused instead on Artima SuiteRunner.

The SuiteRunner project was to a great extent an attempt to redesign JUnit, the de facto standard open source testing toolkit for Java. Because of the existence of JUnit, and dozens of JUnit extensions, the design path for Artima SuiteRunner was much clearer than that of the ServiceUI or Place APIs. But even though we could draw insights from the prior work of others, the design of Artima SuiteRunner nevertheless involved a lot of exploration.

For example, one area I wanted to address better than JUnit was distributed systems testing, including Jini and J2EE systems, but I wasn't sure how to do it. I initially released SuiteRunner without solving the distributed systems testing problem. One day I received a zip file in an unsolicited email from Matt Brauer. Matt had been frustrated with testing J2EE applications with Cactus and JUnitEE, so he made some enhancements to Artima SuiteRunner's code to make it more suitable for testing J2EE systems. One change he made was to add an HTTP server to SuiteRunner so it could serve up the class files of the test to other virtual machines on the network. I thought that was a great idea, one which helped me see a path that will make SuiteRunner more suitable for distributed systems testing.

Getting Lost in Prague and Code

Last week, after I had climbed up the hill and found myself in an unfamiliar neighborhood, seeing the steeple in the distance helped me get my bearings. But instead of heading straight back to the hotel, I decided to try and find my way to the church to which the steeple was attached. I tried several streets I suspected would take me to the church, but all ended in dead-ends. Because of the call of breakfast that morning, I ultimately gave up trying to find a path to that church. But because I could see the steeple, I had confidence there must be some way to get to the church that I could find if I kept trying.

Going down blind alleys, hitting dead ends, and backing up are normal parts of the creative process. And so is getting lost. For about a year, for example, the Place project was at an intellectual standstill. We were, in effect, lost. When a writer gets stuck while writing a story, sometimes just putting the story in a drawer and leaving it for a while is enough to get the writer unstuck. Although I had mentally put the Place project on indefinite hold, about six weeks ago during a jog I suddenly saw a new direction. It occurred to me that I could pull out of the Place API the one third of it that I feel confident about, and move that to its own API. So after a year of being "in the drawer," simmering in the background of my mind, I suddenly found a new way forward in the Place project.

One reason software design is hard is because you are usually aiming at a moving target. The very act of designing, building, and using software clarifies and changes its requirements. This movement is greater in projects that venture into new and uncharted territories, but also exists in projects that rework well-known areas. The best way to deal with the motion is to accept that truly understanding your software's requirements will require a thorough exploration of its design space. Think, discuss, experiment, and test; build, release, and listen to users—these methods of exploration will help you understand your requirements and find your way to a good design.

Interview with Guido van Rossum, Programming at Python Speed, in which he talks about exploring with code:
http://www.artima.com/intv/speed4.html

Talk back!

Have an opinion? Readers have already posted 4 comments about this article. Why not add yours?

About the author

Bill Venners is president of Artima Software, Inc. and editor-in-chief of Artima.com. He is author of the book, Inside the Java Virtual Machine, a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Bill has been active in the Jini Community since its inception. He led the Jini Community's ServiceUI project that produced the ServiceUI API. The ServiceUI became the de facto standard way to associate user interfaces to Jini services, and was the first Jini community standard approved via the Jini Decision Process. Bill also serves as an elected member of the Jini Community's initial Technical Oversight Committee (TOC), and in this role helped to define the governance process for the community. He currently devotes most of his energy to building Artima.com into an ever more useful resource for developers.