Bill Venners writes, "One reason software design is hard is because you are usually aiming at a moving target. The very act of designing, building, and using software clarifies and changes its requirements."

Read this Artima.com article:

http://www.artima.com/objectdesign/getlost.html

Here's an excerpt:

Going down blind alleys, hitting dead ends, and backing up are normal parts of the creative process. And so is getting lost. For about a year, for example, the Place project was at an intellectual standstill. We were, in effect, lost. When a writer gets stuck while writing a story, sometimes just putting the story in a drawer and leaving it for a while is enough to get the writer unstuck. Although I had mentally put the Place project on indefinite hold, about six weeks ago during a jog I suddenly saw a new direction. It occurred to me that I could pull out of the Place API the one third of it that I feel confident about, and move that to its own API. So after a year of being "in the drawer," simmering in the background of my mind, I suddenly found a new way forward in the Place project.

How do you explore design spaces?

One thing the process of designing the Place API taught me is that it is hard to figure out whether you're on the right track towards an optimal solution. It's like a neural network exploring the solution space; there is a chance that it ends up in a sub-optimal solution. Something you won't realise untill you're standing on the top of the solution hill and can see the hills around you. That doesn't have to be a problem, as long as the hill is high enough for its purpose.

The other thing the process taught me is that designing an API can be an exiting and fun exploratory social activity. Another reason why I think software engineering is, and should be, a human factors thing. Besides the advantage that it's much more fun, teamwork increases the chances of realising you're running up the wrong hill.

Design is hard if there is no guiding vision, the art of design is managing that vision. The mark of an effective designer/architect is to develop a vision and to sell it to the client. Then, the process revolves around management of that vision and setting expectations. So long as you and the client share the vision, effective decisions, both strategic and tactical, can be made. If the vision becomes corrupted, then there is no opportunity for strategy, and tactical implementation becomes a gruel.

I liked the writing style of your article, which reminded me of the Bitter java series books...

> Then, the process revolves around
> management of that vision and setting expectations. So
> long as you and the client share the vision, effective
> decisions, both strategic and tactical, can be made.

In projects with small companies and clients that
know little about the coding process let alone
the need for good structure and effective design
managing the expectations becomes the primary tool
for hiding the time and resources you know you
will need to employ.
Hit the milestones or miss them within exceptable
margins and buy the time to explore the design space.
You'll be doing it alone because no one else
there will have a clue what you're doing.
Sub-optimal designs are necessary in getting something
produced to move the project on.
One of the most frustrating aspects of an obsession
with excellent code and good design is seeing the
next step as a flash of inspiration and having to
wait ages before you have the chance to implement it.
Often the change has to be hidden in the next requirement.