Summary
Usage logs can provide useful inputs to user interface and web site designs. But all too often, naive interpretations of log data produce poor (or, at least, unsupported) design decisions. Here are a few of my concerns about the question, "What do the logs say?"

When updating or redesigning user interfaces, beware when people ask "how much is this used?" in design discussions.

Much of the time, the question has an insidious underlying assumption: that current usage of a page, feature, or element is a measure of its usefulness to the user.

I've noticed that when people in design discussions ask how often a page element or feature is used, the question almost always means one of the following:

1. They're looking to defend some preconceived notion. You will notice that they always have an answer if the numbers don't swing their way.

• "It isn't used, so it isn't useful."

  Or maybe:

  • there's a design problem hiding a killer feature.
  • the people who need it aren't getting to this page.
  • it's so poorly implemented that nobody can figure out how to use it, even though it's exactly what they need.

• "It would be useful, but people can't find/don't see it."

  Or maybe:

  • it's a niche feature.
  • it's useless.

• "It's the most used, so it's crucial/the most useful/whatever."

  Or maybe:

  • users click on it because don't see what else they can do.
  • "most used" means "20% of the time", and so biasing a design toward that feature inconveniences the other 80% of users.
  • what they're clicking on is misleading, and doesn't offer what most users expect.

2. They want to use the number to make a decision, regardless of whether the metric means anything. Using logs in this way lets designers punt on the hard questions.

• "We should make this feature prominent because the logs say that's the most useful feature on the page!"

  No, the logs say it's the most heavily-used feature on the page. Not the most useful. Don't confuse the two.

  And maybe:

  • only 3% of the people that use this page ever do anything with it, because the design is broken somehow.
  • the real useful content is hidden or poorly presented.
  • the metric doesn't really measure usefulness, so it's as valid as a coin toss.

• "Only 3% of users click on that: it's useless!"

  Or maybe:

  • those 3% are crucial users.
  • nothing on the page gets more than 3%, so it's as good or better than anything else (though your page may have too much on it).

It seems to me that most useful information you can get from usage logs is contextual. Like usage of one thing relative to something else (though usage does not equal usefulness: you may simply be measuring design artifacts.) Or changes in user behavior after changes in interface (though the potential to mislead yourself there is even more pronounced.) Case in point: a design change that exposes the links in a hidden menu. It's hard to construe a persistent 500% increase in the usage of exposed links relative to hidden links as anything but improvement. That is, until you notice a 40% decrease in overall page usage, because the links obscure something crucial, like a submit button.

My point is that log data are only one input to the design process, and they are data, not information. Interpretations of log data may be information, if your thinking is careful and you're looking diligently for confounding factors, as good scientists do.

Here are my questions:

• How do you truly use activity logs to make informed user interface design decisions?
• Do you have a favorite resource for user activity interpretation?
• Or, conversely, what are your favorite weblog misconceptions and canards?

Talk Back!

Have an opinion? Readers have already posted 2 comments about this weblog entry. Why not add yours?

RSS Feed

If you'd like to be notified whenever Mark Johnson adds a new entry to his weblog, subscribe to his RSS feed.

Digg | del.icio.us | Reddit

About the Blogger

Mark Johnson is a software developer, trainer, writer, and speaker living in Silver Spring, MD. He works at the National Center for Biotechnology Information, writing documentation and tools for open-source bioinformatics software. He is also president of Elucify Technical Communications (www.elucify.com). He has published dozens of articles on Java component technology, enterprise software development, and XML, and is co-author of the book "Designing Enterprise Applications for the J2EE Platform, Second Edition" (Addison-Wesley 2002). He is currently the author of the monthly Enterprise Java Tech Tips newsletter for Sun Microsystems. Mark has been interviewed on CNN, ITworld.com, and JavaWorld.com, and is a member of the National Association of Science Writers.