This post originated from an RSS feed registered with Agile Buzz by Kevin Rutherford. Original Post: process improvement metrics - some questions Feed Title: silk and spinach Feed URL: http://silkandspinach.net/blog/index.xml Feed Description: kevin rutherford on agile software development and life in macclesfield Latest Agile Buzz Posts Latest Agile Buzz Posts by Kevin Rutherford Latest Posts From silk and spinach

Software process improvement is all very well, but how do we know whether it is working?

The topic of measuring the effectiveness of process improvement activities has crossed my path several times in recent days. For example my current client and I are trying to define ways to measure the benefits to their organisation of retaining me. And then there was the designing metrics session at SPA2006; in their handout notes, Jason and Duncan suggest that process improvement be run as an agile project, with its own stories, velocity etc. Not to mention a conversation I had with Deb Hartmann on the same topic; seems she wants to measure - or at least chart - her coaching activities.

I've found almost nothing written about this topic, so at present I'm afraid I have only questions. First though, some basic assumptions. I assume that software process improvement in itself is pointless. As I said earlier this week, the term "process" doesn't capture enough of the behaviour of a system to be useful. And of course "improvement" is a relative term. Improvement compared to what? And towards which goal(s)? According to Goldratt there's only one Goal: maximising profits in both the present and the future. I will assume, then, that software development is always carried out with that goal in mind: the software either reduces our own organisation's costs or increases our sales, or is designed to be sold to other organisations so that they can use it for one of those ends. Either way, the process of producing that software is only a fraction of the whole system, and may contribute a large or small amount to how well we're working towards our organisation's Goal. Furthermore, improving the software development department in isolation may prove beneficial initially, but could be creating a local optimum at the expense of another part of the system, or could simply not be worth doing, if other parts of the system are contributing more to poor overall performance.

All of that notwithstanding, there will be times when an organisation must turn its attention to improving the software development department. Software development will inevitably be the system's primary bottleneck some of the time, and so its contribution to the goal will need to be addressed. But depending on context the "improvement" required may involve improving quality, reducing time to market, improving customer service, ensuring the right software is developed, training people, or any number of other things. That is, the day-to-day dynamics of the overall system's push towards the Goal will create short- and medium-term demands on software development; and the response from software development must be measured in terms of those demands.

Henceforward, I'm going to use the woolly term effectiveness as a shorthand for the software department's contribution to the overall system Goal. Improving the department's effectiveness then equates to reducing costs and/or increasing sales in the overall system. And instead of talking about "software process improvement", I'll discuss "software department effectiveness" instead.

So, back to the top. How can we measure the performance of a software coach? As yet I have no answer; instead I'll ask a few more related questions:

1. If we hire a coach or process facilitator or scrum master, how will we know whether we're getting value for money? Unless the coach doesn't charge, the organisation's Operating Expense has increased, so should we be looking for a corresponding increase in sales?
   
2. Should a software coach charge a percentage of their client's increase in profits? Now and in the future?
   
3. How often is it possible to attribute improved profitability to the actions of a software coach?
   
4. What if the service constitutes training - how can that be directly accounted to profitability?
   
5. Given that increased profitability may be measuable only after a period of time has elapsed, how should the risk be apportioned (between client and coach) during that interval?
   
6. Conversely, how risky is it to define local measures purely on the software department?
   
7. Imagine an ineffective organisation whose biggest problem is not within the software development sub-system. But suppose nevertheless that the organisation blindly hires an agile coach to improve the software department's effectiveness. Is it meaningful to talk about "improvements" when the real problem is elsewhere? Can the changes be measured in any meaningful way?
   
8. Assume that one of the objectives of a coach is to transfer sufficient knowledge so that the client organisation can ultimately continue alone. How can we chart progress towards that goal?