SEI officials say they are not in the business of controlling what companies say about their assessments. Nor will they reveal to the public which companies have been assessed or what the assessments consisted of. "We weren't chartered to be policemen—we're a research and development group," Hayes says.
Indeed, CIOs who look to CMM for guarantees won't find them, says Rick Harris, director of application development for OnStar, a division of GM that provides communications inside the company's vehicles. He recalls confronting a manager from one of his CMM Level 5 offshore outsourcing companies who did not know how to do a testing plan for software. "My people had to train him to do it," he says. On another occasion, Harris's staff discovered that the offshore provider had fallen far behind schedule in one of its projects but had not told him. "You'd think a Level 5 company would have told me months before that the schedule was slipping and we needed to do something," he says.
"Having a higher maturity level significantly reduces the risk over hiring a [company with a lower level], but it does not guarantee anything," says Jay Douglass, director of business development at the SEI. "You can be a Level 5 organization that produces software that might be garbage."
That assessment is borne out by a recent survey of 89 different software applications by Reasoning, an automated software inspection company, which on average found no difference in the number of code defects in software from companies that identified themselves on one of the CMM levels and those that did not. In fact, the study found that Level 5 companies on average had higher defect rates than anyone else. But Reasoning did see a difference when it sent the code back to the developers for repairs and then tested it again. The second time around, the code from CMM companies improved, while the code from the non-CMM companies showed no improvement.
Previous Topic
