Summary
Sounds a bit extreme? Some companies seem to live and breathe benchmarks, responses to studies, and responses to responses. Isn’t it time we realised what benchmarks are really about and stopped it all getting out of hand.

Firstly, I’ll point out here and now that this blog is based on this thread on TSS about Websphere Vs .Net.

This is the first thing that always gets on my nerves - the term ‘Vs.’ when in the context of benchmarking. Why is it there? It’s not some fight at Caesar’s Palace

Anyway, in all of the opinions and flame bait, there was one golden, gleaming treasure of a comment - ‘Every good programmer knows that the only benchmark that holds water is his own.’. We all have to have our own benchmarks, the one that says, “Yeah I’m happy with choosing X over Y”, or whatever, but it wasn’t always like this; we didn’t have to be always self-reliant when comparing products, did we?

When I first started out on this journey into programming, years ago, benchmarks actually meant something; they may have been small in scope but there were plenty of parties involved and they were fair enough comparisons. But nowadays, benchmarks are considered in these huge ‘studies’ that compare only two or three products. And the scope of these studies is often vast, concerning products with huge number of features and configuration options.And most of the objectivity and rational critique gets lost in the sheer size of the thing.

Now, every other week we have someone publish a benchmark about something, and it all turns into a war, people quoting benchmarks like they are sticks to hit each other with. You can always tell when an analyst company does it’s job, and it’s benchmarks are good – people accept the findings, and get on with making their products better – that is most people, bar the vocal zealot few.

One thing I realised is that everyone selectively uses benchmarks to prove their own point of view. Benchmarks are after all statistics, and, you know, you can prove anything with statistics. Oh .Net is better than J2EE and it’s more productive, so why are so many Java community tools being ported to .Net? You see what I did there – it’s called the apples-to-oranges effect. Produce a claim and defend your position with a counter-claim that has nothing to do with the claim you’re disputing. Now if you ask me do I know Java is better than .Net? Overall, I don’t know. There are areas that are better for Java as I’m sure there are for .Net. But this comes back to a scoping issue; trying to say is J2EE better than .Net is such a big question, it’s like saying is day better than night?

I stopped even contemplating Java and .Net benchmarks and studies, the day I, by chance, picked up a copy of MicroMart and I actually got an unbiased, critique of a product; the same morning I had been reading the first TMC comparison for J2EE and .NET. The product was only a PC, and did I have to pay for the study, yes it was about £1.80.

If you’re going to provide studies and benchmarks, make them much more focused and relevant, and broaden the participation. So instead of looking at J2EE Vs. .Net, lets look at Java RMI/Jini ERI as opposed .Net Remoting; JavaMail Vs. Exchange; JMS Vs. MSMQ, etc, etc.. Make them relevant, not this vendor paid-for marketing gimmick. Benchmarks are for techies, marketing is for Business Decision Makers - let’s not try to turn one into the other

Talk Back!

Have an opinion? Be the first to post a comment about this weblog entry.

RSS Feed

If you'd like to be notified whenever Calum Shaw-Mackay adds a new entry to his weblog, subscribe to his RSS feed.

Digg | del.icio.us | Reddit

About the Blogger

Calum Shaw-Mackay is an architect on Java and Jini systems, working in the UK. His interests lie in distributed computing, adaptability, and abstraction. Calum has been using Jini for longer than he would care to mention. His main area for taking the blame (some people would call it 'expertise') is systems integration and distributed frameworks, and is an advocate of using Jini's unique strengths to build adaptable enterprise systems. His opinions are his own. He's tried to get other people to take his opinions off him, but they just won't.