Summary
While I'm all for healthy competition, the open source world might be taking it a bit too far.
Advertisement
The Commoditization of the Platform
At the end of a gruelling week at OSCON, the O'Reilly Open Source Conference in Portland, Oregon, I considered the state of the open source world. One of the advantages of having your company run a conference is that you get to give the keynote speech, which is how come I (and 1500 others) got to hear Tim O'Reilly speak on "The Open Source Paradigm Shift".
O'Reilly is an intelligent and insightful guy, so it was a pleasure to hear his talk. I won't attempt to summarise it -- you can read it in his own weblog (apparently, if I can believe what I overheard on the conference floor, prefaced by remarks to the effect that "you should read this if you want to know what I said rather than what other people's blogs say I said").
From my point of view the most interesting thing was the way he took me back to 1985, when I was a Sun Microsystems Senior Technical Support Specialist and the brave talk in the company was about how we would "bring Unix to the desktop". Right. What actually happened, of course, was that Apollo and Digital (where are they now?) and H-P and Sun and IBM all spent years beating the hell out of each other for dominance of the corporate Unix desktop market, only to wake up and find that a young upstart called Microsoft, more focused on their goal and with fewer competitors trying to exploit their chosen (and by then non-proprietary) hardware platform, had taken control of the desktop while the Unix vendors were still licking the wounds they'd given each other.
The rest, as they say, is history. I have said many uncomplimentary things about Microsoft, but I have never been able to criticize Bill Gates' business instincts. He and his management team saw the opportunity that the disarray among the competition represented and exploited it ruthlessly. The Common Desktop Environment, the Unix world's eventual attempt to arrive at an X Window-based platform that would allow application interoperability, was always going to be too little and too late.
Open Source Diversity
Why are these memories significant at this particular moment? Because Tim O'Reilly very lucidly explained how the open source world was commoditizing the software market to the extent that the new killer apps are Amazon.com, Google, PayPal and the like - web services. By buying or reviewing a book on Google you actually modify the way the application will respond to future inputs, event though the code is determinedly proprietary. So the the operating system and the database and the programming languages are now commodities, and the applications that process the user data represent the new proprietary battleground.
The seminal moment came the next evening, when I was entertained to dinner by my publisher, Sams, with a bunch of my fellow authors. Halfway through the evening an author across the table from me said "I hate X", where X was (if I remember correctly: publishers can be generous with their wine on these occasions) an open source programming language. I berated him for wasting emotional energy - as sensible to hate the color green, or cheese, neither of which will change no matter how vehemently it is vilified. Next day, bells started ringing.
I was reminded especially of the industry response to Sun's introduction of NeWS, the Network extensible Window System. A fine piece of technology based on the PostScript graphical model, using real-world units rather than pixels to describe the shapes to be rendered (and therefore well-to displays of varying resolutions), with the ability to deliver compact interpretable descriptions of the required graphics to the desktop rendering engine rather than bulky bitmaps (and therefore more efficient in its use of network bandwidth,then a scarce resource).
In short, something so clearly fitted to the tasks for which it had been designed that all the competing workstation vendors immediately banded together to support X Window, thus ensuring the death of a beautiful concept. It is not my intention to bemoan the fact that superior marketing is much more important to business success than superior technology. Even I am not such an idealist. But there is a lesson here, relevant to today's situation. If Sun had chosen to abandon NeWS earlier, or if the competition had adopted it rather than competing, there would have been less chance for Microsoft to succeed against them all.
Too Much of a Good Thing?
The open source world is presumably interested in proving the superiority of the open source development model, as contrasted to the proprietary techniques adopted by more traditionally commercial vendors. Yet the Perl programmer sees as "the opposition" not Visual Basic, but Python, or Ruby, or Ocaml. The Debian user "hates" Red Hat, or Suse, or Mandrake rather than Windows. MySQL users compete not against Oracle or Sybase users but against PostgreSQL, MSQL and SQLlite.
Competition is a good thing, but huge amounts of talent and energy are being wasted duplicating in one language functionality that is already available in one or two or three others, simply because of the almost religious fervor with which the devotees of the various open source languages, tools and platforms seek to prove the superiority of their own particular chosen technologies. This is madness, and it's certainly not the way to win mindshare among the uninformed.
My own opinion, for what it's worth, is that we should seek to improve the interoperability of open source systems and components. If a given piece of functionality is available in, say, Perl then it should be easier for a Python or Ruby programmer to use the Perl module than it is to rewrite it (with a few "minor" tweaks and incompatibilities) in the favored language. We should focus on the goal of proving that open is better than proprietary, and deriving the benefits of avoiding such wasteful duplications.
To this end I see both Parrot and Mono as encouraging signs, even despite technical misgivings, in that they seek to provide and support interoperability rather than emphasise the superiority of one technology over others. I particularly like the way that PostgreSQL is so determinedly language-agnostic, having the ability to support components programmed in Perl, Python, Ruby and several other languages.
There's a lot further to go , however, in getting the open source world to wake up and smell the coffee. I couldn't say it any better than the quote from O'Reilly on the front cover of the conference program: "One of the greatest challenges for open source in the next few years is to understand and adapt to the paradign shift implicit in network computing, and to shed the legacy thinking of the desktop era". Interestingly, he was talking about technological developments, but the concept is just as relevantly applied to human behavior.
While I can see the importance of cooperation, I think you are thinking too much about strategy, and open source/free software communities aren't organized strategically.
Good programmers have strong opinions. If you care about what you do, you'll have opinions. We create software that we care about -- strategy is fine for articles, but when you're sitting at the keyboard writing code (or more difficult -- maintaining that code), strategy isn't enough.
Maybe open source can be a commercial success, and people won't have to like what they're doing. Instead of individuals cooperating, corporations will cooperate, and individuals will put in their time doing what their managers tell them to do. But that's not proven -- individuals (some in professional functions, some not) are absolutely what this movement is built upon, and increased commercialization hasn't changed that.
Cooperation sounds nice, but it's not just politics that make it difficult. Digging through other people's code isn't easy. Generalized foreign-function interfaces are unwieldy. The slightly different semantics of different languages and environments will bite you. Ignoring this may be the expedient solution, but we don't to things because they are expedient, we do them because we enjoy them. Many of us create expedient software in our professional jobs, but we want to write *good* software. It doesn't pay off right away, it often doesn't pay off at all when our ambitions extend past our ability, time, or knowledge, but ultimately our success in creating good software is what got us here.
So I'd say strategy is fine and well, but at the periphery of real work. Nobody here is sabotaging anyone else, cutting funny deals, conspiring against competitors. We're not like the old school Unices. We just repeat ourselves a lot, we don't contradict. It's the programming we do that provides the foundation for everything else, and it's the only essential part of this equation. It's gotten us this far.
An interesting article. In many ways it reinforces my belief that one of Open Source's main effects in any market area is to undermine all commercial alternatives to the market leader (thus reinforcing the strength of said market leader: Thus, on the desktop, we now have Microsoft v. OS unix variants. Even Apple has been forced to drop it's own OS and fall into line with one or other camp. For databases: Oracle (and SQL Server) are now unassailable. Languages: Java rules, all other languages being basically of geek or acedemic interest.
There are several alternatives to all of the above, but as you observed, the alternatives have first to fight for survival amongst themselves. They then start picking of the smaller, weaker commercial alternatives. Finally, any attempt to go for the dominant commercial products appears to be mired by a lack of the required stratiegic co-ordination and the constant need to protect themselves from the other open source alternatives biting at their heels.
Good article. The KDE/GNOME rivalry is another example of this sort of thing. Competition may provide extra stimulus to the two teams, but the duplication of effort is huge. And worse, it Balkanizes the application developers, who must chose one or the other API for the apps. Instant 50% reduction in use base!
Ted Lewis's excellent articles on tech competition showed that it continues until one product gets around 70% market share. Then it becomes the defacto standard and competitors fade into irrelevancy. (Examaples: Microsoft Office vs Word Perfect or Ethernet vs Token Ring). One way to reach dominance is through mergers -- but its hard to see open source projects merging.
The situation with dynamic languages such as Ruby, Perl, and Python is a bit different. They do suffer from the NotInventedHere problem. But there is also a strong tendency to use them as integration platforms, providing wrappers to underlying software written in C. In this way they encourage sharing of open-source software, as Steve would like.
> While I can see the importance of cooperation, I think you > are thinking too much about strategy, and open source/free > software communities aren't organized strategically. > That's certainly true, and I wouldn't try to argue that we should change our way of organization. However, I still think that the analogy is strong enough to be a warning to the open source community that things could get too fragmented.
> Good programmers have strong opinions. If you care about > what you do, you'll have opinions. We create software > that we care about -- strategy is fine for articles, but > when you're sitting at the keyboard writing code (or more > difficult -- maintaining that code), strategy isn't > enough. > But surely you can't argue that you don't need strategy at all? One of the problems of open source (IMHO, I am not what you might call an open source guru) is that there are too many indians and almost no chiefs, if you'll excuse the politically incorrect metaphor.
Of course, because open source developments tend to be anarchic it's difficult or impossible to direct them in a strategic way. So the post was really an attempt to influence individuals in the direction of greater collaboration rather than to indicate a grand strategy for the whole community.
> Maybe open source can be a commercial success, and people > won't have to like what they're doing. Instead of > individuals cooperating, corporations will cooperate, and > individuals will put in their time doing what their > managers tell them to do. But that's not proven -- > individuals (some in professional functions, some not) are > absolutely what this movement is built upon, and increased > commercialization hasn't changed that. > People should always like what they do; I think maybe I'm missing the point of that last paragraph. There's room for open source corporations, but clearly the groundswell is individual contributions. That's one of the reasons why so many SourceForge projects get stuck betrween version 0.5 and 0.95 - sometimes it's easier to move forward with experienced collaborators, but it isn't easy to find them in open source, since each individual is pursuing their own interests.
> Cooperation sounds nice, but it's not just politics that > make it difficult. Digging through other people's code > isn't easy. Generalized foreign-function interfaces are > unwieldy. The slightly different semantics of different > languages and environments will bite you. Ignoring this > may be the expedient solution, but we don't to things > because they are expedient, we do them because we enjoy > them. Many of us create expedient software in our > professional jobs, but we want to write *good* software. > It doesn't pay off right away, it often doesn't pay off > at all when our ambitions extend past our ability, time, > or knowledge, but ultimately our success in creating good > software is what got us here. > I won't disagree. Having got here, however, is no guarantee that we'll get "there" - wherever we decide "there" may be. I'm just saying that I suspect the open source community may need to start acting a little more strategically in the future. I'm trying not to either prescribe or proscribe!
> So I'd say strategy is fine and well, but at the periphery > of real work. Nobody here is sabotaging anyone else, > cutting funny deals, conspiring against competitors. > We're not like the old school Unices. We just repeat > ourselves a lot, we don't contradict. It's the > programming we do that provides the foundation for > everything else, and it's the only essential part of this > equation. It's gotten us this far.
Indeed it has, but the very word "us" implies there is a "them", and the evidence suggests that "they" are clearly scared that in some way "we" are about to start eating "their" lunch (which, as it happens, we literally did at OSCON ;-). So it isn't the lack of infighting among the open source community itself, which I do think you unduly minimize, that should concern us. We should be circling the wagons to defend ourselves against "them". Having got this far is no guarantee of further progress, and we need to look out for the critical points where paradigm shifts outdate old ways of behaving.
I personally have not been heavily involved in the Open Source community and am still mostly at the edge looking in. Even so, I thought that perspective might be useful here.
It seems to me, in order for Open Source to be successful, we must first understand why it IS successful. I think this is important, not just for historical purposes, but because I believe attempts to cut against the grain, so to speak, will in fact cut away at its success.
From my view, it appears Open Source is pretty much an accidental success. There never was and really still is not any real intent to succeed. Open Source seems to have started from two fundamental points. The first was Richard Stallman who became irate at the fact a vendor not only refused to fix a problem with their software, but also refused to allow him to fix it himself. The second was Linus Torvalds who seems to have just had nothing better to do and decided he wanted to learn more about computer operating systems.
A lot of ideologies, religions, business models and such have since grown up around Open Source, but the core is still based on these two things: Fix it or get out of my way and How does that work anyway. Without these two driving factors, Open Source does not exist.
Why is this relevant to the future success of Open Source? Simply because cutting across these two fundamental parts of the community is to undermine the very success you seek. As pointed out earlier as well as in many other forums, people work in Open Source because they like it and/or because it is an opportunity to learn. By trying to force, cajole or even suggest people come together you very quickly start to anger and ostracize those upon whom success depends.
You suggest in your article folks should stop re-inventing the wheel in their language of choice, and yet I say, this is one of the only two reasons Open Source exists. So I see much of this duplication not so much a Not Made Here attitude as I see it in an educational respect. People are doing it over not because it is needed, but because they want to know how it works. Its not elitist; its not even selfish. Its personal growth.
Lets face it Linux itself is neither new nor exciting. Linux is just another wheel. Linux exists because a group of people was more interested in learning how wheels were made, than they were in actually using wheels. Because they did not take a commercial interest in their efforts and gave the code away, it dove tailed nicely with the second reason for Open Source; the fix it yourself camp.
Linux was essentially created as a place to learn. Because enough people had a similar interest, it became something usable outside academia and thus the ultimate power tool for the computer do-it-your-selfer. In other words, its a tutorial on steroids.
In your reply to ianb you ask But surely you can't argue that you don't need strategy at all? I would in fact reply, Yes, there is no need for strategy. The reason for this is because I think a universal Open Source strategy is in fact contradictory to the success of Open Source. Open Source is a way for people to learn and share that learning. Sometimes those lessons are hard and yes, just like school, every year those same lessons are repeated for a new batch of students.
This is basically why any analogies to commercial software fall short of reality and neither explains the success of Open Source nor provides a reasonable road map for the future. Open Source succeeds in the same way academia does. It succeeds by constantly pumping out new graduates. Its curriculum improves more in a critical mass way than through any directed plan. The more graduates go through, the more some of those graduates will give back. Over time, things get better.
Just as academia does not fail due to students redoing the same work, or working in competing research, neither will Open Source fail. The main reason commercial ventures fail under these conditions is because they do not do the one thing academia and Open Source do; share. Open Source is the one arena you can actually rob Peter to pay Paul, and still have everyone come out richer for the deal. Commercial ventures cannot do this. Their funds are limited and the end product must pay it back.
Likewise, I think the fear of excessive fragmentation is also needless, mainly for the same reasons. Commercial entities suffer from excessive fragmentation because even in death, they try to keep their secrets. In Open Source, good ideas percolate to the top. No matter how elitist you may be, you are not going to let a good idea go by. And if you do, others will not. Excessive fragmentation in an open community is only a temporary thing. People divide initially on their differences then naturally regroup based on their similarities. However, the new merger is an improvement over either initial camp. The trick is to let this happen on its own rather than trying to mandate it.
Does this mean Open Source, as an entity is incapable of being a commercial success? Yes, I think so. Does this mean commercial ventures will be unable to leverage Open Source and be successful? No, and this is where it gets interesting. The problem is, commercial ventures can ride, but never tame the beast and thats what scares them. People talk about Open Source changing the way we do business and yet no one really seems to understand what that means. It does not mean commercial vendors need to get in the business of herding cats. It means they actually have to change the way they do business! The old rules do not apply anymore. Trying to use them will only spell disaster for those involved.
People need to understand there are two distinct groups here. There is the Open Source community and then there are commercial entities seeking to leverage the output of the Open Source community. Success for one does not mean, nor guarantee success for the other. In fact, success for each cannot even be measured in the same way.
In this sense, the friends of Open Source who are trying to standardize, conventionalize, mediate and regulate the Open Source community for its own good might in fact be more dangerous than blatant competitors. Or rather, those who would be competitors, because I am reminded of an article I wish I could find again. It was entitled The Tao of Competing with Microsoft. Its message was simple; the only way to compete with Microsoft is to NOT compete with them.
In the end, this is why Open Source is and will remain successful. Not because it is superior to Microsoft, but because it simply does not care. Its superiority lies in the fact none of the participants are measuring themselves against Microsoft, but rather, their own, much higher standards. To change this focus, to tame this beast, is to kill it. Those who would become successful leveraging Open Source need to devise a business strategy that fits with Open Source. They cannot succeed by trying it the other way; making Open Source fit their business strategy.
I realize much of this is idealized and simplified, but I think it still describes the core of Open Source success. Everything else can be measured in how it deviates from this norm and thus be rated a success [or failure] because, or in spite of its association with the Open Source core ideals. Open Source, if left to its own devices will always be successful. Whether or not commercial entities can leverage this and ALSO be successful, well, that is another question altogether.
Previous Topic
