Last week Gilad Bracha, Neal Gafter, James Gosling, and Peter von der Ahé floated a proposal on the internet for adding Closures in Java [PDF]. I was excited to see that adding closures to Java was being considered by such influential people, however, reading through the proposal I was reminded of the difficulties of adding such features to a language as mature as Java.

The recent rise in popularity of Ruby, driven by the success of Ruby on Rails, has perhaps raised the profile of closures in the general development community. Just fifteen days ago, for example, Bruce Tate put closures at the top of his list of 10 Things Java Should Steal from Ruby. Others have lobbied for closures for much longer. Gilad Bracha, for example, tells us in Achieving Closure that he has "personally argued for adding closures since 1997/98." And in The Long Strange Trip to Java, Patrick Naughton, one of the original members of the "green team" at Sun from which Java emerged, described in his personal history of Java a scene in which Bill Joy was insisting that closures be added to Java, back when it was called Oak:

[Bill Joy] was often comparing Oak to more complicated and elegant languages like Python and Beta. He would often go on at length about how great Oak would be if he could only add closures and continuations and parameterized types. While we all agreed these were very cool language features, we were all kind of hoping to finish this language in our lifetimes and get on to creating cool applications with it. The more we argued with Bill about making those changes the more strongly he would fight us. After a while it became a choice between not having Bill involved at all or losing control of the language. James [Gosling] and I got into a rather epic battle with Bill in his office in Aspen one evening about this issue. He started out by insulting both of us about how poorly Oak compared to better languages and then he volunteered to resign from being the Live Oak architect if we wanted him to. James and I agreed that would be best and walked out to go across the street to watch "Speed". What a rush.

The next day, Bill was pretty much his normal old nice guy self again, a little relieved, I think, to be out of the role of being directly responsible for our destiny. Bill is annoyingly smart, and I wouldn't put it past him to have planned that whole scenario the night before to force us to defend our positions and take ownership of our project. The interesting thing is that we were right about needing to finish the language even though it had missing features. It was a timing issue, there was only about a three-month window in which the whole Java phenomenon could have happened. We barely made it. It is also interesting that Bill was absolutely right about what Java needs long term. When I go look at the list of things he wanted to add back then, I want them all. He was right, he usually is.

According to Naughton, that exchange with Bill Joy took place in the summer of 1994, twelve years before the Bracha, et. al., proposal appeared. Is it better late than never? Maybe, but maybe not. As Gilad Bracha put it in Achieving Closure:

Since the late 90s, I've brought the topic [of adding closures to Java] up now and again. At times, even I have reluctantly been convinced that it is too late, because we've done so many things that would have been easy with closures in different ways.

A Natural Law of Language Design

There's a natural law in programming language and API design: as backwards compatibility increases, elegance decreases. Backwards compatibility is very important. There's a cost to breaking code, but there's also a cost to not breaking it—complexity in the developer's face.

For example, Joe Walker mentioned in Closures: Cheers, Library Issues and Solutions that while closures are nice, what would be really nice is if Java Collections could have .each() methods like, for instance, Ruby. This would be tough to do because adding more methods to the Collection interface, for example, would cause all existing implementations to not compile. Instead, the methods would likely end up as static methods in the Collections, which require that you pass in the collection as a parameter. It would work, but is arguably not as elegant.

Moreover, even if that were done, there would now be two idiomatic ways to do iteration in Java collections. The new way using passed closures, and the old way using Iterators. Actually there would also be the old, old way: Enumerations.

For several years I have been jealous of the joy people seem to feel when they program in Python or Ruby. I believe the sense of satisfaction stems primarily from feeling, and actually being, more productive in these languages. Although I have no doubt that my choice to use Java for Artima's architecture was the right one, for me Java was a practical choice, not a heart choice. When I work in Java, I wouldn't say I feel joy. I don't feel like I'm riding a fast stallion through beautiful wooded hills, the wind in my long, flowing hair. I feel more like I'm struggling to convince a workhorse to pull a plow.

The closures proposal seems primarily aimed at improving the Java programmer experience, smoothing out some of the rough edges and verbosity of anonymous inner classes, and I think that is a worthy goal. We should try to find ways to improve the usability of the Java language. But because of the natural law of language design, there will be a usability cost of increased surface area that accompanies the benefit of adding closures. Some things done the old way will become obsolete, perhaps deprecated, and officially ignorable. But they will still take up space in the documentation. They will still be in the developer's face, who has to remember they can be ignored. And they can't always be ignored, because important legacy APIs will still do things the old way. For example, the much-used Servlet API still returns Enumeration, not Iterator.

Breaking Source, But Not Binary Compatibility

I've long wondered how I could get the joy of using a language I love, while still getting all the benefits of the Java ecosystem. Earlier this year, I noticed Scala, and realized that perhaps a path may someday open in front of me that will allow me to gain not just the practical benefits of the Java ecosystem, but more joy of programming. I have not yet found an excuse or time to program in Scala, and I can't say whether I would even like it, much less love it. But reading its documentation made me realize that perhaps the best way to beat the natural law problem is to break source compatibility with the Java language, but not binary compatibility with Java class files. This way I can program in a language that is more elegant, fun, and productive, but still call into any Java API, including all the Java APIs we've written for Artima, use tools that perform static analysis on class files, and run my application on highly optimized JVMs.

Looking at Scala, I see it does seem to do all the things Bill Joy purportedly wanted to see in Java back in 1994, including, notably, closures. What Scala and every other alternate language for the JVM currently lacks, however, is anywhere near tool support that the Java programming language enjoys. The Java language has great refactoring IDEs, static analyzers that look for bugs, code complexity, and code style problems. Some of these tools are available to compiled Scala code, because they work off of class files, and there is a Scala plug-in for Eclipse. Nevertheless, the tool support for the Java language is so much greater, I judge it more appropriate to leverage that than to chase fun and productivity by experimenting with a new language. For this path of using a different language for the JVM to actually become practical, some other language besides Java will need to become mainstream enough to motivate the tool support. (And of course, good tool support would help the language become mainstream: a classic chicken and egg problem.)

I asked Martin Odersky, the creator of Scala, for his opinion on the closures proposal and the difficulties of adding features to a language as mature as Java:

I think [Closures in Java] is a very reasonable proposal, done by people who are the experts in the field. It had to resolve some thorny problems related to fitting in with an existing language which is already quite complicated and which was not designed to accommodate the extension. The proposal manages this well, but at the price of some additional complexity. Besides the syntactic complications, there are also the issues of name resolution (when is something a function, when a value?) which look quite tricky and whose ramifications I have not yet fully understood.

Like you, I think there's a law of diminishing returns at work. As languages evolve, each new addition becomes more complicated, because it has to interact with more conflicting constructs. More complicated extensions cause proportionally more difficulties for future extensions, so it might even be a quadratic increase in complexity. At some point, the additions are no longer worth their price. That's the point when sometimes a new language has a chance of replacing an established one.

I will be programming in Java for the forseeable future, because I have a big investment in the Java code I've written for Artima. I do hope to see the Java language continue to evolve and improve, but I also wish for an alternative that makes me even more productive. If I could use the same IDE to go back and forth between the languages, perform refactorings on both languages at the same time, that would give me a more practical transition path. I could start programming some new things in the new language, while continuing to program the existing things in Java. Over time I would expect I'd be doing more and more in the new language.

Do you believe that another language for the JVM could someday become as mainstream as Java? Do you know of any language that might be capable of doing it?