A Conversation with Neal Gafter

---

Last Thursday, December 14, 2006, Neal Gafter gave a presentation entitled Closures for Java at the JavaPolis conference in Antwerp, Belgium. Neal Gafter is a software engineer at Google, where he works on Google Calendar. Together with Gilad Bracha, James Gosling, and Peter von der Ahé, he has also been working on a proposal to add closures to Java. In his JavaPolis talk, he described the goals of adding closures to Java, compared them to anonymous inner classes, and explained and showed the proposed new syntax. Shortly after his talk, Gafter spoke with Bill Venners about the proposal for adding closures to Java.

Using Closures for Control Statements

Bill Venners: What problems do closures try to solve?

Neal Gafter: What I'm try to accomplish with closures is to allow you to write methods that are part of APIs that act like control statements in the language. Because a lot of APIs do have specialized needs. You have the concurrency framework where you'd really like something that acts like the synchronized block, but for the concurrency locks. We did add a statement in JDK 5, the for-each loop, which is specially designed to work with collections. There are a lot of APIs that would benefit from having statement forms, and there are proposals to add even more statement forms in JDK 7. My hope is that by adding one language construct, we won't have to have a never-ending series of additional statement forms because we keep having more and more APIs that we care about. Instead, we can have one language extension and those APIs can include those statement forms.

Bill Venners: One of the examples you gave us in your talk was withLock. Can you elaborate on that?

Neal Gafter: withLock is a method that I hope would be added to the concurrent framework. It basically looks just like a synchronized statement. You say, withLock, open-paren, then a lock variable, close paren, open curly brace, then a bunch of statements, and then close curly brace. It looks just a like a built-in statement form, but it's a method invocation, an invocation of this withLock method. And the withLock method takes two parameters, the lock, and a closure. It does the sort of boilerplate that you would have to do to synchronize a block of code using the concurrency framework yourself, but it's much more convenient to use.

The withLock example is a poster-child for closures. If you don't have closures, and you're never going to have closures, you really want to put something like that in the language. But, again, it's a never-ending series of language additions.

Bill Venners: What makes a block of code a closure, instead of just a block of code?

Neal Gafter: The syntax I just described, where you have a function declaration that looks like it has a block of code after it, that's not legal syntax today. That is a new syntax in the language, and we would define that syntax to mean that you take the block at the end and wrap it in a closure and pass it as an extra parameter to the method.

There is also syntax where you can write closure literals, where you can explicitly pass two or three of them as parameters to a method, and that's a new syntax too. The way you write a closure is, open curly brace, the argument declarations, an arrow (equals-greater-than), a bunch of statements, and then a closed curly brace—that's a syntax that's not in Java right now. That would be the way you write a closure literal in the language. It would be wrapped by the compiler in an interface type when it's passed to the method to which you pass it in as a parameter.

Bill Venners: In your talk, you highlighted the differences between closures and anonymous inner classes, such as the meaning of this and return. When you have a closure, you can return from...

Neal Gafter: ...the enclosing method. Actually, there are some places where there are no enclosing methods. If you wouldn't be allowed to write a return anyway, then you won't be allowed to write a return inside of a closure. But a return from a closure simply returns from the enclosing method. The fact that it's inside of a closure doesn't change the meaning of the return statement.

Closures Under the Covers

Bill Venners: What happens under the covers? In your talk, you showed the interface that has an invoke method in it. Could you describe what really happens?

Neal Gafter: The closure is turned at compile time into an instance of the interface. If you take the example of the withLock method, that method is declared to take a function type as its last parameter. At compile time, the compiler looks at the closure and says, oh, I actually need something of this interface type. (Function type is an interface type.) So it builds an instance of that interface, just like an anonymous inner class, but it's not an anonymous inner class, because the scoping rules for a closure are different from an anonymous inner class's scoping rules. It creates an object that implements that interface where the body of the invoke method does what the code inside the closure says it should do. And it passes that instance as an extra parameter to the withLock method. And inside the withLock method, in order to cause that code to be executed, you have some parameter, like block—you just say block.invoke(). That's all you need to do, and it simply calls the invoke method on that interface from the object that was passed in.

Bill Venners: If that block executes a return, how does it return from the enclosing method, which called withLock?

Neal Gafter: There are actually a few different answers to that question. The simplest answer is that a return from a closure will be implemented using exceptions. So the compiler will create and throw an exception, and that will pass out through the invocation of withLock to the caller that was calling withLock, where it will be caught. And the value that you're returning will be pulled out of the exception, and the code that catches it will actually do the returning from the enclosing method.

As a practical matter, if you use that pattern, where you put a return inside of a closure, it will get inlined by HotSpot, and it will turn into a jump, which is exactly what it would have done if you used the statement in the language instead of a closure. If the closure only executed once, it won't, but if the code is executed more than a couple of times, and the return is executed very frequently in the closure, then HotSpot will inline that. And it will turn into the same code at the machine code level that it would have, if you had done it via a control statement added to the language.

Using Closures with Existing APIs

Bill Venners: One thing I really liked in your presentation was about methods that exist in existing APIs today that you could pass a closure to, even though they never imagined that would be possible.

Neal Gafter: Let's take a particular example. In the concurrency framework there is an interface called Executor. It represents a thread pool, among other things. One of the things you typically do with a thread pool is you submit a Runnable to it. It takes the Runnable, and assigns it to a thread to run now, or as soon as a thread is available. That method is called execute. If you want to use this method, but you want to use a closure, because of the convenience of the syntax, you can. Remember I said the closure conversion takes the closure and figures out what kind of interface type it needs and that's the kind of interface that's built at compile time to be passed it. So if you say, executor.execute(), and then open-close curly braces, and write a block of code inside the curly braces, that block of code is turned into a closure. It's turned into an object that implements Runnable, and the run method does whatever that block says it should do. That object is passed to the execute method of the executor framework.

No change needs to be made to the executor framework because a closure is always converted to whatever interface type you need statically in the context of the invocation of the code. You can use a closure in place of a Runnable, you can use it in place of a Callable—whatever interface you need—you can write a closure literal in that place and pass it to that API.

Bill Venners: So if make a new API after closures are in the language, and I say explicitly that I take a closure...

Neal Gafter: It's just a parameter, right? All you have to do is declare it as the last parameter of the interface.

Bill Venners: So all I have to do to be ready for a closure being passed to me is that the last parameter is an interface?

Neal Gafter: You can receive a closure in any parameter position. The only advantage of having it in the last parameter position is you get to use the special syntax that looks like a statement. That's the point. If it is the first parameter instead of the last parameter, it's no problem. But you would use the closure-literal syntax with the fat arrow instead of putting a block after the invocation. But you can do that. In fact, if you have an API where you need to pass in more than one closure, you have to use the closure literal syntax.

Bill Venners: Basically any existing method call at the end, if it's an interface... It has to be an interface, can't be a class?

Neal Gafter: It has to be an interface, and an interface with one method in it.

Bill Venners: So that narrows it down a little bit the existing APIs that will be able to be called with closures.

Neal Gafter: It does, but there a lot of those—all of the call-back registration stuff in Swing, the executor framework, and so on. There are Callables, Runnables—a lot of places where you could take advantage of this, even java.util.Comparators that you use for sorting. You can write a block of code that describes the Comparator right in the sort invocation. That works just fine. In many of the APIs where people currently use anonymous instances in the invocation, you can use closures instead—but not all of them. If you have something where you really need more than one method to be in there, you can't easily use closures for that. Actually, you can, but not easily. You have to write a method that makes an instance of a class and then invokes the closures for each one. It's not pretty. It's really mainly suitable for the ones that have a single method.

Exception Transparency

Bill Venners: Tell me about exception transparency.

Neal Gafter: The idea of exception transparency is that sometimes you have an API to which you pass in a closure, and if the closure throws an exception, you want your method to be known to throw the same exception so that exception checking works.

A really good example of that is the withLock that works with the new locks. If the block of code in the withLock statement throws some exceptions, you want the compiler to know that those exceptions are going to be propagated out from the withLock call. In the closure specification, therefore, we've added some stuff to the machinery of generics. It's only a little bit; it's not very complicated. We added just enough to make it possible for you to express that the exception propagates out.

Bill Venners: How does this affect the way checked exceptions work?

Neal Gafter: If the block throws, say, a NumberFormatException, then the compiler is going to complain that it's not caught. But if you can declare that in the enclosing method, then the compiler won't complain. You can't declare the withLock method to throw NumberFormatException. That would be silly, because not everyone who uses withLock has to deal with NumberFormatException. In fact, there is no particular exception that you can declare in withLock. But using this new mechanism, you can say whatever type is thrown by the block is thrown by withLock. If no checked exceptions are thrown by the block, then nothing is thrown by withLock. So you have a type parameter for the exception in the API declaration (such as the withLock method), and you say, "The closure throws this exception X, and so do I." That allows you to get the exception checking for APIs where the implementation will propagate the exceptions out.

The reason you need a special syntax for that, the reason you can't just use an ordinary type parameter, is if you use an ordinary type parameter, that will work as long as there is exactly one exception that's thrown. If there is no exceptions thrown, what are you going to fill in there? And what if there are two exceptions thrown? Then you're just out of luck.

The idea is that if you declare a type parameter with the throws keyword, then you're allowed to put multiple types in there for the exceptions, and the way you list them all out, you can't use commas, those would look like extra type parameters, you use a vertical bar. But you don't normally have to specify those type parameters explicitly. You just pass a closure that happens to throw three things, and the caller has to catch those three things. That just works. But if you do want to explicitly pass a type parameter, which is the three different exceptions that can be thrown by a block, you just list the three in that type parameter position with a vertical bar between them.

Bill Venners: So the compiler will always be able to tell by looking at each individual invocation of a method that takes a closure what the exceptions thrown by the block will be, right?

Neal Gafter: The set of exceptions that are thrown by a closure are part of the type of the closure. The interface to which that's converted must declare those exception types. The closure that's received by invoke must declare that in the throws clause of the method of that interface.

Bill Venners: Can you give an example?

Neal Gafter: For example, withLock is a method. It's second parameter is an interface that contains a single method called invoke. Whatever exceptions the passed block throws must be in the throws clause of that method. And the closure conversion checks that.

Writing the withLock method, you don't know what that's going to throw. That's why you use a generic type parameter. But the compiler, when it's looking at the closure, knows exactly what is thrown. It can see that you're calling Integer.parseInt(), for example, and it knows that it throws NumberFormatException. The compiler knows what exceptions are thrown by the closure, and it matches those and makes sure that the interface that the closure is converted to declares all the exceptions that are actually thrown. If it's a generic, then the compiler knows what a generic is supposed to be, it's supposed to be whatever is actually thrown by the closure. It fills in the type parameter on that basis.

Closures and Functional Programming

Bill Venners: The other thing you showed at the end was some functional programming styles. I had written a blog post about how it becomes increasingly difficult to add features to a language over time because you have to be backwards compatible with more and more existing features. If there is a better way to do it, like iterations, you leave in the old way. So we have an Iterator, Enumeration, and with closures we'd be able to pass a closure to an each method.

Neal Gafter: Yes, but I don't think there is any point in doing it. I don't think there is a point in adding an each method to Collection, or Collections, or anything like that. I don't think it does anything for you. And if you do it, you don't need to put it in the interface. If you want to put a static method in Collections, called each that takes a Collection and a closure, and loops over the Collection and calls the closure on each of them, you can do that just fine, there is no backward compatibility problem.

Bill Venners: So you say it we don't need it because we already have the for-each?

Neal Gafter: I'm saying that there is no problem leaving the for-each in, and as a practical matter I don't think you need it. It doesn't do anything for you that you can't already do.

Now, if you had closures before JDK 5, you probably wouldn't have wanted to add that statement to the language. But I don't think there is any harm in having it there.

Bill Venners: So basically your original justification for adding closures was adding control statements. So instead of adding a bunch of them, you can use closures. The for-each was one of those things. We already have that one, so we don't need to use closures to do that.

Neal Gafter: You don't, but the point is there are lots and lots of APIs in the world that would benefit from closures besides collections. It's not the only API in the world that needs to have API-specific control constructs. It's the only one that got them so far, but it's not the only one that deserves them.

Bill Venners: At the end of your talk, you were discussing some of the functional programming styles that can be done with closures. You talked about map. Can you talk about how useful the functional styles might be? When you add closures, can you program in a different style in Java?

Neal Gafter: There is a style of programming that you see in the functional programming community. Some people like it. I think the main benefit from adopting that style is when you start using that with the concurrency framework that takes advantage of that. For example, just having that map operation might be only useful in some cases, but if you can do it in such a way that you invoke the function concurrently, especially if the function is particularly expensive, that becomes a very concise way of writing something that is otherwise very difficult to do. That is what I was alluding to when I mentioned the fork-join concurrency framework that Doug Lea has waiting in the wings. There is no easy way for clients to use that fork-join concurrency framework because there is no way of expressing the API in Java now that makes it convenient to use. If you have closures, you could have this map interface that takes one List of T and this function that maps from T to U, and returns a List of U. That's mildly useful, but when you add a parameter that's an Executor so that you can do that mapping in parallel, it becomes much more useful because you have a way of very conveniently doing fork-join parallelism. There are some cases where there is a huge benefit to do that kind of thing concurrently, if that function is expensive, for example.

I think the most important uses of stuff from the functional programming community will be in a concurrent setting. I don't really have in my mind all of what that looks like, but I hope to get Doug Lea involved early, before I actually put out a prototype or a compiler that does closures. I want to get Doug Lea involved so I can include some of his fork-join concurrent framework, have him express those using closures, so that other people would get a feel for what those actually would look like in JDK 7.

The Next Steps for the Closures Proposal

Bill Venners: Given you work at Google now, what would be the process for getting this proposal into JDK 7?

Neal Gafter: I'm working on this on my own time. The main reason I'm working on this on my own time is that Google's JCP representative has decided that Google does not support this work. So for that reason, I cannot propose a JSR on Google's behalf. But what I can do is provide a prototype that someone, for example, Sun Microsystems, perhaps, could use a basis for a JSR proposal. As a practical matter, I think Sun wants to roll all language changes that might go into JDK 7 into a single JSR anyway. If that's the case, it doesn't make sense to have me be the spec lead for the language change JSR for JDK 7. It will probably be someone at Sun Microsystems, and closures would be included in that, I hope, as a candidate for JDK 7. Doing the prototype, I hope, increases its chance of being considered, because people will have a chance to try it out and see what it actually buys for them.

Bill Venners: What has the response been from your viewpoint?

Neal Gafter: It's very mixed. People seem to have strong feelings about it, one way or another. I don't think there's really a consensus. There are a lot of people who have experience with languages like Ruby that feel that having closures will increase the joy of programming in Java and make it possible to do things where they currently have to fight with the compiler to coerce it to do what they want. There are some people that think that you can actually do with anonymous inner classes all of the things that you can do with closures. I, frankly, don't believe that. That was really the point of my presentation, trying to make that clear. And there are some people that understand the difference in power, but believe that Java programmers are not up the task of taking advantage of closures without shooting themselves in the foot.

Bill Venners: How can they shoot themselves in the foot?

Neal Gafter: For example, Josh Bloch's position is that if we have closures in the language, then we effectively have an extensible language, and if you have extensible languages, different groups will be programming in different dialects of the language, and that means the programmers will not be able to move from one project to the other and understand what the code does. I think that's true, however, programmers that use different API sets already program in different dialects of the language, and there is already a problem that when people start looking at code using an unfamiliar API, they don't understand the code until they learn the API. And closures are just APIs in that sense.

I don't think closures increase the level of difficulty of understanding code that uses an unfamiliar API at all compared to the situation we're in today. If you use one of the twelve XML APIs, and you use a different one when switching from one project to another, you simply have to learn it. If a project uses certain closure methods that you're unfamiliar with, you need to read the documentation and see what those methods do. If you use a good IDE, you can hover over the method name, and it will point you to the JavaDoc. The JavaDoc pops up on the screen and tells you. That's all there is to it. I think that's better than adding two or three language constructs every two or three years, because with those language constructs, you don't get the popups that tell you what the language construct does. You need to learn them whether you're going to use them or not, because they're pervasive. Whereas with APIs, you do have the JavaDoc to help you.

I think it is really an honest difference of opinion. My hope is that having a prototype out there for people to be able to use, and having people using it, will give some actual evidence about whether or not this is a problem, and to what extent it's a problem, and how much does it really solve people's problems to have closures in the language versus not having closures.

Resources

Closures for the Java Programming Language, a proposal by Gilad Bracha, Neal Gafter, James Gosling, and Peter von der Ahé:
http://www.javac.info/

Closures for Java, a webcast of Neal Gafter's JavaPolis talk that he gave just before this interview:
http://www.bejug.org/confluenceBeJUG/display/PARLEYS/Closures+for+Java

Neal Gafter's blog, in which he discusses the closures proposal:
http://gafter.blogspot.com/

Talk back!

Have an opinion? Readers have already posted 19 comments about this article. Why not add yours?

About the author

Bill Venners is president of Artima Software, Inc. and editor-in-chief of Artima Developer. He is author of the book, Inside the Java Virtual Machine, a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Bill has been active in the Jini Community since its inception. He led the Jini Community's ServiceUI project, whose ServiceUI API became the de facto standard way to associate user interfaces to Jini services. Bill also serves as an elected member of the Jini Community's initial Technical Oversight Committee (TOC), and in this role helped to define the governance process for the community.