A Conversation with Yukihiro Matsumoto, Part III

Yukihiro Matsumoto, or "Matz," as he is known online, is the creator of the Ruby programming language. Ruby is an object-oriented language suitable for writing day to day scripts as well as full-scale applications. Matz began work on Ruby back in 1993, because he wanted a language that made him productive while being fun to use. Initially popular in Japan, Ruby has been finding its way into the hearts of programmers all over the world.

On September 24, 2003, Bill Venners met with Yukihiro Matsumoto at the JAOO conference in Aarhus, Denmark. In this interview, which is being published in multiple installments on Artima.com, Yukihiro Matsumoto discusses Ruby's design philosopy, the features of the Ruby language, and becoming a better programmer.

• In Part I: The Philosophy of Ruby, Matz waxes philosophic about design imperfection, the danger of orthogonality, granting freedom with guidance, the principle of least surprise, and the importance of the human in computer endeavors.
• In Part II: Dynamic Productivity with Ruby, Matz discusses morphing interfaces, using mix-ins, and the productivity benefits of being concise in Ruby.
• In this third installment, Matz discusses two kinds of nameless functions in Ruby, blocks and closures.

Loop Abstraction with Blocks

Bill Venners: Ruby supports blocks and closures. What are blocks and closures, and how are they used?

Yukihiro Matsumoto: Blocks are basically nameless functions. You may be familiar with the lambda from other languages like Lisp or Python. Basically, you can pass a nameless function to another function, and then that function can invoke the passed-in nameless function. For example, a function could perform iteration by passing one item at a time to the nameless function. This is a common style, called higher order function style, among languages that can handle functions as first class objects. Lisp does it. Python does it .Even C does it with function pointers. Many other languages do this style of programming. In Ruby, the difference is mainly a different kind of syntax for higher order functions. In other languages, you have to specify explicitly that a function can accept another function as an argument. But in Ruby, any method can be called with a block as an implicit argument. Inside the method, you can call the block using the yield keyword with a value.

Bill Venners: What is the benefit of blocks?

Yukihiro Matsumoto: Basically, blocks are designed for loop abstraction. The most basic usage of blocks is to let you define your own way for iterating over the items.

For example, if you have a list, sequence, vector, or array, you can iterate forward by using the method provided by the standard library. But what if you want to iterate backwards from the end to the beginning? In C, you have to set up four things: an index, a start value, an end comparison, and an increment. This is not good, because it reveals internal details of the list. We want to hide that logic. By using blocks, we can hide the loop logic inside the method or function. So for example by calling list.reverse_each , you can do a reverse iteration over the list without knowing how the list is implemented on the inside.

Bill Venners: I just pass in a block that's going to do whatever I want to do with each element, and it's up to the list to know how to go backwards. In other words, I pass as a block whatever code I would have put inside the for loop in C.

Yukihiro Matsumoto: Yes, and that also means you can define many ways to iterate. You could provide a forward way to iterate, a backward way, and so on. It's up to you. C# has an iterator, but it has just one iterator per class. In Ruby you can have an arbitrary number of iterators if you want. If you have a tree class, for example, which you think people will want to traverse depth first and breadth first, you can provide both kinds of traversal by providing two different methods.

Bill Venners: Let me see if I understand this. In Java, they abstract iteration with Iterators. A client can ask a Collection for an Iterator, for example. But the client must use a for loop that runs through and processes the items returned by the Iterator. Inside the for loop, I have "the code" that I want to perform on each item, so that for loop always shows up in the client code. With blocks, I don't call a method to get an Iterator back, I call a method and pass as a block "the code" I want to process each item of the iteration. Is the benefit of the block approach that it takes a little bit of code, the for loop, out of each client?

Yukihiro Matsumoto: The details of how to iterate should belong to the service provider class. The client should know as little as possible. That was the original purpose of blocks. In fact, in early versions of Ruby, the methods called with blocks were referred to as iterators, because they were designed to iterate. But in the history of Ruby, the role of blocks was later enhanced from loop abstraction to anything.

Bill Venners: For example...

Yukihiro Matsumoto: For example, we can create a closure out of a block. A closure is a nameless function the way it is done in Lisp. You can pass around a nameless function object, the closure, to another method to customize the behavior of the method. As another example, if you have a sort method to sort an array or list, you can pass a block to define how to compare the elements. This is not iteration. This is not a loop. But it is using blocks.

Using Closures

Bill Venners: What makes a block a closure?

Yukihiro Matsumoto: A closure object has code to run, the executable, and state around the code, the scope. So you capture the environment, namely the local variables, in the closure. As a result, you can refer to the local variables inside a closure. Even after the function has returned, and its local scope has been destroyed, the local variables remain in existence as part of the closure object. When no one refers to the closure anymore, it's garbage collected, and the local variables go away.

Bill Venners: So the local variables are basically being shared between the closure and the method? If the closure updates the variable, the method sees it. And if the method updates the variable, the closure sees it.

Yukihiro Matsumoto: Yes, local variables are shared between the closure and the method. It's a real closure. It's not just a copy.

Bill Venners: What is the benefit of a real closure? Once I make a block into an object, what can I do with it?

Yukihiro Matsumoto: You can reconvert a closure back into a block, so a closure can be used anywhere a block can be used. Often, closures are used to store the status of a block into an instance variable, because once you convert a block into a closure, it is an object that can by referenced by a variable. And of course closures can be used like they are used in other languages, such as passing around the object to customize behavior of methods. If you want to pass some code to customize a method, you can of course just pass a block. But if you want to pass the same code to more than two methods -- this is a very rare case, but if you really want to do that -- you can convert the block into a closure, and pass that same closure object to multiple methods.

Bill Venners: OK, but what is the benefit of having the context? The distinction that makes Ruby's closure a real closure is that it captures the context, the local variables and so on. What benefit do I get from having the context in addition to the code that I don't get by just being able to pass a chunk of code around as an object?

Yukihiro Matsumoto: Actually, to tell the truth, the first reason is to respect the history of Lisp. Lisp provided real closures, and I wanted to follow that.

Bill Venners: One difference I can see is that data is actually shared between the closure objects and the method. I imagine I could always pass any needed context data into a regular, non-closure, block as parameters, but then the block would just have a copy of the context, not the real thing. It's not sharing the context. Sharing is what's going on in a closure that's different from a plain old function object.

Yukihiro Matsumoto: Yes, and that sharing allows you to do some interesting code demos, but I think it's not that useful in the daily lives of programmers. It doesn't matter that much. The plain copy, like it's done in Java's inner classes for example, works in most cases. But in Ruby closures, I wanted to respect the Lisp culture.

Next Week

Come back Monday, December 29 for part IV of this conversation with Yukihiro Matsumoto. If you'd like to receive a brief weekly email announcing new articles at Artima.com, please subscribe to the Artima Newsletter.

Ruby in a Nutshell, by Yukihiro Matsumoto, is available on Amazon.com at:
http://www.amazon.com/exec/obidos/ASIN/020161622X/

Programming Ruby: A Pragmatic Programmer's Guide, by Dave Thomas and Andy Hunt, is available on Amazon.com at:
http://www.amazon.com/exec/obidos/ASIN/020161622X/

The Ruby Programming Language, an introduction by Yukihiro Matsumoto:
http://www.informit.com/content/index.asp?product_id=%7BA76D1D1E-AD7D-483E-AB8D-38FB188396C5%7D

An Interview with the Creator of Ruby, by Bruce Stewart:
http://linux.oreillynet.com/pub/a/linux/2001/11/29/ruby.html

Interview with Ruby Create Y. Matsumoto, by S. Ibaraki:
http://www.cips.ca/news/national/news.asp?aID=1224

Talk back!

Have an opinion? Readers have already posted 14 comments about this article. Why not add yours?

About the author

Bill Venners is president of Artima Software, Inc. and editor-in-chief of Artima.com. He is author of the book, Inside the Java Virtual Machine, a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Bill has been active in the Jini Community since its inception. He led the Jini Community's ServiceUI project that produced the ServiceUI API. The ServiceUI became the de facto standard way to associate user interfaces to Jini services, and was the first Jini community standard approved via the Jini Decision Process. Bill also serves as an elected member of the Jini Community's initial Technical Oversight Committee (TOC), and in this role helped to define the governance process for the community. He currently devotes most of his energy to building Artima.com into an ever more useful resource for developers.