I can understand people asking why not JRuby, if not the apparent indignation ;)

But the core problem is async processing and messaging - just porting to JRuby wouldn't get you that much in terms of a system design. So what's more interesting to me over Scala v Ruby, is the choice of Scala Actors for async processing over JMS or RabbitMQ, either of which would be accessible via JRuby. That would be an interesting discussion

> Hmmm, I don't think that really answers the question. It
> answers why they can't just run their old CRuby code on
> JRuby, sure. But that doesn't say why they picked one
> particular JVM language over another. Having decided to
> use the JVM they have to ditch CRuby libraries and use JVM
> libraries instead. But those JVM libraries could be
> consumed from Scala or JRuby or Java or Groovy or Jython
> or Clojure or....
>
Well, it doesn't answer the question of why they picked Scala, instead of any other JVM language. But it highlights that it wasn't a no-op to just port their Ruby app to JRuby, that to some extent it would be as much a "rewrite" to use JRuby as well another JVM language. By the way, this was their Rails app, the twitter.com app, not the message queue that had this problem. The other problem they had with JRuby was performance, and I think that was the main problem with using it for the message queue. I've just got approval on the text change. Here's what they say about JRuby, which I've added to the interview:

Bill Venners: Did you consider JRuby?

Alex Payne: We did. At the time we looked into it, we simply couldn't boot our Rails app on JRuby. Too many of the Ruby Gems we make use of require C extensions, and haven't been ported to JVM-friendly versions. The performance of JRuby was also not even on par with MRI (the C implementation of Ruby), much less a language like Scala. We're open to trying out JRuby again in the future, but we're also hoping that some Ruby patches will help in the meantime.

So performance was also one of the problems with JRuby, which I think helps explain better why they'd prefer Scala over Ruby or JRuby for some things.

I have often heard Rubyists say that although Ruby is slower than Java, for many things it is plenty fast enough, and they are right. The logic goes further, saying that servers are cheap, and programmers expensive, so it makes sense to tradeoff some runtime performance for programmer productivity. And I think that's very often true too, but not always. If you have enough traffic, at some point the cost of servers outweighs the cost of programmers. I'm not sure whether Twitter is past that point, but they get a lot of traffic. And frankly this isn't an intrinsic tradeoff. Other dynamic languages are faster than Ruby, and Scala is too. And people can be quite productive in these other languages too, including Scala.

One thing I've heard in the past is the conclusion that because Twitter had scaling problems, and Twitter uses Ruby on Rails, that therefore Ruby on Rails doesn't scale. Well if that's all you know is those two facts, then you have no evidence at all that Ruby on Rails had anything to do with their scaling problems. As far as I can see, they have Ruby on Rails scaled up pretty well at least now. So I can see how Ruby folks would feel like Twitter may have given them a bit of a black eye, but unless Twitter was publicly blaming Ruby for their scaling problems, then this bit of bad PR isn't Twitter's fault.

Now I read this from the link in this forum:


Twitter: blaming Ruby for their mistakes?

With the Internet abuzz about Google being in talks with Twitter, it seems that Ruby has become the proverbial whipping boy for Twitter's scaling problems. Twitter developer Alex Payne is now preaching Scala is the new Ruby, and panning Ruby for its technical foibles.


From:

http://unlimitednovelty.com/2009/04/twitter-blaming-ruby-for-their-mistakes.html

It certainly didn't come across to me when I interviewed these guys that they are blaming Ruby for all their problems. They had a problem with the performance or Ruby and JRuby in one area, and they solved it by using a different language that's much faster, that's all. They have Ruby running just fine in other areas, and they said so. So I'm not sure what the big deal is. Ruby is a great language in very many ways, but do you have to use it for everything otherwise you're dissing Ruby? That attitude doesn't make sense.

Bill: That helps clear up at least one aspect of the Scala decision. However it still doesn't address the fact that many reasons they gave for leaving "Ruby", and which they preached about as problems with "Ruby", would have been answered by JRuby, regardless of performance. Better GC: check. Built on a 10-year-old VM made for uptime: check. Access to Java libraries: check. And the claimed relative performance of JRuby versus Ruby is confusing as well; either they last tried JRuby a "long time ago" (like over a year), or they were doing something wrong. In either case, we never heard from anyone at Twitter about their JRuby exploration, and we probably could have helped.

Let me be perfectly clear: I don't fault Twitter for choosing Scala for their backend code, especially since they were set on implementing their own messaging system (for better or for worse). Scala is certainly a good choice, and very possibly the best choice. But they can make their case for Scala without making a case against "Ruby" that considers only the C implementation. When a substantial number of their problems could have also been solved by JRuby, it ends up looking like they didn't do their homework. Perhaps that's not true, but I'm not the only one that got that impression.

> performance of JRuby was also not even on par with MRI
> (the C implementation of Ruby), much less a language like
> Scala. We're open to trying out JRuby again in the future,
> but we're also hoping that some Ruby patches will help in
> the meantime.

Thanks Bill. I'd say the question has been answered. For their purposes they needed not only scaling but performance (the two are different, but are often correlated).

> Bill: That helps clear up at least one aspect of the Scala
> decision. However it still doesn't address the fact that
> many reasons they gave for leaving "Ruby", and which they
> preached about as problems with "Ruby", would have been
> answered by JRuby, regardless of performance. Better GC:
> check. Built on a 10-year-old VM made for uptime: check.
> Access to Java libraries: check. And the claimed relative
> performance of JRuby versus Ruby is confusing as well;
> either they last tried JRuby a "long time ago" (like over
> a year), or they were doing something wrong. In either
> case, we never heard from anyone at Twitter about their
> JRuby exploration, and we probably could have helped.
>
> Let me be perfectly clear: I don't fault Twitter for
> choosing Scala for their backend code, especially since
> they were set on implementing their own messaging system
> (for better or for worse). Scala is certainly a good
> choice, and very possibly the best choice. But they can
> make their case for Scala without making a case against
> "Ruby" that considers only the C implementation. When a
> substantial number of their problems could have also been
> solved by JRuby, it ends up looking like they didn't do
> their homework. Perhaps that's not true, but I'm not the
> only one that got that impression.
>
I think it is hard to say because we weren't there, and I also didn't ask about the details of their exploration of JRuby. My sense when I was talking to them was that the answer was simply that they seemed to want static typing, which would disqualify JRuby for them. Maybe they liked the idea of actors too, I don't know. But if that's the case, they wouldn't need to look at JRuby at all, right? So I think it's a leap to conclude they didn't do their homework.

Also, maybe when Alex said "Ruby," he didn't mean to include JRuby. I.e., when he said Ruby is not a good fit for reliable, high performance code, maybe he was talking just about CRuby. But he did say "The Ruby language," so that does seem to encompass JRuby.

>Myself and others I've talked to about this article think Alex makes the mistake that many of us have made in similar circumstances: Attributing success of his rewrite to a new language (or technology) choice, instead of the REAL reason behind his success: The wisdom accumulated by writing a system for the second, third or fourth time and not repeating all of the same mistakes that were made in earlier revisions.

I agree with your point but it's funny you should mention this. I do remember back in the day rails was also promoted using the big rewrite argument:
http://blog.curthibbs.us/articles/2005/03/11/support-for-10x-productivity-increase-with-rails

>Also, maybe when Alex said "Ruby," he didn't mean to include JRuby. I.e., when he said Ruby is not a good fit for reliable, high performance code, maybe he was talking just about CRuby. But he did say "The Ruby language," so that does seem to encompass JRuby.

and

>This is the best writeup I've found about it: http://unlimitednovelty.com/2009/04/twitter-blaming-ruby-for-their-mistakes.html


I think what's causing the confusion is that we judge their decision based on information available at the moment (April 2009).

My understanding is that they made their decision in early 2008 when the tech landscape was much different (both jruby and the available open source message queues have improved *a lot* since)

Hi Obie,

> The issue of typing is a red herring. If you're littering
> kind_of? checks all over your Ruby code then clearly
> you're "doing it wrong".
>
Can you elaborate on this? It does sound a bit like programming in a dynamic language with a static accent. Is the reason that you think this is wrong that by doing this you defeat some of the flexibility of dynamic typing, because after putting in a kind_of? User (sorry I don't know the syntax) that you can only pass a User to that method? If so, why do people put those in there? My guess is that they'd rather fail fast than be flexible. Perhaps the method might have some side effects before it discovers it can't invoke a needed method on the object, and you get a corrupt state resulting. They'd rather fail early before any corruption has a chance to take place than be able to pass something Userlike that isn't a User later. Is that the reason people do it?

Hi Obie,

> This is the best writeup I've found about it:
> http://unlimitednovelty.com/2009/04/twitter-blaming-ruby-fo
> r-their-mistakes.html
>
I think an opportunity may have been missed here. First, there's some inaccuracy in some of these articles. For example, this one:

http://www.technologyreview.com/blog/editors/23282/?nlid=1908

Says:

The company is leaving behind a programming language that has caused it much pain in the past, and instead embracing a new and somewhat obscure language called Scala.

That's not really correct I'd say. They aren't leaving behind Ruby, but still using it, just not for everything they used to use it for. What they are using it for is the Twitter website, and although that may not be the largest portion of Twitter's traffic compared to API traffic, it still must be pretty huge. And what they said in my interview that "Rails works great for doing front-end web development." This is at Twitter. So isn't that a good PR opportunity for Ruby?

Another thing is that I'm confused why it is such a concern to some Ruby folks that Twitter would opt for a different technology at this stage for part of their infrastructure. I thought that was part of the dynamic language story. First, Matz himself explained that the prime focus on Ruby was never performance. It was on making the programmer productive:

http://www.artima.com/intv/tuesday3.html

Matz said:

In my point of view, efficiency and productivity are the same thing, because we focus on programming efficiency, not runtime efficiency. I don't care about runtime efficiency. The computer goes faster and faster every year, so I don't really care about performance. Actually, I care about performance when I'm implementing the Ruby interpreter, but not when I'm designing Ruby. The implementer should care about performance, but the designer shouldn't. If you care about performance, you will focus on machines in design instead of on humans. So you shouldn't focus on performance in design.

And Guido van Rossum, the creator of Python, didn't seem at all bothered by people replacing a Python app with Java or C++ later. In this interview:

http://www.artima.com/intv/speed4.html

He said:

I'm not particularly worried by the fact that people say you can prototype more easily in Python, but eventually the Java version makes it easier to build a robust large system. You can prototype in Python. Once you've explored the space more, you can do the planning and design that the Java version requires. If you start writing in Java knowing as little as you did when you started writing the Python version, you'll waste way more time exploring than actually building the system you'll eventually build.

The Python world has some nice examples of this. Early on, when the Web was just becoming interesting for things like shopping online, a small company called eShop was developing various commerce servers. eShop had proprietary protocols and proprietary applications, and it realized that it should just be able to use a Web server and Web browser. The developers decided to do a prototype in Python. Because they used Python, the eShop developers were the first to release a beta version compared to many other startups doing exactly the same thing. They never actually released working code after their beta release, because the company was acquired by Microsoft. Microsoft spent two or three product revisions to eventually replace all the Python code with C++. But if eShop hadn't started in Python, it would never have released something interesting enough for Microsoft to acquire in the first place.

That also happened with Yahoo. Yahoo Mail started out as a successful Python application. Again, because the developers used Python, they could respond quickly to the user feedback. And that's an application that almost everybody can use. They saw many things wrong with their application, and they responded to that quickly and added new features. Because they were doing something new, they didn't know exactly what people would need from an email Web application. It is different from a program that has your email on your computer. Access times are different. All sorts of things are different. So they were learning about what those differences were. And again, I think Yahoo may now have replaced all the Python code with C++ or some other language, but the Python prototype was essential in order to get there.

So my take on Twitter was they were just doing this. There was a part of their application that they decided to reimplement so it would perform better.

As far as why they didn't do JRuby instead of Scala, I think frankly they wanted the threads, GC, etc., of the Java VM and static typing and a flexible language like Ruby. Well that's not JRuby, because JRuby is statically typed, but Scala fits that perfectly.

So anyway, I think part of the problem is the slant that the reporters took when writing their articles about this. This one seems pretty balanced:

http://www.internetevolution.com/author.asp?section_id=466&doc_id=174573&f_src=internetevolution_sitedefault

And the text of this one seems OK, but the title is a bit sensational:

http://www.theregister.co.uk/2009/04/01/twitter_on_scala/

In the body of the article, they quote Alex Payne saying, "We're still happy with Rails for building user facing features... performance-wise, it's fine for people clicking around web pages." But then the title is "Twitter Jilts Ruby for Scala." The Register tries to be in your face in general, and so making a title like that isn't out of character, but I think it gives the wrong impression. My impression was that Twitter still loves Ruby and especially Rails, they just don't want to use it for everything. I wouldn't call that "jilting" Ruby.

The Unlimited Novelty blog seems to be basically saying, "It's not Ruby's fault, it's the fault of Twitter's engineers." Wouldn't calling out these misrepresentations in the articles, and countering that Twitter's continuing confidence in Rails for their website proves Ruby on Rails to be very scalable be a better response?

Hi Charles,

> Bill: That helps clear up at least one aspect of the Scala
> decision. However it still doesn't address the fact that
> many reasons they gave for leaving "Ruby", and which they
> preached about as problems with "Ruby", would have been
> answered by JRuby, regardless of performance. Better GC:
> check. Built on a 10-year-old VM made for uptime: check.
> Access to Java libraries: check. And the claimed relative
> performance of JRuby versus Ruby is confusing as well;
> either they last tried JRuby a "long time ago" (like over
> a year), or they were doing something wrong. In either
> case, we never heard from anyone at Twitter about their
> JRuby exploration, and we probably could have helped.
>
Can you point us to some stats on the current performance of Ruby and JRuby? I'm wondering if my opinion may be outdated.

One problem I see with some comments here is that they ignore the notion of time and space: Decisions have to be made at a specific time and in a specific context. It's hard to point just one or two things out in these kinds of decisions: rather, the totality of the context must be taken into account.

I also moved away from a Rails app to one using JPA/Java and Flex. When we first built the Rails app, in that time and in that context, Ruby and Rails were the right choice. We could very quickly realize something, get real feedback, etc. Later, a new set of problems made Rails, and even Ruby, a less desirable solution for that problem. I could point out a few issues with Ruby or Rails, but that would miss the point: It was a totality of *all* the factors that made us to switch, not just one or two things.

And some factors are surely psychological. Personally, my problem with the Rails/Ruby solution had to do with refactoring. As I started to better understand the problem domain, I wanted to refactor and clean up many aspects of our Rails app, but I could just never be sure if I missed something. And a few mistakes did slip into production. I didn't have time, or the desire, to write tests for every little method. I figured that there already was technology that I could use to prevent those kinds of errors: the compiler. Why not automate type checking using the compiler? That was an easy benefit to gain - again, in that context and at that time.

That said, if I had to make the decision today, I'd probably pick Scala for the server-side, instead of Java: Mainly because I know Scala a lot better, and also because Scala has matured a lot in the past year.

The whole point is that languages and tools are selected not because of a set of immutable benefits, but because of the totality of some set of (sometimes mutable) benefits in a particular context.

This seems relevant:

http://mail.google.com/mail/?hl=en&tab=wm#inbox/12082144838b9457

Gee guys, calm down a bit. Maybe they didn't use JRuby because they are just not huge Ruby fanboys and given the choice to redevelop and the choice of any language Ruby wasn't their automatic only choice like it would be for you. Not everyone thinks Ruby is gods gift you know!

Another funny take on this discussion:

http://eustaquiorangel.com/posts/twitter_scala_e_ruby