Articles Forum - Twitter on Scala

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Articles Forum
Twitter on Scala

28 replies on 2 pages. Most recent reply: Apr 9, 2009 12:13 PM by Bill Venners

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


		Next Topic

Flat View: This topic has 28 replies on 2 pages [ 1 2 | » ]

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Twitter on Scala

Posted: Apr 3, 2009 10:00 AM

Steve Jenson, Alex Payne, and Robey Pointer, three developers at Twitter, talk with Bill Venners about their use of Scala for production daemon servers at Twitter. Read the interview here:

http://www.artima.com/scalazine/articles/twitter_on_scala.html

What do you think of Twitter's take on Scala and Ruby?

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Twitter on Scala

Posted: Apr 3, 2009 11:17 AM

"As our system has grown, a lot of the logic in our Ruby system sort of replicates a type system, either in our unit tests or as validations on models. I think it may just be a property of large systems in dynamic languages, that eventually you end up rewriting your own type system, and you sort of do it badly. You’re checking for null values all over the place. There’s lots of calls to Ruby’s kind_of? method, which asks, “Is this a kind of User object? Because that’s what we’re expecting. If we don’t get that, this is going to explode.” It is a shame to have to write all that when there is a solution that has existed in the world of programming languages for decades now."

I'm getting my popcorn ready.

Charles Oliver Nutter

Posts: 3
Nickname: headius
Registered: Apr, 2004

Re: Twitter on Scala

Posted: Apr 3, 2009 12:40 PM

I find it remarkable that there's not a single mention of JRuby anywhere in this entire interview. JRuby brings at least some of the JVM characteristics they praise (better GC, built for long-lived processes, native threading) but remains Ruby. Plus JRuby has much-improved performance over the standard implementation. The JRuby team were never approached by anyone from Twitter about helping with their scaling issues. It seems like it would have at least warranted some exploration, and as far as I know, that never happened.

I don't claim that JRuby could have solved all their scaling issues, nor do I expect JRuby will ever perform as well as Scala. But they talk about Ruby and the JVM as if they're mutually exclusive. That's obviously not true.

David Koontz

Posts: 3
Nickname: dkoontz
Registered: May, 2008

Re: Twitter on Scala

Posted: Apr 3, 2009 12:46 PM

I have to agree with Charlie here, JRuby has everything except parity with Java's speed (for now) that Scala boasts. I'm not saying Scala was the wrong choice, just that no mention of JRuby seems odd considering they had a large infrastructure in Ruby, trained Ruby developers, etc. Was JRuby ever considered, and if it was, for what reasons was it rejected?

johny boyd

Posts: 28
Nickname: johnyboyd
Registered: Apr, 2007

Re: Twitter on Scala

Posted: Apr 3, 2009 12:54 PM

Charles, looks like static typing and the functional nature of Scala were key things as well. Obviously JRuby wouldn't have made a difference there ...

Good interview, Bill.

I'd have liked to ask about the steepness (if any) of the 'learning curve', and whether they think Scala is palatable to the 'average' programmer ...

-jb

johny boyd

Posts: 28
Nickname: johnyboyd
Registered: Apr, 2007

Re: Twitter on Scala

Posted: Apr 3, 2009 12:59 PM

btw, I'd also state that any programmer, programming team or architect deciding to write their key product's high-performance messaging system in Ruby (or any scripting language) is NOT doing their job right.

Choose the right tool, folks.

-jb

Charles Oliver Nutter

Posts: 3
Nickname: headius
Registered: Apr, 2004

Re: Twitter on Scala

Posted: Apr 3, 2009 1:18 PM

johny boyd: I can certainly accept that. I was mostly surprised that JRuby was never even considered, nor discussed, nor even mentioned in the entire interview. You *can* have Ruby + JVM characteristics + improved performance, despite the implication in this article that you can't. Maybe it's not as fast, static, or functional as Scala, but it's a damn good choice if you still want to use Ruby.

Joao Pedrosa

Posts: 114
Nickname: dewd
Registered: Dec, 2005

Re: Twitter on Scala

Posted: Apr 3, 2009 1:28 PM

Twitter still has some startup atmosphere to it even though it has probably grown 10 fold at least since it started. Count the use of Scala due to it still being this bigger startup, if you will. :-)

Nice reminder on the lack of mention of JRuby, although they could have said that when they started such experiments they did not expect JRuby to be that evolved yet or something.

JRuby enjoys a lot of what the Java infrastructure has to offer. It sometimes feels like driving a SUV, powerful, big, pleasurable, 4x4... :-) That same feeling some Scala users experiment. Java has a lot to offer to all kinds of languages based on it. I am looking forward to the C# on the JVM some day, why not? :-)

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Twitter on Scala

Posted: Apr 3, 2009 4:13 PM

> I find it remarkable that there's not a single mention of
> JRuby anywhere in this entire interview. JRuby brings at
> least some of the JVM characteristics they praise (better
> GC, built for long-lived processes, native threading) but
> remains Ruby. Plus JRuby has much-improved performance
> over the standard implementation. The JRuby team were
> never approached by anyone from Twitter about helping with
> their scaling issues. It seems like it would have at least
> warranted some exploration, and as far as I know, that
> never happened.
>
> I don't claim that JRuby could have solved all their
> scaling issues, nor do I expect JRuby will ever perform as
> well as Scala. But they talk about Ruby and the JVM as if
> they're mutually exclusive. That's obviously not true.
>
Yes, you're absolutely right. That question was on the tip of my tongue when I asked about the Ruby problems. I.e., I asked "Can you elaborate on what you felt the Ruby language lacked in the area of reliable, high performance code?" The reason JRuby popped into my head is that a lot of what they wanted to solve was solved by switching from the Ruby runtime to the JVM, not the Ruby language to the Scala language. And JRuby would have given them all that as well. But then I felt they kind of answered it. They said they missed static typing. So I moved on and didn't ask about JRuby. But it is a good question I should have asked anyway so we could hear their answer. I'll email them now and see if they can respond to that question here in the forum.

James Iry

Posts: 85
Nickname: jiry
Registered: Nov, 2007

Re: Twitter on Scala

Posted: Apr 4, 2009 8:20 AM

http://twitterapi.quebecblogue.com/2009/04/03/twitter-on-scala/

"Twitter is a well-muscled growing website that provides a micro-blogging ritual. It began its animation as a Ruby on Rails utilization, and alleviate uses Ruby on Rails
to the push most user-facing entanglement pages."

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Twitter on Scala

Posted: Apr 4, 2009 8:50 AM

> http://twitterapi.quebecblogue.com/2009/04/03/twitter-on-sc
> ala/
>
> "Twitter is a well-muscled growing website that provides a
> micro-blogging ritual. It began its animation as a Ruby on
> Rails utilization, and alleviate uses Ruby on Rails
> to the push most user-facing entanglement pages."
>
Hmm. I wonder how they got ahold of the original, unedited interview transcript.

Obie Fernandez

Posts: 608
Nickname: obie
Registered: Aug, 2005

Re: Twitter on Scala

Posted: Apr 4, 2009 11:27 AM

Myself and others I've talked to about this article think Alex makes the mistake that many of us have made in similar circumstances: Attributing success of his rewrite to a new language (or technology) choice, instead of the REAL reason behind his success: The wisdom accumulated by writing a system for the second, third or fourth time and not repeating all of the same mistakes that were made in earlier revisions.

The issue of typing is a red herring. If you're littering kind_of? checks all over your Ruby code then clearly you're "doing it wrong".

I also find it disappointing that JRuby was not mentioned as a viable alternative and speculate that it might have been a more mature choice than Scala.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Twitter on Scala

Posted: Apr 4, 2009 11:56 AM

Hi Obie

> I also find it disappointing that JRuby was not mentioned
> as a viable alternative and speculate that it might have
> been a more mature choice than Scala.
>
We're finalizing a bit of text that we're going to add to the interview about JRuby. It was an oversight on my part to not do this initially. You'll see the official answer once we get the final form of the text approved by the Twitter guys, but the gist is that they did indeed give JRuby a try, and the main problem they had that disqualified it was that their Ruby app used a lot of libraries with C extensions that weren't available for the JVM. So it was not a simple matter of taking their Ruby app and running it on the JVM with JRuby. When they tried that, it didn't run.

James Iry

Posts: 85
Nickname: jiry
Registered: Nov, 2007

Re: Twitter on Scala

Posted: Apr 4, 2009 12:51 PM

Hmmm, I don't think that really answers the question. It answers why they can't just run their old CRuby code on JRuby, sure. But that doesn't say why they picked one particular JVM language over another. Having decided to use the JVM they have to ditch CRuby libraries and use JVM libraries instead. But those JVM libraries could be consumed from Scala or JRuby or Java or Groovy or Jython or Clojure or....

Obie Fernandez

Posts: 608
Nickname: obie
Registered: Aug, 2005

Re: Twitter on Scala

Posted: Apr 4, 2009 12:56 PM

Sigh

This is the best writeup I've found about it: http://unlimitednovelty.com/2009/04/twitter-blaming-ruby-for-their-mistakes.html

Flat View: This topic has 28 replies on 2 pages [ 1 2 | » ]

Next Topic