Comparison of Nine Ruby Implementations

An anonymous reader writes "Zen and the Art of Programming published a new version of The Great Ruby Shootout, which was aimed at testing the performances of multiple Ruby implementations. On the benchmark table this time around are Ruby 1.8 (on GNU/Linux and Windows), Ruby 1.9 (aka Yarv), Ruby Enterprise Edition (aka REE), JRuby 1.1.6RC1, Rubinius, MagLev, MacRuby 0.3 and IronRuby. The results of this comprehensive comparison show that for this set of benchmarks, Ruby 1.9.1 is almost 5 times faster than the notoriously slow Ruby 1.8. Is Ruby finally going to be acceptably fast?"

Comments

Ruby 1.9.1 and JRuby

[mgkimsal2]

Ruby 1.9.1 and JRuby seem to be the 'winners' in most tests. Have a look at the PNG here: http://antoniocangiano.com/images/shootout3/main_time.png

I've heard a lot of good things about the JRuby project, and this test seems to demonstrate that it's probably the closest in terms of speed to the standard Ruby (canonical Ruby?)

Re:Ruby 1.9.1 and JRuby

[Foofoobar]

Well JRuby is an implementation of Ruby in Java which explains the speed but it is still quite slow as it has to 'interpret' everything into java. If you are a fan of Java and like Ruby, I'd suggest Groovy. It's blazingly fast and even puts these to shame. Plus works well Rails, Struts and most MVC frameworks.

Re:Ruby 1.9.1 and JRuby

[mgkimsal2]

See my signature - I'm involved in the Groovy world (via GroovyMag) already. :) I do try to keep tabs on what's going on in other JVM languages, especially JRuby.

Re:Ruby 1.9.1 and JRuby

[Foofoobar]

Um... I suffer from a rare disease known as sig blindness. :)

Re:Ruby 1.9.1 and JRuby

[goombah99]

groovy seems like the ideal language for me. Or at least that's my superficial impression. But it seems like it can seemlessly gain speed by implementing the slow bits into JAVA with very little pain. Or am I mistaken.

I can't actually make any use of groovy however till it catches up with scipy and mayavi.mlab in python. When is that going to happen?

Ultimately what I want is an interpreted language that can be compiled. So for example, in python one rarely actually uses the introspective ability to modify ones self, or even takes advantage of duck typing. instead one usually calls functions with the same type arguments and so forth. So if one just had the ability to switch off the dynamic typing and self-modifying capabilities so that one could compile it it sure would be one sweet language.

I'm wondering if groovy had the same faults or if it's easier to compile it.

Re:Ruby 1.9.1 and JRuby

[Foofoobar]

Well I just played with it a little myself too and it is GREAT for frontend development. You could use it for ALL development if you want but it gives you ('the developer') the option of mixing 'pure java' on the backend with Groovy if you wish (since Groovy compiles to Java anyway from what I remember).

Very flexible, and very powerful. None of that worry of not scaling that there is with Ruby.

Re:Ruby 1.9.1 and JRuby

[K. S. Kyosuke]

"Ultimately what I want is an interpreted language that can be compiled. So for example, in python one rarely actually uses the introspective ability to modify ones self, or even takes advantage of duck typing. instead one usually calls functions with the same type arguments and so forth. So if one just had the ability to switch off the dynamic typing and self-modifying capabilities so that one could compile it it sure would be one sweet language."

I felt a great disturbance in the Force, as if millions of Common Lisp hackers suddenly cried out in terror and were suddenly silenced...

Re:Ruby 1.9.1 and JRuby

[DragonWriter]

Well JRuby is an implementation of Ruby in Java which explains the speed but it is still quite slow as it has to 'interpret' everything into java.

JRuby can do either JIT or ahead-of-time compilation to JVM bytecode, it does not 'interpret' (or even 'compile') everything to Java.

If you are a fan of Java and like Ruby, I'd suggest Groovy. It's blazingly fast and even puts these to shame. Plus works well Rails, Struts and most MVC frameworks.

Well, Rails doesn't run in Groovy, though Groovy has a Rails-inspired framework ("Grails") of its own.

Re:Ruby 1.9.1 and JRuby

[johanatan]

Haskell too!

Re:Ruby 1.9.1 and JRuby

[Sciryl Llort]

# Oh ruby ruby ruby ruby rubeeeey
    I think you're groovy groovy groovy groovy grooveeeey /#

Re:Ruby 1.9.1 and JRuby

[chthon]

Use Common Lisp, CLISP and SBCL can both compile to native executables, while having the best methods available for interactive testing.

Re:Ruby 1.9.1 and JRuby

[rogerdpack]

Seems that grails isn't "tons" faster than rails--see http://docs.codehaus.org/display/GRAILS/Grails+vs+Rails+Benchmark etc. -=R

No Parrot?

[Ed Avis]

Is there a reason why they did not test Cardinal, the Ruby implementation for Parrot? I know it is not production-ready yet but it would be interesting to see how performance compares.

Re:No Parrot?

[Adam Jorgensen]

That seems to be the case with a lot of Parrot projects...

Re:No Parrot?

[facebook12]

We can use Ruby in Facebook application.

Re:Troll Frist Psot

[sveard]

That's the worst cockney accent I've ever heard in my life.

JRuby is the clear winner

[0xABADC0DA]

Look at how many 0.002 and similar instantaneous results there are. Tests were run 5 times and the best result used, so there probably wasn't even a single garbage collect that happened in many of the tests. You can't really say from such short benchmarks, but those tests probably did not include Java's strength of hotspot optimization over time.

There were a couple tests that were much slower in JRuby, but this is countered by the test that failed on Ruby 1.9.1; fast means nothing when you don't finish.

Re:JRuby is the clear winner

[JesseMcDonald]

There were a couple tests that were much slower in JRuby, but this is countered by the test that failed on Ruby 1.9.1; fast means nothing when you don't finish.

Ruby 1.9.1 failed two tests; JRuby failed three.

None of the VMs managed to complete all the tests. Other than Rubinius, which timed out frequently, Ruby 1.9.1 had the fewest errors.

Also, the input sizes for many of the tests numbered in the millions. If that's not enough repetition to take advantage of hotspot optimization, then I'd say they were right to exclude it.

Re:JRuby is the clear winner

[0xABADC0DA]

Ruby 1.9.1 failed two tests; JRuby failed three.

I stand corrected. I blame it on the results being a png so no "highlight all" search results ;-P

Also, the input sizes for many of the tests numbered in the millions. If that's not enough repetition to take advantage of hotspot optimization, then I'd say they were right to exclude it.

Hotspot tracks how much time was spent in the function among other things. With millions of iterations it may be compiled once, but not recompiled using deeper inlining or frequent-type information. In any case, a benchmark that fast is pretty meaningless for any number of reasons.

Re:JRuby is the clear winner

[ESqVIP]

There were a couple tests that were much slower in JRuby, but this is countered by the test that failed on Ruby 1.9.1; fast means nothing when you don't finish.

You should note, though, that one of the failures of YARV was because of code incompatibility (assuming String#[] returns an Integer, which changed in Ruby 1.9). This is an actual worry, because you have to port your code to 1.9 (unlike JRuby, which should be very close to a drop-in solution for 1.8). The other was excessive recursion, something pretty unlikely to find in a "real-world" app.

The JRuby errors were all implementation failures. Pretty minor ones which might be already fixed by now, but still actual mistakes on the JRuby part.

Re:JRuby is the clear winner

[0xABADC0DA]

Let me follow up to answer several replies. Yes, official Ruby 1.9.1 was overall moderately faster than JRuby. JRuby still is the clear winner:

1) JVM has a garbage collector known to work on multi gb sized heap and JVM has a parallel collector that can use multiple processors. So for large programs (programs that actually matter) one would expect JVM to perform better than these microbenchmarks in respect to Ruby 1.9.1.

2) JVM adapts over time to increase the performance of the most-used parts. The benchmarks probably did not run long enough to get optimized as well as they might in an actual program needing to run fast.

3) JVM version wasn't mentioned, so it was probably 1.6, but if it wasn't openjdk then it wasn't the fastest. 1.7 is coming RSN.

4) JVM is getting improvements for dynamic languages in 1.7 with bytecodes for dynamic method calls, escape analysis to use stack for objects, etc.

So in these microbenchmark tests 1.9.1 is slightly faster. The reality for actual programs is unknown but probably closer still. And JRuby should get faster automatically through improvements to JVM, even aside from improvements to JRuby itself.

Platform-based Ruby

[tcopeland]

I heard Rich Kilmer talk about the various Ruby implementations once; his take on them were that each would be used to leverage the underlying platform. In other words, if you want to use Java libraries, you'll use JRuby. If you want to use the Mac libraries (e.g., via Hot Cocoa), you'll use MacRuby. And if you want to use C extensions like RMagick and libxml2, you'll use MRI.

I thought that was an interesting way of looking at the various implementations... each one would be appropriate for a different scenario.

Re:Platform-based Ruby

[tcopeland]

> a different version of a language for whatever platform I want.

Hm, the language would be the same, but the libraries that you would have available or leveraging would change.

Re:Platform-based Ruby

[Surt]

You would write code the same in any version (using the Ruby language). But you would run in a different container depending on your platform. This is no different from using a different compiler for C, or a different runtime for Perl. The availability of different libraries on different platforms is universal for basically all languages.

Re:Platform-based Ruby

[moderatorrater]

The availability of different libraries on different platforms is universal for basically all languages.

Of course there will be, because the platform has different underlying features. The problem this presents is that you're adding another layer on top of it. That statement wasn't like saying that a certain perl program will only run on windows (which would still be considered bad practice btw), but on top of that you'd have to have the right version of the three available interpreters for windows to run it. What happens when someone distributes one program that requires jruby and someone else distributes a program that requires the canonical interpreter? The point of being cross platform is to try to make it so that the code will run on as many platforms as it can; Rich Kilmer wants to fragment it even more. It seems kind of counter productive.

Re:Platform-based Ruby

[Surt]

But seriously, it's the same issue no matter where you go. You can write ruby that will run anywhere, and you can write ruby that will run only on a specific interpreter. You can write c that will link anywhere and you can write c that will link only within a specific compiler.

Encouraging people not to fragment is hopeless, an unfragmentable language would be useless ... there are just too many useful libraries out there.

Is Ruby finally going to be acceptably fast?

[bill_kress]

> Is Ruby finally going to be acceptably fast?

Ruby has always been acceptably fast for most values of acceptably.

In cases where it has not been acceptable, it probably will continue not to be since those are cases where better performance is always welcomed (large volume web services).

Re: Is Ruby finally going to be acceptably fast?

[nettdata]

I guess it depends on the web services.

A lot of LinkedIn stuff is in Ruby... over a billion page views a month and growing.

Their Bumper Sticker app alone has huge numbers.

Interesting read here: http://blog.linkedin.com/2008/06/23/web-scalability-practices-bumper-sticker-on-rails/

Re: Is Ruby finally going to be acceptably fast?

[bill_kress]

If it's always been in ruby, then it's always been fast enough, and remains so-just like I said. Not that a little bump wouldn't help, but sometimes I wonder why people sound like they are disagreeing when they are actually supporting.

or did I read your post wrong?

Re:Troll Frist Psot

[Anonymous Coward]

You should buy a screen reader with cockney support then, mate. Personally, I use the CockneyCobblers3000 Deluxe, it even has support for rhyming slang:

I'll get the jam jar round and we'll get daaaann the rub-a-dub, for a few Uri Gellas.

Comes out, read like this, as:

I will drive the car around to the front entrance. You shall get in, and we will go to the pub. Once there we will purchase and drink at least two pints of Stella Artois lager.

acceptably fast?

[jbeyer05]

What does acceptably fast really mean? If I can write code 5x as quickly relative to Java or C, Ruby will always be fast enough for me. Humans are expensive, CPU cycles are cheap.

Re:acceptably fast?

[Bert64]

CPU cycles, power, memory, all adds up... And once the load exceeds a single box, you have to get into clustering and load balancing.

How many times is your code going to run? Slower code wastes time each and every time it is executed, so the time you saved writing it could end up being lost a thousand times over as you wait for the code to finish it's task.

In this case, acceptably fast is "as fast or faster, using the same or less memory" than other competing options.

My biggest gripe is when people take a perfectly working app tho, and rewrite it in whatever language is fashionable at the time, which seems to be ruby right now... You end up with new code that hasnt been fully debugged yet, which runs at a fraction of the speed of the original, and consumes far more resources.

Ruby gets faster while Python gets slower?

[Aloisius]

It is a bit annoying that Python 3.0 is about 10% slower than 2.6.

I've heard some sensationalist claims that Ruby 1.9.1 is now significantly faster than Python, but of course they always turn out to be some simple test like tail recursion. At the same time, I've seen some awful performance numbers from things like Rails compared to Django.

Anyone have a rough idea of the performance comparison between the two and what bottlenecks people are likely to hit on either side? I know the GIL in Python is sometimes blamed for a lot of problems...

Re:Ruby gets faster while Python gets slower?

[DragonWriter]

Anyone have a rough idea of the performance comparison between the two and what bottlenecks people are likely to hit on either side? I know the GIL in Python is sometimes blamed for a lot of problems...

Ruby 1.9.1 trades green threads for native threads with a GIL, so probably has similar issues to Python (the plan is to eliminate the GIL eventually.) JRuby, I think, uses Java threads (which, on most platforms, IIRC, are native threads) without a GIL.

Re:Ruby gets faster while Python gets slower?

[setagllib]

It's true that JRuby has no overall GIL, but it definitely has plenty of internal fine-grained locks to maintain consistency and determinism. However thanks to Java 6 optimisations, almost all of those locks are completely elided while running a single-threaded program. That's just another way JRuby benefits from JVM technology, and yet another thing a C implementation cannot reasonably implement.

YARV to LLVM

[chrysalis]

I have great hopes in the YARV to LLVM compiler (yarv2llvm), even if you can't read japanese, have a look at the numbers: http://d.hatena.ne.jp/miura1729/20081012/1223785541

Re:PHP is still faster

[resonantblue]

"high level" enough is a relative term. For me, PHP is not enough unless you put it together with one of the MVC frameworks like CakePHP or Symfony. In any case, there are a ton of people (including myself) who like Ruby better for a variety of reasons and we don't have problems with it being too slow (though faster speed is always a welcome improvement). Point is: if you want to use PHP, do it. I don't really care to "persuade" you otherwise. But it's always annoying to see such comparison on a thread like this 'cause this thread has nothing to do with PHP and no one really cares if you like it better :-)

What is it with ruby?

[Bert64]

I don't get all this fascination with ruby, and why it currently seems to be so fashionable that people are willing to take perfectly good projects and rewrite them in this new fashionable language...

Take for example metasploit (see metasploit.com), version 2.x was written in perl and was reasonably quick, version 3 is a complete rewrite in ruby and it's now orders of magnitude slower and more memory hungry than the perl version ever was... Trying to script the CLI based version is now pretty much useless because of how long it takes to initialize, even considering that the hardware in use is a couple of years newer and much faster.

Inefficiency like this is a bad thing, people complain about microsoft bloat, and yet they champion a language which produces code a fraction of the speed of any alternatives... They talk about how 1.9 is 5x faster than 1.8, and is still ridiculously slow compared to anything else?
You think Vista is bloated and slow? imagine if they wrote it in ruby...

Re:What is it with ruby?

[Lord Ender]

It's data-oriented--everything is an object. Your code is just objects and flow control.

Compare that to perl, which is an ugly mess of unix idioms nailed together in a hurry (but polished for years). Even Python requires you to memorize a list of functions to discover the properties of datatypes, rather than just having accessor methods to the data itself (foobar.length, not len(foobar)).

Data-oriented, the way the real world is.

Re:What is it with ruby?

[SoupIsGoodFood_42]

It's not fashion, it's more the work culture of the people who use it. Look at a few Ruby projects and you're more likely to see a project where interface design is taken a bit more seriously as well as following web standards etc. And please, don't give examples to the contrary, as when it come to this sort of thing, it's hardly a certainty and more of a noticeable correlation.

Re:What is it with ruby?

[k8to]

because len(foo) and foo.len() are soooooooooooo different.

They both make use of a common protocol. Only the purist need care.

Re:What is it with ruby?

[nschubach]

I've been reading up on it and use as a quick solution finder. I have already used it for things like file manipulation/creation using regex and simple I/O. Sure, I could code up something in another language, but it's so simple and quick in Ruby.

Re:What is it with ruby?

[Lord Ender]

Wrong. No matter what object I'm working with, I can find its length with the .length property if such property is meaningfull. "len()" on the other hand, only works for certain types of objects. You need to memorize different tricks to find lengths of different data types in non-fully-OO languages.

Having a consistent interface to your data really is a wonderful thing. It means no hacks, no lists of utility functions; things just work as you would expect.

Re:What is it with ruby?

[RAMMS EIN]

I love Ruby, and I completely agree with you. Just because a newly discovered programming language is good, doesn't mean it's a good idea to go and rewrite your perfectly good code in it. It doesn't mean it's a Bad Thing, either, but, if the results are as you state (which I find easy to believe - ruby 1.8 _is_ dog slow), it's hard to argue that it is an improvement.

There is such a thing as the right tool for the job, and Ruby is not (with current implementations) the right tool for things that are CPU intensive and need to be fast. Unless you can isolate the parts that need to be fast and write them as extensions, which is easy to do and gives you the benefits of Ruby for most of your project, while still getting efficient code where it matters.

Re:What is it with ruby?

[RAMMS EIN]

``Having a consistent interface to your data really is a wonderful thing. It means no hacks, no lists of utility functions; things just work as you would expect.''

This is really one of Ruby's greatest strenghts, and why I like it so much. It really feels like it was _designed_, rather than cobbled together.

On the other hand, the object.method notation isn't great for consistency. Invariably, you are going to get methods that don't rightly belong with one of the objects they affect. And then you have to wonder...was it "foo.method bar", "bar.method foo", "method bar, foo", or "SomeClass.method foo, bar"? I'm in favor of "function args" notation everywhere, with an optional namespace prefix so you can have multiple functions of the same name. The way Common Lisp has it.

Re:What is it with ruby?

[tbj61898]

Yep.. it's like an electric car!

Expensive and slow... but the idea behind it is well ahead among others.

Re:What is it with ruby?

[poopdeville]

Wrong. No matter what object I'm working with, I can find its length with the .length property if such property is meaningfull. "len()" on the other hand, only works for certain types of objects.

(I don't do Python, but I know my type theory) What exactly do you think a type is in a language like Ruby or Python?

A "duck type" for an object is the type defined by the methods (and their signatures) it responds to. So tell me, if "length" isn't meaningful for an object, why would "len()" be?

What I'm trying to get at is that you described length and len using different words, but said the same thing about them both. Indeed, while len() only works for certain kinds of objects, length only works on objects for which it is "meaningful".

Still, I do prefer consistent notation, and the . or -> syntax is appropriate for object orientation.

Re:What is it with ruby?

[Lord Ender]

The len() function may work for one datatype. You need different utility functions for different data types. But objects with length can all have .length properties.

Everything-Is-An-Object means working with objects, not poking at things with utility functions. It just makes more sense unless you have been indoctrinated in more primitive language types.

Re:What is it with ruby?

[bill_mcgonigle]

I don't get all this fascination with ruby, and why it currently seems to be so fashionable that people are willing to take perfectly good projects and rewrite them in this new fashionable language...

It's an excellent language, taking most of the good stuff from Perl and giving it proper object orientation. This lets you develop faster and potentially do better things. When time to market is more important than runtime performance then Merb might make sense.

I'm still writing Perl code until the VM is fast enough to use, which it looks like it almost is. I wish Perl 6 were here five years ago, and I honestly wonder what the bottleneck is there.

Re:PHP is still faster

[unity100]

for large web apps, using higher tools is BAD.

you will get innumerable scaling/modification/adaptation issues in future.

its not about 'like'

[unity100]

its about which technology is better.

because, even if we like it or not, we are being pushed stuff that are a pain to work with or problematic for the future, just because of the hype the '2.0' and similar crowds generate. and they are being herded by sources that want to sell certificates, books and make money.

it affects our lives. sooner or later some supervisor or client of yours is gonna shove some hyped tech up your butt, demanding you do stuff with it.

and no, its not about 'bettering yourself' or catching up with technology.

some stuff, are bad.

you'll want example at this point, here, have twitter.

Re:its not about 'like'

[resonantblue]

Dude, comon, it's not about books or making money. I am a former PHP programmer who has switched to Rails; and as far as I'm concerned, it *is* the better platform (that said, if you like PHP, I think it's a fine platform too). I not only like the elegance of the Rails framework, but I like many things about Ruby including the easy use of anonymous functions. Now you may discount anonymous functions as syntactic sugar, but it comes in very useful in so many situations to the point where I don't understand how I lived without it. You could argue it's even a critical part to many applications and even the foundation of Google's Map-Reduce algorithm (Joel on Software has an excellent blog entry about functional programming here: http://www.joelonsoftware.com/items/2006/08/01.html) As for Twitter, that is not a good example because they had a bunch of other issues going on. I happen to work personally at times with the guys from Powerset.com (who use Ruby + Rails extensively). As it turns out, they had a discussion with the head developers at Twitter when they went with Rails and they found out that Twitter's issues were not a result of Ruby or Rails. Don't believe me? Check out point #3 here http://glu.ttono.us/articles/2007/06/21/powerset-to-launch-front-end-on-ruby.

Re:its not about 'like'

[unity100]

http://www.killerphp.com/articles/will-ruby-kill-php/ http://phplens.com/phpeverywhere/?q=node/view/222 well. in my eyes, ruby is just another fad pushed forward to make new money. its not only about the language itself, its the widespread usage.

there has been many languages which were touted to be 'better', rightly or wrongly, that faded away.

Re:its not about 'like'

[resonantblue]

I'm not sure what you're driving at here because:
a.) I never said Ruby will kill PHP
b.) I didn't say Ruby will appeal to everyone

As for "fads," Ruby and PHP have both been around for about 13 years now. Ruby is not some new kid on the block. Rails came along later and really popularized Ruby in the web world, but Ruby as a language is hardly a fad.

faster as in

[unity100]

faster to build.

Re:PHP is still faster

[Bert64]

Yeah i agree there...
Lots of people seem to be converting existing projects to ruby because "it's the in thing", without bringing any practical benefit to the project, but at the same time replacing perfectly good code with new untested code which runs much slower and consumes more resources...

Compare the performance of metasploit 2 to metasploit 3 for a good example.

Who cares?

[Ash Vince]

Ruby is not really a serious programming language to be used in mission critical projects. It is an excellent resource for teaching development best practice due to the patterns it enforces but this comes at a cost. That cost is inflexibility. It frequently restricts thing that are on the borderlines of acceptable practice but in certain very specific circumstances are the best way of achieving a given end.

The only immediate example I can think of in my lunch break is it rigorous enforcement of best practice in database design. The real world is not perfect, sometimes you have to write code that interfaces with truly awful database design to get at legacy data, in this instance forget Ruby. If you are going to have to avoid a platform for certain jobs why use it at all?

Don't get me wrong, I think everyone should learn Ruby in the same way that everyone should learn Pascal. Pascal on the other hand cannot hold a candle to C or C++ in terms of professional development projects due to the flexibility it offers.

Quite frequently in the real world commercial pressures force an interim solution that Ruby cannot hope to offer for the reasons above. You might say that it is better to perform a complete rewrite in these situations but sometimes this is simply not possible due to the number of man hours available for a particular project.

We looked at converting our system to Ruby a few years ago but we simply did not have the developers to perform a complete rewrite in the time available. We could not afford to put all paying work on hold for the 9 months it would have taken our entire development team. In the end we chose PHP as this have us the flexibility we needed while allowing us to migrate the most important parts first but still use the old code for the rarely used pages until we need to modify them. While having the two systems side by side would be possible it would not be practical to get Ruby to pull data from an archaic awful database design that the old required.

Now some people say that Ruby is still perfect for new systems that will never be effected by these factors. I always counter that you never know what is round the corner in business, that is why flexibility is key.

Re:Groovy blazingly fast while jruby is slow?

[Foofoobar]

Looks to me like Groovy outperformed JRuby in many ways. I don't see what you are talking about. And it's still only in BETA!