Twitter On Scala

machaut writes "Twitter, one of the highest profile Ruby on Rails-backed websites on the Internet, has in the past year started replacing some of their Ruby infrastructure with an emerging language called Scala, developed by Martin Odersky at Switzerland's École Polytechnique Fédérale de Lausanne. Although they still prefer Ruby on Rails for user-facing web applications, Twitter's developers have started replacing Ruby daemon servers with Scala alternatives, and plan eventually to serve API requests, which comprise the majority of their traffic, with Scala instead of Ruby. This week several articles have appeared that discuss this shift at Twitter. A technical interview with three Twitter developers was published on Artima. One of those developers, Alex Payne, Twitter's API lead, gave a talk on this subject at the Web 2.0 Expo this week, which was covered by Technology Review and The Register."

Comments

Should have used PHP.

[0100010001010011]

Kidding aside, is this a 'nail' in the coffin of scalable Ruby? 5 years ago people were saying the same thing about PHP scaling but Facebook has done a rather nice job of making it scale. Twitter was supposed to be the poster child of how awesome Ruby and RoR was.

Difference is, Facebook is still using php, Twitter is going toScala.

Re:Should have used PHP.

[stoolpigeon]

It's funny you mention that on a day that facebook has been up and down like a yo-yo.

Re:Should have used PHP.

[Foofoobar]

Never was a nail except for the Ruby community that was in denial. Ever developer in the world except Ruby fanboys knew Ruby's inherent problem was scalability. Benchmarks showed it but they would always slant their own benchmarks to show the opposite. But facts are facts and in the end you can't deny the truth. So this is where we are at. The question is will Ruby fanboys still choose to deny the issues with Ruby or accept that it does have inherent limitations?

Re:Should have used PHP.

[julesh]

Difference is, Facebook is still using php, Twitter is going toScala.

PHP was a mature environment when facebook was launched. RoR was (and still is to a certain extent) a fad environment, popular primarily because of its differentness. People who build sites on a platform because it's the latest thing are less likely to stick with that platform than people who choose a platform that has a solid reputation but is boring. Scala, at a guess, is going to be the next fad platform. Like Ruby, it has some interesting ideas behind it, but it needs a lot of development before we can consider a stable platform for serious applications, I think.

Re:Should have used PHP.

[mini me]

While Facebook uses PHP where Twitter uses Rails, Facebook uses a plethora of languages to make the whole system work. So Twitter really isn't going to Scala any more than Facebook is going to Erlang. Which is the say that they use the best tool for the job, not one tool for every job.

Re:Should have used PHP.

[K. S. Kyosuke]

Ruby is a language. Languages usually don't have problems with scalability.

Re:Should have used PHP.

[digitig]

Really? Try comparing the scalability of, say, FORTRAN or COBOL against any modern language with decent encapsulation.

Re:Should have used PHP.

[Spaham]

yeah, reading the article I kept on thinking "but why do they keep on tryin unknown, unused, under-development languages ??".
Then they explain that they only used research oriented languages... Seems like they weren't programmers to begin with...

Re:Should have used PHP.

[nicolas.kassis]

Except most languages have more than one interpreter/compiler whit varying performance in varying environments. Jython/IronPython/Python all perform a little different. Ruby 1.9 and MacRuby don't perform the same.

Re:Should have used PHP.

[tcopeland]

> Ruby is a language. Languages usually don't have problems with scalability.

Quite right. An application with 8 million users will have scalability challenges regardless of what type of language opcodes are being executed. At some point it's all about architecture.

Re:Should have used PHP.

[mini me]

Everyone knows that Ruby has performance issues. The fact is that the performance problems are not problems for short-lived operations like serving web pages. The article even backs up the claim.

Ruby scales just as well as every other language. The article never once claimed that Ruby does not scale. It claimed Ruby had performance problems; a topic mostly unrelated to scaling.

If Ruby users were in denial about the problems with Ruby, there would not be several projects working on solving the problems. However, like I mentioned above, just because a problem exists in one domain does not mean it exists in another. Ruby will scale and perform just fine under tasks the Ruby fanboys you refer to are using it for (like Rails applications).

Re:Should have used PHP.

[Radhruin]

This isn't a nail in any coffin. The fact of the matter is that Twitter's implementation sucked, and sucked hard. It was not designed to scale in the least, and its authors had no idea how to build scalable websites, but instead learned as they went along.

Starling, for example, was Twitter's initial solution to their scaling problems. It was a way for them to offload tasks into a queue to be completed by workers. Pretty common use case, and indeed there are plenty of industry standard tools to do the job, and do it quickly. Twitter, however, decided to implement their own queue for reasons unknown, and guess what, it sucked. Starling was slow as molasses until it was almost entirely rewritten (still in Ruby, mind) somewhat recently.

Not to mention that listening to a Twitter developer blame Ruby for Twitters problems and pimping Scala is unsurprising. Not only is it typical for an engineer to blame his tools for his own failure, but Payne also has a vested financial interest in pimping Scala as the next big thing due to his book deal.

Basically, Twitter blaming Ruby is absurd. The problem lies with the developers and their lack of experience building scalable web infrastructure. You put the same developers in front of PHP and I guarantee you they'd have failed just as hard.

Re:Should have used PHP.

[Foofoobar]

Ruby is a language. Languages usually don't have problems with scalability.

So you are saying Visual Basic and C are going to have the same scaling challenges and that they scale equally well? Well all I can say it pass that magical crack pipe cause I want a toke too.

Re:Should have used PHP.

[Radhruin]

I find your assertion of differentness being the main reason to use Ruby on Rails to be somewhat offensive let alone uninformed, as it suggests that the multitudes of developers using it are doing so not because of technical merits but because they're buying in to some image of differentness. A cursory examination of the typical Rails project and developer should indicate otherwise. Because you don't find it helpful in your work doesn't mean others don't find it helpful or see real benefits from using the system. Perhaps you should be less dismissive -- you'd find yourself with a lot more interesting stuff to discover!

Re:Should have used PHP.

[Jane Q. Public]

Scala may be unknown, unused, and under-development, but Ruby is over 12 years old and the community that uses it is huge. So it is neither unknown or unused. (It is, however, under development, as most all modern languages are.)

Re:Should have used PHP.

[vlm]

Never was a nail except for the Ruby community that was in denial.

Well, yeah, no kidding. Thats because all I see is an endless circle of people repeating what they heard other people saying:

1) All the cool kids (us) know it won't scale. If you disagree, then you're not cool. Ha ha, thats our proof it doesn't scale.

2) I don't use it, never used it, never will use it, don't know anyone using it, but I am certain it won't work.

3) If it won't work on the worlds biggest websites, I can't use it on the departmental intranet, because I'm as cool as the worlds biggest website, just the world doesn't know it yet.

I can't find anything on google explaining why it "won't scale". Sometimes you can find ridiculous marketing droid babble about "tiers" but nothing technical or even logical.

No one can explain, why it matters if it can't handle the load of one of the top 10 or top 100 websites in the entire world, it therefore is utterly unacceptable on the other 99.999999999% of internet and intranet sites.

Personally, I think it's because the RoR user/devs are too busy making profitable apps to bother discussing foolishness?

Re:Should have used PHP.

[Foofoobar]

And where Facebook and PHP can accomplish this goal for the frontend and the backend (and many other corporate and enterprise PHP sites), Ruby can merely do frontend work. That doesn't seem like scalability to me. That seems very limited. That to me says 'as long as you don't TAX the engine, it can perform'. That's not scaling. That's getting by.

Re:Should have used PHP.

[K. S. Kyosuke]

"Decent encapsulation?" Don't know what that is supposed to mean here, but Common Lisp, e.g., fares quite favourably against Fortran.

Re:Should have used PHP.

[Anonymous Coward]

They started mega-hyping RoR before any of the application servers got out of alpha stage, that's the main reason it "won't scale".

Also Ruby is just slow.

Re:Should have used PHP.

[grumbel]

The problem is that most of those compiler/interpreters suck enormously. Or is there any high level 'scripting' language out there that can actually measure up to a decent Lisp implementation?

Re:Should have used PHP.

[Anonymous Coward]

Ruby and PHP launched the same year.

Re:Should have used PHP.

[Anonymous Coward]

PHP on the backend is as bad if not worse than Ruby (probably more in the worse category). It does not even have threading built in. Try building a php messaging queue and let me know how that goes. Facebook does not use PHP on the backend anymore than twitter was using Ruby for it. They have a ton of different stuff in many languages. I don't think you get what they are trying to do with Scala.

Re:Should have used PHP.

[iluvcapra]

RoR was (and still is to a certain extent) a fad environment, popular primarily because of its differentness.

Huh, I generally use it because it has really good ORM and migrations, and I really like the syntax (coming from Objective-C it's pretty slick. I also used the PHP language when I was starting out, but one day it tried to insist that $myArr[0] and $myArr["0"] actually pointed to the same object, and I have refused to deal with it ever since; I also got tired of typing str_sub_case_insensitive_for_real_safe(haystack, needle) -- or is it needle, haystack? And is this one of those prank functions that fails to substitute the value but still returns a value that evals true? Or if I leave out one of those underscores, am I in fact calling a function that behaves almost exactly the same way but fails under difficult-to-reproduce circumstances? Maybe they've fixed this and the other sundry atrocities? Maybe they've stopped trying to make it into Perl, as compiled by a C++ compiler, and tried fashioning it into an actual dynamic language? I know, I know, some people like PHP, but I think arguments for the superiority of PHP over Ruby (or Python or Scala or Lisp or WebObjects or Perl6 or really anything else) are going to rest completely on the skills of the Zend interpreter writers, and almost never on the quality/readability/maintainability of the code, or the ease of the development process. You can write good safe code in PHP, that is true, but it isn't very ergonomic.

You know, RoR is really good at replacing those old Paradox and FMP database systems. I can see how Facebook might prefer PHP, but people trying to replace little inventory/business processes systems generally only need to support a few dozen users, and don't have an army of developers to keep it running. The Universe is big enough to accommodate the utility of Ruby on Rails and the Twitter developer's stupidity.

Re:Should have used PHP.

[nicolas.kassis]

Also Ruby is just slow.

Not anymore. Go install 1.9 and start using it TODAY.

The ruby community needs to leave 1.8.* to die as quickly as possible.

Re:Should have used PHP.

[K. S. Kyosuke]

Visual Basic was so stupid that Microsoft could have easily made it almost as fast as C - if only they had wanted. I guess they had other priorities.

I must ask one thing: Have you ever heard of Self 93? They made it so good fifteen years ago that it topped at half the speed of C...a language where everything (!) was a message send and *anything* could have been redefined at *any* moment.

If someone threw the necessary money at Ruby, it could have easily the performance of a modern Smalltalk implementation, as Ruby is essentially Smalltalk in diguise. The only major difference I spotted was the fact that Ruby objects behave as hashtables, whereas Smalltalk objects are essentially arrays of references with methods fetching the references ("instance variables") using fixed indices that change only during class redefinitions. I guess you could fake the former with the latter, enjoying a speedup, as even in Ruby most objects of a single class have the same attributes.

But still, I guess that the perf. problems of Twitter are primarily due to Rails and not to Ruby.

Re:Should have used PHP.

[K. S. Kyosuke]

Sorry, this is not the fanboy you are looking for. This one is a Scheme fanboy. ;-)

Re:Should have used PHP.

[zuzulo]

I guess i just dont understand why they decided to use scala rather than a robust and long standing language like erlang. Odd design choice.

Re:Should have used PHP.

[Haeleth]

Indeed; and, reading TFA, I observe that the main reason they're preferring Scala on the backend is that it's made it easier for them to improve the architecture, by (for example) providing better concurrency primitives than Ruby did at the time they started switching.

No reason for Ruby fanboys to get defensive over anything here; they make it clear that they still love Ruby for front-end stuff, but that Scala works better for a different kind of task.

Re:Should have used PHP.

[Foofoobar]

so... your logic is that money makes everything better? so given a piece of shit and enough money, you can engineer that piece of shit in a Ferrari? I'm sure you'll say yes and THINK you can but the fact of the matter is you'll just end up buying me that Ferrari and saying you did. :)

Re:Should have used PHP.

[^Case^]

Difference is, Facebook is still using php, Twitter is going toScala.

PHP was a mature environment when facebook was launched. RoR was (and still is to a certain extent) a fad environment, popular primarily because of its differentness. People who build sites on a platform because it's the latest thing are less likely to stick with that platform than people who choose a platform that has a solid reputation but is boring. Scala, at a guess, is going to be the next fad platform. Like Ruby, it has some interesting ideas behind it, but it needs a lot of development before we can consider a stable platform for serious applications, I think.

What a load of bullcr*p. What are the arguments that RoR is a fad? The fact that people build great businesses on it? The fact that what seems like every other language now has a least one RoR clone - including Microsofts .Net?

I wonder if you've ever even tried implementing anything with RoR. If you had I would expect you to have found plenty of reasons to use RoR over PHP, not because it's different, but because it provides you with a lot of great tools not present in PHP.

Of course I might be mistaken. You might actually have done a serious attempt at using RoR for a project and found it to be worse than PHP. In which case I would love to hear what your objections to RoR are.

Finally I would love to hear what you mean by serious applications. Obviously you don't consider Ruby stable enough for Twitter, Basecamp, Yellow Pages or LinkedIn. Or maybe you just don't consider those serious applications?

Re:Should have used PHP.

[Foofoobar]

Listen to your own Ruby rhetoric. Ruby itself doesn't have threading; it's FAUX threads. And PHP was never meant to be used for app development; it's meant for web development. That would be like trying to build an OS in Visual Basic (now watch some idiot try).

Yeah I understand what they are doing with Scala. And They DID try to do alot of it with Ruby over the years and Ruby people touted it over and over while we said it doesn't scale over and over and pointed out how they were removing more and more Ruby on the backend. Now you Ruby people say it was never on the backend? Unbelievable. You really do live in a world in yoru own mind. Hopefully you didn't build it on Ruby cause there's only room for one before it crashes.

Re:Should have used PHP.

[Anthony_Cargile]

Never was a nail except for the Ruby community that was in denial.

The same thing I said about six months ago - look at the comments on the article. Ruby/Rails never reached the support it needed to be widely deployed, and it's less likely for an ISP/host to deploy a framework than a well-established language. Case in point: more Perl deployments by hosts than Ruby or rails, not to mention PHP/ASP.NET.

Re:Should have used PHP.

[iluvcapra]

Well, it wasn't stable enough for twitter :D

Re:Should have used PHP.

[MarkWatson]

I have used Rails a lot in the last 3 years, but I don't much use PHP except for simple stuff. So, I may be wrong about this, but I believe that PHP does scale slightly better that Rails. So what?

For me, Rails hits the sweet spot for small web applications that need to be written inexpensively and are easy (and inexpensive) to modify and extend.

Rails is all about saving development costs. For large scale systems with many users, writing back end web services in Java or Scala makes a lot of sense.

  I explain to my customers the tradeoffs of runtime efficiency vs. less expensive development. Usually, depending on circumstances, it is an easy enough decision to make.

BTW, the Lift web app framework (written in Scala, slightly similar to Rails) might even make Scala popular for web interfaces - who knows?

Re:Should have used PHP.

[caramelcarrot]

Facebook doesn't use PHP for the backend, it's mostly C++, Python and Erlang.

Re:Should have used PHP.

[robmv]

I think the real change they are making is not replacing a language with another new, they are replacing a basic VM with a more powerful and proved one: Java VM

I think their problem is not the language is the current state of the Ruby VM.

I hope Twitter developers can publish why they choose Scala instead of JRuby (Ruby on the Java VM)

Re:Should have used PHP.

[TeXMaster]

The problem is that most of those compiler/interpreters suck enormously.

Exactly. MRI (Matz' Ruby Interpreter) is known to have some serious scalability issues. Interestingly, one of the main issues with MRI comes from the way gcc compiles the big delegator switch in MRI's core, with a large sparse stack that causes ridiculous memory consumption (and sometimes even leaks). There's a set of 8 patches (the MBARI patchset) that drastically improve the situation. The reduced memory footprint and the much smaller stack also give a noticeable speed increase.

The good news is, these patches are progressively being merged upstream, so it's very likely that future MRI versions will be much better.

Re:Should have used PHP.

[iluvcapra]

Ruby itself doesn't have threading; it's FAUX threads.

By that standard PHP or python doesn't have threads. Ruby at least is able to run green threads and fibers, and PHP can't even do that. I'm not sure what any of this has to do with "application" versus "web" development, since on the "back end" a web app and an app app look a lot alike, and something like twitter definitely needs a message queue that works with memcached.

After the success of Hulu and LinkedIn and the continued adoption of Rails for larger projects (many sited below), I think we can relegate the "Ruby doesn't scale" meme to crank-land, don't you agree?

Re:Should have used PHP.

[digitig]

I don't follow your logic. I only suggested that encapsulation made an improvement to scalability, not that it is the only thing that makes an improvement.

Re:Should have used PHP.

[MarkWatson]

Hello Rob,

Scala runs *much* faster than JRuby.

That said, I use JRuby much more often than Scala - BTW I wish that I new Scala better. I don't have much free time right now, but when I do I was planning on mastering Haskell. However, since Haskell and Scala have some things (like type inferencing) in common, perhaps Scala would be a better choice for my work. Tough call though.

Re:Should have used PHP.

[Foofoobar]

By that standard PHP or python doesn't have threads.

Never said PHP had threads. Hell even Twitter devs says Ruby doesn't have 'true' threads and that they are fake threads. As for LinkedIn and Hulu, watch how fast they dump Ruby; Hulu already is having problems. Seeing that scaling issue yet? PHP is used by PeopleSoft and Financial corporations who need scalability and stability. People who use Ruby apparently don't care about either.

And you seem to flip flop between saying 'Ruby is great on the backend' one second and the dnying it the next. Make up your mind. This is our entire argument about it's scalability. PHP IS the backend AND the frontend for all the sites that use it.

Ruby is just wrapping paper for another language like Scala that can actually scale. :)

Re:Should have used PHP.

[DragonWriter]

Kidding aside, is this a 'nail' in the coffin of scalable Ruby?

No. Its not even the 'nail' in the coffin of scalable Rails, and Ruby isn't Rails.

Twitter was supposed to be the poster child of how awesome Ruby and RoR was.

Since when? When the point-to-an-example site method of arguing for scalability is used with respect to Rails, the site held up is, IME, usually yellowpages.com.

Re:Should have used PHP.

[Anonymous Coward]

Traversing giant class hierarchies for method dispatch, on every method dispatch, is always going to be slow. Even caching is slow, since you will just end up with a copy of the class hierarchy. There's no way around it. This is why C++ and Haskell use "templating" and other similar Turing complete typing constructs to deal with the issue at compile time.

When you start using abstractions 20 layers deep, and have to climb up a tree for every expression (since everything is an object), you will face major difficulties.

Still, Ruby is a fine language, and 1.9 does offer improvements that bring it up to par with Python, for example.

Re:Should have used PHP.

[Trepidity]

It depends partly on what your concurrency looks like. Erlang supports one model of concurrency, a lightweight message-passing one with no explicit threads or shared memory. Scala supports that one as well---less efficiently---but also supports standard Java multithreading, which some people find useful for some purposes.

Re:Should have used PHP.

[zigamorph]

the way gcc compiles the big delegator switch in MRI's core, with a large sparse stack that causes ridiculous memory consumption (and sometimes even leaks).

Couldn't the code be written in a way that gcc can compile efficiently then? Seems like a simpler solution. For a large static switch a perfect hash might be an option.

Re:Should have used PHP.

[lotzmana]

I agree, but wish to add a comment about vertical and horizontal scaling.

Ruby and Python have poor multi-threading. They don't scale well on multi-CPU platforms.

from the interview:
Robey Pointer: Green threads don't use the actual operating systemâ(TM)s kernel threads.

So, a Ruby application can't scale well vertically -- one can't just upgrade the machine with more CPUs for example.

At the same time, no language is inherently prohibiting horizontal scaling, if application design provides for it -- adding more machines onto which the application can run in parallel.

Twitter could've been designed to permit horizontal scaling. Regrettably the article didn't say much about this approach. They are improving the vertical scalability of the application by switching to first-class threads (via the JVM), but are they not eventually going to hit the limits for vertical scaling?!

Re:Should have used PHP.

[Anonymous Coward]

After the success of Hulu and LinkedIn and the continued adoption of Rails for larger projects (many sited below), I think we can relegate the "Ruby doesn't scale" meme to crank-land, don't you agree?

Almost anything can scale, if you throw enough money at it. That doesn't mean it scales well, or efficiently. Servers (and cooling, and rackspace, and power, and administration overhead ...) are not cheap, no matter how many times Rails *developers* claim that they are.

Re:Should have used PHP.

[K. S. Kyosuke]

No, my logic is that Ruby is essentially a member of a well-researched class of languages, for which (the class, not Ruby) high-performance VMs have already been developed (Cincom Smalltalk, Gemstone, Self93, Strongtalk...), but this development was always expensive (this was also the case of Java and .NET, obviously). IMO, your shit(ty?) analogy does not apply here.

Re:Should have used PHP.

[Anonymous Coward]

Actually some languages do have built in features that won't let them scale in certain spaces. If your threading model is nasty or your memory access features too coarse grained or a number of other things get designed wrong at the language level and are by specification broken then you have problems. On top of that if you don't have competition in your compiler space then it doesn't matter if it's not part of the language spec any non scalable parts of the compiler implementation is by default a problem with your language. Same thing for your app server/container if you don't have competition and there are issues with the one container you have.

Not that it invalidates your point but arguing semantics of "languages don't have problems with scalability" meaning the syntax itself doesn't induce scalability problems is being a bit dense intentionally. A language is more than just its syntax it's also the available implementation and tools (I suppose this is the IT version of "if a tree falls in the woods does anyone hear it"...."if a language can't be used to write executable instructions is it really a language?")

And now that my rant is over I'll just say that my experience has been in the C/C++, Java, and M$ areas and I don't know anything about Ruby or Rails so this has nothing to do with the original article.

Re:Should have used PHP.

[SanityInAnarchy]

For that matter, Ruby 1.9 has real threads (but GIL'd) -- this puts its threading in exactly the same class as Python.

Still disappointing, once you know Erlang...

Re:Should have used PHP.

[DragonWriter]

I think the real change they are making is not replacing a language with another new, they are replacing a basic VM with a more powerful and proved one: Java VM

If that was their intent, they could just replace Ruby with JRuby.

I think their problem is not the language is the current state of the Ruby VM.

I think that's one of their problems, but not the main one.

I hope Twitter developers can publish why they choose Scala instead of JRuby (Ruby on the Java VM)

They have. For one component, its because one of the developers threw together a replacement in Scala, they tested alternatives to their existing component including that one, and the one their own developer had written in Scala happened to be best. For some other things, its because they've found that they are, for whatever reasons, doing lots of type tests in their Ruby code (which is, certainly, a code smell and probably symptomatic of basic design problems, but its a fact of where they are) and that they think it makes sense to avoid the problems those have been put into place to address by using static typing.

Re:Should have used PHP.

[DragonWriter]

If someone threw the necessary money at Ruby, it could have easily the performance of a modern Smalltalk implementation, as Ruby is essentially Smalltalk in diguise.

JRuby 1.1.2, has similar performance, and Ruby 1.9.0 better performance, in the alioth shootout, to GNU Smalltalk or Squeak, and worse performance (though Ruby 1.9 by not as much) than VisualWorks Smalltalk. But JRuby performance (1.2.0 is current) has come a considerable way since 1.1.2 (and Ruby 1.9.1 probably has some improvement from 1.9.0), so I don't think modern Ruby is much off from modern Smalltalk implementations.

Re:Should have used PHP.

[DragonWriter]

Or is there any high level 'scripting' language out there that can actually measure up to a decent Lisp implementation?

Since many decent Lisps are high-level dynamic languages (hence, "scripting" languages) clearly so.

Re:Should have used PHP.

[hhr]

The article is about Ruby on Rails. Ruby on Rails is not just a langauge. It is a lanaguage and a web framework. Frameworks very much affect your scalability.

Re:Should have used PHP.

[DuckDodgers]

Scala has the significant advantage that it's built on Java and interoperable with Java. Scala source code compiles directly into .class files. You get the speed of the JVM (which is acceptably quick these days), the ability to easily call Java APIs from within Scala, and the ability to run your Scala code on any machine with the JVM.

It's popular to dislike Java, and even as a well paid Java developer I'm not a huge fan of the language. But Java still is extremely common, and you can even write Java code for your Scala code to use while you're learning Scala.

Scala also keeps Java's strong static typing and adds functional language features. I don't think it needs any development at all to be adapted for mainstream use.

On the other hand, as a C++ developer I found learning Java to be child's play. The learning curve from Java to Scala, for me at least, is noticeably steeper. If anything kneecaps Scala I suspect it will be the barrier to entry, not the language itself.

Re:Should have used PHP.

[DragonWriter]

Couldn't the code be written in a way that gcc can compile efficiently then?

It could be, and IIRC there is some consideration being given to incorporating some or all of the MBARI patches in to Ruby 1.8.x in future releases (which will continue even while the main thrust of new Ruby development has moved to 1.9.x); of course, MRI has been completely replaced with the YARV VM for Ruby 1.9.x, making the whole issue moot for the main line of development.

Re:Should have used PHP.

[DragonWriter]

Listen to your own Ruby rhetoric. Ruby itself doesn't have threading; it's FAUX threads.

Ruby 1.9.x and JRuby both use native threads (well, Java threads for JRuby, but most/all JVMs use native threads), Ruby 1.9.x with a GIL, JRuby without one.

Re:Should have used PHP.

[tcopeland]

> If your threading model is nasty or your memory access
> features too coarse grained or a number of other things get designed
> wrong at the language level and are by specification broken then you have problems

In Ruby's case, you overcome this by using processes rather than threads.

> On top of that if you don't have competition in your compiler space
> then it doesn't matter if it's not part of the language spec any non
> scalable parts of the compiler implementation is by default a problem with your language.

At the level we're talking about - 8M users - I think that compilation optimizations/bottlenecks are not the issue. It's about architecture. It doesn't matter if you're writing assembly language, you'll still need to scale out of one machine's memory space. I'm thinking stuff like Werner Vogel's discussions of CAP.

> A language is more than just its syntax it's
> also the available implementation and tools

Yes, and I'd say that there's also the infrastructure surrounding the language - protocols, operating systems, caching tools, etc. Ruby is well suited to integrating with those sorts of things, which is why there are myriad techniques available for scaling Ruby apps.

All that said, I agree with your proposition - that some languages are well suited to some tasks. As some folks here have suggested, Twitter's messaging bus sounds like a perfect Erlang app.

Re:Should have used PHP.

[RichardJenkins]

Why do people even compare RoR to PHP - I'm pretty sure it gives people the wrong impression:

PHP vs Ruby to compare languages
CakePHP vs RoR compares different framework/language combinations
Pylons vs Django compares different frameworks in the same language

If someone's really trying to decide between PHP and RoR on rails they need to ask themselves why they're looking at RoR instead of raw Ruby. If they're not sure then take a step back and do some more basic research.

Re:Should have used PHP.

[kv9]

I don't get this buzzword-bingo bullshit about Twitter (or whatever the fuck site-du-jour is) in regards to concurrency and scalability. this is not a complex application, it's something that you code one afternoon (in Java/PHP) then throw it in a rack full of HTTP server nodes, a load balancer (shit, even RR-DNS will do) and a RAMSAN for the DB. that's it. stop the drama.

Re:Should have used PHP.

[recharged95]

Have you used facebook lately?

It's slow, the apps bring on the pain, and everything is klunky. <flame-war-hat-on>Like PHP normally is<flame-war-hat-on>. Until facebook has the same number of lifestream transactions as twitter, it's no comparison (and it's smart they are looking/integrating at Scala).

Flame on.

Re:Should have used PHP.

[WgT2]

Well, that explains the reliability issues I've heard about the site.

Re:Should have used PHP.

[CMonk]

Huh? Are you confused? Both companies made the stupid mistake of using a web scripting language to do backend heavy lifting. Twitter is fixing that with Scala. (leaving RoR on the frontend because its really good at web frontends) Facebook already fixed that with ERLANG, not PHP. (leaving PHP on the frontend because of the technical debt they've accumulated) http://www.facebook.com/eblog BTW, PHP is the worst language ever. That is a fact, not my opinion.

Re:Should have used PHP.

[CMonk]

Please mod this post up. PHP is ridiculously bad. When you build things with a huge committee you are likely to get something bad, when you build something with a huge committee of people that have never programmed before, only edited someone's perl CGI, you get PHP.

Re:Should have used PHP.

[david.given]

If someone threw the necessary money at Ruby, it could have easily the performance of a modern Smalltalk implementation, as Ruby is essentially Smalltalk in diguise.

You might be interested in StrongTalk. It's a Smalltalk-80 rewrite with optional strong typing and pervasive JIT, meaning that it's incredibly fast and robust. It is, unfortunately, Windows only at the moment, but it's all BSD licensed --- VM, image, source browser and all --- and they're looking for people to help with a Linux port.

Re:Should have used PHP.

[Anonymous Coward]

so... your logic is that money makes everything better? so given a piece of shit and enough money, you can engineer that piece of shit in a Ferrari?

One of my mentors told me that you can do anything given enough time and money.

Let's take an actual pile of shit and turn it into a Ferrari. Okay. No problem. We will figure out how to turn the shit into energy then restructure the energy into the materials for a Ferrari.

Look, I just invented the transporter. I'm a sci fi writer!

Now, you gave no time-line and no budget so I'm going to ask for 100 trillion dollars and about 500 years. That should do it.

To make Ruby on Rails as performant as Scala all I would need is about a billion dollars and about 10 years.

Re:Should have used PHP.

[Achromatic1978]

I'd flame on, but I'm trying to compose myself after the fits of laughter that the phrase "lifestream transactions" brought on. Seriously?

Re:Should have used PHP.

[Anonymous Coward]

PHP IS the backend AND the frontend for all the sites that use it.

Ruby is just wrapping paper for another language like Scala that can actually scale. :)

Hum, what is your definition of "Backend"? Ruby is used in many more areas then I see php. For example Puppet is a good example of something that scales to where google uses it to stage all their servers and propagate configuration updates. I don't see PHP being used in such settings very much. I rarely see PHP anywhere but in web related tasks.

If you mean backend as in connecting PHP to a database then I don't see anything wrong with Ruby. You seem to think Facebook is using PHP for everything when they clearly don't. While Twitter tried and had varying success trying to build everything with ruby (and it's debatable if ruby was the point of failure. Infrastructure seem much more of an issue here), Facebook didn't try.

Twitter still uses Rails for everything from routing web request to connecting to their database from the web and their api as far as I understand. Thousand of pageviews and request per seconds. Even Facebook doesn't do this. How many clients pull from Facebook every 2 seconds for updates? At the moment Twitter is using ruby to respond. They are trying to move to something else but it's not clear that they will have more success.

As for peoplesoft. All the application I run from them are primarily Java based. OMNI being the biggest one we use here at a state university.

As far as PHP apps go, I have seen many horror stories of people trying to scale Moodle for small campuses. Scalling is mostly all in the way you build it not what you use as an underlying platform.

And PHP+CakePHP is the slowest web framework on the planet. PHP is also slower than ruby now. So go fix your own issues.

Oh and for the java crowd. Look at Blackboard for something that does not scale well at all (and a lot of other crap J2EE based apps I had to maintain with memory leakage comparable to crappy C.

Re:Should have used PHP.

[phrenq]

That's fine for the vast majority of web applications, but you clearly don't understand the scale of the traffic a site like Twitter receives. What do you do when one database machine, no matter how fast, isn't enough? When your load balancer gets overloaded? How about handling that massive search index?

Even a site as simple as Twitter will present you with problems you never expected once it gets that popular. Starting off with something like you suggested is exactly what gets them in the mess they're in now. It must be designed for parallelism and scalability or it will fall over.

That said, if it's properly designed, you can probably make it work in any language, although you can dramatically reduce the number of production machines it takes if you have an efficient compiler/interpreter.

Re:Should have used PHP.

[g8oz]

Scaling to that size is a solved problem with Java/PHP.

With Ruby the problem is still very much open and will remain so until a better implementation comes along. Read the interview, they talk about Ruby using green threads - that alone means that it will never scale the way Java can.

Re:Should have used PHP.

[Anonymous Coward]

Twitter could've been designed to permit horizontal scaling. Regrettably the article didn't say much about this approach. They are improving the vertical scalability of the application by switching to first-class threads (via the JVM), but are they not eventually going to hit the limits for vertical scaling?!

Being on the JVM means they can use tools like azul compute servers depending on their architecture they might be able to use different JVM scaling tricks. Once you are on the JVM you get to cheat with tools like that.

Re:Should have used PHP.

[K. S. Kyosuke]

I now about Strongtalk, I have even mentioned it here ;-), but it is kind of pining for the fjords.

Re:Should have used PHP.

[david.given]

Which is a shame, as it's top-quality pine and the most picturesque of fjords.

I do wonder whether the VM/JIT engine might be useful for implementing other languages, however. Such as Ruby! Just implement a compiler from Ruby source into Strongtalk bytecode, plug it in, watch it work...

Re:Should have used PHP.

[Anonymous Coward]

You put the same developers in front of PHP and I guarantee you they'd have failed just as hard.

So basically you're telling me that the developers at twitter are incompetent. Sounds like getting hired at twitter is a death sentence for your career.

Re:Should have used PHP.

[Anonymous Coward]

> The Universe is big enough to accommodate the utility of Ruby on Rails and the Twitter developer's stupidity. ...And this is exactly the problem with the Ruby on Rails community. Instead of admitting the problems and trying to fix them, they just insult the developers who found them. The developers from Twitter pretty clearly explained the problems they had and why Scala helped to solve them, and are certainly not stupid.

Re:Should have used PHP.

[tcopeland]

> Scaling to that size is a solved problem with Java/PHP.

Java/PHP apps have scaled to that size, but to do it they used the same techniques that a Ruby app would use - caching, async queues, sharding, etc.

> that alone means that it will never scale the way Java can.

That means that to scale a Ruby app you may need to use something other than a thread-per-task model. Java runs into the same problem at a certain point - a JVM (and the hardware it's running on) can only handle so many concurrent native threads. At that point you need to scale using more hardware, processes, message queues, etc.

Re:Should have used PHP.

[jbolden]

I've never seen anything on Allegroserve. What other Lisp Ap servers are there. Haskell is a kissing cousin and Happs exists.

I don't think either is nearly as full featured as ROR, much less the professional quality ones that Scala uses. Playing with Happs is on my todo list.

Re:Should have used PHP.

[ClassMyAss]

If that was their intent, they could just replace Ruby with JRuby.

They couldn't - they tried, and it wouldn't run their stuff (why exactly this is, I have no idea, but that's what Alex said). Maybe it would have been possible with more work, but they decided that they could do the job in less time and with better performance by rewriting pieces of their system in Scala.

I agree that the type tests are a code smell; it may even mean that they're not great Ruby programmers, I don't know, not having seen the code. But in any case, if the development team that they have can spend a bit of time in Scala and write higher quality code that runs faster, that's absolutely what they should do - we're not talking about a startup that needs rapid prototypes anymore, we're talking about a company with massive scaling issues that are starting to outweigh the need to add features quickly. As time goes by and a codebase matures, critical bits of code always migrate to lower levels as you need the performance boost, and I think that's both reasonable and expected. And it doesn't mean RoR sucks or anything, it just means it's not The One True Solution To Everything.

I was going to address the claim that Ruby folks like to make about dev time being more expensive than compute time, but there's no point - people that are experienced with Scala can bang stuff out just as fast as in Ruby, and frankly, it's very easy to go back and forth between the two. You lose a tiny bit of power due to static typing, but it's still an extremely productive language in the right hands.

If they were rewriting in Java, that would be another issue altogether...

Re:Should have used PHP.

[nicolas.kassis]

I don't think Ruby will be able to accommodate their claimed need for a strongly typed language. And they did not really explain the technical details of the problem in depth. And they seem to be saying that their old code required a lot of type checking using kind_of? which is a less then ideal way to do anything. But is it because the code is bad (Alex from Twitter acknoledge the code has issues.)? or Dynamic languages are wrong for what they want to do.

Alot of the gain they might make will probably be due to the fact that they are rebuilding from scratch parts of twitter which often lets you redesign for the better. I think they would have had the same benefit rewriting in ruby.

Re:Should have used PHP.

[Firehed]

Well that's just the problem - it probably was coded in an afternoon (or at least the initial app; not so much on the more recent functionality, namely search). And if you've coded an app in an afternoon, then you didn't give half a thought to scalability when designing the thing.

There are tremendous scaling issues with any site that has millions of users. Doubly so for a site where most of those millions of users are hitting the API about once a minute, every hour of the day (thanks to the plethora of third-party clients). The issue isn't so much with RoR I expect, but probably their DB architecture. When you use these frameworks that do the thinking for you, it's easy to make stupid oversights that wouldn't have come about had it been done by hand by a competent coder. This could cover anything from adding indexes to de-normalizing tables to crazy-advanced SQL replication customization and memcached usage (a la Facebook).

Not to mention that a RAMSAN-based DB is going to be crazy effing expensive - certainly not something they could have accessed before getting stupid amounts of VC funding. I'd suggest that using cloud-based infrastructure might have worked better (since they've already done most of the leg-work for both application scalability and hardware), but that comes with plenty of its own pitfalls. And of course crappy programming is still going to keep costs unnecessarily high when you're being billed by the compute cycle.

Re:Should have used PHP.

[chez69]

I'd agree that a lot of J2EE applications and containers are complete shit (i'm a java developer and I like it, dammit). If you want to write good and scalable java code, a lot of times it's better to go with a lightweight system. an example is the Pojo message beans you can use with spring, or using embedded Jetty instead of a big slow ass websphere to serve up web services. Leaks in java apps crack me up because it's usually somebody who doesn't understand the feature they're using.

Re:Should have used PHP.

[Foofoobar]

Ruby is used in many more areas then I see php

And your point? As I stated earlier, PHP was NEVER meant to be used for APP DEV... only web dev. That was the languages main focus and as a result it does it well when it sticks to that one focus. And it scales and handles page loads like lightning! But app dev? Not in your right mind. Sure it CAN be done.. but you can build Twitter on Ruby as well; it just shouldn't be done because the language is a bad choice due to it's inherent weaknesses: PHP for app dev, Ruby for web dev.

There is no swiss army knife language and PHP accepts that right off the bat. If you think Ruby is that swiss army language, you are only deluding yourself.

Re:Should have used PHP.

[chthonicdaemon]

You know, technically there are more recent Fortran specs. Fortran gets hauled out in the form of FORTRAN77 every time someone wants to talk trash about an old language. However, just like C or C++ or Java, the language has evolved. Fortran 2003 has objects, a pretty cool module system and basic thread support. Fortran 2008 will have explicit support for splitting data over multiple cores (co-arrays). I don't know much about COBOL, except that some of the largest codebases in the world are written in COBOL.

Re:Should have used PHP.

[Firehed]

All you've really done are cite examples of bad programming, not scaling problems tied to a specific language. Obviously when you're dealing with a user base in the millions (Facebook, Twitter), scalability is going to be an issue no matter what language you're using. The rails framework probably doesn't help them, because then they have the overhead of a framework in the mix.

Good development practices are far more important than choosing the right language 99% of the time. If you plan with scalability in mind, you could run the damn app on punch cards and still have it work better than plenty of the PHP horror stories out there. The issue with PHP (and increasingly, Rails) is that it's so accessible to newbie developers that there's a ton more bad code out there that seasoned developers wouldn't make. Yes, when I was first using databases, I managed to pull off something to the effect of "SELECT * FROM .. WHERE .. LIMIT 1" in a loop that ran through about 2100 times, and it was REALLY damn slow. It should have been a single DB query using COUNT with a GROUP BY clause. Because I was first getting started (and was also crunched for time, given the nature of the project), I had written some truly awful code. Everyone's done it before. Now I've got highly reusable personal code libraries that can automatically handle caching and master/slave replicated DBs, rather than attempting to open a connection to the database 90 times per page load.

Look at Wordpress, for god's sake. It must be one of the worst-scaling applications ever written, and even the caching plugins are rather poorly done (they work, but they have to be made in very stupid ways to deal with WPs architecture). But then again, look how big the install is for how much functionality you get. Users, posts, comments, and some of the strangest plugin and template architecture I've seen in my life. It's certainly not that simple anymore, but FFS it just took 6 seconds for it to render the page HTML from LOCALHOST! A high-spec MBP with an upgraded hard drive should NOT take that long to read three rows from a DB table.

But that's not PHP's fault.

Re:Should have used PHP.

[gullevek]

As much I do like coding PHP I completely agree with you. The fact that $arr[0] is the same as $arr['0'] is just insane and that each function has a different order of needle, haystack, replace, etc is just unbearable ... So much that I already have wrapper functions so they are all the same ...

But unless I find years of time and unlimited money there is no way I can ditch all my php code and go to ruby or python or anything else ...

Re:Should have used PHP.

[grimw]

Any RoR developer I've ever talked to was just into it because it was "hip". That's fine and all for toy projects, as I've used that excuse in the past, but it's not an appropriate reason for choosing production-level languages. For that, you just have to sit down and hammer out the details of what the tools/languages are going to offer you or prevent you from doing (easily).

My comments aren't to say that RoR doesn't have benefits that actually make it useful. In fact, that'd be difficult for me to say either way, because I have not used it.

I do know that, in general, ORMs are slow, difficult, and language/library-specific when compared to some more general DB access libraries out there. Really, it doesn't take a lot of code to setup an object that can use a DB in an optimized fashion if you just write that code yourself, and it'll be a lot clearer without all the "magic objects" floating around.

For examples of slow code, try writing Hibernate apps in Java that must use saveOrUpdate() when you could easily write some more explicit code to do any pre-caching for you. Same principle will apply to other ORMs.

Re:Should have used PHP.

[grimw]

I'd guess they compare RoR to PHP because that's the only "compelling" tool in Ruby to use for web development. At least, it's the most well-known one, meaning it's better for production environments, because it will be more stable and have more developer mindshare than X framework in Ruby.

So, I think the real debate is over whether to do things in PHP, where mindshare is massive and there is no ORM (which I think is great... though maybe a PHP ORM does exist) or whether to do things in Ruby+RoR, where mindshare is much smaller and ORMs are in play (you already know my personal opinion about this).

Re:Should have used PHP.

[Ma�djeurtam]

GNU/Hurd launched long before Linux.

What's your point?

Re:Should have used PHP.

[iluvcapra]

The fact that $arr[0] is the same as $arr['0'] is just insane

I'm not even sure if that was what it was, I think the problem I was having on top of that was that I would pop something off the head of $arr and sometimes it would be $arr[0] and sometimes it would be $arr[1], and this is crazy, but I dug in and low and behold I was actually storing values in the array to ["0"] and ["1"], and since arrays and hashes are the same thing in PHP (this is the real sin), it did what it thought was right, and created a non-deterministic bug. Why does it let you pop a value off a hash? UGH!

But one day I discovered Ruby, and the first lesson I learned was that error messages are actually a good thing, instead of the PHP developers practice, wherein they construct the entire language around the principle of guessing what you mean and never reporting errors. Until the interpreter segfaults.

Re:Should have used PHP.

[shutdown -p now]

Scala also keeps Java's strong static typing and adds functional language features. I don't think it needs any development at all to be adapted for mainstream use.

Scala is a great thing, now what it needs is equally great tooling (i.e. IDE support, including refactoring, on the same level as we have for Java). And it's getting there - there is an Eclipse plugin, and a NetBeans one - but it definitely needs more work and polish.

The payoff would be huge, though. Right now, as far as languages go, C# is far more advanced than Java. But Scala is equally more advanced than C# (the only thing on .NET that could compare with Scala is F#, and that's less stable and mature currently... but it will stabilize once VS2010 is released, so there isn't much time). Some significant investment into Scala by one of the big players could help straighten things out a bit (it's good when there are two major competing languages in the same niche, because it forces them to evolve at a more rapid pace, steal each other's features, and learn from each other's mistakes - as Java and C# did for several years, until Java began to stagnate).

About the only thing I don't like about Scala is generic type erasure it inherited from Java to maintain class compatibility; but there are workarounds already, and if Scala becomes The Next Language for the Java platform, it may well deprecate erasure and introduce its own reified generics then.

Now, if only someone would pick it up and throw money (and marketing resources) at it. Like, say, Google...

Re:Should have used PHP.

[Anonymous Coward]

Kidding aside, is this a 'nail' in the coffin of scalable Ruby? 5 years ago people were saying the same thing about PHP scaling but Facebook has done a rather nice job of making it scale. Twitter was supposed to be the poster child of how awesome Ruby and RoR was.

Difference is, Facebook is still using php, Twitter is going toScala.

Except that they don't, were they need true scalability (chat) they use Erlang, so no PHP does not really scale

Re:Should have used PHP.

[julesh]

What a load of bullcr*p. What are the arguments that RoR is a fad? The fact that people build great businesses on it? The fact that what seems like every other language now has a least one RoR clone - including Microsofts .Net?

Here's the thing: just because something is a fad doesn't mean it isn't good. There are plenty of fads that, once the smoke has cleared, it turns out the object of that attention was something really special. The Beatles, for a pop culture example. But what being a fad does mean is that it has gained attention from a lot of people for the wrong reasons. In Ruby-on-Rails' case, those reasons are usually "it was used on this site that was a really cool site, so it must be good". That, I think, is the reason 90% of developers who have started Rails projects in the last couple of years chose the platform. They didn't look at and think about how useful the ORM is, examine the MVC framework and decide that it was a better fit to their application than, say, Struts, and nor did they choose Ruby because they feel that the psuedo-functional style enabled by Ruby's neat closure syntax was the obvious natural way to write programs. They picked Ruby because that's what Rails uses and they picked Rails because that's what Twitter uses.

Just because something is a fad doesn't mean that people can't use it and get good results. But it does mean that some people using it may get better results if they look elsewhere.

I wonder if you've ever even tried implementing anything with RoR. If you had I would expect you to have found plenty of reasons to use RoR over PHP, not because it's different, but because it provides you with a lot of great tools not present in PHP.

Yes, I have tried it, and I agree it's a better platform than PHP. And I say this as a professional PHP developer who has been maintaining a 150kloc code base in PHP for the last 10 years. But what it isn't is so great that _every project_ should be using it. It's a good fit to a certain type of project, which I would say is general CRUD-type applications with a relatively simple domain model and limited reporting complexity. Beyond a certain complexity point of the domain model the lazy fetching will start to become a performance issue, and beyond a certain level of complexity in terms of reports you'll want to be using SQL directly, which I understand can be quite tricky.

Finally I would love to hear what you mean by serious applications. Obviously you don't consider Ruby stable enough for Twitter, Basecamp, Yellow Pages or LinkedIn. Or maybe you just don't consider those serious applications?

Basecamp looks interesting, and I'll admit I hadn't come across it before, so I don't really know enough about it to comment. The others are all very simple applications, fitting well within the scope of the restricted application domain I described above.

Re:Should have used PHP.

[digitig]

You know, technically there are more recent Fortran specs. Fortran gets hauled out in the form of FORTRAN77 every time someone wants to talk trash about an old language. However, just like C or C++ or Java, the language has evolved. Fortran 2003 has objects, a pretty cool module system and basic thread support.

So what you're saying is that language features do affect scalability? Well, that was precisely my point.

Re:Should have used PHP.

[dedo_jozef]

Ruby 1.9/future 2.0 has proper threads.

Re:Should have used PHP.

[LaurensVH]

Correct, but when one implementation is obviously dominant (say, CPython and it's GIL), it becomes a language problem for anyone that wants to write real software in it right now.

Re:Should have used PHP.

[Anonymous Coward]

Happs was superseded by Happstack (an active fork)

Re:Should have used PHP.

[K. S. Kyosuke]

Except that the parent nowhere said "Rails", just "Ruby" (three times), which is precisely why I wrote that.

Re:Should have used PHP.

[Anonymous Coward]

Not sure by your comment if you are saying modern languages beat COBOL or the other way around, but it depends on where you are running your FORTRAN or COBOL. Since COBOL is a primary language of a z/OS mainframe, I would say that many COBOL applications will blow away those so called modern languages.

A new instance of a COBOL based application on a mainframe can be spun up as often as is needed by workload manager to handle increased traffic. The IO performance of the mainframe backplane will generally exceed anything on a mid-tier level, so what you end up with is COBOL applications that far out scale mid-tier systems, and generally take up less floor space in the data center.

Re:Should have used PHP.

[SCHecklerX]

Wouldn't know. I still used embedded perl. You crazy kids and your web backend language of the week.

Re:Should have used PHP.

[robbrit]

It's true. This is the same thing that Twitter is doing - keeping Rails for the front-end and using Scala for the back-end. Much of the grunt work for Facebook is not done using PHP, and much of the grunt work for Twitter will not be done with Ruby.

Re:Should have used PHP.

[Yold]

How can you honestly be a fan of scheme? The language is elegantly simple, however implementing anything non-trivial becomes a real pain. Error messages are extremely cryptic, and changes to complicated code often result in mismatched parens.

I'm curious, have you every attempted to develop anything complex with scheme?

Re:Should have used PHP.

[DuckDodgers]

Some degree of refactoring is nice, of course. On the other hand, Scala has operator overloading and higher order functions. That removes some of the common refactoring requirements you have with Java.

The battle between Scala and F# could be interesting. Scala has the strength of Java's widespread use and now the added strength of open source. But (despite Slashdot prevalent opinion) Microsoft still carries a lot of weight, and F# may be the most powerful option on the .NET framework.

I don't like generic type erasure either.

Re:Should have used PHP.

[Foofoobar]

Well researched? it's a N008 in the world of languages! C,C++,Java,Perl,PHP,Python are well researched but Ruby is still getting it's sea legs. It's a good language bt by no means try to imply that it is mature for it is not.

Re:Should have used PHP.

[Foofoobar]

Not really. Objective-C which is getting a resurgence thanks to the iPhone and Mac popularity takes alot from Smalltalk. Smalltalk is also still used in financial institutions and software and there is a resurgence in smalltalk development recently. It may be old school but a 'mature' language never truly dies.

Re:Should have used PHP.

[Foofoobar]

PHP and Cake are slow as hell. Which is why I use PHPulse... 10X faster than almost all other frameworks for PHP.

Re:Should have used PHP.

[Radhruin]

Any RoR developer I've ever talked to was just into it because it was "hip".

That's funny, none of the ones I've talked to chose it for that reason, and I haven't even heard of people choosing it for that reason. Also I think it's fair to guess that I've talked to more Rails developers than you have.

In fact, that'd be difficult for me to say either way, because I have not used it.

Which is obvious considering the following paragraphs...

Really, it doesn't take a lot of code to setup an object that can use a DB in an optimized fashion if you just write that code yourself, and it'll be a lot clearer without all the "magic objects" floating around.

Do you have any idea what ActiveRecord, even does? Firstly, it's only slightly slower than doing everything you need it to do by yourself with optimized SQL statements. If you need that level of optimization, you can do it within AR. You don't even need to use an ORM in Rails if you don't want to, you could just write standard classes that execute SQL directly on the database connection, again, if you wanted to.

Your magic objects assertion is definitely born of not having any experience with what you're talking about. The reason that there are so many ORMs around is that they are extremely helpful, and indeed at least half of the benefit of using something like Rails is a direct result of the ORM. You would know that if you spent any time getting to know it before talking about it.

Re:Should have used PHP.

[godefroi]

Hey, that's what I need. More LIFESTREAM TRANSACTIONS. That oughta fix me.

Re:Should have used PHP.

[kchrist]

Basecamp looks interesting, and I'll admit I hadn't come across it before, so I don't really know enough about it to comment.

As an aside, Basecamp is where Ruby on Rails originated. Rather than being created from scratch, the Rails framework was extracted from a application that was already in use.

Re:Should have used PHP.

[kv9]

That's fine for the vast majority of web applications, but you clearly don't understand the scale of the traffic a site like Twitter receives.

I clearly don't. this type of webapp generally scales well, their problem is that they chose shitty technologies to "power" it. you want fast? code an Apache module that uses libmysql (or whatever DB it is they use). now that's fast.

What do you do when one database machine, no matter how fast, isn't enough?

I believe they can split the DB, add more RAM, cache a lot (a LOT, cache it in RAM, RAM is cheap), serve static content.

When your load balancer gets overloaded?

in this scenario the load balancer is the least of your problems. what happens if a switch dies? drives die, PSUs blow up, etc etc. use redundancy.

How about handling that massive search index?

I don't know about that, let me google it.

Re:Should have used PHP.

[lewp]

Twitter was going to have problems scaling no matter what they chose. Because of the highly personalized nature of the site, the low ratio of reads to writes, and the way the app encourages you to make small updates constantly, most of the strategies you'd use to increase throughput just don't work and you wind up at "how do we make the app just run faster?" Moving to a faster language is a great way to do that, and something they were probably going to do eventually on the back-end no matter what they started with.

Most sites don't have Twitter's problems. We serve millions of Ruby/Rails-backed pages every day, and we're overall relatively happy with our platform of choice. We even use it in a couple Twitter-esque apps that work just fine, but we don't see near the traffic they do. Rails has its problems, and there are a lot of things it could do better, but it's a good choice for a generic "getting things done quickly on the web" language+framework, and I'm certainly in no hurry to go back to Perl or PHP.

Re:Should have used PHP.

[shutdown -p now]

Some degree of refactoring is nice, of course. On the other hand, Scala has operator overloading and higher order functions. That removes some of the common refactoring requirements you have with Java.

Agreed. At any case, we're getting there. It'll just take time - and if Scala would have corporate backing, very short time indeed. Which is why it is interesting to observe the dances around Sun to decide who will pick up the platform next.

The battle between Scala and F# could be interesting. Scala has the strength of Java's widespread use and now the added strength of open source. But (despite Slashdot prevalent opinion) Microsoft still carries a lot of weight, and F# may be the most powerful option on the .NET framework.

Yes, certainly. F# has the advantage of having a better-suited runtime for a functional language (with custom value types, non-erased generics, and tail calls). It also generally has more terse syntax, and sometimes the abstractions themselves make more sense, IMO (give me ML variants with F# extension of adding methods to them over Scala case classes any day). But at the same time, F# is somewhat mired in the "practical conservatism" of ML (remember, some degree of compliance to core ML language is one of F# design goals - so that you can port OCaml code), while Scala is in a family of itself, and doesn't really need to conform to any existing rules, so it has more new stuff.

Anyway, it is already playing out well for both. "Programming in Scala" references F# as one of sources of inspiration in the intro, and then a couple more times in the book itself for feature comparisons. On the other hand, Don Syme, the F# designer, generally mentions Scala whenever someone asks him of languages similar to F#. I don't think we'll see a feature race as close as it was between C# and Java, if only because Scala and F# have more philosophical differences, but there is still some definite competition there for the same target audience, and that is good.

Re:Should have used PHP.

[Dog-Cow]

You are the most stupid poster I have across lately.

Re:Should have used PHP.

[DragonWriter]

They couldn't - they tried, and it wouldn't run their stuff (why exactly this is, I have no idea, but that's what Alex said).

What they said is that it wouldn't run the Ruby gems they use that include native extensions, which is true of JRuby, but even more emphatically true of everything that is not Ruby, since then you can't use any gems whether or not they use native extensions, so certainly isn't, in and of itself, a competitive advantage to Scala.

Re:Should have used PHP.

[grimw]

Oh, I know a lot about ORMs. You just seem a little angry that people don't agree with your views on them. You calling me inexperienced demonstrates this when, in fact, the contrary about me is true.

Also, they put the ability to write your own code in there for a reason. Reasons like I discussed in the GP. I'll bet that between your code and mine, people will pick up and learn my code faster than your convoluted object model ;-).

Peace.

Re:Should have used PHP.

[DragonWriter]

Hell even Twitter devs says Ruby doesn't have 'true' threads and that they are fake threads.

Twitter devs saying it doesn't make it true. The Ruby language has support for threads. In MRI (1.8.x), they are implemented as green threads. In YARV (1.9.x) and Rubinius (the latter of which is not yet a complete implementation) they are implemented as native threads with something similar to Python's Global Interpreter Lock (GIL) so that only one thread running Ruby code can be running at once (though native extensions that do not need access to the Ruby interpreter internals can release the lock). In JRuby, they are implemented as Java (i.e., native in every, or nearly so, JVM) threads, with no GIL.

Re:Should have used PHP.

[The J Kid]

You, sir, win.

Re:Should have used PHP.

[flibuste]

>>Ruby is a language. Languages usually don't have problems with scalability.
Ruby is an interpreted language. Interpreted Languages often have problems with their runtime's scalability.

There, fixed it for ya.

Re:Should have used PHP.

[Foofoobar]

Ah the third grade response. Jeff Foxworthy lives.

Re:Should have used PHP.

[An Onerous Coward]

I did several googles on "X is the worst language ever"

C++, Java: 5 votes
PHP, 4 votes (including your comment)
C, Perl 3 votes
Ruby, Lisp, Python 2 votes
Scala, 0 votes

What does it mean? Probably not much.

Re:Should have used PHP.

[chthonicdaemon]

My comment was not really supposed to contradict yours, just expand on the tangentially related (and totally on-topic of the overall slant of this whole thread) subject of language racism -- some languages are shunned on the technical merits, but some are shunned without even a second glance.

The whole scalability of languages thing is a bit of a semantic argument for me, as you could make a case both ways. On one hand you could say the implementation (compiler, interpreter, whatever) is responsible for handling issues with scaling, but on the other hand certain language constructs make it easier for compiler writers to do this and for programmers to "get" what is happening.

Re:Should have used PHP.

[JAlexoi]

By that standard PHP or python doesn't have threads.

Python does have threads. Just GIL makes them run serially. Read up on Python more, my good man.

Re:Should have used PHP.

[Anonymous Coward]

you have *no* clue what you are talking about. sorry.

Re:Should have used PHP.

[JAlexoi]

Scala being at version 2.7 now, is definitely not less mature than Ruby. Problem with Ruby, that it stagnated for so many years before Ruby on Rails was created.
And the next thing, you probably rate Ruby's maturity by the age of the interpreter. So don't forget that Scala runs on one of them most developed and one of the most mature VM's out there. Java was released in 1995. (Yes, I am implying that JVM is the most mature one of all.)
So, if you look at Scala as a Java superset, it beats the hell out of anything Ruby can throw.

Re:Should have used PHP.

[Anonymous Coward]

Actually, yes and no. It's usually not the language SPECIFICATION that causes scalability problems; but it is rather often IMPLEMENTATION.

And while it is true that with proper architecture you can avoid many problems, there are still many things for which language runtime (VM) has enough effect to make some implementations unattractice to use. For Perl (and I suspect, Ruby) things like difficulty in properly multi-threading tasks or using in-process caches (with soft/weak references) become significant issues from cost perspective. So you can not really get very much out of a single box -- so while you can get same amount of done with 10x machines, it is kinda nice to be able to get more out of individual boxes too.

For Scala I suppose it's the JVM that really matters, not the language. Not sure how mature JRuby is, and what practical limitations are (JVM is not optimal for languages very different from Java, CLR is supposedly much better).

Re:Should have used PHP.

[tcopeland]

> there are still many things for which language runtime (VM)
> has enough effect to make some implementations unattractice to use.

I agree that the Ruby C implementation has these shortcomings... but I still feel that a large app will farm out things to other processes anyway. I'm thinking memcached vs a weak-referenced threadsafe in-process cache, perhaps. If you start with memcached from the get-go you can move it to another machine, add more memory, etc.

Also, writing correct in-process threadsafe code is a tough row to hoe. Unless your name is Doug Lea :-)

Perhaps a question here is also "do the advantages of the language outweigh the implementation shortcomings". In Ruby's case I think the answer is yes.

Re:Should have used PHP.

[digitig]

My comment was not really supposed to contradict yours, just expand on the tangentially related (and totally on-topic of the overall slant of this whole thread) subject of language racism -- some languages are shunned on the technical merits, but some are shunned without even a second glance.

Well, my personal relationship with FORTRAN is that I did my final year university project in it in 1981 (a simulated annealing program), and was relieved that I'd never have to touch it again. I even refused a promotion a few years later because it would have involved maintenance of a major FORTRAN program. So although I have a huge respect for what FORTRAN accomplished in the early days of computing, I have looked at the language all I want to, thank you!

The whole scalability of languages thing is a bit of a semantic argument for me, as you could make a case both ways. On one hand you could say the implementation (compiler, interpreter, whatever) is responsible for handling issues with scaling, but on the other hand certain language constructs make it easier for compiler writers to do this and for programmers to "get" what is happening.

I am convinced that architectural features of languages can make a huge difference. Another example might be ISO Pascal, which had virtually no features that would allow communication between computers, so making distributed systems virtually impossible (of course, people did and possibly still do do that sort of thing in Pascal, but only by using non-standard language extensions).

Re:Should have used PHP.

[tigersha]

Odersky's work on Scala goes back quite a few years, so its not like Scala is a fly by night operation.

He did not call it Scala back then but designing a FPish language on the JVM is something he has been doing for quite a while. Some of the older work they did formed the basis of Java Generics and that was done back in 1997, when Java was only 3 years old.

http://people.epfl.ch/martin.odersky

Ruby was used in Japan for many years before becoming more widely known in the West (at a time the main webpage was in Japanese). Ruby is OLDER than Python and about as old as PHP and Java.

Ruby was stagnating though, version 1.8, the current common one, was released quite a few years ago.

Personally, I have been using it as a Perl/Pythong general shell scripting language since about 2000.
 

Re:Should have used PHP.

[badkarmadayaccount]

Am I the only one who was more baffled by the malformed XHTML, than by the funny phrase?

Re:Should have used PHP.

[Anonymous Coward]

<?

$arr = array(1 => 'one');
$arr[0] = 'zero';

echo array_shift($arr), "\n";

?>

If you expect the above to print "zero", then it's entirely your fault for not understanding the language's array/hash combo structure. While there are many reasons to bash parts of PHP, your laziness and unwillingness to actually learn how the language works is your problem, not the language's.

If you're going to bitch about a language, bitch about something seriously wrong with it, such as crap like this (ignore the backslashes, slashdot screws up whitespace is line starts with forward slash):

<?

\// two strings evaluated numerically!
\// must test with === or strcmp() to be correct.
var_dump('00010' == '0x00A'); // true!

\// these 3 lines are the equivalent of
\// $a == $b, $a == $c, but $b != $c.
var_dump('foo' == true); // true!
var_dump('foo' == 0); // true!
var_dump(true == 0); // must be true? WRONG!

\// seriously, if 1 == true and 0 == false and true != 0,
\// how can 'foo' equate to both true and 0?!?!?!!!!!!

?>

Now I'm depressed. God-awful having to deal with PHP every day at work. To creators/developers of PHP: fuck you. Captcha: "brutal".

Re:Should have used PHP.

[TeXMaster]

the way gcc compiles the big delegator switch in MRI's core, with a large sparse stack that causes ridiculous memory consumption (and sometimes even leaks).

Couldn't the code be written in a way that gcc can compile efficiently then? Seems like a simpler solution.

This is what some of the MBARI patches do. (Of course, ideally you shouldn't need to rewrite your code to have the compiler produce better stuff.) Anyway, the point is (was) that the scalability issues in Ruby are not intrinsic in the language but a consequence of the implementation.

Re:Should have used PHP.

[DuckDodgers]

I am the exact target market for Scala. I'm a Java developer who has been reading about functional programming for years. I've been trying to pick up Haskell in my spare time, and some very neat concepts 'click' instantly but others drive me bonkers.

So Scala is definitely on my list of must-learns. Just as soon as I can write non-trivial Haskell apps.

Stupid

[AKAImBatman]

They should have just used Java. Wait--

Re:Stupid

[modmans2ndcoming]

actually.... Yes.... There is a reason Google uses Java for their API system. Sun put a lot of time and effort to make JavaEE extremely robust, scalable, and reliable.

Re:Stupid

[Anonymous Coward]

Whoosh!

Re:Stupid

[thatjavaguy]

Err... Scala uses the JVM. So the performance in Scala and Java would be roughly the same.

Re:Stupid

[AKAImBatman]

Congratulations on spotting the joke.

Who gives a shit about twitter?

[snowraver1]

Seriously.

Re:Who gives a shit about twitter?

[ajs]

Well... just to name a few:

• Anyone watching which companies are growing during the recession
• Anyone looking at how the latest high-volume services are building infrastructure on the Web
• Neil Gaiman
• Demi Moore
• The President of the United States (though to be fair, his status isn't updated recently
• Al Gore
• John McCain (currently updating from Hong Kong)
• John Battelle, whose insights about the search industry are often enlightening

This, of course, does not make Twitter a panacea, but it certainly makes it interesting enough to warrant the occasional Slashdot article.

Re:Who gives a shit about twitter?

[FredFredrickson]

Seriously. They've got heavy traffic on a single line comments website! It's pretty much the "hello world" app of web applications (compared to facebook, or, heck, anything google does). Why are we discussing it? It's amateur crap at best, and nobody cares about its performance with its fad language, because it doesn't actually accomplish anything. In fact, it accomplishes in its main goal - what is mostly a side feature of any other site or blog - short comments - rather unspectacularly.

Re:Who gives a shit about twitter?

[nothing2seehere]

Wow, Demi Moore? Really? Now that's an influential user!

Re:Who gives a shit about twitter?

[FredFredrickson]

Wow I just realized what an angry old man I must sound like there. I didn't mean to be that harsh. I'm just don't like twitter, and spending time speaking about its technical *wonders* seems to be a waste of time, since I assume it's about 3 lines of code (Or should be), and mimics most highschool coding class students' first projects.

Wow, there I go again. So angry. I just quit caffeine, you must understand!

Re:Who gives a shit about twitter?

[ajs]

Wow, Demi Moore? Really? Now that's an influential user!

Not really, but given that I was trying to give a sense of the breadth of the people interested as users, commentators and misc. interested parties, I think she fits in the list just fine.

Re:Who gives a shit about twitter?

[Dynedain]

Having a marketing team that makes tweets for you is completely different than being a twitter user yourself.

Of the people you listed, how many subscribe to twitter feeds?

Re:Who gives a shit about twitter?

[Jane Q. Public]

The only one in that whole list I am interested in is Neil Gaiman.

Re:Who gives a shit about twitter?

[snowraver1]

Anyone watching which companies are growing during the recession
... Should get their info from a more complete source.
Anyone looking at how the latest high-volume services are building infrastructure on the Web
.. I'll give you that
Neil Gaiman [twitter.com]
Who? (Turns out to be some author) **Update! I just wrote another page!
Demi Moore [twitter.com]
Who cares?
The President of the United States [twitter.com] (though to be fair, his status isn't updated recently
I would prefer something with more substance...
Al Gore [twitter.com]
Who cares?
John McCain [twitter.com] (currently updating from Hong Kong)
Who cares?
John Battelle [battellemedia.com], whose insights about the search industry are often enlightening
You linked to his site (which actually has content) not to twitter.

Twitter is garbage. Our elected officials are dicking around on their smartphones posting gossip to twitter instead of listening. I live in Canada, and our Members of Parliment have been told repeatedly to turn off their phones while in Parliment, but some refuse to do that and waste time and money posting -- quite literally -- gossip to twitter. Fuck Twitter.

Re:Who gives a shit about twitter?

[Jane Q. Public]

Nothing personal intended here, but not only do you sound like an angry old man, but calling Ruby a "fad language" makes you sound like an ignorant, angry old man. Ruby has been around for over 12 years and is still one of the fastest growing languages around.

Re:Who gives a shit about twitter?

[adwarf]

You forgot the obligatory, "get off my lawn."

Re:Who gives a shit about twitter?

[Hijacked Public]

Demi Moore [twitter.com]
Who cares?

An absolute fuckton shitload of people?

Re:Who gives a shit about twitter?

[Serious Callers Only]

since I assume it's about 3 lines of code (Or should be), and mimics most highschool coding class students' first projects.

That'll work for 100 users, let us know how it works out for 10,000, or 8 million.

I can't comment on the technical wonders (or otherwise) of the twitter code-base, but your comment really just illustrates your ignorance on the topic of programming languages and messaging. The problem of dealing with that many users sharing messages is not trivial, and has very little to do with the choice of programming language.

Re:Who gives a shit about twitter?

[Rycross]

What? How can it have been around for 12 years, still be relatively unused and obscure (compare it to C, C++, Java, C#, Python, Perl, and PHP), and yet be one of the fastest growing languages around? Java is close to 14 years old, and is an order of magnitude more used. C# is less than 12 years old, and likewise is far more used. Twitter is the only project that I personally know of that uses it. Am I missing something?

Re:Who gives a shit about twitter?

[modmans2ndcoming]

Object? Bah... I can write perfectly fine object oriented code in COBOL that will run circles around your.... your... coffee language any day.

Damn punk kids.

Re:Who gives a shit about twitter?

[owlnation]

This, of course, does not make Twitter a panacea, but it certainly makes it interesting enough to warrant the occasional Slashdot article.

No, it really doesn't. Demi Moore? Are you kidding? A middle-aged, talentless, has-been, piece of trailer-trash that's only famous for getting her fake tits out in movies. I'd actually stop using Twitter to avoid her.

Plus the fact that it's perfectly possible to keep up with these people (assuming you are that desperate) in many other ways -- all of them more useful and interesting than the occasional 140 character brainfart.

Twitter adds nothing to humanity. Few people of importance are using it, and even they aren't doing anything important with it. It will be gone in less than 5 years. No one will care. The teenagers will have moved on the the next thing.

That said, Twitter is a pretty good example of bad planning, and how not to do things as a developer. So it is indeed interesting to see on /. how they are trying to plug their many holes, and trying desperately to stop the whole thing crashing down regularly.

Re:Who gives a shit about twitter?

[Jane Q. Public]

To quote Radhruin above: "Anyone who thinks Ruby [hulu.com] on [amazon.com] Rails [zvents.com] can't [scribd.com] scale [yellowpages.com] is as dogmatic in their anti-hype as the original hypers were. The right tool for the right job and all that."

And you can add LinkedIn and many others to those. (His original post contained links.) I know you did not claim that it can't scale; those are just examples of some sites that use it.

As for how it can be 12 years old, rapidly growing, but still not as used as some other languages, is that Ruby remained relatively unknown outside Japan for a few years. So it did not become widely known in the U.S. until relatively recently. But it is a mature language and is in wide use indeed.

Re:Who gives a shit about twitter?

[fm6]

But she's not even a full Moore. She's only a Demi Moore!

Re:Who gives a shit about twitter?

[Anonymous Coward]

Posted snowraver1...

to Slashdot...

in a single sentence...

Who indeed.

Re:Who gives a shit about twitter?

[Anonymous Coward]

So to summarize it: everyone with the attention span of a hummingbird on speed gives a shit about twitter, and people who need the sympathy of those retards. Unfortunatly that qualifies for a rather large part of the population.

Re:Who gives a shit about twitter?

[tyrione]

Wow, Demi Moore? Really? Now that's an influential user!

What we want to know is , ``Does she tweet in the nude or not?''

Re:Who gives a shit about twitter?

[Anonymous Coward]

But she's not even a full Moore. She's only a Demi Moore!

Well, the fullest Moore I know is Michael Moore...

I'll stick with Demi Moore(*), then.

(* from around 20 years ago, I mean)

Re:Who gives a shit about twitter?

[PatrickThomson]

I love how obama's twitter had a bot or a staffer automatically following people who followed him, so that anyone he'd initially handpicked didn't get stalked to oblivion!

Re:Who gives a shit about twitter?

[adavies42]

you are ashton kutcher and i claim my five pounds

Re:Who gives a shit about twitter?

[Scroatzilla]

Scroatzilla is thinking about responding to a pithy anti-Twitter Slashdot post.

Re:Who gives a shit about twitter?

[alexo]

> An absolute fuckton shitload of people?

Can you convert it to SI units please?

Re:Who gives a shit about twitter?

[FredFredrickson]

It's not trivial, no. But it requires a language that lets you get to the core of things- not an abstraction on top of an abstraction. You won't get performance out of multiple layers of abstraction.

It really is a very few lines of code, with a very good database backend, and decent caching. RoR is an abstraction to make it easier for beginners (or lazy coders). It doesn't scale because it's an abstraction, and does things in a more convoluted way.

Either way, we're not talking about the technology of the future that keeps millions of airplanes from crashing, or traffic control systems that scale to the whole world. We're talking about a single-threaded (topic, not process), multiuser message board.

Re:Who gives a shit about twitter?

[snowraver1]

Well... Assuming an average mass of 70Kg, about 14.2 people.

Scala seems to be Java+/-

[ajs]

Scala looks and feels like Java with a tiny bit of Python thrown in for good measure. I'm really not certain why anyone would use it over, say, groovy or just plain Java.

The really odd part is trying to imagine Scala as a reasonable replacement for Ruby or any other higher level language.

Re:Scala seems to be Java+/-

[geekoid]

It's new! It's hot! it's what all the kids are doing! It's what crappy programmers can pretend to adapt to instead of writing solid code!

weeee!!!

Re:Scala seems to be Java+/-

[Anonymous Coward]

Scala compiles to Java Bytecode and runs on an unmodified Java Virtual Machine. It can call any Java Library and you can write code in Scala that is callable from Java.

Scala offers much more than simple syntactic sugar. Take a look at their documentation. Especially the nice support for embedded DSLs is VERY interesting.

Re:Scala seems to be Java+/-

[nicolas.kassis]

Or just jruby which has proven itself to be quite able to handle under load. From what I understand they had issues with their Message Queue which could have benefited from the native threads in jruby. But we are all speculating on what really is going on with Twitter.

And what does scala have over say erlang for concurrency and performance?

Re:Scala seems to be Java+/-

[A.K.A_Magnet]

What? Scala is Java mixed with OCaml -- you get an extremly powerful typing system, but it feels like a "dynamic language" such as Python or Ruby. With the performance of Java. Under the hood, it's a brand-new language, very different from all those: it merges functional and object-oriented programming. Yet, for the regular programmer, it feels like Ruby... until he gets used to more powerful features and learns how to designs more complex libraries as embedded DSLs. All that while running on the JVM and thus giving free access to all Java/JVM libs out there with no overhead. The really odd part is that not more people are using Scala as a replacement as most languages -- except C for device level and Erlang for distributed stuff.

Re:Scala seems to be Java+/-

[Anonymous Coward]

So does Jython. So... yeah, whats the point?

Re:Scala seems to be Java+/-

[hedleyroos]

I'm confused - how does a language "scale"? Can you suddenly have 1 billion items in your array instead of 100 million? If that is indeed the case then like the parent said: it is crappy coding and/or design.

I'm fairly confident you can do twitter in whatever language you want and if designed properly it will scale in the proper sense of the word.

Re:Scala seems to be Java+/-

[A.K.A_Magnet]

I read between the lines that you call C or C++ solid-code, and if I'm not mistaken, you will find that the kids are doing Scala because the code is more solid. Scala benefits from a typing system close to OCaml's which makes Scala code very, very solid -- especially if you keep away Java specifics (such as nullable objects) in your code and take special care when interacting with Java libs that may do so.

If I'm mistaken and you're not talking about C/C++, I hope you are not talking about dynamic languages which offer no guarantee whatsoever; you know as a developer I enjoy actually spending my time on working on the business side of my application -- and how to make it scalable, rather than working on low-level specifics and on testing if every pointer is null before dereferencing them. A type system that does this for me (which Scala or ML's parametrized type Option allows) is a bliss.

Now, I'm not enumerate every language under the sun to see what code you call solid, I guess your answer would be that the code is solid whatever the language it's written in. In the end, it all comes down to binary instructions, right? The question is: how many guarantees do the tools give you? In the case of Scala's compiler, it gives you a lot AND offers you a very enjoyable, lightweight yet powerful syntax.

Re:Scala seems to be Java+/-

[Anonymous Coward]

Yeah "more complex", thats what we need.

Re:Scala seems to be Java+/-

[A.K.A_Magnet]

A language scales if the language itself (not the "runtime" or whatever) can grow with user constructs to suit the ever-changing needs. For instance, in Python, if you write
l = [a]

you will end up with a Python built-in list. Squared-brackets are part of the language syntax and will always refer to Python builtin lists.

In Scala, which is meant to be a scalable language (as the name implies: SCAlable LAnguage), if you write
val l = a :: Nil

you will end-up with a Scala list only because the ::' "keyword" (which in fact isn't) is in the scope, but you could specify another class if you'd like. Use another implementation, yours if you'd like.

All of Scala is made this way and that's why the *language itself* is scalable.

Now, the article talks about scalable architectures, not scalable languages -- they use Scala because they like it :). Yet since Scala compiles to Java bytecode, it offers all the JVM's scability features (clustering with terracotta, ...) and it's a great language to write scalable software in :).

Re:Scala seems to be Java+/-

[AKAImBatman]

I'm confused - how does a language "scale"?

"Scalability" is a multi-faceted term. Most people think of "vertical scaling of the servers" when they hear the term "scalable". Which is to say that the code can handle a higher transaction load on a beefier server.

But there is quite a bit more to scalability than that. There's horizontal scalability of the servers. i.e. Does the software support plugging more boxes in to handle a greater load? Then there's the development scalability. i.e. Are the concerns of the code separated well enough to where multiple development teams can work in parallel without stepping on each other's toes or creating conflicts over code modules? Finally there's organizational scalability. i.e. Is this system designed to allow the organization to effectively split back-end responsibilities as the application grows. (An example of this would be scaling to meet the demands of a large sales force or distributed fulfillment process.)

The language you chose can have an impact on these larger concerns. Various features of the language can support aspects of these needs more naturally than other languages. But at the end of the day, it's still up to the architect to design a system that meets the needs of organization. The language is merely a small cog in the much larger machination of solving the challenges that face the business.

Re:Scala seems to be Java+/-

[halll7]

Something that gets attention from Java developers is Scala's implementation of the Actor concurrency model, which is a fundamental alternative to the shared-state locking that Java provides. Erlang has offered this for many years, but Scala offers it with a more familiar syntax, and is likely to get a more enthusiastic uptake for that reason alone.

As well as this Scala is functional (first-class functions, currying etc) as well as OO, and Scala programs can use the standard Java libraries if they wish, which again makes the transition easier for Java developers.

Re:Scala seems to be Java+/-

[nuttycom]

The combination of static typing, type inference and concise syntax for higher-order functions give you a lot that Groovy and Java just fall short on.

In Java's case, it's the simple fact that you can't reasonably implement HOF's like filter, map, etc. without using anonymous inner classes. If you use anonymous inner classes, you end up with a ratio of like 8:1 of boilerplate:relvant expressions. Believe me, I've tried it. If you want to see just how baroque it gets, take a look at functionaljava.org.

Groovy has HOF, but it's interpreted and it doesn't have real static types. So it's slower and you don't have nearly as much ability to do compile-time type checking.

Scala rocks. I've been using it for several months now building a $50M/year transaction processing system and I never want to go back to writing regular old Java again.

Re:Scala seems to be Java+/-

[Jane Q. Public]

Other languages like Ruby and Python also support DSLs just fine. That isn't the reason Twitter wanted Scala.

Re:Scala seems to be Java+/-

[Anonymous Coward]

It's new! It's hot! it's what all the kids are doing! It's what crappy programmers can pretend to adapt to instead of writing solid code!

And it's what will get those crappy programmers hired into higher-paying, better-connected, better-located, more trend-setting jobs than we programmers who actually know what we're doing could ever hope to find... *grumble*

Re:Scala seems to be Java+/-

[Bruce Perens]

This is the same for Ruby. You can create a list, or you can create some object you wrote in hand-optimized C which just responds to the same operations as a list.

Re:Scala seems to be Java+/-

[Mike Buddha]

The really odd part is trying to imagine Scala as a reasonable replacement for Ruby or any other higher level language.

Particularly with options like Stackless Python available. Python is well supported. Why throw out the baby with the bathwater?

Re:Scala seems to be Java+/-

[Anonymous Coward]

You can get Scala by typing
`from __future__ import braces`
in Python prompt.

Re:Scala seems to be Java+/-

[serviscope_minor]

If I'm mistaken and you're not talking about C/C++, I hope you are not talking about dynamic languages which offer no guarantee whatsoever; you know as a developer I enjoy actually spending my time on working on the business side of my application -- and how to make it scalable, rather than working on low-level specifics and on testing if every pointer is null before dereferencing them. A type system that does this for me (which Scala or ML's parametrized type Option allows) is a bliss.

Sounds like you're doing it (programming C++) wrong. Calling it C/C++ is a dead giveaway as it implies you still believe it's one language. You're probably progamming like C++ is C with funky bits added on. It used to be, years ago. I recommend you buy a recent book on C++. If you're not familiar with modern C++ idioms it will open your eyes, especially if you're working with raw pointers and new/delete or if your code looks mostly like C.

Or, possibly, you're not sticking to Knuth's maxim about premature optimization. C and C++ have so many ways to optimize code it's hard to resist the temptation.

But sure, the type system of C++ isn't as nice as ML, unless ML doesn't have types parameterized on ints like C++. I can't remember if it does or not.

Re:Scala seems to be Java+/-

[A.K.A_Magnet]

I know. I didn't imply no other languages allowed this (especially for "high level" data structures such as list). I was just replying to gp that indeed, the article is about scalable infrastructures/systems rather scalable languages.

On the other hand Scala goes quite far in that approach; Ruby and Scala are also often compared for the ability to define embedded DSLs. In Ruby it is handled with meta-programming whereas Scala has made the choice of offering language-level DSLs (that make use of Scala syntax smartly), thus offering the typing system for the definition and usage of DSLs. People coming from Ruby especially enjoy a strong typing system that helps them and doesn't get in their ways.

Here's something I read tonight on IRC, and I read it quite often :)...
<against_logging> i learned ruby for nothing, i found scala

Now, Ruby is nice -- but just like a lot of people didn't care much before Rails came, right now a lot of people are dismissing Scala because "we don't need another language". Scala is not just another language, really. It finally provides an "enterprise-friendly" functional language: enterprise-friendly, because a lot of companies are sold with Java and the JVM, it's easy to let Scala in. But it gives the power of ML and functional languages, all the libraries written for Java, and the lightweight syntax of Ruby or Python (with a much better performance).

Finally, Scala matters because this is the Free alternative to F#, which is basically Ocaml for .NET. OCaml is Free (even if the policy is a bit restrictive with contributions), but it isn't part of a larger platform such as the JVM or the CLR. In the upcoming years, functional languages will get a boost because they handle multithreading better (they make it more practical to use immutable data) and because finally people understand that a typing system makes complex programs easier to re-use and maintain. Scala is very much the contender for the Free Software community -- and it's not only more innovative than F# (which is very similar to Ocaml, which is awesome but has been around for a while), but also in my opinion offers developers from the world of Java & dynamic languages with a stairway to functional programming: they can learn step by step.

Re:Scala seems to be Java+/-

[sams67]

"Scala looks and feels like Java with a tiny bit of Python thrown in for good measure." Then you haven't used it properly. Scala is fundamentally a functional programming language with OO. You can also work in procedural manner , in which case, yes, it is much the same as Java .. but then you are missing out on some really nice features.

Re:Scala seems to be Java+/-

[A.K.A_Magnet]

In fact I used to program a lot in C++ and I loved it. I was very much in Bjarne's school of C++ and programming it just like He said (even among ##C++ freenode helpers/idlers at a point); definitely not programming it like C at all (I still love C). But my points are still valid. Sure, the const modifier in C++ is great to get the typing system to prevent writing on some refs. Sure, the typing system isn't bad for an imperative language. But it is still light years behind ML & fp, and that is something Scala does very nicely and it keeps the object oriented paradigm.

One of the things I love most in fp (and Scala does it right) is that "everything is an expression that returns a value". For example, an "if" is not a statement in ML or Scala, it's an expression and you are guaranteed that both branches will return the same type, so you can write: val x = if (...) y else z

It doesn't look like much but when you get used to it and what you can make of it, it's something that I miss in imperative languages. I often think of Scala as C++ done right (note: these are EXTREMELY different languages, I'm talking about the feeling): not too much low-level details to care about (for instance, Java's "all objects in the heap, only refs/primitives on the stack", and always have a gc, simplified a LOT programming compared to C++'s static & automatic objects on the stack vs. objects in the heap). But unlike Java (and very much like C++), Scala is fun to program in and gives the developer a feeling of freedom and power.

Anyway I'm not expecting I will convince a C++ programmer over a post -- but I convinced myself that C++ is fundamentally flawed. And whether you trust me or not, I guarantee you that I know C++ very well. A programming language is all about giving you the right tools and abstractions. Quoting Saint-Exupéry, "Perfection is reached not where there is nothing left to add, but when there is nothing left to remove.". In that regard, Scala is a lot closer to it than C++. You should give it a try :).

PS: My code has never been evil :).

Re:Scala seems to be Java+/-

[Trepidity]

I tend to think of D as C++ done right, although I like Scala more, for mostly different reasons. I agree that C++ is fairly fundamentally flawed, mostly because the things that were bolted onto C and then independently developed interact very badly. There are all sorts of edge cases and special-case rules for typename resolution, implicit conversions, method dispatch, how that all interacts with templates, etc.

One of the better explications of specific issues, IMO, is the C++ FQA, which point-by-point dismantles the C++ FAQ Lite.

Re:Scala seems to be Java+/-

[DragonWriter]

Scala is not just another language, really.

Yes it is, really.

It finally provides an "enterprise-friendly" functional language:

In what way is Erlang not an "'enterprise-friendly' functional language"?

But it gives the power of ML and functional languages, all the libraries written for Java, and the lightweight syntax of Ruby or Python (with a much better performance).

I won't comment on the first part, but it certainly doesn't have "the lightweight syntax of Ruby or Python". It does, OTOH, have a somewhat lighter syntax than Java or C++.

Finally, Scala matters because this is the Free alternative to F#, which is basically Ocaml for .NET.

Certainly, its a Free alternative to F#, but its hardly the free alternative, as there are others. Scala's probably, of those targetting .NET or the JVM, the one getting the most attention at the moment, though.

Re:Scala seems to be Java+/-

[zuperduperman]

Others have said it all, but I will throw in my lot too: groovy has a number of problems that weigh very heavily on productivity: the first and most important is that it's a dynamic language taken to an extreme. This means that the compiler never knows the type of any object and lets just about anything pass without complaining. For example, the following compiles fine but explodes at runtime:

  int i = "cat"

You see, groovy is *soooo* groovy, that someone just might have modified things under the hood at runtime so that the = operator *will* work to assign a string to an int. So the compiler really can only verify that the most rudimentary syntax of the language is right and nothing else. If you start developing a large code base with this almost non-existent compiler checking of types, things start to fall apart quickly. You don't see this much at the start because most groovy is written on top of strongly typed java libraries which do have the benefit of strict typing. Once you start developing large groovy code bases in their own right it starts to hurt. The other problems are very poor error reporting (inscrutable 3000 line stack traces when exceptions occur) and poor IDE support (which I suspect partly flows from the loose nature of the language).

Scala solves all these problems. Thus I'm currently considering dropping groovy for Scala.

Re:Scala seems to be Java+/-

[Fnkmaster]

No, it's an ML dialect for the JVM, like F# is in the .NET universe. ML/OCaml and similar languages have been around for years and years, so it's not just a new fad.

Scala just compiles down to Java bytecode which means it can interoperate with the huge quantity of existing Java libraries with basically zero incremental effort, and allow the benefits of writing code in a modern ML-like language (functional expressiveness, static typing, etc.).

Re:Scala seems to be Java+/-

[DragonWriter]

The really odd part is trying to imagine Scala as a reasonable replacement for Ruby or any other higher level language.

For something Ruby is really ideal for, like, say, the stuff Twitter isn't replacing, its maybe not. For implementing message queues, one of the things Twitter is replacing, Scala isn't exactly a bad choice. (If someone put a gun to my head and told me to implement a message queuing system from scratch where performance, development time, and scalability were major concerns, I'd probably choose Erlang personally, but Scala seems like a not-unreasonable choice -- quicker to develop in than Java or C++, better performance than most dynamic languages, and no obvious barriers in the particular problem domain.)

Re:Scala seems to be Java+/-

[steelfood]

Typesafety is good for development ease--extensibility, ease of maintenance, and generally readable code, not scalability. Scalability more depends on a forward-looking design and implementation. These two concepts are orthogonal. For example, forseeing that you might have more than 64K users and using a 32-bit int to store your user ID's instead of an unsigned short. A good library is a plus for scalability, but not required, since it's possible to develop your own massively-scalable library. Thus, even something coded in asm can be scalable. As I've stated before, it depends on design and implementation.

Unless what you mean by scalability has nothing to do with mine.

Re:Scala seems to be Java+/-

[ClassMyAss]

From Bill Venners, apparently the person that interviewed them about this (pasted from here):

We're finalizing a bit of text that we're going to add to the interview about JRuby. It was an oversight on my part to not do this initially. You'll see the official answer once we get the final form of the text approved by the Twitter guys, but the gist is that they did indeed give JRuby a try, and the main problem they had that disqualified it was that their Ruby app used a lot of libraries with C extensions that weren't available for the JVM. So it was not a simple matter of taking their Ruby app and running it on the JVM with JRuby. When they tried that, it didn't run.

(Now quoting parent):

And what does scala have over say erlang for concurrency and performance?

I'd say that maybe it's that Alex has written a book on Scala, whereas they don't have someone on staff that's written a book on Erlang?

Use what you know, as long as it can get the job done...

Re:Scala seems to be Java+/-

[Raenex]

I tend to think of D as C++ done right, although I like Scala more, for mostly different reasons.

Now you've peaked my curiosity. Why do you prefer Scala over D?

Re:Scala seems to be Java+/-

[Trepidity]

Some of it is probably just personal preference and how I file away the languages in my head--- D in my mind is "C++ done right", but aesthetically still feels like a nicer C++, so I think of using it in situations where I might otherwise use C++. I think of Scala more as a mixed-paradigm OO/functional language on the JVM, rather than an upgrade to Java. In particular, it's got lightweight anonymous function syntax and first-class symbols with Lisp-style single-quote syntax, which come in handy in some kinds of programming.

I wouldn't say I'm an expert on their pros and cons though, especially as concerns D.

Re:Scala seems to be Java+/-

[Anonymous Coward]

Mr-Must-Contradict much? Typical Slashdot arrogant reply Mr. Wikipedia. Where in his post does grandparent say that typesafety is good for scalability? Solid is more than just scalable: it means low bugs, proper behavior, reusable and maintainable. A good type-system does make the code more solid, since some properties are implicitly proven for the code if it compiles.

Re:Scala seems to be Java+/-

[Anonymous Coward]

OCaml is another functional programming language with OO. And without curly braces.

Proving that..

[hexghost]

Twitter's developers care more about being cool and hip and using the latest tool so that they remain popular, than they do about having a site that stays up 7 days a week.

Re:Proving that..

[AKAImBatman]

Twitter's developers care more about being cool and hip

Not to be too pedantic, but doesn't that sum up Twitter as a whole?

Re:Proving that..

[joe_bruin]

Twitter's developers care more about being cool and hip and using the latest tool so that they remain popular, than they do about having a site that stays up 7 days a week.

Exactly. Scalability problems arise from poor implementation, not from language choices. Scalable platforms have been implemented in the past with PHP, ASP, Perl, C, Java, and I'm sure with Ruby, Python, or your favorite new language. Twitter is a massive-scale site, they should be looking at deep engineering, not a buzzword platform that promises easy scalability for dummies.

Scala may help them alleviate problems they've hit in the Rails framework. What will help them with the problems they hit in Scala?

Re:Proving that..

[FredFredrickson]

Yeah, I mean, don't let them get too caught up with features or anything. Maybe some day, they'll add some features, but right now, they're just selling cool.

Re:Proving that..

[go_epsilon_go]

Amen to that. Instead on focusing on the latest and greatest fad, new developers should try to do the following: Learn the craft, not only the tools. A little of algorithmic complexity and data structures won't kill you. Throw a little math for good measure. Abstraction and modularization is the key for good architectures, it does not matter if you prefer procedural, functional or object oriented techniques. Just my 2 cents.

Re:Proving that..

[Algan]

What will help them with the problems they hit in Scala?

Erlang :)

Seriously speaking, when you cannot have long lived processes due to memory leaks, you're having a language/platform problem. When you only have green threads, you're having a language/platform problem. Can you architect around it? Probably, but it may be sub-optimal, and bottom line is, why bother when there are better tools available?

Re:Proving that..

[burris]

According to the rebuttals in the comments of the blog post in one of my sibling posts here, part of Twitter's scalability problem was poor implementation of the Ruby interpreter. Lots of small objects cause the heap to get fragmented and eventually it runs out of memory. Java interpreters have better GC and you can swap out different GC algorithms in some of them.

Why does everyone assume the people at Twitter are a bunch of newbies who don't know about deep engineering? Is it just because their analysis didn't lead them to your preferred buzzword?

Re:Proving that..

[Anonymous Coward]

Remember, twitter is not saying ruby and rails aren't good enough for their web interfaces. What they had a problem with was specifically their queuing system in pure ruby.

Rails has been plenty powerful and scalable for them (apparently).

Re:Proving that..

[Anonymous Coward]

"What will help them with the problems they hit in Scala?"

They'll go to Java, which has proven abilities in a large scale environment. Hence the benefits of scala (Java compatibility).

Re:Proving that..

[Anonymous Coward]

no wrong, it is a language problem. If a language does not offer true concurrency then it will have issues with more load. It will also have issues once you start to run it on an 8, 16, 32 core CPU. Languages like PHP, Ruby are going to have to adapt to this or suffer.

The fact the languages like Scala and Erlang are popular now is for a reason. Perhaps you should familiarize your self with such concepts as process isolation and message based concurrency and why for example a server written in erlang will still be taking requests at 80,000+ concurrent users and Apache wont.

Re:Proving that..

[Thomas Mertes]

Seed7 :-)

Sorry, couldn't resist...

Greetings Thomas Mertes

Seed7 Homepage: http://seed7.sourceforge.net/
Seed7 - The extensible programming language: User defined statements
and operators, abstract data types, templates without special
syntax, OO with interfaces and multiple dispatch, statically typed,
interpreted or compiled, portable, runs under linux/unix/windows.

Re:Proving that..

[jknoe]

An issue with Ruby is the maturity of all the dependencies required to run Twitter. Many of them have not been tested at scale and some of them are a continuous drain on engineering resources. Things are getting better, but it isn't always wonderful.

In the JVM there's often a wider choice of third party packages for any given dependency. It's a more mature stack. Writing in Scala gives you access to an enormous repository of high quality JVM code.

That being said, Java really isn't used in much web development at Internet scale, and some critical pieces are missing. Memcache client libraries are wanting, for example. Thus, out pops Smile, a memcache client written in Scala.

Re:Proving that..

[DragonWriter]

Seriously speaking, when you cannot have long lived processes due to memory leaks, you're having a language/platform problem. When you only have green threads, you're having a language/platform problem.

The first is an implementation (or "platform") problem, not a language problem, unless the language requires memory leaks, which would be unusual. The second could be either, I suppose (though again, a language is more likely to determine thread semantics, not whether they are implemented as native threads on the underlying platform), but since Ruby as a language has threads, and only in certain implementations are they only green threads, its also, in this case, a platform, not a language problem.

Neither of these problems, particularly, exists with JRuby, which is also the best performing Ruby 1.8.x implementation* overall. Now, JRuby won't run the gems Twitter was using that use native extensions in C, but then neither will Scala (or Erlang, for that matter), obviously.

Useful!

[castorvx]

Twitter using new-and-fancy programming languages has a way of load testing them for all of us.

I'm not sure I'd have the balls to take a 5 year old development platform/framework and drop it into something that sees so much traffic. Hopefully they share their experiences in some form.

Re:Useful!

[Captain Spam]

Hopefully they share their experiences in some form.

If downtime and lag each count as "some form", I think we'll have us plenty of data to observe!

Re:Useful!

[Anonymous Coward]

If there's anything we learned about these guys so far it's that they're really good at making really poor technology choices.

Whoever buys Twitter is going to throw _all_ their code away.

Good thinking,

[geekoid]

replace one language that wasn't tested on that scale and replace it with another one that wasn't tested on that scale.

Good thinking~

Oh look, twitter is down..again.

Re:Good thinking,

[AKAImBatman]

As the joke I posted above insinuates, Scala runs on top of the Java platform. And unlike Ruby, it focuses on the use of the platform's features. So the platform is more than tested enough. Why they feel the need to use Scala rather than straight-up Java is one of life's great mysteries. But for now, their platform should be fine.

Whether the code they write is scalable and holds up under loads or not is an entirely different topic.

Re:Good thinking,

[The Slashdolt]

Read this and all will become clear:
Event-Based Programming without Inversion of Control

Re:Good thinking,

[Anonymous Coward]

Summary: Blah, blah, blah, nothing to do with the subject at hand, blah, blah, blah.

Re:Good thinking,

[burris]

Maybe they use Scala because writing Java code is painful by comparison. Tons of boilerplate, every exception has to be caught in every scope, no pattern matching, no named arguments, and on and on. For people like me, without Scala the JVM wouldn't even be under consideration, though I admit that Java has been more usable since it got generics.

Re:Good thinking,

[The Slashdolt]

From the Article:
Scala is different from other concurrent languages in that it contains no language support for concurrency beyond the standard thread model offered by the host environment. Instead of specialized language constructs we rely on Scala's general abstraction capabilities to define higher-level concurrency models. In such a way, we were able to dene all essential operations of Erlang's actor-based process model in the Scala library.

However, since Scala is implemented on the Java VM, we inherited some of the deciencies of the host environment when it comes to concurrency, namely low maximum number of threads and high context-switch overhead. In this paper we have shown how to turn this weakness into a strength. By dening a new event-based model for actors, we could increase dramatically their efficiency and scalability. At the same time, we kept to a large extent the programming model of thread-based actors, which would not have been possible if we had switched to a traditional event-based architecture, because the latter causes an inversion of control.

Re:Good thinking,

[Anonymous Coward]

Whether the code they write is scalable and holds up under loads or not is an entirely different topic.

Duh. Of course it's Scalable. It says so, right there in the title!

Thank you, I'll be here all week, posting on Slashdot because I don't have a job!

Re:Good thinking,

[Anonymous Coward]

I've seem some Scala benchmarks that showed it to be faster than even java in some occasions.

One things is sure, It does not have the overhead of Jruby or groovy.

Re:Good thinking,

[Aloisius]

And why do you need more than process-level concurrency (or currency at all?) for a system as simple as Twitter's? They aren't exactly doing heavy calculations.

Re:Good thinking,

[kipton]

For people like me, without Scala the JVM wouldn't even be under consideration, though I admit that Java has been more usable since it got generics.

The creator of Scala, Martin Odersky, was instrumental in the design and development of Java's generics. I believe he is also responsible for (some version of) Sun's javac implementation.

For a reference, see
http://www.artima.com/forums/flat.jsp?forum=276&thread=221233

In my opinion, Scala is excellent. So far I've used Scala to generate code which leverages NVIDIA GPU's for computational physics. I've also used Scala to do simple combinatorics and text processing. The latter is particularly pleasant -- a parser-combinator framework is included in the standard library which makes it a snap to define and process arbritrary grammars. My biggest frustration with Scala has been the somewhat spotty IDE support (compared to Java). However, the situation is improving and decent plugins for Eclipse, Netbeans and IDEA are all on the horizon.

Re:Good thinking,

[chez69]

well, sometimes you want more then one message consumer per process.

Re:Good thinking,

[master_p]

Lot's of acronyms and buzzwords in order to explain the simple concept of passing callbacks to functions. I feel sorry for the IT industry that has to reinvent itself every few years. On the other hand, it keeps the economy going :-).

Re:Good thinking,

[Anonymous Coward]

sigh.. ffs.

Every exception doesn't have to be caught at every level. This is not true. It has never been true. It will never be true.

There are checked exceptions you are forced to catch. People who aren't stupid wrap them in a runtime exception and then it's happy fun time for all.

Java's 'thing' is having confidence in your code. So checked exceptions are about making sure people catch things they definitely need to catch. It's up to APIs to use them correctly and not abuse them. Some do, some don't.

Re:Good thinking,

[The Slashdolt]

They're using this for their message passing, not their web stuff. For example, they accept tweets from sms, internet apps, etc. This system is used to handle all of those disparate messages in a performant manner. It's not about the calculation it's about concurrent message handling. Millions of people sending messages to them simultaneously.

Re:Good thinking,

[HAWAT.THUFIR]

Tons of boilerplate

"boilerplate" is generally taken care of in the ide.

Which sidesteps the fact that Java has too much boilerplater, particularly in comparison to C#

Re:Good thinking,

[TwinkieStix]

In Java, we have the built in RuntimeException library that can re-throw any other Exception is completely implicit. We have the built in java.util.regex pattern matching library, Map can be used to pass named arguments, varargs are available to take arbitrary parameters to a function, and very little boilerplate when frameworks like Spring are used in conjunction with annotations.

That pretty much covers any complaint you might have had I guess, and these features have been around for the better part of a decade (except varargs which are a few years old).

Re:Good thinking,

[burris]

I'm not talking about regexps, I'm talking about object pattern matching in switch/case statements.

Obligatory...

[dex.pdx]

Perl would be better.

p.s. I don't want to here any of this Perl is ugly crap either - it's not my fault I don't need my language to -make- me write readable code. A good Perl programmers code is readable anyways.

Re:Obligatory...

[ttfkam]

p.s. I don't want to here any of this Perl is ugly crap either

How can we trust you with Perl when we can't even trust you with English?

Re:Obligatory...

[Anonymous Coward]

Too bad there are no good perl programmers~

Re:Obligatory...

[glenstar]

Give the guy a break! His sentence is perfectly valid Perl... although it would be much faster if he used the Pussiferous version of the crap keyword (cr*p).

Scala? Not for me!

[bogaboga]

Scala is not for me I can confidently say. I am too old to learn a new [programming] language. The languages I know will suffice for now.

There is a saying too: "You cannot teach old dogs new tricks."

Re:Scala? Not for me!

[Anonymous Coward]

You know, at first I thought this post was off-topic, because it's an incredibly pointless life update that none of us could possibly care about.

Then I realized that the story is about Twitter, and suddenly I think it's the most relevant post so far.

Re:Scala? Not for me!

[nothing2seehere]

I am too old to learn a new [programming] language. The languages I know will suffice for now.

Understood - and incidentally, I would like fries with that, Grandpa.

Re:Scala? Not for me!

[proton]

There is a saying too: "You cannot teach old dogs new tricks."

And in the Mythbuster way, thats totally busted! (BOOM)

As for the language, use whatever floats your boat. It could be total crap and still work (like PHP for me).

Re:Scala? Not for me!

[Anonymous Coward]

It's OK, Scala has been around since the 80's. They just released Scala-5 last year:

http://en.wikipedia.org/wiki/Scala,_Inc

Re:Scala? Not for me!

[shutdown -p now]

There is a saying too: "You cannot teach old dogs new tricks."

There's another saying, though - I hear it's a Korean one - that says "When the dog grows old, you can always cook it".

Worth keeping that in mind.

Re:Scala? Not for me!

[JAlexoi]

Literally your own personal problem that you have not been excersizing your brain enough.
I have a colleague who is 46 y/o and he just finished learining Ruby, Ruby on Rails and ObjectiveC. Just for the fun of it.
He is now very comfortable in developing in most programming languages known to man kind :)

Aw, Java and Python had a baby!

[gpig]

Isn't she cute :)

Re:Aw, Java and Python had a baby!

[Anonymous Coward]

Hopefully she isn't as slow as one of her parents. /cough Java.

Re:Aw, Java and Python had a baby!

[Anonymous Coward]

Scala is what you'd get if you did

from __future__ import braces

in Python.

Re:Aw, Java and Python had a baby!

[Anonymous Coward]

Isn't she cute :)

I thought that already happened with jython

Re:Aw, Java and Python had a baby!

[Anonymous Coward]

Yet more proof that if you only know Python, everything looks like Python.

Really, there is nothing much of Python in Scala at all. Its influences are entirely from the academic functional-programming tradition, and specifically the ML family of languages.

That's why it includes a bunch of neat features that Python doesn't have, like compile-time typechecking, proper anonymous functions, pattern matching, etc.

Re:Aw, Java and Python had a baby!

[bnenning]

Yet more proof that if you only know Python, everything looks like Python.

I think it's more that Python doesn't "look like" anything, in that there's very little excess syntax. All languages that share that property will look similar to some extent.

Re:Aw, Java and Python had a baby!

[binarylarry]

So let me get this straight.

You're talking about Java and Python... and you're saying *Java* is slow?

If so, you are hilariously misinformed about platform/language performance these days. And well forever, since python is one of the slowest languages/platforms available today.

Not that python is bad, it's fine for what it does. It's just extremely slow.

Re:Aw, Java and Python had a baby!

[Anonymous Coward]

Python? fuck no!
Java and OCaml had a baby!

Re:Aw, Java and Python had a baby!

[ClassMyAss]

Welcome to 1998.

In the real world circa 2009, I routinely port C++ code to Java, where it runs almost as fast (usually within 20% or so) as the C++, sometimes even a bit faster if the algorithms are not very cache-friendly.

The JVM is very fast, especially if you're using it in a production environment with the right flags set.

Scala is great

[burris]

If I want to use any Java software then I'll use Scala. I see people bashing Scala, saying the languages they know are good enough or they can just use jython/jruby/groovy, but they clearly know little about Scala.

One thing that's nice about Scala that Java, Jython, JRuby, and Groovy all lack is it's powerful type system and pattern matching. Once you get used to good pattern matching like in Scala, SML, OCaml, or Haskell you won't want to go back. Plus you get all the benefits of running on the JVM at high speed (unlike all the aforementioned JVM languages, except Java itself.)

Honestly, you should check out Scala before you bash it. It's a very good choice wherever you might choose Java, which is a good choice for the back end. Twitter's developers are smart and experienced. They didn't choose Scala just to be cool. It is a powerful tool that can get the job done in an elegant way.

Re:Scala is great

[AbbyNormal]

Why wouldn't Groovy be a suitable replacement? (Curious about Scala).

Re:Scala is great

[burris]

Groovy is very slow. Maybe that will get better in Java 7. There are probably other meaningful differences that I would be able to point out if I were better versed in Groovy.

Re:Scala is great

[Anonymous Coward]

Why wouldn't Groovy be a suitable replacement? (Curious about Scala).

Groovy is dynamically typed and slow as python or ruby.

Re:Scala is great

[Anonymous Coward]

Groovy typing is optional. You can use strong typing same as Java.

def num = 1;

num = "aaa" //ok

Integer num2 = 2;

num2 = "aaa" //compiler error

Scala inferred typing it brilliant though. It saves me a lot of time when writing tests and gives me confidence in general when programming.

Re:Scala is great

[listen]

Maybe because its a slow, me-too, mashed together dynamic language rather than a cohesive, statically typed language, designed in a peer reviewed environment with modularity, performance, safety, concurrency, and expressivity in mind?

Re:Scala is great

[Anonymous Coward]

running on the JVM at high speed

As someone who has yet to see the JVM beat me in a foot race, let alone run me over, I call your bluff.

Re:Scala is great

[nadavwr]

Right, let's start with the perceived problem: Java's syntax. What's the problem with Java's syntax? have a look at Steve Yegge's "Execution in the Kingdom of Nouns". While I can't wholeheartedly agree with many of the points, the gist of it is right on spot -- in Java there's a strong emphasis on nouns and very little is left for the verbs, but verbs are important for overall legibility. Having good (conceptual) support for verbs means that your Runnables and Callables are implied and the code becomes more fluent.

"Good support for verbs" translates to computerese as "functional programming". Ruby and Groovy owe much of their power to features associated with functional programming, such as lambda expressions.

Enter Scala: Scala achieves a brevity of syntax you see in Groovy and Ruby without sacrificing the static type system (it uses type inference). Since Groovy and Ruby are dynamically typed, they are inherently slower than Scala. Scala's performance characteristics on the JVM are similar to those of Java itself.

Without starting a flame war, let's accept that some jobs are better fitted for statically typed languages (e.g. a rapidly evolving project that is prone to regressions). The nieche Scala fits into is actually a testament that developers want the power and benefits of Ruby's syntax, in situations where Ruby (and Groovy, Ioke, BeanShell, etc...) *will* *never* *ever* *be* *used* (e.g. when maintaining performance and type checking are absolute requirements). Scala does its best to do just that.

A JVM language occupying a similar nieche is Fan, with a growing community and tool support coming right up.

Oddly enough, JavaFX (the language) has a syntax that is somewhat similar to Scala, and also some of the language features lacking in Java.

btw: the Scala wiki hosts a piece of sample code by Warren Henning. Never mind the sample, the main comment reads: "THIS. IS. SCALAAAA!!!"

Re:Scala is great

[Anonymous Coward]

> Twitter's developers are smart and experienced

I totally agree wrt Scala, but that's going too far. I went to a tech talk at Twitter where they were supposed to discuss their experience with Scala. Instead, they changed to subject to ask us whether we liked to format our code with a 2 spaces, 4 spaces, or 8 spaces. Oh my god! When others commented that the Scala coding standard stated the answer, the developers stated that it wasn't their style and they thought that they should write their own style guide. It wasn't that Scala's style is bad, just insanely idiotic NIH.

After many requests, they refused to actually talk about what we had actually come out to listen to. Instead we got a really crappy presentation on other non-sense and shown alpha quality code for a memcached client (since Java's wasn't cool enough). When asked whether they used Spring or a DI framework, Alex stated that he didn't understand what DI meant (had to be explained) and that he saw no value in it. His coworker stated that their code was so pretty that DI wasn't needed, but only necessary due to Java's verbosity. Total freakin morons.

Alex announced his book and admitted that the name was meant to confuse readers and redirect purchases of Martin's awesome text. So even worse then the garbage talk, he was frank about having a lack of integrity.

So no way in hell would I call those Twitter guys good. Perhaps before that "tech talk", but having met them.. no.

Re:Scala is great

[Anonymous Coward]

Is that comment supposed to be sarcastic? The reverse of that comment would be more accurate.

Re:Scala is great

[Anonymous Coward]

I met alex at a conference last year. Well, in the bathroom. He was tweeting on his blackberry or some shit while I waited to take a piss. After a couple minutes I bashed his head into the wall and pissed on him as he lay on the floor groaning. Tweet that cocksucker.

Re:Scala is great

[DragonWriter]

Enter Scala: Scala achieves a brevity of syntax you see in Groovy and Ruby without sacrificing the static type system (it uses type inference).

While, certainly, Scala code is less verbose than Java, it is noticeably more verbose, in most cases, than Python, Ruby, or Groovy.

Since Groovy and Ruby are dynamically typed, they are inherently slower than Scala.

Groovy, as I understand, supports but does not require static typing.

The nieche Scala fits into is actually a testament that developers want the power and benefits of Ruby's syntax,

Scala's syntax, which is almost completely unlike Ruby's.

A JVM language occupying a similar nieche is Fan, with a growing community and tool support coming right up.

Fan seems to fit much better into Groovy's niche (the almost-Ruby-but-intimately-tied-to-the-JVM niche, such as it is) than Scala's. Scala's real niche, it seems to me, is closer to Java's own niche.

Re:Scala is great

[nadavwr]

Enter Scala: Scala achieves a brevity of syntax you see in Groovy and Ruby without sacrificing the static type system (it uses type inference).

While, certainly, Scala code is less verbose than Java, it is noticeably more verbose, in most cases, than Python, Ruby, or Groovy.

Marginally so, and mostly where type inference breaks down (e.g. public api's, but that's a Good Thing). If you code in Scala against standard Java libraries you will get verbose code same as you will with Groovy and Ruby. To properly judge Scala, look at samples coded against 'native' Scala libraries. And don't forget all the code you won't be writing for runtime type checking with Ruby or Python.

A very trivial comparison between Groovy and Scala can be seen here. A comparison of BDD-style tests between Scala, Java, and Ruby can be seen here.

Since Groovy and Ruby are dynamically typed, they are inherently slower than Scala.

Groovy, as I understand, supports but does not require static typing.

Not quite. From my understanding, if you declare an identifier to be of a certain type in Groovy and then go ahead and treat as something else, a type error would still only occur at runtime -- no type-related compilation errors! That's called strong typing, not dynamic typing. Please someone correct me if I'm wrong about Groovy on this. Also, using strongly typed variables in Groovy seems to be slower (as per the previous link, may have changed since).

The nieche Scala fits into is actually a testament that developers want the power and benefits of Ruby's syntax,

Scala's syntax, which is almost completely unlike Ruby's.

Agreed, but Scala and Ruby are both very malleable and share some very attractive features. To clarify my point: Java developers want some features popularized by Ruby (closures!), which Scala delivers.

A JVM language occupying a similar nieche is Fan, with a growing community and tool support coming right up.

Fan seems to fit much better into Groovy's niche (the almost-Ruby-but-intimately-tied-to-the-JVM niche, such as it is) than Scala's. Scala's real niche, it seems to me, is closer to Java's own niche.

Agreed about Scala's nieche being closer to Java's. I think we can fit Fan somewhere between Scala and Groovy: Fan's type system actually does support both static&dynamic (unlike Groovy, which is dynamic with support for strong&weak typing).

Mod down

[Anonymous Coward]

OP is just a twitter sock puppet.

more like

[toby]

Java and Erlang did. Python isn't particularly 'functional' despite some recent syntactic grafts.

There you go again!

[Jane Q. Public]

Ruby does not have a problem scaling. Neither, for that matter, does even Rails. (As the companies that run Basecamp, Campfire, LinkedIn, Lighthouse, and many others will tell you.)

The fact is that the Twitter folks tried to write their own message queue in Ruby, when there was absolutely no reason to do so: there were plenty of pre-made message queues already available for Ruby, and already optimized. Not only did they choose to write their own, unnecessarily, they did it badly.

And not only that, but Alex Payne has a hidden agenda: he is trying to push Scala to boost interest in the book about Scala he just wrote!

Please get some facts before digging up this long-dead and well-buried "Ruby or Rails doesn't scale" bullshit again.

Re:There you go again!

[Radhruin]

Anyone who thinks Ruby on Rails can't scale is as dogmatic in their anti-hype as the original hypers were. The right tool for the right job and all that.

Re:There you go again!

[mckinnsb]

Mod parent up. He is dead on.

Re:There you go again!

[chromatic]

Alex Payne has a hidden agenda: he is trying to push Scala to boost interest in the book about Scala he just wrote!

If that's damning criticism, it shows the stupidity of someone for believing that a technical book will sell enough copies that the author will earn more than minimum wage from writing it (a $39.95 book from O'Reilly with two authors gives Payne about $1 per copy -- he might earn $12k over three years).

Re:There you go again!

[Jane Q. Public]

As many programmers can tell you, publishing a successful book these days is worth a lot more than just the dollars it brings in directly; there is recognition and reputation, and perhaps a better-paying job down the road. Whether Payne is interested in the latter I do not know, but my point is simply that the book royalties are sometimes the least of the returns for being a successful author.

Re:There you go again!

[Anonymous Coward]

Please get some facts before digging up this long-dead and well-buried "Ruby or Rails doesn't scale" bullshit again.

Says the person getting his/her 'facts' from Obie Fernandez's insane flamebait blog post? Holy crap, dude!

When the Twitter folks wrote their own message queue, there were very limited options on the market. Seeing as Obie Fernandez has failed to even begin to explain, in technical terms (rather than saying "... made me throw up in my mouth") what's wrong with their implementation, forgive me if I don't consider this damning evidence

Moreover, if you're going to reference Basecamp, Campfire, Lighthouse, et al, perhaps you should also reference the ridiculous effort and resources that they expend in scaling Rails?

Rabid fanboyism does Rails and Ruby a disservice. I wouldn't touch that community with a 10 foot pole.

Re:There you go again!

[Nevyn]

It's good to see you didn't read the comments, where the Twitter admins/developers explain why the arguments in the blog aren't true. Meh, personally I hate Java and Ruby in equal amounts ... but just saying "Ruby can scale" over and over, isn't a winning argument.

Re:There you go again!

[moderatorrater]

Not a single site you posted is anywhere near the volume of traffic that Twitter's seen. Hulu's the closest, but the heavy lifting is done outside of RoR.

Re:There you go again!

[maxume]

If you announce it to the world, it isn't 'hidden'. He may very well have an agenda, but he isn't being particularly disingenuous about it.

And really, Twitter never said 'Ruby and Rails don't scale', they said 'This other stuff works better for what we are trying to do, for various reasons'.

Re:There you go again!

[Aloisius]

From the Scala website:

This change was driven by the companies need to reliably scale their operation to meet fast growing Tweet rates, already reaching 5000 per minute during the Obama Inauguration

In what parallel universe it is difficult to build a message queue capable of handling 83 messages per second? I built a fault-tolerant group message passing system 10 years ago that handled 30,000 messages per second on a dinky machine. Hell, Oracle's built in message queue system can handle more than 83 messages per second with ACID!

I will never, ever, ever understand the engineering choices of the Twitter team.

Re:There you go again!

[Radhruin]

Actually, Scribd is the closest, and had more traffic than twitter until the recent media frenzy.

Pointing out that the "heavy lifting" is done outside RoR isn't saying much. If you consider the heavy lifting of the entire Hulu service to be video encoding and streaming, well, obviously that's not done in Rails, that'd be stupid.

Personally, I would consider pushing that many web requests "heavy lifting" in the domain of problems for which Rails is a potential solution.

Re:There you go again!

[iluvcapra]

I dunno, the busy ones are all in the same ballpark, definitely the same magnitude, and twitter is a remarkably simple service compared to linkedin or hulu. As for "heavy lifting", a page view is a page view; pageviews do not suddenly get easier to service because that page has a video on it.

That said, none of them come near facebook.

Re:There you go again!

[moderatorrater]

pageviews do not suddenly get easier to service because that page has a video on it.

No, they get considerably harder. Hulu, if I remember correctly, dynamically alters the bitrate to compensate slow connections and improve the quality on faster ones. It also puts a load on the server that's throwing out the video regardless of the bitrate.

You're using Alexa's rank rather than their pageviews, which shows a considerably different picture. Also, unless I'm missing something, linkedin is written in java, not ruby on rails, it just had a rather high profile experiment with ruby on rails and a facebook app.

Re:There you go again!

[TheNarrator]

The core service platform at LinkedIn runs on Java

They do have a Facebook application that runs on Ruby.

Re:There you go again!

[tieTYT]

Jane Q. Public: Either you didn't read the comments of that blog or you're spreading FUD. Here is a comment from Alex Payne from that article:

Hoo boy. First of all, I hope you've had a chance to read my general reply to the articles about my Web 2.0 Expo talk [1] and this response to a vocal member of the Ruby community [2]. I sound like a pretty unreasonable guy filtered through the tech press and Reddit comments, but I hope less so in my own words.

Secondly, the quote at the top of your post is from my coworker, Steve Jenson, who's been participating in the discussion on this post.

On JRuby: as Steve said, we can't actually boot our main Rails app on JRuby. That's a blocker. Incidentally, if you know of anyone who has a large JRuby deployment, we'd be interested in that first-hand experience. If you don't, it might be a little early to say it would solve all our problems.

It's also incorrect to say that the way JRuby and Scala make use of the JVM is exactly the same. Much like our other decisions haven't been arbitrary, our decision to use Scala over other JVM-hosted languages was based on investigation.

On our culture: if you'd like to know about how we write code, or how our code has evolved over time, just ask us. We're all on Twitter, of course, but most of the engineers also have blogs and publish their email addresses. There's no need to speculate. Just ask. There's not a "raging debate" internally because we make our engineering decisions like engineers: we experiment, and base our decisions on the results of those experiments.

It's definitely true that Starling and Evented Starling are relatively immature queuing systems. I was eager to get them out of our stack. So, as Steve said, we put all the MQ's you think we'd try through their paces not too long ago, and we knocked one after another over in straightforward benchmarks. Some, like RabbitMQ, just up and died. Others chugged on, but slowly. Where we ran into issues, we contacted experts and applied best practices, but in the end, we found that Kestrel fit our particular use cases better and more reliably. This was not the hypothesis we had going into those benchmarks, but it's what the data bore out.

We get a lot of speculation to the tune of "why haven't those idiots tried x, it's so obvious!" Generally, we have tried x, as well as y and z. Funnily enough, I was actually pushing to get us on RabbitMQ, but our benchmarks showed that it just wouldn't work for us, which is a shame, because it advertises some sexy features.

Personally, I'm extremely NIH-averse; I research open source and commercial solutions before cutting a new path. In the case of our MQ, one of our engineers actually wrote Kestrel in his free time, so it was bit more like we adopted an existing open source project than rolled our own. Pretty much the last thing we want to be doing is focusing on problems outside our domain. As it so happens, though, moving messages around quickly is our business. I don't think it's crazy-go-nuts that we've spent some time on an MQ.

I hope my colleagues and I have been able to answer some of your questions. As I said, in the future, please consider emailing us so we can share our experience. Then, we can have a public discussion about facts, not speculation. Perhaps, as commenter sethladd suggested, the onus is on us to produce a whitepaper or presentation about our findings so as to stave off such speculation. Time constraints are the main reason why we haven't done so.

[1] http://al3x.net/2009/04/04/reasoned-technical-discussion.html
[2] http://blog.obiefernandez.com/content/2009/04/my-reasoned-response-about-scala-at-twitter.html#IDComment18212539

Re:There you go again!

[Achromatic1978]

And not only that, but Alex Payne has a hidden agenda: he is trying to push Scala to boost interest in the book about Scala he just wrote!

HAHAHA. OMG. Were you actually serious when you said that with a straight face, posting a rebuttal by a guy with links to HIS RAILS BOOKS in the sidebar of said rebuttal?

Regardless of the merits of either argument, that's pretty fucking laughable, don't you think?

Re:There you go again!

[chez69]

I use an open source MQ at work and it's never the bottleneck in high message rates (> 100 msgs/second easily)

Re:There you go again!

[Firehed]

Let me know when Alexa takes the once-a-minute API calls to Twitter into account. Or even has vaguely accurate numbers, for that matter. Google Analytics data (while not being publicly available) would be reasonably accurate on the page views front, but my four or five page views a day to twitter.com are irrelevant next to the 60 API calls an hour my desktop client makes every hours of the day.

Re:There you go again!

[Just Some Guy]

twitter is a remarkably simple service compared to linkedin or hulu

I totally disagree with that. LinkedIn and Hulu serve mostly static pages, or at least static enough that a good cache could offload a large portion of their workload. Almost every page from Twitter would require a lot of queries.

Re:There you go again!

[godefroi]

I wrote an application that uses an MSSQL server to implement a message bus (with ACID), and even on old hardware (mid-range desktop P4, 1GB memory, singe disk), running the app and the database on the same machine, I saw rates in the ~250/second range.

Re:There you go again!

[JAlexoi]

Oh, sure RoR can scale. Problem is that in the end, you will be getting close to a point where putting more H/W is more expensive, then the man power to fix the concurrency issues.
And if you look at the posterchild sites, you mostly see something other then RoR hauling the most heavy load.
I have been in this debate for a really long time now. And, literally, nothing has persuaded me to move from my current set of languages, to scale the back end to ridiculous levels. Sorry Ruby zealots I am already immune to you venoms.

Here is an interesting discussion on alternatives

[tieTYT]

http://unlimitednovelty.com/2009/04/twitter-blaming-ruby-for-their-mistakes.html

This blog post takes the attitude that Twitter didn't move to Scala because ROR had a problem, but because the in-house messaging system Twitter created performed poorly. The author does not work at Twitter but many of the Twitter developers (including Alex Payne) respond in the comments. I found the article to be very interesting and the comments even more so. They give a sense of how much research Twitter did before this change.

Should have used Erlang!

[K. S. Kyosuke]

'Nuff said. Those twitter posts were once described as "internet SMS messages", didn't they? Short messages...phone systems and heavy networking...Ericsson...Erlang...

Re:Should have used Erlang!

[Anonymous Coward]

Indeed, I don't know why they are going with a multi-paradigm language, when they really need a pure functional language like erlang.

Re:Should have used Erlang!

[K. S. Kyosuke]

I bet the keywords here are not "multi-paradigm" or "purely functional", but "maturity", "libraries", "track record" and "community" (it seems to me that Erlang is mostly used for demanding networking applications and many Erlangistas (?) on mailing lists etc. might have precisely the experiences necessary for building something like Twitter).

Mod Up

[Jane Q. Public]

The problems they had were not due to Rails. The problems arose from poor implementation of their message queue... some bad engineering decisions.

http://unlimitednovelty.com/2009/04/twitter-blaming-ruby-for-their-mistakes.html

And some of their recent Scala noise might be due to the fact that Alex Payne just wrote a book about Scala that was just released, or is about to be.

Twitter on Scalia!

[Steve1952]

At first I read the title as "Twitter on Scalia". Justice Scalia is one of the most conservative US Supreme Court justices, and I'm not sure that this would have been a happy combination.

Re:Twitter on Scalia!

[bersl2]

Same here.

I hope it's at least fast.

[rduke15]

Well, at a first glance, it looks just as rigid and boring as Java.

I guess that can be considered Falmebait?

Anyone remember this quote?

[..] but it's certainly the case that over the last twenty years or so, many Computer Scientists have come out in opposition to the Art of Programming. In trying to make programming predictable, they've mostly succeeded in making it boring. And in so doing, they've lost sight of the idea that programming is a human pursuit. They've designed languages intended more to keep the computer happy than the programmer. Was any SQL programmer ever happy about having to declare a value to be varchar(255)? Oops, now it's a key, and can't be longer than 60. Who comes up with these numbers?

Re:I hope it's at least fast.

[obarthelemy]

Yeah for Unpredictable Programming ! That should be fun indeed :-)

Preferred Language = probably PERL, I would guess, but

- variables can only be accessed via pointers to pointers that point to pointers. Sometimes.

- New function: Rand_UP (something): does "something" to "something", not clear what, and sometimes, the other way round. MUST be used at least twice per page.

Cost of developers vs. cost of hardware

[janwedekind]

From The Register article:

"Investors now want to know that you're not going to be paying tons and tons of money for servers because you decided to build in a stack that isn't as efficient as possible," he (Alex Payne) said.

I think in the beginning it matters a lot what language you use. If the scale of your Web business is small, the salaries of your staff will be the biggest item on the list of expenses. I guess now they'll have quite a number of computers and having to buy 20% more servers will cost them quite a bit. If they are saying now that Scala is good for them, it does not mean anything. At this point it probably would be profitable for them to port a large part of their software to COBOL. A more forward-looking approach would be to improve the Ruby VM. But maybe they need somebody else to do it for them.

Occasional?!

[Anonymous Coward]

the occasional Slashdot article.

Im going to post anonymously not to conceal my identity (that will be obvious by the rest of this post) but to deflect the accusations of karma whoring.

Twitter is not getting the warranted "occasional Slashdot article". It is getting a lot more than that. A couple of days ago, maybe a week, I mentioned it in another comment. That day Twitter had 3 articles in the front page of Slashdot, arguably none of them being news by itself except for the "novelty" that is to involve Twitter in activities that, otherwise, would be totally unremarkable.

To illustrate that, I will replace the mentions to "Twitter" on those articles by "Instant Messenger" or "Social Network", depending on the context. Notice that I understand that Twitter is a little bit more that "Instant messaging" in a sense that it is more permanent and involves more than one media, but it fits the definition,

"a form of real-time communication between two or more people based on typed text. The text is conveyed via devices connected over a network such as the Internet"

Proposal Suggests UK Students Study Wikipedia and Social Networks
Build Your Own Open Source Instant messaging Power Meter
Researchers Can ID Anonymous Social Network users

In the first example, Twitter is only one of the many new tecnologies that would potentially be studied by the UK brats. In the second, it could be any protocol, and it did not impressed the audience (only 31 comments).

But the third one is the stranger. It mentions something, placing enormous emphasis on the fact that it is about "identifying anonymous twitterers", that were already covered before in a more neutral point of view here on Slashdot, in the article Google Researchers Warn of Automated Social Info Sharing, three months before.

It is not occasional, it is habitual, and I don't believe it warrants all that coverage. But yet, there are two stories again on the front page of Slashdot covering peripheral interactions of Twitter with the rest of the world: Organized Online, Students Storm Gov't. Buildings In Moldova (people using instant and relatively anonymous communication to organize themselves, a novelty since the creation of the telegraph) and this very one.

Seems like Second Life all over again.

How many saw Scala and thought about...

[Anonymous Coward]

the Commodore Amiga?

Re:How many saw Scala and thought about...

[68030]

Thought? I was STILL thinking about it until your post D:

Right.

[moniker127]

I was looking at the syntax of scala- and it looks like they're trying to do web development in C. Why can't people just learn a new syntax?

It isn't going to kill you. It really wont.

Re:Right.

[ClassMyAss]

I was looking at the syntax of scala- and it looks like they're trying to do web development in C.

You've never seen actual Scala code, then, as no decent Scala I've ever seen looks anything at all like C.

Good Scala should look more like typed Ruby than anything else...

Re:Right.

[DragonWriter]

I was looking at the syntax of scala- and it looks like they're trying to do web development in C.

A lot of what they are replacing with Scala is things like their message queueing system, which isn't "web development", except in the extremely loose sense in that it is used in a very large system which happens to have a web-facing interface available.

Why can't people just learn a new syntax?

The one thing Scala is, compared to almost any language you care to name (and certainly C) is "a new syntax".

pageviews, not rank

[moderatorrater]

You are correct that Hulu is surpassed by scribd in rank, but if you look at the pageviews instead of the ranking, you'll see that twitter and hulu get more pageviews than scribd and have gotten more since at least january. Reach incorporates unrelated metrics such as unique visits, etc, which doesn't have as much of an effect as just sheer pageviews. Funny enough, that would make scribd's ridiculous bounce rate give it a higher overall rank.

Re:pageviews, not rank

[Radhruin]

Roger, good point.

Temporarily....

[Cloud K]

.... irrelevant, now that you lot have gone and Slashdotted it :P

Wondered why I couldn't get a word in edgeways without seeing the over-capacity whale.

They wanted a type system...

[patniemeyer]

The most interesting part of the article to me was that they said they wanted the benefits of a type system, which then ended up reproducing in large part in their code. They also wanted the stability of the JVM and they use Java collections from Scala. Hmmm... I wonder what other language they could have chosen that would have had all of those features, an existing messaging system, and developers on every street corner?

Re:They wanted a type system...

[Anonymous Coward]

The most interesting part of the article to me was that they said they wanted the benefits of a type system, which then ended up reproducing in large part in their code. They also wanted the stability of the JVM and they use Java collections from Scala. Hmmm... I wonder what other language they could have chosen that would have had all of those features, an existing messaging system, and developers on every street corner?

Your knowledge of Scala is clearly nil if you think that Java's type system (and other language features) even begin to hold a candle to Scala's

Psht.

[kkrajewski]

I program in PDP-11 assembly, which is then translated into C, compiled into Java bytecode, and executed on a JVM. I call it Assemblacava, and it's the wave of the future.

Re:Psht.

[Anonymous Coward]

Clever. Why did you leave out the macro processor you used, huh?!

Re:Psht.

[Anonymous Coward]

PDP-11 assembler sucks. You should try PDP-10 assembler for a *real* taste of power.

Re:Psht.

[Qbertino]

I program in PDP-11 assembly, which is then translated into C, compiled into Java bytecode, and executed on a JVM. I call it Assemblacava, and it's the wave of the future.

Dude, that is *so* 2008! I programm in Sharp 1403H Opcode which is piped into an optimized generator for x86 opcode which is written in eLisp and even runs on my old slightly rigged m105 Palm without a hitch - very cool!. Sharp 1403H Opcode where the future and money is. Get with the proramm!

Oh, and btw., the portables that run it natively have upwards of 300 hours of battery life and are way smaller than these bulky netbooks from last year. Now who's gonna beat that? Hah!

Twitter should have used Perl

[Anonymous Coward]

Every scripting language tries to do what Perl does. None of them have CPAN.

I've tried repeatedly to use Ruby and RoR. For trivial projects, they are fine. Scalability may come, but Ruby GEMS needs to be rewritten in Perl to remove the HUGE memory footprint to maintain your GEMS. You all know what I mean.

Yep, perl is what Twitter needs.

Re:Twitter should have used Perl

[binarylarry]

Taco, is that you?

Re:Personally, I think they should've...

[steelfood]

...gone with minifig scale.

But I approve of this move in general. I think the putting together little bricks to make something much bigger really fits with idea of a lot of little comments to "put together" a person.

Actually, this is pretty complex

[Stu Charlton]

Twitter is not a trivial application to scale, considering the wide disparity in listeners to follower ratios, that views are dynamically generated by interpolating many-to-many message streams, and that each message is persistent forever.

As an analogy, It's like managing an IRC server, with persistent messages that are full-text indexed, with one channel per user, and unlimited number of users can join each other's channels. When you join a new user's channel, your chat log is automatically (and quickly) re-woven with messages from that channel according to relative time series of these messages. And, there's a global channel that everyone can watch to see what any user in any channel is saying at any time.

Now do this, all the while avoiding netsplits (i.e. missing messages), allowing retracts of almost message, recent or historical, and ensuring the channel history (eventually) reflects that change. And handle sudden bursts of activity among unpredictable sets of channels because they're all attending the same conference, or a burst of network-wide high activity because people are watching the World Cup or Obama's inauguration.

The point is that, while the idea is simple, the variability of use and disparity of activity is what makes life interesting; the messaging & DB architecture that works well for recent activity, for example, doesn't help for having reasonable persistent random-access to historical messages.

In all, Twitter has gotten a *lot* more reliable the past several months than it was a year ago.

scala vs erlang

[xkcd150]

i figured a lot of people would mention erlang, and thought someone might be interested in this writeup i read the other day http://yarivsblog.com/articles/2008/05/18/erlang-vs-scala/

This is great

[melted]

Java is a flaming turd of a programming language compared to Scala. About time someone made it outdated. Hopefully other "fad" sites follow the trend and switch to Scala as their language of choice, and then serious dev houses follow. Frankly, after reading the Scala book, I can't see a good reason for a team with little to no legacy Java code to even bother with Java.

old trick for old dogs

[speedtux]

Scala-like languages have been around for decades; it's just that now people are finally starting to use them.

Ooooh New Shiny

[Anonymous Coward]

I'll buy your book Mister. I'll pay lots and lots of money for your book. Gimme NOW!

Buzzwords? Nope, Just Software Engineering.

[TwinkieStix]

Words like "inversion of control" and "Actors" are not new buzzwords. They are part of the functional vocabulary of real software engineers, and they are easily over a decade old. Check out the book with the ISBN 0201633612, published in 1994 and is considered a classic containing what I believe is all of the "buzzwords" you are talking about.

It's the idea of abstracting specifics like "callback functions" to patterns such as IoC that separates "programmers", "hackers", and even "computer scientists" from "software engineers".

I have a degree in Computer Science and I am a professional Software Engineer, and let me tell you, Software Engineering is not the same as Computer Science. Both studies have their place, and both are important.