Scaling Large Projects With Erlang

Delchanat points out a blog entry which notes, "The two biggest computing-providers of today, Amazon and Google, are building their concurrent offerings on top of really concurrent programming languages and systems. Not only because they want to, but because they need to. If you want to build computing into a utility, you need large real-time systems running as efficiently as possible. You need your technology to be able to scale in a similar way as other, comparable utilities or large real-time systems are scaling — utilities like telephony and electricity. Erlang is a language that has all the right properties and mechanisms in place to do what utility computing requires. Amazon SimpleDB is built upon Erlang. IMDB (owned by Amazon) is switching from Perl to Erlang. Google Gears is using Erlang-style concurrency, and the list goes on."

Comments

Scala

[fils]

People may also want to check out Scala at:
http://www.scala-lang.org/

It also uses the Erlang style concurrency approach and runs on the JVM with class compatibility with other JVM languages, ie Java, Groovy, etc.

Re:Scala

[bonefry]

There is a significant difference between Scala and Erlang.

Erlang uses green threads. And green threads have advantages and disadvantages over native threads.

For instance Erlang is bad at IO but on the other hand it can spawn millions of threads, something that the JVM has a hard time doing because native threads are limited by the kernel.

Re:Scala

[Richard W.M. Jones]

"Last time you checked" was some time last century in that case. Linux kernels have been able to support at least 100,000 threads for ages.

That doesn't mean that using shared memory concurrency is a good idea though. When your computer comes with 10s or 100s of cores you'll realise that maybe SMP wasn't the best model of concurrency to choose. That's where models such as map-reduce, Erlang's shared nothing concurrency, message passing, and MPI come into their own. Even today they are useful because you'll be able to scale your program across multiple machines.

Rich.

Re:Scala

[Jamie Lokier]

Linux threads stopped using the LDT on x86 in 2002. This change went mainstream over subsequent years, and is nowadays always used on x86.

There was once a limit on the number of processes, too, due to each process having an entry in the GDT. That has long been removed too.

Stupid article

[IamTheRealMike]

Wow, it's not often I strongly criticise articles around here, but that was total garbage.

For the smart ones that didn't RTFA, here's a quick summary:

• I like Erlang.
• Big companies like Google and Amazon make things fast by using concurrency.
• Erlang supports (one type of) concurrency.
• Thus Google and Amazon are [probably] using Erlang.
• Thus everyone should learn Erlang.

For the record, I work for Google and we don't use Erlang anywhere in the codebase. Google Gears restricts you to message passing between threads because JavaScript interpreters are not thread-safe, so it's the only way that can work. Visual Basic threading works the same way for similar reasons. It's not because eliminating shared state is somehow noble and pure, regardless of what the article would have you believe, and in fact systems like BigTable use both shared-state concurrency and message passing based concurrency.

The article says this:

Architects (but also university-professors for that matter) still think they can build current and future industrial-grade and internet-grade systems with the same technologies as they did 10-15 years ago.

But in fact the Google search engine, which is one of the larger "industrial-grade, internet-grade" systems I know of, is written entirely in C++. A language which is much the same as it was 10-15 years ago. Thus the central point of his argument seems flawed to me.

Seeing as the article is merely an advert for Erlang, I'll engage in some advocacy myself. If you have an interest in programming languages, feel free to check out Erlang, but be aware that such languages are taking options away from you, not giving you more. A multi-paradigm language like version two of D is a better way to go imho - it supports primitives needed to write in a functional style like transitive invariance, as well as a simple lambda syntax, easy closures and first class support for lazyness.

However it also compiles down to self-contained native code in an intuitive way, or at least, a way that's intuitive to the 99.9% of programmers used to imperative languages, unlike Erlang or Haskell. It provides garbage collection but doesn't force you to use it, unlike Java. It doesn't rely on a VM or JIT, unlike C#. It provides some measure of C and C++ interopability, unlike most other languages. And it has lots of time-saving and safety-enhancing features done in a clean way too.

Re:Why Erlang doesn't matter

[stonecypher]

1) Actually, there are quite a few good reasons for this, largely around the complete elimination of mutexing and locks. Just because you don't understand the purpose doesn't mean there wasn't one.

2) Oooooh, a language is faulty because it has a syntax with which you are not familiar. Immediately kill all non-Java clones!

3) They're just lists of numbers; they're neither ASCII nor Latin1. There is unicode parsing in the XMERL module.

Please wait until you know a language before criticizing it.