Slashdot Mirror
Scaling Large Projects With Erlang
Delchanat points out a blog entry which notes,
"The two biggest computing-providers of today, Amazon and Google, are building their concurrent offerings on top of really concurrent programming languages and systems. Not only because they want to, but because they need to. If you want to build computing into a utility, you need large real-time systems running as efficiently as possible. You need your technology to be able to scale in a similar way as other, comparable utilities or large real-time systems are scaling — utilities like telephony and electricity. Erlang is a language that has all the right properties and mechanisms in place to do what utility computing requires. Amazon SimpleDB is built upon Erlang. IMDB (owned by Amazon) is switching from Perl to Erlang. Google Gears is using Erlang-style concurrency, and the list goes on."
They were right!
Perhaps the systems would be better running efficiently rather than sufficiently?
"The two biggest computing providers of today"?
What the hell does that mean?
Also, is it just me or does the article intro sound like it was written by someone who has taken way too many marketing classes?
TFA states "Because I don't want to be hooked into the (proprietary) Google stack (Python, Django, BigTable, GoogleOS) just yet" ... IMHO neither Python nor Django are proprietary. Or even proprietary in a way that the AWS stack is not?
"running as sufficiently as possible"?
Sometimes as a nation we must ask ourselves, is our children learning?
You see? You see? Your stupid minds! Stupid! Stupid!
People may also want to check out Scala at:
http://www.scala-lang.org/
It also uses the Erlang style concurrency approach and runs on the JVM with class compatibility with other JVM languages, ie Java, Groovy, etc.
1. Multicore ready.
Erlang will use them. Write your application in Erlang and it's done for you.
2. Scales well.
As an example, http://yaws.hyber.org/ scales very nicely when loads increase. Your basic LAMP/LYMP setup runs much better on vanilla hardware.
3. Designed for telecom
The architects designed the language to run in a telecom environment so things like upgrades can be done while the application is running.
Yaws in particular needs your help. Failover clustering inside the yaws server would be wonderful. Right now, it uses CGI to process other languages. It does it flawlessly, but a more direct solution might be a nice project.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
"you need large real-time systems running as sufficiently as possible."
Should that not be efficiently as possible?
You obviously haven't looked very closely at any of the "market leader" software lately.
Software from the Big Guys is more and more designed to sell (think forced upgrades) bigger, faster systems. You don't do this by making your software efficient.
The logic behind many software updates these days is "Will this release require sufficient resources that customers will be persuaded to upgrade to new hardware?"
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
You obviously don't read summaries, articles, or headlines. The logic behind your post is "rant about something for no reason at all."
I think the summary (and article) are somewhat poorly written, but that doesn't shadow the fact that functional languages are becoming more and more interesting these days with concurrency becoming so important.
I'd like to learn one, but there are several out there.. What I'd like to see is a good in-depth comparison of different concurrent functional languages: why would I choose Haskell, or Erlang, or OCaml, for example? Are they all interpreted? (Does one exist that compiles?) Which ones support concurrency? What language features do they boast, and what are the advantages and disadvantages of these features? Do they have a complete set of libraries?
Anyone know of an article like this? I've been searching for a while. Every article on functional languages I've found seems to concentrate on a particular one, but I can't find something helping me decide which one is most worth learning.
If you want to build computing into a utility, you need large real-time systems running as sufficiently as possible.
But if you want to build sprockets into a weasel you need small batch-mode systems running as necessarily as possible.
If the poster had anything interesting to say (I'd guess not, but who knows!), it was totally obscured by his lack of grasp of the English language.
Actually, Gears doesn't use Erlang either. What he means is that Gears threading doesn't allow for shared state (is it really threading then?). Instead threads communicate back to the browser by message passing.
It's remarkably deceptive indeed to even imply that Gears and Erlang are connected. Message passing based concurrency isn't exactly new or limited to Erlang, and can be implemented in any language.
I'm not sure what the point of this piece is. I've looked at Erlang and didn't see much of anything to get me excited. It's a functional language, which like most of them have unnecessarily weird syntax and force immutable state. I don't really see what this buys you over a language like D 2 (or hell, even C++) in which you can write in a functional message passing style if you like, but then still use imperative shared state whenever useful, convenient or performant.
Given that this statement appears almost halfway through the blog post, I would say that it was already too late for that.
'The tyrant will always find pretext for his tyranny.' - Aesop's Fables
1. Invariable variables.
This appears to have been done for no reason other than the designer's preference. In fact, it's not strictly true -- variables can be unbound, and later bound. They just can't be re-bound once bound.
2. Weird syntax.
Why, exactly, are there three different kinds of (required) line endings? It seems as though the syntax is designed to be as different from C as possible, while maintaining at least as many quirks. Moreso, even -- when constructing normal, trivial programs, you're going to hit most language features head-on and at their worst. Where's my 'print "hello\n"' that works most other places?
I don't believe the important features of Erlang are mutually-exclusive with the sane syntax of, say, Ruby or Python.
3. Not Unicode-ready.
Strings are defined as ASCII -- maybe latin1. But there's no direct unicode support in the language -- if you're lucky, there are functions you can pipe it through.
There are other things I haven't mentioned, mostly implementation-specific -- things like the fact that function-reloading cannot be done when you natively-compile (with hipe) for extra speed. My plan is to take the features I actually like from Erlang and implement them elsewhere, in a language I can actually stomach for its real tasks.
Don't thank God, thank a doctor!
Wow, it's not often I strongly criticise articles around here, but that was total garbage.
For the smart ones that didn't RTFA, here's a quick summary:
For the record, I work for Google and we don't use Erlang anywhere in the codebase. Google Gears restricts you to message passing between threads because JavaScript interpreters are not thread-safe, so it's the only way that can work. Visual Basic threading works the same way for similar reasons. It's not because eliminating shared state is somehow noble and pure, regardless of what the article would have you believe, and in fact systems like BigTable use both shared-state concurrency and message passing based concurrency.
The article says this:
But in fact the Google search engine, which is one of the larger "industrial-grade, internet-grade" systems I know of, is written entirely in C++. A language which is much the same as it was 10-15 years ago. Thus the central point of his argument seems flawed to me.
Seeing as the article is merely an advert for Erlang, I'll engage in some advocacy myself. If you have an interest in programming languages, feel free to check out Erlang, but be aware that such languages are taking options away from you, not giving you more. A multi-paradigm language like version two of D is a better way to go imho - it supports primitives needed to write in a functional style like transitive invariance, as well as a simple lambda syntax, easy closures and first class support for lazyness.
However it also compiles down to self-contained native code in an intuitive way, or at least, a way that's intuitive to the 99.9% of programmers used to imperative languages, unlike Erlang or Haskell. It provides garbage collection but doesn't force you to use it, unlike Java. It doesn't rely on a VM or JIT, unlike C#. It provides some measure of C and C++ interopability, unlike most other languages. And it has lots of time-saving and safety-enhancing features done in a clean way too.
Erlang is a language that has all the right properties and mechanisms in place to do what utility computing requires.
Well, except that it's darned inconvenient to actually write the applications in it.
Google Gears is using Erlang-style concurrency, and the list goes on."
Yup, and it makes more sense to add "Erlang-style concurrency" to existing languages than to throw out everything and switch to Erlang.
Andy
The enthusiasm for "cloud computing" may evaporate when Xmas rolls around.
I went to a talk at Stanford by the architect of Amazon's web services. It came out in questioning that the real motivation between Amazon's low-priced web services is that their load in the Xmas shopping season is about 4x the load for the rest of the year. Their infrastructure is sized for the November-December peak, so for ten months of the year they have vast excess capacity. That's why Amazon's web services are so cheap.
Don't expect good response time during the shopping season. Although this Xmas might be OK, due to the recession.
It's a functional language you dolt, of course the "variables" are invariant. The only way in which it was the designers preference is that he presumably wanted all the niceties that come with declarative languages; provability, implicit parallelism, etc.
1. Invariable variables.
This appears to have been done for no reason other than the designer's preference. In fact, it's not strictly true -- variables can be unbound, and later bound. They just can't be re-bound once bound.
Why is any syntax that's not filched from C weird? Frankly I'm not that fond of C's syntax it _can_ lead to very unreadable code (Speaking as someone with 20 years experience with C). If you'd done a modicum of research you'd realise that Erlang models its syntax on Prolog, and Prolog like Lisp has very regular syntax.
Note that I'm not claiming that Erlang, or Prolog, or Lisp syntax is that great either, but by holding up C as some kind of gold standard you've automatically lost that argument.
2. Weird syntax.
Why, exactly, are there three different kinds of (required) line endings? It seems as though the syntax is designed to be as different from C as possible, while maintaining at least as many quirks. Moreso, even -- when constructing normal, trivial programs, you're going to hit most language features head-on and at their worst. Where's my 'print "hello\n"' that works most other places?
I don't believe the important features of Erlang are mutually-exclusive with the sane syntax of, say, Ruby or Python.
Not sure about Unicode, you may actually have made a valid point here.
3. Not Unicode-ready.
Strings are defined as ASCII -- maybe latin1. But there's no direct unicode support in the language -- if you're lucky, there are functions you can pipe it through.
There are other things I haven't mentioned, mostly implementation-specific -- things like the fact that function-reloading cannot be done when you natively-compile (with hipe) for extra speed. My plan is to take the features I actually like from Erlang and implement them elsewhere, in a language I can actually stomach for its real tasks.
Joe Armstrong's Erlang textbook is interesting, but I did not have time to learn the language and recode the part of our current project that would benefit from it. So I did what any sensible person would do: raided the concepts, and used them to redesign the critical parts of the application. I was initially provoked into doing this because, in the book, the comparison at one point between the Erlang and Java way of doing something is just plain wrong. When I thought out how I would actually do it in Java, I realised that it helps to stick to a language you know well.
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
They built the Facebook Chat backend using Erlang. Scaling something from 0 users one day to tens of millions of active users the next day is a challenge, and they decided Erlang was the right tool for the job. http://www.facebook.com/note.php?note_id=14218138919&id=9445547199
TFA more or less says that IMDB is switching from Perl to Erlang. So I looked at the link and here's what I got:
(From here
We are looking for developers with experience building web scale distributed systems. We are currently working in Perl but have plans to use Java, Erlang and any other language that we think will suit our purposes. We aren't looking for expertise in any of those, particularly, but we expect that you will be an expert in the systems you know. We do require that you be passionate about testing (unit, integration, fault-injection) and code quality. Experience with relational databases (Oracle, MySQL, etc), embedded databases (BerkeleyDB, CDB, MonetDB, etc) and Linux are a big plus.
I'll leave anyone to draw his own conclusions.
Erlang vs. Stackless python: a first benchmark is a very good discussion of lots of niggling details in benchmarking a concurrency language. The comments are quite good.
Every mans' island needs an ocean; choose your ocean carefully.
It should be noted that Facebook's relatively new chat feature, which allows Facebook users to send instant messages to all their online friends as well as see status changes, notifications, and feed stories in near real time, was developed using Erlang. http://www.planeterlang.org/story.php?title=Facebook_chat_is_developed_in_Erlang
fewer
There are no karma whores, only moderation johns