Scaling Large Projects With Erlang

Delchanat points out a blog entry which notes, "The two biggest computing-providers of today, Amazon and Google, are building their concurrent offerings on top of really concurrent programming languages and systems. Not only because they want to, but because they need to. If you want to build computing into a utility, you need large real-time systems running as efficiently as possible. You need your technology to be able to scale in a similar way as other, comparable utilities or large real-time systems are scaling — utilities like telephony and electricity. Erlang is a language that has all the right properties and mechanisms in place to do what utility computing requires. Amazon SimpleDB is built upon Erlang. IMDB (owned by Amazon) is switching from Perl to Erlang. Google Gears is using Erlang-style concurrency, and the list goes on."

Comments

Erlang: The Movie !

[Enlightenment]

They were right!

Re:Erlang: The Movie !

[mini me]

Hello, Robert.

Sufficiently?

[Anonymous Coward]

Perhaps the systems would be better running efficiently rather than sufficiently?

Perl + erlang

[Anonymous Coward]

Perlang

hard to read after

[puto]

"you need large real-time systems running as sufficiently as possible."

Should that not be efficiently as possible?

Re:hard to read after

[jc42]

"you need large real-time systems running as sufficiently as possible."

Should that not be efficiently as possible?

You obviously haven't looked very closely at any of the "market leader" software lately.

Software from the Big Guys is more and more designed to sell (think forced upgrades) bigger, faster systems. You don't do this by making your software efficient.

The logic behind many software updates these days is "Will this release require sufficient resources that customers will be persuaded to upgrade to new hardware?"

Re:hard to read after

[The End Of Days]

You obviously don't read summaries, articles, or headlines. The logic behind your post is "rant about something for no reason at all."

Re:hard to read after

[Nursie]

I work for one of the "Big Guys".

We spend a lot of time looking at efficiency and trying to do more with what we've got. This is because we're interested in selling software for everyone's hardware and not just our own. You can't get everyone to switch to exclusively our stuff, but we can still get into their operations by being the best.

Re:hard to read after

[orasio]

Maybe efficiency is not the main goal.
We are talking about utility-like quality of service.
That is not necessarily efficient.
I am sure the power grid would be much more efficient if it didn't have to account for peak usage. But they do, because their core value is not efficiency, but high availability.
I don't know if the guy meant something like that, but "efficiency" is not the right word here, either.

Re:hard to read after

[antifoidulus]

You obviously don't read summaries, articles, or headlines. The logic behind your post is "rant about something for no reason at all."

Dude, if you threw out all those posts, each story would have what, maybe 10 comments per story at most? And yes, I'm guilty of the above sin, quite often actually :P

Re:hard to read after

[antifoidulus]

And if you threw out posts with redundant grammar in them, you would have even less posts per story!

Re:hard to read after

[davester666]

Quick, somebody forward this article to twitter!

Re:hard to read after

[Anonymous Coward]

There is an inherent logic in the way big guys as you called them work. They optimize the efficiency of R&D processes not efficiency of the resulting programs.
BTW: I wonder what makes anybody think Erlang will change it in anyway? It did not in the company that developed it. I like the ideas that were used in the development of language though.

Re:hard to read after

[Anonymous Coward]

Now this post is why the 'Funny' moderation doesn't give karma.

Re:hard to read after

[The Clockwork Troll]

fewer

Re:hard to read after

[OwnedByTwoCats]

And apparently the author of the piece doesn't really care about 100% power availability. Because he doesn't mention the care and feeding of UPSs. Infrastructure services are there and you can use them... until they're not.

First, I plugged TiVo and the cable box into the wall outlets. At least once a week I'd come home and my wife would be fuming because there was a power glitch and the cable box shut itself off.

Now there's a UPS for the TiVo and the cable box. But it only sustains the devices for 15 minutes. Sometimes the power is out for longer than that.

Erlang! It's Concurrently Sufficient!

[Anonymous Coward]

I mean, It's Sufficiently Concurrent!

...wait, what?

Bingo!

[Anonymous Coward]

Buzzwords that matter.

Huh?

[The Breeze]

"The two biggest computing providers of today"?

What the hell does that mean?

Also, is it just me or does the article intro sound like it was written by someone who has taken way too many marketing classes?

Re:Huh?

[K. S. Kyosuke]

Maybe they referred to Amazon EC2 and Google Application Engine?

Re:Huh?

[augustwest2112]

Also, is it just me or does the article intro sound like it was written by someone who has taken way too many marketing classes?

Too many marketing classes and not enough English classes.

Re:Huh?

[fermion]

Definitely market copy. Extremely general, not useful information, indiscriminate name dropping, with unintended consequences.

For instance, by dropping the imdb name, it is now my impression that this Erlang thing is best at destroying otherwise useful sites by making them less reliable and more annoying to users. Who in their right mind would want to do that. Oh, marketing people, thats who!

Re:Huh?

[SolitaryMan]

In the article the guy says he is a consultant. This explains everything

Re:Huh?

[extrasolar]

It's not just you.

Re:Huh?

[DRobson]

Let's not forget Network.com

Moore's Law

[ryanscottjones]

Eh... Moore's Law says that we'll all be on the same processor in a couple years anyway.

Re:Moore's Law

[Anonymous Coward]

It does?

Re:Moore's Law

[ryanscottjones]

It does?

Sigh... no, it doesn't.

Re:Moore's Law

[uassholes]

Of course it does. Haven't read about Jupiter Brains and The Singularity. We'll all be disembodied minds swimming in Computronium eventually.

http://en.wikipedia.org/wiki/Computronium

Anyway, I think that the webertubes need a condescension filter to block the passage of "Sigh..."

Re:Moore's Law

[ryanscottjones]

Exactly. That was pretty much my tongue-in-cheek point. The greater our processing power, the less we need things like Erlang. Follow that out and you have your Computronium. Don't really see how this is "offtopic," but I'm over it.

Proprietary?

[2stein]

TFA states "Because I don't want to be hooked into the (proprietary) Google stack (Python, Django, BigTable, GoogleOS) just yet" ... IMHO neither Python nor Django are proprietary. Or even proprietary in a way that the AWS stack is not?

Who wrote the summary? GWB?

[Junior J. Junior III]

"running as sufficiently as possible"?

Sometimes as a nation we must ask ourselves, is our children learning?

Scala

[fils]

People may also want to check out Scala at:
http://www.scala-lang.org/

It also uses the Erlang style concurrency approach and runs on the JVM with class compatibility with other JVM languages, ie Java, Groovy, etc.

Re:Scala

[bonefry]

There is a significant difference between Scala and Erlang.

Erlang uses green threads. And green threads have advantages and disadvantages over native threads.

For instance Erlang is bad at IO but on the other hand it can spawn millions of threads, something that the JVM has a hard time doing because native threads are limited by the kernel.

Re:Scala

[Cyberax]

Scala has actors, which are allow you to do something _like_ green threads: http://lamp.epfl.ch/~phaller/doc/ActorsTutorial.html

Re:Scala

[jonabbey]

Modern JVMs on the modern Linux Kernel can spawn quite a hellacious amount of threads these days, actually.

The problem with Java is the shared-state synchronization that is often necessary, and the extra work required to distribute state to threads across different VMs. A functional language and programming style could work quite well on top of the JVM, though, and could leverage RMI and some kind of message port facility for the distribution.

Re:Scala

[metaconcept]

Modern JVMs on the modern Linux Kernel can spawn quite a hellacious amount of threads these days, actually.

Thousands? Or millions, like Erlang?

Re:Scala

[TheRaven64]

Last time I checked, the Linux kernel was limited to around 8000 threads per process on x86, since it used an LDT entry for TLS on each one (and, if you didn't properly dispose of your threads, would leak the LDT entry, causing some really difficult-to-track bugs). I believe a modern JVM uses an N:M threading model, and removes locks if all threads that can access a one are on the same OS thread. I doubt it scales as well as beam, which is designed from the ground up to handle insane numbers of (potentially very short-lived) threads, however.

Re:Scala

[Richard W.M. Jones]

"Last time you checked" was some time last century in that case. Linux kernels have been able to support at least 100,000 threads for ages.

That doesn't mean that using shared memory concurrency is a good idea though. When your computer comes with 10s or 100s of cores you'll realise that maybe SMP wasn't the best model of concurrency to choose. That's where models such as map-reduce, Erlang's shared nothing concurrency, message passing, and MPI come into their own. Even today they are useful because you'll be able to scale your program across multiple machines.

Rich.

Re:Scala

[Anonymous Coward]

And green threads have advantages and disadvantages over native threads.

Inevitable parallel: Green technology have advantages and disadvantages over the technologies of natives.
Inevitable comic strip: "Since our corporate strategy includes the word green all over it, we are selecting Erlang for our SOA implementation for the threads are green!"

Re:Scala

[TheRaven64]

Note that I said 'per process' - each process has its own LDT, and so each one can support 8K threads, so you can get 100K processes with 13 processes easily. This might not still be the case - implementing TLS using an extra register would avoid this limitation but would remove one GPR, and they are quite scarce on x86. The other option, updating the LDT every few thread context switches introduces a lot of TLB churn.

I quite agree that shared memory concurrency is a bad idea, however. Unfortunately, until you have message passing instructions in the hardware, you're stuck emulating message passing on top of shared memory, which leads to cache coherency issues and a host of other problems.

Re:Scala

[Jamie Lokier]

Linux threads stopped using the LDT on x86 in 2002. This change went mainstream over subsequent years, and is nowadays always used on x86.

There was once a limit on the number of processes, too, due to each process having an entry in the GDT. That has long been removed too.

Re:Scala

[Just Some Guy]

I ignored references to Scala until recently because I figured someone had updated the Amiga multimedia language, Scala, and I wasn't particularly interested in an updated version of something I'd never use.

Seriously, is it so hard to search for prior uses of a name these days?

Seen next month: Cobol, the interactive music description language.

Why Erlang Matters

[mpapet]

1. Multicore ready.
Erlang will use them. Write your application in Erlang and it's done for you.

2. Scales well.
As an example, http://yaws.hyber.org/ scales very nicely when loads increase. Your basic LAMP/LYMP setup runs much better on vanilla hardware.

3. Designed for telecom
The architects designed the language to run in a telecom environment so things like upgrades can be done while the application is running.

Yaws in particular needs your help. Failover clustering inside the yaws server would be wonderful. Right now, it uses CGI to process other languages. It does it flawlessly, but a more direct solution might be a nice project.

Re:Who wrote the summary? GWB?

[Anonymous Coward]

Well, are they?

Re:Who wrote the summary? GWB?

[Anonymous Coward]

Man this thread seems to have had a can of *whoosh* unleashed upon it. GP was quoting a GWB speech.

Comparison of functional languages?

[radarsat1]

I think the summary (and article) are somewhat poorly written, but that doesn't shadow the fact that functional languages are becoming more and more interesting these days with concurrency becoming so important.

I'd like to learn one, but there are several out there.. What I'd like to see is a good in-depth comparison of different concurrent functional languages: why would I choose Haskell, or Erlang, or OCaml, for example? Are they all interpreted? (Does one exist that compiles?) Which ones support concurrency? What language features do they boast, and what are the advantages and disadvantages of these features? Do they have a complete set of libraries?

Anyone know of an article like this? I've been searching for a while. Every article on functional languages I've found seems to concentrate on a particular one, but I can't find something helping me decide which one is most worth learning.

Re:Comparison of functional languages?

[AndyS]

Brief answer:

All three languages have both interpreters and compilers (ocaml is part of the base distribution, haskell has a number of compilers, and Erlang apparently has a compiler)

They all support concurrency, all in slightly different ways. They all have a lot of libraries.

Ocaml is sort of a functional language that includes object oriented features and also has very good performance numbers. It allows mutable updates, including arrays and references. For threading I believe it has the usual mutexes and so on, but nothing more sophisticated (but I could be wrong)

Haskell is a pure functional language. It tends to be a test bed for programming language ideas. It has some interesting features that can screw with your mind - it's lazy (which means that it only evaluates things when required), and pure (manipulating state can be interesting). It has mutex support, but also (in GHC) support for software transactional memory, which can be used to simulate erlang style concurrency.

Not really an expert on Erlang, but to my knowledge it pushes you to a 'mini-server' model, where you write each component as a mini server which then performs a single job, and you then spend most of your time sending messages to other processes. The Erlang system then distributes this across multiple machines for you and handles fault tolerance etc.

Re:Comparison of functional languages?

[Richard W.M. Jones]

OCaml compiles down to native code, which about 10-20% slower than C. Faster than C in a few (narrow) cases.

Haskell is also compiled to native code, but difficulties with the execution model mean it's pretty slow for any practical use.

Erlang is interpreted - the execution model is similar to Perl or Python - which means its slow on single cores, but of course the whole point of Erlang is to run in highly concurrent, distributed machines. There is a project to use OCaml for the performance-critical, single threaded parts, and Erlang for coordinating the parallelism.

Of course, this is probably missing the point. Unless you're doing intensive numerical work, you probably don't need the performance. The real advantage of these languages is how your code will be much smaller, easier to understand, safer, and faster to write.

Rich.

Re:Comparison of functional languages?

[GatesDA]

Can't point you to a comparison article, but one language you should consider is Scala. It compiles to the Java platform, and thus can interact almost transparently with existing Java code and libraries, and uses Erlang's concurrency model. It can do both functional and imperitive, object-oriented tasks. It's statically-typed, but with features I didn't think were possible outside a dynamic language, such as duck-typing (only compile-time checked!)

It's very powerful, but sometimes hard to figure out. Not my ideal language, but the closest I've found.

Official site:
http://www.scala-lang.org/

The busy Java developer's guide to Scala:
http://www.ibm.com/developerworks/views/java/libraryview.jsp?search_by=scala+neward

Scala for Java refugees:
http://www.codecommit.com/blog/scala/roundup-scala-for-java-refugees

Re:Comparison of functional languages?

[Anonymous Coward]

Just a minor correction: Erlang has native code compilation on quite a few architectures -- try the "+native" flag. Most projects seem content with just using the VM interpreter, though.

Best,
Thomas Lindgren

Re:Comparison of functional languages?

[shallot]

The real advantage of these languages is how your code will be much smaller, easier to understand, safer, and faster to write.

YMMV, but for the typical person, Erlang code is hardly easier to understand, at least compared to a typical imperative language. There are numerous obscure parts of the syntax - such as the pushing variables onto an array [like|this] or referencing unused arguments _likethis (especially fun in arcane constructs such as having [_|likethis] in function arguments) - and obviously the general problem of being unable to randomly create "normal" variables, as well as an unhealthy addiction to tail recursion.

In my programming classes, Erlang was one of those things that no students actually *learned* - they just forced themselves to use it because it was required, but once their grade was in the gradebook, nobody ever looked back.

Re:Comparison of functional languages?

[larry bagina]

Erlang is a functional language.

Re:Comparison of functional languages?

[bcrowell]

This is an excellent article about functional programming.

Ocaml has a very good reputation. It compiles to native code, and has performance comparable to C. It's agnostic about programming style. You can use it as if it was C with garbage collection, or you can us it in an fp style, or you can mix the two. There are good libraries. Because of the way the garbage collector is designed, it doesn't support SMP to the extent that erlang does. The design of ocaml includes some conscious trade-offs of elegance for performance. In particular, strings are not lists.

Erlang's claim to fame is that any code you write will automagically take advantage of SMP.

Haskell's design focuses on elegance, sometimes at the expense of performance. It's strongly focused on the functional style.

Re:Comparison of functional languages?

[TheRaven64]

Both of the things you mention are exactly the same as Prolog, so I would expect any programmer familiar with languages outside of the Algol family to have no problem with it. You apparently missed the fact that [this|Syntax] is for linked lists, not for arrays. The closest thing to an array in Erlang is the tuple.

Re:Comparison of functional languages?

[Anonymous Coward]

YMMV, but for the typical person, Erlang code is hardly easier to understand, at least compared to a typical imperative language.

Y supongo que después nos dirás que para la persona "típica," el español es un idioma difícil de comprender, al menos comparado con el inglés.

Re:Comparison of functional languages?

[jbolden]

Erlang = massively parallel systems
OCaml = most practical / mainstream

Haskell = best language to make sure you really "get it" because the constructs just don't exist to do it your old way.

LISP/Scheme = Most books by far. Best language for really seeing that code is data and data is code. Very old fashioned though.

Re:Comparison of functional languages?

[SnowZero]

Of course, this is probably missing the point. Unless you're doing intensive numerical work, you probably don't need the performance.

...or if you are a company with a large web presence. Using something that runs at half the speed of C++ is not a big deal if you only have 2 servers. However, if you have 1000, then its wasting a lot of money, and it's worth the extra programming time to tune the system. This is kind of why I like OCaml (though I don't get to use it at work), it's a very good balance of a mostly-functional language, speed, and pragmatic features.

Re:Comparison of functional languages?

[Anonymous Coward]

You're stupid.

Re:Comparison of functional languages?

[QuestionsNotAnswers]

One downside of Erlang and OCaml is poor native Unicode support. Erlang has a Unicode library, but from my experience I would prefer my strings to be a language feature (that is used consistently thoughout all libraries). Appears Haskell has it's problems with Unicode too.

Re:Comparison of functional languages?

[A Numinous Cohort]

Have you had a look at Clojure? It is a Lisp dialect that runs on the JVM with good Java interop and has built-in support for STM concurrency.

Deceptive

[sentientbrendan]

>The two biggest computing-providers of today, Amazon as well as Google, are building their concurrent offerings on top of really concurrent programming languages and systems

Google is largely a C++ company, a language that doesn't include explicit support for concurrency (although the next version, C++0x, will).

They mention erlang only being used in a relatively small project that most of google's own software doesn't support yet.

Note, that google gears is used in the excellent google reader software (although not much else).

Re:Deceptive

[IamTheRealMike]

Actually, Gears doesn't use Erlang either. What he means is that Gears threading doesn't allow for shared state (is it really threading then?). Instead threads communicate back to the browser by message passing.

It's remarkably deceptive indeed to even imply that Gears and Erlang are connected. Message passing based concurrency isn't exactly new or limited to Erlang, and can be implemented in any language.

I'm not sure what the point of this piece is. I've looked at Erlang and didn't see much of anything to get me excited. It's a functional language, which like most of them have unnecessarily weird syntax and force immutable state. I don't really see what this buys you over a language like D 2 (or hell, even C++) in which you can write in a functional message passing style if you like, but then still use imperative shared state whenever useful, convenient or performant.

Re:Deceptive

[Richard W.M. Jones]

... force immutable state

You're probably looking for an impure functional language like OCaml. You get to use purity when you want to (it helps in reasoning about code), but drop to impurity when you need the performance.

Rich.

Re:Deceptive

[IamTheRealMike]

Well, OK. Most functional languages I know require immutable state, but I'll look more closely at OCaml. How though, is it better than v2 of D, which appears to offer most things that functional languages offer (except perhaps an easy pattern matching syntax) but also a whole lot more?

Re:Deceptive

[Richard W.M. Jones]

I don't really know much about D, but OCaml nowadays has loads of libraries, excellent features like macros, type safety everywhere, pattern matching, a super-powerful object system (not used very much, mind you), functors, and it compiles down to fast, tight machine code. If you use Debian or Fedora you can just 'apt-get' or 'yum install' the whole system and almost all of the popular libraries.

Rich.

Re:Deceptive

[Richard W.M. Jones]

I should add that banning or confining mutable state is fairly important for efficient generational garbage collection.

This is often overlooked (for many programmers a good GC is "invisible"), but if you allow mutable state, then you allow for pointers which point from old heaps to young heaps. That complicates and slows down the garbage collection because you have to keep a special list of those pointers. With immutable data, no such pointers can ever exist. The OCaml compiler/runtime allows mutable data, but has to track which structures can be mutable and handle them specially.

Personally it's been ages since I used a mutable structure. Once you understand how to use immutable structures, they are just so much more convenient. Of course there's a big leap that has to happen in your head before you understand how to use them naturally. Functional programming has several big leaps like that though.

Rich.

Re:Deceptive

[jbolden]

Huh? Real Mike, you got me here. I'm hard pressed to even see how D is remotely like a functional language:
-- order of execution is explicit
-- functions have side effects
-- there are explicit looping structures
etc...

Re:Deceptive

[bar-agent]

Personally it's been ages since I used a mutable structure. Once you understand how to use immutable structures, they are just so much more convenient. Of course there's a big leap that has to happen in your head before you understand how to use them naturally. Functional programming has several big leaps like that though.

But immutable data seem really inefficient. Rather than altering one value in the data structure or collection, you have to copy the entire thing to a new location with one value changed! Now, granted, since the other values in the structure or collection are immutable, you can just alias the data from the old location to the new, but that is still a lot of pointer-copying and allocation.

Re:Deceptive

[Richard W.M. Jones]

Seems really inefficient. but may not be. It really depends on how much data you need to copy versus how much data you can share. And how fast copying is on your processor (probably very fast indeed). And whether your GC strategy can be made globally more efficient because of the immutable data. There's no single answer to this and naturally it depends on the precise application.

Rich.

Gibberish

[SpinyNorman]

If you want to build computing into a utility, you need large real-time systems running as sufficiently as possible.

But if you want to build sprockets into a weasel you need small batch-mode systems running as necessarily as possible.

If the poster had anything interesting to say (I'd guess not, but who knows!), it was totally obscured by his lack of grasp of the English language.

Re:Gibberish

[Deliveranc3]

by his lack of grasp of the English language.

Lol... (/English Major)

Wrong parenthesis added for humour :P

Re:Who wrote the summary? GWB?

[Anonymous Coward]

Oops, wrong link. I isn't learning.

http://www.youtube.com/watch?v=aAUrToY33tI

TFA is misleading

[Anonymous Coward]

TFA implicitely states that Google is using Erlang or some concurrent programming languages. That's wrong: they use C++, Java and Python, and prett much nothing else (apart for specialized stuff for MacOS apps for instance).

Re:TFA is misleading

[Ambidisastrous]

Google wrote their own language for parallelism called Sawzall. I'm not sure where they're using it in particular (possibly as a front-end to MapReduce), but they've published some research papers about it.

Too late

[Fnord666]

Anyhow, this post was not intended to be a rant about old-school technology solutions vs. current and future technology problems.

Given that this statement appears almost halfway through the blog post, I would say that it was already too late for that.

Why Erlang doesn't matter

[SanityInAnarchy]

1. Invariable variables.
This appears to have been done for no reason other than the designer's preference. In fact, it's not strictly true -- variables can be unbound, and later bound. They just can't be re-bound once bound.

2. Weird syntax.
Why, exactly, are there three different kinds of (required) line endings? It seems as though the syntax is designed to be as different from C as possible, while maintaining at least as many quirks. Moreso, even -- when constructing normal, trivial programs, you're going to hit most language features head-on and at their worst. Where's my 'print "hello\n"' that works most other places?

I don't believe the important features of Erlang are mutually-exclusive with the sane syntax of, say, Ruby or Python.

3. Not Unicode-ready.
Strings are defined as ASCII -- maybe latin1. But there's no direct unicode support in the language -- if you're lucky, there are functions you can pipe it through.

There are other things I haven't mentioned, mostly implementation-specific -- things like the fact that function-reloading cannot be done when you natively-compile (with hipe) for extra speed. My plan is to take the features I actually like from Erlang and implement them elsewhere, in a language I can actually stomach for its real tasks.

Re:Why Erlang doesn't matter

[Anonymous Coward]

Strings are defined as ASCII

IIRC, Erlang doesn't have actually have strings, it has character lists where every char is stored as a 32 bit int plus a pointer to the next char (8 bytes per char). If you're primarily concerned with processing text, it's the wrong tool for the job.

Re:Why Erlang doesn't matter

[zmooc]

Though I agree with you on 2 and 3, I'm not so sure about 1, but I might be wrong on that. As I understand it, you should look at variables in functional programming languages like Erlang more like those in a mathematical formula; such programs can be proven correct a lot easier, and since variables are effectively immutable, it facilitates forking the line of execution in a way that would not be possible without all kinds of semaphores and other concurrency stuff than if variables where not immutable. You should see this practice more like taking "pure functions" (http://en.wikipedia.org/wiki/Functional_programming#Pure_functions) to another level, namely to within functions themselves.

But I might be wrong, if so, please educate me:)

Re:Why Erlang doesn't matter

[stonecypher]

1) Actually, there are quite a few good reasons for this, largely around the complete elimination of mutexing and locks. Just because you don't understand the purpose doesn't mean there wasn't one.

2) Oooooh, a language is faulty because it has a syntax with which you are not familiar. Immediately kill all non-Java clones!

3) They're just lists of numbers; they're neither ASCII nor Latin1. There is unicode parsing in the XMERL module.

Please wait until you know a language before criticizing it.

Re:Why Erlang doesn't matter

[Anonymous Coward]

Erlang's syntax comes from Prolog. This is not so strange with the first versions of Erlang having been implemented in Prolog. The syntax is very natural. Like the english language you have . and , doing different things. How hard can it be?

Re:Why Erlang doesn't matter

[Anonymous Coward]

1. Invariable variables.
This appears to have been done for no reason other than the designer's preference. In fact, it's not strictly true -- variables can be unbound, and later bound. They just can't be re-bound once bound.

On the contrary, there are very good reasons for having single-assignment variables. It makes the code more similar to plain mathematics, which makes it easier to reason about, and significantly reduces the number of programming errors. And you don't have to take that from me - there are some 20 years of experience at Ericsson and elsewhere with writing huge telecom applications in Erlang.

2. Weird syntax.
Why, exactly, are there three different kinds of (required) line endings? It seems as though the syntax is designed to be as different from C as possible, while maintaining at least as many quirks. Moreso, even -- when constructing normal, trivial programs, you're going to hit most language features head-on and at their worst. Where's my 'print "hello\n"' that works most other places?

I don't believe the important features of Erlang are mutually-exclusive with the sane syntax of, say, Ruby or Python.

The syntax is certainly different from C, Ruby, or Python, but this is because it is derived from the Prolog syntax. Furthermore, it is actually pretty systematic, once you get over those initial differences. It is a poor programmer who cannot master both worlds.

3. Not Unicode-ready.
Strings are defined as ASCII -- maybe latin1. But there's no direct unicode support in the language -- if you're lucky, there are functions you can pipe it through.

A standard unicode library is still missing, but can be hoped for. At least, there is nothing in the basic representation of strings that prevents full unicode support (*cough* Java *cough*).

There are other things I haven't mentioned, mostly implementation-specific -- things like the fact that function-reloading cannot be done when you natively-compile (with hipe) for extra speed.

That's simply wrong. Dynamic code upgrade still works, native or not. It's the unloading of older native code from memory that is not being done (this is safe, but could be a memory leak in a very long running server).

My plan is to take the features I actually like from Erlang and implement them elsewhere, in a language I can actually stomach for its real tasks.

Well, good luck, and see you in 20 years. Meanwhile, the rest of us will be over here, getting stuff done with the language we have. For my part, I don't know anything else that lets me be as productive, at least for general problem solving.

Re:Why Erlang doesn't matter

[SanityInAnarchy]

As I understand it, you should look at variables in functional programming languages like Erlang more like those in a mathematical formula; such programs can be proven correct a lot easier, and since variables are effectively immutable

All of this is based on the premise that Erlang is a functional language. It's not purely-functional, and I just don't see the point of doing it half-assedly. Erlang is effectively an imperative language dressed up like a functional language.

And they're not immutable -- they can be unbound. As I understand it, this unboundedness is detected at runtime, not compiletime. If it was detected at compiletime, you'd have a valid point.

it facilitates forking the line of execution in a way that would not be possible without all kinds of semaphores and other concurrency stuff

Except that's not how Erlang does concurrency. It does concurrency with explicit "processes" (green threads) and message-passing.

Now, it does make these very easy, and you can get it to distribute processes among a few real OS threads (one per core) -- so it's still very cool. But you're thinking of languages like Haskell, which can be automagically threaded. Erlang is manually threaded, it's just much easier to think in threads (or "processes") -- they're effectively a language feature.

Re:Why Erlang doesn't matter

[SanityInAnarchy]

Actually, there are quite a few good reasons for this, largely around the complete elimination of mutexing and locks.

...What? No, the elimination of mutexing and locks is made possible by a shared-nothing architecture.

Oooooh, a language is faulty because it has a syntax with which you are not familiar.

Hey, I mentioned Ruby. I don't mind LISP, either.

The point is not that the language is unfamiliar, the point is that it's inconsistent (and unfamiliar) for no good reason. I use English, but I could make a lot of the same criticisms about it.

They're just lists of numbers;

In that case, the argument becomes, "Erlang has very poor text-processing, if any at all."

If Erlang has text-processing functions that are designed to operate on these "lists of numbers", then yeah, it's pretty much going to be ASCII. And how are Erlang source files read? Could be "neither ASCII nor Latin1" if you like, but they can't be Unicode unless the parser is actually Unicode-aware.

Re:Why Erlang doesn't matter

[SanityInAnarchy]

Furthermore, it is actually pretty systematic, once you get over those initial differences. It is a poor programmer who cannot master both worlds.

I've written a decent amount of Erlang. I can do it.

Point is, why would I want to?

That's simply wrong. Dynamic code upgrade still works, native or not. It's the unloading of older native code from memory that is not being done (this is safe, but could be a memory leak in a very long running server).

Ah, you're right.

I do consider it a fairly severe deficiency that you get to choose between interpreted bytecode and a memory leak.

Well, good luck, and see you in 20 years.

Done, in Ruby. Took a weekend. I'm polishing it now.

Re:Why Erlang doesn't matter

[SQL Error]

2) Oooooh, a language is faulty because it has a syntax with which you are not familiar.

Yes.

Where is Lisp today? Smalltalk?

On the other hand, languages that offered the same features with a familiar syntax have taken over the market.

Re:Why Erlang doesn't matter

[daviddennis]

Since strings are just sequences of ordinary numbers, it would seem like you could develop Unicode by just using numbers in unicode character code range. A string could be a tuple with the encoding and the sequence, like this:

{ utf8, [ character1, character2 ... ] }

After reading through the documentation, my main gripe is that it's a totally different way of thinking, and so I have no clue of the first way to do what is accomplished with routine ease in other languages. The Erlang getting started guide admits that there are many things that would be, by their own admission, "impossible".

For example, the tutorial says that "guards" (conditional execution for you and me) cannot use user defined functions, to guarantee there are no side effects. Great in theory but what if you want to develop a regular expression library? How would it even be possible to do?

So what's the advantage of this language over, say, using threads in a language like Ruby or Perl? It seems to me that it's extremely difficult to master and lacks features that exist in virtually every interpreted language in existance today.

D

Re:Why Erlang doesn't matter

[aconbere]

Actually, there are quite a few good reasons for this, largely around the complete elimination of mutexing and locks.

...What? No, the elimination of mutexing and locks is made possible by a shared-nothing architecture.

Oooooh, a language is faulty because it has a syntax with which you are not familiar.

Hey, I mentioned Ruby. I don't mind LISP, either.

The point is not that the language is unfamiliar, the point is that it's inconsistent (and unfamiliar) for no good reason. I use English, but I could make a lot of the same criticisms about it.

It's not that it's syntax is /inconsistent/ Erlang is actually incredibly consistent, it's just very different. Once you learn the 3 or 4 quirks that separate it from other languages those 3 or 4 quirks are very consistently applied.

Take for instance the punctuation (not line ending characters as is suggested).

Commas separated arguments in function calls, data constructors, and patterns. Periods separate functions.

Semi-Colons separate clauses. (this is the trickiest, but can be thought of as signifying the existence of multiple cases of pattern matching).

They're just lists of numbers;

In that case, the argument becomes, "Erlang has very poor text-processing, if any at all."

If Erlang has text-processing functions that are designed to operate on these "lists of numbers", then yeah, it's pretty much going to be ASCII. And how are Erlang source files read? Could be "neither ASCII nor Latin1" if you like, but they can't be Unicode unless the parser is actually Unicode-aware.

Re:Why Erlang doesn't matter

[aconbere]

As I understand it, you should look at variables in functional programming languages like Erlang more like those in a mathematical formula; such programs can be proven correct a lot easier, and since variables are effectively immutable

All of this is based on the premise that Erlang is a functional language. It's not purely-functional, and I just don't see the point of doing it half-assedly. Erlang is effectively an imperative language dressed up like a functional language.

OH come on this is bullshit. Even lisp and scheme aren't "purely functional" the fact that the actor model breaks the functional model clearly makes almost no difference in practice.

And they're not immutable -- they can be unbound. As I understand it, this unboundedness is detected at runtime, not compiletime. If it was detected at compiletime, you'd have a valid point.

it facilitates forking the line of execution in a way that would not be possible without all kinds of semaphores and other concurrency stuff

Except that's not how Erlang does concurrency. It does concurrency with explicit "processes" (green threads) and message-passing.

Now, it does make these very easy, and you can get it to distribute processes among a few real OS threads (one per core) -- so it's still very cool. But you're thinking of languages like Haskell, which can be automagically threaded. Erlang is manually threaded, it's just much easier to think in threads (or "processes") -- they're effectively a language feature.

Re:Why Erlang doesn't matter

[edalytical]

Lisp: Emacs, Undergrad Computer Science, AI research...

Smalltalk: Objective-C, Squeak, Seaside, F-Script...

Re:Why Erlang doesn't matter

[Anonymous Coward]

I've written a decent amount of Erlang. I can do it.

Point is, why would I want to?

Given the gross misunderstandings demonstrated in your comments, you may have written some Erlang, but you sure as hell didn't understand it

Done, in Ruby. Took a weekend. I'm polishing it now.

You wrote a SMP-capable micro-stack green thread scheduler in Ruby, over the weekend? It supports transparent multi-node messaging?

Color me impressed. I've seen other implementations, such as libthread, but they were written in C, using a fair bit of OS-specific assembly to support micro-stack green threads, and didn't support multi-node messaging -- just process support

If you pulled this off in Ruby, which doesn't even -support- native threads, I'll be seriously impressed -- especially given the lack of understanding inherent in your previous comments. Perhaps I misjudged you

Or, more likely, you're just another know-it-all shithead who implemented 1% of the solution to a problem you don't even understand, and are now trumpeting it on the blogowww-twittersphere.

Re:Why Erlang doesn't matter

[magus_melchior]

2) Oooooh, a language is faulty because it has a syntax with which you are not familiar. Immediately kill all non-Java clones!

What makes this rebuttal a ridiculous straw man is the fact that GP didn't even mention Java. He did mention C (with some derision), Ruby, and Python.

Here's a clue: what is familiar to you (Erlang) may have syntactic inconsistencies. Okay, so you took the time and effort to learn it, and more power to you. However, because syntax is a design decision rather than an implementation decision, it says a whole lot about the design when there are three different ways to end a statement. Perhaps next time, you could explain why Erlang was designed that way instead of responding with a useless hyperbole.

Re:Why Erlang doesn't matter

[Anonymous Coward]

I think you don't understand "shared-nothing". Immutable variables and objects are the core of a shared-nothing architecture. When your variables are immutable, they are not "shared state", and there can be no race conditions in accessing them.

You can have a "shared-nothing architecture" in a language with mutable variables by establishing and enforcing conventions, but then you have to think about it all the time and trust other people not to mess it up. In a language like Erlang, where all variables are immutable, a shared-nothing architecture is automatic. Your code (and more importantly, other people's code) is always automatically thread-safe and parallelizable.

Considering that thread safety is perhaps the hardest thing to achieve in traditional languages, and concurrency is the next hot topic in computer science, having it built into the language starts to sound like a good idea.

Re:Why Erlang doesn't matter

[SanityInAnarchy]

You wrote a SMP-capable micro-stack green thread scheduler in Ruby, over the weekend?

Nope. I used the existing Thread API. It is theoretically, but not actually, SMP-capable. It should be portable to JRuby, which uses Java threads.

What I did was port the message-passing to a Ruby paradigm -- so, much simpler than it sounds. Essentially, I wrote a proxy for objects -- every method call to such an object stacks up in a queue, and is sent (in the object's thread).

It's not so much a direct port of Erlang as a mapping of what I found valuable in the Erlang philosophy. A class can be written with almost no thought to concurrency, and then used concurrently.

It supports transparent multi-node messaging?

No, Ruby supports that on its own, with DRb.

What I haven't figured out is the exception handling.

If you pulled this off in Ruby, which doesn't even -support- native threads

Ruby 1.9, which does. Unfortunately, it takes a page out of Python and uses a global interpreter lock, so only one Ruby bytecode can be executing at a time -- but again, I believe JRuby does real (actually concurrent) threads.

Or, more likely, you're just another know-it-all shithead who implemented 1% of the solution to a problem you don't even understand, and are now trumpeting it on the blogowww-twittersphere.

That's exactly why I'm not -- no blog, no big announcements until it's done.

Re:Why Erlang doesn't matter

[SanityInAnarchy]

OH come on this is bullshit.

Which seems to be your way of saying "exactly right."

Even lisp and scheme aren't "purely functional"

Right. Haskell is.

Re:Why Erlang doesn't matter

[TheRaven64]

...What? No, the elimination of mutexing and locks is made possible by a shared-nothing architecture.

To eliminate locks, you need to satisfy the condition that no data is both aliased and mutable. Erlang does this by making sure that the only mutable data structure is the process dictionary, which is not aliased. This means that in a shared-memory system message passing is just pointer copying.

The other issue, that variables are single static assignment, is due to the fact that Erlang evolved from Prolog, where there are no variables, just unbound terms.

Re:Why Erlang doesn't matter

[TheRaven64]

What is the connection between writing a regular expression library and guards? Think of guards as an optional type system - by default Erlang is untyped, but you can provide constraints on input to clauses using guards. If you want a regular expression library then you just need to define a set of functions for processing regular expressions - you might do this by creating a DFA which corresponds to the regex, with an Erlang process for each node and have them consumer the input by each consuming the head character from a string and then passing the rest to the next node depending on the contents, for example. Or you might use a more functional approach, depending on the amount of concurrency you need.

So what's the advantage of this language over, say, using threads in a language like Ruby or Perl? It seems to me that it's extremely difficult to master and lacks features that exist in virtually every interpreted language in existance today.

Scalability. I've written code in Erlang that scaled happily to a 64-CPU machine without even thinking much about concurrency. Try doing that in Ruby or Perl. It's trivial to master - the syntax is close to Prolog and the semantics are functional or CSP depending on the scope. Any half-decent programmer should be able to learn Erlang in an afternoon.

Re:Why Erlang doesn't matter

[Anonymous Coward]

Damn, you're absolutely right, only products that have taken over the market are worth using... quick pass me a copy of Windows and give me a VBscript reference manual, burn all the other language and OS texts I have, after all it's all just academic nonsense right?

Really, who gives a flying fuck if lisp and smalltalk aren't used everywhere, in many ways they are like Unix; people in the know use them, and the rest of this half-baked industry that force feeds everyone an elvis sized diet of Windows and Flash and other all the other frivolous shite that gets passed off as high tech these days can go fuck 'em selves while we get on with the job of building the infrastructure that supports everyone's daily lives. You know the stuff I mean, stockmarkets, nuclear power control systems, fly-by-wire for passenger aircraft, the stuff that can't afford to be simply good enough, it has to be provably correct.

Re:Why Erlang doesn't matter

[Anonymous Coward]

Where is Lisp today? Smalltalk?

Running Orbitz.com and DabbleDB. Lisp also allowed Paul Graham to make a lot of money.

Re:Why Erlang doesn't matter

[Anonymous Coward]

You wrote a SMP-capable micro-stack green thread scheduler in Ruby, over the weekend?

Nope. I used the existing Thread API. It is theoretically, but not actually, SMP-capable. It should be portable to JRuby, which uses Java threads.

What I did was port the message-passing to a Ruby paradigm -- so, much simpler than it sounds. Essentially, I wrote a proxy for objects -- every method call to such an object stacks up in a queue, and is sent (in the object's thread).

It's not so much a direct port of Erlang as a mapping of what I found valuable in the Erlang philosophy. A class can be written with almost no thought to concurrency, and then used concurrently.

Your class can't be used concurrently, because Ruby is not actually SMP-capable. It can't be used on multiple systems, because the messaging can't be used across nodes.

Single-machine, non-concurrency queue-based message passing systems are what we usually call "linked lists". Comparing your hobby-horse linked-list message queues to Erlang is foolish hubris.

It supports transparent multi-node messaging?

No, Ruby supports that on its own, with DRb.

DRb is just another RPC implementation. By your logic, I could call Python's xmlrpc libraries an implementation of transparent multi-node concurrent messaging.

However, you missed the "transparent" and "concurrent".

In erlang, processes and (immutable) messages are first class entities, whether they're operating locally or remotely. Sending a local process a message is the same as sending a remote process a message. The message and protocol formats are well-defined, such that one can implement an Erlang "node" in any language -- including Java, Ruby, or C. Unlike most RPC systems, Erlang doesn't attempt to hide (or ignore the possibility of) underlying network/system failure, but instead, is built to handle failure through the use of supervisors.

A language-specific reflective RPC library doesn't really count as a replacement, though it does make a pat answer.

What I haven't figured out is the exception handling.

You mean one of the some of the most complex pieces of code in Erlang -- the software that manages a supervision tree of processes, handling failure across the local system as well as remote nodes?

Good luck with that

If you pulled this off in Ruby, which doesn't even -support- native threads

Ruby 1.9, which does. Unfortunately, it takes a page out of Python and uses a global interpreter lock, so only one Ruby bytecode can be executing at a time -- but again, I believe JRuby does real (actually concurrent) threads.

So no concurrency.

Or, more likely, you're just another know-it-all shithead who implemented 1% of the solution to a problem you don't even understand, and are now trumpeting it on the blogowww-twittersphere.

That's exactly why I'm not -- no blog, no big announcements until it's done.

Then please stop commenting on these systems, because I don't think you're going to understand what makes this problem space so complex until you've actually finished a solution for the whole space, instead the small percentage of it that you actually understand.

Re:Why Erlang doesn't matter

[Jack9]

http://www.paulgraham.com/avg.html

Re:Why Erlang doesn't matter

[aconbere]

OH come on this is bullshit.

Which seems to be your way of saying "exactly right."

It's my way of saying that even in languages that have pushed the boundaries of the functional paradigm, this doesn't matter.

The reality is that functional programing even when "impure" has proven to be a highly effective model, and pretending like impurities within the model discredit that, is just silly.

Even lisp and scheme aren't "purely functional"

Right. Haskell is.

Re:Why Erlang doesn't matter

[SanityInAnarchy]

Your class can't be used concurrently, because Ruby is not actually SMP-capable.

But JRuby is.

I should note that, in a simple test, Erlang got about three times slower when I enabled SMP mode.

It can't be used on multiple systems, because the messaging can't be used across nodes.

Except it can, with DRb. Which also provides sloppy SMP support -- just run two nodes on the same machine.

Single-machine, non-concurrency queue-based message passing systems are what we usually call "linked lists".

Erm, what? I know what a linked list is, and I don't see the comparison.

DRb is just another RPC implementation.

...and Erlang's RPC isn't?

In erlang, processes and (immutable) messages are first class entities, whether they're operating locally or remotely.

Again, I have to go, "huh?"

DRb allows objects to be created and "passed" between the machines. I could do:

remote_object = DRB::whatever
remote_object.some_method :arg1, :arg2 ...

How is this different than Erlang's PIDs, other than being syntactically cleaner?

Sending a local process a message is the same as sending a remote process a message.

Yep. Got that, as demonstrated above. The fact that Ruby uses method_missing to accomplish this is an implementation detail.

The message and protocol formats are well-defined, such that one can implement an Erlang "node" in any language

DRb is perhaps not as well-defined -- I think most Rubyists would rather speak an appropriate protocol, rather than a language-specific RPC.

And yes, Erlang's RPC is language-specific -- it supports the features needed to run Erlang programs. Implementations in other languages doesn't make it natural to the other language.

You mean one of the some of the most complex pieces of code in Erlang -- the software that manages a supervision tree of processes, handling failure across the local system as well as remote nodes?

Yeah. I'm not sure if I want to duplicate that, or find something simpler, though I still don't see it as incredibly complex. The trick is to make it simpler for the end-user (application programmer) -- I'm not convinced Erlang does that.

So no concurrency.

Congrats for not reading. Allow me to summarize for your short attention span: JRuby.

Then please stop commenting on these systems

Why should I?

This is Slashdot. If you want more intelligent comments, educate me or mod me down.

Re:Why Erlang doesn't matter

[jbolden]

Actually if you read the LISP documentation they agreed that being purely functional was a huge advantage. SICP for example spends a long time talking about how dangerous any kind of side effect is, and trying to justify why one would use an environmental evaluation (object oriented for example) vs a purely functional method. They also talk about the advantages of laziness vs. eagerness which is another big difference between Haskell and Erlang.

In the end LISP went with eager to simplify computation (its from the early 50s). Until monadic programming was invented too many problems (like a random number generator) were just too complex. I don't know they would make the same choices today.

Re:Why Erlang doesn't matter

[Anonymous Coward]

1) unbound variables are detected at compile time.
2) when passing data to another local thread, the fact that the data (like a list, tuple, or binary) can't change means the RT can pass it by reference. Same for closures

Re:Why Erlang doesn't matter

[jbolden]

Where's my 'print "hello\n"' that works most other places?

Well
 
#!/usr/bin/env escript
main(_) -> io:format("Hello World\n").

And for something less trivial here is 99 bottles of beer:
simple more standard erlang

Re:Why Erlang doesn't matter

[jbolden]

Where is LISP. Slowly taking over all the other languages. More and more ideas are incorporated into mainstream languages with every generation (say a decade). The only thing missing from most scripting languages now is the interchangeability of code and data.

Re:Why Erlang doesn't matter

[jbolden]

So what's the advantage of this language over, say, using threads in a language like Ruby or Perl? It seems to me that it's extremely difficult to master and lacks features that exist in virtually every interpreted language in existance today.

Because in practice if you don't stay pure you'll end making assumptions that are false in a multi execution system. It is very hard for people to picture all things that go wrong if the environment their code executes doesn't execute in the same order they expect.

The whole industry of relational databases exists because once we were out of batch: Read a record, modify, write was too difficult. The issue of what happens if the record has changed since you started, made VSAM type (COBOL) structures impossible. So we invented a whole application to handle record locking and a whole sub programming language.

The goal is to not have to do that every time you want to be safely parallel for some object in the universe.

Re:Why Erlang doesn't matter

[Anonymous Coward]

Your class can't be used concurrently, because Ruby is not actually SMP-capable.

But JRuby is.

I should note that, in a simple test, Erlang got about three times slower when I enabled SMP mode.

JRuby is a non-standard Ruby implementation. From their feature list: "Most builtin Ruby classes provided". Emphasis mine. Additionally, see below why support for OS threads isn't enough to implement M:N actor:thread scheduling.

As for Erlang SMP speed -- first, they implemented SMP support. Then, they started to optimize it -- It has the architecture necessary to improve performance. Ruby, in contrast, still has a GIL, which means every single component and extension will need to be reviewed for thread-safety before the GIL can be removed (which means, never).

It can't be used on multiple systems, because the messaging can't be used across nodes.

Except it can, with DRb. Which also provides sloppy SMP support -- just run two nodes on the same machine.

How many "nodes" are you going to run on the same machine? Each ruby process consumes a great quantity of resources, the elimination of that resource drain is the entire purpose behind Erlang's tiny-stack microthreads -- they consume very few resources, so you can have a great many of them.

Single-machine, non-concurrency queue-based message passing systems are what we usually call "linked lists".

Erm, what? I know what a linked list is, and I don't see the comparison.

You've created a dead-simple message queue based on delivery of messages to a thread-owned linked-list. Erlang implements externally scheduled pattern-based actor message delivery, using thread-scheduled microstacks to allow for an enormous number of concurrently deliverable processes. One key element of these implementations is that they do NOT consume a thread per actor. Since you're so up on JRuby, I recommend Philipp Haller's papers on his actor library for Scala.

Assuming the implementation of thread-scheduled actor message delivery in Ruby (rather than JRuby), you'll still be stuck with the GIL, making the whole exercise, well, moot. If you use JRuby, you still don't control scheduling of microthreads, leaving you with 1:1 correspondence between an executing actor and a real thread, whereas Erlang is capable of maintaining M:N scheduling of *executing* actors to real threads.

This M:N correspondence truly matters -- without it, blocking operations inside of an actor mean that the entire thread is blocked and useless for other purposes. It's not possible to implement the M:N model without control over your execution stack, which means true microthreads are unimplementable in both JRuby and Ruby.

DRb is just another RPC implementation.

...and Erlang's RPC isn't?

No. RPC is a limited form of message passing -- RPC is something you build ON TOP OF the message passing model. If you skip the "message passing" step and go straight to RPC, you wind up with something that ignores the complexity of the network -- the network fails, messages get lost, dropped, or ignored. Given this ignorance of the medium, RPC interfaces assume synchronous message delivery and response, which then precludes asynchronous delivery and response.

In erlang, processes and (immutable) messages are first class entities, whether they're operating locally or remotely.

Again, I have to go, "huh?"

DRb allows objects to be created and "passed" between the machines. I could do:

remote_object = DRB::whatever remote_object.some_method :arg1, :arg2 ...

How is this different than Erlang's PIDs, other than being syntactically cleaner?

Re:Why Erlang doesn't matter

[daviddennis]

I was under the impression from reading the tutorial that guards were the only way to create an IF style conditional in the languge and so no user defined functions in guards meant no user defined function in the IF statement (which I did notice exists) as well. Did I get this wrong?

Oh, it's definitely easy to learn. I could certainly learn the language, in the sense of being able to write small programs with it, in a short time.

But the methods needed to create more complex software seem very alien to me. The way it looks right now is that problems are broken into tiny steps and it seems like it would be very hard to get it to do something at all complex. And yet as you say entire complex telephone systems are written in it.

Are there CGI or web form programs written in it anywhere? Seeing something like that would be a good way for me to understand the language a bit better, although I suppose it makes little sense in terms of concurrency.

Thanks for your response!

D

Re:Why Erlang doesn't matter

[TheRaven64]

Erlang if statements can only be done using guard clauses, but equality is a guard clause and you can use the result of other functions in guard clauses, so you can call your own functions and branch on the results. You can also match patterns in a very powerful way.

I don't know of any CGI or web form programs - there's a web server written in Erlang and the most popular (and most feature-complete) Jabber server is written in it, and this is probably a good place to look if you want some example code.

Re:Why Erlang doesn't matter

[speedtux]

1) Actually, there are quite a few good reasons for this, largely around the complete elimination of mutexing and locks. Just because you don't understand the purpose doesn't mean there wasn't one.

The existence of mutable variables doesn't mean that you need locks. You can use Erlang-style message passing between threads, and still have the convenience of mutable variables where it is safe.

2) Oooooh, a language is faulty because it has a syntax with which you are not familiar. Immediately kill all non-Java clones!

Well, this particular syntax comes from Prolog, and it wasn't any good in Prolog either.

3) They're just lists of numbers; they're neither ASCII nor Latin1. There is unicode parsing in the XMERL module.

That's another thing Erlang inherited from Prolog. Given the importance of strings, representing them as lists of numbers is ridiculous. And properly processing unicode requires a lot more than just "parsing".

Re:Why Erlang doesn't matter

[Anonymous Coward]

All of this is based on the premise that Erlang is a functional language. It's not purely-functional, and I just don't see the point of doing it half-assedly. Erlang is effectively an imperative language dressed up like a functional language.

Actually, much less so than e.g. O'Caml, Lisp and most Schemes. In Erlang, the value referred to by a variable can never be changed "under your feet". All mutable state is modelled (conceptually) as if belonging to some process.

And they're not immutable -- they can be unbound. As I understand it, this unboundedness is detected at runtime, not compiletime. If it was detected at compiletime, you'd have a valid point.

What on earth are you talking about? I can only tell you that "as you understand it" is wrong. You may have got it mixed up with the interactive shell, in which you can "unbind" variables for convenience, but that is not a feature of the programming language.

It seems to me that you are loudly criticizing language features that you barely understand.

Re:Why Erlang doesn't matter

[Anonymous Coward]

Where is LISP. Slowly taking over all the other languages. More and more ideas are incorporated into mainstream languages with every generation (say a decade). The only thing missing from most scripting languages now is the interchangeability of code and data.

But that's the point. If you add the good features of uncommon languages to common languages, why continue to use the uncommon languages? To make code more unreadable to the many coders who don't know every esoteric language? Bragging rights? Give me a break.

Re:Why Erlang doesn't matter

[jbolden]

Well no to use the advantages of these esoteric languages today and not 25 years from now. The people using LISP in the 1950s hard garbage collection, reentrant functions, complex data structures....

Re:Why Erlang doesn't matter

[krog]

I have two points to make.

First, Lisp's syntax and semantics allow the programmer to manipulate code and data using the same structures, functions and parsers. That is huge in its own right, allowing really powerful compile-time macros as well as all sorts of metaprogramming that may be possible, but is not quite as straightforward in other languages. That is but one reason why someone might use a Lisp, as opposed to a language which merely appropriated 85% of Lisp's cool features.

Second, I have absolutely zero sympathy for programmers who can't walk up to a new language and understand most of what's going on within a few minutes, given some reference materials. (And that zero goes double for any language that remotely looks like C or Fortran.) Of course true expertise takes time, but the concepts being expressed in all but the most esoteric corner languages are the same. A programmer who can't comfortably read (most) languages is either no good, or green as grass, or just not trying.

Re:Why Erlang doesn't matter

[SanityInAnarchy]

How many "nodes" are you going to run on the same machine?

If the only problem is lack of SMP support, I'll run as many as I have CPUs. Or fewer, given that the target app doesn't need resources.

Each ruby process consumes a great quantity of resources, the elimination of that resource drain is the entire purpose behind Erlang's tiny-stack microthreads

Ruby 1.9 has "fibers". (Disclaimer: I haven't read anything beyond that they exist.)

You've created a dead-simple message queue based on delivery of messages to a thread-owned linked-list.

That's the implementation, right. I'm doing the simplest thing that can possibly work, and I'm trying to spec out the API as much as possible in the meantime.

Erlang implements externally scheduled pattern-based actor message delivery

I actually have no idea what that means. Trying to deconstruct...

"Pattern-based" -- I figure that most often, the patterns are used to fork based on the "type" of message received, where "type" is explicitly stated in the message. Given that, I'm simply sending arbitrary method calls and branching that way. If people want to do something more advanced, they can implement method_missing, or have the method itself branch based on whatever.

"actor" -- from Wikipedia:

in response to a message that it receives, an actor can make local decisions, create more actors, send more messages, and determine how to respond to the next message received.

Sounds like exactly what I've created; an object attached to a thread. The difference is, mine uses OO primitives, and I like Ruby syntax, whereas Erlang uses functional primitives, and I don't like Erlang syntax.

One key element of these implementations is that they do NOT consume a thread per actor.

That's, again, an implementation detail. I'm building the simplest thing that can possibly work, and I'm also trying to write specs -- where I can, I'm trying to make it the least predictable that will meet the spec.

Once that's done, I can tune the implementation. I don't believe Ruby will forever be incapable of doing proper concurrency; Python actually likes the GIL, whereas Ruby seems to be doing it for the benefit of C extensions. Therefore, any non-C implementation (JRuby, IronRuby), or any alternative implementations (Rubinius), won't be so constrained.

If you skip the "message passing" step and go straight to RPC, you wind up with something that ignores the complexity of the network -- the network fails, messages get lost, dropped, or ignored.

Which causes an exception to be raised, just as with message passing in Erlang. (Again, correct me if I'm missing something.)

Given this ignorance of the medium, RPC interfaces assume synchronous message delivery and response, which then precludes asynchronous delivery and response.

Asynchronous delivery is reasonably possible -- in fact, for the system I'm building, it's the default. Synchronicity is easy, because I want to make the easy things easy, but it requires an additional method call -- for example:

object.method # some intermediate object
object.method.now # actual result
object.method.then{...} # callback

you need to version your objects (or upgrade your entire network in lockstep)

True of protocols, also. (Keep in mind, calling any method on a DRb "proxy object" will result in a message being sent to the other end, where the real object resides.)

It's an error to conflate messages with objects

Actually, objects are components of the message here. The message is a method call.

I read it, you're just missing the point. JRuby has no microthreads and isn't Ruby. I could say that IronPython provides t

Re:Why Erlang doesn't matter

[Free the Cowards]

Just which popular languages offer the same features as Lisp and Smalltalk? I'm dying to know because I'd love to start using them. Most of the popular languages I know of are horrible, and the non-horrible ones still don't come close to offering the same features as Smalltalk or Lisp.

Re:Why Erlang doesn't matter

[icepick72]

...What? No, the elimination of mutexing and locks is made possible by a shared-nothing architecture.

Not responding specifically to your post, but to the thread and the article. This place seems fitting.

I call bullshit on the concept that: if data is not shared we can get rid of locks. Message passing occurs between objects, objects come from classes and classes are data types; therefore your objects that are doing the message passing are data (complex, encapsulated, maybe inherited, maybe using composition) and you need locks inside an object to ensure two simultaneous calls (or message passes) to it do not ruin each other.

Re:Why Erlang doesn't matter

[SageMusings]

Lighten up, Francis.

Re:Why Erlang doesn't matter

[landonf]

As I understand it, you should look at variables in functional programming languages like Erlang more like those in a mathematical formula; such programs can be proven correct a lot easier, and since variables are effectively immutable

All of this is based on the premise that Erlang is a functional language. It's not purely-functional, and I just don't see the point of doing it half-assedly.

Of course Erlang is not purely functional. If it was, the only thing it could do is warm up your CPUs, and even that is technically a side-effect. A compiler could optimize away your whole purely-functional application and not produce any output.

I'm also not really sure what you mean be "half-assedly" in reference to purely-functional -- a full system that supports no side-effects wouldn't actually be useful for anything.

To your earlier point, Erlang doesn't support unbinding variables outside of the interactive shell. I think that you've misunderstood the meaning of an unbound variable, or I've misunderstood what you mean by "they're not immutable -- they can be unbound".

Re:Why Erlang doesn't matter

[fnc]

The first two problems are based on the early implementations that were made in Prolog.

The weird syntax is basically result of getting Prolog without unification plus processes, but I don't think is that bad (I first hated it, but after some use I liked it more than Lisp, basically because I like pattern matching more than using car and cdr).

The invariable variables are also from this source. In logic programming, variables can be bounded or not, the implementation search for possible assignments. I'm not sure if these are particulary usefull for Erlang, but similar variables are used by Oz programming language for to obtain deterministic concurrency, also known as dataflow computing, that is a lot easier to reason about and programming than full concurrency.

What makes Erlang relevant is a solid implementation, a well tested VM and many libraries/utilities for the context of distributed programming (unfortunally text processing is not in this niche); and excellent documentation (the concurrent design patterns that they documented and implemented are valuable even if you don't want to program in Erlang).

Re:Why Erlang doesn't matter

[Yenya]

But there's no direct unicode support in the language -- if you're lucky, there are functions you can pipe it through.

3) They're just lists of numbers; they're neither ASCII nor Latin1. There is unicode parsing in the XMERL module.

Which is exactly the problem that GP discussed. There is a huge difference between a true in-language support of Unicode (such as the one in Perl) and just "the Unicode parsing library". In Perl there is a difference between "string of bytes" and "string of characters", and this distinction is made when the string is created (i.e. in the I/O layer when it is read from the file, or in the source code pragma when it is a literal constant). And then all things work as expected (conversion between upper and lower case, word boundaries, string length in bytes or in characters, etc.) - differently on a character string than on a byte string.

This means, that the only place you have to actually "handle" unicode is the point where the data enter your program or leave it. All the other things (libraries etc) just magically work without needing a change. This cannot be easily done without an in-language support (C or maybe Erlang).

Re:Why Erlang doesn't matter

[Rysc]

Mad props to you for the unheralded Stripes reference.

Re:Why Erlang doesn't matter

[pbaer]

I just tested this 1 is wrong. Variables can NOT be bound unbound and rebound. You are thinking of the f() function which *only* works in the interactive emulator (erl). It's purpose is to stop you from running out of sane variable names while trying things out. The f/0 and f/1 only work inside the shell, if you try to call it in a compiled program it does nothing and you will get an error.

Re:Why Erlang doesn't matter

[Anonymous Coward]

Smalltalk is not in use anywhere. Please do not attempt to learn it. Move on, the coffee is ready. Go go. Shoo.

Re:Why Erlang doesn't matter

[Anonymous Coward]

1) Matching and memory (gc) efficiency.

2) Erlang is a functional language. The "weird" syntax comes from correct syntax. It is a learning curve involved learning correctness.

3) Unicode is currently in the works.

Re:Why Erlang doesn't matter

[SanityInAnarchy]

I'm sorry, that sentence really wasn't clear...

I meant that variables, once bound, cannot be unbound and then rebound. My point was that since a variable can first be unbound, and later bound, I don't really see the point to making them "unbound-then-bind-once" instead of "bind-anytime".

I understand why data structures are immutable, though that's annoying. I can sort of understand why one might want to make them "bind-once", where the compiler never allows a variable to be unbound. I don't understand why they can have an unbound state, and then be bound, but not later re-bound.

Re:Why Erlang doesn't matter

[SanityInAnarchy]

Matching and memory (gc) efficiency.

That explains immutable data structures (though I still wish they could be copy-on-write), but not bind-once variables.

Erlang is a functional language.

Sort of. Not really. In the half-functional state it's in now, I'd much rather have a rich imperative language with the same pervasive share-nothing message-passing.

Unicode is currently in the works.

Lemme know when it's done. Until then, it doesn't really count. From other comments I've read here, it doesn't look particularly promising, and certainly not backwards compatible.

Re:Why Erlang doesn't matter

[SanityInAnarchy]

What on earth are you talking about?

Apparently, my English syntax is sloppy here. I mean that they start in an unbound state, and can later be bound.

Is this unbound state detected at compiletime?

Re:Why Erlang doesn't matter

[SanityInAnarchy]

Message passing occurs between objects

Technically, processes. I've also heard them called "actors".

objects come from classes and classes are data types

Erlang supports neither objects nor classes, in the sense you're thinking.

you need locks inside an object to ensure two simultaneous calls (or message passes) to it do not ruin each other.

At a very low level, possibly. But those message passes, in Erlang, are actually being sent to a "process", not an object -- and events within a process are entirely single-threaded. That is, messages will pile up in a "mailbox", and the process is usually in a loop processing messages out of its mailbox.

I've modeled something similar in Ruby. I use objects, but each object has one Ruby Thread associated with it. If I dig deep enough, I find that the Queue structure I'm using to send messages does, in fact, use locks -- but that's an implementation detail.

What I think is vastly more important than having no locks, anywhere (which, as you suggest, may not be possible) is to hide the locks from the programmer. Think of them as GOTOs -- sure, on a fundamental level, there are jump commands being executed by the CPU all the time, and I'd almost argue that understanding GOTO is fundamental to understanding programming. But we don't use GOTO -- we use higher-level structures like loops, functions, even objects. I say we shouldn't use locks (or thinly-veiled hacks like synchronize()); we should use higher-level structures like message-passing, map/reduce, etc.

That said, the Erlang runtime is pretty impressive.

Re:Why Erlang doesn't matter

[SanityInAnarchy]

If you're primarily concerned with processing text, it's the wrong tool for the job.

Fair enough, but it seems likely that you would want both -- that is, both processing text, and scaling up to large clusters and core counts.

Simple (contrived) example: search engine.

That's my main frustration with Erlang -- "right tool for the job" inevitably forces you to use multiple "tools" to do anything really interesting.

Stupid article

[IamTheRealMike]

Wow, it's not often I strongly criticise articles around here, but that was total garbage.

For the smart ones that didn't RTFA, here's a quick summary:

• I like Erlang.
• Big companies like Google and Amazon make things fast by using concurrency.
• Erlang supports (one type of) concurrency.
• Thus Google and Amazon are [probably] using Erlang.
• Thus everyone should learn Erlang.

For the record, I work for Google and we don't use Erlang anywhere in the codebase. Google Gears restricts you to message passing between threads because JavaScript interpreters are not thread-safe, so it's the only way that can work. Visual Basic threading works the same way for similar reasons. It's not because eliminating shared state is somehow noble and pure, regardless of what the article would have you believe, and in fact systems like BigTable use both shared-state concurrency and message passing based concurrency.

The article says this:

Architects (but also university-professors for that matter) still think they can build current and future industrial-grade and internet-grade systems with the same technologies as they did 10-15 years ago.

But in fact the Google search engine, which is one of the larger "industrial-grade, internet-grade" systems I know of, is written entirely in C++. A language which is much the same as it was 10-15 years ago. Thus the central point of his argument seems flawed to me.

Seeing as the article is merely an advert for Erlang, I'll engage in some advocacy myself. If you have an interest in programming languages, feel free to check out Erlang, but be aware that such languages are taking options away from you, not giving you more. A multi-paradigm language like version two of D is a better way to go imho - it supports primitives needed to write in a functional style like transitive invariance, as well as a simple lambda syntax, easy closures and first class support for lazyness.

However it also compiles down to self-contained native code in an intuitive way, or at least, a way that's intuitive to the 99.9% of programmers used to imperative languages, unlike Erlang or Haskell. It provides garbage collection but doesn't force you to use it, unlike Java. It doesn't rely on a VM or JIT, unlike C#. It provides some measure of C and C++ interopability, unlike most other languages. And it has lots of time-saving and safety-enhancing features done in a clean way too.

Re:Stupid article

[burris]

I'm not going to disagree with most of your post, I think you're spot on. However, your suggestion of D is totally off. I like the D programming language quite a bit and version 2 is going to be really cool. However, even version 1 of D is not ready for prime-time. Version 2 of D is unstable and not recommended for production by even the author himself. All of the other languages you mentioned such as Erlang or Haskell are much more mature.

Also, "most other languages" have a foreign function interface for C, including Erlang, Haskell, Python, Java, Perl, Ruby, etc... In fact, I can't think of a well known programming language actually used by people other than the author that does not have an FFI... It is true that in most cases the FFI of other languages is more difficult to use than the one in D, but they are there.

Re:Stupid article

[Jerry Coffin]

But in fact the Google search engine, which is one of the larger "industrial-grade, internet-grade" systems I know of, is written entirely in C++. A language which is much the same as it was 10-15 years ago. Thus the central point of his argument seems flawed to me.

Not to mention Sabre and the VISA credit card processing system -- both use TPF, which is a lot older than Erlang. Meanwhile, many libraries have used the Tandem/Compaq/HP Non-stop system for years -- in fact, I'm pretty sure some have been up without a single interruption since before Erlang was invented. IBM has also had CICS for decades as well; IIRC, it's used by quite a few banks and hotel reservation systems. Likewise, substantial parts of the updates to the Air Traffic Control system currently in progress are being written in various older languages (more in Ada than any other, but it's certainly not the only one).

Now, don't get me wrong: none of these is a panacea for scalable programming. Nonetheless, the people who think they can create highly scalable systems using technology that's a lot more than 10 or 15 years old have some pretty good support. In fact, I can't think of a single really large scale system developed using only technology newer than that.

From a viewpoint of programming languages, there have been quite a few with support for various forms of parallelism over the years. Some (e.g. Concurrent Pascal) were oriented primarily toward tightly coupled processing. Others (e.g. Occam) were oriented primarily toward loosely coupled parallelism.

IMO, you just about need at least some degree of both. You can't take anywhere close to full advantage of a tightly coupled machine using only a loosely-coupled programming model. Likewise, if you want to scale to lots of separate machines, you just about need a more explicitly loosely coupled programming model.

If, however, you use a loosely coupled model even when it's not really appropriate, you may end up needing scalability, even when you otherwise wouldn't -- i.e. your code will be enough slower that it requires multiple machines, even when code using a more appropriate model wouldn't have required it.

Re:Stupid article

[IamTheRealMike]

Yes, D is very young and has problems. But then again, what language didn't? It's easy to forget but Python was first released in 1991. It took many years before it became mainstream (and some would say it's still not there yet).

The post-mortem is an interesting document, but I disagree with the authors conclusions. The compilers are buggy, well, C++ had exactly the same problem for a long time but still was a huge success. In particular, the trend seems to be basing new compilers on LLVM, which has a pretty robust optimization core. Frontend bugs are by comparison pretty trivial and easy to fix. Another few years and I think this problem will be licked - and besides, lots of C++ code has workarounds for compiler issues. Same thing for class libraries.

You're right about C-level FFIs. However D provides a simple C++ FFI which as far as I know is unique. Such a thing would be very useful for a company like Google which has a lot of C++ code, as it'd simplify binding considerably (I don't mean to imply anything about the future direction of the codebase, by the way).

The argument about parallelism is a more interesting one. But I disagree with that too :) D provides exactly what is needed for automatic sharding of work across cores (or machines). Specifically the combination of transitive invariance, reflection and purity enforcement is a very powerful one.

Essentially, if you can write your code to consist of non-trivial trees of pure functions, then it's perfectly safe to parallelise something like this:

foreach (item; list) {
    fooResults[item] = someTransform(item);
    barResults[item] = anotherTransform(item);
}

If someTransform and anotherTransform are both pure, by implication their parameters are transitively invariant, and thus they can both be invoked in parallel (because the compiler knows "item" can't be changed). What's more both calls can be invoked simultaneously as well.

Once the compiler knows these things, making this code run in parallel is simply another compiler optimization. That's the whole theory behind how functional languages can be super easy to parallelize. But in fact the key concepts can be applied to imperative languages as well, with the advantage that you can still have temporary mutable state within the function scopes - you just can't modify the heap, or anything reachable through your arguments.

D has keywords that let the compiler know and enforce function purity.

Now as it happens I doubt that any D compiler today implements this optimisation - it's sophisticated and transitive invariance is newly introduced in D2. But all the pieces of the puzzle are there. This also lets the compiler do calculations on data structures available at compile time.

Re:Stupid article

[Anonymous Coward]

• Erlang supports (one type of) concurrency.
• Thus Google and Amazon are [probably] using Erlang.

Amazon's SimpleDB does use Erlang, but it does not use the Erlang concurrency model -- they wrote their own because they didn't feel that Erlang's was robust enough.

It's unlikely that any of the retail site uses Erlang, because Erlang's not integrated with the Amazon communication framework. C++ and Java are formally supported internally for server applications and clients are written for C++, Java, Perl, and Ruby. SimpleDB wouldn't care, as it doesn't work inside the Amazon framework; it's an outgrowth of the A9 search engine.

I never looked at the IMDB code, so I'm not sure how it's set up, but they also mention Java in the same breath as Erlang. This suggests that they have a very particular application in which they plan to use Erlang, some number of use cases where they plan to use Java, and will continue to use Perl for things like generating content.

Re:Stupid article

[jbolden]

Right but what I can't do is is be sure in looking at your code block that someTransform didn't have a side effect that pollutes item when I want to run anotherTransform. Nor can I be sure that one of these two doesn't have some something like a "last" so I can't go through list in whatever order I want.

Re:Stupid article

[IamTheRealMike]

Yes you can, that's the point of the "pure" keyword and transitive invariance. If the someTransform function is defined pure you know it will not change item, and the compiler can enforce that.

Re:Stupid article

[jbolden]

OK that's useful. Does it also prevent all side effects (like a file write)? What about the issue of one of them having a last statement?

Re:Stupid article

[burris]

Once the compiler knows these things, making this code run in parallel is simply another compiler optimization. That's the whole theory behind how functional languages can be super easy to parallelize. But in fact the key concepts can be applied to imperative languages as well, with the advantage that you can still have temporary mutable state within the function scopes - you just can't modify the heap, or anything reachable through your arguments.

Sounds great in theory but in the real world you don't get much concurrency. Once Bright finishes D2 he'll discover this too, just as the Haskell people did. For the forseeable future, it's going to require the same humans that write the software to help the compiler generate concurrent code. Languages that have had purity for many years are trying to integrate more powerful tools into the language and standard libraries for writing concurrent code, such as STM.

In the mean time, D is still doesn't have a stable compiler and a standard library most of its users can agree on. D doesn't even have a debugger for non-windows machines (not sure about windows.) The GDB patches are long abandoned and the commercial debugger with beta D support is unusably buggy in my experience.

Re:Stupid article

[IamTheRealMike]

I don't know about looping through the filesystem. That's shared state that the compiler probably can't perceive, and thus an evasion of the type system. But it'd be a pretty weird thing to do anyway. It can certainly prevent side-effects within the realm of the type system, for instance, pure functions can allocate memory using "new" but can't change any existing memory on the heap, or read global state that isn't also marked invariant.

Re:Stupid article

[IamTheRealMike]

Yeah, I am not expecting this to be a pill that magically makes programs concurrent. In particular, writing programs to be trees of transforms would be quite hard in some cases and potentially very inefficient, as you trade parallelism for lots of extra load on the GC. The days when a compiler does it automatically are some way off I think, but I can imagine this simplifying the task of doing some calculations concurrently when used properly. It won't replace STM or locked threading but is another useful tool in the toolbox - and available quite cheaply.

Re:Stupid article

[Anonymous Coward]

Once the compiler knows these things, making this code run in parallel is simply another compiler optimization.

That's true iff you are compiling for a strictly controlled runtime environment. If the target is a generic environment, this is a naive statement, since at compile time you cannot know the resources available (or busy) at run time.

At best, at compile time, you can identify which parts of the object code can be run concurrently if resources are available at run time. This is not a free lunch at compile time, it will not make code smaller, and if the environment is resource-constrained (e.g. fewer processing cores than expected, possibly down to a fraction of one single core), the probes for resource availability will slow down execution at run time.

Moreover, unless someTransform and anotherTransform are provably non-mutating of item at run time it is unsafe to split the computation trees and subtrees that are rooted at each. This implies unusual (for an Algol-like language, especially) hygiene in the namespace, or inlining at compile time.

At best, now, you have a pool of worker threads waiting so that when you enter into a looping/recursion block you can split the work up among the waiting threads on some spanning modulus that is some multiple of the flat part of the loop unwound for a single core. That may buy a speedup for that loop proportional to the number of additional worker threads acquired.

This approach is also useful for mutating data, and is probably more often used for that in the real world.

In pure FP, research technology may opportunistically fix some computations that knowable at compile time into strictly constrained compute trees with no computational dependencies except directly between each parent node and its children in the rootward direction only. Subtrees of these computations can be farmed off dynamically. This maps identically to an unrolled loop as above. However, in future there may be ways to constrain unknowable-at-compile-time computational activity into DAGs (this would require language B&D) or opportunistically explore computations at run time and throw away (unwind from) work already done if a bad dependency or side-effect is found. The latter approach has been used with some success implemented in syntactic sugar in Scheme-like languages, but has the drawback of very high energy use in the worst case (unwinding from a mostly complete distributed computation and restarting). A similar approach (and problem) is seen in software transactional memory.

Ultimately the problem comes down to programmer discipline; either a computation is safe for opportunistic distribution or it is not, but compilers are not yet good enough to decide that automatically at compile time, and automatic decision taking at run time is not free. Making the wrong decision can lead to using fewer resources than are available, or outright errors.

Languages which make it much easier for the programmer(s) to remain disciplined are therefore useful.

Re:Stupid article

[Shinobi]

While the article is crap, you make some pretty funny assertions too.

Just because Google used C++ does NOT make the central point of the article flawed. Google is not known for reliability, either at the application level or the server level. Take a look at something that requires far more reliability instead, such as large telecom networks. There you will see Erlang and similar languages used, because it's much easier to make a solid and reliable design, and to keep it maintained.

As for age, Erlang hasn't changed that much more than C++ has, in its almost 20 years of existence either.

As for "removing options", any language removes some options from the programmer. If multitude of options is one of the primary criteria of selecting any given language over far more critical concerns, then I question the competence of the person selecting the language. It ties in with software engineering, and how it should be treated like any other engineering discipline.

Re:Stupid article

[jbolden]

I'm not sure you are following.

Lets say someTransform has a line like:
  cout << item
then I can't execute the loop out of order. Same issue of it has a line like:
  if (test (item)) {last;}

What I was wondering is does it prohibit those sorts of things?

Re:Stupid article

[Teunis]

*heh* No argument. Next they'll be calling MPC or other concurrency systems "erlang".... ;)

It's handy looking at both the original post and yours for ideas though - thank you. I'm working on a concurrent LISP (there's a few of them out there but none AFAIK are currently maintained) for intent at managing my code and linux environments much more effectively. I've got a long way to go and more than once have considered just switching to some other language to finish the project. As I want it to scale down to ARM/mmu-free architecture and up to large distributed environments, some of the metathoughts (philosophies? architecture intents?) make my head hurt. Oh well - it is fun.

Cheers!

Learning it

[caluml]

I started getting interested in Erlang a year ago. I bought Joe Armstrong's book about it, and, when I was in a place for a few weeks with no internet connectivity, I settled down with my laptop to learn it.

However, I got stuck a: By the lack of documentation (perhaps I could have downloaded it, but over a mobile phone it would have been painful), and b: one of the tutorials in his book seemed to include a library that he had written, to which there was no source code, and the others then built upon that, which scuppered me.

I got dispirited, but I'm still interested. I guess I'll end up learning it when I need to use it for something (like I've learned everything in life). Anyone know where the equivalent of Javadocs are for the core Erlang libraries?

Re:Learning it

[e4liberty]

Erlang documentation has been improving over the last few releases. The Erlang core and OTP libraries are documented here. -- e

Re:Learning it

[caluml]

Yes, I've just upgraded from 11.something to 12.something, and it seems a lot better now.

Might have another crack at it.

no new language needed

[speedtux]

Erlang is a language that has all the right properties and mechanisms in place to do what utility computing requires.

Well, except that it's darned inconvenient to actually write the applications in it.

Google Gears is using Erlang-style concurrency, and the list goes on."

Yup, and it makes more sense to add "Erlang-style concurrency" to existing languages than to throw out everything and switch to Erlang.

Re:no new language needed

[Anonymous Coward]

Erlang is a language that has all the right properties and mechanisms in place to do what utility computing requires.

Well, except that it's darned inconvenient to actually write the applications in it.

Tell that to Ericsson, the creators of the language, whose AX301 ATM switch is powered by 1.7 million lines of Erlang (the largest functional program ever) and which achives 'nine 9s'
  reliability, i.e. an uptime of 99.9999999% (which translates to 31 milliseconds of downtime a year).

This, IMHO, makes it one of the most successful languages ever designed.

And no, it is not an imperative language with function bits bolted on, as someone suggested.

It's not perfect, but at least it's not Java, which looks as if it was designed (badly) in the (early) 1970s.

Re:no new language needed

[Anonymous Coward]

How the fuck is it "darned" inconvenient. It's no more or less convenient than any other language.

Someone please explain why it is that this article has managed to bring out all those slashdot-ers who are recovery from surgery to remove their heads from their collective arses.

Erlang is a language that has all the right properties and mechanisms in place to do what utility computing requires.

Well, except that it's darned inconvenient to actually write the applications in it.

Google Gears is using Erlang-style concurrency, and the list goes on."

Yup, and it makes more sense to add "Erlang-style concurrency" to existing languages than to throw out everything and switch to Erlang.

Re:no new language needed

[Anonymous Coward]

How the fuck is it "darned" inconvenient. It's no more or less convenient than any other language.

Well, why don't you go back to writing in Fortran then?

Someone please explain why it is that this article has managed to bring out all those slashdot-ers who are recovery from surgery to remove their heads from their collective arses.

Speaking about yourself?

For the record, I was using Erlang 10 years ago, and functional languages 20 years ago. I've written some moderately sized programs in it. Believe me: it's inconvenient.

But you wouldn't know: your head is still firmly stuck in your ass.

Re:no new language needed

[Anonymous Coward]

And no, it is not an imperative language with function bits bolted on, as someone suggested.

No, it's a Prolog subset with some distributed bits bolted on.

It's not perfect, but at least it's not Java, which looks as if it was designed (badly) in the (early) 1970s.

So does Erlang.

Re:no new language needed

[Anonymous Coward]

Why would I want to use Fortran you dolt?

I too have been using Erlang and other functional languages for between 10 and 20 years.

I still don't understand in what way you think it's inconvenient. Care to actually explain?

How the fuck is it "darned" inconvenient. It's no more or less convenient than any other language.

Well, why don't you go back to writing in Fortran then?

Someone please explain why it is that this article has managed to bring out all those slashdot-ers who are recovery from surgery to remove their heads from their collective arses.

Speaking about yourself?

For the record, I was using Erlang 10 years ago, and functional languages 20 years ago. I've written some moderately sized programs in it. Believe me: it's inconvenient.

But you wouldn't know: your head is still firmly stuck in your ass.

Re:no new language needed

[speedtux]

Why would I want to use Fortran you dolt?

Because you said "It's no more or less convenient than any other language." If it's no more or less convenient than any other language, in particular, it's no more or less convenient than Fortran. So, why not use Fortran, you dolt?

I still don't understand in what way you think it's inconvenient. Care to actually explain?

As an example, take strings:

http://schemecookbook.org/Erlang/StringBasics

Few would recommend Erlang as a high-performance string manipulating language. Strings in erlang are simply lists of characters, with a bit of syntactic sugar to allow you to easily construct such lists as text enclosed within quotation marks. In fact, to quote Sendmail's excellent case studyin implementing their Sendmail load balancing "Client Daemon" in Erlang:

But Erlang's treatment of strings as lists of bytes is as elegant as it is impractical. The factor-of-eight storage expansion of text, as well as the copying that occurs during message-passing, cripples Erlang for all but the most performance-insensitive text-processing applications.

I couldn't believe it that in 2008, Erlang still hasn't fixed this, and this is just one of many problems with Erlang for real-world applications.

I'm all for functional, safe languages. I like syntax different from C/Java. The problem is that the functional programming community produces one language disaster after another by always screwing up on one or more important things: string handling, numerics, syntax, error messages, usability, I/O, etc.

In the end, most applications do the "heavy lifting" in parts like string handling, disk I/O, and numerics. It's a lot easier to get the job done writing that in a language that has excellent syntactic and library support and than add on a little bit of communications on top. And there are excellent Erlang-style communications libraries for other languages.

Mis-designed languages like Erlang can still solve some problems (CouchDB) really well, but a language that I can use even for 50% of my jobs simply isn't worth bothering.

Languages like C and Java keep winning because designers of languages like Erlang are too lazy, arrogant, and stupid to do their job right. In different words, languages like Erlang is what's killing functional programming.

Consider Stackless Python

[Anonymous Coward]

http://www.stackless.com/

I will usually bend over backward to use Python just because I find it very easy to write self documenting code. I have to maintain my own code and find it easy to work with code that I wrote years ago. In fact, my cow-irkers (who don't use Python at all) can easily follow what I've done. The language is sane enough that I can get help from C++ people if I get into a thinko. (a thinko is like a typo but way way worse) It also means that I can still help people with their C and C++ problems even though I have hardly written a line in those languages in the last couple of years.

If I had to write something concurrent, I would try Stackless Python. I looked at the mailing list archive and the project is active (so you can get help) and there are major projects that use it. It looks like it is worth checking out. (Obviously, I haven't used it yet)

Re:Consider Stackless Python

[GaryOlson]

Erlang vs. Stackless python: a first benchmark is a very good discussion of lots of niggling details in benchmarking a concurrency language. The comments are quite good.

Re:Consider Stackless Python

[Shinobi]

Stackless is, essentially, junk. Requires way too much overhead, and it's nowhere near fault-tolerant enough. If you want to see a real-world application where it fails for those reasons, just look at EVE Online, their entire glue layer is made up of Stackless Python, and they have ridiculous overheads. Writing that in Erlang would have saved lots of processing power, and even more developer time in the form of design and bug testing.

To give you an indication of just how useful Erlang is, significant parts of the worlds mobile telephony networks are run on Erlang, and many landline telecom networks run on Erlang too.

Wings 3D is written in Erlang...

[Fallen Andy]

See here. (It's an open source subdivision modeller).

Andy

"Cloud computing" is an Xmas artifact

[Animats]

The enthusiasm for "cloud computing" may evaporate when Xmas rolls around.

I went to a talk at Stanford by the architect of Amazon's web services. It came out in questioning that the real motivation between Amazon's low-priced web services is that their load in the Xmas shopping season is about 4x the load for the rest of the year. Their infrastructure is sized for the November-December peak, so for ten months of the year they have vast excess capacity. That's why Amazon's web services are so cheap.

Don't expect good response time during the shopping season. Although this Xmas might be OK, due to the recession.

Re:"Cloud computing" is an Xmas artifact

[datablaster]

Santa Claus, take note: Just what do you do with your sleigh and reindeer the rest of the year? That 11-month excess capacity could be out makin' you money Jan thru Nov.

Re:"Cloud computing" is an Xmas artifact

[Chang]

While the origin of EC2 in 2006 is certainly related to peak capacity requirements at Amazon, it is certainly way beyond that point now.

Two Christmas seasons have come and gone without major capacity problems on EC2.

The reality is that EC2 has grown far beyond its roots as a way for Amazon to amortize their peak capacity by reselling it and it has turned into a small but growing profit center and publicity success for Amazon.

Re:"Cloud computing" is an Xmas artifact

[Anonymous Coward]

Don't expect good response time during the shopping season. Although this Xmas might be OK, due to the recession.

EC2 runs on dedicated hardware, completely separate from Amazon's retail business. The point is, Amazon has to buy a huge amount of server capacity that's only needed for three months of the year. After, it's moved to scale the AWS fleets for the next year, practically for free as the revenue generated during the holidays fully covers the purchase.

Re:"Cloud computing" is an Xmas artifact

[Anonymous Coward]

I'm very surprised by this answer, because I attend such a conference and the guys told us that nothing was shared between their infrastructure and Amazon Webservices. If this is true, this is bad.

Get a clue!

[Anonymous Coward]

It's a functional language you dolt, of course the "variables" are invariant. The only way in which it was the designers preference is that he presumably wanted all the niceties that come with declarative languages; provability, implicit parallelism, etc.

1. Invariable variables.
This appears to have been done for no reason other than the designer's preference. In fact, it's not strictly true -- variables can be unbound, and later bound. They just can't be re-bound once bound.

Why is any syntax that's not filched from C weird? Frankly I'm not that fond of C's syntax it _can_ lead to very unreadable code (Speaking as someone with 20 years experience with C). If you'd done a modicum of research you'd realise that Erlang models its syntax on Prolog, and Prolog like Lisp has very regular syntax.
Note that I'm not claiming that Erlang, or Prolog, or Lisp syntax is that great either, but by holding up C as some kind of gold standard you've automatically lost that argument.

2. Weird syntax.
Why, exactly, are there three different kinds of (required) line endings? It seems as though the syntax is designed to be as different from C as possible, while maintaining at least as many quirks. Moreso, even -- when constructing normal, trivial programs, you're going to hit most language features head-on and at their worst. Where's my 'print "hello\n"' that works most other places?

I don't believe the important features of Erlang are mutually-exclusive with the sane syntax of, say, Ruby or Python.

Not sure about Unicode, you may actually have made a valid point here.

3. Not Unicode-ready.
Strings are defined as ASCII -- maybe latin1. But there's no direct unicode support in the language -- if you're lucky, there are functions you can pipe it through.

There are other things I haven't mentioned, mostly implementation-specific -- things like the fact that function-reloading cannot be done when you natively-compile (with hipe) for extra speed. My plan is to take the features I actually like from Erlang and implement them elsewhere, in a language I can actually stomach for its real tasks.

Erland textbook: read and adapt

[Kupfernigk]

Joe Armstrong's Erlang textbook is interesting, but I did not have time to learn the language and recode the part of our current project that would benefit from it. So I did what any sensible person would do: raided the concepts, and used them to redesign the critical parts of the application. I was initially provoked into doing this because, in the book, the comparison at one point between the Erlang and Java way of doing something is just plain wrong. When I thought out how I would actually do it in Java, I realised that it helps to stick to a language you know well.

Cloud computing, not internal services

[jmcbain]

This article is discussing the front-facing programmatic interface to cloud computing. It is not referring to the internal code of your company.

Dunno if Google uses Erlang, but Facebook does

[Anonymous Coward]

They built the Facebook Chat backend using Erlang. Scaling something from 0 users one day to tens of millions of active users the next day is a challenge, and they decided Erlang was the right tool for the job. http://www.facebook.com/note.php?note_id=14218138919&id=9445547199

sufficient efficiency?

[JustNiz]

>> you need large real-time systems running as sufficiently as possible.

lol. Are they running vista then? oh wait.. that would be INsufficient...

Going from Perl to Erlang, eh?

[thanasakis]

TFA more or less says that IMDB is switching from Perl to Erlang. So I looked at the link and here's what I got:

(From here

We are looking for developers with experience building web scale distributed systems. We are currently working in Perl but have plans to use Java, Erlang and any other language that we think will suit our purposes. We aren't looking for expertise in any of those, particularly, but we expect that you will be an expert in the systems you know. We do require that you be passionate about testing (unit, integration, fault-injection) and code quality. Experience with relational databases (Oracle, MySQL, etc), embedded databases (BerkeleyDB, CDB, MonetDB, etc) and Linux are a big plus.

I'll leave anyone to draw his own conclusions.

Re:Going from Perl to Erlang, eh?

[sigzero]

Yeah...that is kind of stretching the "IMDB is re-writing in Erlang".

...switching from Perl to Erlang

[Anonymous Coward]

Oh no! What's to become of poor pudge?? Will he rather fight than switch? Or will he roll over like he does for his favorite political thugs?

Web apps not large-scale, something is off

[Tablizer]

I don't get it. Web transactions are generally like little programs. Concurrency is only needed when sharing info, which is what the database is used for. They need a big-ass database, not a big-ass language unless they are doing online games or the like. Are they trying to re-invent a navigational RAM database or something? Something doesn't add up.

Mozart/Oz

[synthespian]

http://www.mozart-oz.org/

I'll just cite another "competitor":

"The Mozart Programming System is an advanced development platform for intelligent, distributed applications. The system is the result of a decade of research in programming language design and implementation, constraint-based inference, distributed computing, and human-computer interfaces. As a result, Mozart is unequalled in expressive power and functionality. Mozart has an interactive incremental development environment and a production-quality implementation for Unix and Windows platforms. Mozart is the fruit of an ongoing research collaboration by the Mozart Consortium.

Mozart is based on the Oz language, which supports declarative programming, object-oriented programming, constraint programming, and concurrency as part of a coherent whole. For distribution, Mozart provides a true network transparent implementation with support for network awareness, openness, and fault tolerance. Mozart supports multi-core programming with its network transparent distribution and is an ideal platform for both general-purpose distributed applications as well as for hard problems requiring sophisticated optimization and inferencing abilities. We have developed many applications including sophisticated collaborative tools, multi-agent systems, and digital assistants, as well as applications in natural language understanding and knowledge representation, in scheduling and time-tabling, and in placement and configuration."

self sufficient? Re:Sufficiently?

[Fubari]

I read that as "If you want to build computing into a utility, you need large real-time systems running as self sufficiently as possible."

imdb?

[Anonymous Coward]

IMDB (owned by Amazon) is switching from Perl to Erlang

You know, other than the spam blogs that just cut and paste the front of Slashdot, I can't find any reference to back this statement up. Anyone have a link that isn't a spam blog?

So what are Ericsson doing?

[dread]

I can tell you that NO development that is outside of core network componens (I have only limited insight there) is done in Erlang. Not the billing systems, the messaging components (SMS/MMS/Voicemail), not the IVRs. But sure. Whatever. If you really want to build an old school exchange then by all means, get your SS7 stack and Erlang away. Unfortunately the movements towards IMS aren't really helping Erlang internally. And neither is LTE.

Re:So what are Ericsson doing?

[Anonymous Coward]

More and more three-letter-acronym-boxes in IMS are (re)implemented in Erlang so I wouldn't say IMS aren't helping.

Concurrency Fad?

[Tablizer]

People are turning to currency-friendly languages as kind of a fad. A solid need has not been identified for it yet except maybe action gaming. Outside of that, a smart use of the database takes care of most if not all of the alleged "concurrency problems" that are being generated by new chip sets.

I agree that database architecture may need an adjustment to take advantage of more RAM, but that shouldn't require a significant change in the way applications are written. DB's already insulate most apps from implementation issues including concurrency. I'm not saying that concurrency app languages don't have their use, but lets make sure there is really a problem before we make a mad dash to claimed solutions. The IT industry has a tendency to get carried away with fake or exaggerated problems.

Research is fine, but lets approach actual implementation smartly. Maybe I've been around too long, but this smells like yet another fad.
     

Re:Concurrency Fad?

[jbolden]

Since this sounds practiced thought I'd comment that Software Transactional Memory essentially uses the approach you are suggesting. Then just certain parts of the compiler need to be functional not the apps themselves.

Re:Concurrency Fad?

[Anonymous Coward]

Yeah, this whole TCP/IP end-to-end internet thing is just a fad too.

Re:Concurrency Fad?

[Tablizer]

Do you have a sure-shot way to tell fads from non-fads?

Facebook chat

[dristoph]

It should be noted that Facebook's relatively new chat feature, which allows Facebook users to send instant messages to all their online friends as well as see status changes, notifications, and feed stories in near real time, was developed using Erlang. http://www.planeterlang.org/story.php?title=Facebook_chat_is_developed_in_Erlang

Have you never heard of functional programming?

[Estanislao Mart�nez]

1. Invariable variables. This appears to have been done for no reason other than the designer's preference. In fact, it's not strictly true -- variables can be unbound, and later bound. They just can't be re-bound once bound.

Um, have you never heard of functional programming? Erlang's a pure functional language, a design choice that opens up a number of optimization possibilities. The most obvious one is that the fact that since programs can't mutate memory, there is no memory synchronization or locking required. There are other advantages to it, too: operations on pure functional data structures don't destroy older states of the data structure, meaning that no thread ever sees a shared data structure in an inconsistent state.

To tell you the truth, I'd prefer a language where mutable memory is optional (e.g., O'Caml), but I don't think that Erlang's design choice is by any means stupid. I do think the syntax is funny--it was largely copied from Prolog and I don't like Prolog's syntax--but come on, this is also a minor point. The only good point you've brought up is the awful support for Unicode.

Re:Have you never heard of functional programming?

[SanityInAnarchy]

Erlang's a pure functional language

Erlang is a vaguely functional language. Haskell is a pure functional language.

Re:Have you never heard of functional programming?

[Anonymous Coward]

Erlang's a pure functional language

Erlang is a vaguely functional language. Haskell is a pure functional language.

I don't think you know what "purely functional" actually means.

Haskell may have monads, but those don't make it purely functional. If Haskell didn't support side-effects, you wouldn't be able to use it for -anything-.

Ignorance of history

[roca]

It's very misleading to start talking about message passing and no-shared-memory as "Erlang-style concurrency". There is a huge history of languages and frameworks in this area that predate Erlang by decades. It does no credit to Erlang when its advocates ignore this.

Re:Ignorance of history

[Anonymous Coward]

Actually, the communication model of Erlang is the Communicating Sequential Processes, proposed by C.A.R. Hoare in the seventies.
http://en.wikipedia.org/wiki/Communicating_Sequential_Processes

Eat Their Own Dogfood

[d'baba]

As a former Amazon.com(tm) warehouse employee & a software engineer in several completely separate companies, I'll get a positive sense out of Erlang's effectiveness when they switch their in-house warehouse receive/ship tools to it from perl.

Ob. dogfood.

Wouldn't get through my spam filter

[leftie]

Sounds like the beginning of a Nigerian scam mail.

"The two biggest computing providers of today are offering to you opportunities to become financially enriched by building their concurrent offerings on top of really concurrent programming languages and systems. All you have to profit is send $20,000 via Western Union to Ubulu@lagosnet.ng...

Why Erlang Does Not Matter

[canuck57]

This is really a duplicate of previous posts and sales ploy. A hard sell of Erlang.

Me, I never plan on learning or using it. There are plenty of good languages out there without another. But what really turns me off it their hyperbole on threading and multiprocessing. C, C++ and Java thread nicely thank you. They work across platforms, have standards, relatively mature and run on big and small. C and C++ are also fast. Using an ORB or RPCs isn't that tough.

The limitation today in tapping multiprocessing is the lack of good designs by the people. Nothing else. Those who do not believe this ought to read up on RPCs, semaphores, mutex and the like. Or just do Java. However Java does show not knowing the fundamentals can often result in big fat inefficient bloatware.

Real time is also by design and not the specific language but tends towards C. It is why device drivers are written in C. C can be real time and can be made very lean and fast. vxWorks, pSOS and there is even a Linux RTOS out there, all are in C.

I invested in the 80's in learning C, and shortly after C++. It has no limitations in applied computing. Limitations are purely the carbon based units design.

Re:Why Erlang Does Not Matter

[Anonymous Coward]

You think you are a Real Programmer. Concurrency must be done the Hard Way (tm). There must be locks etc, there must be lots of additional work to make a program concurrent, and the debugging must be hard but that is OK since your reward is finding that Singleton construction method that wasn't thread-safe.

Which is good for a consultant because he can bill endless hours and spew tech talk while doing so.

Me, I just want to get things done, and I get my rewards from doing them elegantly.

Re:Why Erlang Does Not Matter

[rootooftheworld]

this got moded down?! mods', dont let me find you!

There is no silver bullet.

[ireallylovelinux]

It seems when something new is developed everyone thinks that this will solve all the problems all the other languages didn't. First it was ruby on rails and now we have erlang, guess what it wont.

Re:There is no silver bullet.

[DozePih]

Define "new" please.

Take a look at Reia - Python-like on Erlang VM

[Anonymous Coward]

http://wiki.reia-lang.org/wiki/Main_Page

CouchDB

[pixelcort]

Apache CouchDB, an Incubator project, uses Erlang. It is a document-oriented database with MapReduce views/indicies support. Its documents are stored as JSON objects and its MapReduce functions are typically written in JavaScript.