Slashdot Mirror

Mnesia is a database that is being used in telco does this. It is fast, fault tolerant and distributed.

More information on Erlang website

Mnesia is a database that is being used in telco does this. It is fast, fault tolerant and distributed.

More information on Erlang website

For enterprize? Why not Erlang?
At least it's fault tolerant and runs well 24/7.
Production times is much faster using it than using Java.
It can be combined with Java, Python, it has native CORBA,
native fault-tolerant built-in database.

I agree. What I've also found out is that the programming language can help you a lot with the task of writing reliable software. If you haven't looked at Erlang yet, please do.

Lisp has one of the best object-oriented paradigm implementation, Meta-Object Protocol among languages with both scripting and bytecompiling capabilities.

Scheme has been proved as a good language for GUI and configuration: GIMP, Sawfish, TeXmacs.

OCaml has all the power as Lips, just in syntax conviniect for many Java/C-poisoned brains to read faster. No wonder there are many real-world applications on it.

Haskell... I just love how it demonstrates that OOP is not everything (and even not enough) :)

Sun works for telecom industry - why not consider Erlang?

And don't ignore Mozart - it's multi-paradigm pradigm might be just what we all will thing as the best in 3-5 years.

The list is not complete, of course. And it's inspired by Functional Programming.

My main point here is: each of above languages, would it be in hands of Sun marketed instead of Java (with all that money invested to), would have quality of implementation much better than Java.

In fact, I am impressed how such poorly designed language as Java succeed so far on the market. It wouldn't without so much money behind. And without so many classes written by Sun to compensate the poor design of the core language itself.

Would Sun invest so much efforts and money to FP language then the result would be much better. Because quality is why FP matters.

Writing a robust program in Erlang is extremely simple by using supervisors and following design principles like these. For example, my Erlang IRC bot is near impossible to completely crash. If the server connection dies, there's a supervisor that'll keep trying to get it back up; All the while the other processes of the bot have no idea there's no connection -- they don't need to know. If I've written a faulty module and the code misbehaves, only that single process is affected. If possible, a process that has crashed will be automatically restarted and operations resume as if nothing had happened.

Hot-swapping of code allows me to debug the bot without restarting it. If I discover a bug in the irc parsing routines, I simply fix it in the code, recompile, and reload the offending module -- while the bot is running. The new code replaces the old, and it just works.

There's plenty more, but I would suggest checking Erlang's website and reading the FAQ and examples. Oh, and here's a nifty benchmark :)

Writing a robust program in Erlang is extremely simple by using supervisors and following design principles like these. For example, my Erlang IRC bot is near impossible to completely crash. If the server connection dies, there's a supervisor that'll keep trying to get it back up; All the while the other processes of the bot have no idea there's no connection -- they don't need to know. If I've written a faulty module and the code misbehaves, only that single process is affected. If possible, a process that has crashed will be automatically restarted and operations resume as if nothing had happened.

Hot-swapping of code allows me to debug the bot without restarting it. If I discover a bug in the irc parsing routines, I simply fix it in the code, recompile, and reload the offending module -- while the bot is running. The new code replaces the old, and it just works.

There's plenty more, but I would suggest checking Erlang's website and reading the FAQ and examples. Oh, and here's a nifty benchmark :)

Erlang is built on the OS model to abstract the hardware and act as a multiplateform development environment.
His threading mechanism is very efficient and do not use the OS one.
You can have a look at: http://www.erlang.org/

Ace's Hardware needs to research real servers before talking about their "scalable" servers. Their numbers are really saying that their box performs like a dog.

For those of you interested in this topic here is a few pointers and words of wisdom.

Server scalabilty and performance has three basic metrics, thruput (urls/sec), simultaneous connections, and performance while overloaded. Of course, you could add latensy but I'd argue that with the correct design latency is directly proportional to the real work you are doing, bad design insertes arbitrary waits.

I know of a HTTP Proxy by a large ISP that does user authentications & URL authorization (re: database), header manipulation, and on-the-fly text compression at 3000 urls/sec for 2000-4000 simultaneous connections and maintains that performance under load by sheding connections, all this on a dual 1GHz Intel PIII box running a Open Source OS that starts with "L". That is a maximum of 260 Million URL/day, three orders of magnitude greater performance than Ace's Hardware stats.

The simple answer to the question "How do I create a scalable fast network server?" is Event-driven GOOD & Threads BAD. Event driven network communication is two to three orders of magnitude better performing than thread/thread-pool based network communications. See Dan Kegel's C10K web page. That means you must use non-blocking IO to client sockets and databases. Once you accomplish that small feat, dynamic content just consumes CPU; with 2.8 Ghz Xeon processors you have plenty of cycles for parsing HTML markup or whatever. Threads cause cache thrashing, and context switching. While thread programmers don't see the cost in their code, just read the kernel code and you'll see how much work HAS TO BE DONE to switch threads. Event driven programming just takes some state lookups (array manipulation) and a callback (push some pointers onto the stack and jump to a function pointer).

Desgin is FAR MORE IMPORTANT than which runtime you use (execution tree, byte code, or straight assembly). I have done some very high load network programming with Perl using POE.

Python has Twisted Python

Java has the java.nio and the brilliant event/thread hybrid library SEDA by Matt Welsch.

I am also looking into the programming language Erlang which builds concurrancy and event driven programming into the language. Further, Erlang is used by some big telco manufacturers to great effect (high performance and claimed 99.9999999% nine-nines reliability on a big app).

Comparing Java and Erlang/OTP?

Both names are used to reference a programming language, a large library and a run-time system that is based on a virtual machine.

Java is (depending on your view and mode) either a dumbed down version of C++ or a version of C++ adapted for use with a virtual machine.

Erlang however is not an imperative language (where you state the control flow explictly) but a functional language (where the focus is on function evaluation) and a concurrent one too (which means the language system makes it easy for you to work with parallel processes on different CPU nodes).

Both systems come with large libraries. However Erlangs focus is rather concurrency and high availibility while client side Java provides much GUI support.

It is not unusual to do the high level programming in Erlang while interfacing to Java GUI code or to C++ performance intensive stuff.

Regards,
Marc

I always found erlang(www.erlang.org) a pretty productive language and practical too.

But the learning curve for most people seems too much. OO on the other hand looks similar to more common programming paradigms. Unfortunately I have always found it does not scale well.

Erlang is a general-purpose programming language and runtime environment. Erlang has built-in support for concurrency, distribution and fault tolerance.

In other words, it's a small concurrent functional programming language developed by Ericsson after their experiments with Lisp and Prolog.

Although, you will not love it if have already poisoned by OO "snake oil"

If it's a case then try Dylan

Erlang is a general-purpose programming language and runtime environment. Erlang has built-in support for concurrency, distribution and fault tolerance.

In other words, it's a small concurrent functional programming language developed by Ericsson after their experiments with Lisp and Prolog.

Although, you will not love it if have already poisoned by OO "snake oil"

If it's a case then try Dylan

If fork() expense is not an issue, then process context switch and process state memory requirements are definitely issues. Create a few ten thousand processes on your machine, and you'll bring your machine down to its knees. Your memory will be wasted, your CPU will spend most of the time (if not all) context switching.

Of course the solution is a real language with built-in concurrency, message passing, distribution and fault tolerance like Erlang.

Why Erlang

Not some perversion like pthreads under Java or pthreads in C. Erlang processes have very lightweight message passing and process management overhead. Lighter than your OS (because all Erlang processes run inside the Erlang VM), several orders of magnitude lighter than Java, and no mutexes, semaphores, and other such bullshit to worry about. In addition you get the ability to distribute your processes seamlessly over networked machines. This is a language built-in feature.

The only library in C which lets you do something similar is state threads from SGI.

State Threads

Speaking of Java and threads, I think it's past time for someone to seriously think about creating a language with even more first class structures for dealing with parallelism.

Erlang

Sir, allow me to kick your arse with Erlang!

• Our nations' taxes paid for it.
• Our nations' safety depend on it.
• Our travellers' time is valuable.
• The developers need to know that the world is watching their keystrokes,
  so that they are encouraged to press the correct buttons.

This is the reason you get dialtone reliably when you pick up the 'phone.
It would work for Air Traffic Control too.

The languages syntax. Nothing more. For example, you won't benefit from Erlang's very efficient thread implementation with a (AFAIK imaginary) Erlang.NET. Just see the various ports of languages to the JVM (and there are many, that isn't a feature of .NET only) - all of them are crippled in one way or another.

Java by itself is an OOP that doesn't suck as much as it could (it's better than Satan^H^H^H^H^HC++, and is palatable to managers in the same way Smalltalk isn't).

Add AspectJ, and you're really cooking! What do you get? Custom profiling and tracing/logging support. Method dispatch re-writing (yes I know this is the cure, not the disease, which is policy or contract enforcement), mixins, inheritence hierarchy re-writing...

Of course, you could look at Erlang instead. It's a neat functional language that solves multi-threading, communication, event handling, and other issues that you long for when switching from an imperative language. I'm still trying to make this one fly with my bosses...

It that isn't your primary concern (and frankly, it seldom is), there are lots of interesting FPLs, like Erlang, which has support for parallel, distributed, fault-tolerant computing as a primary design goal.

However, as for the "critical mass", it's a chicken and egg problem. There are way less useful libraries for FPLs, and the documentation you get is mostly not quite written for the "pragmatic programmer".

However, I also would recommend a FPL if you want to write good code quickly and have fun doing so - I agree with the OCaml-Team chosing "The programming tool of choice for the discriminating hacker" as a motto for the language.

No way, Lisp is way outdated, and it isn't even a pure functional language. People will want to use Haskell, Erlang, or Goatsex instead.

Have you actually used a Lisp environment in the past twenty years or so? Every major Common Lisp implementation has a compiler, and at least one doesn't even have an interpreter. What specifically about the condition system do you find inadequate compared to other languages? It definitely offers support at a level at least as high as something like C++ or Java.

As far as doing real work in functional languages, perhaps you ought to ask the Erlang people about that. Or check out this link.

Does everybody on slashdot think that the only salient feature of Python is that it doesn't use curly braces?

Get over it!

I am being quite serious. If that modest syntax change is enough to keep you from considering a language, you're doomed as a programmer to linguistic provencialism that will keep you from seeing some really elegant ways to simplify and modularize your code. Ever programmed in Erlang? Haskell? Scheme? Prolog? You might end up preferring a more mainstream language after all is said and done, but the experience of seeing the new ways of doing things will certainly make those mainstream programs better.

You'll never get that experience, though, if you get scared by the syntactic differences between those languages and C (which are vast). So do yourself a favor and try to see beyond a language's syntax.

This sounds like the kinds of problems that the concurrent functional programming people love. See Erlang or perhaps some concurrent variation of Haskell like Eden.

Regards,

Zooko

It's called Mnesia: Mnesia

Also Visit Erlang

It's not necessarily SQL based.

• OCaml is multithreaded (linuxthreads was born as a way to get it to work on linux), although a single program won't take advantage of a multiprocessor. Core dump is impossible using its static strong typing, and the optimizing native-code compiler provides with great performance.
• JOCaml is an extension of OCaml based on the Join-Calculus, the latest and greatest paradigm for distributed programming (can be seen as actor-based programming done right, as based on a well-understood algebra). Can take advantage not only of multiprocessors, but of processor farms, or any distributed architecture, even heterogeneous architectures. Only bytecode can migrate, native code modules and primitives must be compiled into the servers.
• Erlang is based on a paradigm quite similar to jocaml, but is designed for industrial applicability rather than hacker coolness; it has dynamic typing, a pure functional core (despite the logic programming syntax, it only has matching, not unification), enriched with explicit asynchronous communication primitives, and an implementation that every phone call you make depends on, if you use British Telecom. If you need tens of thousands of threads and/or lots of nodes, this is what you need.
• There is also Mozart, an distributed implementation of Oz, a real logic programming language (has unification).
• Haskell and Mercury also have extensions for concurrent and distributed programming in the works, but I admit I don't know how usable they are for real programming.

-- Faré @ TUNES.org

Just because I'm sitting at school right now trying to finish a small project in it, I'd like to add Erlang to that list. It's kind of neat, actually.

Erlang (developped by the Swedish telecom company Ericsson) is an Open Source distributed operating system that runs on top of a host OS such as Unix or MS Windows. Erlang is based on high-level language paradigms, which makes it refreshingly different from all these C-based OSes. I think it deserves to be better known.

For a rather comprehensive list of operating systems, check out the OS review subproject of the Tunes project. Of course, since Tunes is The Ultimate OS, it is distributed also (its only disadvantage is that it (currently?) doesn't exist).

For many applications, writing your own makes good sense. But even then you can get a flying start by using components made by others with high availability and reliability in mind. By building your own using fine-grained components you keep control over what you're building while also having some comfort in knowing that others have successfully done similar things using the same tools. And been happy with the results.

One example is the erlang programming language and libraries, which were developed specifically for writing high availability telephone systems. This is open source, available as tar.gz for solaris or in debian, red hat, BSD...

People have gone on to use these tools (both on Solaris and other OSes) to build high availability web systems (e.g. lodbroker) and robust email systems (e.g. bluetail).

But Ericsson doesn't want to be out there on its own as regards using functional languages. Therefore Ericsson has released the whole of the Erlang system as Open Source. For those who like such things, commercial support is also available.

So why haven't Functional Languages caught on? Someone once said "perception is reality", and peoples' perception of functional programming is that it is something you learn at school because it is good for you, like learning Latin. Then you forget all about it and use C++ or Java like everybody else. Popular perceptions are:

• FP is hard to learn and requires a PhD in mathematics
  This may be true for some functional languages, but Erlang is trivially easy to learn.
• FP are very inefficient
  Not true any longer. True the rely on recursion and other techniques which used to be inefficient - but implementation using tail recursion of last call optimization makes recursion is just as fast as normal iteration.
• You can't hire programmers who know FP
  Firstly if a programmer can't learned to be productive in a language like Erlang within a week, then they aren't worth hiring. Secondly, the number of people who are learning Scheme, ML etc at school is increasing day by day.
• You can't buy tools for FP
  And what's wrong with emacs then? If you are one of these people who like writing programs by clicking in funny boxes, then FP's aren't for you. But for the rest of us, functional programs are much closer to specifications than those funny boxes. Erlang, for example has a rich set of tools like debuggers available free.
• There aren't any libraries
  Wrong, yes there are. Have a look at the Erlang open source web and you will be amazed.
• Learning FP won't enhance my career prospects
  Unfortunately, probably true.

But Ericsson doesn't want to be out there on its own as regards using functional languages. Therefore Ericsson has released the whole of the Erlang system as Open Source. For those who like such things, commercial support is also available.

So why haven't Functional Languages caught on? Someone once said "perception is reality", and peoples' perception of functional programming is that it is something you learn at school because it is good for you, like learning Latin. Then you forget all about it and use C++ or Java like everybody else. Popular perceptions are:

• FP is hard to learn and requires a PhD in mathematics
  This may be true for some functional languages, but Erlang is trivially easy to learn.
• FP are very inefficient
  Not true any longer. True the rely on recursion and other techniques which used to be inefficient - but implementation using tail recursion of last call optimization makes recursion is just as fast as normal iteration.
• You can't hire programmers who know FP
  Firstly if a programmer can't learned to be productive in a language like Erlang within a week, then they aren't worth hiring. Secondly, the number of people who are learning Scheme, ML etc at school is increasing day by day.
• You can't buy tools for FP
  And what's wrong with emacs then? If you are one of these people who like writing programs by clicking in funny boxes, then FP's aren't for you. But for the rest of us, functional programs are much closer to specifications than those funny boxes. Erlang, for example has a rich set of tools like debuggers available free.
• There aren't any libraries
  Wrong, yes there are. Have a look at the Erlang open source web and you will be amazed.
• Learning FP won't enhance my career prospects
  Unfortunately, probably true.

Pointed out above in this list - Erlang is an FP language used in serious production. The ERTS, running production telecom applications, does much of the work needed to support high availability clustering.

The language bears the hallmarks of ruthless pragmatism in its feature set. It was designed to serve a real, specific set of needs and appears to do so very well. But, it also makes you want to try it for just about any distributed app that comes along.

The FP nature of the language with very light weight processes and IPC suggests that this kind of FP may be even better suited to extreme programming and similar rapid development disciplines than OO.

If you need an example of functionnal programming used in real life, you should give a look at Erlang web page:
www.erlang.org

This language is used by Ericsson as a strategic advantage in their software business.

Debugging Functional languages can be quite messy and of course i don't know of any good debugging utilities for languages like (S)ML that are purely function in approach.

I think this is not true.

You should give a try to the Erlang functionnal language.
Their debugger is really amazing.
Really useable and even really useful as a learning tool.

Check:
http://www.erlang.org/

There is no question that Plan 9 is more elegant and clean, but it also lacks a lot of functionality. Whether it's easier to add all that stuff to Plan 9 (and whether the result will still be elegant and clean), or whether it's easier to add Plan 9 ideas to Linux, to me, is still an open question. Both paths would lead to roughly the same end result: a complex system with better support for distributed computing.

Either way, the open source release of Plan 9 is a good starting point because it gets those ideas out into the mainstream.

Again, I must go in my usual rant: is there any reason why we insist in still writing operating systems in low-level programming languages like C (or C++, which I insist on considering as a low-level language)? It may be that low-level languages are better suited for writing the low-level stuff like the bootstrap code and the immediate hardware drivers (but even then, I don't see why a high-level language couldn't be extended with the appropriate functions to do the task), but some things in modern operating systems are definitely archaic.

Why is it for example that we still maintain the memory/filesystem dichotomy? It is about as absurd as requiring that a programmer have to handle the mainboard-level cache by hand. Why is it that whenever any program wants to save data, it must painstakingly convert it into a binary representation? Why is it that subroutines and programs still have to be distinguished? Why is it so painful to have reflexivity (e.g. for user-mode Linux you have to recompile a whole new kernel, the one already in place is not reflexive / reentrant in that sense)? Why is it that for security measures we rely on hardware control (aka MMU's) rather than formal invariance proofs? Why is it that our processors are so big and bloated and distribution / (asymmetric) multiprocessoring / clustering is so far behind?

For more information about what a high-level operating system might be, I refer to the Hurd (which is high-level, but at the cost of an abstraction inversion, because it is still written in C; notice that the Hurd Really Runs), to Erlang, which deserves to be better known because it's really impressive, and to the all-ambitious Tunes project.

A language that might be well suited to genetic programming and proving math theorems is Erlang (http://www.erlang.org). It supports all sorts of neat stuff, like hot swap code changes, and the ability to run both the new and old code at the same time. Your program could evolve.

There's a lot of info on genetic programming available on http://www.geneticprogramming.com.

The best way of dealing with complexity is to simplify and limit features to the essentials. JavaScript, graphics, fancy layout, etc. all can present an bottomless hole for time and bandwidth.

Another thing to realize is that the relational database you are using is probably one of your biggest bottlenecks. Relational databases are slow. MySQL is actually one of the faster ones, but that's because it doesn't make a lot of the guarantees or provide a lot of the features that a "real" relational database provides. You can get a lot of performance from your relational database by tuning it, but ultimately, the architecture and functionality itself present a limit. If speed is of the essence, consider using dbm, plain files, or memory mapped files (Apache has several "databases" internally, and that's its approach.)

For a single person project, Java is probably not the best implementation language. It really shines for multiprogrammer projects. You may find that Python, PHP, or Ruby are better choices. Perl and Tcl are also widely used, but they are also the oldest of the scripting languages and have a lot of rough edges and clunkiness. The performance of all of them is excellent when used as server plug-ins. Perl and PHP have by far the largest libraries and toolsets.

Some of the packages I haven't used but that look interesting are the following. Enhydra takes a much better approach to dynamic HTML than most other packages, I think. Erlang / Eddie address scalability, reliability, and clustering in a really clean way (but you have to learn a new language). Zope I think has a lot of good ideas, but I'm not sure I'd use it for a large server.

Most load balancing solutions use some kind of network routing hacks. That's efficient, but can be a pain to configure. However, there is now a load balancing module for Apache that works as a proxy; that's probably also worth looking into.

The Coda file system is a free, next-generation AFS; while I'm not sure Coda is mature enough yet to be used in heavy-duty applications, systems like it help with scalability by replicating files automatically and have been used on some really large web sites.

I would stay away from commercial "application server platforms". They often are mostly repackaged open source software, and they are expensive and complex, and while they claim to be general, they are also usually created with fairly specific commercial applications in mind ("shopping cart", etc.). I think those kinds of packages are worthwhile for corporate developers that work with large teams of programmers and need consistency, documentation, training, and support.

So, to recap, this kind of stuff is still a lot of work and it seems like you are ahead actually. But there are a lot of proven open source tools available that have been used on really big projects. There are also a lot of open source developments coming around that may make this kind of project a easier.

Distributed computing is a cool idea, but (there's always a but) you still need to write your programs to be distributed

> But besides Netscape, which `corporation' has
> actually gave us any interesting source code
> (not hidden behind 1,0000 lines of legal crap)?

Ericsson has Open Sourced the programming
language/compiler/etc Erlang
and also the web-farm thingy (written in Erlang)
called Eddie.

They're both pretty significant hunks of code.