Slashdot Mirror
The Great Computer Language Shootout
kato writes: "Doug Bagley has posted results from benchmarking of 29 different language implementations solving 25 different problems (he's written ~600 of the 725 programs so far). The languages include C/C++, Perl, Python, Eiffel, BASH, Tcl, and OCaml. The problems range in complexity from "Hello, World!" to the Sieve of Eratosthenes and Matrix Multiplication. The results can be sorted by speed, memory usage, or lines of code. You can also give one particular program more weight than another (if you are doing more client/server code than "Hello, World!") and find the faster/smallest/shortest language implementation. I can see many of my programs being written in OCaml from now on." Update: 07/04 12:42 PM by CT : The site is apparently now redirecting people back here. I guess technically thats an error message, just not a helpful one. Update: 07/05 8:40 PM by M : Please don't email. The link is broken. We know. The guy is running a server at home on a metered connection, and doesn't want any more traffic.
I am not dismissing the value of this work, but so long as developer & maintainer time is more valuable than processor time, when you choose a language, you should consider how well each language will make the developers' and (especially) the maintainers' jobs easier.
An example of how this can be considered is found in the appendices of "A Gentle Introduction to Software Engineering" (2,565 KB; MS Word format) by the Air Force's Software Technology Support Center (STSC). The appendices come from a document written in March 1996, so it's a little dated (predates the C++ standard, does not address Java, Python, or Perl), but the idea is there. (I like that it keeps repeating "Bad programmers can write bad code in any language, but a good language facilitates the production of good code by good programmers.".)
Including the performance achievable with each language is already in the formulation offered by STSC. The obvious caveat is that you look at the performance of benchmarks related to your application.
cb
cb
Oooh! What does this button do!?
http://pauillac.inria.fr/~furuse/generics/
For real-world use, would you choose a language that compiled out to run 5% faster if it was also 20% harder to maintain?
IT has a misplaced fixation on speed.
--
Sheesh, evil *and* a jerk. -- Jade
He forgot native Assembly for whatever platform he's working on.
And he forgot to test each implementation in single processor, SMP, or Beowulf cluster (Imagine one!) on his platform.
For that matter, he forgot to test it on many different platforms.
And with different hard drive an memory types too.
And at different elevations... computing runs faster in the Colorado air...
And he forgot to test it with or without water cooling or a Peltier cooler... or both...
Finally, he forgot what the sun looks like.
A much fairer contest is the pseudoknot benchmark. The idea is to take one real-world task (not a partial task like matrix multiplication), and get experienced programmers to write a program to solve the problem in whatever is the most natural way for some language. The results are then benchmarked on equivalent hardware.
Of course it's not representative of all programs. Pseudoknot is a floating point-intensive search problem, which is not the sort of thing that I do most of the time. Still, it's a better example than a self-confessed beginner trying out toy problems.
sub f{($f)=@_;print"$f(q{$f});";}f(q{sub f{($f)=@_;print"$f(q{$f});";}f});
Or how difficult it would be to fix a bug or add a new feature. Or how robustly it would perform in the presence of other kinds of failure (e.g. unexpected input, hardware failure etc). Or how easy or difficult it would be for a larger group of people to work on the same program. Or how easy it would be to adapt his programs to work with other pre-existing programs.
sub f{($f)=@_;print"$f(q{$f});";}f(q{sub f{($f)=@_;print"$f(q{$f});";}f});
If you don't think "Hello World" is worthwhile, enter a weight of 0 for it and press Recalculate Scores. That's why Doug provided the flexible interface. Of course the server is probably too slow for you to access right now.
I'd strongly disagree with that. Nobody, nowhere, writes optimal code. What good does it do you if you can prove that language FooBar has the best optimal implementation of problem X, but nobody in the real world actually writes it that way? (Hint: none at all)
I think the important metric is not that the programs are optimal, but that they are representative of what a programmer of average skill in a particular language would produce. After all, one of the benefits of a good language is how easy it is to use -- and we presume that this ease of use will be reflected by better-written benchmarks. It's a lot easier, for instance, to write good code in Java or Smalltalk than in assembly, so why shouldn't those languages be able to show some benefit from that in a test like this?
Of course, ease of use is impossible to quantify in a test like this, but I'd argue that shooting for optimal (i.e. written by somebody far more skilled than an average programmer) will seriously distort one's expectations of real-world usability of these languages.
ZFS: because love is never having to say fsck
Here's a good example of this:
for(int i = 0; i < vector.size(); i++)
do_something(vector.element_at(i));
This code potentialy has a O(n^2) complexity if the size of vector is calculated dynamically each time size() is called versus
int size = vector.size();
for(int i = 0; i < size; i++)
do_something(vector.element_at(i));
which should have O(n) complexity with regards to traversing the vector. The interesting thing is that the previous version could potentially be O(n) depending on how the size() method is implemented in the vector class. But how many people actually know if the size() method in the vector class in whatever library they use (e.g. Java, C++, etc) actually use lazy evaluation or overeager evaluation? <br><br>
NOTE: Regardless of which language used. If a benchmark is run using vectors which dynamically calculates their size and contain a large amount of elements then the code in the first part will ALWAYS run slower than the code in the second part even if the comparison is Java and C/C++.
--
size() doesn't use "lazy" or "overeager" evaluation, it's an accessor method.
Evaluation is deemed overeager or eager if the value is computed as you go along instead of just when it is needed. The fact that the C++ vector implementation increments and decrements a counter on pushes and pops instead of when size() is called is eager evaluation .
--
Me thinks that his email server will be slashdotted before his website is with every geek in teh world telling him how to improve the programs he wrote.
Which is a good thing - I'd love to see before and after results with after being after all teh /.'er changes are added - course that assumes /.ers would be able to agree on the best route of action. ROFLMAO!
Seriously - very interesting project - he'll ctach flack for being a newbie I'm sure - but a great endeavour all around. More power to him!
Top Most Bizarre/Disturbing Error Messages
Furthermore, type declarations are not there for speed, they are there for correctness. ML programs (O'CAML or SML) usually run correctly once they compile; with CL, you spend a lot of time unit-testing for silly type errors.
I think an updated version of CL would be great, something that is based on UNICODE, throws out the old pathname and character set cruft, gets rid of some other obsolete features, defines error handling and reflection carefully, and adds threads, sockets, and binary I/O. But I don't see it happening. Most of the people who want CL-like interactivity are now using Python or Java+Beanshell. The syntax isn't as nice, but they are so much more practical.
I see no mention of how long it took him to code any of these...
I can see many of my programs being written in OCaml from now on.
Ok, about O'caml [O'caml = Objective Caml = a variant of ML with objects]. I had to use it for a PL class last semester. It's kind of nice, but don't expect it to be in any way like any language you've ever used (unless you know some other ML variant, of course).
In many ways it's very nice. In other ways it will drive you absolutely insane. For example, O'caml has very strong typing rules (which are good). It has almost no diagnostics when it decides it doesn't like your code (which is very very bad). It also apperantly sometimes gets very confused. For example, sometimes you'll get something like this:
foo.ml, line 100, chars 16-30: this expression has type int, but is used here with type int
This can be a little maddening after a few hours.
Other annoyances: no function overloading. So to print a string, use print_string. To print an int, use print_int. And so on. This is just how it works (I think you can write a wrapper which will check the type and dispatch the right call, but that's fairly irritating in and of itself).
Another thing that irritates me (probably just because I'm from a C/C++/Perl background), is that sometimes you use no ';' to end a statement, sometimes one, sometimes a pair.
Also, I found the documentation to be very erratic (the modules docs are quite complete, but try finding simple examples of how to do OO, without reading the BNF grammar they put in the docs - that's no way to learn a language).
Nice stuff: strong typing, cool type matching stuff, bytecode and native compilers, seems like a decent module system.
But if you don't already know it, good luck. You'll need it.
Apparently the server was running Commodore 64 BASIC.... on native hardware.
sig fault
I don't know about that. In C you would have to include headers and define the main function. In perl easy things are easy so it would just be
shoot($foot);
Thanks perl!
This Wiki Feeds You TV and Anime - vidwiki.org
-
The combination of jar-file packaging, pure interfaces, and dynamic class-loading by name make it easy to use a "pluggable component" approach. In my opinion, this goes beyong traditional OO development in making large-scale development manageable.
-
Automatic memory management and no pointers, love it or hate it, saves boatloads of time in chasing down those old runaway pointer bugs. Of course, you can still leak memory, but you have to work a little harder to do it.
-
The built-in Java libraries are as good as or better than the C runtime, if you ask me.
-
It looks damned nice on the resume these days.
Of course, if you need to write anything that is CPU-bound and has to be balls-to-the-wall fast, you are still better off using C++, or probably C or assembler. But most "enterprise" applications are not CPU bound, anyway."Rub her feet." -- L.L.
Uh, how is this possible? Even if the semantics of all these languages were formally specified (which, of course, they're not) and even if the semantics included execution time information (which I've never ever seen before), the task of proving optimality seems impossible. No, I take that back, it is impossible.
Because code transformation in OCaml is not pretty. Owning to CL's clean and uniform syntax, macro in CL is simple and elegant; this metaprogramming capability provides it with unlimited expressiveness and adaptibility.
Because OCaml doesn't provide the option to delay deciding types. In CL, I write generic code to speed up development; I then (optionally) declare types in bottleneck code segments to speed up program execution -- it makes perfect sense (remember the 90/10 rules).
Because OCaml doesn't have a Meta Object Protocol.
CMUCL is a highly optimized, Free compiler and execution system for Common Lisp (http://cmucl.cons.org/cmucl). Please come and help making it better (http://www.telent.net/cliki)!
Personally I like C because I can shoot myself in the foot faster and with less effort with it.
... is my favorite language: Brainfuck.
Here
As pointed out by a previous poster, this author is a beginner in most of these languages. I had an interesting experience in a graduate level CS class here in Berkeley on optimizing a matrix-matrix multiply routine. (Interestingly enough, the class was on parallel computing -- the point of the exercise was to learn just HOW INCREDIBLY important the serial part of one's algorithm is, even when one has oodles of processors available).
The results are interesting for a number of reasons.
(1) The "naive" algorithm, even with optimization, performed at about 50 MFlops (the Suns we used had a theoretical peak of 333 MFlops).
(2) With excellent optimization, pulling out all the stops, and using all the tricks available (unrolling loops, deallocating pointers to local variables, etc.), teams of just two students working for a week could get EXCELLENT performances (ie, within 10-20% of theoretical peak), approaching those of Sun's built-in library, and exceeding those of some existing libraries (like PHiPAC).
(3) Different groups with different approaches got very widely disparate results -- some barely exceeding those of the naive algorithm.
In sum, how ones goes about coding an algorithm can make ENORMOUS differences in the performance of a code. This is particularly true with numerical algorithms in C and other languages with pointers, where some compilers have great trouble optimizing routines using pointers, since the values are not known at compilation time. Taking this into account, I wouldn't give this guy's results much credence at all.
However, with the help of a lot of experts in the various languages, it will be possible to get a much better appraisal of the relative performances of different languages. A close analogy exists with these benchmarks and the SPEC open standards evaluations for CPUs. The only fair way they found, to compare across all CPUs and compilers, was to allow a very strict non-optimal compilation, and no-holds barred compilation. The same is true here -- we need to get teams to go no-holds barred in the creation of the best possible codes for each language.
Bob
Science, like Nature, must also be tamed, with a view turned towards its preservation.
Hrm. Never even heard of Ocaml.
It's the Irish version of Perl. (ouch.)
--
"that's not encryption - it's a new perl script that I'm working on..." - from some Matrix parody
...it sounds like the PL/I of the new millenium! (smirk)
"Rub her feet." -- L.L.
Because it has all the great functional features that can make Lisp programmers happy.
Because it has a wonderful OO model which can make all OO programmers happy.
Because it has super fast compilers that can make C and C++ programmers happy.
Because it is great for imperative programming and for functional programming.
Because it is great for procedural programming and for OO programming.
Because it is as multiplatform and portable as Java.
Because it is designed to please everyone without compromising on anything, and, put more simply, because it can reconciliate the C, Java, Lisp and C++ community.
Because it can even be used indifferently as a scripting or a system language.
Because it is great for teaching AND for the real world.
Because its compilers are libre software and its design and developement are made in a very open fashion.
I took the liberty of benchmarking 25 spoken languages by comparing their methods for expressing the phrase "Wuzzzzzzaaaaaah" from the budweiser commercial. Some of these may not be optimized as I am a bit of a newbie at some of these languages. Here is an excerpt:
American English: "Wuzzzzzzzzzzzzzahhh!"
British English: "What is up?"
Japanese: "Konichiwaaaaaaaaaaaaa!"
Spanish: "Holaaaaaaaaaaaaa!"
Welsh: "Wyffffffwyfffffffffffff!"
The binary language of water vaporators:
"0100000000000000000000000000000000"
Australian English: "I'm not saying any such thing. Give me a Fosters'."
Irish English: "I'm not saying any such thing. Give me another pint of Guiness."
Javanese: " I'm not saying any such thing. Give me another pint of Guiness. No, I'm not the Irish guy in Javanese traditional dress, I'm from Java!"
microsoftword.mp3 - it doesn't care that they're not words...
In other words, if engineers spent more time studying best practices and efficient algorithm design, that might well improve performance more than spending time in religious wars about the merits of pet languages.
In my experience, basic performance tuning can easily create an order of magnitude improvement regardless of the language. And in my experience, most engineers don't know how to do basic performance tuning. That's got to be more important than language selection in the average case.
Tim