Slashdot Mirror

← Back to Stories (view on slashdot.org)

The Great Computer Language Shootout Revived

Posted by timothy on from the sputters-to-life dept.

An anonymous reader writes "Doug Bagley's famous Great computer Language Shootout more or less died in 2001 out of lack of support by its own author. A group of developers have decided to revive it and update it with the latest versions of each compiler and interpreter available on the Debian distribution. The good news is, a wiki has been set up so that people can help improve the shootout, discuss the implemetations of the programs, and suggest optimizations."

51 comments

  1. Judging by the lack of comments... by Anonymous Coward · · Score: 2, Funny

    Judging by the lack of comments in here, I'd say it isn't just the author who lost interest.

    1. Re:Judging by the lack of comments... by turgid · · Score: 2, Funny

      No man. We're all flame-warred out from the other day.

      --
      Stick Men
  2. And that'd be...? by Anonymous Coward · · Score: 2, Funny
    ...and update it with the latest versions of each compiler and interpreter available on the Debian distribution.

    I'm fighting the urge to ask how C compiled with gcc 1.3 will compare to COBOL...

    1. Re:And that'd be...? by turgid · · Score: 1
      On an IBM 360?

      No! Please! Not the JCL again! Anything but the JCL!

      --
      Stick Men
    2. Re:And that'd be...? by Detritus · · Score: 1
      Your job has resulted in an ABEND and a core dump.

      Try again, puny mortal.

      --
      Mea navis aericumbens anguillis abundat
  3. gcj vs Sun? by BRSloth · · Score: 3, Informative

    They could have added comparsions between gcj and Java.

    At first, when I saw the Java comparsions, I tought "ok, that will send the last report that says that Sun Java is faster than g++ to space!" but checking the "Implementations" section, it showed that they used gcj for its java comparsion.

    1. Re:gcj vs Sun? by Anonymous Coward · · Score: 0

      Sun's implementation is not available as free software or as a debian package.

    2. Re:gcj vs Sun? by bonniot · · Score: 1

      Actually, it seems that they are using gij, which is the Java interpreter from the GCC team. It would be interesting to also get the results when natively compiled with gcj.

      --
      Watch great movie opening scenes!
  4. Flawed by Jordy · · Score: 4, Informative

    Going through some of these sample programs, I see some serious flaws. The C implementation of regex just calls pcre. The C implementation of Hello World calls fputs instead of write(). The C++ implementation of hash uses sprintf and strdup. The C and C++ implementations of the fibonatchi sequence test are recursive. The tests themselves are so short that you are measuring the time to load the binary into memory and cleanup for half the tests.

    Really there should be some automated way to submit a replacement for some of these tests that gets peer reviewed. That way each language has the "best" implementation for the language.

    Of course, it would have to be considered the "best practices" use of the language as you could always write C in C++ (call write() for Hello World for instance).

    --
    The world is neither black nor white nor good nor evil, only many shades of CowboyNeal.
    1. Re:Flawed by Vaevictis666 · · Score: 3, Informative
      Keep in mind that the purpose is to test a "method"

      The fibonacci test, for example, is not testing fibonacci number generation, it's testing recursion. (Same deal with the Ackerman test) If you were allowed to do it iteratively, you're not testing the right thing.

      Also, if you want to submit a replacement that you think will do better, feel free. Just make sure it's doing what the test spec says.

      Please, do read the methodology page, it's there for a reason.

    2. Re:Flawed by Jordy · · Score: 2, Informative

      Ok, I can understand that one. The rest I'm not sure about. Surely using strdup() and sprintf() in a C++ program to hash is wrong or using fputs() to output a string. Plus, how the heck did they use 2 megs of memory for Hello World?

      Static:

      jordy 21022 0.0 0.0 388 60 pts/1 T 12:39 0:00 /home/jordy/hello

      Dynamic:

      jordy 21038 0.0 0.0 1344 228 pts/1 T 12:40 0:00 /home/jordy/hello

      --
      The world is neither black nor white nor good nor evil, only many shades of CowboyNeal.
    3. Re:Flawed by Brandybuck · · Score: 1

      Going through some of these sample programs, I see some serious flaws.

      Well of course! How else are you going to promote your own favorite language unless you sabotage those you don't like?

      --
      Don't blame me, I didn't vote for either of them!
    4. Re:Flawed by Sigma+7 · · Score: 4, Informative
      The C implementation of regex just calls pcre.
      Yes. If you have a better regular expression library, you're welcome to submit as a more efficient submission.

      The C implementation of Hello World calls fputs instead of write()
      fputs() is an ANSI C standard function designed to write text to an output stream or file. Write() is not ANSI C compatable, requires knowledge of the length of the string (either truncating if the length parameter is too low, or writes garbage if the length parameter is too high), and is designed for binary output rather than text.

      When writing a portable C application, you should *NEVER* assume that a platform is using anything other than the ANSI C libraries except when absolutly necessairy (e.g. a mutlimedia library.)

      The C++ implementation of hash uses sprintf and strdup.
      Correct it then. The webpage containing the benchamarks has links available to contact the maintainers of the website so that you can submit a more efficient function.

      This, of course, assumes that the HM class is not affected by the buf[] string being overwritten. If it is, then you'll have to find another way to eliminate the sprintf/strdup inefficiency.

      The C and C++ implementations of the fibonatchi sequence test are recursive.
      The fibonacci numbers test is explicitly stated that the procedure must be use recursion. If you want, you can ask the website maintainer to add a new test containg the iterative counterpart.

      Just because there is an absolute best way of doing things does not mean that it is permitted in the test. Read the Testing Methodology for more information.

      The tests themselves are so short that you are measuring the time to load the binary into memory and cleanup for half the tests
      That's why the webpage provides an option to exclude startup time in the performance metrics. The option to do so is located at the top of the table, just below the Title naming the test.
    5. Re:Flawed by Anonymous Coward · · Score: 0

      The C++ implementation of hash uses sprintf and strdup

      As was pointed out in the many comments concerning Java's speed when compared to C++, that hash code for C++ is seriously screwed up. It leaks memory like crazy, and one of the examples doesn't even do the same thing as the Java example because the code uses hash_map::operator[]() to test for element existence, instead of hash_map::count(). The former checks to see if the a value for the key exists, creates a default-initialized value for the key if it doesn't exist, and returns a reference to the value. The latter merely returns a 0 if it doesn't exist or a 1 if it does. The output is the same only because the default-initialized value of an integer is zero, which evaluates to false.

      I agree that better programs are needed. The person writing the program should be very experienced in the language, not someone who can barely get his code to work.

    6. Re:Flawed by Jordy · · Score: 4, Insightful

      fputs() is an ANSI C standard function designed to write text to an output stream or file. Write() is not ANSI C compatable, requires knowledge of the length of the string (either truncating if the length parameter is too low, or writes garbage if the length parameter is too high), and is designed for binary output rather than text.

      write() is POSIX/BSD/SVr4 which is certainly fine for the vast majority of platforms. Even with doing the strlen() and a const, it is cheaper than fputs() due to the locking fputs() puts on stdout (admittedly nothing compared to loading the binary into memory). Further, even if you wanted to use ANSI C, they should have just used puts().

      Correct it then. The webpage containing the benchamarks has links available to contact the maintainers of the website so that you can submit a more efficient function.

      Every single test for every single language is almost certainly flawed in some way. If they wanted to do it right, they should have allowed open submission and peer review of every test so that they could be written and rewritten then tested.

      The point is, the person running this is certainly not an expert in every single language. I doubt very much that they can make a proper decision between four different and seemingly correct implementations of the hash implementation. Just submitting replacements doesn't make much sense because *my* implementation could raise just as many questions as the up right now.

      This, of course, assumes that the HM class is not affected by the buf[] string being overwritten. If it is, then you'll have to find another way to eliminate the sprintf/strdup inefficiency.

      Oddly enough, the "correct" implementation from a best practices standpoint may be slower by using std::string instead. It all depends on what your definition of "correct" is. That is why an open peer review process would be helpful as the feedback would educate people as to why certain design choices were made.

      The question you have to ask yourself is, do the results on this page reflect reality in any way?

      --
      The world is neither black nor white nor good nor evil, only many shades of CowboyNeal.
    7. Re:Flawed by Anonymous Coward · · Score: 0

      "
      Really there should be some automated way to submit a replacement for some of these tests that gets peer reviewed. That way each language has the "best" implementation for the language."

      Did you actually read the article ?

      There is a wiki just for that !

    8. Re:Flawed by 0x0d0a · · Score: 2, Insightful

      As was pointed out in the many comments concerning Java's speed when compared to C++, that hash code for C++ is seriously screwed up. It leaks memory like crazy, and one of the examples doesn't even do the same thing as the Java example because the code uses hash_map::operator[]() to test for element existence, instead of hash_map::count().

      Funny that you should say that, because hash tables are one of the easiest ways to "leak" memory in Java -- to have stray references floating around.

      --
      May we never see th
    9. Re:Flawed by E_elven · · Score: 1

      Did anyone notice the 'C++' network server? It uses FORK() for crying out loud! Aaaaargh.

      .

      --
      Marxist evolution is just N generations away!
    10. Re:Flawed by Hognoxious · · Score: 0
      The person writing the program should be very experienced in the language, not someone who can barely get his code to work.
      From what I've seen, the latter is a more vaid test, in that it's more representative of the typical 'real world' situation. YMMV, of course.
      --
      Confucius say, "Find worm in apple - bad. Find half a worm - worse."
    11. Re:Flawed by Anonymous Coward · · Score: 0

      From what I've seen, the latter is a more vaid test, in that it's more representative of the typical 'real world' situation. YMMV, of course.

      And yet we wonder why software development gets no respect and random Joes with little training can take jobs from us...

    12. Re:Flawed by jbolden · · Score: 1

      That way each language has the "best" implementation for the language.

      The whole point was what a medium - good programmer does rapidly. An immitation of real life corporate programming where the programmer sort of cares, they think about it a little but not too hard and the levle of programmer is good but not great. The purpose was never to test the languages using "best implementation" but rather measure likely implementations of common problems.

    13. Re:Flawed by baruz · · Score: 1

      I don't think that a region of memory can be called properly leaked if you still have a reference to it that can be deallocated. If you want to free it up, release the hash table, and the garbage collector will reclaim it. Truly leaked memory is memory that your program has allocated, is untraceable from any references you currently hold, and which the system will not give back to you because you never told the system that it could reclaim it. Properly garbage collected systems such as Java, Lisp, and Smalltalk cannot leak memory; other resources, perhaps, but not memory.

      --
      He was a verray parfit gentil knight.
  5. If it's true, it's great news by GCP · · Score: 2, Interesting

    I find it quite frustrating that there is such huge inertia in programming languages. Even when languages have some remarkable advantage, programmers won't use it because "nobody uses it."

    I don't deny that popularity provides some huge advantages. I just hope that there can be mechanisms to bootstrap popularity, and this Great Language Shootout is (potentially) the sort of thing the could do that.

    Some languages, for example, have significant advantages in some specialty area: Erlang in parallelism, Lisp as a language for writing specialty languages, bug minimization approaches taken by Eiffel or the functional languages, etc. They may be awful for other types of work, but the fact that a hammer is a terrible saw doesn't make it any less of a great tool for the right problem.

    If there were an active, busy site frequented by developers at which there were a wide variety of benchmarks that allowed uncommon languages to show off their specialty advantages, more developers would do some of their work in those languages, and the increasing popularity would result in better implementations and a flourishing of new languages.

    Right now, the popular languages don't come close to taking full advantage of what we have learned about programming, and yet you would have to be either an academic or a fool to put a lot of effort into designing a great new language. The way things are right now, a great language designer probably has a better chance of being hit by lightning than he has of ever making his great new language popular.

    Anything that reduces this language inertia (and YES that includes .Net) is a Good Thing.

    --
    "Those who have never entered upon scientific pursuits know not a tithe of the poetry by which they are surrounded."
    1. Re:If it's true, it's great news by Brandybuck · · Score: 2, Insightful

      Probably because most of us want to be expert in a few languages than mediocre in many. It's like human language. How many people do you know are fluent in more than one language? Two languages? By the time you get to fluency in three languages, they're getting rare.

      You can't pick the language for the task, because in all probability you don't have the time to learn a new language for each task. There are many times when I have to write a piece of code in an "inappropriate" language simply because I don't have time to learn the appropriate language. I know what it's like to be in the middle of some C++ code thinking, "gee, [Java|Lisp|Ruby|etc] would be perfect here...".

      As with human language, some people are naturals when it comes to being fluent in multiple programming languages. Unfortunately, I am not one of their number.

      --
      Don't blame me, I didn't vote for either of them!
    2. Re:If it's true, it's great news by Anonymous Coward · · Score: 1, Insightful

      Multi-language development is incredibly useful - the last program I wrote has a C/C++/OCaml server with a GUI client written in Delphi.

      Unfortunately, it also means that nobody can easily maintain the program who isn't fluent in all four languages, which is why, while useful, it's also an incredibly bad idea for anything critical.

      The program in question is a quick and dirty solution to a migration issue, and using the mixture of languages allowed me to leverage all their strengths and produce a fast and robust program in record time (the C++ and about half of the C were existing code that OCaml's FFI allowed me to reuse), but I would never consider it for production software - the up-front cost savings would be swallowed by the maintenance costs.

    3. Re:If it's true, it's great news by GCP · · Score: 1

      As with human language, some people are naturals when it comes to being fluent in multiple programming languages. Unfortunately, I am not one of their number.

      That's unfortunate, because the ability to quickly learn to handle new specialty tools has significant competitive advantages--dare I call it "survival value"?

      And with all the things one needs to learn to move beyond junior programmer these days (things like XML, DHTML, accessibility, SQL, MS Excel, version management utilities, scripting for metaprogramming tasks, etc., etc.) it just doesn't make sense to turn up your nose at improved programming languages.

      --
      "Those who have never entered upon scientific pursuits know not a tithe of the poetry by which they are surrounded."
    4. Re:If it's true, it's great news by Anonymous Coward · · Score: 0

      It is great news. I first learned of Ocaml through the shootout.

      A post to a story somewhere on Slashdot recently (unfortunately, I can't remember where) linked to a paper on quantifying variance in performance due to programmer versus language.

      I would very much like to see the Shootout expanded to include such things--i.e., more compilers, the quantifying the impact of invividual programmer, etc.

      The data from the shootout can be very interesting in itself. With the original shootout, I did cluster analyses to statistically analyze patterns of performance among languages. What you ended up with was a classification system for programming languages based on empirical performance characteristics. It was interesting to see how these languages clustered together. I shared it with the original shootout maintainer, and he was very friendly about the whole thing.

      Anyway, while there may be flaws in the shootout, it had many benefits as well. I hope to see it continued and improved.

    5. Re:If it's true, it's great news by Brandybuck · · Score: 1

      That's unfortunate, because the ability to quickly learn to handle new specialty tools has significant competitive advantages--dare I call it "survival value"?

      I can definitely learn a new language. There is no problem with that. But if they expect me to aquire complete mastery of the new language in a week, I will have to be honest and say I cannot do it. Frankly, of forty experienced software developers at my work, I can think of one who might be able to perform this feat.

      Can YOU become an expert in Lisp in one week? Heck, start with the easy ones, what about Ruby or Python? Not just knowing the language, but attaining an expert level of proficiency in it.

      --
      Don't blame me, I didn't vote for either of them!
    6. Re:If it's true, it's great news by GCP · · Score: 2, Insightful

      Can YOU become an expert in Lisp in one week?

      I've actually spent a number of years using Lisp. I've been paid to write software in somewhere between a dozen and two dozen languages, depending on your definition of language, and I assure you that your "acquire complete mastery of a new language in a week" test is completely irrelevant.

      If you haven't even started to look at a language until a week before you need to be a master, it's not the fault of the project. You simply aren't qualified for that particular job.

      But, unless you never do anything until someone else forces you to do it, you usually have years in which to get to know a new language. Over that time, you will either determine that it is not useful enough to add to your toolbox or you will have added it to the extent you feel it warrants.

      So all I'm talking about is the advantages of a site like the Great Language Shootout that gives you a lot of information with which to make your decisions over that period of years that I'm referring to.

      --
      "Those who have never entered upon scientific pursuits know not a tithe of the poetry by which they are surrounded."
    7. Re:If it's true, it's great news by LardBrattish · · Score: 2, Insightful

      It's more than that.

      In the real world someone is paying to get software done. That someone does not want to pay us to learn a new language just so a function doesn't need to be written because in language X it's a one liner. Write the equivalent function or find a suitable existing library. If you program in C/C++ the library probably does exist. In Haskell or Eiffel...

      Also real commercial products do not exist in a vacuum. You will probably have existing code that does at least part of the job. The person paying the bills will not want to pay to port all of that code. They will want to leverage it.

      That's why the new cool language du jour does not rule the world. It's frustrating if the language you want to use is demonstrably better but factor in the cost of finding more progammers that can use the language effectvely & most of the advantages you perceive evaporate.

      --
      What are you listening to? (http://megamanic.blogetery.com/)
    8. Re:If it's true, it's great news by DavidNWelton · · Score: 1
      I find it quite frustrating that there is such huge inertia in programming languages. Even when languages have some remarkable advantage, programmers won't use it because "nobody uses it."


      You want to look up "positive network externalities" from the field of economics. Basically, the idea says that something is more valuable the more people use it. For instance, if you are the only one to have a mobile phone, it's a worthless hunk of plastic, but if all your friends have one too, it's a convenient way to keep in touch. The same thing goes for programming languages, to some degree, because you can share and learn from others.
      --
      http://www.welton.it/davidw/
    9. Re:If it's true, it's great news by GCP · · Score: 1

      Now *you* look up what I posted. I never said I couldn't understand why popularity mattered. I said that something like the Great Shootout could serve to bootstrap popularity. In an area like programming languages, which is subject to strong network effects, anything that could get the snowball started for a wider variety of languages would be useful--as I said.

      --
      "Those who have never entered upon scientific pursuits know not a tithe of the poetry by which they are surrounded."
    10. Re:If it's true, it's great news by Brandybuck · · Score: 1

      But, unless you never do anything until someone else forces you to do it, you usually have years in which to get to know a new language.

      But that contradicts you're earlier statement. Earlier you said that "the ability to quickly learn" new lanaguages was a survival advantage. Now you're saying I have years in which to do it.

      "Survival advantage" in context means that you must quickly adapt to and use the language your pointy headed buffoon has a hardon for. You can't produce good code in this situation. You can't learn Latin in a week to a level capable of reading Cicero.

      --
      Don't blame me, I didn't vote for either of them!
  6. Same Story as Yesterday by xp · · Score: 1

    Isn't this the same story as the one yesterday about how Java is faster than C++ under "suitable" conditions.

    ----
    Notes on Stuff

    1. Re:Same Story as Yesterday by Anonymous Coward · · Score: 0

      No it isn't, and it's a pity that slashdoters seem to prefer to waste their time with hundreds of posts of flamefests, complaining that the code snippets of the shootout are flawed, than actually submitting solutions in the wiki which is at their disposal.

      Is the community spirit slowly leaving Slashdot ?

  7. Programmers where? by Anonymous Coward · · Score: 0

    And where exactly can you find a good suppy of, say, Erlang programmers? Compare that with yucky java.

  8. Oh Boy! by Rhesus+Piece · · Score: 2, Funny

    Oh boy! There is a "Manufacture your own results" cgi! Now we can all play the "change the weighting so your language of choice wins" game!

    I changed everything to '1', and Ocaml won.

  9. System Calls by Detritus · · Score: 2, Insightful
    The C implementation of Hello World calls fputs instead of write().

    write(2) is a system call on UNIX systems, not a part of the C language or the C standard library. It has no place in a portable C program. You might as well argue that the program should have use DosWrite() (OS/2 system call) or $QIO (VMS system call).

    --
    Mea navis aericumbens anguillis abundat
    1. Re:System Calls by Greg+Lindahl · · Score: 1


      There's this little standard called "POSIX". A very large number of OSes, including many that aren't Unix, support POSIX. OpenVMS is one example.

    2. Re:System Calls by Detritus · · Score: 1

      A large number of operating systems do not support POSIX, support an incomplete subset, or have a broken implementation. If it isn't in ISO/ANSI C, it isn't standard or portable.

      --
      Mea navis aericumbens anguillis abundat
  10. Actually, they made a new hashtable test by Anonymous Coward · · Score: 0

    If you got to their wiki, they explicitly recognize the problem.

    That's why they set up another test for hashes (namely Hash II).

    Therefore, when evaluating the final result, all you have to do is to set a weight of 0 to the test Hash I, and the resulting evaluation is Ok.

    As flawed as it is (but which comparison isn't ?), the shootout probably is the most comprehensive language comparison available.

    1. Re:Actually, they made a new hashtable test by Anonymous Coward · · Score: 0

      If you got to their wiki, they explicitly recognize the problem.

      That's why they set up another test for hashes (namely Hash II).

      The wiki says that they were trying to reduce the time spent generating the keys. They did not acknowledge the fact that the C++ program modifies the hash when the others do not.

  11. C still kicks by wazerface · · Score: 1

    for(i=0;language[GCC]>language[i];i++) too_true++; Better make too_true a long long.

  12. The Python results seem suspicious by Colonel+Panic · · Score: 2, Insightful

    Apparently they used Psycho to get the Python numbers. This is why the Python numbers look so good. However, Psycho is not the standard distribution of Python and it doesn't allow the full flexibility of standard Python. This should be noted in the results. They should create a set of results using the 'standard' Python distro as well. ...otherwise, why not just let us use Inline::C in Perl (or in Ruby) - basically this would allow you to put C code into your Perl/Ruby script (which wouldn't be fair).

  13. Missing a lot of languages by LWATCDR · · Score: 1

    Where is Fortran, Pascal, and Basic?
    I know these are old but you can get them all under Linux.
    Pascal at http://www.gnu-pascal.de/gpc/h-index.html
    Fortran at http://world.std.com/~burley/g77.html This is g77 I do not know if G95 is ready yet.
    And I have seen a few Basics.

    --
    See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
  14. Debian is dying by Anonymous Coward · · Score: 0

    fact> debian is dying
    no new releases...
    they still have the linux kernels 2.2.x and 2.4.x
    archaic installer
    it's full of bugs / the most buggy and unsecure open source os. linuxsecurity.com/advisories

  15. Facts about it by Anonymous Coward · · Score: 0

    Lack of comments, lost interest in something old = Debian is dying
    (Score:5, Funny)

    1. Re:Facts about it by Anonymous Coward · · Score: 0

      the last book about debian linux was released in 2000.
      the last book about freebsd was released last month.
      fact: debian is dying!!!

  16. books by Anonymous Coward · · Score: 0

    there's no interest in writing a single book about debian. last book about debian was released in 2000. debian is dying