Performance Benchmarks of Nine Languages

ikewillis writes "OSnews compares the relative performance of nine languages and variants on Windows: Java 1.3.1, Java 1.4.2, C compiled with gcc 3.3.1, Python 2.3.2, Python compiled with Psyco 1.1.1, Visual Basic, Visual C#, Visual C++, and Visual J#. His conclusion was that Visual C++ was the winner, but in most of the benchmarks Java 1.4 performed on par with native code, even surpassing gcc 3.3.1's performance. I conducted my own tests pitting Java 1.4 against gcc 3.3 and icc 8.0 using his benchmark code, and found Java to perform significantly worse than C on Linux/Athlon."

Comments

Accurate?

[Nadsat]

Not sure of the accuracy. Benchmark is on a loop:

32-bit integer math: using a 32-bit integer loop counter and 32-bit integer operands, alternate among the four arithmetic functions while working through a loop from one to one billion. That is, calculate the following (while discarding any remainders)....

It also relies on the strength of the compiler, not just the strength of the language.

Why did VB do so bad on IO.

[nberardi]

Why did VB do so bad on IO compared to the other .Net benchmarks? They were pretty much equal up until the IO benchmarks? Any chance of getting the code published that was used to test this?

Speed or accuracy?

[derek_farn]

The Java performance is best explained by an article by Prof Kahan: "How JAVA's Floating-Point Hurts Everyone Everywhere" http://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf also see "Marketing vs. Mathematics" http://www.cs.berkeley.edu/~wkahan/MktgMath.pdf I suspect the relatively poor floating-point performance of gcc is also caused by the desire to acheive accurate results.

Re:Trig functions...

[mengel]

Probably the Microsoft languages use the Intel trig instructions.

In the case of Java, you find that the Intel floating point trig instructions don't meet the Java machine spec. So they had to implement them as a function.

It all depends if you want accuracy or speed.

Re:They should benchmark development time

[ultrabot]

Someone should do a study on the time taken to design, implement and debug a resonably complex chunk of code under C++ and Java. I'm pretty sure that the result would show the huge advanatage of Java over C++.

The difference b/w Java and C++ would be dwarfed by the difference b/w Java and Python. Java may be 30-40% more productive than C++, but Python is 1000% more productive than Java. And yes, this applies to larger projects. J2EE may come to its own w/ projects that have hundreds of mediocre programmers, but if you have a mid-size team of highly skilled developers creating something new & unique (something like Zope or Chandler), Python will trounce the competition.

Re:Wow

[be-fan]

According to these benchmarks it doesn't.

The short of it is that GCC 3.2.1 is highly competitive with ICC 7.0, except for two cases:

FP-intensive code on the Pentium 4
Code that allows Intel C++ to auto-generate SSE vector code for it

Re:Wow

[LinuxInDallas]

There was an interesting article in Dr Dobb's a few months back. They did a performace (C++) comparison of 6 or so compilers, gcc included. The end result was that performace wise (execution AND code size) gcc came in last place in all their testing. However, gcc did win when it came to conformance to the C++ standard as it was the only compiler that supported all the language features.

Re:Trig functions...

[dnoyeb]

Just because Swing is slow does not make it crappy. It meets nicely what it was designed to do. I use swing applications all the time. Today we have 1GHz processors, its not even an issue any longer, but it wont be allowed to die...

Eclipse is nice, I love eclipse. But I dont mistake it as a Swing replacement. AWT has a purpose, as does Swing and SWT, they are all different.

I believe AWT should be as fast as SWT because its also natively implemented.

I just sped the Python version by 7x and 1.5x

[b0rken]

IMO this benchmark is nonsense, and the way the Python code is written is even worse. I looked at the "trig" and I/O benchmarks. In the i/o benchmark, the output is assembled in the stupidest way possible:

linesToWrite = [myString]
for i in range(ioMax - 1):
linesToWrite.append(myString)

Changing this to 'linesToWrite = [myString] * ioMax' dropped time on my system from 2830ms to 1780ms (I'd like to note that I/O on my system was already much faster than his *best* I/O score, thank you very much Linux)

In the trig test, I used numarray to decrease the runtime from 47660.0ms to *6430.0ms*. The original timing matches his pretty closely, which means that numarray would probably beat his gcc timings handily, too. Any time you're working with a billion numbers in Python, it's a safe bet that you should probably use numarray!

I didn't immediately see how to translate his other mathematical tests into numarray, but I noted that his textual explanation in the article doesn't match the (python) source code!

(My system is a 2.4GHz Pentium IV running RedHat 9)