Performance Benchmarks of Nine Languages

ikewillis writes "OSnews compares the relative performance of nine languages and variants on Windows: Java 1.3.1, Java 1.4.2, C compiled with gcc 3.3.1, Python 2.3.2, Python compiled with Psyco 1.1.1, Visual Basic, Visual C#, Visual C++, and Visual J#. His conclusion was that Visual C++ was the winner, but in most of the benchmarks Java 1.4 performed on par with native code, even surpassing gcc 3.3.1's performance. I conducted my own tests pitting Java 1.4 against gcc 3.3 and icc 8.0 using his benchmark code, and found Java to perform significantly worse than C on Linux/Athlon."

Comments

Accurate?

[Nadsat]

Not sure of the accuracy. Benchmark is on a loop:

32-bit integer math: using a 32-bit integer loop counter and 32-bit integer operands, alternate among the four arithmetic functions while working through a loop from one to one billion. That is, calculate the following (while discarding any remainders)....

It also relies on the strength of the compiler, not just the strength of the language.

Speed or accuracy?

[derek_farn]

The Java performance is best explained by an article by Prof Kahan: "How JAVA's Floating-Point Hurts Everyone Everywhere" http://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf also see "Marketing vs. Mathematics" http://www.cs.berkeley.edu/~wkahan/MktgMath.pdf I suspect the relatively poor floating-point performance of gcc is also caused by the desire to acheive accurate results.

Re:Trig functions...

[mengel]

Probably the Microsoft languages use the Intel trig instructions.

In the case of Java, you find that the Intel floating point trig instructions don't meet the Java machine spec. So they had to implement them as a function.

It all depends if you want accuracy or speed.

Re:They should benchmark development time

[ultrabot]

Someone should do a study on the time taken to design, implement and debug a resonably complex chunk of code under C++ and Java. I'm pretty sure that the result would show the huge advanatage of Java over C++.

The difference b/w Java and C++ would be dwarfed by the difference b/w Java and Python. Java may be 30-40% more productive than C++, but Python is 1000% more productive than Java. And yes, this applies to larger projects. J2EE may come to its own w/ projects that have hundreds of mediocre programmers, but if you have a mid-size team of highly skilled developers creating something new & unique (something like Zope or Chandler), Python will trounce the competition.

Re:Wow

[be-fan]

According to these benchmarks it doesn't.

The short of it is that GCC 3.2.1 is highly competitive with ICC 7.0, except for two cases:

FP-intensive code on the Pentium 4
Code that allows Intel C++ to auto-generate SSE vector code for it

I just sped the Python version by 7x and 1.5x

[b0rken]

IMO this benchmark is nonsense, and the way the Python code is written is even worse. I looked at the "trig" and I/O benchmarks. In the i/o benchmark, the output is assembled in the stupidest way possible:

linesToWrite = [myString]
for i in range(ioMax - 1):
linesToWrite.append(myString)

Changing this to 'linesToWrite = [myString] * ioMax' dropped time on my system from 2830ms to 1780ms (I'd like to note that I/O on my system was already much faster than his *best* I/O score, thank you very much Linux)

In the trig test, I used numarray to decrease the runtime from 47660.0ms to *6430.0ms*. The original timing matches his pretty closely, which means that numarray would probably beat his gcc timings handily, too. Any time you're working with a billion numbers in Python, it's a safe bet that you should probably use numarray!

I didn't immediately see how to translate his other mathematical tests into numarray, but I noted that his textual explanation in the article doesn't match the (python) source code!

(My system is a 2.4GHz Pentium IV running RedHat 9)