Performance Benchmarks of Nine Languages

ikewillis writes "OSnews compares the relative performance of nine languages and variants on Windows: Java 1.3.1, Java 1.4.2, C compiled with gcc 3.3.1, Python 2.3.2, Python compiled with Psyco 1.1.1, Visual Basic, Visual C#, Visual C++, and Visual J#. His conclusion was that Visual C++ was the winner, but in most of the benchmarks Java 1.4 performed on par with native code, even surpassing gcc 3.3.1's performance. I conducted my own tests pitting Java 1.4 against gcc 3.3 and icc 8.0 using his benchmark code, and found Java to perform significantly worse than C on Linux/Athlon."

Comments

Trig functions...

[eples]

I am not a compiler nerd (IANACN?), so maybe someone else can answer the following simple question:

Why are the Microsoft languages so fast with the Trig functions?

Re:Trig functions...

[mengel]

Probably the Microsoft languages use the Intel trig instructions.

In the case of Java, you find that the Intel floating point trig instructions don't meet the Java machine spec. So they had to implement them as a function.

It all depends if you want accuracy or speed.

Re:Trig functions...

[fuzzbrain]

SWT libraries use native widgets. I think the parent post is perhaps overstating the speed increase of SWT over the latest versions of Swing. The main thing I like about SWT java apps vs swing apps is that they fit into the rest of the desktop better. Another benefit is that SWT and gcj fit well together. Again the benefit here isn't so much speed as reduced download sizes for those users who don't have java installed already.

Re:Trig functions...

[dnoyeb]

Just because Swing is slow does not make it crappy. It meets nicely what it was designed to do. I use swing applications all the time. Today we have 1GHz processors, its not even an issue any longer, but it wont be allowed to die...

Eclipse is nice, I love eclipse. But I dont mistake it as a Swing replacement. AWT has a purpose, as does Swing and SWT, they are all different.

I believe AWT should be as fast as SWT because its also natively implemented.

Re:Trig functions...

[be-fan]

Actually, this is a myth. Modern GC methods do not introduce a huge amount of overhead, when you take into account a few things:

- Algorithms for allocating memory in a manual management scheme can be quite complicated. Look up on how large glibc's memory allocator is. Memory allocation algorithms for GCs tend to be much simpler, often as simple as a simple pointer increment.

- Deallocation algorithms for manual memory management are often even more complicated than the allocation algorithms. They are nearly always slower. Plus, objects are deallocated one at a time. Deallocation algorithms for GC can be simpler, but most importantly, the GC can deallocate large numbers of objects at once. This is, of course, more efficient.

- Copying GCs can compact holes in memory, which makes for better cache utilization.

Depending on the problem at hand, a GC can be a little slower or about the same. For a functional programming style (which has a particular pattern of memory usage) GC is usually faster than manual management. For programs that tend to operate in phases, allocating large numbers of objects gradually and freeing them at once, GC will also be faster.

The real problem with GC is that it affects latency. Modern GCs only freeze the app for a fraction of a second, but that's a large amount of time for something like a game or movie. There are some work arounds for this, though. Latency-sensitive apps can disable GC and use manual memory management. Or, they can use a real-time garbage collector, which has guaranteed latency, but does incur a large fixed overhead. Of course, these problems can be worked around, as evidenced by the fact that major PS2 games like Jak and Daxter were written in a GC'ed language (Common Lisp). Most proponents of GC will tell you that such work-arounds are a good deal easier than hunting down memory leaks and dangling pointers!

Accurate?

[Nadsat]

Not sure of the accuracy. Benchmark is on a loop:

32-bit integer math: using a 32-bit integer loop counter and 32-bit integer operands, alternate among the four arithmetic functions while working through a loop from one to one billion. That is, calculate the following (while discarding any remainders)....

It also relies on the strength of the compiler, not just the strength of the language.

Why did VB do so bad on IO.

[nberardi]

Why did VB do so bad on IO compared to the other .Net benchmarks? They were pretty much equal up until the IO benchmarks? Any chance of getting the code published that was used to test this?

Re:Wow

[finkployd]

Well, for performance it does. For cross platform compilation it rocks the house. If you really want performance you need to be using something like Intel's C compiler (which oddly was not tested)

Finkployd

.NET Languages and IL

[ClubStew]

Why benchmark the various ".NET languages" (those languages whose compilers target the CLR)? Every compiler targeting the CLR produces Intermediate Languages, or more specifically MSIL. The only differences you'd find is in optimizations performed for each compiler, which usually aren't too much (like VB.NET allocates a local variable for the old "Function = ReturnValue" syntax whether you use it or not).

Look at the results for C# and J#. They are almost exactly the same, except for the IO which I highly doubt. Compiler optimizations could squeeze a few more ns or ms out of each procedure, but nothing like that. After all, it's the IL from the mscorlib.dll assembly that's doing most the work for both languages in exactly the same way (it's already compiled and won't differ in execution).

When are people going to get this? I know a lot of people that claim to be ".NET developers" but only know C# and don't realize that the clas libraries can be used by any languages targeting the CLR (and each has their shortcuts).

Would like to see...

[CaptainAlbert]

...some analysis of the code generated by Visual C++ and gcc side by side, particularly for those trig calls. If there's that great a discrepancy between the runtimes, that's a good clue that either one of the compilers is under-optimising (i.e. missing a trick), or the other is over-optimising (i.e. applying some transformation that only approximates what the answer should be). I didn't see any mention of the numerical results obtained being checked against what they ought to be (or even against each other).

As any games/DSP programmer will tell you, there are a million ways to speed up trig providing that you don't *really* care after 6dps or so.

OK, maybe I'm just bitter because I was expecting gcc 3.1 to wipe the floor. :)

What about coder's performance?

[G4from128k]

Given the ever accelerating clockspeed of processors, is the raw performance of langauges that big an issue? Except for CPU-intensive programs (3-D games, high-end video/audio editing), current CPUs offer more than enough horsepower to handle any application. (Even 5-year old CPUs handle almost every task with adequate speed). Thus, code performance is not a big issue for most people.

On the other hand, the time and cost required by the coder is a bigger issue (unless you outsource to India). I would assume that some languages are just easier to design for, easier to write in, and easier to debug. Which of these langauges offers the fastest time to "bug-free" completion for applications of various sizes?

Speed or accuracy?

[derek_farn]

The Java performance is best explained by an article by Prof Kahan: "How JAVA's Floating-Point Hurts Everyone Everywhere" http://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf also see "Marketing vs. Mathematics" http://www.cs.berkeley.edu/~wkahan/MktgMath.pdf I suspect the relatively poor floating-point performance of gcc is also caused by the desire to acheive accurate results.

Re:They should benchmark development time

[ultrabot]

Someone should do a study on the time taken to design, implement and debug a resonably complex chunk of code under C++ and Java. I'm pretty sure that the result would show the huge advanatage of Java over C++.

The difference b/w Java and C++ would be dwarfed by the difference b/w Java and Python. Java may be 30-40% more productive than C++, but Python is 1000% more productive than Java. And yes, this applies to larger projects. J2EE may come to its own w/ projects that have hundreds of mediocre programmers, but if you have a mid-size team of highly skilled developers creating something new & unique (something like Zope or Chandler), Python will trounce the competition.

IBM Java

[PourYourselfSomeTea]

Using the IBM Java VM, I've been able to achieve consistently cutting my runtimes in half over the Sun VM. Anyone currently using the Sun VM for production work should test the IBM one and consider the switch.

My application that I benchmarked is data and network and memory intensive, although not math intensive, so that's what I can speak for. We consistently use 2 GB of main memory and pump a total of 2.5 TB (yes, TB) of data (doing a whole buch of AI style work inside the app itself) through the application over it's life cycle, and we cut our total runtime from 6 days to 2.8 days by switching to the IBM VM.

Re:Wow

[be-fan]

According to these benchmarks it doesn't.

The short of it is that GCC 3.2.1 is highly competitive with ICC 7.0, except for two cases:

FP-intensive code on the Pentium 4
Code that allows Intel C++ to auto-generate SSE vector code for it

Re:Wow

[LinuxInDallas]

There was an interesting article in Dr Dobb's a few months back. They did a performace (C++) comparison of 6 or so compilers, gcc included. The end result was that performace wise (execution AND code size) gcc came in last place in all their testing. However, gcc did win when it came to conformance to the C++ standard as it was the only compiler that supported all the language features.

Re:Under Windows...

[Umrick]

Actually, I'm quite comfortable with the performance numbers Python turned in. I use Python quite a bit, and for the things the benchmark was run on, it's the kind of area I'd find looking for bottlenecks, and in turn implement in C or C++.

Python's huge win is not in speed, but in the ability to express the program in a very concise and easy to understand way.

The fact that Psyco can provide huge speed ups via a simple import is just icing.

Performance not important? Umm , not quite...

[Viol8]

I was a bit surprised by this quote in the article:

"Even if C did still enjoy its traditional performance advantage, there are very few cases (I'm hard pressed to come up with a single example from my work) where performance should be the sole criterion when picking a programming language. I"

I can only assume from this that he has never done or known anyone who has done any realtime programming. If you're going to write something
like a car engine management system performance is the ONLY critiria, hence a lot of these sorts of systems are still hand coded in assembler , never
mind C.

I just sped the Python version by 7x and 1.5x

[b0rken]

IMO this benchmark is nonsense, and the way the Python code is written is even worse. I looked at the "trig" and I/O benchmarks. In the i/o benchmark, the output is assembled in the stupidest way possible:

linesToWrite = [myString]
for i in range(ioMax - 1):
linesToWrite.append(myString)

Changing this to 'linesToWrite = [myString] * ioMax' dropped time on my system from 2830ms to 1780ms (I'd like to note that I/O on my system was already much faster than his *best* I/O score, thank you very much Linux)

In the trig test, I used numarray to decrease the runtime from 47660.0ms to *6430.0ms*. The original timing matches his pretty closely, which means that numarray would probably beat his gcc timings handily, too. Any time you're working with a billion numbers in Python, it's a safe bet that you should probably use numarray!

I didn't immediately see how to translate his other mathematical tests into numarray, but I noted that his textual explanation in the article doesn't match the (python) source code!

(My system is a 2.4GHz Pentium IV running RedHat 9)