Slashdot Mirror
Linux Number Crunching: Languages and Tools
ChaoticCoyote writes " You've covered some of my past forays into benchmarking, so I thought Slashdot might be interested in Linux Number Crunching: Benchmarking Compilers and Languages for ia32. I wrote the article while trying to decide between competing technologies. No one benchmark (or set of benchmarks) provides an absolute answer -- but information helps make reasonable decisions. Among the topics covered: C++, Java, Fortran 95, gcc, gcj, Intel compilers, SMP, double-precision math, and hyperthreading."
...what they are doing.
It's pretty clear this guy is not a Java programmer, or at least has limited knowledge of how to write efficient, well-designed programs in Java. Amazingly, that's actually not relevant. To get to the bottom line skip to the bottom of this post.
For starters, he's using 2-dimensional arrays, so of course Java is going to be slower. 2-dimensional arrays in C++ are actually 1-dimensional arrays with some syntactic sugar thrown in. In Java they are arrays of arrays (really akin to using vectors of pointers to vectors). Experienced Java programmers (and for that matter good C++ programmers) will use a Matrix class to both clean up the code structure and to provide a much more optimised 1-dimensional array structure for manipulations of the Matrix.
His benchmark doesn't have a warm-up run, and doesn't run nearly long enough to cause HotSpot to optimise the code thoroughly. Sure, this is fair if what you're measuring is how well the Java VM can execute a relatively short-run program (quick answer: not very well at all), but it is no indication of the kind of performance that Java can achieve.
The code essentially is not written like a standard Java program (and indeed, is a pretty evil C program if you ask me). I mean, any method with 30+ local variables is suspect in my book (particularly amusing is that some of them appear to be never used). As such, it doesn't play very well against the JVM's optimiser, which is oriented towards Java code with much different structure.
His TEST_LENGTH loop is long enough that he's potentially better off catching an array out of bounds exception than constantly checking the index.
This doesn't affect Java performance, but it's just silly to have a while(true) loop and then have a break on an if at the end of a block is just silly (it might trigger some optimization for a specific, broken C optimizer, but it's just silly to have that on a standard benchmark).
But ultimately this all doesn't matter. Why? Because the bottom line is this benchmark makes heavy uses of standard runtime functions (the trig and square root function). This means that large portions of this code are essentially out of the realm of the compiler, and in the hands of the runtime library. Doing stuff like this I can get widely different results just be linking in different floating point libraries. So, while he highlights the *compilers*, in fact what this is really stressing is the quality of the floating point libraries of the particular runtimes he's chosen. As such, it provides no insight in to the relative merits of using different compilers or languages. At worst, this might be misleading even for the specific application he has in mind. At best it suggests how a few different math libraries perform. The conclusions this guy draws from this are just rediculous.
sigs are a waste of space
1. Fortran is no faster than C++, given decent compilers (Intel C++).
2. The g++ compiler is a pile of crap - it gave execution times more than double the times given by Intel C++. IMHO the difference is more than can reasonably be brushed off by saying that Intel know their own CPU better, or g++ has more features, or whatever the g++ fans will say. A difference as big as this means that - as an optimising compiler - g++ is broken.