Linux Number Crunching: Languages and Tools
ChaoticCoyote writes " You've covered some of my past forays into benchmarking, so I thought Slashdot might be interested in Linux Number Crunching: Benchmarking Compilers and Languages for ia32. I wrote the article while trying to decide between competing technologies. No one benchmark (or set of benchmarks) provides an absolute answer -- but information helps make reasonable decisions. Among the topics covered: C++, Java, Fortran 95, gcc, gcj, Intel compilers, SMP, double-precision math, and hyperthreading."
Interesting numbers. Have you considered benchmarking Octave or rlab also? (Or is there a native MATLAB for Linus now?)
There is a quick and dirty intro to K over at Kuro5hin.
Some more links for more inforation:
Kernigan's benchmark test
more examples
Kx: the people who make K and KDB
Ah ha! Someone who understands what benchmarks are for and how to use them - it sometimes seems like the corporate world uses numbers from benchmarks only when they prove their claims. Of course, that's the difference between open source and the business world - open source (ideally) looks at every benchmark result and asks "now how can we get all of these numbers better than the competition?" while more traditional businesses ask, "Which of these numbers make our product look the best?". *shrug* its just nice to see benchmarks used properly, is all.
I always find Intel C++ shines in all benchmarks. I wondered if anyone has ever tried to compile linux out of it? I know it might hurt your ideology but just for the fun of it. :)
I'm hardly a Java junky, but I've spent a lot of time recently with the language and I've heard a lot of complaints from my peers about Java being slow. Most of the time, just like this author, they're wrong! Java isn't slow, but sometimes you do have to program more thoughtfully to make Java fast.
First things first, though. No one would ever claim that JDK 1.4 is the ultimate Java speed demon. Even the "HotSpot" in server mode is going to be slow if your code isn't written well. But the author fails to do any profiling, and fails to give anyone even a hint as to why Java doesn't perform well. But I shouldn't get on him about his coding, or lack of profiling... neither issue is the reason his test showed Java to be slow.
The real problem: Firs, I'll cut him some slack for not profiling. However, I won't cut him any slack for using an interpreter instead of a JIT compiler. Java's been shown time and time again to be as fast as FORTRAN/C++ when using a good compiler, rather than an interpreter. *sigh* When will the madness end? A 0.07 second query to Google should explain that one to even a novice. Java IS fast. Interpreted byte-code is slow. Java != interpreted byte code; Java is a language.
Anyway, here's a link to a weak, biased, and not so rigorous argument backing up that statement. But, it's an easy read for Java newbies, so I'll risk posting it anyway: Java is Fast for Dummies(tm)
Can anyone suggest a good compiler for floting point number crunching on Athlon based systems? (For Linux, and preferably free or not too expensive)
He, who dies with the most toys, wins
Yes, but indeed if you're really looking to benchmark only, comparing a row-based database engine with a column-based one is like comparing an apple to an orange. Both are fruits, both give you calories, but they're quite different.
Now as we're going off-topic from the original submission, one could benchmark KDB with Sybase IQ Multiplex. Here you're talking about 2 column-based db engines. In my testing, KDB is indeed up to an order of magnitude (10x) faster than Sybase IQ which is itself 2 orders of magnitude (100x) faster than row-based database engines.
However, as the article in the post says, benchmarks don't give the whole story.
Apart from the usual learning curve issues and available management tools (which KDB sadly lacks compared to Sybase IQ), there is one fundamental difference between the 2 db engines (and Oracle, DB2, Sybase ASE, etc...):
KDB is single-process, and does not pool memory. I'm not saying this is bad, but it makes for very interesting architectural issues when designing a system. For example, if you're going to use KDB, you're better off with the fastest possible single-CPU system. The best platform for KDB is probably the fastest Intel P4 Xeon, dual-processor, and as much RAM as possible on the machine. One processor will be used exclusively for KDB, the other for the OS. To grow, you'll implement a farm of those.
On the other hand, the other major DB engines generally perform much better in multi-CPU systems such as 16-way Sun servers. They pool the memory and use all the CPUs you'll give them. This makes for a more expensive single system, but an easier implementation if your application is larger than what a single dual Intel box can provide. In such a case, KDB will need one write engine and multiple read engines, significant storage pooling issues, etc...
Anyway, one last point regarding column-based database engines: they are certainly amazing for reporting and most read commands. Where they lose to row-based engines is in inserts, and in selects that return data from a large number of columns.
In the former case, you trick KDB and Sybase IQ into performing batch inserts (where the loading of columns will only be "wasted" once per batch). In the latter case, you're going to be hurt with KDB and Sybase IQ whatever you do, as they'll have to load in memory all the columns out of which you need the data.
Bottom line:
If you need OLTP (lots of inserts/updates) and aren't worried about extreme speeds, go for Oracle, Sybase ASE, DB2, etc...
If you need fast reporting with very quick time to market, go for Sybase IQ Multiplex.
If you need the absolute ultimate in reporting speed and have the time and resource to apply to it, go for KDB.
time python -O almabench.py user: 22m19.354s
gcc -ffast-math -O3 almabench.cpp -lm time ./a.out
user: 0m50.348s
C++ is only 27 times faster than Python for planetary simulations.
Almabench.py is my own conversion from the cpp source. I will send it to the author for possible addition to the benchmark.
-- Imperial units must die --
Just to put some things into perspective, here are some numerical results. These were obtained on my dual-athlon 1.4GHz w/ 1GB of RAM. The task was to compute the TE and TM surface currents induced on a circular cylinder 10 wavelengths in circumferece and having a relative permittivity equal to 4-j2. The program simultaneously solves the perfect electric conducting case. The surface was discretized into 60 cells using 120 unknowns (way overkill, but just to prove the point) using the Integral Equation Asymptotic Phase method.
g77 Compiler (-O2 -malign-double -funroll-loops): 24.11s
Lahey Compiler (equivalent paramters): 16.45s
As you can see, there's really no comparison, except that the lahey-created binary uses about 10% more RAM than does the one created with g77. This is just a summary comparison as I did not go into measuring the difference in the error of the two results compared to a reference solution. I'm assuming that both solutions are about the same with regard to accuracy.
There is no question that you can have a memory leak in a Java program. However, Java does remove a number of potential ways to create a memory leak that do occur in C++ (heck, 90% of C++ programs I see will leak heap memory if you have exceptions). Anyone who claims that "While it is possible to carefully write a Java program that doesn't leak, I don't think it's any easier than making a leakproof C++ program" either hasn't written a lot of C++, or hasn't written a lot of Java. ;-)
sigs are a waste of space
With numerical code there will be little difference regardless of which version of Linux your use or if you use Windows for that matter
Anyone who's spent any time working with Sun's Wireless Toolkit to develop for mobile devices will have witnessed pretty serious memory leakage firsthand. I know I do! After starting an emulator 10-20 times to test code it's using so much RAM that it's necessary to kill and restart the Toolkit to get anything like reasonable performance back again!
;-)
I'll agree that Java makes memory management much simpler, (I've spent a lot of time hacking x86 assembler, Pascal, C and C++ over the years) but bad programming can lead to leaks just as well. You tend to discover leaks pretty quickly with a mobile phone that has only 200K of RAM to play with though
Code, Hardware, stuff like that.
Using object orientation in C++ adds a very small overhead (in the vicinity of 10% if you're using virtual functions). Now I understand that the article was mostly concerned with benchmarking the languages, and I applaud the writer for specifiying that benchmarking is no "silver bullet". But I really want to stress:
Somebody has to develop and maintain the freaking program too!
Object orientation helps anything but the most trivial of applications to acvhieve better modularity and reuseability. Anyway your program is going to spend most of its time in development, not running, so anything that can help that process along is going to be a big help to your project. Please check out the benefits of the SW industries "best practices" and apply them to your project.
You will save days and months of development time, during which you can run your finished program to your heart's content.
A)bort, R)etry or S)elf-destruct?
Using both Sun's and IBM's Java virtual machines I get the following results from 'time'.
real 0m35.173suser 0m35.140s
sys 0m0.030s
The results vary from 35.130 to 35.160. When I run the c++ test with the following compiler options : -march=i686 -mcpu=i686 -O (somehow the mmx and sse option are rejected)
real 0m43.467suser 0m41.790s
sys 0m0.010s
Can somebody explain this please ?
Yes, indeed, what about fp (functional programming) and numerics? :)
Funny thing is that the fp people have invested lots of brainpower into advanced functional programming techniques. Symbolic and logics math, yes I have seen software for it. But numerics?
Is it possible that a functional language beats FORTRAN eg in eigenvector calculations?
Regards,
Marc
In case any fellow beginning C++ programmer was wondering: This is because objects are only guaranteed to be destructed when an exception is thrown if and only if this exception is caught.
So no matter what you're doing in your program, its main() should always look a bit like this to make sure every exception is caught:
(Sorry, I can't seem to save indentation with /. HTML here)
Please note that this way, you also always assure to have a valid shell return code (EXIT_SUCCESS from your function() would be the OK code).
42. Easy. What is 32 + 8 + 2?
Here's some the "better" parts of Java:
Don't get me wrong. Java's fine for certain applications. For lightweight networking stuff, I think it's almost unparalleled. It's also pretty good for prototyping C++ stuff. It's good for lightweight tasks that break down logically into threads -- Java has nice threading support.
My beef is that Java is not, despite its supporters' loud claims (which have been going on for years), remotely performance-competitive with C.
The language simply has some foundational performance limitations in it. It was designed that way, and tweaking implementations cannot get around that.
I agree that there are some nice things about Java.
Rapid Development
Damn straight. Java is a great prototyping language.
Hotspot
Not bad, but not that incredible, either. The benchmarks I've seen haven't shown HotSpot to be incredible, and besides, competitors like C (gcc) have branch-profiling code of their own.
Secure Software
True. There are some improvements. But buffer overflows are less and less common in C (due to *excellent* libraries like glib), and have been fixed in other languages without anywhere near the performance hit of Java (like Ocaml).
One of the big factors remaining is just HOW you write code in Java.
It may be a personal thing, but I have a deep dislike of languages where you have to modify your regular coding style to get decent performance at a given point. It used to be BASICs...you'd use some nasty trick and you could actually get decent performance out of the thing. Then MATLAB. *God* I hate vectorizing operations. I expect that a MATLAB guru simply does this in his sleep, but I find it incredibly frusterating to totally rethink code in an any areas where performance matters.
These things slow Java down, btu also make it more uniform which makes it easier (faster) to
A fair number of the uniformity improvements in Java could have come from simply tweaking syntax (int[50] x instead of int x[50], for example).
I'm all for modern language features...I just think that doing anything that implies a necessary performance hit is a bad idea. If someone wants a given feature, they can slap it on top. I can make C++ have a virtual function, but I can't make Java run quickly.
If you made every function in your C++ classes virtual, used RTTI and Strings to do runtime linking, etc. your C++ programs would be slower too!
Ya, but Stroustroup went to a lot of work to ensure that you only "pay for what you use".
So, I'm not out to bash Java as a usable language. It has some major pluses. However, specifically in the performance arena, Java definitely has issues.
May we never see th
Yes, look on Google for the language SISAL. That apparently beats Fortran.
Sisal was developed at Los Alamos IIRC and those dudes have some of the fastest supercomputers in the world so they should know.
The thing with FL's is not to use a lazy language, but a stract one (SISAL being a good example) should be able to be optimized quite heavily.
The dangers of excessive individualism are nothing compared to the oppressiveness of excessive collectivism
"it has always been clear that Java is inferior to native code applications in terms of raw power. Computational performance depends on the effective use of a target processor's instruction set; such optimization can not be expected of Java given its goals of universality and portability."
This statement is simply not true! The portability nature of Java does not conflict with a virtual machine that does processor-specific optimizations. Take a look at Sun's HotSpot VM source code (it's publicly available!) In the IA32-specific code, you'll see lots of run-time switches to enable specific P4 optimizations, for example.
"Perhaps Java's Just-in-Time compiler could be enhanced to perform processor-specific run-time optimizations; on the other hand, doing so would require different JVMs (Java Virtual Machines) for different architectures, or a single bloated JVM with run-time architecture detection."
This already exists in the current HotSpot VM. There's an IA32 binary, which includes optimizations for several versions of IA32. It does not include PowerPC or SPARC code, as that's in a different binary.
" The "ia32" world is already fragmented between Pentium III, Pentium IV, Itanium, Athlon, and Opteron architectures, each having unique requirements for optimization;"
That's the challenges a VM writer has to deal with. And the HotSpot team did a great job in managing this complexity.
In the future, if you (or anyone else, for the matter) takes the time to write a paper, you should do more research. Some of the statements above are simply misleading.
Oh, yes, it does. If you want GNU C/C++ to inline trig functions and/or use the machine instructions, you can get it to do that, too. But there are good reasons why it doesn't do that by default. There are other options you can give GNU C/C++ to tell it to make assumptions that let it optimize more. It's really for you to decide what tradeoffs you want.
The point of using a high-level language is to avoid the need to play in assembly-land.
For day-to-day programming, I agree. But if you do benchmarks, you have to understand assembly language and look at it. There are a lot of other low-level details you need to look at and understand as well (e.g., caching). That's not so that you can tune your benchmarks for it, it's so that you can determine whether your benchmark is actually doing what you think it is doing.
The trick is picking the right level of hardware abstraction -- do you write for an assembler
Unfortunately, your code doesn't test the efficiency of abstraction at all--your code is something that could have been written in Fortran 77. Once you actually start moving to higher levels of abstraction, Fortran 95's capabilities are limited, and Java's performance becomes abysmal. Pretty much only C++ has all the necessary hooks to write efficient high-level numerical code at this point.
Note, incidentally, that even if your benchmarks are representative, the Pentium4 is probably still not the best solution in terms of price/performance. That's another important factor to be taken into account when benchmarking: how much does that performance actually cost.
(Incidentally, I hope JPL isn't using this kind of code for actual navigation.)
I know it costs a little money, but i would be very intrested to see the PGI compiler set tested up there as well.. I've seen on alot of CFD code have 10x speedups over g77 with pgf77. I do know that PGI has a 15 day trial license for their compiler, that should be long enough for a test run of the almabench to run.
But with 90% of the code these days in business (which is the majority of custom code written in the world), Java's speed is acceptable and its portability and memory garbage collection outweigh any speed advantages of C/C++ in a Gigahertz world.
Sure. Java's fine for doing a front end or simple database accessor. Hell, Hypercard was great for this.
It's just not so great to do mainstream apps in, and people claiming that Java is "just as fast as C", which sometimes gets these people to waste time trying to implement their app in Java, gets my gander.
Look at Corel's suite. They implemented the *entire stupid thing* in Java because it was going to be the "next big thing" and eventually be as "fast as C", according to the frenzied shouts of some of the Java supporters. Then they had to throw the whole thing out because of performance issues. I can't even imagine the cost, both direct and strategic (cost in time sitting on your ass while your competitor does something) to Corel. Same thing happened to Mozilla...the thing was originally going to be in Java. That idea got nixed...
The other thing that vaguely pisses me off is that almost all of the things that Java does that make performance *suck* really aren't necessary. You can design a fast language that has most of its security model. Ocaml is quite safe -- moreso than Java -- but the Ocaml people imposed almost no overhead from C, because they avoided adding features that required runtime overhead, but went all out on things that could be somehow finagled into compile-time work. They *do* do most array bounds checking, but that's about it. Ocaml has GC, portability, and type safety...it just does almost all its work at compile time. I'd be willing to use Ocaml (well, actually I don't like functional languages much, but I'd endorse its performance) for almost any of the situations where I'd use C or C++.
I'm not a huge fan of Ocaml itself, but it's a model for what *should* have been done with Java. If a feature is going to nail you performance-wise -- if users are going to have to do contortions to get decent performance -- you should think long and hard before adding it. Everything that sounded "cool" ended up in Java, which resulted in countless wasted CPU cycles for users around the world.
I don't see why, as processors get faster, people feel the need to keep computers equivalently slow. Why not take advantage of the constant-time improvement? We can do better optimizations, do caching, because now we have the memory and extra constant time to do so. We have better body of knowledge about compilers, so why can't we *improve* performance instead of hurting it? It's *stupid*.
May we never see th