Java Faster Than C++?

jg21 writes "The Java platform has a stigma of being a poor performer, but these new performance benchmark tests suggest otherwise. CS major Keith Lea took time out from his studies at student at Rensselaer Polytechnic Institute in upstate New York's Tech Valley to take the benchmark code for C++ and Java from Doug Bagley's now outdated (Fall 2001) "Great Computer Language Shootout" and run the tests himself. His conclusions include 'no one should ever run the client JVM when given the choice,' and 'Java is significantly faster than optimized C++ in many cases.' Very enterprising performance benchmarking work. Lea is planning next on updating the benchmarks with VC++ compiler on Windows, with JDK 1.5 beta, and might also test with Intel C++ Compiler. This is all great - the more people who know about present-day Java performance, the better.""

Comments

Could use a good analysis

[GillBates0]

The results are very non-intuitive. An extra layer between the program -> CPU implies an extra amount of overhead - be it any layer (VM at the Application layer, VM at the OS layer, or even at the CPU layer (hyperthreading)).

I looked at his results page quite extensively, but failed to find a good analysis/justification of the results. Just saying that the Server JVM is better than the Client JVM is *not* enough.

I want to know where the C++ overhead comes from, which Java manages to avoid - does the JVM do better optimization because it is given a better intermediate code (bytecode)? Is it better at doing back/front end optimizations (unlikely given gcc's maturity).

I tried to look for possible discrepancies in the results, but the analysis will definitely take more time - and I think it's the job of the experimenter to do a proper analysis of the results. Liked his choice of benchmarks though.

Re:Could use a good analysis

[jfengel]

His examples are all non-GUI things; they're pure CPU benchmarks. That's one major case where Java is certainly slower than C++.

Most of his tests are big loops (primes, string concatenation, etc.) These are cases where (as a sibling poster mentioned) hot path analysis can do you a world of good. A heavily tuned C++ program can do it just as well, or better, but the point of using a high-level language is that you don't have to do those optimizations yourself; you write the code in whatever way seems natural and you let the compiler optimize.

In a long-running Java program, you don't have that extra layer between the program and the CPU. The JIT does a real native compilation and passes control off to it. Once that's started, it runs just as fast as any other assembly code. Potentially faster, given that the JIT can look at the current run and optimize based on the way the code is going: the precise CPU it's running on, where things are in memory, how far it can afford to unroll a loop, what loop invariants it can lift, etc. It can even replace code as it runs.

The question then is, does the one-time (albeit run-time) optimization do more good than it costs?

That's especially easy on a hyperthreaded system. In a C++ program, these loops will run in a single thread on a single CPU, so if the JIT compiler runs on the other (virtual) CPU, you get its effort for free. Even the garbage collector can run on the other CPU, so you get the convenience of memory management with no total performance cost. (You do burn more CPU cycles, but you use up no extra wall-clock time.)

GCC is very mature, but it doesn't have the option of changing the code at run time. Especially on modern CPUs with their incredibly deep pipelines, arranging your code to avoid pipeline stalls will depend a lot on runtime considerations.

Also, Java has a few advantages over C++ in optimization. It's very easy to analyze Java programs to be certain that certain memory locations absolutely will not be modified. That's much harder in languages with native pointers. Those invariants allow you to compile out certain calculations that would have to be done at runtime in a C/C++ program. You can even start spreading loop cycles over multiple CPUs, but I'm pretty certain that the present JVMs aren't that smart.

These results are toy benchmarks, and not really indicative of real performance, even on purely non-GUI code. But I wanted to outline the reasons why the results aren't just silly, and they do have a theoretical basis.

I don't actually care hugely about performance

[Omnifarious]

I care that Java is an inconvenient pain to develop in and use. I care that I have to start a mini-OS just to run a Java program. I care that the language is under the control of one vendor. I care that the 'intialization == resource allocation' model doesn't work in Java. I care that the type system is too anemic to support some of the more powerful generic programming constructs. I care that I don't get a choice about garbage collection. I care that I don't get to fiddle bits in particular memory locations, even if I want to.

I think Java is highly overrated. I would prefer that a better C++ (a C-like memory model, powerful generic programming, inheritance, and polymorphism) that lacked C++'s current nightmare of strangely interacting features and syntax.

I use Python when I don't need C++s speed or low-level memory model, and I'm happier for it. It's more flexible than Java, much quicker to develop in, and faster for running small programs. Java doesn't play well with others, and it was seemingly designed not to.

Besides, I suspect that someone who knew and like C++ really well could tweak his benchmarks to make C++ come out faster again anyway. That's something I've noticed about several benchmarks that compare languages in various ways.

Re:Sorry, no.

[shadowmatter]

First, it's been known for awhile that Java is a poor performer when writing to the console, for whatever reason. Second, your Java timing probably include the time to startup the VM (not that this is wrong).

If you have a program that runs for awhile (so the startup time is small compared to the time the program takes to run), and does not do intensive output to the console, then Java is a reasonable choice in my opinion. Combined with SWT, Java applications can be quite snappy (see Eclipse, Azureus), and the end user will probably never know the difference.

- shadowmatter

Some performance myths

[vlad_petric]

First of all, g++ actually sucks big time in terms of performance. Intel C Compiler, with inter-procedural optimizations enabled, produces code that's almost always 20->30% faster than g++. I've actually once compiled C code with g++ and it was visibly slower than the same code compiled with gcc ... oh well.

Now, regarding java performance ... Java isn't slow per se, JVMs and some apis (most notably swing) are. Furthermore, JVMs usually have a slow startup, which gave java a bad name (for desktop apps startup matters a lot, for servers it's hardly a big deal). Java can be interpreted, but it doesn't have to be so (all "modern" JVMs compile to binary code on the fly)

Bytecode-based environments will, IMNSHO, eventually lead to faster execution than with pre-compilation. The reason is profiling and specialized code generation. With today's processors, profiling can lead sometimes to spectacular improvements - as much as 30% performance improvements on Itanium for instance. Although Itanium is arguably dead, other future architectures will likely rely on profiling as well. If you don't believe me, check the research in processor architecture and compiling.

The big issue with profiling is that the developper has to do it, and on a dataset that's not necessarily similar to the user's input data. Bytecode environments can do this on-the-fly, and with very accurate data.

What about gcj?

[joshv]

I'd be interested in comparing the speed of the native code generated by gjc to the that of JVM.

-josh

java *can* be fast...

[alphafoo]

A year and a half ago I proposed building a standalone server-type application using Java, and my client scoffed at me because "everyone knows Java is slow". It was 1.4.2 on rh8.0 running on standard dual xeons. It ran pretty fast, and then I profiled it. Repeatedly. I replaced some of the stock library routines with my own faster ones or ones I found on sourceforge, found the most monitor-contentious areas and tuned them, played around with different GC strategies for the given hardware, and ended up with something that is amazingly fast. Scaled to 400+ HTTP requests per second and over a thousand busy threads, per node. Some of the speed bumps came for free, like when NPTL threads came available in the 2.4 kernel.

I am starting on a new standalone server now doing something different, but I am going to stick with Java, and will be happy to see what 1.5 does for me.

But I have seen Java run slow before, and I will tell you this: in every instance it is due to someone writing some needlessly complicated J2EE application with layer upon bloaty layer of indirection. All the wishing in the world won't make one of those behemoths run fast, but it's not fair to blame Java. Maybe blame Sun for EJB's and their best practices, or blame BEA for selling such a pig.

Stuff I like in the Java world:

• sun's 1.4.2 on hyperthreaded xeons
• Jetty (fast!)
• Piccolo XML parser (fast!)
• Lea's concurrency library
• Grosso's expirable cache [click]
• hibernate
• JAM on Maven [click]
• eclipse

Can I have an infinite budget to write the code?

[IvyMike]

Here is my experience with C++ vs. Java: At my company, we had a specialized image viewing program. The original program was written in C++ years ago, and performance sucked even on modern machines. It probably had a dozen man-years of time in it. We decided to re-write it in java.

We knew java in theory should be worse than C++ at manipulating large blocks of raw data, so we spent some time architecting, prototyping, and profiling java. We quickly learned the limitations and strengths.

The result? After 4 engineers worked for 6 months, we had a program that was rock solid, had more features, had a modern UI, and was WAY faster. Night and day; the old program felt like a hog, and the new program was zippy as anything. And the new code is fewer lines, and (in our opinion) way more maintainable. Since the original release, we've added severeal new features after day or two of work; the same features never would have happened on the old version, because they would have been too risky.

So the question is this? Could we have re-written or refactored the C++ program and gotten the same speed benefits? No doubt, such a thing is possible. But we are all convinced there is NO WAY we could have done it with as little effort. The C++ version would have taken longer to write and debug.

Now you're talking Profiling

[GillBates0]

and that's a topic that gets me all worked up (my Master's Thesis touch on Program Profiling).

So, if the JIT computes Hot/Cold Paths, and optimizes the Hot paths, then it should work better and better on successive runs (as more and more profiling information is gathered). On the other hand, there will be cases where it performs worse, as profiles are gathered for specific inputs.

That means that if an average of say 5 runs (on the same input) is taken, it will have an unfair advantage (since gcc did NOT have the advantage of profiling information (see man gprof or similar)). Using Profiling as an optimization tool is *always* unfair unless both tools are provided with the advantage of the same profiling information. This is a valid question for the author then: if the JIT/javac/JVM uses profiling information, gcc should too, for fair comparison.

PS: I have seen this argument being made by my Professor and audiences at compiler conferences.

Re:A few points...

[Cthefuture]

I've been playing with those benchmarks for ages. I use them any time a new language comes out or if I just want to do some independent testing.

A couple points:

- The "Great Shootout" benchmark times are sometimes way off because the run-time was too short to get an accurate reading. In those cases the tests should have been run with higher values to really stress the machine. That doesn't appear to be an issue in this test though (assuming his graph values are in seconds).

- Many of the C++ tests are not optimized. That is, they use C++ features like the iostream stuff (cout, and friends) which is extremely slow. The C versions are available and very fast. C++ is pretty much just an extension of C. You don't need to use C++ features if they slow you down. Another one is the hash stuff. In the C++ hash benchmark there are some goofy mistakes made by using the brackets [] operator where it forces several unnecessary lookups. You can also substitute a better STL hashing function that is faster (like MLton's insanely fast hasher).

- The test could be done by comparing C to Java. Anything in C++ can be made as fast as an equivalent C version but there are not many programmers that know how. Just assume anything in C++ will run as fast as a C version, and if it doesn't then you did something wrong. The hash tests would be easier in C++ though. If they were written properly they would kill the Java version.

With that said, I'm going to try these tests myself because I do not believe the results to be accurate. but who knows...

Re:Expert results

[Xugumad]

Also, a quote from the article:

"I was sick of hearing people say Java was slow, when I know it's pretty fast"

Nice, unbiased viewpoint there...

Explanation

[gillbates]

Reviewing the console log, we find that when java programs were tested with a large number of iterations, Java only performed better on one test.

• We don't know which OS was used. While each C++ program must have been loaded entirely each time, the JVM may very well have remained cached in RAM between tests - hence a faster startup time, which explains:
• Java is actually slower than C++, but because the JVM was already cached in RAM, it ran faster on those tests which involved a relatively small number of iterations. However, when the number of iterations was increased, Java was always slower than C++, with the exception method call and object instantiation:
• Object instantiation isn't really relevant because of the fact that C++ programs call the OS for every single memory request, where as Java can pool it. This test measured the speed of the kernel's malloc more than the speed of C++.
• In most of the C++ code, IO is placed in the inside loops, meaning that the program is really testing the throughput of libc and the OS, as opposed to the efficiency of the generated code.
• An interesting note: the Java client won none of the benchmarks.

I know that Java has many strengths, but speed isn't one of them. Looking at the results, we see the g++ runtimes are much more consistent than those of Java - on some tests, the Java Server is faster than the client by a factor of 20!? How could a programmer code without having any realistic expectation of the speed of his code. How embarrassed would you be to find that your "blazingly fast" app ran slower than molasses on the client machine, for reasons yet unknown?

When it comes to speed, compiled languages will always run faster than interpreted ones, especially in real-world applications.

But discussions of language speed are a moot point. What this really tested was the implementation, not the language. Speed is never a criteria upon which languages are judged - a "slow" language can always be brought up to speed with compiler optimizations (with a few exceptions). I suspect that if C++ was interpreted, and Java compiled, we'd see exactly the opposite results.

In short, the value of a language consists not in how fast it runs, but in what it enables the programmer to do.

Function calls

[BenjyD]

Why does the example use a recursive fibonnaci sequence algorithm? It's so slow, and the runtime is dominated by the function call time.

For example:

[bdr@arthurdent tests]$ time ./fib_recurse 40
165580141
real 0m3.709s
user 0m3.608s
sys 0m0.005s

time ./fib_for_loop 40
165580141
real 0m0.006s
user 0m0.002s
sys 0m0.002s

I think a lot of these benchmarks are showing that the Hotspot optimiser is very good at avoiding function call overheads.

Same for hash.cpp

[Tim]

I'm futzing around with the other hash benchmark, and sure enough, making only a trivial change to the code (eliminating the unnecessary strdup in the second hash lookup), gets me about a 30% improvement in performnace.

This guy is a tool.

Re:Um, it's online

[ipfwadm]

I've read many times that it actually does ONLY result in a hint to collect. Unless you can prove otherwise, I'm apt to lean in that direction. Which, actually means that the 1:1 implementation is a more realisitic apples to apples comparison. Can I prove that? No. I'm still hoping a java guru will come in with some insightful tidbits. ;)

Write a java app that does nothing but repeatedly call System.gc(). Run it with the -verbose:gc option, and watch the garbage collector go. Mind you, this is not 100% proof, but the fact that it prints out [Full GC] over and over again makes me lean pretty strongly in the direction of "the garbage collector actually runs in response to a System.gc(), it's not just a hint".

One thing I would like Java to do is to allow me to delete objects manually. There are times when the garbage collector really sucks, but 95% of the time it's sufficient, in my experience. And yes, this experience comes from real-world apps.

Regarding the original topic, I would bet that there are cases where Java really could give C++ a run for its money. However, one liability that Java has compared to C at least is that making everything an object adds a whole lot of object overhead. I had to write a file search routine as part of a Java app, and originally wrote it strictly in Java. The sheer number of File objects that get created by such a routine is ridiculous, and there's really no way to reduce the overhead by reusing the objects -- File objects are immutable. Calling out to JNI resulted in a 3 to 5x performance boost. Does this one example prove anything? No, but it's a heck of a lot more real-world than simply appending a string to itself a few dozen times...