Java Faster Than C++?

jg21 writes "The Java platform has a stigma of being a poor performer, but these new performance benchmark tests suggest otherwise. CS major Keith Lea took time out from his studies at student at Rensselaer Polytechnic Institute in upstate New York's Tech Valley to take the benchmark code for C++ and Java from Doug Bagley's now outdated (Fall 2001) "Great Computer Language Shootout" and run the tests himself. His conclusions include 'no one should ever run the client JVM when given the choice,' and 'Java is significantly faster than optimized C++ in many cases.' Very enterprising performance benchmarking work. Lea is planning next on updating the benchmarks with VC++ compiler on Windows, with JDK 1.5 beta, and might also test with Intel C++ Compiler. This is all great - the more people who know about present-day Java performance, the better.""

Comments

Um, it's online

[JoshuaDFranklin]

If you want, you can read the actual author's piece instead of a news story about it:

The Java is Faster than C++ and C++ Sucks Unbiased Benchmark

Re:Um, it's online

[Anonymous Coward]

It also does stupid things. Like this:

int c = 0;
for (int i=n; i>0; i--) {
sprintf(buf, "%d", i);
if (X[strdup(buf)]) c++;
}

When this would have worked just fine:

int c = 0;
for (int i=n; i>0; i--) {
if (X[atoi(i)]) c++;
}

The code is dumb, yes, but you are wrong, nonetheless. That code won't even compile. I think you meant itoa(), which would be about the same as sprintf in terms of functionality.

That for() loop is not equivalent to the Java code's for loop, either. In the java code, he used

if (ht.containsKey(Integer.toString(i, 10))) c++;

which means that he should have used

if (X.count(somestringrepofi)) c++;

X[somestringrepofi] will create an entry for the key if it is not found, making it very different from containsKey().

Re:Um, it's online

[GooberToo]

The more I read the code, it sure is starting to look more and more like an apples and oranges comparison; which is usually what happens when people do java benchmarks.

As typically observed, I'm seeing the benchmarks take serious advantage of java's GC mechanism, whereby, they never pay the piper. With C++, the piper is constantly being paid. All of the benchmarks which allocate objects and then delete them, are therefore, invalid. To be fair, I think, you either need to add a System.gc() line in the java code where C++ is doing its deletes or you need to implement your own new and delete operators to function more in line with what java is doing. Until you do either one of those things, the comparisons where objects are being allocated and deallocated are invalid. And frankly, I'm still not sure adding System.gc() is even fair, on either side. The reason is, calling System.gc() simply hints that it's a good idea to collect. There is nothing which requires the collection to take place. So, technically, the call could still be many times faster. On the other hand, I don't know enough about how they handle their gc-hint logic nor am I aware of exactly how much overhead is involved in the actual collection process. If it occurs too often, the shift in workload may be too unfair. Nonetheless, it's a point of very serious contention.

Just for kicks, I modified objinst.java with, "if( i%(n/1000) == 0 ) System.gc() ;", on the lines that the C++ code had it's delete. When I timed it, it was over twice as slow as the C++ code (24+s vs 55+s). Worse, when I ran it with a 1:1 ratio of delete:System.gc(), I simply got tired of waiting, having waited over 5 minutes.

So, basically, I'm not nearly as impressed as I first was. Simplistically, it's starting to look like a serious apples and oranges comparison. Elsewhere, you can find other examples of just plain bad code. Where again, with correct C++ code, came in about twice as fast at the Java code, whereby, more optimizations were still possible with the C++ code.

So, it looks like we're seeing a combination of things here. Looks like a combination of bad code, ideal corner cases for java's hot spot, and invalid comparisons with memory allocation.

Sadly, I'm once again seriously disappointed in java.

Anyone got a match?

[nebaz]

Here's some kindling...

vi is better than emacs
bsd is better than linux
gnome is better than kde
.
.
.
anything else?

oh yeah...
my dad can beat up your dad.
And you smell funny.

Languages vs Compilers

[Anonymous Coward]

Java and C++ are language. Languages aren't "faster" or "slower", but compilers for them might be. I find it somewhat underhanded to put the languages in the header when it's really comparing compilers.

Not to mention, inter-language compiler benchmark[et]ing is notoriously difficult to get 'right'. The programs tested are often stupid (doesn't do anything meaningful), or constructed by a person with more skill/bias for one language than the other.

A few points...

[mindstrm]

There seem to be some unanswered questions here..

- How equivalent were the benchmarks? Where they programmed in an optimum way for their respective compilers and libraries? I'm sure the java ones were.. what about the C++ ones? The author states he doesn't understand G++ very well.

G++ is also known to not produce the best results.

"I rant it with -O2"

My guess is many of the tests were not implemented properly in c++.

The main clue would be this... I can understand java having better than expected performance.. but there is no way I can accept that java is that much FASTER than properly done C++... it doesn't make any sense.

Re:A few points...

[Cthefuture]

I've been playing with those benchmarks for ages. I use them any time a new language comes out or if I just want to do some independent testing.

A couple points:

- The "Great Shootout" benchmark times are sometimes way off because the run-time was too short to get an accurate reading. In those cases the tests should have been run with higher values to really stress the machine. That doesn't appear to be an issue in this test though (assuming his graph values are in seconds).

- Many of the C++ tests are not optimized. That is, they use C++ features like the iostream stuff (cout, and friends) which is extremely slow. The C versions are available and very fast. C++ is pretty much just an extension of C. You don't need to use C++ features if they slow you down. Another one is the hash stuff. In the C++ hash benchmark there are some goofy mistakes made by using the brackets [] operator where it forces several unnecessary lookups. You can also substitute a better STL hashing function that is faster (like MLton's insanely fast hasher).

- The test could be done by comparing C to Java. Anything in C++ can be made as fast as an equivalent C version but there are not many programmers that know how. Just assume anything in C++ will run as fast as a C version, and if it doesn't then you did something wrong. The hash tests would be easier in C++ though. If they were written properly they would kill the Java version.

With that said, I'm going to try these tests myself because I do not believe the results to be accurate. but who knows...

Could use a good analysis

[GillBates0]

The results are very non-intuitive. An extra layer between the program -> CPU implies an extra amount of overhead - be it any layer (VM at the Application layer, VM at the OS layer, or even at the CPU layer (hyperthreading)).

I looked at his results page quite extensively, but failed to find a good analysis/justification of the results. Just saying that the Server JVM is better than the Client JVM is *not* enough.

I want to know where the C++ overhead comes from, which Java manages to avoid - does the JVM do better optimization because it is given a better intermediate code (bytecode)? Is it better at doing back/front end optimizations (unlikely given gcc's maturity).

I tried to look for possible discrepancies in the results, but the analysis will definitely take more time - and I think it's the job of the experimenter to do a proper analysis of the results. Liked his choice of benchmarks though.

Re:Could use a good analysis

[jfengel]

His examples are all non-GUI things; they're pure CPU benchmarks. That's one major case where Java is certainly slower than C++.

Most of his tests are big loops (primes, string concatenation, etc.) These are cases where (as a sibling poster mentioned) hot path analysis can do you a world of good. A heavily tuned C++ program can do it just as well, or better, but the point of using a high-level language is that you don't have to do those optimizations yourself; you write the code in whatever way seems natural and you let the compiler optimize.

In a long-running Java program, you don't have that extra layer between the program and the CPU. The JIT does a real native compilation and passes control off to it. Once that's started, it runs just as fast as any other assembly code. Potentially faster, given that the JIT can look at the current run and optimize based on the way the code is going: the precise CPU it's running on, where things are in memory, how far it can afford to unroll a loop, what loop invariants it can lift, etc. It can even replace code as it runs.

The question then is, does the one-time (albeit run-time) optimization do more good than it costs?

That's especially easy on a hyperthreaded system. In a C++ program, these loops will run in a single thread on a single CPU, so if the JIT compiler runs on the other (virtual) CPU, you get its effort for free. Even the garbage collector can run on the other CPU, so you get the convenience of memory management with no total performance cost. (You do burn more CPU cycles, but you use up no extra wall-clock time.)

GCC is very mature, but it doesn't have the option of changing the code at run time. Especially on modern CPUs with their incredibly deep pipelines, arranging your code to avoid pipeline stalls will depend a lot on runtime considerations.

Also, Java has a few advantages over C++ in optimization. It's very easy to analyze Java programs to be certain that certain memory locations absolutely will not be modified. That's much harder in languages with native pointers. Those invariants allow you to compile out certain calculations that would have to be done at runtime in a C/C++ program. You can even start spreading loop cycles over multiple CPUs, but I'm pretty certain that the present JVMs aren't that smart.

These results are toy benchmarks, and not really indicative of real performance, even on purely non-GUI code. But I wanted to outline the reasons why the results aren't just silly, and they do have a theoretical basis.

Expert results

[otterpop81]

Some of the C++ tests would not compile. I've never been very good at decoding GCC's error messages, so if I couldn't fix a test with a trivial modification, I didn't include it in my benchmarks.

That's Great! I can't figure out GCC's error messages, but I offer definitive proof that Java is faster than C++. Nice.

I don't actually care hugely about performance

[Omnifarious]

I care that Java is an inconvenient pain to develop in and use. I care that I have to start a mini-OS just to run a Java program. I care that the language is under the control of one vendor. I care that the 'intialization == resource allocation' model doesn't work in Java. I care that the type system is too anemic to support some of the more powerful generic programming constructs. I care that I don't get a choice about garbage collection. I care that I don't get to fiddle bits in particular memory locations, even if I want to.

I think Java is highly overrated. I would prefer that a better C++ (a C-like memory model, powerful generic programming, inheritance, and polymorphism) that lacked C++'s current nightmare of strangely interacting features and syntax.

I use Python when I don't need C++s speed or low-level memory model, and I'm happier for it. It's more flexible than Java, much quicker to develop in, and faster for running small programs. Java doesn't play well with others, and it was seemingly designed not to.

Besides, I suspect that someone who knew and like C++ really well could tweak his benchmarks to make C++ come out faster again anyway. That's something I've noticed about several benchmarks that compare languages in various ways.

One example of why the tests are BS

[mypalmike]

From methcall.cpp:

int
main(int argc, char *argv[]) {
int n = ((argc == 2) ? atoi(argv[1]) : 1);

bool val = true;
>> Toggle *toggle = new Toggle(val);
for (int i=0; i<n; i++) {
val = toggle->activate().value();
}
cout << ((val) ? "true" : "false") << endl;
delete toggle;

val = true;
NthToggle *ntoggle = new NthToggle(val, 3);
for (int i=0; i<n; i++) {
val = ntoggle->activate().value();
}
cout << ((val) ? "true" : "false") << endl;
>> delete ntoggle;

return 0;
}

Why allocate and deallocate an object within the scope of a function? Well, in C++, there's no reason, so this is bad code. You can just declare it as a non-pointer and it lives in stack space. But guess what? You can't do that in Java: all objects are allocated on the heap.

That, and using cout instead of printf, are probably why this is slower than the "equivalent" Java.

-_-_-

Re:my arse

[kaffiene]

*sigh* have you people never heard of runtime optimisations? There are some things you can optimise at runtime (like runtime constants) which are *impossible* to optimise at compile time.

This whole "x is written in y, so x can't be faster than y" rubbish is just that - rubbish.

Re:Sorry, no.

[shadowmatter]

First, it's been known for awhile that Java is a poor performer when writing to the console, for whatever reason. Second, your Java timing probably include the time to startup the VM (not that this is wrong).

If you have a program that runs for awhile (so the startup time is small compared to the time the program takes to run), and does not do intensive output to the console, then Java is a reasonable choice in my opinion. Combined with SWT, Java applications can be quite snappy (see Eclipse, Azureus), and the end user will probably never know the difference.

- shadowmatter

Some performance myths

[vlad_petric]

First of all, g++ actually sucks big time in terms of performance. Intel C Compiler, with inter-procedural optimizations enabled, produces code that's almost always 20->30% faster than g++. I've actually once compiled C code with g++ and it was visibly slower than the same code compiled with gcc ... oh well.

Now, regarding java performance ... Java isn't slow per se, JVMs and some apis (most notably swing) are. Furthermore, JVMs usually have a slow startup, which gave java a bad name (for desktop apps startup matters a lot, for servers it's hardly a big deal). Java can be interpreted, but it doesn't have to be so (all "modern" JVMs compile to binary code on the fly)

Bytecode-based environments will, IMNSHO, eventually lead to faster execution than with pre-compilation. The reason is profiling and specialized code generation. With today's processors, profiling can lead sometimes to spectacular improvements - as much as 30% performance improvements on Itanium for instance. Although Itanium is arguably dead, other future architectures will likely rely on profiling as well. If you don't believe me, check the research in processor architecture and compiling.

The big issue with profiling is that the developper has to do it, and on a dataset that's not necessarily similar to the user's input data. Bytecode environments can do this on-the-fly, and with very accurate data.

been there, done that

[Anonymous Coward]

1) javac (Sun's Java compiler) is written in Java. You can even access it programmatically at runtime if you really want to.

2) While it's not an id game, IL2 Sturmovik is a critically-acclaimed fight simulator that was written almost entirely in Java.

java *can* be fast...

[alphafoo]

A year and a half ago I proposed building a standalone server-type application using Java, and my client scoffed at me because "everyone knows Java is slow". It was 1.4.2 on rh8.0 running on standard dual xeons. It ran pretty fast, and then I profiled it. Repeatedly. I replaced some of the stock library routines with my own faster ones or ones I found on sourceforge, found the most monitor-contentious areas and tuned them, played around with different GC strategies for the given hardware, and ended up with something that is amazingly fast. Scaled to 400+ HTTP requests per second and over a thousand busy threads, per node. Some of the speed bumps came for free, like when NPTL threads came available in the 2.4 kernel.

I am starting on a new standalone server now doing something different, but I am going to stick with Java, and will be happy to see what 1.5 does for me.

But I have seen Java run slow before, and I will tell you this: in every instance it is due to someone writing some needlessly complicated J2EE application with layer upon bloaty layer of indirection. All the wishing in the world won't make one of those behemoths run fast, but it's not fair to blame Java. Maybe blame Sun for EJB's and their best practices, or blame BEA for selling such a pig.

Stuff I like in the Java world:

• sun's 1.4.2 on hyperthreaded xeons
• Jetty (fast!)
• Piccolo XML parser (fast!)
• Lea's concurrency library
• Grosso's expirable cache [click]
• hibernate
• JAM on Maven [click]
• eclipse

Re:He used g++ to compare C++ with Java...

[ky11x]

g++'s goal is modularity for ease of porting in cross-platform cross-compiling. aggressive optimization is not one of its strengths. the point of such benchmarks is really not a language comparison, but a comparison between the code generated by the most optimized compilers for that language on a specific platform. Using g++ for this simply causes the study to lose credibility

Troll

[stratjakt]

This test proves that Sun's optimized Java compiler and VM are faster on Red Hat than gcc.

Gcc is designed for compatibility with a wide range of architectures, and is not optimized for a single one. He also (apparantly) used stock glibc from Red Hat. And only one "test", the method call test, showed java to be a real winner. And even then, it's server-side Java, which is meaning less when you talk about it as a day-to-day dev language (ie; creating standalone client-side apps).

Intel's (heavily optimized) C++ compiler should be a damn sight faster, and so should VC++.

This "comparison" is so limited in scope and meaning, that this writeup should be considered a troll.

Hell, read his lead-in:

"I was sick of hearing people say Java was slow, when I know it's pretty fast, so I took the benchmark code for C++ and Java from the now outdated Great Computer Language Shootout and ran the tests myself."

Ie; I set out to prove Java is teh awesome and c++ is teh suck!

If anything it proves something I've known intuitively for a long time. gcc does not produce x86 code that's as fast as it could be. That's a trade-off for it being able to compile for every friggin cpu under the sun.

I can't wait till RMS takes personal offense and goes on the attack.

One more...

[Rufus88]

anything else?

Yeah, Kuro5hin is better than Slashdot.

Re:He used g++ to compare C++ with Java...

[mwillis]

Intel gives their c++ compiler away free for non-commercial hobbyist use on linux.

The windows version has a free trial that runs for 30 days.

Try it. See if it makes a difference. If it doesn't, torch it. If you find it makes your critical code run 2x faster, then... have a look at what a computer that runs 2x faster will cost you, and then decide what to do.

Java performance "truths" change over time

[eduardodude]

Check out this recent IBM Developerworks article which looks at how modern JVMs handle allocation and garbage collection.

Some very surprising tidbits there. For instance:

"Performance advice often has a short shelf life; while it was once true that allocation was expensive, it is now no longer the case. In fact, it is downright cheap, and with a few very compute-intensive exceptions, performance considerations are generally no longer a good reason to avoid allocation. Sun estimates allocation costs at approximately ten machine instructions. That's pretty much free -- certainly no reason to complicate the structure of your program or incur additional maintenance risks for the sake of eliminating a few object creations."

Read the article for an excellent nuts-and-bolts analysis of many current performance considerations. From the posts in this thread, it's pretty clear a lot of people haven't looked into what's actually done in a server JVM these days.