Java VM & .NET Performance Comparisons

johnhennessy writes "Just came across some good virtual machine performance benchmarks (on Mono's mailing list). It covers executing java bytecode via a host of different Java runtimes and also the mono runtime. Not only does it give numbers for IBM's runtime (1.4.2 and 1.3.1), Sun's runtime (1.5.0 and 1.4.2), GCJ, Mono, Jikes and much more! These numbers are also given for both Intel and Opteron (where relevant). Before the flames begin, don't forget to read the authors description of how the benchmark was carried out. Hopefuly this should inspire educated discussion on ones favourite JVM/CLR."

Comments

Re:GCJ slower than a native JVM?

[murphee]

Sigh... for the last time: Java code executed with Sun or IBM JVMs is not interpreted, but compiled to native and heavily optimized code. Before anybody complains: yes, the Sun VM uses mixed-mode execution, which means that code is interpreted first until it has been used for a certain amount of times, then it's compiled. It might seem like a contradiction, but that's mostly a performance improvement (turning byte code into native code with optimizations costs time... often more time than it would take to simply interpret the code; not to mention the memory overhead of creating native code for every one-off method). IBMs VMs always JIT compile their code, but they have different levels of JIT Compilers; the first time code is used, a baseline compiler simply transforms byte code into native code. If that code is heavily used, it is optimized and the old code is replaced with the faster version. Again: No Bytecode interpretation, Dynamic compilation! Next week: Why Java != JavaScript

Re:GCJ slower than a native JVM?

[murphee]

The big advantage of VMs and dynamic compilers is that they can take advantage of the CPU that they are running on. Ie. if you use gcj to compile software, you have to compile optimize it for a specific CPU (instruction order is important and makes a difference, as different OutOfOrder engines on different CPUs (think Intel vs. AMD) act differently). It also means more complicated deployment, for instance: if your code uses SSE2, your program only runs on Pentium4 and other CPUs that have SSE2. If you don't want that, you have to factor out the functions that make use of SSE2 and provide replacements that don't use it (which also means having to check the CPU at startup and choosing the libs appropiately).
This doesn't matter for a compiler working at runtime, as it knows about the capabilities of the CPU and just uses them if possible (I know that the JVM uses SSE when available, though I haven't found out what exactly it is used for).
A JIT/DynamicCompiler also knows everything about your CPU, for instance Cache sizes,... and can exploit that knowledge to further optimize the code or data layout.

Also: you have to keep in mind, that Java (like .NET) is a dynamic environment, ie. you can load classes at runtime. This proves to be a bit of a problem with static compilation, or I should say: with optimization. Hotspot (Suns Dynamic Compiler or VM and also other JVMs) do some optimizations that improve OOP code, like devirtualization (which removes the vtable lookup overhead for virtual method calls) and even virtual method inlining.
Now: these optimizations have one problem: they may be accurate when the code is generated... but what happens when a new class is loaded? For instance, the compiler devirtualizes a method in class A, because Class Hierarchy Analysis tells him, that this method is never overridden. So... 10 minutes later, a class is loaded, that is derived from class A, and overrides the aforementioned method... so... the code is suddenly incorrect. In this case, the dynamic compiler, takes back the old optimization, and everything is OK again.
This kind of scenario makes it necessary for many optimizations to have a system (compiler) working at runtime.

Mind you: Runtime code loading or generation is a point where gcj gets really slow, because then it has to run that code with gij (the interpreter). There are people that got Eclipse compiled with gcj, but they ran into that problem too (the Plugins in Eclipse are loaded at runtime; they worked around that by compiling the plugins as shared libraries).