I'm the author of the "Java on Linux" article in the August Issue of Linux-Journal. Please note that I never claimed this benchmark to be "rigorous" or "complete" -- because it isn't. All it is is a simple benchmark to measure the performance of Java programs in terms of how many objects can be created, used, and discarded in a given amount of time. I believe this is fairly typical behaviour for OO programs, but, it does in no way justice to all aspects of a Java implementation (such as JDBC, Java 2D, Swing, etc.). In fact, no single benchmark could do this.
Quite a few people sent me e-mail with comments, suggestions, and criticism. I prepared a summary to these questions and mailed it to the authors. I'd like to post the summary to this discussion group as well, since many of the questions and objections turn up here too.
So here it is....
REPLY TO LINUX-JOURNAL ARTICLE ON JAVA IMPLEMENTATIONS
================================================ ======
1. GENERAL
1.1 IBM JDK 1.3
At the time I wrote the article, the IBM JDK 1.3 was not available
yet for Linux and therfore it wasn't in the original article.
When it was released, I immediately ran the benchmark again
and obtained the best results I have ever seen on Linux. I sent
a short summary with the results to LJ and asked if they could
add a "last minute news" box with the results to the end of the
article.
Well, the LJ editors decided to place the IBM JDK 1.3 results in
the introduction section of the article, where they are a bit
out of place.
1.2 Article Headline: "The Java interpreter on Windows is faster than
the fastes JIT available on Linux..."
This headline was selected by the LJ editors, not by myself.
After running the benchmark with the IBM JDK 1.3, it is actually
no longer true, but I wasn't able to rewrite the article in the
short time after IBMs JDK 1.3 became avaible and before the
article went to press.
1.3 Borland Delphi
Someone pointed out that Delphi would be simpler than Java
and as efficient as C. I'm not familiar with Delphi, so I cannot
comment on this. However, as far as I know, there are no open
source implementations of Delphi, so at least for me it's not
a candidate for the open source programming language I mentioned
casually at the end of the article.
1.4 Why was Tower/J not included in the benchmark ?
Tower/J is a commercial product. Although they do provide
evaluation licenses, you have to justify _why_ you want one.
Since I was unsure if doing a simple benchmark for LJ would
be justification enough, I didn't bother to ask.
2. QUALITY OF THE JAVA BENCHMARK
2.1 The benchmark is not indicative for real world Java applications.
Yes, this is true. No single benchmark can do justice to all
aspects of a Java implementations, so I picked something very
simple (ie. creating objects and calling methods on them) which
_all_ programs do (well, at least all reasonable OO programs).
2.2 Flyweight Pattern
Someone pointed out that applying the flyweight pattern would
improve the performance of the code by a factor of 5. This is
certainly true, since flyweight eliminates the overhead of
creating 500k objects by using just one object and associated
IDs. I agree that this is what every sensible software engineer
should use in a real world program of production quality.
However, the intent of the benchmark was to compare the relative
performance of different Java implementation on Linux in terms
of "how much does creating an object cost ?", so saving on the
number of objects to be created wasn't an option.
3. THE C++ BENCHMARK
3.1 Why is there such a big difference in performance between compiled
C++ code and compiled Java code ?
I'm not a compiler implementer (ask Per Bothner of Cygnus Support,
he would probably know), but I think that the overhead for garbage
collection and synchronization mechanisms for multithreading are
likely explanations.
3.2 C++ Optimization
I didn't use any optimizations when compiling the C++ program,
because I didn't use any on the Java benchmark either.
3.3 By reference vs. by value semantics
Someone pointed out that replacing "ptarr" (which is an array
of pointers to "PerfTest" objects) with an array of "PerfTest"
objects will result in a 20 fold performance improvement.
I can believe this, since it eliminates 500k calls to "new
PerfTest".
However, this is not the point, since Java has no comparable
mechanisms
for this. In Java, all objects have "by reference" semantics, there
is no "by value" semantic for objects. This is one of the reasons
why Java is simpler than C++ (and also why it's slower than C++;-)
)
3.4 Missing "#include" statements
The #includes at the beginning of the C++ program (listing 2)
should
read like this:
#include
#include
#include
Since I sent the original article in HTML format to LJ,
their HTML converter probaly ate the includes, thinking
they were some kind of obscure markup tags.
Slashdot Mirror
I'm the author of the "Java on Linux" article in the August Issue of Linux-Journal. Please note that I never claimed this benchmark to be "rigorous" or "complete" -- because it isn't. All it is is a simple benchmark to measure the performance of Java programs in terms of how many objects can be created, used, and discarded in a given amount of time. I believe this is fairly typical behaviour for OO programs, but, it does in no way justice to all aspects of a Java implementation (such as JDBC, Java 2D, Swing, etc.). In fact, no single benchmark could do this.
= ======
;-)
Quite a few people sent me e-mail with comments, suggestions, and criticism. I prepared a summary to these questions and mailed it to the authors. I'd like to post the summary to this discussion group as well, since many of the questions and objections turn up here too.
So here it is....
REPLY TO LINUX-JOURNAL ARTICLE ON JAVA IMPLEMENTATIONS
===============================================
1. GENERAL
1.1 IBM JDK 1.3
At the time I wrote the article, the IBM JDK 1.3 was not available
yet for Linux and therfore it wasn't in the original article.
When it was released, I immediately ran the benchmark again
and obtained the best results I have ever seen on Linux. I sent
a short summary with the results to LJ and asked if they could
add a "last minute news" box with the results to the end of the
article.
Well, the LJ editors decided to place the IBM JDK 1.3 results in
the introduction section of the article, where they are a bit
out of place.
1.2 Article Headline: "The Java interpreter on Windows is faster than
the fastes JIT available on Linux..."
This headline was selected by the LJ editors, not by myself.
After running the benchmark with the IBM JDK 1.3, it is actually
no longer true, but I wasn't able to rewrite the article in the
short time after IBMs JDK 1.3 became avaible and before the
article went to press.
1.3 Borland Delphi
Someone pointed out that Delphi would be simpler than Java
and as efficient as C. I'm not familiar with Delphi, so I cannot
comment on this. However, as far as I know, there are no open
source implementations of Delphi, so at least for me it's not
a candidate for the open source programming language I mentioned
casually at the end of the article.
1.4 Why was Tower/J not included in the benchmark ?
Tower/J is a commercial product. Although they do provide
evaluation licenses, you have to justify _why_ you want one.
Since I was unsure if doing a simple benchmark for LJ would
be justification enough, I didn't bother to ask.
2. QUALITY OF THE JAVA BENCHMARK
2.1 The benchmark is not indicative for real world Java applications.
Yes, this is true. No single benchmark can do justice to all
aspects of a Java implementations, so I picked something very
simple (ie. creating objects and calling methods on them) which
_all_ programs do (well, at least all reasonable OO programs).
2.2 Flyweight Pattern
Someone pointed out that applying the flyweight pattern would
improve the performance of the code by a factor of 5. This is
certainly true, since flyweight eliminates the overhead of
creating 500k objects by using just one object and associated
IDs. I agree that this is what every sensible software engineer
should use in a real world program of production quality.
However, the intent of the benchmark was to compare the relative
performance of different Java implementation on Linux in terms
of "how much does creating an object cost ?", so saving on the
number of objects to be created wasn't an option.
3. THE C++ BENCHMARK
3.1 Why is there such a big difference in performance between compiled
C++ code and compiled Java code ?
I'm not a compiler implementer (ask Per Bothner of Cygnus Support,
he would probably know), but I think that the overhead for garbage
collection and synchronization mechanisms for multithreading are
likely explanations.
3.2 C++ Optimization
I didn't use any optimizations when compiling the C++ program,
because I didn't use any on the Java benchmark either.
3.3 By reference vs. by value semantics
Someone pointed out that replacing "ptarr" (which is an array
of pointers to "PerfTest" objects) with an array of "PerfTest"
objects will result in a 20 fold performance improvement.
I can believe this, since it eliminates 500k calls to "new
PerfTest".
However, this is not the point, since Java has no comparable
mechanisms
for this. In Java, all objects have "by reference" semantics, there
is no "by value" semantic for objects. This is one of the reasons
why Java is simpler than C++ (and also why it's slower than C++
)
3.4 Missing "#include" statements
The #includes at the beginning of the C++ program (listing 2)
should
read like this:
#include
#include
#include
Since I sent the original article in HTML format to LJ,
their HTML converter probaly ate the includes, thinking
they were some kind of obscure markup tags.