Hope these help shed some light on what's going on with these benchmarks. As you can see, even as poor as g++ does, it's still proving to be faster than java, and that's using ideal corner cases for java.
If you follow the link about java's gc, I attempted to let the 1:1 gc implementation run all night. I killed it this morning. It never completed. Clearly a 1:1 implementation does not appear to be a fair comparison, just the same, it highlights that any real java application where gc does become a factor, java's performance does go out with the baby and bathwater.
This is what I've "learned", but I have idea as the the validity of it. It does seem to jive with everything else I've heard and read, so I don't currently have reason to question it.
It appears that we're both right! What? Ya, it appears that System.gc() does not immediately cause the collector to run, well, not always and not in the same thread of execution. Furthermore, it's also considered a hint because if you call it and the collector is already running, it supposed to ignore the additional requests. Lastly, it appears that the collector runs as its own, very low priority thread. That means, it probably won't get much runtime on systems under heavy loads. Especially when the code in question is a tight loop.
So, I do agree that a 1:1 implementation is not fair, just the same, it is technically a hint or request to collect, nothing more. Furthermore, the completion of System.gc() does not appear to actually indicate that collection is anywhere near completion.
Just because the benchmark is flawed, doesn't mean that Java was somehow bad.
I didn't assert that Java was bad. Personal disappointment is certainly my right.
Especially since your benchmark was a thousand times more flawed. Since you have shown little knowledge about Java (every beginning Java programmer learns that you should avoid calling gc directly), I suggest you stop making these stupid comments./end rant
Actually, my "benchmark" was not flawed at all. It did exactly what it was supposed to do. If you took it at face value, then that's your problem and not a fault of my results. What my result attempted to prove was that these benchmarks were flawed and that they were purposely hiding java's gc overhead. What my "benchmark" also showed is that java's gc overhead can be significant. It also highlighted the fact that since c++'s new and delete operators can be overloaded, a fair comparison would be to make the c++ implementation perform the same collection deferment.
I think results did exactly what I setup out to do. Take them with a grain of salt but walk away with the lessons learned.
I wish I could find what I originally read. It stated that it simply issued a request for the gc to collect. Under a high load (tight loop, in this case), chances are, the gc will not collect. Accordingly, I have no problem finding many posts by "java gurus" which seem to be under the same impression I am (that clearly doesn't make it true though). Also, even here, others have piped in and offered that gc is run in a low priority thread whereby, the "hint" is actually a mechanism which attempts to wake up the gc, assuming it's not already running. Thusly, while I do agree that calling System.gc() on a one to one basis is not fair (clearly not an apples to apples comparison), I'm still not convinced that simply calling System.gc(), and it returning, means that all memory has been collected.
Regardless, I do believe I have a legitimate point and that these benchmarks are fundimentally flawed.
Hopefully you'll realize that the comparison is still false and everything I stated is spot on. In fact, you didn't state (java wise) anything that I didn't already state.
Just the same, to be fair, there is absoluetely no reason why delete and new couple be replaced to function in the same deferred manner. So, avoiding GC in java and forcing MM in C++ is clearly a case of a piss-poor benchmark. To create an apples to apples comparison, either java must be forced to collect during it's run or the c++ implementation must be forced to defer. Only then will you have an honest benchmark.
Until such time, as I originally stated, all of the benchmarks which create tons of objects and then delete them, are invalid. As is, the java benchmarks hide java's true performance. Which, is exactly my point.
Nope. But, I can make some fairly safe assumptions. As Java can be capped as to the maximum memory that it will use. Accordingly, any server application (which, IMO, is what these attempt to capture), is not going to be configured to be paging it's workload. Accordingly, if you're running a java server and you're paging, something is horribly wrong.
I completely agree with you. Your assessment matches my own, which I've offered in other threads. It's just that making the changes in the java code was the fastest path to gleen some measure of comparability. I completely agree that my changes were completely lacking scientific method but they were 100% reproducible, which does seem to lend some credence to them (with a large grain of salt).
As for JVM start up, I think it's completely fair and just to add it in. For short run utilities, it reflects real world delays. For long run applications, it will become noise. As such, in either case, it's realistic and completely fair.
There is no doubt that hot spot can certainly make note worthy improvements. Heck, python's own pysco seems to offer additional proof of concept. Just the same, clearly these benchmarks are quickly proving that they do not reflect real world applications in the least.
FYI, I attempted to let the 1:1 implementation run all night. It never completed. I had to kill it this morning. So clearly, more is going on under the covered with System.gc() than is typically known (at least be me). So, I think it does support the argument that a 1:1 implementation is clearly not an apples to apples comparison either.
Wow. Fair enough. That squarely places the java implementation, assuming the c++ version were to be completely optimized, more than half the performance.
Anyone who is worried about the speed differences between the two languages wouldnt use g++, and wouldnt use java.
How do you plan to support such a statement? To me, you're implying that anyone using Java cares nothing for performance. Yet, you then turn and say the same thing for g++, which of course, is simply not true. While g++ is clearly not the faster compiler around, it generally doesn't do that bad of a job.
Which to me doesnt say a whole frickin lot.
I happen to agree, especially since more looking indicates that this is more an example of poorly writen C++ code using ideal performance corner cases for java's hot spot technology. I especially agree with your assessment since all of these benchmarks appear to purposely avoid any GC during their short runtimes. Which means, it's not even starting to reflect real world java application runtimes, which seems to common among almost all java benchmarks.
I hope you will not confuse me with a java zealot as I am NOT a java guy. Feel free to look up my profile and read the many other messages I've posted on this subject.
Now then, after all that, I still argue that you simply can not dismiss these benchmarks because java was compared to gnu's tools. If you insist on doing so, you must then disregard every samba vs win or apache vs iis benchmark because apache and samba are typically both compiled with gnu's tools. Likewise, so is the kernel. You ready to toss all of those winning benchmarks away?
Simple fact is, there are many holes with these benchmarks but the compiler used does not appear to be one of them.
If I still haven't brought you around to my viewpoint, then I guess we'll have to agree to disagree.
Seems I've been validated in my action of marking you a foe. You might want to take a look at your self to figure out why I lashed out at you. And, bluntly, it's because you asked for, just as you did now. You were as wrong then as you are now.
Go back a reread what I posted. I did offer a polite counter point. You attacked, and insanely, you clearly don't even have a leg to stand one. That's what puts you over the edge of being an idiot.
Go on. Prove me wrong. Go back and reread my post. Act like a child you want, but that hardly validates your childish behavior.
And, I'll point out, again you dodge supporting your view point. The reason is fairly obvious. You clearly don't have a leg to stand on so your last resort is to attack the person. I think I'm seeing a pattern here.
Here's the previous message to which you attacked:
I guess you're happy to ignore the fact that g++ is the defacto compiler on the Linux platform? You happy to ignore that gnu's compilers are used on the applications that just about everything else is benchmarked against. [so, how can this make the comparison unfair??]
You're just not happy with the results. [is it rude to point ot the obvious here?]
Frankly, I've not look at much code yet, so I'm trying to figure out if the benchmarks actually have any merit. But, dismissing them because you didn't like the results (which is what your excuse boils down to), is simply not fair. Ya, g++ is clearly not the fastest compiler but it is the compiler that most Linux applications are compiled with. Dismissing it because you simply want to, is what's not fair.
[so, if it's unfair, why, since just about all linux applications are compiled via gnu's tools?]
So, please let me know how I attacked you? How was I not polite? You asserted many things, can you support any of them?
However it is not really valid to say that calling System.gc() is the same as deleting ONE indiviudal object in C++.
Why not? According to the java docs, it's a hint that the GC should considering gc'ing some time soon. Technically, the majority of the time, according to java's docs, it should do nothing at all. So, if the majority of the calls are more or less an empty function call, what's the harm in doing it? Well, other than making the java application pay for it's allocation costs.
Has the JVM changed such that it no longer hints? Are you saying that it now actually forces collection to occur?
Troll??? WTH? Talk about ignorant trolls. What a dolt.
It's a benchmark. We're benchmarking performance.
And your point is?
Compiling it with a compiler which is not designed to optimize well is NOT a good benchmark.
Why?
And by the way, this site is News for Nerds, not News for Linux Zealots.
RTFA. It was benchmarked on Linux.
Java runs on Windows too. In fact, MOST if not nearly all Java code runs on Windows systems by sheer weight of numbers.
And....if you have a point, make it.
Besides... you've already proven in other threads that you don't know jack about programming, so please, butt out troll.
What an ignorant, small minded troll you are.
Basically, you're calling me a troll because you clearly don't know anything about programming, insist that a a test is invalid just because you think g++ is slow, and because you think the tests should be run on windows? WTH? Can you prove that you're any more of an idiot? Worse, I offered a polite and direct counter point and your response is to attack and avoid answering any of the raised questions. What a tiny mind you have.
That's simply not true if you expect benchmarks to reflect real world applications. Sure, it might be true for trivial utilities but for applications that stay alive for long periods of time chugging along, which is probably the majority of the worlds useful applications, it makes the results invalid. Or, at a minimum, sets greatly unrealistic expectations.
The theory behind garbage collection isn't just that it allows the programmer to avoid the effort of watching when to delete things. It's that garbage collection can actually improve performance on certain workloads.
This is true..."on some workloads". I certainly recognize that fact. I hoped I had more or less pointed that out in my original message. At any rate, for real world workloads, I would offer that such expectations are not realistic.
Forcing a garbage collection for every delete is completely unfair, since it does a full scan of memory, as opposed to just twiddling bits to free a single data value.
Well, technically, even with the 1:1 implementation, it's not forcing collection. It's actually hinting to the GC system, that it should consider doing so. The GC is 100% free to ignore such a request. This is per the java specifications. Which, makes plenty of sense because doing so tends to greatly increase the window where GC can improve performance in various workloads. The problem is, I honestly don't know where implementation and specification lay. I've read many times that it actually does ONLY result in a hint to collect. Unless you can prove otherwise, I'm apt to lean in that direction. Which, actually means that the 1:1 implementation is a more realisitic apples to apples comparison. Can I prove that? No. I'm still hoping a java guru will come in with some insightful tidbits.;)
There's no memory leak for these benchmarks... both C++ and Java free all memory used when the process exits. Perhaps you'd prefer a longer-running test with lot of garbage generation (forcing gc to run at some point).
Yes, I'm aware that there are no leaks. That's not my complaint. My complaint is that just about every java benchmark I've ever seen is so artificially small that they all appear to purposely avoid GC ever becoming part of the benchmark. Since most major applications simple don't start, run for 20 seconds and exit, I don't consider the presumed bevaior to be valid.
The secondary point I was attempting to raise (along with awareness to the hidden gc issue) is that once Java is forced to start collecting, it's pays a significant cost. A cost, I'd hazzard a guess, that brings about the common complaints of Java. And that, I offer, is the difference between labratory tests and real world application.;)
I can't argue with that. As is, it certainly makes sense to have a hash as lightweight as is required. In this case, it appears that your implementation fits the bill nicely. Just the same, I'm hoping that some Java guy will jump in and offer some insight so that we could perhaps run a true apples to apples comparison.
Personally, I'm content to accept your implementation as valid as long as people are aware of the caveat.
The more I read the code, it sure is starting to look more and more like an apples and oranges comparison; which is usually what happens when people do java benchmarks.
As typically observed, I'm seeing the benchmarks take serious advantage of java's GC mechanism, whereby, they never pay the piper. With C++, the piper is constantly being paid. All of the benchmarks which allocate objects and then delete them, are therefore, invalid. To be fair, I think, you either need to add a System.gc() line in the java code where C++ is doing its deletes or you need to implement your own new and delete operators to function more in line with what java is doing. Until you do either one of those things, the comparisons where objects are being allocated and deallocated are invalid. And frankly, I'm still not sure adding System.gc() is even fair, on either side. The reason is, calling System.gc() simply hints that it's a good idea to collect. There is nothing which requires the collection to take place. So, technically, the call could still be many times faster. On the other hand, I don't know enough about how they handle their gc-hint logic nor am I aware of exactly how much overhead is involved in the actual collection process. If it occurs too often, the shift in workload may be too unfair. Nonetheless, it's a point of very serious contention.
Just for kicks, I modified objinst.java with, "if( i%(n/1000) == 0 ) System.gc();", on the lines that the C++ code had it's delete. When I timed it, it was over twice as slow as the C++ code (24+s vs 55+s). Worse, when I ran it with a 1:1 ratio of delete:System.gc(), I simply got tired of waiting, having waited over 5 minutes.
So, basically, I'm not nearly as impressed as I first was. Simplistically, it's starting to look like a serious apples and oranges comparison. Elsewhere, you can find other examples of just plain bad code. Where again, with correct C++ code, came in about twice as fast at the Java code, whereby, more optimizations were still possible with the C++ code.
So, it looks like we're seeing a combination of things here. Looks like a combination of bad code, ideal corner cases for java's hot spot, and invalid comparisons with memory allocation.
Sadly, I'm once again seriously disappointed in java.
Keeping in mind, I'm not a java guy, but I did see something that I want to question. Above, you have "hashme", which is clearly about as simplistic as simplistic can get. I do not see the equivelent in the java code. So, I'm assuming that means it's using some other hash method which is built in. Is the built in hash method comparable to your very simplistic hash method? If not, you're benchmarking an apples and orages difference.
Aside from that concern, everthing else looks good. Care to address my concern?
but there is no way I can accept that java is that much FASTER than properly done C++
Well, bluntly, you need to better understand the technology so you can adjust your expectations. While I've only started looking at the code, it does appear that these benchmarks represent many ideal situations whereby java's hotspot technology can make many aggressive optimizations.
These benchmarks, by no stretch of the imagination mean that Java is faster than C++. It does, however, clearly show that in ideal situations, hotspot can clearly make huge leaps in performance.
Worth noting, psyco for python (a hot-spot "like" technology), also has ideal situations where it actually performs faster than c. Again, this is because it's able to more aggressively optimize in certain situations with more knowledge than the compiler had during compilation. Does this mean that, in whole, python+psyco is also faster than c? Of course not. But, it certainly does lend support to the notion that java and it's hot spot technology can not be dismissed out right, as I see many racing to do.
The big question, which I currently do not have an answer to is, do these benchmarks have any meaning in the least, in the real world? If so, in what situations do they reflect?
In stead of panicing about these results, let's take them for what they are, lick on a grain of salt whlie we're at it, and figure out if they mean anything in the real world. I suspect that they don't, but that's just my bias talking. In the mean time, I'm going to be looking at some code.
I guess you're happy to ignore the fact that g++ is the defacto compiler on the Linux platform? You happy to ignore that gnu's compilers are used on the applications that just about everything else is benchmarked against.
You're just not happy with the results.
Frankly, I've not look at much code yet, so I'm trying to figure out if the benchmarks actually have any merit. But, dismissing them because you didn't like the results (which is what your excuse boils down to), is simply not fair. Ya, g++ is clearly not the fastest compiler but it is the compiler that most Linux applications are compiled with. Dismissing it because you simply want to, is what's not fair.
Using g++ for this simply causes the study to lose credibility
I'm sorry, but that statement holds no merit. The g++ compiler is considered a defacto implementation for many, many platforms, including Linux. While g++ is clearly not the fastest of compilers, it is certainly used-a-plenty. If you're running Linux, chances are, all of your applications have been compiled with it. Furthermore, the java that was being used in the same benchmark, was probably also compiled with it (it's compiled c++, right?). So, your dismissal is nothing more than a ploy. Seriously, if you think your logic holds merit, then you must dismiss any and every benchmark you've ever read about that was run on Linux.
Now then, what you can't say from these bechmarks is that java is faster than C++. These benchmarks don't prove that. What they do say is, for the benchmarks ran, java was running close to and in some cases faster than GNU's g++ compiler.
The simple fact that java is actually getting in the same ballpark as g++ is certainly slashdot news worthy. Also, keep in mind that I've also considered myself to be a pro-C++ kind of guy. Frankly, everytime I start to learn Java, I get disgusted. I actually don't care for the language much. But...it's clearly getting faster and deserves it's hard fought recognition.
Doh! Right afterward I posted that, I figured it out. I had to change the, "-cp java", part to, "-cp.", and it works fine.
In case anyone is curious, the results of the tests above, which was randomly picked, were measured as: cpp: 24.72user 0.04system 0:25.30elapsed 97%CPU java-server: 17.71user 0.59system 0:18.79elapsed 97%CPU java-client: 17.86user 0.66system 0:19.95elapsed 92%CPU
Color me impressed. Now then, my system is not setup to do real benchmarking and I was running many other processes which may of stolen some CPU here and there. Just the same, the results do appear to match (this test anyways; it's the only one I ran so far). I also ran each result three times and they were all running very close each previous run, so, I'm willing to accept them for what they are -- reproduceable.
I guess I'll have to spend some time thinking about what's going on and actually look at the code to make sure all is fair. Thus far, I am impressed.
Slashdot Mirror
Please share your updated java code.
reasons like this and and this are why you shouldn't get too bothered by these benchmarks.
Hope these help shed some light on what's going on with these benchmarks. As you can see, even as poor as g++ does, it's still proving to be faster than java, and that's using ideal corner cases for java.
If you follow the link about java's gc, I attempted to let the 1:1 gc implementation run all night. I killed it this morning. It never completed. Clearly a 1:1 implementation does not appear to be a fair comparison, just the same, it highlights that any real java application where gc does become a factor, java's performance does go out with the baby and bathwater.
Cheers.
Thanks! Chances are, I'm right in the middle of those other posts. ;)
Cheers!
This is what I've "learned", but I have idea as the the validity of it. It does seem to jive with everything else I've heard and read, so I don't currently have reason to question it.
It appears that we're both right! What? Ya, it appears that System.gc() does not immediately cause the collector to run, well, not always and not in the same thread of execution. Furthermore, it's also considered a hint because if you call it and the collector is already running, it supposed to ignore the additional requests. Lastly, it appears that the collector runs as its own, very low priority thread. That means, it probably won't get much runtime on systems under heavy loads. Especially when the code in question is a tight loop.
So, I do agree that a 1:1 implementation is not fair, just the same, it is technically a hint or request to collect, nothing more. Furthermore, the completion of System.gc() does not appear to actually indicate that collection is anywhere near completion.
Just because the benchmark is flawed, doesn't mean that Java was somehow bad.
/end rant
I didn't assert that Java was bad. Personal disappointment is certainly my right.
Especially since your benchmark was a thousand times more flawed. Since you have shown little knowledge about Java (every beginning Java programmer learns that you should avoid calling gc directly), I suggest you stop making these stupid comments.
Actually, my "benchmark" was not flawed at all. It did exactly what it was supposed to do. If you took it at face value, then that's your problem and not a fault of my results. What my result attempted to prove was that these benchmarks were flawed and that they were purposely hiding java's gc overhead. What my "benchmark" also showed is that java's gc overhead can be significant. It also highlighted the fact that since c++'s new and delete operators can be overloaded, a fair comparison would be to make the c++ implementation perform the same collection deferment.
I think results did exactly what I setup out to do. Take them with a grain of salt but walk away with the lessons learned.
It doesn't say that at all
I wish I could find what I originally read. It stated that it simply issued a request for the gc to collect. Under a high load (tight loop, in this case), chances are, the gc will not collect. Accordingly, I have no problem finding many posts by "java gurus" which seem to be under the same impression I am (that clearly doesn't make it true though). Also, even here, others have piped in and offered that gc is run in a low priority thread whereby, the "hint" is actually a mechanism which attempts to wake up the gc, assuming it's not already running. Thusly, while I do agree that calling System.gc() on a one to one basis is not fair (clearly not an apples to apples comparison), I'm still not convinced that simply calling System.gc(), and it returning, means that all memory has been collected.
Regardless, I do believe I have a legitimate point and that these benchmarks are fundimentally flawed.
This is true...but c++'s delete isn't exactly free either. ;)
LOL. Clearly a java zealot here.
Hopefully you'll realize that the comparison is still false and everything I stated is spot on. In fact, you didn't state (java wise) anything that I didn't already state.
Just the same, to be fair, there is absoluetely no reason why delete and new couple be replaced to function in the same deferred manner. So, avoiding GC in java and forcing MM in C++ is clearly a case of a piss-poor benchmark. To create an apples to apples comparison, either java must be forced to collect during it's run or the c++ implementation must be forced to defer. Only then will you have an honest benchmark.
Until such time, as I originally stated, all of the benchmarks which create tons of objects and then delete them, are invalid. As is, the java benchmarks hide java's true performance. Which, is exactly my point.
Are you sure?
Nope. But, I can make some fairly safe assumptions. As Java can be capped as to the maximum memory that it will use. Accordingly, any server application (which, IMO, is what these attempt to capture), is not going to be configured to be paging it's workload. Accordingly, if you're running a java server and you're paging, something is horribly wrong.
I completely agree with you. Your assessment matches my own, which I've offered in other threads. It's just that making the changes in the java code was the fastest path to gleen some measure of comparability. I completely agree that my changes were completely lacking scientific method but they were 100% reproducible, which does seem to lend some credence to them (with a large grain of salt).
As for JVM start up, I think it's completely fair and just to add it in. For short run utilities, it reflects real world delays. For long run applications, it will become noise. As such, in either case, it's realistic and completely fair.
There is no doubt that hot spot can certainly make note worthy improvements. Heck, python's own pysco seems to offer additional proof of concept. Just the same, clearly these benchmarks are quickly proving that they do not reflect real world applications in the least.
FYI, I attempted to let the 1:1 implementation run all night. It never completed. I had to kill it this morning. So clearly, more is going on under the covered with System.gc() than is typically known (at least be me). So, I think it does support the argument that a 1:1 implementation is clearly not an apples to apples comparison either.
Wow. Fair enough. That squarely places the java implementation, assuming the c++ version were to be completely optimized, more than half the performance.
I think that one has been completely debunked.
Thanks.
Anyone who is worried about the speed differences between the two languages wouldnt use g++, and wouldnt use java.
How do you plan to support such a statement? To me, you're implying that anyone using Java cares nothing for performance. Yet, you then turn and say the same thing for g++, which of course, is simply not true. While g++ is clearly not the faster compiler around, it generally doesn't do that bad of a job.
Which to me doesnt say a whole frickin lot.
I happen to agree, especially since more looking indicates that this is more an example of poorly writen C++ code using ideal performance corner cases for java's hot spot technology. I especially agree with your assessment since all of these benchmarks appear to purposely avoid any GC during their short runtimes. Which means, it's not even starting to reflect real world java application runtimes, which seems to common among almost all java benchmarks.
I hope you will not confuse me with a java zealot as I am NOT a java guy. Feel free to look up my profile and read the many other messages I've posted on this subject.
Now then, after all that, I still argue that you simply can not dismiss these benchmarks because java was compared to gnu's tools. If you insist on doing so, you must then disregard every samba vs win or apache vs iis benchmark because apache and samba are typically both compiled with gnu's tools. Likewise, so is the kernel. You ready to toss all of those winning benchmarks away?
Simple fact is, there are many holes with these benchmarks but the compiler used does not appear to be one of them.
If I still haven't brought you around to my viewpoint, then I guess we'll have to agree to disagree.
Seems I've been validated in my action of marking you a foe. You might want to take a look at your self to figure out why I lashed out at you. And, bluntly, it's because you asked for, just as you did now. You were as wrong then as you are now.
Go back a reread what I posted. I did offer a polite counter point. You attacked, and insanely, you clearly don't even have a leg to stand one. That's what puts you over the edge of being an idiot.
Go on. Prove me wrong. Go back and reread my post. Act like a child you want, but that hardly validates your childish behavior.
And, I'll point out, again you dodge supporting your view point. The reason is fairly obvious. You clearly don't have a leg to stand on so your last resort is to attack the person. I think I'm seeing a pattern here.
Here's the previous message to which you attacked:
I guess you're happy to ignore the fact that g++ is the defacto compiler on the Linux platform? You happy to ignore that gnu's compilers are used on the applications that just about everything else is benchmarked against. [so, how can this make the comparison unfair??]
You're just not happy with the results. [is it rude to point ot the obvious here?]
Frankly, I've not look at much code yet, so I'm trying to figure out if the benchmarks actually have any merit. But, dismissing them because you didn't like the results (which is what your excuse boils down to), is simply not fair. Ya, g++ is clearly not the fastest compiler but it is the compiler that most Linux applications are compiled with. Dismissing it because you simply want to, is what's not fair.
[so, if it's unfair, why, since just about all linux applications are compiled via gnu's tools?]
So, please let me know how I attacked you? How was I not polite? You asserted many things, can you support any of them?
However it is not really valid to say that calling System.gc() is the same as deleting ONE indiviudal object in C++.
Why not? According to the java docs, it's a hint that the GC should considering gc'ing some time soon. Technically, the majority of the time, according to java's docs, it should do nothing at all. So, if the majority of the calls are more or less an empty function call, what's the harm in doing it? Well, other than making the java application pay for it's allocation costs.
Has the JVM changed such that it no longer hints? Are you saying that it now actually forces collection to occur?
Troll??? WTH? Talk about ignorant trolls. What a dolt.
It's a benchmark. We're benchmarking performance.
And your point is?
Compiling it with a compiler which is not designed to optimize well is NOT a good benchmark.
Why?
And by the way, this site is News for Nerds, not News for Linux Zealots.
RTFA. It was benchmarked on Linux.
Java runs on Windows too. In fact, MOST if not nearly all Java code runs on Windows systems by sheer weight of numbers.
And....if you have a point, make it.
Besides... you've already proven in other threads that you don't know jack about programming, so please, butt out troll.
What an ignorant, small minded troll you are.
Basically, you're calling me a troll because you clearly don't know anything about programming, insist that a a test is invalid just because you think g++ is slow, and because you think the tests should be run on windows? WTH? Can you prove that you're any more of an idiot? Worse, I offered a polite and direct counter point and your response is to attack and avoid answering any of the raised questions. What a tiny mind you have.
That's not a bug in the tests, it's a feature.
;)
;)
That's simply not true if you expect benchmarks to reflect real world applications. Sure, it might be true for trivial utilities but for applications that stay alive for long periods of time chugging along, which is probably the majority of the worlds useful applications, it makes the results invalid. Or, at a minimum, sets greatly unrealistic expectations.
The theory behind garbage collection isn't just that it allows the programmer to avoid the effort of watching when to delete things. It's that garbage collection can actually improve performance on certain workloads.
This is true..."on some workloads". I certainly recognize that fact. I hoped I had more or less pointed that out in my original message. At any rate, for real world workloads, I would offer that such expectations are not realistic.
Forcing a garbage collection for every delete is completely unfair, since it does a full scan of memory, as opposed to just twiddling bits to free a single data value.
Well, technically, even with the 1:1 implementation, it's not forcing collection. It's actually hinting to the GC system, that it should consider doing so. The GC is 100% free to ignore such a request. This is per the java specifications. Which, makes plenty of sense because doing so tends to greatly increase the window where GC can improve performance in various workloads. The problem is, I honestly don't know where implementation and specification lay. I've read many times that it actually does ONLY result in a hint to collect. Unless you can prove otherwise, I'm apt to lean in that direction. Which, actually means that the 1:1 implementation is a more realisitic apples to apples comparison. Can I prove that? No. I'm still hoping a java guru will come in with some insightful tidbits.
There's no memory leak for these benchmarks... both C++ and Java free all memory used when the process exits. Perhaps you'd prefer a longer-running test with lot of garbage generation (forcing gc to run at some point).
Yes, I'm aware that there are no leaks. That's not my complaint. My complaint is that just about every java benchmark I've ever seen is so artificially small that they all appear to purposely avoid GC ever becoming part of the benchmark. Since most major applications simple don't start, run for 20 seconds and exit, I don't consider the presumed bevaior to be valid.
The secondary point I was attempting to raise (along with awareness to the hidden gc issue) is that once Java is forced to start collecting, it's pays a significant cost. A cost, I'd hazzard a guess, that brings about the common complaints of Java. And that, I offer, is the difference between labratory tests and real world application.
I can't argue with that. As is, it certainly makes sense to have a hash as lightweight as is required. In this case, it appears that your implementation fits the bill nicely. Just the same, I'm hoping that some Java guy will jump in and offer some insight so that we could perhaps run a true apples to apples comparison.
;)
Personally, I'm content to accept your implementation as valid as long as people are aware of the caveat.
Good work btw.
The more I read the code, it sure is starting to look more and more like an apples and oranges comparison; which is usually what happens when people do java benchmarks.
;", on the lines that the C++ code had it's delete. When I timed it, it was over twice as slow as the C++ code (24+s vs 55+s). Worse, when I ran it with a 1:1 ratio of delete:System.gc(), I simply got tired of waiting, having waited over 5 minutes.
As typically observed, I'm seeing the benchmarks take serious advantage of java's GC mechanism, whereby, they never pay the piper. With C++, the piper is constantly being paid. All of the benchmarks which allocate objects and then delete them, are therefore, invalid. To be fair, I think, you either need to add a System.gc() line in the java code where C++ is doing its deletes or you need to implement your own new and delete operators to function more in line with what java is doing. Until you do either one of those things, the comparisons where objects are being allocated and deallocated are invalid. And frankly, I'm still not sure adding System.gc() is even fair, on either side. The reason is, calling System.gc() simply hints that it's a good idea to collect. There is nothing which requires the collection to take place. So, technically, the call could still be many times faster. On the other hand, I don't know enough about how they handle their gc-hint logic nor am I aware of exactly how much overhead is involved in the actual collection process. If it occurs too often, the shift in workload may be too unfair. Nonetheless, it's a point of very serious contention.
Just for kicks, I modified objinst.java with, "if( i%(n/1000) == 0 ) System.gc()
So, basically, I'm not nearly as impressed as I first was. Simplistically, it's starting to look like a serious apples and oranges comparison. Elsewhere, you can find other examples of just plain bad code. Where again, with correct C++ code, came in about twice as fast at the Java code, whereby, more optimizations were still possible with the C++ code.
So, it looks like we're seeing a combination of things here. Looks like a combination of bad code, ideal corner cases for java's hot spot, and invalid comparisons with memory allocation.
Sadly, I'm once again seriously disappointed in java.
Keeping in mind, I'm not a java guy, but I did see something that I want to question. Above, you have "hashme", which is clearly about as simplistic as simplistic can get. I do not see the equivelent in the java code. So, I'm assuming that means it's using some other hash method which is built in. Is the built in hash method comparable to your very simplistic hash method? If not, you're benchmarking an apples and orages difference.
Aside from that concern, everthing else looks good. Care to address my concern?
Fair enough. Thanks for the reply.
I have not looked at the changed code, but I do reproduce your results:
;)
java-server: 14.93user 0.10system 0:15.67elapsed 95%CPU
g++-O2: 8.10user 0.03system 0:08.38elapsed 96%CPU
That places g++ about twice as fast as java, at that benchmark, assuming the code is correct.
but there is no way I can accept that java is that much FASTER than properly done C++
Well, bluntly, you need to better understand the technology so you can adjust your expectations. While I've only started looking at the code, it does appear that these benchmarks represent many ideal situations whereby java's hotspot technology can make many aggressive optimizations.
These benchmarks, by no stretch of the imagination mean that Java is faster than C++. It does, however, clearly show that in ideal situations, hotspot can clearly make huge leaps in performance.
Worth noting, psyco for python (a hot-spot "like" technology), also has ideal situations where it actually performs faster than c. Again, this is because it's able to more aggressively optimize in certain situations with more knowledge than the compiler had during compilation. Does this mean that, in whole, python+psyco is also faster than c? Of course not. But, it certainly does lend support to the notion that java and it's hot spot technology can not be dismissed out right, as I see many racing to do.
The big question, which I currently do not have an answer to is, do these benchmarks have any meaning in the least, in the real world? If so, in what situations do they reflect?
In stead of panicing about these results, let's take them for what they are, lick on a grain of salt whlie we're at it, and figure out if they mean anything in the real world. I suspect that they don't, but that's just my bias talking. In the mean time, I'm going to be looking at some code.
I guess you're happy to ignore the fact that g++ is the defacto compiler on the Linux platform? You happy to ignore that gnu's compilers are used on the applications that just about everything else is benchmarked against.
You're just not happy with the results.
Frankly, I've not look at much code yet, so I'm trying to figure out if the benchmarks actually have any merit. But, dismissing them because you didn't like the results (which is what your excuse boils down to), is simply not fair. Ya, g++ is clearly not the fastest compiler but it is the compiler that most Linux applications are compiled with. Dismissing it because you simply want to, is what's not fair.
Using g++ for this simply causes the study to lose credibility
I'm sorry, but that statement holds no merit. The g++ compiler is considered a defacto implementation for many, many platforms, including Linux. While g++ is clearly not the fastest of compilers, it is certainly used-a-plenty. If you're running Linux, chances are, all of your applications have been compiled with it. Furthermore, the java that was being used in the same benchmark, was probably also compiled with it (it's compiled c++, right?). So, your dismissal is nothing more than a ploy. Seriously, if you think your logic holds merit, then you must dismiss any and every benchmark you've ever read about that was run on Linux.
Now then, what you can't say from these bechmarks is that java is faster than C++. These benchmarks don't prove that. What they do say is, for the benchmarks ran, java was running close to and in some cases faster than GNU's g++ compiler.
The simple fact that java is actually getting in the same ballpark as g++ is certainly slashdot news worthy. Also, keep in mind that I've also considered myself to be a pro-C++ kind of guy. Frankly, everytime I start to learn Java, I get disgusted. I actually don't care for the language much. But...it's clearly getting faster and deserves it's hard fought recognition.
Doh! Right afterward I posted that, I figured it out. I had to change the, "-cp java", part to, "-cp .", and it works fine.
In case anyone is curious, the results of the tests above, which was randomly picked, were measured as:
cpp: 24.72user 0.04system 0:25.30elapsed 97%CPU
java-server: 17.71user 0.59system 0:18.79elapsed 97%CPU
java-client: 17.86user 0.66system 0:19.95elapsed 92%CPU
Color me impressed. Now then, my system is not setup to do real benchmarking and I was running many other processes which may of stolen some CPU here and there. Just the same, the results do appear to match (this test anyways; it's the only one I ran so far). I also ran each result three times and they were all running very close each previous run, so, I'm willing to accept them for what they are -- reproduceable.
I guess I'll have to spend some time thinking about what's going on and actually look at the code to make sure all is fair. Thus far, I am impressed.