Actually, Google is bloating like crazy. If you want "just search", I'd suggest you go for All The Web. They're now owned by Overture, and the site is very focused on the web.
You should get out more. There are tons of search engines which are quite clear about what are paid ads and what are not. There are tons of search engines which do text only ads. Indeed Overture owns one of them, and it's results are about as relevant as Google's.
Modern C++ Design actually does a good job describing some of the more sophisticated metaprogramming work you can do with C++. It actually demonstrates that you can determine a lot of type information when metaprogramming thanks to C++ template semantics. Really cool stuff.
Modern C++ Design actually does a good job describing some of the more sophisticated metaprogramming work you can do with C++. It actually demonstrates that you can determine a lot of type information when metaprogramming thanks to C++ template semantics. Really cool stuff.
The boost guys have developed MPL, which really does seem to work reasonably well.
The boost lambda library, if you can get it to compile, gives you most of what you'd want with regard to lambdas.
You can sort of use the "any" stuff in the boost library to accomplish this. Honestly though, with C++'s already complex semantics, I would not want to add type inferencing. I suspect things would get FAR too unpredictable.
Why Apple wouldn't work with IBM and Motorolla to make gcc the best compiler possible for PPC I don't know
They are doing exactly that. However, it is quite hard to get gcc well tuned as compared to modern compilers. Keep in mind that Intel has also been contributing to x86, and as we can see their own compiler is much better. The planes for gcc v3.4 and v3.5 are going to dramatically change the way the compiler works, and make it much easier to produce a high performance compiler for all platforms.
Sorry, I should have been more clear: gcc produces faster code than CodeWarrior (which is what we are talking about here). CodeWarrior *compiles* code more quickly.
even PGCC and EGCS doesn't mean a great deal in terms of current x86 processors
Those projects were folded in to the main gcc project a while ago, but the focus did not die. Indeed, there have been significant contributions from a number of vendors and individuals on tuning the x86 backend since then. The pressure to improve the x86 backend was so strong that the project was actually restructured. As anyone who works on gcc which backend receives the majority of optimization effort, and they'll tell you gcc.
Apple could've done just fine by using Dell's already published SPEC results instead of retesting.
No they couldn't have. There were too many variables between Dell's configuration and whatever optimized configuration Apple used. Dell's numbers demonstrate how wide a performance variance you can get simply by using different software.
If Apple had used an "optimized" configuration and achieved numbers 2x Dell's, this thread would instead be about how the numbers are BS and meaningless because Apple was using software tricks to make their hardware look better. Indeed, that has been the traditional feedback that Apple has received (in many cases, correctly) for previous benchmarks they've published.
The notion of controlling variables in an experiment is a really basic one, and it really disturbs me how people don't seem to be able to grasp this basic scientific concept.
Take a look at any and all of IBM's literature on the performance of the PPC 970.
The spec int scores of the power4 chip are lower than the p4 scores
You are missing the point. At no point did I compare the scores of the Power4 system to the scores of the P4. My point is not that the Power4 is faster than the P4. In fact, if you were trying, you couldn't have come up with a greater misunderstanding of the point.
If Apple wants to prove that they have a superior system....
This is where I think you are missing the point. Apple is not trying to prove they have a superior system. Well, actually they are, but they *aren't* trying to prove that they have a faster system. With the SPEC benchmarks they published, they are trying to prove that they have a faster processor.
[Apple are goint to need to] put out a benchmark showing that they are actually faster than Intel machines.
For the system performance comparisons, they chose to use application benchmarks. They did about 4 different benchmarks as I recall, and to use your parlance, they "totally kicked the crap" out of the Intel machines.
Of course, I'm sure that as far as you are concerned, so long as you can take two numbers from benchmarks with totally different controlled conditions that show the Intel machine winning, it really doesn't matter how many other reasonable benchmarks suggest the other way around. We all know the Intel machine is faster, and it's just a matter of finding the one set of test data that proves it!;-)
Sounds like a pretty broken compiler to me, unless that was padding code which was never executed. I've never seen anything like that even with optimisations turned off, even with ancient versions of gcc.
I have to think at this point you are simply deliberately trolling, but I will try to clarify this in case I am wrong and it's just that your critical thinking skills are weak.
Let's try algebra, as prose does not seem to be working:
let x be the 3GHz P4
let y be the 2GHz PPC970
let z be the 1.5GHz Power4
let o(n) be the SPEC performance of n when using the optimal compiler/heap/OS/etc.
let A(n) be the SPEC performance of n when using the compiler/heap/OS/etc. in Apple's benchmarks
we already know that o(y) > o(z), based on IBM's own tests.
You are arguing that:
o(x) > A(y)
somehow proves that x is faster than y
I am point out that:
o(z) > A(y)
Based on your reasoning above, z is faster than y, but we know that z is slower than y. This is a negative proof demonstrating the flaw in your reasoning.
To restate: the I am mentioning the Power4 CPU to demonstrate that your statement that the P4 has published benchmarks which "kick the crap" out of the numbers in Apple's benchmark's of the G5 is essentially meaningless, and says little about the relative performance of the P4 vs. the G5.
Unfortunately that really only proves that those results used a better compiler/heap/etc. This is a really simple concept folks. I'm sure if they had used a nice compiler along with all kinds of crazy compiler, OS and system optimizations, an even larger group of people would be saying that Apple was misrepresenting the benchmarks.
I'm going to say this once again: the Apple benchmark chose software consistency over potential performance winds in an attempt to isolate the performance benefits of the hardware.
It's worth noting that IBM's own 1500MHz Power4 CPU "kicks the crap" out of the published results for the G5. Guess what? The Power4 processor is the slower processor. IBM's own SPEC benchmarks show that the G5 at 1.8GHz should "kick the crap" out of that 1500MHz CPU, let alone a 2GHz model.
For the record, IBM's own SPEC tests for the 970 put it at 937/1051 when running at 1.8Ghz. Extrapolating this would suggest that the 970 using an "optimised" test bed, would score slightly below those ICC results for integer, while beating it for floating point. So amazingly, this performance differences demonstrated by Apple's benchmarks actually correspond to performance differences that exist in the case where both sides are using an "optimised" test bed.
Sadly, I haven't seen any 970 SPEC_*_rate benchmarks using IBM's compiler yet, but given that those benchmarks are much more influenced by motherboard design, it's quite possible those numbers would be misleading anyway. That being said, for the SPEC_*_rate benchmarks, IBM's 1500MHz Power4 system definitely "kicks the crap" out of the Dell system your suggesting.
Sorry, I should clarify. I was not talking about how fast gcc compiled source code, but rather how fast the compiled code execute (how fast the compiler actually compiles source is not really relevant for SPEC... unless we're talking about the gcc test, but now we're getting silly).
VC7 and VC7.1 are a big leap forward in this area, and I am not confident that gcc produces faster results. I can tell you though that back when I measured it, gcc was definitely producing faster code than VC6.
Which you would think would be the compiler that Apple and IBM have put the most time into in the last couple of years, right?
Certainly Apple has put more effort into GCC. IBM perhaps, but almost certainly their GCC contributions have primarily been on the x86 and IA64 side of things. In terms of PPC backends, I suspect IBM at the very least has put in much more work into their own compiler.
The real point I'm sure you are trying to get at though is that you suspect that GCC has been optimised much more for PowerPC than x86. Let me assure you that you could not be more wrong. If you recall the days of egcs and pgcc you'll realize that x86 optimisation with gcc has been a strong area of interest for gcc maintainers. Indeed, until VC++7 came out, gcc was one of the fastest x86 compilers out there. The PowerPC backend on the other hand was essentially useless until Apple started working on it. By comparison, IBM's PPC compiler has been used by Big Blue for RS/6000 SPEC benchmarks for over a decade. It also doesn't have gcc's ancient architecture and cross-platform needs, so it is literally light years ahead of gcc.
Saying: "IBM PPC Compiler : gcc PPC backend:: Intel x86 compiler : gcc x86 backend from 1997" is probably a fair statement.
In the end, SPEC is about measuring how fast something can be done in the real world
No, actually SPEC CPU was designed as a benchmark for a CPU (hence the name). It's openly acknowledged that it is influenced by other components in the system, so it is, in practice more of a measure of how all those components peform under high CPU load.
It is actually quite common to control the various variables in the system in order to compare how specific components improve performance (my god! using scientific methods!!!! it's crazy!!!). For example, SPEC is a good way to measure the relative performance of various Intel compilers. It's also helpful for measuring performance of various heap implementations. It's also a very good way to compare chipsets.
Honestly, everyone on this list soaks up similar style benchmarks that Anandtech and Tomshardware do. Why people choose to decide the process in this particular benchmark is flawed is beyond me.
The observation is generally that GCC is optimised for G5 and not sufficiently optimised for Intel.
That would be a horribly misinformed observation then. Ask the people who work on the respective back ends and you'll quickly discover that the reverse is true.
And there is no evidence for a more optimised compiler for the G4 and G5 available to Apple.
Again, quite misinformed. For starters there are Motorola's (for the G4) and IBM's (for the G5) compilers.
Also note that arguably GCC's code generation on Intel has *no* bearing on code generation on PPC. They're different beasts.
That's a bit of an overstatement. True, each back end has processor and architecture specific optimisations available to it, but a large number of GCC's optimisations are platform neutral.
Regardless it's true that despite Apple's efforts, there will always be differences in the compiler backends which could impact the benchmarks. It is fair to say that the choice of GCC was probably the best one (out of a host of poor choices) in terms of providing a "real world" compiler that would employ similar optimisations on both platforms. Again, if anything it hurt the PowerPC benchmarks much more than the x86 ones.
If this is true, then why does Apple compile OS X with GCC?
As I said in the previous post, read the first page of VeriTest's report.
For some strange reason it seems people would rather read some silly Slashdot thread rather than read the relevant material, so I'll quote the report here in order to give some people a clue:
"To be able to directly compare the performance of the hardware systems, the same compiler - GCC, with similar settings were used on both platforms."
You simply can't isolate the CPU from the compiler.
A rediculous assertion. If that were true then you'd have no basis for your claim that the Intel compiler produces faster code than gcc. Apple's benchmark is a pretty good existance proof that one can try to isolate the two.
If I were to lock you in a room with the SPEC benchmarks, whatever compilers you wanted, and two machines that were identical except for one has a 3GHz P4 and the other has a 2.4GHz P4. Would you be unable to determine which CPU was faster, rather than which CPU/compiler combination was faster?
Your solution is to normalize the compiler.
This is not my solution. It's not even a problem. It is merely a statement of fact that this was part of what Apple was trying to do with their benchmark, and that the benchmark is valid in that context.
[..two mysterious binaries scenario here..] Is it impossible to compare performance?
Of course not. However, the performance comparison would be of CPU and compiler (and likely as well a host of other system components). With this limited bit of experimental data, you could not be certain how much of the performance differences would be because of CPU's or the other components in the system. If you didn't have the source code, it'd also be hard to isolate differences there. Indeed, differences in the source code could easily outweigh any other differences.
Gcc is great, I use it all the time, but for performance, I'll use Intel's icc. The reason the Intel compiler isn't used [..host of problems with the Intel compiler..]
You just listed off a ton of reasons (and there are others) why one might want to use the gcc instead of the Intel compiler. Guess what? Those of us in the real world frequently have to address BOTH some of those factors as well as performance. This is why people almost invariably never use the exact configuration that produces the best SPEC benchmarks for their work.
Heck, even when Netscape & Microsoft were fighting like cats and dogs for market share, Netscape was compiling their Windows browser (as well as their Windows servers) using VC++, not the Intel compiler or any other compiler that likely would have given them faster executables. It's a simple concept:
even when peformance is critical, there are other factors which would drive you to pick something other than the choice which gives you the best overall performance
No other vendor does this optimization because it's so lame.
And more importantly, customers (well, those who care to know the details) know that this optimization is not likely something that'd be in place for the particular scenario they are trying to address.
The general guideline for the SPEC benchmark is to try to find a system configuration most like what you would likely use for your needs, and compare it with other systems which meet the same criteria. You should also compare the SPEC benchmark which most closely resembles what you would want to do. So, if you are a Intel/Linux user, the Intel portion of the tests Apple selected are actually likely to be more representative of the how those systems are going to perform for you than the SPEC benchmarks made with Windows 2003 server, either Microsoft's or Intel's compiler, a commercial heap solution, etc.
I can't believe your even arguing this point. A basic notion of science is doing experiments where you control variables in hopes of isolating a particular variable. That's all that is going on here.
The whole notion of CPU benchmarks is to try to isolate the CPU as much as possible. You can choose to ignore the principle, but it invalidates the notion of doing the benchmark at all.
Nobody compiles performance sensitive code with gcc on an x86, they either use Microsoft's compiler or Intel's compiler....
Most Linux kernel's are compiled with gcc. Most apache servers are compiled with gcc. gcc is typically compiled with gcc. Many embedded systems, which have severe performance constraints, are compiled with gcc. Indeed, most of the places I have worked which used C or C++ code for Unix systems ended up compiling it with gcc. In many cases, while code may be performance sensitive, there are other factors which make it preferable to use something other than the absolute fastest compiler out there.
If Apple does not have a similarly powerful compiler...
If you know about Metrowerks, then I'd hope you'd realize that Apple has a variety of options in terms of compilers out there for the PowerPC platform.
Certainly, few people are using gcc for compiling binaries for Windows, as it is not the preferred development tool for that platform. However, throughout much of gcc's history, it has produced more efficient code than the preferred Windows compiler of the moment (for example until VC++7 came out, VC++ was a little less than "fast"), and it developers still did not use it. Actually the vast majority of developers doing performance sensitive work do not use the fastest compiler. Look at how many use VC++ (if it were otherwise, I think you'd find there would only be one professional compiler on the market at a time).;-)
Similarly, microquill's heap is not used by most software developers. (If I had a time for the number of times someone wrote their own custom allocator when then could have used microquill's and gotten faster and bug-free results....;-)
[..list of vendors supposedly not using gcc..]
For the record, Acrobat reader for Linux is compiled with gcc, so is OS X. Sun's JVMs for Linux and OS X are compiled with gcc. I believe Microsoft actually does compile one of their products with gcc. gcc is the dominant compiler on the two platforms in the benchmark: OS X and Linux.
Hell, one of the SPECint CPU benchmarks is gcc compiling itself. Clearly the SPEC people think gcc is used quite a bit.
I am suggesting that Apple's use of gcc is for the sole purpose of handicapping x86 systems for comparison purposes.
Hmm... I guess you missed my point earlier that if anything using gcc penalizes the PPC systems. If you check around you'll discover that gcc's x86 backend has been a much better performer than its PPC backend, and any unreleased modifcations that Apple has done are unlikely to have completely bridged the gap.
Again, the purpose of this test was not to see how fast you can make these machines go. A benchmark is a tool for measuring performance, but it doesn't have to be used strictly to measure the fastest possible performance. Using your argument it would be pointless to perform x86 SPEC_*_rate benchmarks on anything less than Lawrence Livermore's Xeon MCR cluster.
FYI, in the past vendors have used RAM-disks, or RAM-cached RAID's which never had to seek during the benchmark. That has become somewhat out of fashion as vendors have been pressured to demonstrate benchmarks on "real world" systems.
BTW, I believe that the optimizations you mention Sun doing are not legal for the SPEC_*_base benchmarks, which were again expressly introduced to prevent clever compiler optimizations from producing misleading results.
Slashdot Mirror
C++ can get pretty damn these days, but yeah, I'd throw Lisp, Smalltalk, and for totally different reasons, APL into the pool.
Actually, Google is bloating like crazy. If you want "just search", I'd suggest you go for All The Web. They're now owned by Overture, and the site is very focused on the web.
Specifically they run Linux, as I recall.
You should get out more. There are tons of search engines which are quite clear about what are paid ads and what are not. There are tons of search engines which do text only ads. Indeed Overture owns one of them, and it's results are about as relevant as Google's.
Modern C++ Design actually does a good job describing some of the more sophisticated metaprogramming work you can do with C++. It actually demonstrates that you can determine a lot of type information when metaprogramming thanks to C++ template semantics. Really cool stuff.
;-)
Of course it does look ugly as hell.
Modern C++ Design actually does a good job describing some of the more sophisticated metaprogramming work you can do with C++. It actually demonstrates that you can determine a lot of type information when metaprogramming thanks to C++ template semantics. Really cool stuff.
;-)
Of course it does look ugly as hell.
Huh. I've been enjoying Dark Age of Camelot quite a bit. You might want to give it a shot if you are consistently being disappointed.
Why Apple wouldn't work with IBM and Motorolla to make gcc the best compiler possible for PPC I don't know
They are doing exactly that. However, it is quite hard to get gcc well tuned as compared to modern compilers. Keep in mind that Intel has also been contributing to x86, and as we can see their own compiler is much better. The planes for gcc v3.4 and v3.5 are going to dramatically change the way the compiler works, and make it much easier to produce a high performance compiler for all platforms.
Sorry, I should have been more clear: gcc produces faster code than CodeWarrior (which is what we are talking about here). CodeWarrior *compiles* code more quickly.
As anyone who works on gcc which backend receives the majority of optimization effort, and they'll tell you gcc.
Yikes! Wasn't paying attention there. That should read:
"Ask anyone who works on gcc which backend receives the majority of optimization effort, and they'll tell you x86."
even PGCC and EGCS doesn't mean a great deal in terms of current x86 processors
Those projects were folded in to the main gcc project a while ago, but the focus did not die. Indeed, there have been significant contributions from a number of vendors and individuals on tuning the x86 backend since then. The pressure to improve the x86 backend was so strong that the project was actually restructured. As anyone who works on gcc which backend receives the majority of optimization effort, and they'll tell you gcc.
Apple could've done just fine by using Dell's already published SPEC results instead of retesting.
No they couldn't have. There were too many variables between Dell's configuration and whatever optimized configuration Apple used. Dell's numbers demonstrate how wide a performance variance you can get simply by using different software.
If Apple had used an "optimized" configuration and achieved numbers 2x Dell's, this thread would instead be about how the numbers are BS and meaningless because Apple was using software tricks to make their hardware look better. Indeed, that has been the traditional feedback that Apple has received (in many cases, correctly) for previous benchmarks they've published.
The notion of controlling variables in an experiment is a really basic one, and it really disturbs me how people don't seem to be able to grasp this basic scientific concept.
When did IBM say the Power4 was slower
;-)
Take a look at any and all of IBM's literature on the performance of the PPC 970.
The spec int scores of the power4 chip are lower than the p4 scores
You are missing the point. At no point did I compare the scores of the Power4 system to the scores of the P4. My point is not that the Power4 is faster than the P4. In fact, if you were trying, you couldn't have come up with a greater misunderstanding of the point.
If Apple wants to prove that they have a superior system....
This is where I think you are missing the point. Apple is not trying to prove they have a superior system. Well, actually they are, but they *aren't* trying to prove that they have a faster system. With the SPEC benchmarks they published, they are trying to prove that they have a faster processor.
[Apple are goint to need to] put out a benchmark showing that they are actually faster than Intel machines.
For the system performance comparisons, they chose to use application benchmarks. They did about 4 different benchmarks as I recall, and to use your parlance, they "totally kicked the crap" out of the Intel machines.
Of course, I'm sure that as far as you are concerned, so long as you can take two numbers from benchmarks with totally different controlled conditions that show the Intel machine winning, it really doesn't matter how many other reasonable benchmarks suggest the other way around. We all know the Intel machine is faster, and it's just a matter of finding the one set of test data that proves it!
Sounds like a pretty broken compiler to me, unless that was padding code which was never executed. I've never seen anything like that even with optimisations turned off, even with ancient versions of gcc.
The Power4 may be on a different mobo and use different memory. It is possible that 'z' is faster then 'y'.
Okay, now I know you are trolling. IBM made both chips, presumably this makes them pretty good experts on which is faster.
In any event, both 'y' and 'z' are much slower than 'x' in any measurement ever performed.
Cool! You get bonus troll points! On a thread that was spawned by a benchmark which showed the PPC 970 beat the P4! You're the man!
Let's try algebra, as prose does not seem to be working: Based on your reasoning above, z is faster than y, but we know that z is slower than y. This is a negative proof demonstrating the flaw in your reasoning.
To restate: the I am mentioning the Power4 CPU to demonstrate that your statement that the P4 has published benchmarks which "kick the crap" out of the numbers in Apple's benchmark's of the G5 is essentially meaningless, and says little about the relative performance of the P4 vs. the G5.
Unfortunately that really only proves that those results used a better compiler/heap/etc. This is a really simple concept folks. I'm sure if they had used a nice compiler along with all kinds of crazy compiler, OS and system optimizations, an even larger group of people would be saying that Apple was misrepresenting the benchmarks.
I'm going to say this once again: the Apple benchmark chose software consistency over potential performance winds in an attempt to isolate the performance benefits of the hardware.
It's worth noting that IBM's own 1500MHz Power4 CPU "kicks the crap" out of the published results for the G5. Guess what? The Power4 processor is the slower processor. IBM's own SPEC benchmarks show that the G5 at 1.8GHz should "kick the crap" out of that 1500MHz CPU, let alone a 2GHz model.
For the record, IBM's own SPEC tests for the 970 put it at 937/1051 when running at 1.8Ghz. Extrapolating this would suggest that the 970 using an "optimised" test bed, would score slightly below those ICC results for integer, while beating it for floating point. So amazingly, this performance differences demonstrated by Apple's benchmarks actually correspond to performance differences that exist in the case where both sides are using an "optimised" test bed.
Sadly, I haven't seen any 970 SPEC_*_rate benchmarks using IBM's compiler yet, but given that those benchmarks are much more influenced by motherboard design, it's quite possible those numbers would be misleading anyway. That being said, for the SPEC_*_rate benchmarks, IBM's 1500MHz Power4 system definitely "kicks the crap" out of the Dell system your suggesting.
Sorry, I should clarify. I was not talking about how fast gcc compiled source code, but rather how fast the compiled code execute (how fast the compiler actually compiles source is not really relevant for SPEC... unless we're talking about the gcc test, but now we're getting silly).
VC7 and VC7.1 are a big leap forward in this area, and I am not confident that gcc produces faster results. I can tell you though that back when I measured it, gcc was definitely producing faster code than VC6.
Which you would think would be the compiler that Apple and IBM have put the most time into in the last couple of years, right?
:: Intel x86 compiler : gcc x86 backend from 1997" is probably a fair statement.
Certainly Apple has put more effort into GCC. IBM perhaps, but almost certainly their GCC contributions have primarily been on the x86 and IA64 side of things. In terms of PPC backends, I suspect IBM at the very least has put in much more work into their own compiler.
The real point I'm sure you are trying to get at though is that you suspect that GCC has been optimised much more for PowerPC than x86. Let me assure you that you could not be more wrong. If you recall the days of egcs and pgcc you'll realize that x86 optimisation with gcc has been a strong area of interest for gcc maintainers. Indeed, until VC++7 came out, gcc was one of the fastest x86 compilers out there. The PowerPC backend on the other hand was essentially useless until Apple started working on it. By comparison, IBM's PPC compiler has been used by Big Blue for RS/6000 SPEC benchmarks for over a decade. It also doesn't have gcc's ancient architecture and cross-platform needs, so it is literally light years ahead of gcc.
Saying: "IBM PPC Compiler : gcc PPC backend
In the end, SPEC is about measuring how fast something can be done in the real world
No, actually SPEC CPU was designed as a benchmark for a CPU (hence the name). It's openly acknowledged that it is influenced by other components in the system, so it is, in practice more of a measure of how all those components peform under high CPU load.
It is actually quite common to control the various variables in the system in order to compare how specific components improve performance (my god! using scientific methods!!!! it's crazy!!!). For example, SPEC is a good way to measure the relative performance of various Intel compilers. It's also helpful for measuring performance of various heap implementations. It's also a very good way to compare chipsets.
Honestly, everyone on this list soaks up similar style benchmarks that Anandtech and Tomshardware do. Why people choose to decide the process in this particular benchmark is flawed is beyond me.
Actually, the Apple compiler is faster than Codewarrior. IBM's PowerPC compiler is much faster than both of them though.
The observation is generally that GCC is optimised for G5 and not sufficiently optimised for Intel.
That would be a horribly misinformed observation then. Ask the people who work on the respective back ends and you'll quickly discover that the reverse is true.
And there is no evidence for a more optimised compiler for the G4 and G5 available to Apple.
Again, quite misinformed. For starters there are Motorola's (for the G4) and IBM's (for the G5) compilers.
Also note that arguably GCC's code generation on Intel has *no* bearing on code generation on PPC. They're different beasts.
That's a bit of an overstatement. True, each back end has processor and architecture specific optimisations available to it, but a large number of GCC's optimisations are platform neutral.
Regardless it's true that despite Apple's efforts, there will always be differences in the compiler backends which could impact the benchmarks. It is fair to say that the choice of GCC was probably the best one (out of a host of poor choices) in terms of providing a "real world" compiler that would employ similar optimisations on both platforms. Again, if anything it hurt the PowerPC benchmarks much more than the x86 ones.
If this is true, then why does Apple compile OS X with GCC?
As I said in the previous post, read the first page of VeriTest's report.
For some strange reason it seems people would rather read some silly Slashdot thread rather than read the relevant material, so I'll quote the report here in order to give some people a clue:
"To be able to directly compare the performance of the hardware systems, the same compiler - GCC, with similar settings were used on both platforms."
You simply can't isolate the CPU from the compiler.
A rediculous assertion. If that were true then you'd have no basis for your claim that the Intel compiler produces faster code than gcc. Apple's benchmark is a pretty good existance proof that one can try to isolate the two.
If I were to lock you in a room with the SPEC benchmarks, whatever compilers you wanted, and two machines that were identical except for one has a 3GHz P4 and the other has a 2.4GHz P4. Would you be unable to determine which CPU was faster, rather than which CPU/compiler combination was faster?
Your solution is to normalize the compiler.
This is not my solution. It's not even a problem. It is merely a statement of fact that this was part of what Apple was trying to do with their benchmark, and that the benchmark is valid in that context.
[..two mysterious binaries scenario here..] Is it impossible to compare performance?
Of course not. However, the performance comparison would be of CPU and compiler (and likely as well a host of other system components). With this limited bit of experimental data, you could not be certain how much of the performance differences would be because of CPU's or the other components in the system. If you didn't have the source code, it'd also be hard to isolate differences there. Indeed, differences in the source code could easily outweigh any other differences.
Gcc is great, I use it all the time, but for performance, I'll use Intel's icc. The reason the Intel compiler isn't used [..host of problems with the Intel compiler..]
You just listed off a ton of reasons (and there are others) why one might want to use the gcc instead of the Intel compiler. Guess what? Those of us in the real world frequently have to address BOTH some of those factors as well as performance. This is why people almost invariably never use the exact configuration that produces the best SPEC benchmarks for their work.
Heck, even when Netscape & Microsoft were fighting like cats and dogs for market share, Netscape was compiling their Windows browser (as well as their Windows servers) using VC++, not the Intel compiler or any other compiler that likely would have given them faster executables. It's a simple concept:
even when peformance is critical, there are other factors which would drive you to pick something other than the choice which gives you the best overall performance
No other vendor does this optimization because it's so lame.
And more importantly, customers (well, those who care to know the details) know that this optimization is not likely something that'd be in place for the particular scenario they are trying to address.
The general guideline for the SPEC benchmark is to try to find a system configuration most like what you would likely use for your needs, and compare it with other systems which meet the same criteria. You should also compare the SPEC benchmark which most closely resembles what you would want to do. So, if you are a Intel/Linux user, the Intel portion of the tests Apple selected are actually likely to be more representative of the how those systems are going to perform for you than the SPEC benchmarks made with Windows 2003 server, either Microsoft's or Intel's compiler, a commercial heap solution, etc.
I can't believe your even arguing this point. A basic notion of science is doing experiments where you control variables in hopes of isolating a particular variable. That's all that is going on here.
I disagree that you should keep elements similar.
;-)
;-)
The whole notion of CPU benchmarks is to try to isolate the CPU as much as possible. You can choose to ignore the principle, but it invalidates the notion of doing the benchmark at all.
Nobody compiles performance sensitive code with gcc on an x86, they either use Microsoft's compiler or Intel's compiler....
Most Linux kernel's are compiled with gcc. Most apache servers are compiled with gcc. gcc is typically compiled with gcc. Many embedded systems, which have severe performance constraints, are compiled with gcc. Indeed, most of the places I have worked which used C or C++ code for Unix systems ended up compiling it with gcc. In many cases, while code may be performance sensitive, there are other factors which make it preferable to use something other than the absolute fastest compiler out there.
If Apple does not have a similarly powerful compiler...
If you know about Metrowerks, then I'd hope you'd realize that Apple has a variety of options in terms of compilers out there for the PowerPC platform.
Certainly, few people are using gcc for compiling binaries for Windows, as it is not the preferred development tool for that platform. However, throughout much of gcc's history, it has produced more efficient code than the preferred Windows compiler of the moment (for example until VC++7 came out, VC++ was a little less than "fast"), and it developers still did not use it. Actually the vast majority of developers doing performance sensitive work do not use the fastest compiler. Look at how many use VC++ (if it were otherwise, I think you'd find there would only be one professional compiler on the market at a time).
Similarly, microquill's heap is not used by most software developers. (If I had a time for the number of times someone wrote their own custom allocator when then could have used microquill's and gotten faster and bug-free results....
[..list of vendors supposedly not using gcc..]
For the record, Acrobat reader for Linux is compiled with gcc, so is OS X. Sun's JVMs for Linux and OS X are compiled with gcc. I believe Microsoft actually does compile one of their products with gcc. gcc is the dominant compiler on the two platforms in the benchmark: OS X and Linux.
Hell, one of the SPECint CPU benchmarks is gcc compiling itself. Clearly the SPEC people think gcc is used quite a bit.
I am suggesting that Apple's use of gcc is for the sole purpose of handicapping x86 systems for comparison purposes.
Hmm... I guess you missed my point earlier that if anything using gcc penalizes the PPC systems. If you check around you'll discover that gcc's x86 backend has been a much better performer than its PPC backend, and any unreleased modifcations that Apple has done are unlikely to have completely bridged the gap.
Again, the purpose of this test was not to see how fast you can make these machines go. A benchmark is a tool for measuring performance, but it doesn't have to be used strictly to measure the fastest possible performance. Using your argument it would be pointless to perform x86 SPEC_*_rate benchmarks on anything less than Lawrence Livermore's Xeon MCR cluster.
FYI, in the past vendors have used RAM-disks, or RAM-cached RAID's which never had to seek during the benchmark. That has become somewhat out of fashion as vendors have been pressured to demonstrate benchmarks on "real world" systems.
BTW, I believe that the optimizations you mention Sun doing are not legal for the SPEC_*_base benchmarks, which were again expressly introduced to prevent clever compiler optimizations from producing misleading results.