Slashdot Mirror
Apple's G5 Speeds Challenged
An anonymous reader was the first of a seemingly infinite stream of people to submit a URL to an argument that makes the case that the G5 isn't quite what Apple wants you to think of it. The evidence? Apple's own press material. Worth a read.
...and benchmark different too!
Is this truly the only Earth I can live on?
Sheesh, evil *and* a jerk. -- Jade
translation
i am too stupid to round up.
Are you MORE than your SPINAL COLUMN?
Apple is always a little sketchy when it comes to speed measurements. I can't count how many questionable run-offs Steve Jobs has demonstrated during his keynotes.
They're always a little suspect. I love Apple as much as anyone, but their talk of the megahertz myth and the amazing clock cycle of the G4/G5 and the biased tests they use are starting to sound a little shrill. Apple needs to admit that their machines aren't as fast as the fastest Intel has to offer. They're much cleaner and much more elegant, though, and that's why they're in the market. That's what they should stress, since it actually attracts customers -- rather than THE NEED FOR SPEED.
Well, it certainly isn't the first time that a company has used a benchmark to make a product look better than it is, and it certainly won't be the last time. I think what we should all learn from this is as follows. Don't worry about Statistics, Benchmarks, or any Media Hype. Just go to the store, buy whatever kind of computer you want that floats ur boat, Be it a Mac, Linux Box, Windoze Box, or god forbid, a compaq. Set it up, get broadband internet, and read lots of Slashdot and play Starcraft.
I have no regrets, this is the only path.
My whole life has been "UNLIMITED BLADE WORKS"
I mean computers are so fast that there's very little that I might want to do at a consumer level that makes a difference. Most applications are responsive on my ancient 500MHz Pentium 3.
The only things that really need speed are things like 3d rendering, video compression and compiling large appllications. 3D rendering in games is influenced by the speed of the graphics card a lot more than the speed of the CPU, so we're left with the long slow scenes. Personally, it makes very little difference to me if a rendering a scene or compressing a video takes 30 minutes rather than 40. If I can kill 30 minutes, I can kill another 10 quite easily.
In the past, I'd have been able to tell you whether I was using a 20MHz or a 25Mhz 386 just by using it. I can hardly detect the difference between a 1.5GHz machine and a 3.0GHz machine without using a benchmark.
In the end, it's just numbers.
I watched the video. (http://stream.apple.akadns.net/ - requires QuickTime). Now, I'm sure there's many ways you could tweak the benchmarks and so forth but the Photoshop and Mathematica benchmarks rocked. The G5 was 2x faster than the Xeon.
I used to get involved doing benchmarking back in the good old days of Whetstone when I worked on supercomputers. Every manufacturer had a different nasty tweak to the compilers that were pulled out only when it was time to do benchmarks for a customer. The mantra then as now was: the best benchmark is the app you want to run (since most buyers of supercomputers write their own apps, porting them for a benchmark was a possibility).
The G5's may not be the hottest thing on the planet but they're close enough to get Apple back in the ball game. Nice systems architecture, nice case and the claim is they're quiet as well. Oh, and don't forget you can put in 8GB of RAM. Now even OS X doesn't need to swap :-)
And what's more, when you start running programs that use more than 2GB of data, the 64-bit machine is going to beat the pants off the 32-bit one, since the 32-bit machine (i.e. intel) is going to have to resort to slow and hacky solutions such as segments and paging. The intel may me "faster" but only as long as 32-bit are enough for you. The days of 32-bit machines are numbered, just as they were for 16-bit machines when 32-bit machines started to appear.
Stick Men
I was going to mod you down, but...
Apple has deliberately turned off processor features on the other platforms that would have led to their 'fastest in the world' claim being untrue. That's the point of the article. Cross-platform benchmarking IS hard, but deliberately crippling what you benchmark against in order to look better makes it seem that your software/hardware/whatever just isn't as good as what you're comparing it to...
Game dev and music blog
The 1GHz backplane is the real news. No processor benchmark test really takes into account the total real speed of the system when running applications.
The fast backplane will speed up IO, which is a common bottleneck. 1GHz for a PC backplane is huge. The only machine I had seen a 1GHz backplane in so far is a HP-UX server. It cost wayyy more than $2000 or even $3000.
I really believe that with this new chip alliance with IBM Apple will finally be able to put that "the OS is really cool, but PCs are always faster" stuff behind them.
Yesterday was a good day for apple.
If you're talking about this (section entitled "Apple Copies Ideas From Microsoft") then you'll find that he admits that Microsoft copies stuff from Apple, but that Apple have copied things from Microsoft too. Which wouldn't seem a too unreasonable claim.
If you're going to claim someone is a troll, the least you could do is give us an example which isn't guaranteed to mislead us.
Avantslash - View Slashdot cleanly on your mobile phone.
Are you going to deny that Apple cheated at the benchmarks by disabling various optimizations on the competition? Are you going to deny that most software uses integer math, as one "software coder" clearly did (hint: i write a lot of software, and integer math practically always dominates)?
The guy may, or may not be a troll. However, the sheer amount hate mail, and the level of it, was stunning. What kind of people write stuff like that? Very few of them even attempted to address the guys points, and those that did made a hash job of it (nobody uses int math? wtf?).
The fact is that anybody outside the Mac community, having read that essay, is going to come away with a bad impression of said community. Nobody deserves to get hate mail like that for pointing out the other side of the statistics.
Anyone who's followed the computer industry for more than a couple minutes knows that there are lies, damned lies, and benchmarks.
Go use a machine, for tasks you'd typically perform -- that's the only benchmark that matters.
But if you must assign a number to the size of your virtual phallus, by all means, benchmark away...
While the article linked does indeed make it look somewhat shady, it's worth pointing out that a major weakness of his argument is that he implies (credibly) that this lab test commissioned by Apple is not trustworthy, and then compares it with tests by Dell and Intel, which he seems to present as implicitly true. How do we know that's the case? If Apple did indeed gain anything by mucking about with the configurations (and it sounds like they did), who's to say that they did anything more than offset similar mucking about on the other side of the fence?
I can vouch for those unfortunate enough to have worked in the grocery industry (and have an idea of what that kind of mechandising entail) but this is hardly insightful. It happens on nearly everything that you buy.
When asking the pricing managers (which work for the chain, not an individual store) they replied that there was a study once done, indicating that there is a psychological tendancy to shy away from certain "maker" numbers as being too big. For example, the masses statistically believed that twenty dollars was too much to pay for item x, but for some reason, nineteen ninety-nine was not too much to pay for the same item. Funny thing is that with the same item, eighteen dollars would again be too much, but seventeen ninety-five wouldn't.
Even if the study is flawed or bogus, it is still being taught in the "front-line" marketing schools, (ie. grocery, drug-store, clothing, etc.) , and so I expect we will see nineteen ninety-five for many many years to come.
Well.. you can have my 8-bit commodore 64 for free.. so the bang for the buck is infinite..
I had a desktop UNIX (Solaris) running on my desktop 64-bit hardware (Sun Blade 100) a couple of years ago.
Yeah, me too. But unlike the Mac, I could not run Office, Photoshop, function as a web server, surf the web, compile code, run bioinformatics searches, do molecular modeling and have wonderful text aliasing all at the same time. Now with OS X, I can do all this and network seemlessly with Wintel and UNIX machines while maintaining my sanity by only having one software library to keep up with and have one system on my desk instead of three. Oh, and when I am on the road (like now on the other side of the country), I can take all of this with me by using a Powerbook.
No other company has been able to give me these tools, and for that.....I have to say, "Thank you Apple Computer".
Visit Jonesblog and say hello.
Believe it or not, some of us want both a desktop UNIX on 64 bit hardware AND legitimate benchmarks. I don't see why one excuses the other.
Listen, Apple made a good product because they needed to stay in business. They didn't do it out of the good of their hearts. And their good product in no way changes the fact that I don't appreciate being lied to by corporations.
Don't get me wrong, this is not the world's biggest lie or corporate misdeed. I don't put much faith in benchmarks anyway, and I wouldn't make my decision between a Mac or a PC based on them (although for others the specs might be more important). But it's still sleazy. And it's very unfair to act like it's "ungrateful" or "trollish" to demand that Apple set up legitimate benchmarking tests.
Misleading Prices
Both Apple and Dell are guilty of using misleading prices. For example, Apple gives the price of the low-end G5 as "$1999", and the high-end G5 as "$2999". In other words, they have subtracted $1 from a $3000 computer to make it seem cheaper, which is absolutely ridiculous. This demonstrates that both Apple and Dell are willing to mislead people when stating their prices.
Next crackpot, please.
The G5 benches were provided by Apple, they optimised it as much as they could.
The Dell/Intel benches were provided by Dell/Intel, they optimised them as much as they could.
However, what he didn't include was benchmarks for a G5 which had been crippled by Dell and Intel..
I usually hate analogies, but sometimes it's my only way of getting my point across:
If Ford tweaks their engines and suspension set up before a test. OK!
If Nissan tweaks their engines and suspension set up before a test. OK!
If Nissan tweaks their engines and suspension set up, and pours sugar in the Ford's "gas" tank before the test. NOT OK!
A. Benchmarking is a black art, and benchmark results more often than not bear little or no relation to reality (i.e. the actual performance you will get, today, running your particular workload). Talk to anyone who does it for a living and they are the first to admit that.
B. Benchmarks are very rarely impartial. Whoever is footing the not inconsiderable bill for a properly-done benchmark will have a result they want to see, and the benchmarkers can do a lot to make sure they do see it.
C. "Perception is reality" is a well-known saying in marketing. It doesn't actually matter whether the perception is correct. If Joe Sixpack believes he has bought the fastest PC in the world, he will be happy. More so since he most likely has nothing on hand to compare it to.
D. The speed this industry moves at, there will be a faster one along in a month or less, so if you really want something faster, wait for it.
E. All this debating about which is faster is more like masturbating. And "Masturbation, although an inherently pointless way to pass time, is at least enjoyable. Comparing PC performance is equally pointless, but rather less fun. The conventional epithet applied to those who engage in the former to excess is equally applicable to those who persist in the latter."
The guy may, or may not be a troll. However, the sheer amount hate mail, and the level of it, was stunning. What kind of people write stuff like that? Very few of them even attempted to address the guys points, and those that did made a hash job of it (nobody uses int math? wtf?)
Did you notice how almost all of the hatemail was addressing him in the third person?
He went onto a discussion board somewhere about the post (probably MacNN, probably one of the worst reputation Mac websites in terms of brainpower) and just cherry picked the comments he could take apart easily.
It's not like he actually *got* that hatemail. He didn't even post an email address with the article.
Isn't it funny how you can bend things to make you look favorable - just like Apple may have done?
OK, I mostly skimmed the article, but he's among other things complaining that they turned off SSE2. May I mention that from what I could gather, the benchmarks used on the Apple platforms had NO Altivec optimisation? With that in mind it seems that disabling SSE2 was simply done to level the field. If there had been Altivec optimisations, then for comparison's sake it would make much better sense to use Altivec and SSE2. Actually, they might have chosen to disable SSE too, but they didn't!
The other feature he's complaining about is the disabling of hyperthreading. From other benchmarks I've seen before, hyperthreading in SMP systems usually results in equal or slower performance, or at most a 10% addition in certain benchmarks. It was probably better to leave it off.
Finally, about the discrepancy between Veritest's/Dell's/Intel's benchmarks, this is to be expected. Veritest compiled the benchmarks with GCC 3.3, and certainly used different compiling options and different testing options than Dell used. Unless you use the same options and methodology on every test, comparing benchmarks is useless.
I'm not saying Veritest and Apple didn't do their best to look good, of course they did! But at least you have to give them credit for going with an independent firm with a full report (where everything is laid out), instead of absurd and evidently fabricated application benchmarks like they've done in the past.
As another poster mentioned, benchmarking is HARD, and harder across platforms, especially on a new CPU platform with no optimizations and no way to use some of the CPU features. When we get a benchmark version that allows for full use of al the features of the 970 (G5) and the x86 CPUs, then we might get a clearer picture. It also doesn't remove the fact that these machines are MUCH, MUCH better than the G4s, or that Apple also promised the processor would scale to at least 3GHz within a year.
Oh of course one of his arguments about his righteousness is "Look at all these Mac fanatics who flame me". He's not much better than them, from what I can see. One fanatic from one camp doesn't make all of them fanatics, and doesn't validate his points one iota (neither does flaming him destroy his points, which is why intelligent rebuttal would be better, but I have the feeling he would most likely not publish that).
If he explores all forms and substances Straight homeward to their symbol-essences; He shall not die.
There is no saying which is right, and I don't think this guy was really trying to. If you read his writeup, he says that Apple claims a certain Dell model benchmarks at value $X, while Dell claims that the same model can do $Y.
He doesn't actually say that one or the other is correct -- he says that the most charitable thing you can do is split the difference and go with the average -- and the kicker is that even that midway point is higher than what Apple claims for the G5.
You've got a good point, but I think this guy is aware of it as much as you are. He's not saying that each vendor's analysis is authoritative, but that the truth probably lies somewhere in the middle, and that middle ground might or might not look to be in Apple's favor (in fact, it doesn't seem to be in Apple's favor).
DO NOT LEAVE IT IS NOT REAL
-Lucas
At work, we just bechmarked the Dell systems a month ago and got very similar results to Apple for the "base" rate. The article seems to be quoting the "peak" rate for the Dells. It's not valid to compare peak rates yet because gcc 3.3 and os 10.3 aren't really fully optimized yet.
The article also complains that using the NAGWare compilers is not a valid test since they're too slow. But I think the NAGWare compiler is a more vallid comparison than intel's compiler because most real-world computing is done with NAGWare because it fully implements the F95 spec and is more portable. In addition NAGWare is well tested for accuracy and it also very much cheaper.
The Dell benchmark numbers are pure fantasy. They never occur in real-world use.
There are 10 types of people in this world, those who can count in binary and those who can't.
I just read the veritest document myself. This guy did not show the complete picture at all. First, in favour of Apple, these tests were run on Mac OS X 10.2.7. I don't think this is the 64-bit kernel that can really take advantage of the G5. I'm not a mac head, but I'm guessing this is Smeagol?
So right from the start the G5 is seriously crippled in these tests. Especially if they don't even take advantage of 64-bit as seems to be the case. Now, on the the other points that the spl dude makes.
Sure the special malloc library seems a bit unfair. But then again, do these tests really focus on memory allocation? I'd think they are limited by CPU power more than memory allocation. In any case, it'd be nice if we saw results without this library.
Now for the comfusing part. The 8300 only has a single cpu. For the base tests, they use hyperthreading and an SMP kernel. They do the exact same test for the 650's base test, hyperthreading with a single processor and an SMP kernel. The G5 system is run with a single processor as well.
So it seems this test is well balanced and fair. The confusing part is the rate tests.
For the 8300 they have no hyperthreading and a uniprocessor kernel. For the 650 they have no hyperthreading and an SMP kernel with two processors. The G5 system is run with two processors. It's unclear why they chose not to use hyperthreading on the rate test. It could be that hyperthreading actually reduced the scores of these tests. I'm no expert on the SPEC tests and hyperthreading, but what I do know is that hyperthreading is an intelligent technology. It can't always increase speed, it depends on what kind of code it's running. In the rate test it's possible that hyperthreading is unable to yeild any improvements, in which case the overhead of enabling hyperthreading may make the scores worse than without hyperthreading.
At anyrate, the tests were a LOT more fair than the dpl guy makes them out. And considering that the G5 could be seriously crippled by not running 64-bit and who knows what other optimisatoins, I'd say that the numbers are still impressive.
The author claims the test is biased mostly because:
1. On x86 hyperthreading was disabled
2. on x86 SSE was disabled
3. on PPC a custom malloc was used
4. on PPC a different set of optimizations were used
1. I admit is seems odd that this was disabled. I think it's effect would be little, but it should be turned on
2. So was the PPC's AltiVec. I recall that SPEC wants FP and INT performance from the ALU sections, not SIMD
3. And I'm sure that there are many "tweaks" for x86 that are transparent within the GCC 3.3 code generators
4. Again, each CPU has different optimizations, either allow them all or disable them all - on both platforms, command line switched or embedded
What I think would be interesting for Apple to do to help settle all this (You know, spread around some of that $4B+ they have lying around):
Purchase two of the fastest model of 1st tier systems they can get that run on x86.
Using four different testing labs, send one machine to each lab (2 x86, 2 G5). Instruct each lab to perform any software/configuration optimizations they feel necessary to get the most performance out of the machine. Then they run a standardized set of benchmarks. They each fully document the changes they've made and the results.
Apple (or perhaps a 5th lab) colates the data and produces a final result.
Or some open source minded person with some extra bandwidth(ha) could create a web site where PCers and Macers could post their own results from the benchmarks. With sufficient results posted, the "noise" would get filtered out and the results would become statistically useful.
Article X: The powers not delegated... by the Constitution...are reserved...to the people