Gigahertz Mac Finally SPEC'd

FrkyD writes "C't magazine puplished a story with the results of a test they designed using a Mac OS X-adapted benchmark suite by the Standard Performance Evaluation Corporation (SPEC) entitled CPU2000. SPEC allows comparisons to be made within a certain framework with the Intel competition. They compared the G4/1 GHz running Mac OS X with a PIII/1 GHz (Coppermine) running Windows and Linux."

Comments

So. they are fast, but not WHOA fast....

[Toraz Chryx]

gee, who'd have thought...

Now, where's the PowerPC chips made on IBM's new process and running at 40Ghz?... :)

Re:So. they are fast, but not WHOA fast....

[easter1916]

That'd be the copper chips first used in the eServer iSeries (the machine formerly known as the AS/400).

It all comes down to application

[Strog]

Until a more optimized compiler comes along then it looks like for general purpose applications clock speed will mostly be the indicator of performance.

Having said that, there will always be applications that are optimized enough to kick some butt on a G4 like Photoshop, etc. If you are a programmer then it is nice to not be limited on registers on a RISC cpu. Choose the right tool for the right job. If it comes down to a push then use your favorite. :P

Re:It all comes down to application

[mr100percent]

Unfortunately SPEC is optimized for Intel processors, and is NOT written to even see the Altivec. Plus, it doesn't even see the G4's extra pipelines or registers. In short, it's not coded to see any advantage of the G4, while any other "Hello World" program compiled to PPC would run laps around the SPEC app.

Linux vs. Windows

[crow]

I found this from the article to be interesting:

With a SPECint_base value of 306 Apple's 1 GHz machine under Mac OS X ran almost head to head with the equally clocked Pentium III, combined with Linux and GCC, with a SPECint_base value of 309. Under Windows, the bad quality of Microsoft's run-of-the-mill compiler, which pushed the system down to a SPECint_base value of 236

That means Linux is over 30% faster than Windows!

Too bad they didn't give similar floating point numbers (or at least I didn't find them in the article), especially seeing as how the Mac is faring so poorly against the Linux PIII in that area.

Re:Linux vs. Windows

[foobar104]

That means Linux is over 30% faster than Windows!

Of course it doesn't. It means that GCC is somewhat better at compiling the SPECint_base benchmark than Visual Studio is.

I won't pretend to be educated about the inner workings of SPECint, but one would suppose that, because it's purported to be a hardware benchmark rather than an OS benchmark, it is completely independent of the standard C library, or any other OS-level service. One would expect the compiled benchmark to just run pure code inside the CPU, without any system calls or any of that stuff.

So the same benchmark compiled with the same compiler but run under two different OSs should return exactly the same result, within a certain statistical margin.

Somebody with more time on their hands could either test this hypothesis, or confirm that it's already been done by somebody else.

Re:Linux vs. Windows

[leviramsey]

Of course, since everything on a standard Linux machine (kernel, libraries, X, etc.) is compiled with GCC, while I imagine that everything that Windows runs is compiled with Visual Studio's compiler, if GCC consistently turns out code that's faster than Visual Studio, we could extrapolate that Linux is significantly faster than Windows.

Re:Linux vs. Windows

[RevAaron]

we could extrapolate that Linux is significantly faster than Windows.

No, you couldn't. Writing code in C is generally faster than writing it in Java. However, surely you've seen a C program do something slower than a Java program. For example, starting up Nautilus compared to running a hello world program in Java. While this example is taken to an extreme, it still holds true- there is a big difference in the way hello world in Java and Nautilus does things, the same with Linux and Windows, and the speed of the C compiler says absolutely nothing about the amount of abstraction, the amount of bloat, the amount of optimization, the sheer amount of code or the quality of code written as a part of neither Linux nor Windows. Among plenty of other factors.

Re:Linux vs. Windows

[Anonymous Coward]

Air Jordan shoes can make you jump higher than wearing your Dutch granny's wooden clogs, generally. But if you give the clogs to Mr. Jordan, and put the Nikes on yourself, it doesn't mean you're going to be able to outjump him, does it?

Re:Linux vs. Windows

[foobar104]

[I]f GCC consistently turns out code that's faster than Visual Studio, we could extrapolate that Linux is significantly faster than Windows.

Maybe you could, if GCC could be thus characterized. But there's no evidence in this article that points to that conclusion. Rather, this article says that GCC did a better job of compiling the SPEC benchmarks. As everybody knows-- or should know-- benchmarks are to real applications as fish are to bicycles.

Re:Linux vs. Windows

[Anonymous Coward]

I think the SPEC benchmarks are held in high regard because they ARE real world applications. GCC, GZIP etc...In the past people used code fragments and artifical loops and what not but SPEC uses applications we typically (at least in the UNIX world) use everyday.

They're a good representation of the types of tasks people do with workstations -- file compression, compilation etc. It may not be completly fair for Microsoft since they use different tools than we use. WinZip instead of GZip(?), but I think most of the underlying algorithms are the same and unless we all use the same code base it's hard to make a fair comparison.

Re:Linux vs. Windows

[Graymalkin]

Actually it means GCC is 30% faster than Visual Studio's compiler which is notoriously shitty. You're also basing your aguement on too few details. You don't know which compiler flags were used so you can't compare -O3 optimized code to VS optimized code. The VS compiler is not the world's greatest compiler and I think they should have gone with the Borland C++ compiler for Windows. Your compiler makes a big difference in the speed at which code is going to run.

Snide comments on "supercomputer" show bias

[jdb8167]

I know people are going to claim that the SPEC marks aren't susceptible to bias but the SPEC suite only test traditional architectures. As far as I know, they don't test for SIMD vector processing like the altivec.

No one ever claimed that the FP alone on the G4 was at supercomputer status, just that the G4 in conjunction with Altivec could crunch at FLOPs at "supercomputer" speeds.

Keep in mind that OS X is hardly optimized for this kind of test. OS X has just recently reached the point where it is useful as a general purpose platform. But Apple is making a big push in the scientific computing area so I expect that you will find vast improvements in the SPEC FP suite in the future.

Re:Snide comments on "supercomputer" show bias

[morbid]

Yes, but PIII has SSE which is very similar to altivec. AFAIK there aren't many compilers out there that can do vectorisation, and GCC 3.0.x doesn't do it, so I'm working on this:
libSIMD
I haven't done any SSE or AltiVec yet, only 3DNow!.

Re:Snide comments on "supercomputer" show bias

[Matthias Wiesmann]

As far as I understand the problem, altivec is very performant but only handles single precision floats, not doubles.

While single precision floats are largely enough for multi-media processing (filters, compression, etc...), in general, number crunching is done in double precision and the floating point tests of SEPC reflect this. You don't always need double for scientific calculations, but this is altoghter another discussion.

Maybe one day we'll see a multimedia component of SPEC or Altivec will support double precision numbers (the author even mentions this at the end of the article) but until then Altivec is out and this has nothing to do with a bias of the author.

As for OS X being optimised for this kind of stuff, we are talking applications that nearly never call the OS for anything, so the impact of OS X is probably nil. The truth is, floating point calculation is not really important for most users and both Intel and PowerPC processors are optimised for integer calculations. There was a good article about this on Ars Technica.

One reason I could see to explain the large difference lies in the compiler: there has been much more work on gcc to optimise for the Intel instruction set than for the PPC instruction set. Like most RISC processors, the performance of a PPC processor is hugely influenced by the compiler.

Re:Snide comments on "supercomputer" show bias

[Matthias Wiesmann]

Actually, there seem to be a variant of gcc that supports altivec

Re:Snide comments on "supercomputer" show bias

[morbid]

Hmmm... it doesn't look all that advanced, and it's not that much different to what I've been doing in my library. Never mind, as time passes, I'm sure there will be "seamless" vectorisation built into gcc.

Re:Snide comments on "supercomputer" show bias

[jdb8167]

Are you sure about the OS not having much impact on the test? I've read that there is a floating point library in Mac OS X that is significantly slower than the equivalent in OS 9. Also, the virtual PC people have spent months on OS X getting the speed up on VPC due to issues in OS X with priorities and time-slicing.

I don't know much about how these benchmarks are written or how the compilers actually generate FP code but if they use a standard OS library that isn't particularly optimized then that would show up in the SPEC FP tests.

SPEC doesn't just measure CPU speed, it measures it in conjunction with the complete system that is being used to run the test. Unless they've changed their charter, this was always acknowledged by the SPEC consortium.

I would love to know what kind of impact OS X has on the benchmarks. Has anyone done the equivalent study using Yellow Dog Linux?

Re:Snide comments on "supercomputer" show bias

[peter]

They said they ran in single user mode, no GUI, on OSX and linux. That would probably mean that it was basically a text-console FreeBSD system. With no other processes running, the OS would have to waste time on purpose to slow down the benchmark!

Of notu to GNU and Linux Users

[gus goose]

All tests indicated that the GCC Compiler produced better results than MSoft's C. It is not clearly indicated whether the GCC Results were from Linux bachines, but I presume so.

gus

Processor only test

[gouldtj]

I guess, I am supprised by the results. In just USING the various machines I would say that there is about a 1:2 ratio between the MHz on a PPC to an x86 processor. But then I started to think about them. In reality the processors aren't what make the difference. It is everything else. Macs don't tend to use as much 'soft' stuff, and they make better use of the GPU, so the processor is left to do more 'processing'. I guess they're will never be an completely perfect 'user' test, I'll just have to go with feel. But in my experience those SPEC numbers don't tell the whole story.

It might be interesting to see a comparison with Linux running on both machines... Anyone have one of these?

Re:Processor only test

[Anonymous Coward]

actually you are wrong, under OSX at least most of the graphics are handled by the cpu due to the PDF renderer most surfaces have to be rendered on the CPU then copied over to the graphics card for display.

until more advanced GPU's come along (that can do antialiased biezer curves) all the rendering will be done on the cpu.

Re:Processor only test

[Anonymous Coward]

Problem with this is that gcc on x86 is likely vastly superior to gcc on ppc. So again the ppc may not fare as well as x86. Plus I doubt there is much difference in OS X's gcc compared to the Linux gcc on ppc.

which libraries?

[AnOminous CowHerd]

On the SPEC reports I've seen, they usually provide the list of compiler flags and libraries.

I don't see that info here.

Is it possible that unoptimized libraries like libm would hobble the Mac's results under OS X?

FYI: Good articles on P4 v G4e architecture....

[jeblucas]

For those interested, arstechnica had some great articles a while ago on the processor families and the different ways they handle instructions.

Part I.

Part II.

and what version of Darwin??

[EccentricAnomaly]

I wonder which version of Darwin that they've been using? I've been running some floating-point benchmarks on OS X and I've seen a very large improvement from Darwin ver 5.1 to Darwin ver. 5.3.

my own experience

[jchristopher]

My own experience tells me that a 500mhz Mac (iBook, 640 MB RAM) runs OSX and common apps (browser, mail, newsreader, IM) at roughly the speed of a Pentium II 300 with Windows 2000. That's terrible.

Even a lowend PC these days ($700 or so) will run Windows FAST, whereas Apple's lowend end runs OS X slowly.

Most of the Mac's "speed problems" lie in the OS, not the hardware. Linux on the iBook described above flies.

Re:my own experience

[HypodermicEyes]

Sure, but that's not really relevant to the article. SPEC is a measure of the capability of a CPU+Memory Subsystem+compiler in performing Real Work, where Real Work is number/symbol crunching. You know - linear algebra, compiling, those kinds of things. SPEC couldn't care less about measuring how responsive a GUI is, and rightly so. Assuming the OS has decent process+priority management (as it should for Mach+BSD), the GUI bloat should not be relevant.

I was actually quite surprised by the poor performance of the G4. Although if the other posters are correct in stating that SPEC doesn't test very well for single-precision floats nor SIMD, it's probably not a very good test of the capabilities of the CPU. Of course, Apple has crippled their systems with PC133. That doesn't help either.

Re:my own experience

[HypodermicEyes]

If I may add something... there's a long thread on exactly this subject in comp.arch right now. Look for the thread called "SPEC2k results for G4". There's some very interesting comments from people that mostly seem know what they're talking about.

Re:my own experience

[Llywelyn]

Your experience, in essence, is not the norm for mac users.

My own experience is that a 300 MHz G3 will blow a 500 MHz Pentium out of the water, thats running MacOS 9.

System configurations matter, memory matters, &c.

Re:my own experience

[jchristopher]

That's right, but I'm talking about OS X, not OS 9.

OS 9 is not a fair comparison - it's more like Windows 98 than 2000. Mac users who want a stable system must use OS X. The equivalent PC operating system is Windows 2000.

Windows 2000 is far faster than OS X with the same amount of memory.

Re:my own experience

[Anonymous Coward]

Ah, but you're also running over the G3 vs. G4 divide.

OS X on a G4 makes a massive difference, even at the same speed.

In addition, the other guy's comments are valid - if you run at Thousands, it negates some of the G4s advantage, etc. Just dropping the default install on and marvelling at how slow it's running isn't necessarily a good gut benchmark.

Christ, the default W2K server (and now XP Home/Pro) install includes IIS - like you NEED a web server eating up CPU time and memory, all because MS is too lazy to update their help system's engine?

p4

[Anonymous Coward]

I want g4 v p4 specs!

Assembler anyone?

[Tokerat]

if compilers are making such a huge problem in attaining accurate benchmarks, why doesn't someone hand code some assembly for these machines?

It's not even reasonable to take readings when you KNOW you're data will be inaccurate. Sheesh. Anyone who can code VB will call themselves a "computer scientist" these says...

Re:Assembler anyone?

[Anonymous Coward]

Because SPEC tests the entire system, including the compiler, you twit.

Why a P3?

[george399]

Call me crazy, but why is there a benchmark between a PIII and a G4.
Wouldn't a P4 be a better test?

Re:Why a P3?

[doooras]

i don't think a 1GHz Pentium 4 exists.

Re:Why a P3?

[Anonymous Coward]

Don't you understand? The writer of the article was trying to make Macs look slow...Although a 1GHz PIII ~= a 1.5GHz PIV, the Mac looks worse when compared to the PIII.

Re:Why a P3?

[xazkep]

C'mon. It's all personal.

Re:Why a P3?

[Anonymous Coward]

Particularly in light of the fact that the PIII 1ghz is now 2 years old.

Re:Why a P3?

[Anonymous Coward]

Better Question: Why isnt there a $1500 x86 SPEC test vs a $1500 Apple.

In case you don't read the article...

[Johnny Mnemonic]

Buried in this article is this note: and switched off the second supporting processor of the dual machines. Which means that the Dual 1Gs were only run as single Gig machines--and would therefore be much faster in the real world, so cost comparisons should be made accordingly.

only realistic for cross platform unoptimized apps

[Anonymous Coward]

From what I gather this test used neither MMC (term?) nor AltiVec optimizations, which means this is test can only be (and isn't necessarily) valid for running generic unoptimized applications. The scope of this test is such that its correlation with any users' experience of either computer is extremely limited. I would gather the main thing this would be useful for is if you like to disable your second GHz processor and compile GNU and other GPL'ed programs all day. If that's your bag, perhaps this is helpful otherwise this is nearly irrelevant. I'm much more concerned with floating point operations as I do audio and graphics work and with perceived user performance as I tend to use my computer more than I tend to compile code on my computer. Well, that's not completely true but I don't expect comparison tests on compiling Objective-C with Carbon and Cocoa APIs across chip sets.

Just make your compiler produce Altivec then!

[Anonymous Coward]

The whole idea with SPEC is that it test a number of very optimized real-world codes written in standard programming languages.

The rules are simple: You can do anything you want to your system, compiler, libraries, optimization flags, but you are NOT allowed to touch the code.

This is *GOOD* since it means any optimization introduced by the hardware vendor or compiler authors will benefit all programs, not only hand-tuned assembly.

So, it's completely OK to use vector processing (and some of the benchmarks would benefit from it), but must do it in the compiler and not hand-tune each executable.

Not necessarily

[Wesley Felter]

Some apps can't take advantage of 2 CPUs.

Re:Not necessarily

[fiddlingNero]

I wouldn't necessarily consider this a bad thing. My p4 1.8 quite frequently makes me miss my 2x p3 500. While I agree that 2GHz > 2x 1GHz on raw performance level (given the same chip), the usability level of the latter often exceeds that of the former. That is, I out multi-task my computer all the freakin' time.

Re:Not necessarily

[CaptCosmic]

But, you are ignoring the fact that Mac OX X itself supports dual processors and will distribute processes and threads between the two processors. Thus, it will run faster than just a single processors system.

Now, whether the SPEC benchmarks would reflect that is another question entirely. Not knowing anything about how they are written, I can't make any predictions. However, if they are simply single-threaded tests, then, no, they won't show much of an improvement on the dual systems.

Comparing sci/pro app execution speeds

[RemiT]

Hmm... more real world tests for the new G4 duals (all disabled processor, integer spec benchmarks aside)?
For the 'silver standard' DNA sequencing application BLAST, Apple claims that a dual 1 GHz G4 sequences up to 5 times as fast as a 2 GHz Pentium 4. (Sounds like we may have at least a fair RISC:CISC speed advantage there). Can't complain about the digital video rendering speed either. Mathematica seems to kick @ss. And then there's Maya... yas....
Remind me again why these G4 boxes are too expensive for the professional working scientist or media producer. Say, can anyone spec me a cheaper dual P4 box with builtin 1000-base-T for cluster or render farm building?
Warning: Failure to religiously upgrade your PC's Red Hat installation may be an early sign of OS X intoxication.

same OS would have been better

[gnu-sucks]

They should have loaded a BSD or Linux (same kernel versions) onto each computer, just to rule out OS benchmarking. Not to mention similar graphics cards, ram, hard disk, etc. This is one of the worst benchmarks I've ever seen. What a joke!

Never call the OS???

[vanguard]

As for OS X being optimised for this kind of stuff, we are talking applications that nearly never call the OS for anything, so the impact of OS X is probably nil.

That doesn't sound right. Most unix systems, OSX included (and NT FYI) don't allow direct hardware calls. You can only access system resources through operating sytem APIs.

DOS is the only system I know of that lets you access the hardware directly. (I think NT let's you access graphics systems directly too. However, that has nothing to do with this test).

Vanguard