Slashdot Mirror

Anyone know where the iBench test site can be found? The only link I found ( http://www.veritest.com/benchmarks/i-bench ) is dead.

Question: What's strange is their involvment in the Overheard a series of websites. What's interesting is that the TCG has ownership of Veritas. Answer: The company that did a test on MS Server 2003 and found it to be 300% faster than other OS's. They wereVeritest have actually merged with symantec, and who could forget the 1999 partner ship between Microsoft and symantec?

Thats because the slashdot post is stupidly pointing at the wrong article. The real one can be found here:
http://www.veritest.com/clients/reports/microsoft/ microsoft_IT_Pro.pdf

Verisoft's client list reads like a global Who's Who of the industry. AMD, Apple, Cisco, Dell, Google, Intel, IBM. Google, Nokia, Oracle, PeopleSoft, Philips, Sony, Sun. Hundreds of others. Veritest Clients. This is not a company whose reports will be casually ignored or dismissed at the enterprise level.

Microsoft Windows Server 2003 vs. Linux
Competitive File Server Performance
Comparison

Test report prepared under contract from Microsoft

Executive summary
Microsoft commissioned VeriTest, a
division of Lionbridge Technologies,
Inc., to conduct a series of tests
comparing the File serving
performance of the following server
operating system configurations
running on a variety of server
hardware and processor
configurations:

You haven't cited your numbers for the dell.

Veritest PowerMac G5 comparison That's a PDF, I'm sorry to say.

This is the report your Apple scores came from. When Veritest ran the G5 against a P4 3 GHz Dell, the dell had an FP base of 693. You quote a 2.8 GHz dell at 1209.

Does this mean that a 2.8 GHz P4 is twice as fast as a 3 Ghz P4, or does it mean you made a big mistake somewhere? You *do* know that CPU benchmarking requires a careful equalization of all other parameters? For example, the G5 in this test had one processor physically removed and the Dell had its hyperthreading disabled.

Of course, if you don't trust that website, how about ZDNet or even compare the numbers yourself. There's Veritest's Apple numbers, versus the offical published numbers from SPEC. There's also this site which goes into detail about the benchmark. They used -ffast-math on PPC, but not on x86, for instance. They explicitly turned off hyperthreading, which obviously hurt the Dell machine during the MP tests.

Then again, as the old saying goes, there's three types of lies. Lies, damn lies and benchmarks.

Because I don't use Photoshop and don't do much media processing, I'm not all that interested in how those application-level benchmarks play out. The things I care about are integer-based. Although I've done some crossplatform benchmarking of my typical workload before, it's time-consuming to get that all set up the right way. I may return to that at some point, but I've found that SPEC CPU2000 roughly corresponds, at least in the integer tests. The rate tests don't measure communication costs between processes on multiple processors, so they're a bit of a best case for multiprocessor systems. Still, they can be revealing.

Apple publishes a base 17.2 SPECint_rate for the dual 2GHz G5 (not submitted to SPEC? they point here. Among the published scores at spec.org is a dual Opteron 248 running Linux and using gcc.

It scores 27.4.

Xi Computing charges $200 more for the dual 248 over the 246 the parent article was writing about. So deduct ~10% of MHz from that 27 if you want both boxes to be dual 2GHz....

" The IT Week Labs tests used Ziff-Davis NetBench file server benchmark with 48 client systems. We selected a low-specification but otherwise modern server for our tests. We used an HP ProLiant BL10 eClass Server fitted with a 900MHz Pentium III chip, a single 40GB ATA hard disk and 512MB of RAM. We did not tune any of the software to improve performance."

Easy: you yourself point out that 1100 * 15.7 = 17.27 ... not 17.6.

Since the call for papers for the new Top 500 list was Oct 1, and the BBC show aired on Oct 9 with a companion BBC News story dated Oct 12, you'd hope that VT was simply regurgitating the figure that has already been sent to the Top 500 organization.

And why are you trolling around with one of those super-old benchmarking stories? We've already established that every manufacturer does what they can to show their products in the best possible light. At least Apple documented their test results and methods in full.

So acually, your logic doesn't make any sense: you jump to the conclusion that it's not real results - even though real results already exist and have been submitted, and the entire story is pretty much about that process, making performance figures a critical piece to get accurate - and that they must have just multiplied some benchmark number by 1100. Then, even though the subject of your own post indicates your recognition that "it doesn't add up", you still apparently assume that the results are somehow doctored, this time for the worse, and you manage to weave in one of the stories that tries to make it look like Apple lied with its benchmarks - which it didn't - which is unrelated to the current issue! How does it "assume" the original scores were accurate?? YOU are assuming that they're just multiplying. You might have been onto something if the multiplication actually came out, but it doesn't, meaning that is NOT what they did.

Bravo, +1 Troll.

What that hell is this?!?!

These exactly disclosures were available on the day of the G5 launch. The link with the raw data is here:

Veritest PDF

And you can find a thorough debunking of the whole thing on my site here

This is very, very old news.

Apple hired an independant benchmarking firm

No, they didn't. Independent would mean that they weren't getting paid by anyone who has anything to gain by the results one way or another. As it turns out, apple hired them and they're not independent at all. M$ hires 'independent' think tanks to issue reports and lobby the government all the time.

MS creates "independent" thinktanks whose only income comes from MS and pays them to find the results they want. Veritest is a company that specializes in benchmarks and has done testing for dozens of major companies (see here).
The mythical oraganizations MS quotes usually only have one customer, MS. While it is true that Apple paid Veritest for their services, it's pretty hard to find someone who benchmarks for free and is willing to sign an NDA to keep quiet until the announcement. Veritest, at least, has a reputation to uphold as a fair, independant, tester.

is here

B) everyone knows how buggy and crappy GCC is on the PowerPC

Jeff

The machine (Mac and PC) specifications are in the VeriTest document.

I thought that yesterday's veritest g5 specint report was funny when it comes to describing how they configured the systems. They compared OSX vs. Redhat 9.

One pages 5-6, the describe the apple process for each of the 2 configurations. Each config is about 1/2 page or 24 lines. Besides control panel stuff, you must edit /etc/hostconfig manually, physically remove a processor and reboot twice.

The redhat config, on page 7, is only 9 lines long, requires no file editing, and has only the initial boot to select 1 or 2 processors.

It seems that easy-of-configuration is a reason to use these new machines with linux!

That may well be true, but it's also not really relevant. Being a known troll doesn't mean that his 'essay' is wrong - something that a lot of people seem to have missed.
I thought most of the same things as he did when I read the detailed description of the benchmarks, and if you understand anything about benchmarks then you probably would have too.

You can get the pdf which describe the process here.

The thing that everyone needs to remember though is that EVERYONE CHEATS, the point of releasing benchmarks to the public is very rarely to ensure that they can make an informed discussion, it is to make your product look better than everyone else's. They should be viewed with a healthy dose of scepticism.

Another thing to remember - especially if you are about to flame someone - is that few people, if any, are saying that these new Macs are anything but very good. What they are saying, is that Apple may not be presenting the facts in a clear and unambiguous fashion.

Be grateful that the specifics of the testing are available.

it would be Veritest. After reading the PDF myself, they draw their own conclusions that the G5 outperforms the two Dell machines they tested it against.

So how would Apple be at fault for saying they have the fastest CPU benchmarks based on independent testing? Unless of course "a very stupid man decided to take the large stack of money."

I find it kinda strange how the SPEC results varied quite markedly between what Intel had listed and what was on display on the Apple website. So, I decided to look into it. First thing I did was goto Veritest's site and look at how they did the benchmark.

I noticed something odd... Veritest decided to run TWO different tests in the P4, one with and without Hyper-Threading enabled. Hyper-Threading is enabled by default on the P4 processor. Odd.

Then I decided to goto www.Spec.org and do a benchmark search for Intel P4:

SPEC.org results:
SPECint2000 : 1200
SPECfp2000 : 1229
SPECint_rate2000 : 14.1
SPECfp_rate2000 : 13.7

Apple.com results:
SPECint2000 : 889
SPECfp2000 : 693
SPECint_rate2000 : 10.3
SPECfp_rate2000 : 8.07

And yes, I did choose the latest results for the Intel P4.

It is very clear that the results obtained by Veritest and put forth in their report is of a P4 3.06 GHz with Hyper Threading DISABLED. The last I checked, HT is a feature which is enabled by default. WHY would someone purposely disable HT? Purposely make their CPU run SLOWER? Hmmm...

Come on Apple, do a serious system comparison, the best versus the best. Not the best versus a crippled system.

Looking at pages 7 and 8 of the Veritest performance comparison quoted on Apple's website, I see that hyperthreading (HT) was turned off for the Intel P4 SPEC CPU2000 Rate Base testing. Am I missing something here? I wonder what the Intel benchmarks would have looked like with hyperthreading (HT) on?

I think the one thing to remember about the "offical" SPEC results is the choice of compilers - the ones I saw on the SPEC page used Microsoft's compilers. The ones in Apple's test use GCC 3.3 for both Apple's and Dell's test systems. You can get a PDF detailing the tests, results, and configurations of the systems at http://www.veritest.com/clients/reports/apple/defa ult.asp, VeriTest's site, which handled the testing...

Those statistics were tested outside of Apple by veritest

http://www.veritest.com/clients/reports/apple/appl e_performance.pdf

Perhaps by using the NAGWare Fortran 95 compiler like they say in the Veritest Test Report

According to Apple's web site, they tested their machine against two Dell Intel boxes (Dell Dimension 8300 (P4) and Dell Precision 650 (Dual Xeon)) running Red Hat Linux 9.0 Professional (at Apple's request).

Intel states that Red Hat Linux 9.0 Professional is one of the Linux OS's currently available that "include optimizations for HT Technology and are currently eligible to carry the Intel® Pentium® 4 Processor with HT Technology logo".

Apple commissioned the benchmark from a company called Veritest. The full report (in .PDF format) including all hardware and software used is available from Veritest's web site.

This could make Intel take notice! Of course, this benchmark comes on the same day that Intel announces the 3.2GHz Pentium IV (and Xeon) processors. Go figure!

Of the published data on both (in SPEC processor benchmarks), Apple's Power Mac G5 generated a SPECfp_base2000 score of 840 and SPECint_base2000 score of 800, while Intel claims that their new 3.2 GHz processors get a SPECfp_base2000 score of 1252 and a SPECint_base2000 score of 1221.

And the SPIN goes on!

Er, after all their boasting, they use Apache+Unix for hotmail still?

hmmm...
Veritest-Novell
Maybe RedHat should pay for some test results too.