PCMark Memory Benchmark Favors GenuineIntel

javy_tahu writes "A review by Ars Technica disclosed that PCMark 2005 Memory benchmark favors GenuineIntel CPUID. A VIA Nano CPU has had its CPUID changed from the original VIA to fake GenuineAMD and GenuineIntel. An improvement of, respectively, 10% and 47% of the score was seen. The reasons of this behavior of FutureMark product are not yet known."

Comments

Money

[TheRealMindChild]

The reasons of this behavior of FutureMark product are not yet known

Easy. Intel paid them to make it that way.

Re:Money

[Shadow Wrought]

Yep. They increased the L2 Cash size.

Re:Money

[SimonGhent]

Easy. Intel paid them to make it that way.

If anyone can come up with a better explanation I'd be interested to hear it.

Re:Money

[ArcherB]

Easy. Intel paid them to make it that way.

If anyone can come up with a better explanation I'd be interested to hear it.

TFA offers the following:

At the very least, this suggests some incredibly sloppy coding on Futuremark's part, as the company may be enabling or disabling CPU optimizations based on a processor's vendor name in CPUID instead of actually checking CPUID for SIMD support. In this case, PCMark 2005's memory subsystem test doesn't appear to be aware that Nano supports SSE2 and SSE3, and is instead running a decidedly less-optimized code path. There are two factors, however, that make this explanation a bit difficult to swallow.

First, there's the issue of timing. PCMark 2005 was released (obviously) in 2005, and was obviously coded with an eye towards supporting current and future processors. This is standard operating procedure for Futuremark, which always builds benchmarks designed to last for at least a year, and often two. VIA's C5N-T (Nehemiah) core may have only supported MMX and 3DNow!, but the C7 launched in 2005, and that processor supported SSE2 and SSE3 from day one. Even if proper extension support wasn't built into the first version of PCM2K5, we tested version 1.2.0, and that patch was released on or around 11-29-2006.

Second, there's the issue of performance when Nano is identified as AuthenticAMD. If performance between the AMD and Intel CPUIDs was identical, there wouldn't really be a story here, but it isn't, and that's curious. Futuremark could plausibly argue that VIA's C3/C7 processors weren't exactly on the radar back in 2004-2005, but AMD and K8 certainly were, and K8 launched with full SSE and SSE2 support, with SSE3 added in 2005.

There's more, but I don't want to quote the entire article.

Re:Money

[ShieldW0lf]

Moral of the story is, when you're dealing with code like this, where it has the capacity to influence who receives billions of dollars and who doesn't, well, you can't trust it if it's closed source and not subject to public scrutiny.

Closed source test suites cannot be trusted, shouldn't even be considered by potential purchasers, and have been misleading the public for years and years. This is mute evidence to the fact.

Re:Money

[Eponymous Cowboy]

I'll give 10:1 odds that Futuremark simply compiled their benchmark with Intel's C++ compiler.

I wrote a detailed explanation back in 2005 about how the Intel C++ compiler generates separate code paths for memory operations to make AMD processors appear significantly slower, and how you can trick the compiled code into believing your AMD processor is an Intel one to see incredibly increased performance. See this article for additional details.

Re:Money

[Anonymous Coward]

Mods, while you're at it, mod me +5 insightful for pointing out that the parent post was modded +4 insightful for pointing out that its parent was redunant...

Re:Money

[clickclickdrone]

This is just a classic example of amateur (poster) vs professional (Intel dev team).

Writes an (anonymous) Intel representative.

Re:Money

[Fumus]

What I don't get is why game developers don't release freeware benchmark versions of their engines.
Saying that a config has 9000 points is pretty much useless. Saying that it gets an average of 40FPS in the UT3 benchmark at high details, and 1680x1050 is much more informative.

Unfortunately, this also is a little bit more complicated, and as we know everything simpler is more popular with the dumb masses.

Re:Money

[Anonymous Coward]

Here, sir, is the Internet, which you have won fair and square.

Re:Money

[Goaway]

"Everything I don't know how to do is easy!"

GenuineIntel

[Plantain]

I'm a GenuineIntel, mod me 47% higher!

Re:GenuineIntel

[Metorical]

People running Intel can get Score: 7.35, Funny?

Possible semi-benign explaination?

[davidwr]

This definitely requires clarification from the creator of the benchmark.

It is possible that the benchmark uses the CPUID to change how the benchmark works, for example, to work around known flaws in a given chip. If this is the case, then the problem is not "omyghoshitplaysfavorites" but rather lack of full disclosure that the benchmarks are not directly comparable across different chips. In the most benign scenario, this could be someone at the benchmark creator's shop forgetting to tell the documentation team. This is still a very serious issue, but it's not fraud.

Re:Compiler Optimization?

[afidel]

CPUID can be intercepted so it shouldn't be a big deal to grab the call and return whatever you want.

Moronic or Corrupt?

[OmniGeek]

Does it really matter whether the cause was "incredibly sloppy coding" or "Intel bribed them?" Either way, their benchmark cannot be trusted, and trustworthiness is ESSENTIAL for a benchmark. If anyone pays serious attention to this (which, having read TFA, it seems to merit), then FutureMark is toast.

Re:Do I understand this correctly?

[enoz]

GenuineIntel/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008072820 Firefox/3.0.1

I can already feel the speed!

Re:Closed Source Benchmarks?

[Sir_Lewk]

Obviously the only real benchmark is bogomips.

Re:troll? really? mod up again!

[deanoaz]

>>> Contrary to the claims of OSS proponents, the code isn't really more trustworthy if it's open, because not all of us are programmers. If we were (hell, even if most of us were), that'd be true. As things are, though, closed source is only slightly less trustworthy than open source.

I disagree. At this point there is controversy. It will be explained by the vendor and people will have to either accept the explanation or not.
If it were open source, the facts of how the code behaves could be determined by third parties and publicized. We wouldn't have to take anyone's word for it.