Slashdot Mirror
PCMark Memory Benchmark Favors GenuineIntel
javy_tahu writes "A review by Ars Technica disclosed that PCMark 2005 Memory benchmark favors GenuineIntel CPUID. A VIA Nano CPU has had its CPUID changed from the original VIA to fake GenuineAMD and GenuineIntel. An improvement of, respectively, 10% and 47% of the score was seen. The reasons of this behavior of FutureMark product are not yet known."
The reasons of this behavior of FutureMark product are not yet known
Easy. Intel paid them to make it that way.
"When life gives you lemons, don't make lemonade. Make life take the lemons back!" -- Cave Johnson
This definitely requires clarification from the creator of the benchmark.
It is possible that the benchmark uses the CPUID to change how the benchmark works, for example, to work around known flaws in a given chip. If this is the case, then the problem is not "omyghoshitplaysfavorites" but rather lack of full disclosure that the benchmarks are not directly comparable across different chips. In the most benign scenario, this could be someone at the benchmark creator's shop forgetting to tell the documentation team. This is still a very serious issue, but it's not fraud.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Code that only used SSE3 or some such on the basis of the CPU ID might explain it but conspiracy is more fun to believe. Lies, Damn Lies and Benchmarks.
This program was made possible by a grant from the Ultra-Humanite, and viewers like you.
Could it be that FutureMark uses the GenuineIntel and AMD flags to enable processor specific extensions? and then does a whole bunch of math with those extensions and never bothers to check the result?
This would indicate some really terrible code on FutureMarks part, and VIA should be flagging those op-codes as illegal op-codes, but it might be possible that something like this could happen. It is even possible that the CPUID checks are duplicated in some library somewhere that actually gets the correct code sequence right, and the main FutureMark code disables the advanced functions of the library whenever the GenuineIntel and AMD flags are missing. Thus FutureMark may feature both code sequences that work and those that don't, and the resulting incompatibilities are what causes the issues.
Why would you even consider running a benchmark program you don't have source code for and cannot compile yourself? (If you are worried about random compiler differences messing up the results, you can check an MD5 sum of the final binary against the published one, but it is important that you can reproduce the binary from source and you can read the source to find out what it does.)
If compilers like ICC cripple their code depending on CPUID, that will just lead all manufacturers to set CPUID to GenuineIntel, just as moronic websites (with help from Microsoft) ensured that all browsers call themselves 'Mozilla'.
-- Ed Avis ed@membled.com
In all likelihood, this probably IS the case, but that still goes a long way to discredit Futuremark as it shows their benchmarks were certainly NOT fairly tested.
+1 IDisagreeSoHeMustBeATrollOrAnAstroturferOrAShill
You can't. That's why it was discovered now. Intel and AMD don't let you change the CPUID results on their CPUs. Via DOES let you change it. (You could hack the benchmark to change the checks, but then your results are invalid because you changed the benchmark code)
Either way, that's not an excuse. As Ars points out, if it is just checking for something like SSE2 the Nano has that. If you want to make an optimized code path it should be based on if a feature is reported as present or not, not who made the CPU.
It's just really REALLY fishy.
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
That's a pretty good analogy.
If Futuremark is indeed enabling CPU features based upon the CPUID, then this situation is a lot like the webpages that render incorrectly in Firefox unless the user agent is set to Internet Explorer.
Does it really matter whether the cause was "incredibly sloppy coding" or "Intel bribed them?" Either way, their benchmark cannot be trusted, and trustworthiness is ESSENTIAL for a benchmark. If anyone pays serious attention to this (which, having read TFA, it seems to merit), then FutureMark is toast.
"My strength is as the strength of ten men, for I am wired to the eyeballs on espresso."
>>> Contrary to the claims of OSS proponents, the code isn't really more trustworthy if it's open, because not all of us are programmers. If we were (hell, even if most of us were), that'd be true. As things are, though, closed source is only slightly less trustworthy than open source.
I disagree. At this point there is controversy. It will be explained by the vendor and people will have to either accept the explanation or not.
If it were open source, the facts of how the code behaves could be determined by third parties and publicized. We wouldn't have to take anyone's word for it.
If 'the people' in Amendment 2 are 'the state' then Amendments 1, 2, 4, 9, and 10 benefit the state, not you.
$randomInternetDude
If the source is open, you have multiple samples of $randomInternetDude to choose from. And it's not really random either. More like $internetDudeWithUnhealthyInterestInGameEngineProgramming, who I would expect to know a thing or two.
And you can always learn enough to verify it for yourself, if you have the source.
Better than trusting $corporatePrMan, anyway.
The problem with this argument is that with open source software, you don't just have to trust a single random guy for your information. When the source is open, it is often the case that MANY people in the online community will examine the code, and through discussion there emerges a consensus which is far more reliable than the opinion of just one random guy. That isn't to say that the community as a whole is never wrong, but it's vastly more trustworthy and reliable than just some $randomInternetDude.
Shawn Asmussen