FutureMark Confirms nVidia's Benchmark Cheating

jlouderb writes "As first reported by ExtremeTech, Futuremark has confirmed that nVidia is cheating on its 3DMark2003 benchmark through eight driver optimizations. The 3D graphics performance war just keeps getting more and more interesting!" See our previous story.

Comments

Doom3

[Blaster Jaack]

From what I read from [h]ardOCP's benchmark with doom3 It kills nvidia's card. And who cares aren't you suppose to optimize your card?

They also have another benchmark here where they compare the 5900 ultra and the radeon 9800 pro. In that article it says that NVIDIA told them not to use 3DMark03 I recommend reading that article

Performance Difference Due to These Cheats

[Anonymous Coward]

A test system with GeForceFX 5900 Ultra and the 44.03 drivers gets 5806 3DMarks with
3DMark03 build 320.
The new build 330 of 3DMark03 in which 44.03 drivers cannot identify 3DMark03 or the tests in
that build gets 4679 3DMarks - a 24.1% drop.
Our investigations reveal that some drivers from ATI also produce a slightly lower total score on
this new build of 3DMark03. The drop in performance on the same test system with a Radeon
9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to
8.2% difference in the game test 4 result, which means that the test was also detected and
somehow altered by the ATI drivers. We are currently investigating this further.

Re:This is why artificial benchmarks don't matter

[rmarll]

Partially true... Trouble is, there aren't any games out yet that exploit pixel/vertex shader features to the extent that Futuremark does. And that gives us insight into how hardware will perform on next generation games. It's not a be all end all benchmark, even by futuremark's PR. It is a tool to be used along side current generation titles to measure differing aspects of hardware.

It is by Nvidia's negligence that the optimisations were found. That's why (among other things) the beta program exists with those features. I think we can probably expect this and other cheat hampering features in future versions.

Who found it?

[anon*127.0.0.1]

I wonder why this driver cheat was discovered by Extremetech? If you're a video card manufacturer, wouldn't you have your engineers go over every one of the competitions driver releases with a fine-toothed comb, just hoping to find some kind of cheat? You'd think ATI has better testing facilities are resources then ET.

Certainly any negative publicity for NVidia is good for ATI and vice versa.

Re:Who found it?

[User 956]

Well, if you read Hard|OCP last week, you might have gotten the impression that Extremetech was making the whole thing up. They said "I have a feeling that Et has some motives of their own that might make a good story"

Right, like maybe getting a fix posted? Oh, wait, looks like Hard|OCP is taking credit for that:

Futuremark has released a patch for 3DMark 2003 that eliminates "artificially high scores" for people using NVIDIA Detonator FX drivers. This is in response to the news item we posted last week. According to the PDF on Futuremark's site, the patch causes a 24.1% drop in score for NVIDIA..."

I'm amazed at the OCP's coverage of this whole deal. They didn't break the story, so they cast doubt on ExtremeTech's findings, and allude to suspicious "motives" that were never proven.

Then, when the fix is released, they claim the fix is released "in response to a news item we posted last week", as if they're directly responsible. A week ago they're bashing ExtremeTech for even insinuating driver cheating, and this week they're taking credit for getting the fix released (as if they broke the story themselves).

Re:ATI Did The Same...

[TrancePhreak]

No, ATI forced you to medium quality no matter what so that it would seem like high quality scores were better.