FutureMark Confirms nVidia's Benchmark Cheating

jlouderb writes "As first reported by ExtremeTech, Futuremark has confirmed that nVidia is cheating on its 3DMark2003 benchmark through eight driver optimizations. The 3D graphics performance war just keeps getting more and more interesting!" See our previous story.

Comments

Don't don the tinfoil hats prematurely...

[dvanduzer]

According to the ExtremeTech article, it's entirely plausible that this isn't entirely intentional on NVidia's part:

nVidia believes that the GeForceFX 5900 Ultra is trying to do intelligent culling and clipping to reduce its rendering workload, but that the code may be performing some incorrect operations. Because nVidia is not currently a member of FutureMark's beta program, it does not have access to the developer version of 3DMark2003 that we used to uncover these issues.

So it's quite likely that NVidia was just anticipating optimizations and not outright "cheating."

Re:Don't don the tinfoil hats prematurely...

[jandrese]

I see you didn't read the article. Nvidia is actually detecting 3dmark and substituting in more efficent renderers and dropping the back buffer clearing at certain points to get higher FPS scores.

Something else that may shock you: it appears that ATI is doing the same thing, although to a much lesser extent.

Re:Don't don the tinfoil hats prematurely...

[Ashran]

If you would read the article you'd know that changing a few bits here and there (nothing affecting the actual code flow, just replacing one register with another) removed the 'bug' and the score dropped by 25%.

PDF Mirror

[Cable_Monkey]

http://198.3.92.62/3dmark03_audit_report.pdf Just don't kill me now. ;-)

For those of you too lazy to read the article...

[XaXXon]

The "optimization" relied on the benchmark camera being on 'rails'. It always shows the exact same angles, and there are some things that the benchmark would have the graphics card render, even though it's impossible for the viewer to see.

HOWEVER, in the development version of 3dmark 2k3, you can take the camera "offroading". When you do that, it becomes apparent that things are being drawn incorrectly -- that there are hard-coded limits that result in the video card doing less work than the program requests.

For those of you whining about how they should use "real life" games for benchmarks, this technique could be applied to anything where the camera path is predetermined. It has nothing to do with 3dmark 2k3 specifically.

As an ex-NVidia employee

[Anonymous Coward]

Let me just say that this occurs not just on this test, but on all imaginable tests, as well as all games that are somewhere used as benchmarks. Many of the cheats are hard to detect because they don't break the test in the way that this cheat did. For instance, at some point there was a trick for a test with lots of occlusion to clip (discard) polygons that would eventually be occluded. However, these discarded polygons were actually calculated at run-time and not precomputed, so if you changed the test, it would still work right. For Quake (I or II, can't remember) they had a hack where they wouldn't need to clear the framebuffer. That version of Quake would do a glClear at each frame, which takes some time, and prior to framebuffer compression, there was a hack where you wouldn't need to clear the framebuffer if you swapped the Z-check and only used half of the Z span every frame. That hack's probably been backed out now because with framebuffer compression, you're actually better off doing the glClear each frame.

Anyway, I'm posting this as an AC for obvious reasons.

Re:/.'ed already???

[mskfisher]

Yeah, here's a mirror of that 760k file - though it won't be up for long, since I've only got 1.9 GB of transfer left for this month.
Be nice and download the zip or the bzip2'd version instead, if you're able.

Worse than that!

[siskbc]

Wrong - as they point out in the article, these "optimizations" are usually reductions in quality. They don't just improve performance.

According to the article, that's only half the story. I could almost accept it if they were "optimizing" in the sense that, in certain situations, they slightly reduced image quality for a significant gain. That's kind of sketchy, as the card isn't then doing what it's claiming, but you could argue, perhaps, that the tradeoff is worth it. And if this activity were optional, it might be a benefit.

What they're doing here is different, and much worse. They're actually detecting what program is running - whether it is 3D Mark or not. Effectively, what it does is disobey 3DMark, and only 3DMark, when it issues certain commands that would reduce throughput. That has no purpose but to deceive.

So, not only are these not optimizations in that they don't really improve performance, they're not optimizations in that they don't even take effect when you run a program not called 3DMark.

Quite frankly, I think this could be considered false advertising and nVidia should get in deep shit for this. This is the worst kind of cheating, and quite frankly, this could be what puts nVidia down the Voodoo path. I don't know whether I'll ever buy another of their cards.

Re:Worse than that!

[ergo98]

This is the specific point of the parent and grandparent post: ATI's action was questionable and bordering on fraudulent, but they were "optimizing" a game that people actually play, with a specific branch for quake that altered settings accordingly: 99.9+% of the times that this "optimization" would take effect would be people actually playing the game, versus gathering benchmark numbers. The quake hack didn't have a "if (bBenchmarking)" condition.

From what it sounds like, nvidia purportedly altered something for the specific purposes of deceiving a benchmark. A benchmark has the sole purpose of benchmarking, so there is absolutely no justification for "optimizations" for a benchmark.

The point is that ATI had a pretty tenuous justification (that they were optimizing for Quake 3 as it's the engine behind a large number of games), but if this is the case then nvidia has none.

Re:Isn't this standard practice?

[yamla]

Actually, it is against the law, at least in Canada.

380. (1) Fraud -- Every one who, by deceit, falsehood or other fraudulent means, whether or not it is a false pretence within the meaning of this Act, defrauds the public or any person, whether ascertained or not, of any property, money or valuable security or any service [is guilty of fraud, a criminal offence]...

Nvidia (and ATI before) are guilty of using deceit to attempt to sell more video cards. Thus, they are guilty of fraud.

Re:They didn't cheat!

[egomaniac]

I thought the same thing, until I actually RTFA. This is blatant cheating. Everything looks fine until you take the camera off the rails, and then there are clipping and display problems galore.

Further, the problems change depending on which part of the demo you're in (for instance, the "background not being cleared" bug conveniently only shows up in the part of the space demo where a largely black sky is being displayed, and so no background clear is necessary). This is cheating, plain and simple.

Re:This is why..

[captain_craptacular]

Tom's Hardware and other Intel fanboysites alike

Funny, I seem to remember Toms Hardware being rabidly AMD fanboyish about 1.5 years ago when AMD still had the fastest processor. I'm not saying they aren't biased fanboys, what I'm saying is they're fairweather fans.

To keep it on-topic, I also seem to remember ATI doing the exact same thing nVidia is now doing with quake "optimization" for the 8500 cards... Do a google search for "quake quack"