Slashdot Mirror
FutureMark Confirms nVidia's Benchmark Cheating
jlouderb writes "As first reported by ExtremeTech, Futuremark has confirmed that nVidia is cheating on its 3DMark2003 benchmark through eight driver optimizations. The 3D graphics performance war just keeps getting more and more interesting!" See our previous story.
There a lies, damm lies and statistics .
I remember SPEC benchmarking ment something, and companies putting special routines to make chips seems faster than they were.
Thats why "Real world testing" is important. While not always the greatest comparison, its much better in most cases.
From what I read from [h]ardOCP's benchmark with doom3 It kills nvidia's card. And who cares aren't you suppose to optimize your card?
They also have another benchmark here where they compare the 5900 ultra and the radeon 9800 pro. In that article it says that NVIDIA told them not to use 3DMark03 I recommend reading that article
A test system with GeForceFX 5900 Ultra and the 44.03 drivers gets 5806 3DMarks with
3DMark03 build 320.
The new build 330 of 3DMark03 in which 44.03 drivers cannot identify 3DMark03 or the tests in
that build gets 4679 3DMarks - a 24.1% drop.
Our investigations reveal that some drivers from ATI also produce a slightly lower total score on
this new build of 3DMark03. The drop in performance on the same test system with a Radeon
9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to
8.2% difference in the game test 4 result, which means that the test was also detected and
somehow altered by the ATI drivers. We are currently investigating this further.
Partially true... Trouble is, there aren't any games out yet that exploit pixel/vertex shader features to the extent that Futuremark does. And that gives us insight into how hardware will perform on next generation games. It's not a be all end all benchmark, even by futuremark's PR. It is a tool to be used along side current generation titles to measure differing aspects of hardware.
It is by Nvidia's negligence that the optimisations were found. That's why (among other things) the beta program exists with those features. I think we can probably expect this and other cheat hampering features in future versions.
I wonder why this driver cheat was discovered by Extremetech? If you're a video card manufacturer, wouldn't you have your engineers go over every one of the competitions driver releases with a fine-toothed comb, just hoping to find some kind of cheat? You'd think ATI has better testing facilities are resources then ET.
Certainly any negative publicity for NVidia is good for ATI and vice versa.
I am NOT a man!
I am a free number!
Here is an interesting quote from the article that seems to have been overlooked so far.
"Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.
Gasp, what a shock. Everyone seems to be guilty of having cheated on synthetic benchmarks at some time. This has happened before, it will happen again.
No, ATI forced you to medium quality no matter what so that it would seem like high quality scores were better.
-]Phreak Out[-
>> What's worse than that, though, is that they are still trying to pretend that it's not the case.
Since when? Jen-Hsun Huang admits defeat (But promises a comeback):
"Tiger Woods doesn't win every day. We don't deny that ATI has a wonderful product and it took the performance lead from us. But if they think they're going to hold onto it, they're smoking something hallucinogenic."
is 3dmark03 a synthetic benchmark or a eye-candy?
if i remember correctly some of the people who funded futuremark had something to do with a demo named "second reality". a good old school demo on 2 discs.
if 3dmark was TRULLY a bench it would then resort on code that we find in games!! opts are expected for thoses...even more for stuff...
what if you told carmack that the opt he made for quake and tweaked openGL implementation are just cheats? Sure you remember 3dfx ogl implementation and riva128 drivers...
what if you told ppl from the 'scene that their demo sucks because they don't properly handle Z buffering.
They all rely on tricks.(beter than opts or cheating from a coder point of view), even processors rely on thoses. they're based on user experience, not bogomips or whatever. page-flipping was a inproper behavior at a time when VESA was not VESA but scene called mode-X, eventually it became best practice. Sprites asm hard-coding was the same and most 2d shooters are based on that.
I'm pretty sure ppl at futuremark include some kind of sleazzy code in their bench as coders always do.
the only difference b/w cheating and proper optimization is only PR. if nvidia told us "wow! we made an optimization that runs 3dmark faster" as it would with a game none would complain.
it's just that for a lot of us 3dmark is supposedly an untouchable thing. It's not. it should reflect real world 3d. and in real life you expect those kind of code workaround.
then i ask myself a question... why doesn't futuremark distribute freely a playable bench.
why put us in front of a demo claiming it's a synthetic bench and then why aren't we believing it?
because it'a a lie. either they're real world gaming and tricks are OK, either they're pure demos and tricks are not options.
I do live television special effects for sports, and while I care a -GREAT DEAL- about the performance of the graphics cards (If I screw up, millions of people see it.. right away), we have small enough volume (you only need one system to make graphics for millions of people), that ATI and NVidia don't just hand out their cards to us.
Would I prefer it that way? (Who doesn't like free goodies??) Heck yes! I'd like to get the latest card and evaluate its robustness (Very important to television...) right away so that I can qualify it for use in our systems only a few weeks after it comes out instead of months.
On that note, I'm also constrained by lack of support for Linux on the latest cards (at times). For example, the 9800 doesn't yet have an accelerated linux driver. Dangit! Now, I love the 9700 pro, but I'd love to have that 256 meg on-card.. It is amazing how quickly you can eat up texture memory when you're doing things the card manufacturers didn't think of (like chroma-keying, video mapping, interlaced frame rendering, blah blah)