Slashdot Mirror
FutureMark Confirms nVidia's Benchmark Cheating
jlouderb writes "As first reported by ExtremeTech, Futuremark has confirmed that nVidia is cheating on its 3DMark2003 benchmark through eight driver optimizations. The 3D graphics performance war just keeps getting more and more interesting!" See our previous story.
There a lies, damm lies and statistics .
I remember SPEC benchmarking ment something, and companies putting special routines to make chips seems faster than they were.
Thats why "Real world testing" is important. While not always the greatest comparison, its much better in most cases.
Comment removed based on user account deletion
These cheats could be used in game-benchmarks as well. At least in case of 3DMark, we have proper methods of detecthing those cheats.
Lesbian Nazi Hookers Abducted by UFOs and Forced Into Weight Loss Programs - -all next week on Town Talk.
From what I read from [h]ardOCP's benchmark with doom3 It kills nvidia's card. And who cares aren't you suppose to optimize your card?
They also have another benchmark here where they compare the 5900 ultra and the radeon 9800 pro. In that article it says that NVIDIA told them not to use 3DMark03 I recommend reading that article
A test system with GeForceFX 5900 Ultra and the 44.03 drivers gets 5806 3DMarks with
3DMark03 build 320.
The new build 330 of 3DMark03 in which 44.03 drivers cannot identify 3DMark03 or the tests in
that build gets 4679 3DMarks - a 24.1% drop.
Our investigations reveal that some drivers from ATI also produce a slightly lower total score on
this new build of 3DMark03. The drop in performance on the same test system with a Radeon
9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to
8.2% difference in the game test 4 result, which means that the test was also detected and
somehow altered by the ATI drivers. We are currently investigating this further.
We should have a constant for each 3d company that we can multiple their benchmarks agains...
...
Maybe nvidia is 0.80 and ATI is 0.90
so then 100fps on a geFrorce card, is really 80 fps, and it would be 90 on an ATI...
Partially true... Trouble is, there aren't any games out yet that exploit pixel/vertex shader features to the extent that Futuremark does. And that gives us insight into how hardware will perform on next generation games. It's not a be all end all benchmark, even by futuremark's PR. It is a tool to be used along side current generation titles to measure differing aspects of hardware.
It is by Nvidia's negligence that the optimisations were found. That's why (among other things) the beta program exists with those features. I think we can probably expect this and other cheat hampering features in future versions.
Don't let yourself be fooled by the article... Though they hint at the drivers detecting the program, did it ever occur to you that 3dmark's code could have been incorrect? I see no hard proof that the drivers actually "detected" the program running. Just because the article says "the drivers F'd up, we fixed it, then it worked, so it must be detecting our program", doesn't mean that is true. Perhaps they were doing strange things to mess up nVidia... There's a motive in everything, maybe this is 3dmark attempting to extort something from nVidia.
Thats why "Real world testing" is important. While not always the greatest comparison, its much better in most cases.
:)
"Real world testing" is great, if you're just a gamer. The problem is with independent developers who want to know about the performance of a card. Not only are features important but also how well can the card perform. I put both in consideration when I'm looking for a card. I don't consider current game benchmarks much because those games won't matter in 6 months, or by the time I finally finish my game.
-B
I wonder why this driver cheat was discovered by Extremetech? If you're a video card manufacturer, wouldn't you have your engineers go over every one of the competitions driver releases with a fine-toothed comb, just hoping to find some kind of cheat? You'd think ATI has better testing facilities are resources then ET.
Certainly any negative publicity for NVidia is good for ATI and vice versa.
I am NOT a man!
I am a free number!
Here is an interesting quote from the article that seems to have been overlooked so far.
"Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.
Gasp, what a shock. Everyone seems to be guilty of having cheated on synthetic benchmarks at some time. This has happened before, it will happen again.
No, ATI forced you to medium quality no matter what so that it would seem like high quality scores were better.
-]Phreak Out[-
It's the exact same thing. Both companies tried to get higher performance out of their hardware on one specific piece of software by writing different routines for that software. Don't try to tell me that higher fps in Quake3 didn't help ATI sell more cards. The claim was made by ATI and people testing it on Quake3 that on a certain hardware spec, it got this performance. It's all marketing. Hell, I'm more likely to buy a card based on real world performace like a game than based on a benchmark, so as far as I'm concerned, ATI did something more underhanded here. They cheated in real world performance at the cost of image quality. All nVidia did was cheat a stupid benchmark.
do not read this line twice.
>> What's worse than that, though, is that they are still trying to pretend that it's not the case.
Since when? Jen-Hsun Huang admits defeat (But promises a comeback):
"Tiger Woods doesn't win every day. We don't deny that ATI has a wonderful product and it took the performance lead from us. But if they think they're going to hold onto it, they're smoking something hallucinogenic."
is 3dmark03 a synthetic benchmark or a eye-candy?
if i remember correctly some of the people who funded futuremark had something to do with a demo named "second reality". a good old school demo on 2 discs.
if 3dmark was TRULLY a bench it would then resort on code that we find in games!! opts are expected for thoses...even more for stuff...
what if you told carmack that the opt he made for quake and tweaked openGL implementation are just cheats? Sure you remember 3dfx ogl implementation and riva128 drivers...
what if you told ppl from the 'scene that their demo sucks because they don't properly handle Z buffering.
They all rely on tricks.(beter than opts or cheating from a coder point of view), even processors rely on thoses. they're based on user experience, not bogomips or whatever. page-flipping was a inproper behavior at a time when VESA was not VESA but scene called mode-X, eventually it became best practice. Sprites asm hard-coding was the same and most 2d shooters are based on that.
I'm pretty sure ppl at futuremark include some kind of sleazzy code in their bench as coders always do.
the only difference b/w cheating and proper optimization is only PR. if nvidia told us "wow! we made an optimization that runs 3dmark faster" as it would with a game none would complain.
it's just that for a lot of us 3dmark is supposedly an untouchable thing. It's not. it should reflect real world 3d. and in real life you expect those kind of code workaround.
then i ask myself a question... why doesn't futuremark distribute freely a playable bench.
why put us in front of a demo claiming it's a synthetic bench and then why aren't we believing it?
because it'a a lie. either they're real world gaming and tricks are OK, either they're pure demos and tricks are not options.
I do live television special effects for sports, and while I care a -GREAT DEAL- about the performance of the graphics cards (If I screw up, millions of people see it.. right away), we have small enough volume (you only need one system to make graphics for millions of people), that ATI and NVidia don't just hand out their cards to us.
Would I prefer it that way? (Who doesn't like free goodies??) Heck yes! I'd like to get the latest card and evaluate its robustness (Very important to television...) right away so that I can qualify it for use in our systems only a few weeks after it comes out instead of months.
On that note, I'm also constrained by lack of support for Linux on the latest cards (at times). For example, the 9800 doesn't yet have an accelerated linux driver. Dangit! Now, I love the 9700 pro, but I'd love to have that 256 meg on-card.. It is amazing how quickly you can eat up texture memory when you're doing things the card manufacturers didn't think of (like chroma-keying, video mapping, interlaced frame rendering, blah blah)
Less than 8%. It was 8% in one benchmark, and ~0 in all others, which ended up being 1.9%. ATI's probably fudging the pixel/vertex shader programs in Game 4. The same behavior wasn't seen with NVIDIA, which makes it very unlikely that FutureMark's changes did anything. In addition, the changes were meaningless ones (removing a splash screen, switching registers around, etc.) which do nothing except change the exact bits of the program.
With NVIDIA, it was about 25% lower overall, which I don't believe they spelled out in terms of per-game change, but it's a LOT of driver cheating. A lot more than ATI's, in any case.
Sure. So it may be in violation of, say, OpenGL specifications. I don't know the licensing details, but OpenGL might prohibit NVidia from using its logo or claiming compatibility until that's fixed. That's about as close as you can get to a "legal" remedy.
The market remedy is far simpler. Just don't buy NVidia products if you don't agree with the way they do business. Litigation should really not be the first resort.
This is obviously done quite deliberately to fool the benchmark.
So is shortening pipeline stages to achieve higher clock rates. I think they are both sleazy practices, preying on the least informed consumer, but it does not constitute fraud. Their product really does 3.06 GHz, or 300 FPS. It's your assumption that it translates directly to general performance that is misinformed.
McDonald's has "America's favorite fries", based on sales. If you conclude that it means they taste best, there's no fraud involved here.
Optimization would be a better, but mathematically equivalent algorithm.
I understand your outrage, but your energy is misdirected.
The 3-D graphics industry has always been about "deception". The most commonly used lighting model consists of "ambient", "diffuse", and "specular" lights. These lights are not mathematically accurate, in the sense that they do not simulate real world lights. Instead, they produce a rough approximation. Hell, the basic concept of subdividing an object into polygons is a deception.
Point is, this industry has never been about mathematical correctness, but apparent visual quality. It would be valid for you to ask for magnified screenshots along with FPS ratings, which would then tell a more complete picture of the card's performance, but it's quixotic to ask for mathematically equivalent optimizations in this industry. You're not going to get it, even with the most honest vendors.
Does the DMCA make this type of exposure illegal?
I'd go on a Vegan diet but the delivery time from Vega is too long. --brownkitty
The problem here is not Nvidia. It is the dummies who wrote Futuremark.
It would be TRIVIAL to make it so that when run the program chooses a random number seed that modifies the path and the firections in which the camera looks. This would make it impossible to cheat by creating clipping planes that are out of view, and it would still allow them to create reproduceable results. All they would have to do is pick a set of random numbers and run the demo with those seeds on the different cards and compare results.
Other cheats aren't as easy for them to prevent, but the culling thing is pretty simple to stop.
I've stuck with Matrox, mostly I don't do games and it's very stable.
I will not buy nvidia due to them not allowing specifications out about the cards so a Linux driver can be created. I won't run their binary only driver or any binary only driver.
In fact any binary only code allows for this sort of tweaking or worse.
I'd go on a Vegan diet but the delivery time from Vega is too long. --brownkitty
Beyond3d is reporting on the ATI part of this issue.
ATI's official statement:
The 1.9% performance gain comes from optimization of the two DX9 shaders (water and sky) in Game Test 4 . We render the scene exactly as intended by Futuremark, in full-precision floating point. Our shaders are mathematically and functionally identical to Futuremark's and there are no visual artifacts; we simply shuffle instructions to take advantage of our architecture. These are exactly the sort of optimizations that work in games to improve frame rates without reducing image quality and as such, are a realistic approach to a benchmark intended to measure in-game performance. However, we recognize that these can be used by some people to call into question the legitimacy of benchmark results, and so we are removing them from our driver as soon as is physically possible. We expect them to be gone by the next release of CATALYST.
There isn't really much difference bewteen a "cheat" and a true optimization. As long as the "cheated" driver produces acceptable results, and produces them faster, I don't see what the problem is.
Some of the cheats potentially reduce image quality, but we are talking about OpenGL and DirectX here - nobody really aims for 100% visual quality, and indeed there is no target to shoot for since neither standard specifies "correct" rendering down to the pixel level.
You might complain that 3DMark is being treated specially, that other software wouldn't receive the same speedups. That is true. But application-specific optimization has a long history. Just look at Windows - the more recent versions detect and flag certain programs that are known to break or run slowly due to compatibility issues. Nobody says Windows is "cheating" because it refuses to install a driver that its internal knowledge base knows will trash your system. In the CAD world, video card makers almost always tweak drivers to support specific CAD and 3D applications. (3DLabs' control panel used to have a box where you could select "optimize for AutoCAD/3D Studio/Maya/etc...")
ATI should be happy that NVIDIA engineers are wasting time fixing specific benchmarks when they could instead be improving performance in general. But I wouldn't read much more than that into this.
Making your buying decision based on a synthetic benchmark, rather than in-context with your intended application, is always going to distort the picture. (Looking at SPEC benchmarks, Itanium blows the competition away - just tell that to the millions of people who are *not* buying IA64 chips!)
If you, the OpenGL developer, end up writing the next wildly-successful game, I'm sure NVIDIA will be happy to tweak their drivers for it.