Slashdot Mirror
FutureMark Confirms nVidia's Benchmark Cheating
jlouderb writes "As first reported by ExtremeTech, Futuremark has confirmed that nVidia is cheating on its 3DMark2003 benchmark through eight driver optimizations. The 3D graphics performance war just keeps getting more and more interesting!" See our previous story.
So it's quite likely that NVidia was just anticipating optimizations and not outright "cheating."
I thought that ATI did the same with their Radeon 8500 drivers 2 years ago, making their Quake 3 scores look better by "cheating". Isn't that just status quo in the video card manufactoring world.
http://198.3.92.62/3dmark03_audit_report.pdf Just don't kill me now. ;-)
The "optimization" relied on the benchmark camera being on 'rails'. It always shows the exact same angles, and there are some things that the benchmark would have the graphics card render, even though it's impossible for the viewer to see.
HOWEVER, in the development version of 3dmark 2k3, you can take the camera "offroading". When you do that, it becomes apparent that things are being drawn incorrectly -- that there are hard-coded limits that result in the video card doing less work than the program requests.
For those of you whining about how they should use "real life" games for benchmarks, this technique could be applied to anything where the camera path is predetermined. It has nothing to do with 3dmark 2k3 specifically.
How does McDonalds lie to you?
Here is a short list.
my religion lies somewhere between buddhism and super monkey ball - pamphlet?
Let me just say that this occurs not just on this test, but on all imaginable tests, as well as all games that are somewhere used as benchmarks. Many of the cheats are hard to detect because they don't break the test in the way that this cheat did. For instance, at some point there was a trick for a test with lots of occlusion to clip (discard) polygons that would eventually be occluded. However, these discarded polygons were actually calculated at run-time and not precomputed, so if you changed the test, it would still work right. For Quake (I or II, can't remember) they had a hack where they wouldn't need to clear the framebuffer. That version of Quake would do a glClear at each frame, which takes some time, and prior to framebuffer compression, there was a hack where you wouldn't need to clear the framebuffer if you swapped the Z-check and only used half of the Z span every frame. That hack's probably been backed out now because with framebuffer compression, you're actually better off doing the glClear each frame.
Anyway, I'm posting this as an AC for obvious reasons.
the pdf for bittorrent
Yeah, here's a mirror of that 760k file - though it won't be up for long, since I've only got 1.9 GB of transfer left for this month.
Be nice and download the zip or the bzip2'd version instead, if you're able.
0x0D 0x0A
Lets not forget that about 4 months ago Nvidia deemed 3D Mark2003 a poor representation of real world scenarios, so how could they be "Cheating" if they pointed this out before hand? and what about all the other FX5900 benchmarks where Nvidia had a steady 20 to 30% lead on ATI? This article was posted before the FX cards were released, Nvidia's not trying to "SNEAK" anything by us here. "The primary goal of any benchmark is to arm the consumer with the right information to make the best possible purchase decision. As the gamers' benchmark, 3DMark 03 must emulate as closely as possible the kind of experience that the gaming enthusiast will expect on their machine. It must exercise graphics hardware in the same manner that consumer games will. The graphics features, rendering paths, and effects must all emulate games, or the consumer will be misinformed and their expectations misguided. 3DMark 03 combines custom artwork with a custom rendering engine that creates a set of demo scenes that, while pretty, have very little to do with actual games. It is much better termed a demo than a benchmark. The examples included in this report illustrate that 3DMark 03 does not represent games, can never be used as a stand-in for games, and should not be used as a gamers' benchmark. The ultimate injury to the consumer of such a benchmark is three-fold. First, of course, the consumer is misguided. A purchase decision based on ineffectual data will lead consumers to wrong conclusions. Second, it causes graphics hardware manufacturers to focus attention and engineering resources on optimizing for artificially fabricated cases that are a-typical of games. Such optimizations generally do nothing to improve real game performance, and provide no benefit to the consumer. Finally, the extra engineering effort focused on such benchmarks reduces the effort available for activities beneficial to consumers--improving the actual gaming experience." http://www6.tomshardware.com/column/20030219/index .html
Of course if the article title was, "Everybody cheats on our benchmark!" then that would do more to undermine their benchmark than anything else. Instead they made the focus of the article the fact that NVidia is cheating.
Lasers Controlled Games!
Mirror. Slashdot into oblivion.
According to the article, that's only half the story. I could almost accept it if they were "optimizing" in the sense that, in certain situations, they slightly reduced image quality for a significant gain. That's kind of sketchy, as the card isn't then doing what it's claiming, but you could argue, perhaps, that the tradeoff is worth it. And if this activity were optional, it might be a benefit.
What they're doing here is different, and much worse. They're actually detecting what program is running - whether it is 3D Mark or not. Effectively, what it does is disobey 3DMark, and only 3DMark, when it issues certain commands that would reduce throughput. That has no purpose but to deceive.
So, not only are these not optimizations in that they don't really improve performance, they're not optimizations in that they don't even take effect when you run a program not called 3DMark.
Quite frankly, I think this could be considered false advertising and nVidia should get in deep shit for this. This is the worst kind of cheating, and quite frankly, this could be what puts nVidia down the Voodoo path. I don't know whether I'll ever buy another of their cards.
-Looking for a job as a materials chemist or multivariat
Nvidia (and ATI before) are guilty of using deceit to attempt to sell more video cards. Thus, they are guilty of fraud.
Oceania has always been at war with Eastasia.
I thought the same thing, until I actually RTFA. This is blatant cheating. Everything looks fine until you take the camera off the rails, and then there are clipping and display problems galore.
Further, the problems change depending on which part of the demo you're in (for instance, the "background not being cleared" bug conveniently only shows up in the part of the space demo where a largely black sky is being displayed, and so no background clear is necessary). This is cheating, plain and simple.
ZFS: because love is never having to say fsck
http://torrents.slash0.org/ /. seems to be down right now though
its basicly a BitTorrent tracker for
0x0D 0x0A
Tom's Hardware and other Intel fanboysites alike
Funny, I seem to remember Toms Hardware being rabidly AMD fanboyish about 1.5 years ago when AMD still had the fastest processor. I'm not saying they aren't biased fanboys, what I'm saying is they're fairweather fans.
To keep it on-topic, I also seem to remember ATI doing the exact same thing nVidia is now doing with quake "optimization" for the 8500 cards... Do a google search for "quake quack"
They who would give up an essential liberty for temporary security, deserve neither liberty nor security
ATI had their own cheating debacle a few years back.
Quake 3 vs Quack 3
The problem with real world testing: Should I go out and buy 3 video cards and then return 2 to the store? Especially with CompUSA's 15% "restocking" fee...
From the article:
Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.
No, because most reviewers would point out any serious image problems (usually they take screen shots as well). What's at issue here is that with the "optimizations", the card isn't doing what its supposed to be doing, its using shortcuts that won't be detected due to the known path of the benchmark. If it were a game, it would be less of an issue because it would either work or it wouldn't.
Image a benchmark that calculated pi to 10,000 digits ten million times then exited. You could optimize it by realizing that it never actually used the values it calculated and "optimize" the entire program to a NOP (This has been done :^). Imagine the speed up! But it invalidates the benchmark, because you didn't do what you were supposed to do.
Or a Mongol training course where the wall was out of sight of the judges. Horde A cheats and runs around the wall so they get a better time than Horde B, who goes over it. When Gengis brings Horde A to invade China he gets slaughters because Horde A isn't very good at climbing walls, esp. Great Walls, despite they better time. Fortunately, Gengis had spies scretly watching the walls (this is whey he is Khan) and slaughters Horde A.
Thus concludes todays episode of "History Bastardizations Great and Small"
You are in a maze of twisted little posts, all alike.
I really don't think that this is still a valid argument. I'm on my second Radeon now, and preformance/stability is excellent under both Linux and Windows.
In the last few years, I can only think of a couple crashes that were possibly caused by the drivers. I can't say for sure, but those crashes were probably caused by overheat, and not by the mythical shitty ATI drivers.
I don't see any reason to even bother with nVidia, and unless I become dissatisfied with ATI, I don't even give nVidia a second thought.
Dom
They've never done you any harm. And except for recent accusations of revenue massaging, they don't lie.
Well, friend. It's time you learn that nothing is sacred. Yes, Virginia, even Coca-Cola lies and squashes people to keep its bottom line intact. Read the sad and infuriating tale of judicial corruption and corporate fraud of Bob Kolody vs. Coca-Cola. I was outraged for days.
If it's for-profit but free, you're not the customer -- you're the product (e.g., the Slashdot Beta's "audience").
Let's not jump on nVidia too harshly for this. Sure, this spectacle seems to have gained a lot more publicity than ATi's own cheating ( link link link ). At least when nVidia cheated in 3DMark, they publically denounced synthetic benchmarks.
You forgot this part:
"And my doctor thinks this twitch will eventually go away"
PSST... all previous references refer to the big blower fan on the unit... ;)
NVidia immidiately put out a rebuttal to these claims, and I'm not sure why they weren't reported along with this article. But, I guess I really can't say that I'm not used to biased or ignorant reporting from slashdot.
From Bluesnews (from an unlinked CNet article):
"Recently, there have been questions and some confusion regarding 3DMark 03 results obtained with certain Nvidia" products, Futuremark said in the statement. "We have now established that Nvidia's Detonator FX drivers contain certain detection mechanisms that cause an artificially high score when using 3DMark 03."
A representative at Nvidia questioned the validity of Futuremark's conclusions. "Since Nvidia is not part of the Futuremark beta program (a program which costs of hundreds of thousands of dollars to participate in), we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer," the representative said. "We don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad."
I know more than you drink.