Slashdot Mirror
FutureMark Confirms nVidia's Benchmark Cheating
jlouderb writes "As first reported by ExtremeTech, Futuremark has confirmed that nVidia is cheating on its 3DMark2003 benchmark through eight driver optimizations. The 3D graphics performance war just keeps getting more and more interesting!" See our previous story.
You don't base your findings on one benchmark. Whenever I go to a site like tomshardware.com they have several different ways to benchmark. Each card has its own strengths, and if a card has cheated it will show up like that.
Paint.NET, a Free Image Editor, with Source Code Available!
Wrong - as they point out in the article, these "optimizations" are usually reductions in quality. They don't just improve performance.
0x0D 0x0A
WHAT?? My FX 5800 Leaf Blower only has a range of five feet and not six? I want a refund!
/^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$/i
While this isn't a huge suprise, I am happy that there are smart folks out there who spend time to uncover this kind of information. Kudos to you for your efforts!
Videocard Benchmarks are about as believable as the the 'World's Best Grampa' award.
-n
http://www.remix.net/
How can company proceed to do its business while blatantly lying to its customers!!??
Oh wait, my medication just kicked in. It's just business as usual. I will just go on checking my MSN e-mail, while watching MSNBC, drinking my Coke and eating my McDonalds burger.
Never mind.
Wearing pants should always be optional.
So it's quite likely that NVidia was just anticipating optimizations and not outright "cheating."
Thank you for submitting this to Slashdot. With Futuremark slashdotted to death, NOBODY will be able to get the evidence! *manical laughter*
This has been done for many years, even the last decade. A good friend of mine works and has worked for almost every major video card company in the buisness for the last decade. What is his job? Make sure THEIR video card gets the best scores on the latest and greatest video cards.
I am sorry to tell you all, but just because Nvidia was CAUGHT this time, doesn't mean they haven't been "cheating" (by optimizing for a specific benchmark) for the last 6 years.
I would bet every driver release contains code to help out benchmarks and even specific games. Why do you think Nvidia just said with there latest driver release " *Up to 30% faster frame rates ( *With Unreal Tournament 2002)".
Its just once in a great while someone notices a performance jump TOO big, or just wants some news worthy-ness and decides to put out a nice PDF file.
- Jeff
Modesty is one of life's greatest attributes
From what I read from [h]ardOCP's benchmark with doom3 It kills nvidia's card. And who cares aren't you suppose to optimize your card?
They also have another benchmark here where they compare the 5900 ultra and the radeon 9800 pro. In that article it says that NVIDIA told them not to use 3DMark03 I recommend reading that article
9th grade, you told me cheaters never make money
well 'pbhtbhtbthbth'
http://198.3.92.62/3dmark03_audit_report.pdf Just don't kill me now. ;-)
A test system with GeForceFX 5900 Ultra and the 44.03 drivers gets 5806 3DMarks with
3DMark03 build 320.
The new build 330 of 3DMark03 in which 44.03 drivers cannot identify 3DMark03 or the tests in
that build gets 4679 3DMarks - a 24.1% drop.
Our investigations reveal that some drivers from ATI also produce a slightly lower total score on
this new build of 3DMark03. The drop in performance on the same test system with a Radeon
9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to
8.2% difference in the game test 4 result, which means that the test was also detected and
somehow altered by the ATI drivers. We are currently investigating this further.
The "optimization" relied on the benchmark camera being on 'rails'. It always shows the exact same angles, and there are some things that the benchmark would have the graphics card render, even though it's impossible for the viewer to see.
HOWEVER, in the development version of 3dmark 2k3, you can take the camera "offroading". When you do that, it becomes apparent that things are being drawn incorrectly -- that there are hard-coded limits that result in the video card doing less work than the program requests.
For those of you whining about how they should use "real life" games for benchmarks, this technique could be applied to anything where the camera path is predetermined. It has nothing to do with 3dmark 2k3 specifically.
Partially true... Trouble is, there aren't any games out yet that exploit pixel/vertex shader features to the extent that Futuremark does. And that gives us insight into how hardware will perform on next generation games. It's not a be all end all benchmark, even by futuremark's PR. It is a tool to be used along side current generation titles to measure differing aspects of hardware.
It is by Nvidia's negligence that the optimisations were found. That's why (among other things) the beta program exists with those features. I think we can probably expect this and other cheat hampering features in future versions.
No, because ATI did a much worse job of cheating. Nvidia got a 24% boost out of some of the benchmarks while the best ATI could do was a measly 8%. This clearly shows that ATI must cheat harder if they want to keep up with Nvidia.
I read the internet for the articles.
Let me just say that this occurs not just on this test, but on all imaginable tests, as well as all games that are somewhere used as benchmarks. Many of the cheats are hard to detect because they don't break the test in the way that this cheat did. For instance, at some point there was a trick for a test with lots of occlusion to clip (discard) polygons that would eventually be occluded. However, these discarded polygons were actually calculated at run-time and not precomputed, so if you changed the test, it would still work right. For Quake (I or II, can't remember) they had a hack where they wouldn't need to clear the framebuffer. That version of Quake would do a glClear at each frame, which takes some time, and prior to framebuffer compression, there was a hack where you wouldn't need to clear the framebuffer if you swapped the Z-check and only used half of the Z span every frame. That hack's probably been backed out now because with framebuffer compression, you're actually better off doing the glClear each frame.
Anyway, I'm posting this as an AC for obvious reasons.
Yeah, here's a mirror of that 760k file - though it won't be up for long, since I've only got 1.9 GB of transfer left for this month.
Be nice and download the zip or the bzip2'd version instead, if you're able.
0x0D 0x0A
This is a problem with Nvidia. The only reason they are competing well with ATI is because they cut so many corners to get their benchmark scores up. It certainly would be nice if Nvidia concentrated on real-world apps and games but it seems like they do not. If you look at the benchmarks historically between ATI and Nvidia's closely competing cards, you'll find that they are closely matched in default runs. However, try turning on 4x anti-aliasing or anisotropic filtering and watch the older, slower, ATI cards beat out the shiniest new Nvidia cards. ATI's image quality has always been superior to Nvidia's. They are all about quantity and need to be focusing more on quality.
According to the article, that's only half the story. I could almost accept it if they were "optimizing" in the sense that, in certain situations, they slightly reduced image quality for a significant gain. That's kind of sketchy, as the card isn't then doing what it's claiming, but you could argue, perhaps, that the tradeoff is worth it. And if this activity were optional, it might be a benefit.
What they're doing here is different, and much worse. They're actually detecting what program is running - whether it is 3D Mark or not. Effectively, what it does is disobey 3DMark, and only 3DMark, when it issues certain commands that would reduce throughput. That has no purpose but to deceive.
So, not only are these not optimizations in that they don't really improve performance, they're not optimizations in that they don't even take effect when you run a program not called 3DMark.
Quite frankly, I think this could be considered false advertising and nVidia should get in deep shit for this. This is the worst kind of cheating, and quite frankly, this could be what puts nVidia down the Voodoo path. I don't know whether I'll ever buy another of their cards.
-Looking for a job as a materials chemist or multivariat
I wonder why this driver cheat was discovered by Extremetech? If you're a video card manufacturer, wouldn't you have your engineers go over every one of the competitions driver releases with a fine-toothed comb, just hoping to find some kind of cheat? You'd think ATI has better testing facilities are resources then ET.
Certainly any negative publicity for NVidia is good for ATI and vice versa.
I am NOT a man!
I am a free number!
I thought the same thing, until I actually RTFA. This is blatant cheating. Everything looks fine until you take the camera off the rails, and then there are clipping and display problems galore.
Further, the problems change depending on which part of the demo you're in (for instance, the "background not being cleared" bug conveniently only shows up in the part of the space demo where a largely black sky is being displayed, and so no background clear is necessary). This is cheating, plain and simple.
ZFS: because love is never having to say fsck
Just think about this the next time you do a 5MB driver download. How much of that code is specifically for detecting and defeating benchmarks? How much of the cheats are part of the instability problems in your system?
No, ATI forced you to medium quality no matter what so that it would seem like high quality scores were better.
-]Phreak Out[-
The problem with 'real world testing' when it comes to video cards aimed at the gamers market is basically the difference between a few lousy FPS between the two top-of-the-line cards (and each have similar features, performance-wise) will be virtually indistinguishable in most cases.
I think people shouldn't get all macho when it comes to this stuff. Honestly, it's like the difference between a 350 hp engine and a 351 hp engine. It doesn't amount to a hill of beans worth of difference except on paper.
Get over it people.
My journal has hot
No, you should buy 3 video cards from Fry's, then put older video cards (which need not even have the same bus on them) in the boxes, and return THOSE to the store. Those bastards will just shrinkwrap 'em and re-sell them.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
In terms of the clipping planes, you're right.
But the nVidia driver also substituted a shader for one of the water effects, which degraded/modified the image quality.
And past history has shown that companies are willing to sacrifice quality for performance (see ATI's Radeon 8500 drivers and Quake 3 for an example)...
It's almost like this is a cold war, of sorts, between the testers/benchmarkers and the card manufacturers.
0x0D 0x0A
Dear nvidia / ATI / etc.,
Please optimize your drivers and hardware for the actual applications and games I run, not the synthetic benchmarks designed to simulate workloads. Benchmarks don't use your products, end-users do.