Slashdot Mirror
FutureMark Confirms nVidia's Benchmark Cheating
jlouderb writes "As first reported by ExtremeTech, Futuremark has confirmed that nVidia is cheating on its 3DMark2003 benchmark through eight driver optimizations. The 3D graphics performance war just keeps getting more and more interesting!" See our previous story.
You don't base your findings on one benchmark. Whenever I go to a site like tomshardware.com they have several different ways to benchmark. Each card has its own strengths, and if a card has cheated it will show up like that.
Paint.NET, a Free Image Editor, with Source Code Available!
Wrong - as they point out in the article, these "optimizations" are usually reductions in quality. They don't just improve performance.
0x0D 0x0A
WHAT?? My FX 5800 Leaf Blower only has a range of five feet and not six? I want a refund!
/^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$/i
This has been done for many years, even the last decade. A good friend of mine works and has worked for almost every major video card company in the buisness for the last decade. What is his job? Make sure THEIR video card gets the best scores on the latest and greatest video cards.
I am sorry to tell you all, but just because Nvidia was CAUGHT this time, doesn't mean they haven't been "cheating" (by optimizing for a specific benchmark) for the last 6 years.
I would bet every driver release contains code to help out benchmarks and even specific games. Why do you think Nvidia just said with there latest driver release " *Up to 30% faster frame rates ( *With Unreal Tournament 2002)".
Its just once in a great while someone notices a performance jump TOO big, or just wants some news worthy-ness and decides to put out a nice PDF file.
- Jeff
Modesty is one of life's greatest attributes
I see you didn't read the article. Nvidia is actually detecting 3dmark and substituting in more efficent renderers and dropping the back buffer clearing at certain points to get higher FPS scores.
Something else that may shock you: it appears that ATI is doing the same thing, although to a much lesser extent.
I read the internet for the articles.
http://198.3.92.62/3dmark03_audit_report.pdf Just don't kill me now. ;-)
A test system with GeForceFX 5900 Ultra and the 44.03 drivers gets 5806 3DMarks with
3DMark03 build 320.
The new build 330 of 3DMark03 in which 44.03 drivers cannot identify 3DMark03 or the tests in
that build gets 4679 3DMarks - a 24.1% drop.
Our investigations reveal that some drivers from ATI also produce a slightly lower total score on
this new build of 3DMark03. The drop in performance on the same test system with a Radeon
9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to
8.2% difference in the game test 4 result, which means that the test was also detected and
somehow altered by the ATI drivers. We are currently investigating this further.
The "optimization" relied on the benchmark camera being on 'rails'. It always shows the exact same angles, and there are some things that the benchmark would have the graphics card render, even though it's impossible for the viewer to see.
HOWEVER, in the development version of 3dmark 2k3, you can take the camera "offroading". When you do that, it becomes apparent that things are being drawn incorrectly -- that there are hard-coded limits that result in the video card doing less work than the program requests.
For those of you whining about how they should use "real life" games for benchmarks, this technique could be applied to anything where the camera path is predetermined. It has nothing to do with 3dmark 2k3 specifically.
Let me just say that this occurs not just on this test, but on all imaginable tests, as well as all games that are somewhere used as benchmarks. Many of the cheats are hard to detect because they don't break the test in the way that this cheat did. For instance, at some point there was a trick for a test with lots of occlusion to clip (discard) polygons that would eventually be occluded. However, these discarded polygons were actually calculated at run-time and not precomputed, so if you changed the test, it would still work right. For Quake (I or II, can't remember) they had a hack where they wouldn't need to clear the framebuffer. That version of Quake would do a glClear at each frame, which takes some time, and prior to framebuffer compression, there was a hack where you wouldn't need to clear the framebuffer if you swapped the Z-check and only used half of the Z span every frame. That hack's probably been backed out now because with framebuffer compression, you're actually better off doing the glClear each frame.
Anyway, I'm posting this as an AC for obvious reasons.
Yeah, here's a mirror of that 760k file - though it won't be up for long, since I've only got 1.9 GB of transfer left for this month.
Be nice and download the zip or the bzip2'd version instead, if you're able.
0x0D 0x0A
Just think about this the next time you do a 5MB driver download. How much of that code is specifically for detecting and defeating benchmarks? How much of the cheats are part of the instability problems in your system?
The problem with 'real world testing' when it comes to video cards aimed at the gamers market is basically the difference between a few lousy FPS between the two top-of-the-line cards (and each have similar features, performance-wise) will be virtually indistinguishable in most cases.
I think people shouldn't get all macho when it comes to this stuff. Honestly, it's like the difference between a 350 hp engine and a 351 hp engine. It doesn't amount to a hill of beans worth of difference except on paper.
Get over it people.
My journal has hot