Slashdot Mirror
FutureMark Confirms nVidia's Benchmark Cheating
jlouderb writes "As first reported by ExtremeTech, Futuremark has confirmed that nVidia is cheating on its 3DMark2003 benchmark through eight driver optimizations. The 3D graphics performance war just keeps getting more and more interesting!" See our previous story.
3DMark03 is such an unreliable and poorly programmed piece of code that it is all but completely useless as a benchmark. This was not an attempt at artificially inflating 3DMark scores it was simply a bug in the drivers that affected the calculations when clipping and culling.
I think that Kyle at HardOCP said it best when discussing this issue he said "Finding a driver bug is one thing, but concluding motive is another." Look up his comments on Thursday May 15, 2003 if you want to read more.
Page 1 of 7
,00.asp
Futuremark Corporation
May 23rd, 2003
To the Media, Futuremark's customers and business partners:
Audit Report: Alleged NVIDIA Driver Cheating on 3DMark03
After the launch of 3DMark03 Build 320 Futuremark has received reports from the members of its BETA Program concerning certain anomalies with 3DMark03 and Nvidia drivers. ExtremeTech (www.extremetech.com) has published an article1 on suspecting NVIDIA drivers to improperly boost scores on Futuremark's 3DMark®03. Some of these anomalies have also been reported by Beyond3D2. Alarmed by all these reports Futuremark has conducted a thorough internal audit regarding this matter and has verified that certain NVIDIA drivers indeed seem to have detection mechanisms, which are triggered by components of the 3DMark03 program. We have identified eight such mechanisms.
In our testing, all identified detection mechanisms stopped working when we altered the benchmark code just trivially and without changing any of the actual benchmark workload. With this altered benchmark, NVIDIA's certain products had a performance drop of as much as 24.1% while competition's products performance drop stayed within the margin of error of 3%. To our knowledge, all drivers with these detection mechanisms were published only after the launch of 3DMark03. According to industry's terminology, this type of driver design is defined as 'driver cheats'.
We are publishing this document to report to our customers in detail about our findings. Main reason behind publishing this document is to answer the criticism presented against synthetic benchmarks and their reliability when testing hardware performance. The document follows a question/answer format.
How Were These Driver Cheats Found?
Members of Futuremark's BETA program3 first noticed how parts of the tests in 3DMark03 were rendered differently on different hardware. When testing NVIDIA hardware on 3DMark03 with socalled developer's version's free camera enabled, they noticed how some parts of tests were rendered strangely, and informed Futuremark of their findings. Futuremark investigated further and our findings show that certain NVIDIA drivers seem to detect when 3DMark03 is running and then replace the 3DMark03's rendering requests with manually implemented alternative rendering operations. These alternative rendering operations reduce the amount of rendering work and thereby increase the obtained benchmark result.
1 Extremetech: http://www.extremetech.com/article2/0,3973,1086025
2 Beyond3D: http://www.beyond3d.com/#news5856
3 Futuremark's BETA program is an open, fee based
Page 2 of 7
Why Does This Matter - It Is Just a Synthetic Benchmark?
We acknowledge with great pride how big a role 3DMark has in the PC industry. 3DMark score has become perhaps the most influential metric of PC performance. Enthusiasts, professional hardware reviewers and OEMs all depend on 3DMark results to a great extent.
We have a tremendous responsibility towards our users, who count on us and on our products when making important decisions. Thus, it matters a great deal that no one is able to take advantage of 3DMark - or any other significant benchmark - with unfair means.
Well designed synthetic benchmarks are excellent tools to objectively compare performance and to reveal different architectures' strengths and weaknesses. Some commentators have argued for only using benchmarks based exclusively on games; however there are severe problems with relying only on this approach:
Game benchmarks only demonstrate how the hardware performs for that particular game and do not indicate the overall performance of the hardware.
Any cheats potentially included in drivers are much easier to hide in game benchmarks.
Finally, synthetic benchmarks can stress the hardware in a variety of ways, allowing reviewers to explore performance in particular areas and extrapolate performance of a hardware f
> just keeps getting more and more interesting!
Uhh, no it's not.
Antiquated competence won't be a job skill forever.
I may be taking this a bit OT, but...
How do you know there have been murders since the beginning of human existance? And don't just quote me a line in a book that's a few thousand years old; that's not proof. I want the proverbial smoking gun. Show me the corpses!
Bad analogy, good sentiment.
What the PDF documents is a smoking gun. The only way to get the displayed results would be to hard-code the clipping plane. While doing so does improve the raw fps while the camera's on the rails, it goes against the very nature of comparitive testing. Its not something you can do in a real environment when the camera isn't on rails.
Page 1 of 7 ,00.asp
Futuremark Corporation
May 23rd, 2003
To the Media, Futuremark's customers and business partners:
Audit Report: Alleged NVIDIA Driver Cheating on 3DMark03
After the launch of 3DMark03 Build 320 Futuremark has received reports from the members of its
BETA Program concerning certain anomalies with 3DMark03 and Nvidia drivers. ExtremeTech
(www.extremetech.com) has published an article1 on suspecting NVIDIA drivers to improperly
boost scores on Futuremark's 3DMark®03. Some of these anomalies have also been reported by
Beyond3D2. Alarmed by all these reports Futuremark has conducted a thorough internal audit
regarding this matter and has verified that certain NVIDIA drivers indeed seem to have detection
mechanisms, which are triggered by components of the 3DMark03 program. We have identified
eight such mechanisms.
In our testing, all identified detection mechanisms stopped working when we altered the
benchmark code just trivially and without changing any of the actual benchmark workload. With
this altered benchmark, NVIDIA's certain products had a performance drop of as much as
24.1% while competition's products performance drop stayed within the margin of error of 3%. To
our knowledge, all drivers with these detection mechanisms were published only after the launch
of 3DMark03. According to industry's terminology, this type of driver design is defined as 'driver
cheats'.
We are publishing this document to report to our customers in detail about our findings. Main
reason behind publishing this document is to answer the criticism presented against synthetic
benchmarks and their reliability when testing hardware performance. The document follows a
question/answer format.
How Were These Driver Cheats Found?
Members of Futuremark's BETA program3 first noticed how parts of the tests in 3DMark03 were
rendered differently on different hardware. When testing NVIDIA hardware on 3DMark03 with socalled
developer's version's free camera enabled, they noticed how some parts of tests were
rendered strangely, and informed Futuremark of their findings. Futuremark investigated further
and our findings show that certain NVIDIA drivers seem to detect when 3DMark03 is running and
then replace the 3DMark03's rendering requests with manually implemented alternative rendering
operations. These alternative rendering operations reduce the amount of rendering work and
thereby increase the obtained benchmark result.
1 Extremetech: http://www.extremetech.com/article2/0,3973,1086025
2 Beyond3D: http://www.beyond3d.com/#news5856
3 Futuremark's BETA program is an open, fee based cooperation program between Futuremark
and the PC industry at large. BETA program members have access to pre-release builds of
upcoming benchmarks and to a so-called developer build. The developer build is exactly the
same as the public version of the benchmark, but with additional functionality. Amongst other
things, the developer build has a 'free camera' mode, where the user can manually move the
camera around while the test is running.
Page 2 of 7
Why Does This Matter - It Is Just a Synthetic Benchmark?
We acknowledge with great pride how big a role 3DMark has in the PC industry. 3DMark score
has become perhaps the most influential metric of PC performance. Enthusiasts, professional
hardware reviewers and OEMs all depend on 3DMark results to a great extent.
We have a tremendous responsibility towards our users, who count on us and on our products
when making important decisions. Thus, it matters a great deal that no one is able to take
advantage of 3DMark - or any other significant benchmark - with unfair means.
Well designed synthetic benchmarks are excellent tools to objectively compare performance and
to reveal different architectures' strengths and weaknesses. Some commentators have argued for
only using benchmarks based exclusively on games; however there are se
bah! no one reads the articles anyway :)
Winter 2010: With Glowing Hearts
For what it is worth, they do say that ATI is also guilty in certain driver versions of cheating.