Slashdot Mirror
FutureMark Confirms nVidia's Benchmark Cheating
jlouderb writes "As first reported by ExtremeTech, Futuremark has confirmed that nVidia is cheating on its 3DMark2003 benchmark through eight driver optimizations. The 3D graphics performance war just keeps getting more and more interesting!" See our previous story.
You don't base your findings on one benchmark. Whenever I go to a site like tomshardware.com they have several different ways to benchmark. Each card has its own strengths, and if a card has cheated it will show up like that.
Paint.NET, a Free Image Editor, with Source Code Available!
Wrong - as they point out in the article, these "optimizations" are usually reductions in quality. They don't just improve performance.
0x0D 0x0A
While this isn't a huge suprise, I am happy that there are smart folks out there who spend time to uncover this kind of information. Kudos to you for your efforts!
Videocard Benchmarks are about as believable as the the 'World's Best Grampa' award.
-n
http://www.remix.net/
This has been done for many years, even the last decade. A good friend of mine works and has worked for almost every major video card company in the buisness for the last decade. What is his job? Make sure THEIR video card gets the best scores on the latest and greatest video cards.
I am sorry to tell you all, but just because Nvidia was CAUGHT this time, doesn't mean they haven't been "cheating" (by optimizing for a specific benchmark) for the last 6 years.
I would bet every driver release contains code to help out benchmarks and even specific games. Why do you think Nvidia just said with there latest driver release " *Up to 30% faster frame rates ( *With Unreal Tournament 2002)".
Its just once in a great while someone notices a performance jump TOO big, or just wants some news worthy-ness and decides to put out a nice PDF file.
- Jeff
Modesty is one of life's greatest attributes
Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.
It not about cheating... but about how much you cheat.
"Engineers do the work of man, Physicists do the work of God"
I would be amused to see ATI try and sue over this considering that they also appeared to cheat the benchmark on game test 4. I wonder if this is because they weren't able to catch and manipulate any other tests. New benchmark for driver writers: how effectively can the coder cheat the performance benchmarks?
This is a problem with Nvidia. The only reason they are competing well with ATI is because they cut so many corners to get their benchmark scores up. It certainly would be nice if Nvidia concentrated on real-world apps and games but it seems like they do not. If you look at the benchmarks historically between ATI and Nvidia's closely competing cards, you'll find that they are closely matched in default runs. However, try turning on 4x anti-aliasing or anisotropic filtering and watch the older, slower, ATI cards beat out the shiniest new Nvidia cards. ATI's image quality has always been superior to Nvidia's. They are all about quantity and need to be focusing more on quality.
When they did nothing more then change the splash screen, the nvidia card gave out different results. That seems to be detecting the benchmark and cheating.
Try reading the article.
Just think about this the next time you do a 5MB driver download. How much of that code is specifically for detecting and defeating benchmarks? How much of the cheats are part of the instability problems in your system?
Funny, I seem to remember Toms Hardware being rabidly AMD fanboyish about 1.5 years ago when AMD still had the fastest processor. I'm not saying they aren't biased fanboys, what I'm saying is they're fairweather fans.
Isn't that the definition of a good reviewer? Fans of the current top of the line stuff - damn their history?
To keep it on-topic, I also seem to remember ATI doing the exact same thing nVidia is now doing with quake "optimization" for the 8500 cards... Do a google search for "quake quack"
Case in point...
The problem with 'real world testing' when it comes to video cards aimed at the gamers market is basically the difference between a few lousy FPS between the two top-of-the-line cards (and each have similar features, performance-wise) will be virtually indistinguishable in most cases.
I think people shouldn't get all macho when it comes to this stuff. Honestly, it's like the difference between a 350 hp engine and a 351 hp engine. It doesn't amount to a hill of beans worth of difference except on paper.
Get over it people.
My journal has hot
I think it's awesome that Futuremark has come out swinging on this one. NVidia has obviously cheated horribly on these benchmarks. ATI aparently has also taken the low road on these but not as low as NVidia.
NVidia is losing. Their chips and cards are worse than ATI's. What's worse than that, though, is that they are still trying to pretend that it's not the case. They need to seriously sit down and work on their designs but instead they are pissing money away working on cheating on benchmarks. That is a really bad sign for a company. It means managament is diverting money away from becoming successful twords appearing to be successful. A mentality like that is disasterous to the real value of a company.
SELL! SELL NOW! Buy again when they have fixed their mangement and design issues.
Contravertial != Overrated. Reply if you disagree, I'll read it.
set softtabstop=4 shiftwidth=4 expandtab nocp worlddomination
"Who came out with a standard API that ALL manufactures could use without resorting to the arcane obfuscation of OpenGL? That's right, cuntfaces...
It was Microsoft."
Right. All manufacturers... whose hardware works with windows. I'll take cross platform compatability thank you very much.
Before you might argue that nobody uses OpenGL, what about all those licensees of the Quake 3 engine? And what about all those who will license the Doom 3 engine?
In terms of the clipping planes, you're right.
But the nVidia driver also substituted a shader for one of the water effects, which degraded/modified the image quality.
And past history has shown that companies are willing to sacrifice quality for performance (see ATI's Radeon 8500 drivers and Quake 3 for an example)...
It's almost like this is a cold war, of sorts, between the testers/benchmarkers and the card manufacturers.
0x0D 0x0A
Now that's just fucking wrong! I can understand returning a card because it didn't do as well as you thought or not worked at all. But to get a free product out of it. Damn, have some moral backbone.
Life is not for the lazy.
Dear nvidia / ATI / etc.,
Please optimize your drivers and hardware for the actual applications and games I run, not the synthetic benchmarks designed to simulate workloads. Benchmarks don't use your products, end-users do.