Slashdot Mirror
FutureMark Confirms nVidia's Benchmark Cheating
jlouderb writes "As first reported by ExtremeTech, Futuremark has confirmed that nVidia is cheating on its 3DMark2003 benchmark through eight driver optimizations. The 3D graphics performance war just keeps getting more and more interesting!" See our previous story.
Test with the applications/games people really use, and they can't optimize for them without, well, optimizing for them! If they want to make Quake III faster, great.
You don't base your findings on one benchmark. Whenever I go to a site like tomshardware.com they have several different ways to benchmark. Each card has its own strengths, and if a card has cheated it will show up like that.
Paint.NET, a Free Image Editor, with Source Code Available!
While this isn't a huge suprise, I am happy that there are smart folks out there who spend time to uncover this kind of information. Kudos to you for your efforts!
Videocard Benchmarks are about as believable as the the 'World's Best Grampa' award.
-n
http://www.remix.net/
Calling them optimizations gives what nVidia is trying to do a level of legitimacy which is undeserved. If you read the Futuremark paper, you will see that they are clearly cheating.
It would be as if a CPU manufacturer substituted its own algorithms stealthily in a CPU performance benchmark and only when running that benchmark.
Sure, you get a higher number, but you aren't measuring what the benchmark designer intended to measure.
This has been done for many years, even the last decade. A good friend of mine works and has worked for almost every major video card company in the buisness for the last decade. What is his job? Make sure THEIR video card gets the best scores on the latest and greatest video cards.
I am sorry to tell you all, but just because Nvidia was CAUGHT this time, doesn't mean they haven't been "cheating" (by optimizing for a specific benchmark) for the last 6 years.
I would bet every driver release contains code to help out benchmarks and even specific games. Why do you think Nvidia just said with there latest driver release " *Up to 30% faster frame rates ( *With Unreal Tournament 2002)".
Its just once in a great while someone notices a performance jump TOO big, or just wants some news worthy-ness and decides to put out a nice PDF file.
- Jeff
Modesty is one of life's greatest attributes
Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.
It not about cheating... but about how much you cheat.
"Engineers do the work of man, Physicists do the work of God"
Different graphics cards have different strengths and weaknesses - much moreso than in previous years.
eg. Fillrate, Vertex manipulation, Texture rasterizer, Shader technology, Texture sampling techniques, Shadow buffering etc.. etc...
Some cards will be better than other at these tasks, and some games will take advantage of differing ratios of these technologies.
The unreal engine has a reliance on poly-count and texture resolution, and it looks like the doom engine will tend to tax shader, and multitexture units more than the polygon throughput side of things.
In other words, gfx cards are now so flexible that their abilities in these individual areas must be assessed in isolation depending on your choice of game/engine/technology.
As little as 2 years ago all that mattered was fillrate, and this was essentially what the direct3d/opengl api's could stress in hardware.
IMO, price seems to be the most useful benchmark for the newest cards.
\\ Mitch
The problem with comments like these is why should Nvidia HAVE to buy into the FutureMark program, which is a "monopoly" in the benchmark market.
...
I believe Nvidia's stance is "we don't really care for 3dmark all that much"
Colossians 2:8
Never attribute to malice what can be explained by stupidity.
I would be amused to see ATI try and sue over this considering that they also appeared to cheat the benchmark on game test 4. I wonder if this is because they weren't able to catch and manipulate any other tests. New benchmark for driver writers: how effectively can the coder cheat the performance benchmarks?
No harm except for the sugar and caffeine that they pump into your body. If you don't think it does any harm then just try and quit for a week. When you start feeling the effects of withdrawal let me know.
I've been caffeine free for 6 months now and the week long migraine when I quit was well worth it.
...Slashdot was to host a BitTorrent of this and similar files for faster, cooperative downloading?
I've said it before, and I'll say it again: doing this would be a win-win situation. It's a pity that the editorial team are too busy playing with MAME/whatever to actually do something of real benefit to the wider community.
"Accept that some days you are the pigeon, and some days you are the statue." - David Brent, Wernham Hogg
When they did nothing more then change the splash screen, the nvidia card gave out different results. That seems to be detecting the benchmark and cheating.
Try reading the article.
Just think about this the next time you do a 5MB driver download. How much of that code is specifically for detecting and defeating benchmarks? How much of the cheats are part of the instability problems in your system?
Funny, I seem to remember Toms Hardware being rabidly AMD fanboyish about 1.5 years ago when AMD still had the fastest processor. I'm not saying they aren't biased fanboys, what I'm saying is they're fairweather fans.
Isn't that the definition of a good reviewer? Fans of the current top of the line stuff - damn their history?
To keep it on-topic, I also seem to remember ATI doing the exact same thing nVidia is now doing with quake "optimization" for the 8500 cards... Do a google search for "quake quack"
Case in point...
So, ATI results drop from this new patch too? Doesn't this mean that ATI is also cheating? If so, then how do we know that there isn't more cheats ATI is using, as this new patch is only made to exploit the nVidia ones. ATI has access to the developer version of 3D mark, so they could hide their cheats much more efficiently.
The problem with 'real world testing' when it comes to video cards aimed at the gamers market is basically the difference between a few lousy FPS between the two top-of-the-line cards (and each have similar features, performance-wise) will be virtually indistinguishable in most cases.
I think people shouldn't get all macho when it comes to this stuff. Honestly, it's like the difference between a 350 hp engine and a 351 hp engine. It doesn't amount to a hill of beans worth of difference except on paper.
Get over it people.
My journal has hot
Benchmark cheating on video drivers has been going on ever since video driver benchmarking. I worked for a graphics card manufacturer for a few years and a driver development company for a few years and remember a few interesting "optimizations". (This was in the 1991-1997 timeframe.)
We had cheats -- excuse me, optimizations -- that were specific to a particular version of Winbench. Other "cheats" were true optimizations... they were just optimizations you'd probably only see in the benchmark program. A few were known to reduce performance in the "real world" but produced higher Winmarks... so we turned them on only when we detected Winbench was running.
One of the major motivations towards benchmarks that use real applications/games was because of these "optimizations" that were only useful in the benchmarking program. Optimizations for those are still cheats, in the sense that they are intended to improve benchmark results, but at least they are optimizations that have a real-world positive effect.
This is basically a "nothing new under the sun" story. But it's good for these kinds of articles to come out every once in a while, because it helps keep things from getting too out of hand.
-(realname) posting anonymously.
I think it's awesome that Futuremark has come out swinging on this one. NVidia has obviously cheated horribly on these benchmarks. ATI aparently has also taken the low road on these but not as low as NVidia.
NVidia is losing. Their chips and cards are worse than ATI's. What's worse than that, though, is that they are still trying to pretend that it's not the case. They need to seriously sit down and work on their designs but instead they are pissing money away working on cheating on benchmarks. That is a really bad sign for a company. It means managament is diverting money away from becoming successful twords appearing to be successful. A mentality like that is disasterous to the real value of a company.
SELL! SELL NOW! Buy again when they have fixed their mangement and design issues.
Contravertial != Overrated. Reply if you disagree, I'll read it.
set softtabstop=4 shiftwidth=4 expandtab nocp worlddomination
"Who came out with a standard API that ALL manufactures could use without resorting to the arcane obfuscation of OpenGL? That's right, cuntfaces...
It was Microsoft."
Right. All manufacturers... whose hardware works with windows. I'll take cross platform compatability thank you very much.
Before you might argue that nobody uses OpenGL, what about all those licensees of the Quake 3 engine? And what about all those who will license the Doom 3 engine?
You could not prove to any court that NVidia is using deceit. NVidia improved their driver so that a certain set of operations runs faster. There is nothing deceitful about this.
Even if they were to state on the box that they have the card that performs best on the 3DMark2003 benchmark, it would still be a truthful statement. Logically, it's a flaw of the benchmark that it is able to be exploited.
If there is any deceit involved, it would be if someone were to claim that the result of this one benchmark conclusively proves that the NVidia card is superior.
Now that's just fucking wrong! I can understand returning a card because it didn't do as well as you thought or not worked at all. But to get a free product out of it. Damn, have some moral backbone.
Life is not for the lazy.
Dear nvidia / ATI / etc.,
Please optimize your drivers and hardware for the actual applications and games I run, not the synthetic benchmarks designed to simulate workloads. Benchmarks don't use your products, end-users do.
Hell I'd read the articles if they weren't slashdotted 90% of the time, such as the current story.
"A good conspiracy is an unprovable one." -Conspiracy Theory
You have just described an optimization, not a cheat. The point of cheats is that they take advantage of knowledge that's not available to normal processes. If your "cheat" takes no such advantage (e.g. calculating its shortcuts at runtime based only on the actual rendering data) then it's actually an optimization.
Have a look at these benchmarks and tell me if nVidia made a cheat for all of them as well. http://www6.tomshardware.com/graphic/20030512/inde x.html
Tom's Hardware does a really good job of actually testing the hardware with multiple softwares to get an accurate test of the card.
BTW -- Those "enhanced" drivers did nothing for my score.
The point is that ATI had a pretty tenuous justification (that they were optimizing for Quake 3 as it's the engine behind a large number of games), but if this is the case then nvidia has none.
Only if they didn't understand what they were doing, which I doubt. Since there aren't many quake-based games that are named quake.exe, and at the time, Quake 3 was an aging game used mainly for benchmarks, and the stunning similarity between the two, you're just searching for a way to justify it.
This case certainly isn't black and white. If you recall, 3dmark and nvidia are kind of in this PR war right now because 3dmark uses general shader code, which the ATI specifically handles better, whereas the nvidia handles specially optimized shader code quicker. Considering that either card is made for these fictional "next generation games" which the gaming media has been warning us of since the days of the Geforce 3, I've decided that it doesn't really matter either way. My old Geforce 4 MX is STILL chugging the newest games away just fine, despite all the doomsayers prophecies, and the gaming media is still warning us about these mythical games that are supposedly just around the corner, so I think I'll hold off my next video card purchace somewhat.
It's been a long time.
"Real World Testing" in general means that they're testing the card on games that are out on the shelf, finished products, right now; i.e. games which were targeted at video cards years old. In other words, one card does 150fps at the highest quality settings, another does 155fps, and when both of them are run on my 80hz refreshing monitor, the results are exactly the same.
Instead, I want testing that approximates the sorts of games that I'll want to buy years from now. Unfortunately those games don't exist yet. In lieu of those games existing, I can look at these eye candy benchmarks to get some idea of what the performance of video cards will be once they're pushed to their limits. How many polygons or how many dynamic lights can programmers squeeze into a scene before the frame rate drops to something unacceptable? How fast can the card whip through those pixel shader programs that everybody is going to be rendering fur and metal and such with in soon? That's what these sorts of benchmarks are supposed to do: tell me how my prospective new purchase will perform on games in the future.