Slashdot Mirror
More 'Application-Specific' Optimizations in NVidia Drivers
EconolineCrush writes "Futuremark and NVIDIA have been embroiled in a spat over various cheat/optimizations in 3DMark03 for several weeks now. Last week, the soap opera appeared to be over; Futuremark and NVIDIA released a joint statement in which Futuremark clarified that NVIDIA was optimizing its drivers for 3DMark03 rather than cheating. This story, however, appears to be far from over. Tech Report has uncovered a new series of optimizations in NVIDIA's Detonator FX drivers that affect image quality in even Futuremark's latest 3DMark03 build. What's more, if you rename the 3DMark03 executable, the optimizations disappear."
well luckily for me I've renamed all my executables files 3DMark03.exe for some time now.
Mike
Ooh ooh I scored a 19341 on my 3dMark test so that means I can play Quake now?
Aren't we smart enough not to be pulled in my marketing hyperzor?
Tom
Someday, I'll have a real sig.
They always use the 3DMark results as though it's some sort of holy scripture, and as though a benchmark can indicate how well it will work in a real everyday situation. Every industry optimises for benchmarks. From a marketing point of view, it's insane not to.
The only reliable way to test is by testing it withthe applications it's used for. Get some actual games, and see what the frame rate is. If they optimise for those tests then it doesn't matter! It means they're oiptmised for real world situations.
...to profit off of their customers hopes and dreams. This kind of crap is really showing off the sorry state of our modern societies. we're scum. dirty scum.
Machine9dotNet
Just ask Microsoft.
Oh...wait...
Never mind...
a) Is this indicative of a high level strategy by NVidia's management, who's marketing department is pressuring them to have higher 3DMark2003 scores than ATI?
OR
b) Has some low level device driver programmer (intern?) looked to get some easy brownie points by "optimising" the drivers for 3dMark2003 in a slightly clunky way?
Either is quite interesting :) I've been a victim/perpetrator of both in the past.
If we could have the Open Source developer community review and improve these drivers we would not encounter any problems with them. The experience and integrity of the Open Source developer community would be vital for the consumer to take Nvidia cards seriously in the market.
Benchmarks would reflect the actual performance of the card instead of skewing the results in order to garner favorable reviews.
Only when we allow Nvidia to see the benefits of Open Source can we free the graphic benchmark software from the clutches of Matrox.
Wearing pants should always be optional.
I tried various tricks too. Oddly, renaming it to Outlook.exe made it crash.
You can't trust results for older games, and you can't trust benchmarks evidenlty. I think the best thing is just to wait for the new generation of games which will surely clear things up.
It doesn't make sense to buy a card to run Doom 3 when the game isn't out. Here is a clue, when Doom 3 does come out I will be able to buy something as powerful as the FX 5900 for $150.
I'm going to go into an offtopic rant now. It is sad that we have huge displays and crazy-go-nuts graphics processors on computers, but consoles will probably always beat PCs for game size. Game makers are too scared to release a DVD only game, so our games are limited to 700MB by disk, and don't even get me started on controllers.
Slashdotter are stupid and biased.
Really? I mean, I've been playing 3d FPS games since the original Quake/QuakeWorld, and the only thing that has mattered since then is what kind of score you can get on a timedemo. I don't care if a card can get 200000 frames/sec in glxgears or the 3dmark tests - I care about how it works in the latest/greatest 3d FPS. In Quake/QuakeWorld, you *needed* to get at least 40 frames/sec. In Quake2, 60 was an ideal minimum. With Quake3, it changed to 125f/s because there were/are some trick jumps/moves you can only do with a minimum of 125f/s framerate. And of course, when playing online, your connection had to be able to get enough data to feed the card as well.
So throw the benchmark software out, fire up Q3 or whatever, and let us know how the card really performs.
Don't blame me, I voted for Kodos
If renaming the file changes the optimizations, why doesn't anyone (read: not me) take a gander at the detonator drivers and figure out what OTHER games it's tuned to?
"Draco dormiens nunquam titillandus."
If nVidia continues to have these bulky video cards which take two PCI slots and make noise like a whale they just might go by the way side just like 3DFX.
nVidia is walking a tight rope and for the first time in six years I'm actually going to consider buying an ATI. Come September 30th there is 90% chance that I'll have an ATI card on my machine.
-----
One is born into aristocracy, but mediocrity can only be achieved through hard work.
Sad, but true.
Most people will shop around, to make sure the features they're looking for are simply there and work. Beyond that, they don't do the research to understand which version is better unless they're forced to.
You are... I am... And a bunch of /. readers are too. But the millions of teenagers that grew up without technical skills but love games and subscribe to gaming magazines are looking at the benchmarks to decide what to ask for X-Mas and what to beg Mom to buy at CompUSA. I've seen it happen in person with a girlfriends younger brother. The difference between Quake II scores of 110 FPS to 112 FPS was a world of difference. Of course, the card with another 2 FPS was bought! (Not actual numbers, but I remember the difference was in fact 2 FPS!)
That said, what NVidia is doing is cheating, plain and simple. No laywer or press release can spin it otherwise. Well, they try, but the truth hurts in its simplicity. Change the .exe name and the cheats dissapear. And they are not "optimizations" because when the cheats are working they reduce the quality of the rendered image.
Rather than an Nvidia problem, this is a benchmark problem. I don't know why people keep crying about this rather than fixing the benchmark.
Why isn't the benchmark a supervisor that renames the real benchmark to some random name, then runs it.
Seems to me the trick is to stay one step ahead of the marketers.
Comment removed based on user account deletion
as to why synthetic benchmarks are useful please see the following:Beyond3D. This website is probably the best site for info on 3d hardware.
later,
"Im drowning here, and you're describing the water!"
Does anyone else see the irony in the fact that the "Graphics" topic doesn't have a picture?
"...At the end of the day"..."when everyone goes home, you're stuck with yourself." RIP Layne Staley
Forget about Intel vs. AMD, RDRAM vs. DDR. This is the real political intrigue now. Two cheating hardware companies and the benchmark tool company who hasn't got the guts to stand behind the truth of the matter.
Once again, and this can't be stated strongly enough - synthetic benchmarks really don't tell you what you think you're hearing. Indicative? Yes. Conclusive? Absolutely not. Don't listen too deeply to them.
When this much money is at stake, don't expect to hear the truth from any angle associated with these companies. Remember, we're dealing with marketers and lawyers here...
Most major vendors have been "optimizing" for Quake III ever since it became the informal benchmarking standard... I think Futuremark has blown the issue up a little since they weren't on really good terms with nVidia before this started.
But really, 3dmark has always been a "gee-whiz" pretty demo of current graphic card abilities, but never a reliable benchmark. In fact, no one program/game can be a reliable benchmark, since performance must be judged on a variety of applications. Only then do you get some kind of idea of where the "real world" performance lays.
The competition between ATI and nVidia is good for us customers; they both have excellent cards now. ATI has the fastest, while nVidia's drivers (yeah linux support is flakey I know) seem a bit more stable than ATIâ(TM)s.
Really, between the two companies, it is hard to make a really "wrong" choice.
So yeah, everyone, these aren't the droids you are looking for, you can go about your business...
"To confine our attention to terrestrial matters would be to limit the human spirit." -Stephen Hawking
Dang! I just replaced my GeForge MX with an ATI 9700 - now I have to rename all of my EXEs from 3dmark03.exe to quake.exe
Users want to know how a piece of hardware performs. When a hardware vendor takes a shortcut it improve results against a specific benchmark, it is subverting the purpose of the benchmark and is unethical. By 'optimizing' or 'cheating', NVIDIA simply has created a situation where the benchmark is not indicative of real-world performance, and consumers lose a source of factual data.
It would not surprise me to see that much of this is an attempt by NVIDIA to marginalize the value of FutureMark 3d 2003. If a benchmark isn't favorable to a piece of hardware, then make the benchmark a 3-ring circus with these antics - then nobody trusts the benchmark at all.
A sad way to do business and I can't say when my GF3 Ti 200 will be replaced, but it when it is I will not be using NVIDIA. Apparently they don't trust users to make a decision based on an honest assessment of facts.
I opened the optimized screen and the renamed .exe screen in two tabs and have been scrolling to each corner flipping back and forth between them, and I've got to say I actually think the image quality is higher on the optimized one. If you look at the book in the background, you get a hint of text on it in the optimized version, where it's blank in the renamed one. And the bevels on the edges of the desks are a lot clearer in the optimized one. And there's a really jagged edge on the carpet under the left desk in the renamed one that gets fixed in the optimized one. It's not all good, of course. There are some textures on the left wall that are brighter in the renamed one, but it's hard to tell which one would be better without seeing it in action. (My system gets about 10fps on that test, not really enough.) And finally the optimized one has one thing that looks obviously worse, and that's the cross pieces on the rear window, they're a little strange.
So, anyways, even though it's bad that they change the settings to get higher scores on the benchmark, I'd like to know how to change those settings myself, if it improves performance that much and looks (arguably) no worse, or even better.
How about optimizing the driver for all applications???
Or is that just being silly?
When I want to test-drive a new car, do I build a driving simulator based on the car and load in my route to work so I can see how it would handle during a typical commute? Of course not, I just take the car for a test drive, because sometimes the best way to test something is just to use it.
Benchmarks are inheriently flawed because a benchmark is not what your buy the card for. Card manufacturers are always going to "teach to the test" by optimizing their cards and drivers for whatever the reviewers are using to review the cards. So why not take advantage of this and use popular applications to benchmark? The venerable Quake3 framerate test is one example of this, but I would arge that all benchmarks should be numerical data taken from the performance of real world applications. That way, at least those apps will perform as advertised, and any other apps that take advantage of the same features on the card will benefit.
To be fair, it is the /. crowd that 3dmark appeals to. 3dmark has never been more than a simulation of a game and as such should be considered useless. Games that people actually play are the only real benchmarks as they represent real-world results, the only results that matter. Nvidia optimizes it's drivers for games(benchmarks) and so does every other 3D video card manufacturer, is that cheating? No thats improving performance by tailoring the drivers for a specific enviroment. Thats optimization. Is compiling for athlons cheating? Course its not.
The only cheating is on futuremark's part for selling a product that they claim is a valid benchmark for 3d gaming when it is nothing of the sort.
[Just Shut Up and Do What I say]
My Radeon 9700 pro works fine for me.
Pretty much any graphics card has drivers for XFree86, the rest should work with the vesa driver.
I mean, in this age were we are supposed to tweak the bejesus out of everything (just look at all the people with computers in refridgerators :) ), why not simply let the users choose themselfs?
;)
Just add a new page in the driver settings where you can add an exe-file and then allow the user to activate the different "tweaks/optimizations". It would be more honest and people who want more speed than looks would be happy as a clam.
"But won't the user be confused by all the options?" I hear you ask. Maybe, but have you looked at the BIOS setup lately? Now, the wrong settings here can potentially blow the socks of your precious processor, but the wrong tweak setting will only make DOOM XII look ugly.
Anyway, thats my five cents on this issue. Enough with this and onwards to new drivers and more piethrowing
I read the original Extremetech article which details them using the beta version of 3dmark2k3 to "stop" the demo and move the camera outside of the normal rendered path which revealed that nvidia played with their drivers to take a load off the card by not fully renedering everything seen "outside" of the normal view. I would no consider this a cheat, but an optimization. who cares what is not seen by the camera? I give nvidia a pat on the back for this.
On the other hand when you enable the 8x aniso filtering and the driver deliberately reduces image quality to gain a few more marks, I would consider this unfair cheating. If I ask for 8x aniso you better damn well give it to me.
Lastly I think it's stupid for anyone to rely on any single benchmarking program to gauge the performance of a card, that's why I always read THG articles when I want to know how hardware performs, Tom uses 3dmark as one of many applications that he benchmarks new hardware with. and frankly I don't really care about 3dmark scores, I always look at the scores for real games.. I wanna see how my gaming is going to be affected, not how sum st00pid benchmark runs.
C:\earth\humans\del *.m0ronz
If I rename my quake3's q3.exe to 3DMark03.exe will I get some extra fps? :-)
Ok ok, I shut up....
Absolutely! I can't believe that some people are saying this isn't cheating!! Here's the deal, prior to any optimisation/cheating... App programmer tells card to render something -> Card renders it the way app programmer tells it to -> User sees what app programmer intended. After optimisation... App programmer tells card to render something -> Card renders it the way app programmer tells it to, but faster -> User sees what app programmer intended. After cheating... Programmer tells card to render something -> Driver programmer decides that, actually app programmer doesn't know what he/she is talking about and shouldn't have told it to render the thing that way and that they know a much better way to render it -> Card renders it the way the driver programmer tells it to, which (surprise!) is faster -> User sees what driver programmer intended. The point is that what the app programmer and driver programmer intended are different things. This, in itself is not a cheat. The cheat comes in when the driver programmer doesn't tell people about the change and instead let's people think that a difference in FPS between competitor cards is because of differences in power, rather than differences in what they are trying to render. It's a matter of trust that graphics cards render things the way the app tells them to. To do otherwise is cheating. Plain and simple.
This is not an nVidia problem? This is a benchmark problem, because they weren't wily enough to prevent nVidia from cheating tremendously and repeatedly?
People are crying about this because they rely upon benchmarks as a gague of how powerful a card is, and make purchasing decisions around such knowledge. Sure, some of them forget that a %5 difference is meaningless in real-world performance, but that doesn't mean that the overall scores are meaningless. nVidia's last round of cheats pretended that the card was 25% faster overall than it actually was. This particular cheat ads between 8 and 18 percent to the total, with the largest false total going to the most expensive card.
In other words, if you bought a $300 nVidia card on the strength of this benchmark, you bought a card that is %40 slower than it should be because nVidia went out of its way to lie about the speed.
ATI optimized for the test, they re-ordered the way in which the card handled executions similar to the way someone might re-order their day for maximum efficiency. It was a cheat, but a minor one that only added 2% to the score. nVidia's cheat involved dropping instructions entirely, equivalent to doing more in a day by checking things off your list without actually doing them, letting the food rot in the kitchen and the dirty laundry pile up.
The sad fact of the matter is that nVidia now has a big problem, in that their fastest card which managed to eeek out ATI's fastest cards can no longer claim that crown, and yet it is ATI's turn next to introduce faster cards. Their technology can compete, but can't demand the premium that graphics card developers rely upon to survive. Furthermore, this cheat comes after nVidia promised to clean up their act and remove all cheats from their driver. Not only did they cheat, they promised to clean up their act and yet cheated again in the very driver that is supposed to be clean. Their public image is bloody shot, significantly worse than ATI's was over their Quake 3 debackle. ATI fell back on their technology and released superior cards, but nVidia doesn't seem to be able to head down that road.
To get back to the poster's original position, the benchmark should try to outsmart the developers, though being a test for future games they have to stay abreast of display technology more than cheating techniques. But blaming the benchmark not the nVidia for cheating is like discovering that during a Car-and-Driver top-test the Ford team took a shortcut shaving %40 off of the race course and congratulating them on their ingenuity.
What they did was indefensible. They knew it, they appologized, and they did it again.
The ______ Agenda
Why's that sad? Who really cares? Most people just want a decent card, and either don't care enough or don't have time to devote themselves to some absurd quest to find the perfect graphics card. You'll never get the latest technologies in a consumer-grade product, so as long as you're getting a good price and it works, why should they care? Stop making this out to be some monstrous injustice, because it's not.
Actually, Tech-Report did check what happens to the ATI drivers as well, just visit their website and see. And like they said in the article, the differences they found were negligible for ATI (1-2%) and I would not be surprised if image quality remained the same. After all, there ARE legitimate optimizations one can write for a graphics engine to run better on a given video card architecture, just look at Doom3. The optimisations for GFFX are endless when it comes to the new Doom engine, and I doubt that Id would want to get caught reducing game image quality to let a certain card have an fps advantage.
P.S.I've been visiting Tech-Report for a while now and I find no reason to doubt their honesty quite yet, but then again - i semi-trusted nVidia too.
Benchmarks are meant to test card performance. Optimizing for them is wrong, ok? Optimizing for real games is ok, because there's benefit to the end user (better game play). But for synthetic benchmarks, it's just fucking cheating and lying.
Also, compiling for athlons isn't cheating, as long as 2+2 is still equal to 4, and not 3.96. The new set of issues (hell, the original set as well) involve the program not doing what it's supposed to do: Produce the best quality images. The first set found involved it completely screwing up if you went off the path, and the second set (the current one) involve lowering visual quality. ATi got slammed when they did it with quake3, nVidia deserves to take even more shit for doing it with something where the only benefit to the user is that the number at the end is higher. Thus, they're doing it to lie. ATi just did it cuz the cards at the time were crap, but it DID make a REAL game somewhat faster (though uglier).
I'm surprised this comes up so often. I would have thought that by now NVidia and all the other companies would have big notices on the wall of their dev offices:
"When cheating benchmarks, DO NOT use the name of the benchmark app as an indicator"
Last week when this topic was on slashdot somebody posted that its possible to run these benchmarks rendering them through a software driver that conforms 100% to the DX9 spec. Slow as hell but you end up with EXACTLY what you should get. Maybe the 3DMark authors could render all this stuff in software so they'll know exactly what SHOULD be rendered by a video card and then have a new section in their benchmarks that compares what should have been rendered to what actually was. So, you run the benchmark and you end up with something like this: Nvidia FX: frames per second: 187, accuracy of rendered image: 62% ATI Radeon: frames per second: 177, accuracy of rendered image: 94%
Ok, most people don't understand that there is no one correct method for everything. NVIDIA has it's own rendering path for AF. They want 3dmark to take advantage of their optimization, so they detect when 3dmark is run and force it to take the faster rendering path. The net result is an undetectable quality difference (i can't tell the difference at all) -- and not necessarily quality degredation...just difference. It's not like when ATI made it's hacks so that quake 3 was visually horrible on ati cards. Honestly, all this hoopla is just that. 3dmark's use of generic rendering paths does not properly benchmark the abilities of the vid cards that have optimized paths. Obviously, game developers will take advantage of nvidia's rendering path for AF on nvidia cards and ati's rendering path on ati cards and default to the generic path for everything else. 3dmark is not a good method of benchmarking video cards. your favorite game is the best benchmark. obviously, there need to be some changes made (at 3dmark, nvidia, and ati) to address using vendor specific optimizations. and sensationalist journalists like this need to leave their biases out of their reports (or at least think critically about what they're reporting and realize that the larger issue is the innapropriateness of 3dmark as a proper benchmarking utility).
So suspend them for 8 games, unless they are a big superstar and appeal.
And go back in the archives to see if they ever said, "You can test me for steriods any time -- except right now, of course."
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
so if an executable prefetches libraries, is that called "executable doping"?
I can see in now...
--headline--
Microsoft admits to executable doping, anonymous sources heard talking about how they "knew" their executables loaded too quickly. Microsoft spokesperson read a public apology to all executables that had ben resource starved because of the malaction of their executables.
Yes Francis, the world has gone crazy.