Slashdot Mirror
More 'Application-Specific' Optimizations in NVidia Drivers
EconolineCrush writes "Futuremark and NVIDIA have been embroiled in a spat over various cheat/optimizations in 3DMark03 for several weeks now. Last week, the soap opera appeared to be over; Futuremark and NVIDIA released a joint statement in which Futuremark clarified that NVIDIA was optimizing its drivers for 3DMark03 rather than cheating. This story, however, appears to be far from over. Tech Report has uncovered a new series of optimizations in NVIDIA's Detonator FX drivers that affect image quality in even Futuremark's latest 3DMark03 build. What's more, if you rename the 3DMark03 executable, the optimizations disappear."
Ooh ooh I scored a 19341 on my 3dMark test so that means I can play Quake now?
Aren't we smart enough not to be pulled in my marketing hyperzor?
Tom
Someday, I'll have a real sig.
They always use the 3DMark results as though it's some sort of holy scripture, and as though a benchmark can indicate how well it will work in a real everyday situation. Every industry optimises for benchmarks. From a marketing point of view, it's insane not to.
The only reliable way to test is by testing it withthe applications it's used for. Get some actual games, and see what the frame rate is. If they optimise for those tests then it doesn't matter! It means they're oiptmised for real world situations.
You can't trust results for older games, and you can't trust benchmarks evidenlty. I think the best thing is just to wait for the new generation of games which will surely clear things up.
It doesn't make sense to buy a card to run Doom 3 when the game isn't out. Here is a clue, when Doom 3 does come out I will be able to buy something as powerful as the FX 5900 for $150.
I'm going to go into an offtopic rant now. It is sad that we have huge displays and crazy-go-nuts graphics processors on computers, but consoles will probably always beat PCs for game size. Game makers are too scared to release a DVD only game, so our games are limited to 700MB by disk, and don't even get me started on controllers.
Slashdotter are stupid and biased.
Really? I mean, I've been playing 3d FPS games since the original Quake/QuakeWorld, and the only thing that has mattered since then is what kind of score you can get on a timedemo. I don't care if a card can get 200000 frames/sec in glxgears or the 3dmark tests - I care about how it works in the latest/greatest 3d FPS. In Quake/QuakeWorld, you *needed* to get at least 40 frames/sec. In Quake2, 60 was an ideal minimum. With Quake3, it changed to 125f/s because there were/are some trick jumps/moves you can only do with a minimum of 125f/s framerate. And of course, when playing online, your connection had to be able to get enough data to feed the card as well.
So throw the benchmark software out, fire up Q3 or whatever, and let us know how the card really performs.
Don't blame me, I voted for Kodos
Sad, but true.
Most people will shop around, to make sure the features they're looking for are simply there and work. Beyond that, they don't do the research to understand which version is better unless they're forced to.
You are... I am... And a bunch of /. readers are too. But the millions of teenagers that grew up without technical skills but love games and subscribe to gaming magazines are looking at the benchmarks to decide what to ask for X-Mas and what to beg Mom to buy at CompUSA. I've seen it happen in person with a girlfriends younger brother. The difference between Quake II scores of 110 FPS to 112 FPS was a world of difference. Of course, the card with another 2 FPS was bought! (Not actual numbers, but I remember the difference was in fact 2 FPS!)
That said, what NVidia is doing is cheating, plain and simple. No laywer or press release can spin it otherwise. Well, they try, but the truth hurts in its simplicity. Change the .exe name and the cheats dissapear. And they are not "optimizations" because when the cheats are working they reduce the quality of the rendered image.
Rather than an Nvidia problem, this is a benchmark problem. I don't know why people keep crying about this rather than fixing the benchmark.
Why isn't the benchmark a supervisor that renames the real benchmark to some random name, then runs it.
Seems to me the trick is to stay one step ahead of the marketers.
That's right, so Nvidia should instead release specs (not under an NDA) on their chips, just like almost every other IC manufacturer in the world.
Most major vendors have been "optimizing" for Quake III ever since it became the informal benchmarking standard... I think Futuremark has blown the issue up a little since they weren't on really good terms with nVidia before this started.
But really, 3dmark has always been a "gee-whiz" pretty demo of current graphic card abilities, but never a reliable benchmark. In fact, no one program/game can be a reliable benchmark, since performance must be judged on a variety of applications. Only then do you get some kind of idea of where the "real world" performance lays.
The competition between ATI and nVidia is good for us customers; they both have excellent cards now. ATI has the fastest, while nVidia's drivers (yeah linux support is flakey I know) seem a bit more stable than ATIâ(TM)s.
Really, between the two companies, it is hard to make a really "wrong" choice.
So yeah, everyone, these aren't the droids you are looking for, you can go about your business...
"To confine our attention to terrestrial matters would be to limit the human spirit." -Stephen Hawking
Users want to know how a piece of hardware performs. When a hardware vendor takes a shortcut it improve results against a specific benchmark, it is subverting the purpose of the benchmark and is unethical. By 'optimizing' or 'cheating', NVIDIA simply has created a situation where the benchmark is not indicative of real-world performance, and consumers lose a source of factual data.
It would not surprise me to see that much of this is an attempt by NVIDIA to marginalize the value of FutureMark 3d 2003. If a benchmark isn't favorable to a piece of hardware, then make the benchmark a 3-ring circus with these antics - then nobody trusts the benchmark at all.
A sad way to do business and I can't say when my GF3 Ti 200 will be replaced, but it when it is I will not be using NVIDIA. Apparently they don't trust users to make a decision based on an honest assessment of facts.
I opened the optimized screen and the renamed .exe screen in two tabs and have been scrolling to each corner flipping back and forth between them, and I've got to say I actually think the image quality is higher on the optimized one. If you look at the book in the background, you get a hint of text on it in the optimized version, where it's blank in the renamed one. And the bevels on the edges of the desks are a lot clearer in the optimized one. And there's a really jagged edge on the carpet under the left desk in the renamed one that gets fixed in the optimized one. It's not all good, of course. There are some textures on the left wall that are brighter in the renamed one, but it's hard to tell which one would be better without seeing it in action. (My system gets about 10fps on that test, not really enough.) And finally the optimized one has one thing that looks obviously worse, and that's the cross pieces on the rear window, they're a little strange.
So, anyways, even though it's bad that they change the settings to get higher scores on the benchmark, I'd like to know how to change those settings myself, if it improves performance that much and looks (arguably) no worse, or even better.
When I want to test-drive a new car, do I build a driving simulator based on the car and load in my route to work so I can see how it would handle during a typical commute? Of course not, I just take the car for a test drive, because sometimes the best way to test something is just to use it.
Benchmarks are inheriently flawed because a benchmark is not what your buy the card for. Card manufacturers are always going to "teach to the test" by optimizing their cards and drivers for whatever the reviewers are using to review the cards. So why not take advantage of this and use popular applications to benchmark? The venerable Quake3 framerate test is one example of this, but I would arge that all benchmarks should be numerical data taken from the performance of real world applications. That way, at least those apps will perform as advertised, and any other apps that take advantage of the same features on the card will benefit.
To be fair, it is the /. crowd that 3dmark appeals to. 3dmark has never been more than a simulation of a game and as such should be considered useless. Games that people actually play are the only real benchmarks as they represent real-world results, the only results that matter. Nvidia optimizes it's drivers for games(benchmarks) and so does every other 3D video card manufacturer, is that cheating? No thats improving performance by tailoring the drivers for a specific enviroment. Thats optimization. Is compiling for athlons cheating? Course its not.
The only cheating is on futuremark's part for selling a product that they claim is a valid benchmark for 3d gaming when it is nothing of the sort.
[Just Shut Up and Do What I say]
My Radeon 9700 pro works fine for me.
Pretty much any graphics card has drivers for XFree86, the rest should work with the vesa driver.
I read the original Extremetech article which details them using the beta version of 3dmark2k3 to "stop" the demo and move the camera outside of the normal rendered path which revealed that nvidia played with their drivers to take a load off the card by not fully renedering everything seen "outside" of the normal view. I would no consider this a cheat, but an optimization. who cares what is not seen by the camera? I give nvidia a pat on the back for this.
On the other hand when you enable the 8x aniso filtering and the driver deliberately reduces image quality to gain a few more marks, I would consider this unfair cheating. If I ask for 8x aniso you better damn well give it to me.
Lastly I think it's stupid for anyone to rely on any single benchmarking program to gauge the performance of a card, that's why I always read THG articles when I want to know how hardware performs, Tom uses 3dmark as one of many applications that he benchmarks new hardware with. and frankly I don't really care about 3dmark scores, I always look at the scores for real games.. I wanna see how my gaming is going to be affected, not how sum st00pid benchmark runs.
C:\earth\humans\del *.m0ronz
Absolutely! I can't believe that some people are saying this isn't cheating!! Here's the deal, prior to any optimisation/cheating... App programmer tells card to render something -> Card renders it the way app programmer tells it to -> User sees what app programmer intended. After optimisation... App programmer tells card to render something -> Card renders it the way app programmer tells it to, but faster -> User sees what app programmer intended. After cheating... Programmer tells card to render something -> Driver programmer decides that, actually app programmer doesn't know what he/she is talking about and shouldn't have told it to render the thing that way and that they know a much better way to render it -> Card renders it the way the driver programmer tells it to, which (surprise!) is faster -> User sees what driver programmer intended. The point is that what the app programmer and driver programmer intended are different things. This, in itself is not a cheat. The cheat comes in when the driver programmer doesn't tell people about the change and instead let's people think that a difference in FPS between competitor cards is because of differences in power, rather than differences in what they are trying to render. It's a matter of trust that graphics cards render things the way the app tells them to. To do otherwise is cheating. Plain and simple.
This is not an nVidia problem? This is a benchmark problem, because they weren't wily enough to prevent nVidia from cheating tremendously and repeatedly?
People are crying about this because they rely upon benchmarks as a gague of how powerful a card is, and make purchasing decisions around such knowledge. Sure, some of them forget that a %5 difference is meaningless in real-world performance, but that doesn't mean that the overall scores are meaningless. nVidia's last round of cheats pretended that the card was 25% faster overall than it actually was. This particular cheat ads between 8 and 18 percent to the total, with the largest false total going to the most expensive card.
In other words, if you bought a $300 nVidia card on the strength of this benchmark, you bought a card that is %40 slower than it should be because nVidia went out of its way to lie about the speed.
ATI optimized for the test, they re-ordered the way in which the card handled executions similar to the way someone might re-order their day for maximum efficiency. It was a cheat, but a minor one that only added 2% to the score. nVidia's cheat involved dropping instructions entirely, equivalent to doing more in a day by checking things off your list without actually doing them, letting the food rot in the kitchen and the dirty laundry pile up.
The sad fact of the matter is that nVidia now has a big problem, in that their fastest card which managed to eeek out ATI's fastest cards can no longer claim that crown, and yet it is ATI's turn next to introduce faster cards. Their technology can compete, but can't demand the premium that graphics card developers rely upon to survive. Furthermore, this cheat comes after nVidia promised to clean up their act and remove all cheats from their driver. Not only did they cheat, they promised to clean up their act and yet cheated again in the very driver that is supposed to be clean. Their public image is bloody shot, significantly worse than ATI's was over their Quake 3 debackle. ATI fell back on their technology and released superior cards, but nVidia doesn't seem to be able to head down that road.
To get back to the poster's original position, the benchmark should try to outsmart the developers, though being a test for future games they have to stay abreast of display technology more than cheating techniques. But blaming the benchmark not the nVidia for cheating is like discovering that during a Car-and-Driver top-test the Ford team took a shortcut shaving %40 off of the race course and congratulating them on their ingenuity.
What they did was indefensible. They knew it, they appologized, and they did it again.
The ______ Agenda
Why's that sad? Who really cares? Most people just want a decent card, and either don't care enough or don't have time to devote themselves to some absurd quest to find the perfect graphics card. You'll never get the latest technologies in a consumer-grade product, so as long as you're getting a good price and it works, why should they care? Stop making this out to be some monstrous injustice, because it's not.
Benchmarks are meant to test card performance. Optimizing for them is wrong, ok? Optimizing for real games is ok, because there's benefit to the end user (better game play). But for synthetic benchmarks, it's just fucking cheating and lying.
Also, compiling for athlons isn't cheating, as long as 2+2 is still equal to 4, and not 3.96. The new set of issues (hell, the original set as well) involve the program not doing what it's supposed to do: Produce the best quality images. The first set found involved it completely screwing up if you went off the path, and the second set (the current one) involve lowering visual quality. ATi got slammed when they did it with quake3, nVidia deserves to take even more shit for doing it with something where the only benefit to the user is that the number at the end is higher. Thus, they're doing it to lie. ATi just did it cuz the cards at the time were crap, but it DID make a REAL game somewhat faster (though uglier).
I'm surprised this comes up so often. I would have thought that by now NVidia and all the other companies would have big notices on the wall of their dev offices:
"When cheating benchmarks, DO NOT use the name of the benchmark app as an indicator"