Slashdot Mirror
More 'Application-Specific' Optimizations in NVidia Drivers
EconolineCrush writes "Futuremark and NVIDIA have been embroiled in a spat over various cheat/optimizations in 3DMark03 for several weeks now. Last week, the soap opera appeared to be over; Futuremark and NVIDIA released a joint statement in which Futuremark clarified that NVIDIA was optimizing its drivers for 3DMark03 rather than cheating. This story, however, appears to be far from over. Tech Report has uncovered a new series of optimizations in NVIDIA's Detonator FX drivers that affect image quality in even Futuremark's latest 3DMark03 build. What's more, if you rename the 3DMark03 executable, the optimizations disappear."
well luckily for me I've renamed all my executables files 3DMark03.exe for some time now.
Mike
Ooh ooh I scored a 19341 on my 3dMark test so that means I can play Quake now?
Aren't we smart enough not to be pulled in my marketing hyperzor?
Tom
Someday, I'll have a real sig.
They always use the 3DMark results as though it's some sort of holy scripture, and as though a benchmark can indicate how well it will work in a real everyday situation. Every industry optimises for benchmarks. From a marketing point of view, it's insane not to.
The only reliable way to test is by testing it withthe applications it's used for. Get some actual games, and see what the frame rate is. If they optimise for those tests then it doesn't matter! It means they're oiptmised for real world situations.
a) Is this indicative of a high level strategy by NVidia's management, who's marketing department is pressuring them to have higher 3DMark2003 scores than ATI?
OR
b) Has some low level device driver programmer (intern?) looked to get some easy brownie points by "optimising" the drivers for 3dMark2003 in a slightly clunky way?
Either is quite interesting :) I've been a victim/perpetrator of both in the past.
If we could have the Open Source developer community review and improve these drivers we would not encounter any problems with them. The experience and integrity of the Open Source developer community would be vital for the consumer to take Nvidia cards seriously in the market.
Benchmarks would reflect the actual performance of the card instead of skewing the results in order to garner favorable reviews.
Only when we allow Nvidia to see the benefits of Open Source can we free the graphic benchmark software from the clutches of Matrox.
Wearing pants should always be optional.
I tried various tricks too. Oddly, renaming it to Outlook.exe made it crash.
You can't trust results for older games, and you can't trust benchmarks evidenlty. I think the best thing is just to wait for the new generation of games which will surely clear things up.
It doesn't make sense to buy a card to run Doom 3 when the game isn't out. Here is a clue, when Doom 3 does come out I will be able to buy something as powerful as the FX 5900 for $150.
I'm going to go into an offtopic rant now. It is sad that we have huge displays and crazy-go-nuts graphics processors on computers, but consoles will probably always beat PCs for game size. Game makers are too scared to release a DVD only game, so our games are limited to 700MB by disk, and don't even get me started on controllers.
Slashdotter are stupid and biased.
Really? I mean, I've been playing 3d FPS games since the original Quake/QuakeWorld, and the only thing that has mattered since then is what kind of score you can get on a timedemo. I don't care if a card can get 200000 frames/sec in glxgears or the 3dmark tests - I care about how it works in the latest/greatest 3d FPS. In Quake/QuakeWorld, you *needed* to get at least 40 frames/sec. In Quake2, 60 was an ideal minimum. With Quake3, it changed to 125f/s because there were/are some trick jumps/moves you can only do with a minimum of 125f/s framerate. And of course, when playing online, your connection had to be able to get enough data to feed the card as well.
So throw the benchmark software out, fire up Q3 or whatever, and let us know how the card really performs.
Don't blame me, I voted for Kodos
If nVidia continues to have these bulky video cards which take two PCI slots and make noise like a whale they just might go by the way side just like 3DFX.
nVidia is walking a tight rope and for the first time in six years I'm actually going to consider buying an ATI. Come September 30th there is 90% chance that I'll have an ATI card on my machine.
-----
One is born into aristocracy, but mediocrity can only be achieved through hard work.
Sad, but true.
Most people will shop around, to make sure the features they're looking for are simply there and work. Beyond that, they don't do the research to understand which version is better unless they're forced to.
You are... I am... And a bunch of /. readers are too. But the millions of teenagers that grew up without technical skills but love games and subscribe to gaming magazines are looking at the benchmarks to decide what to ask for X-Mas and what to beg Mom to buy at CompUSA. I've seen it happen in person with a girlfriends younger brother. The difference between Quake II scores of 110 FPS to 112 FPS was a world of difference. Of course, the card with another 2 FPS was bought! (Not actual numbers, but I remember the difference was in fact 2 FPS!)
That said, what NVidia is doing is cheating, plain and simple. No laywer or press release can spin it otherwise. Well, they try, but the truth hurts in its simplicity. Change the .exe name and the cheats dissapear. And they are not "optimizations" because when the cheats are working they reduce the quality of the rendered image.
Rather than an Nvidia problem, this is a benchmark problem. I don't know why people keep crying about this rather than fixing the benchmark.
Why isn't the benchmark a supervisor that renames the real benchmark to some random name, then runs it.
Seems to me the trick is to stay one step ahead of the marketers.
Comment removed based on user account deletion
as to why synthetic benchmarks are useful please see the following:Beyond3D. This website is probably the best site for info on 3d hardware.
later,
"Im drowning here, and you're describing the water!"
Does anyone else see the irony in the fact that the "Graphics" topic doesn't have a picture?
"...At the end of the day"..."when everyone goes home, you're stuck with yourself." RIP Layne Staley
Hopes and dreams? We're talking about video card benchmarks!
This space intentionally left blank.
Forget about Intel vs. AMD, RDRAM vs. DDR. This is the real political intrigue now. Two cheating hardware companies and the benchmark tool company who hasn't got the guts to stand behind the truth of the matter.
Once again, and this can't be stated strongly enough - synthetic benchmarks really don't tell you what you think you're hearing. Indicative? Yes. Conclusive? Absolutely not. Don't listen too deeply to them.
When this much money is at stake, don't expect to hear the truth from any angle associated with these companies. Remember, we're dealing with marketers and lawyers here...
Most major vendors have been "optimizing" for Quake III ever since it became the informal benchmarking standard... I think Futuremark has blown the issue up a little since they weren't on really good terms with nVidia before this started.
But really, 3dmark has always been a "gee-whiz" pretty demo of current graphic card abilities, but never a reliable benchmark. In fact, no one program/game can be a reliable benchmark, since performance must be judged on a variety of applications. Only then do you get some kind of idea of where the "real world" performance lays.
The competition between ATI and nVidia is good for us customers; they both have excellent cards now. ATI has the fastest, while nVidia's drivers (yeah linux support is flakey I know) seem a bit more stable than ATIâ(TM)s.
Really, between the two companies, it is hard to make a really "wrong" choice.
So yeah, everyone, these aren't the droids you are looking for, you can go about your business...
"To confine our attention to terrestrial matters would be to limit the human spirit." -Stephen Hawking
Users want to know how a piece of hardware performs. When a hardware vendor takes a shortcut it improve results against a specific benchmark, it is subverting the purpose of the benchmark and is unethical. By 'optimizing' or 'cheating', NVIDIA simply has created a situation where the benchmark is not indicative of real-world performance, and consumers lose a source of factual data.
It would not surprise me to see that much of this is an attempt by NVIDIA to marginalize the value of FutureMark 3d 2003. If a benchmark isn't favorable to a piece of hardware, then make the benchmark a 3-ring circus with these antics - then nobody trusts the benchmark at all.
A sad way to do business and I can't say when my GF3 Ti 200 will be replaced, but it when it is I will not be using NVIDIA. Apparently they don't trust users to make a decision based on an honest assessment of facts.
How about optimizing the driver for all applications???
Or is that just being silly?
When I want to test-drive a new car, do I build a driving simulator based on the car and load in my route to work so I can see how it would handle during a typical commute? Of course not, I just take the car for a test drive, because sometimes the best way to test something is just to use it.
Benchmarks are inheriently flawed because a benchmark is not what your buy the card for. Card manufacturers are always going to "teach to the test" by optimizing their cards and drivers for whatever the reviewers are using to review the cards. So why not take advantage of this and use popular applications to benchmark? The venerable Quake3 framerate test is one example of this, but I would arge that all benchmarks should be numerical data taken from the performance of real world applications. That way, at least those apps will perform as advertised, and any other apps that take advantage of the same features on the card will benefit.
To be fair, it is the /. crowd that 3dmark appeals to. 3dmark has never been more than a simulation of a game and as such should be considered useless. Games that people actually play are the only real benchmarks as they represent real-world results, the only results that matter. Nvidia optimizes it's drivers for games(benchmarks) and so does every other 3D video card manufacturer, is that cheating? No thats improving performance by tailoring the drivers for a specific enviroment. Thats optimization. Is compiling for athlons cheating? Course its not.
The only cheating is on futuremark's part for selling a product that they claim is a valid benchmark for 3d gaming when it is nothing of the sort.
[Just Shut Up and Do What I say]
Absolutely! I can't believe that some people are saying this isn't cheating!! Here's the deal, prior to any optimisation/cheating... App programmer tells card to render something -> Card renders it the way app programmer tells it to -> User sees what app programmer intended. After optimisation... App programmer tells card to render something -> Card renders it the way app programmer tells it to, but faster -> User sees what app programmer intended. After cheating... Programmer tells card to render something -> Driver programmer decides that, actually app programmer doesn't know what he/she is talking about and shouldn't have told it to render the thing that way and that they know a much better way to render it -> Card renders it the way the driver programmer tells it to, which (surprise!) is faster -> User sees what driver programmer intended. The point is that what the app programmer and driver programmer intended are different things. This, in itself is not a cheat. The cheat comes in when the driver programmer doesn't tell people about the change and instead let's people think that a difference in FPS between competitor cards is because of differences in power, rather than differences in what they are trying to render. It's a matter of trust that graphics cards render things the way the app tells them to. To do otherwise is cheating. Plain and simple.
This is not an nVidia problem? This is a benchmark problem, because they weren't wily enough to prevent nVidia from cheating tremendously and repeatedly?
People are crying about this because they rely upon benchmarks as a gague of how powerful a card is, and make purchasing decisions around such knowledge. Sure, some of them forget that a %5 difference is meaningless in real-world performance, but that doesn't mean that the overall scores are meaningless. nVidia's last round of cheats pretended that the card was 25% faster overall than it actually was. This particular cheat ads between 8 and 18 percent to the total, with the largest false total going to the most expensive card.
In other words, if you bought a $300 nVidia card on the strength of this benchmark, you bought a card that is %40 slower than it should be because nVidia went out of its way to lie about the speed.
ATI optimized for the test, they re-ordered the way in which the card handled executions similar to the way someone might re-order their day for maximum efficiency. It was a cheat, but a minor one that only added 2% to the score. nVidia's cheat involved dropping instructions entirely, equivalent to doing more in a day by checking things off your list without actually doing them, letting the food rot in the kitchen and the dirty laundry pile up.
The sad fact of the matter is that nVidia now has a big problem, in that their fastest card which managed to eeek out ATI's fastest cards can no longer claim that crown, and yet it is ATI's turn next to introduce faster cards. Their technology can compete, but can't demand the premium that graphics card developers rely upon to survive. Furthermore, this cheat comes after nVidia promised to clean up their act and remove all cheats from their driver. Not only did they cheat, they promised to clean up their act and yet cheated again in the very driver that is supposed to be clean. Their public image is bloody shot, significantly worse than ATI's was over their Quake 3 debackle. ATI fell back on their technology and released superior cards, but nVidia doesn't seem to be able to head down that road.
To get back to the poster's original position, the benchmark should try to outsmart the developers, though being a test for future games they have to stay abreast of display technology more than cheating techniques. But blaming the benchmark not the nVidia for cheating is like discovering that during a Car-and-Driver top-test the Ford team took a shortcut shaving %40 off of the race course and congratulating them on their ingenuity.
What they did was indefensible. They knew it, they appologized, and they did it again.
The ______ Agenda
Why's that sad? Who really cares? Most people just want a decent card, and either don't care enough or don't have time to devote themselves to some absurd quest to find the perfect graphics card. You'll never get the latest technologies in a consumer-grade product, so as long as you're getting a good price and it works, why should they care? Stop making this out to be some monstrous injustice, because it's not.