Slashdot Mirror
More on Futuremark and nVidia
AzrealAO writes "Futuremark and nVidia have released statements regarding the controversy over nVidia driver optimzations and the FutureMark 2003 Benchmark. "Futuremark now has a deeper understanding of the situation and NVIDIA's optimization strategy. In the light of this, Futuremark now states that NVIDIA's driver design is an application specific optimization and not a cheat."" So nVidia's drivers are optimized specifically to run 3DMark2003... and that's not a cheat.
Time to update Websters. Cheat just got new semantics.
I think [H]ardOCP stated it best as "Futuremark didn't want to get sued by Nvidia". Nvidia has the legal and financial resources to totally ruin Futuremark and they know it.
And now Futuremark has totally invalidated their own benchmark software by declaring it "open season" for hardware manufacturers to distort the "tests" in any way shape or form they desire to make the numbers higher.
N.
"Nothing strengthens authority so much as silence." - Charles de Gaulle
Looks like someone is scared of somebody elses lawyers. Yuck! This is obviously Futuremark trying to appease Nvidia.
That is the way it sounds, isn't it?
"Application-specific optimization". . . In other words, "We're not cheating, we're just adding code to our driver to make sure our card works really well with benchmarking software." Of course, if it works better with benchmarking software than it does with real-world applications, that is cheating, isn't it?
It actually reminded me of the axiom, "That's not a bug, it's a feature!"
!#@%*)anks for hanging up the phone, dear.
Cripes already. No one even BOTHERS with #DMark anymore, and after this fiasco no one is ever going to bother with them again. Gamers will use REAL EVERYDAY GAMES to see what runs the fastest again. Looking at some goofy simulation app coming up with scores and people buying into the company and people tricking drivers for particular tests is just crappy and makes 3DMark 100% invalid to any of my concerns in the future. I will only trust reviews that benchmark the latest and greatest games that I will be buying these cards for, whoever can run them fastest at that particular time IS WHAT IM GOING TO BUY. Peroid. Enough of this 3DMark BS.
So the question in my mind is did Futuremark learn something from the discussions? Is there something it was ignoring in its tests?
I'm trying to not be a cynic and assume a big fat envelope was passed under the table. That what Nvidia did was legitimate.
Those that are following this, should check the pictures on the previous article. The quality of the nvidia "optimized" version sucked (showed big artifacts). That's no optimization, unless there was no image quality loss.
"This card is optimized for quake as long as you follow the left trail, the right trail will just look like crap but nobody follows it anyway".
If a benchmark doesn't measure performance related to real-world applications, what's the point? If a driver is optimized to run a benchmark faster, that SHOULD mean that the real world apps should run faster, too. If not, the benchmark is useless.
If you want to, you can prerender the whole fucking test, stick it in your driver and just play it off instead of actually rendering when Futuremark is running, that would be an "application-specific optimization" too.
The benchmark is ment to reflect performance in the actual game, the reason it takes the same path is merely to make the results comparable. What ATI was something the game *could* have achieved in game, if the operations were properly sequenced. What Nvidia did is to fake a performance it can't actually give if a person had followed the exact same path in the game. That is cheating.
It is pathetic by Nvidia, and it's pathetic by Futuremark to present this press statement. Get some backbone and integrity.
Kjella
Live today, because you never know what tomorrow brings
Now that all the latest games have benchmarking modes, what do we need FutureMark for?
If NVidia wants to do application-specific optimizations that make UT2003 go faster, then that would be great. That's what they should be doing. Those are optimizations that genuinely benefit the user.
Is this truly the only Earth I can live on?
It's not an optimization if it does not produce the same results! Recall that the shader code that the driver used did not produce the same visual results as the shader code it replaced.
... based purely on application-specific pattern matching, which by its very nature is fragile. As was demonstrated so aptly by the 'off the rails' mode in Futuremark. This isn't an accidental bug: it is obvious that such mechanisms are highly fragile, and are almost certain to cause bad rendering on these applications when they are modified in small ways.
More tellingly, the driver deliberately flaunts the D3D spec by omitting buffer clears, mucking about with clip planes, etc.
As others have said, Futuremark's statement is just covering their legal arse. If someone modifies their code to get better scores in some benchmarks while introducing deliberate bugs (i.e. incorrect rendering), it's a cheat in my book.
Well, after seeing the future mark scores, it looked to me as nvidia fx chips are blowing away ati and its gf4 line.
But I found a really nice german benchmark site 3dcenter.org that had to be the best benchmarks ive ever seen, they actually use the games on each and lists the fps.
Looks like the FX/GF4 5200/4200 4600/5600 (non ultra) are the same. And the ATI 9700PRO/9800 are faster than the 5800 Ultra.
After reading these benchmarks, you can really tell nvidia tweaked the SHIT out its drivers for futuremark...
NVidia did things that were clearly NOT legitimate, and FutureMark caught them at it. There's a PDF report on FutureMark's Web site (assuming it hasn't met with an "accident" by now) detailing the dirty deeds. Chief among them, IMHO, was a trick where the driver was supposed to draw and update positions of stars in a night sky (involving clearing the background) as one moved along a 3D path; if one stays on the exact preprogrammed track of the demo, it looks OK. BUT... if you turn around (possible in the beta mode of the benchmark) you see that the driver SKIPPED clearing the background; the stars smear like mad. There is NO POSSIBLE WAY their driver was behaving legitimately. (Especially since changing the benchmark's fingerprint oh-so-slightly caused all these quirks to vanish; they were detecting the demo and screwing with things if it was being run...) The rest is just fear-of-pissing-off-the-800-pound-gorilla. A FutureMark developer admitted as much in a newsgroup posting. Sigh...
"My strength is as the strength of ten men, for I am wired to the eyeballs on espresso."
All video card drivers have algorithms to keep from rendering unneccessary portions of the map. What NVidia did, I believe, was to bypass those algorithms and hard-code into the driver the portions that needed to be rendered. That wouldn't work in a real game, where the card must decide what to render at run-time based on user input. Therefor, it's cheating, no different from including an MPEG of the entire 3Dmark demo and showing it in lieu of actually rendering it.
It's hard to be religious when certain people are never incinerated by bolts of lightning.
What a load of crap. This is one of those things that when you think about it too much a bunch of false lines of logic get drawn and you come up with a nonsensical answer. Either that or 3DMark is trying to avoid a lawsuit from nVidia, which no doubt has been threatened.
The point of a benchmark is to test dissimilar systems against common references to get an idea of how they perform against each other in such a way that you have an apples to apples comparison.
If 3DMark writes their program in a way that allows optimization paths for a specific GPU, then it is no longer a benchmark.
You now no longer have an idea of how fast the card REALLY runs as there is no guarantee that game writers will use GPU-specific optimizations. It's the same thing as MMX...Nobody sees the benefits if it's not hardcoded into the program, so what's the point if being uberfast in a benchmark if you won't necessarily see the same results in the real world?
-R
I guess you just need to buy another card at that point.
That seems a bit more appropriate to the story, doesn't it.
There is nothing so silly as other peoples traditions, and nothing so sacred as our own.
Don't get me wrong, there is nothing wrong with application-specific optimizations.
But this misses the whole point of 3dmark 2003. Different developers stress the pixel and triangle pipelines in different ways to produce a whole boatload of effects. While major games and engines are often optimized-for, there is no guarantee that ATI or Nvidia will sit down and optimize for the game you just bought.
That said, 3dmark 2003 should be considered a relational tool for generic perfrormance. Consider it a good bet that if two cards perform similarly and acceptably, the two cards should be able to run almost any DX8/DX9 game off the shelf acceptably.
The fact that Nvidia's unopitmized drivers perform significantly behind ATI's unoptimized drivers in 3dmark 2003 raises a significant question:
We all know how well the 5900 does in Quake III, Serious Sam 2, UT2003, etc, but how does it do in ?
I want to know that if I take *insert random DX8 game here* home to play, IT WILL PERFORM WELL. That is the entire point of having a benchmark like 3dmark. To do application-specific optimizations for it is to nullify the entire point of the benchmark.
Man is the animal that laughs.
And occasionally whores for Karma.
It's not like saying 1+1 = 3. It's more like saying what's 7+7+7+7+7+7? Well, it's the same as 7*6, but guess which one is faster to calculate?
It's more like saying "What's 7+7+7+x+7+7?" For the benchmark program, x happens to be 7, leading to 7*6, but the general case is actually 7*5+x. No attempt is made to check an arbitrary game to see if x is 7. It only applies the optimization if the executable is a particular benchmark.
__
Do ya feel happy-go-lucky, punk?
I would mod this down, but I'd rather argue (so much more fun! ;-) ). As someone who has worked on a very large game (team of >50, sold 500K copies so far), someone who works at a small studio now (20 people), and someone who develops at home on the side, (whew) I can say that size and clout has little to do with how much attention video card manufacturers are willing to give you. All that really matters is that they see an interesting prospect, and a way for their card to look "better."
I currently have no clever signature witicism to add here.