Slashdot Mirror
More on Futuremark and nVidia
AzrealAO writes "Futuremark and nVidia have released statements regarding the controversy over nVidia driver optimzations and the FutureMark 2003 Benchmark. "Futuremark now has a deeper understanding of the situation and NVIDIA's optimization strategy. In the light of this, Futuremark now states that NVIDIA's driver design is an application specific optimization and not a cheat."" So nVidia's drivers are optimized specifically to run 3DMark2003... and that's not a cheat.
Yes, there is an increase in speed, but there was also a degradation in quality in the test. See here.
That's the whole issue...
"I turn away with fright and horror from the lamentable evil of functions which do not have derivatives."
Total BS, 3DMark2003 is already geared toward specific vendors. I've personally analyzed the data from the driver (since I'm writing one), and they totally favor ATI with the heavy use of PS 1.4 shaders. In fact, the data changes completely if PS 1.4 support isn't claimed. (3x more geometry is sent)
Also, PS 1.4 shaders don't always translate 'up' to PS 2.0 hardware very well, which is why (IMHO) Nvidia started all this hub-bub in the first place.
The only vendor that natively supports PS 1.4 is ATI.
They should have created PS 1.1 shaders for the masses, and then if 2.0 hardware is detected, had 2.0 shaders for everything.
And their "DX9" onyl test is a piece of crap too. They use one or two new instructions in the VS, and PS2.0 is only used for the sky. big whoop. No branching in the VS, two sided stencil, or anything cool.
It's sad the OEMs put alot of stock in 3Dmark, they don't seem to realize that gamers play games all day, not benchmarks.
...and that's not a cheat.
Well, they are right, it's not a cheat. FutureMark, however, has just admitted that as a performance bench mark that might indicate suitability for other tasks their rating sucks.
I think you don't understand what was done. Nvidia cheated on the benchmark because they new exactly what needed to be rendered. This kind of "optimization" won't help any other application, but it doesn't mean the benchmark is bogus. The same thing could be done by putting optimizations in for time_demo (or something similar) in quake. It will help that specific demo, but it won't help quake.
Jack Valenti and Orrin Hatch will be first up against the wall when the revolution comes.
That would be worse than useless. No two test runs could be usefully compared, so the utility of the benchmark itself is called into question. You'd be reduced to aggregation of test results in the hope that they statistically settle out towards some sort of 'true' average for a given card/driver combo.
The inherent value of a benchmark is the notional "apples to apples" comparison, and you've taken even that away. There would now be no reason at all to use 3DMark.
"So on one hand, honey is an amazingly sophisticated and efficient food source. On the other hand it's bee backwash."
What would be great...
If someone was to reverse engineer the drivers, remove the "Optimisation", recompile and compare results. See what percent the "Optimisations" fudged the results.
Don't have to. In the previous story, they stated how they simply removed the condition that the driver used to switch on this optimisation and, as you said, saw what percent the optimisation fudged the result.
Snooze and you lose your sushi.
But did you also not read that ATI CHEATS? Oh, that's right -- ATI and FutureMark are on good terms. There's three kinds of lies in this world -- lies, damn lies and benchmarks.
I used up all my sick days, so I'm calling in dead.
I think the idea was to test new technologies that haven't been implemented yet in Quake 3 or Unreal Tournament 2003 (like in the upcoming DOOM III).
A quote of a quote in their 10/26/98 press release:
Quack, quack.
actually, Gates did say something similar, but the "640K ought to be enough for anyone!" quote itself is misquoted, and I could cite a source if I had the stupid book with me. Stephen Manes' and Paul Andrews' "Gates" has the actual quote in it, (if someone has the book could they please post it? mine's packed away. thanks.) And in all fairness, the quote is taken out of context...it was said in like 1980 or 1981 when people were coming off the CP/M-80 machines with 64K memory onto the brand spankin' new 16-bit 8088-based IBM PCs.
My journal has hot
I can sort of see the argument here, but it basically ruins the point of having a standard interface like DirectX
Shrug... welcome to reality. DirectX, OpenGL, etc. don't properly model the hardware in some cases leading to much worse performance than should be available.
It's not like saying 1+1 = 3. It's more like saying what's 7+7+7+7+7+7? Well, it's the same as 7*6, but guess which one is faster to calculate?
And it's not quite like that either, I know, because the bit that FutureMark is tentatively agreeing with is Nvidia changing the shader precision, which can lead to a loss of quality (so maybe it is 1+1 = 1.999999999998).
ATI did pretty much the same thing with their drivers, leading to a much slimmer 1.9% improvement. Of course, it's unclear how much of Nvidia's improvement was from the shader changes (which FM is considering) versus from the other modifications they made (like clipping issues). The latter points are not in question by FM - they are cheating.
That's not in question though... what's in question is NVidia's changing of the shader precision from FP32 to INT12. It's entirely reasonable that this is a tradeoff that would be made to improve speed at a slight cost to render quality -- since the speed improvement can be substantial.
Not clearing the back buffer and other "on rails" cheats are still classified as cheats. It's merely the shader changes that are being considered as possibly OK. Which, btw, is similar to what ATI did. But I don't know that ATI's changes sacrificed visual quality at all. (Not that they haven't done that in the past...)
One key point, though: NVIDIA's shader precision is much higher than ATI's at its highest settings. NVIDIA's middle precision, however, is lower than ATI's maximum. This means that comparing the performance of the two is kind of a crap shot, because you can't configure the two to use the same precision.
A deep unwavering belief is a sure sign you're missing something...
Problem is, NVIDIA didn't just optimize. Their application specific "optimization" made the images look worse. And when you couldn't notice it, it was because they were clipping outside the camera angle (becuase they knew exactly where the camera was, something they can't do when you're playing UT2003)
Like the original statement by futurmark said, optimization is great. But when you change the image intended by the software designer in order to make it go faster, that's not an optimization. For god's sake, I can turn all the details to low on UT2003 and get it to go faster, but that's besides the point
Reason games specific benchmark don't fly for me (although now that Futuremark has issued this statement, I'm sure ATI will start cheating as well, making the whole thing useless) is that synthetic benchmarks can test features new to the cards that games may not yet have implemented. So I have an idea how it'll perform with future games.
Warning: Opinions known to be heavily biased.
And he also said flat out that Nvidia's "optimization" was a cheat. By throwing away data in a manner that was not possible from the data given by the engine the Nvidia engineers cheated, a real world application would never make those optimizations because as soon as the viewpoint changes the optimization is null and the results are incredibly horrible images.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
> Of course, if it works better with benchmarking software than it does
...to such a breath-taking
:-)
> with real-world applications, that is cheating, isn't it?
I like it when card manufacturers optimize their driver to achieve high
Quake III Arena frame rates, because coincidently Quake III Arena is my
favourite game (and actually the ONLY game I play). I don't care if the
drivers are good by thoughtful design, or by re-engineering the Q3A code
path and then constructing a driver that is an exact fit (possibly with
penalties for other games).
While your favourite game probably is not "Futuremark", I can see that a
lot of people (including buyers) are happy when their card performs well
with that application. So, I won't consider this cheating.
Take advantage of this race: promote your own favourite game until it is
popular enough to be included in mag benchmarks. Then the card manufacturers
will optimize their drivers for YOUR game, too!
degree, that others will bash on them for "cheating".
Marc
PS: Hey, this FPS shooter cheats, too. It's not modeled down to the molecule,
but rather makes up the characters from triangles. Bah, CHEATING!
An "application specific optimisation" still displays a picture that is of adertised quality, still crunches all the numbers. A "cheat" drops the quality to bump up the frame rate when it sees a particular application running, without reporting the drop in quality.
Nvidia's card produces the advertised quality at all times, it just understands what 3Dmark2003 is trying to do and knows how to do it particularly well. It's a fine line they didn't cross.
Guess you should read the entire article.
Future Mark build 320 vs. 330. (330 doesnt have the nvidia cheat...)
3,215 - ATI Radeon 9800 Pro 256MB w/ 3DMark03 Build 330
2,821 - Nvidia GeForce FX 5900 Ultra w/ 3DMark03 Build 330
Nvidia using the cheat had - 3,458.
The ATI 9800 Pro is faster.