Slashdot Mirror
Futuremark Replies to Nvidia's Claims
Nathan writes "Tero Sarkkinen, Executive Vice President of Sales and Marketing at Futuremark, has commented on the claims by Nvidia that 3DMark2003 intentionally puts the GeforceFX in bad light, after Nvidia had declined becoming a member of Futuremark's beta program. This issue looks like it will get worse before it gets better." ATI also seems to be guilty of tweaking their drivers to recognize 3DMark.
People keep begging that nvidia release their drivers under a open license. Well i guess we now know why they don't. /Esben
"Nobody really checks their email any more. They just delete their spam"
This is why you need many forms of evaluations to properly test something. Just running one program to show you pretty pictures is not going to give any meaningful result. You need to stress test the card in other ways.
And, since one of the main reasons people will buy this is to play flashy and pretty games, ignoring the performance in those games is rediculous.
3dMark isn't a standard. It's a business, who makes money by charging hardware developers LOTS of money to be included in their "BETA" program. In real life(TM), manufacturer-specific optimizations matter. Many games will look better and run faster if they use vendor-specific OpenGL extensions, for instance. For a gamer looking to buy the fastest card to run his favorite game, he should look for benchmarks on that game. FutureMark is trying to make a business by predicting behavior of games that aren't out. Well, either the game you want to play is out or it isn't. If it's out, buy your card based on benchmarks for it. If it's not, wait until it's out before you spend your money. There is no guarantee that 3dMark is a good predictor of DirectX 9 performance.
When Quake III runs at 300 FPS on my system under my 9700 Pro with 4x AA, I could care less about 3DMark and what ATI or Nvidia tweak. If the games run smooth and they look good, then go with it. Truth is, the ATI looks better than the Nvidia card under QIII, WCII, JKII, and pretty much everything else I've been playing.
The issue with low FPS is a game problem 9 out of 10 times. The faster the video card, the less the game development houses work to streamline and improve their framerate.
Open Source benchmarks are a bad idea because Closed Source drivers can still be used which may or may not contain these "cheats".
Better to have open source drivers so we can inspect the driver for cheating/optimisations.
Infact, open source them all if the hard numbers are that important!
All this does is make 3DMARK look worthless as a benchmarking app. All it has now is some value as a pretty looping demo or stress testing application. I run it to make sure the card works and the drivers are installed properly (as in runs all tests) and thats it. The little number it spits up at the end is worthless.
I dont even bother with 3DMark scores when I read reviews, I skip straight to the tested games and get a look at the FPS at various levels of detail.
Then it's easy to realize that card A gives 201 FPS, card be gives 199 FPS, and the answer is: buy whichever is cheaper.
This gives me much more useful information that relates to what I want the card for - playing games.
I don't need no instructions to know how to rock!!!!
If it was a generic optimization (and probably should have been), there'd be no issue.
ATI recognized the 3dmark executable and special cased for it. Which is misleading and wrong. The performance is enhanced for 3DMark and 3DMark alone.
I don't need no instructions to know how to rock!!!!
Rewriting shaders behind an application's back in a way that changes the output under non-controlled circumstances is absolutely, positively wrong and indefensible.
Rewriting a shader so that it does exactly the same thing, but in a more efficient way, is generally acceptable compiler optimization, but there is a range of defensibility from completely generic instruction scheduling that helps almost everyone, to exact shader comparisons that only help one specific application. Full shader comparisons are morally grungy, but not deeply evil.
The significant issue that clouds current ATI / Nvidia comparisons is fragment shader precision. Nvidia can work at 12 bit integer, 16 bit float, and 32 bit float. ATI works only at 24 bit float. There isn't actually a mode where they can be exactly compared. DX9 and ARB_fragment_program assume 32 bit float operation, and ATI just converts everything to 24 bit. For just about any given set of operations, the Nvidia card operating at 16 bit float will be faster than the ATI, while the Nvidia operating at 32 bit float will be slower. When DOOM runs the NV30 specific fragment shader, it is faster than the ATI, while if they both run the ARB2 shader, the ATI is faster.
When the output goes to a normal 32 bit framebuffer, as all current tests do, it is possible for Nvidia to analyze data flow from textures, constants, and attributes, and change many 32 bit operations to 16 or even 12 bit operations with absolutely no loss of quality or functionality. This is completely acceptable, and will benefit all applications, but will almost certainly induce hard to find bugs in the shader compiler. You can really go overboard with this -- if you wanted every last possible precision savings, you would need to examine texture dimensions and track vertex buffer data ranges for each shader binding. That would be a really poor architectural decision, but benchmark pressure pushes vendors to such lengths if they avoid outright cheating. If really aggressive compiler optimizations are implemented, I hope they include a hint or pragma for "debug mode" that skips all the optimizations.
John Carmack