Slashdot Mirror
Futuremark Replies to Nvidia's Claims
Nathan writes "Tero Sarkkinen, Executive Vice President of Sales and Marketing at Futuremark, has commented on the claims by Nvidia that 3DMark2003 intentionally puts the GeforceFX in bad light, after Nvidia had declined becoming a member of Futuremark's beta program. This issue looks like it will get worse before it gets better." ATI also seems to be guilty of tweaking their drivers to recognize 3DMark.
It won't be fast enough next year.
People keep begging that nvidia release their drivers under a open license. Well i guess we now know why they don't. /Esben
"Nobody really checks their email any more. They just delete their spam"
This is why you need many forms of evaluations to properly test something. Just running one program to show you pretty pictures is not going to give any meaningful result. You need to stress test the card in other ways.
And, since one of the main reasons people will buy this is to play flashy and pretty games, ignoring the performance in those games is rediculous.
The problem isn't that benchmarks lie. We all know they do. The problem is we don't know how they lie. Creating open source benchmark applications can show how the driver is excirsed so everyone who wants to know or learn where cards and drivers are strong and weak. Everyone is on the level if everyone can look at the code that came up with numbers. Not to mention there are things to learn from code in benchmarks that excirse the fringe elements of graphics cards and drivers.
The alternative is what we have now: hand waving voodoo. Not only do we have to take the vendor's word they aren't monkeying around with the driver to match execution of the benchmark but now we have to question where the aligence of the benchmark makers.
3dMark isn't a standard. It's a business, who makes money by charging hardware developers LOTS of money to be included in their "BETA" program. In real life(TM), manufacturer-specific optimizations matter. Many games will look better and run faster if they use vendor-specific OpenGL extensions, for instance. For a gamer looking to buy the fastest card to run his favorite game, he should look for benchmarks on that game. FutureMark is trying to make a business by predicting behavior of games that aren't out. Well, either the game you want to play is out or it isn't. If it's out, buy your card based on benchmarks for it. If it's not, wait until it's out before you spend your money. There is no guarantee that 3dMark is a good predictor of DirectX 9 performance.
When Quake III runs at 300 FPS on my system under my 9700 Pro with 4x AA, I could care less about 3DMark and what ATI or Nvidia tweak. If the games run smooth and they look good, then go with it. Truth is, the ATI looks better than the Nvidia card under QIII, WCII, JKII, and pretty much everything else I've been playing.
The issue with low FPS is a game problem 9 out of 10 times. The faster the video card, the less the game development houses work to streamline and improve their framerate.
All this does is make 3DMARK look worthless as a benchmarking app. All it has now is some value as a pretty looping demo or stress testing application. I run it to make sure the card works and the drivers are installed properly (as in runs all tests) and thats it. The little number it spits up at the end is worthless.
I dont even bother with 3DMark scores when I read reviews, I skip straight to the tested games and get a look at the FPS at various levels of detail.
Then it's easy to realize that card A gives 201 FPS, card be gives 199 FPS, and the answer is: buy whichever is cheaper.
This gives me much more useful information that relates to what I want the card for - playing games.
I don't need no instructions to know how to rock!!!!
If it was a generic optimization (and probably should have been), there'd be no issue.
ATI recognized the 3dmark executable and special cased for it. Which is misleading and wrong. The performance is enhanced for 3DMark and 3DMark alone.
I don't need no instructions to know how to rock!!!!
I know my problems are in actuality MS/XP refresh rate problems. But it still inexcusable that it exists at all and I can't blame MS completely. It's too implausible, and reeks of a lack of cooperation between video card manufacturers, MS, and game developers.
Video cards have a simple job when it comes to resolution and refresh rate: When using resolution X, use the best refresh rate Y, and if I have to tell it what that is, so be it. They can't do this.
A lot of this is due to games and their poor detection of capabilities, and lack of effort to try for the best refresh rate. However, it's hard to pin it all on them. Games generally don't have trouble detecting if you have EAX capability, or detecting how many axis are on your joystick, or whether you have a third button on your mouse. Sure, I've seen problems in these areas too, but the video card situation feels like someone just invented VGA yesterday and the video card manufacturers are struggling to make it all work.
I'm using the Cat 3.4 drivers. I can set the refresh separately too, but a few times both the ATI driver tabs and the XP display properties reported that I was in one refresh rate, but my monitor OSD said differently. Inexcusable. It was due to that "maximum capability" setting, and as a result it didn't mind lying to me as long as it avoided going over the maximum. Glad it was able to "protect" me.
But that's not the half of it. I set the refresh rate, and when entering a game, it changes, usually back to 60hz. When entering games, the resolution changes a lot, and it seems completely random what it ends up on.
Other problems I'm having include that mode switching in general takes three times as long as the NV card. Switching back to Windows from games results in a very long black screen, and until Cat 3.4 came along, I couldn't switch out of CS/HL at all without crashing the entire OS.
Let me give you an example of my typical day. I set the display capability to 1280x1024 @ 85hz because I want 85hz in CS. In CS, I accidentally had the mode set for 1600x1200. With ATI3.3, it would crash. With ATI3.4, it would actually draw a 1600x1200 screen in a 1280x1024 window. Yep, it was cut off, with part of the screen literally extending off the monitor into the void.
I change the capability to 1600x1200 @ 75hz and play a while. I quit and fire up BF1942, which due to CPU constraints, runs better at 1024x768. But, I'm at 75hz, because I have no way to tell the card that while it only supports 1600x1200 @ 75hz, it does 85hz in every other mode. I have to change the capability to 1280x1024 @ 85hz. BF1942 runs.
I run another game at 1600x1200. Unlike CS, where it drew off the screen, this one would simply blackscreen as a result of trying to go into 1600x1200 @ 85hz (since 85hz is my "maximum" resolution.) I reboot, and the first few times it happened, I looked for game patches before realizing that this stupid ATI driver was the cause.
The constant mode switches between games take several seconds, and perform an odd "screen wiping" effect that reeks of cheesy hardware. The NV switches modes smooth as butter.
I'm scared as hell to hook this thing into my TV. It might try to pump 2048x1024 @ 100hz at it and cause an explosion.
# Erik
"Three of four 3DMark03 demos don't use new DirectX9 shaders at all"
No, but they use shaders which are generally only supported on DX9 cards and a few older ATI cards. Just because you have a PS2.0 card that doesn't mean you have to use PS2.0 if PS1.4 can do the same: why deliberately make more work for yourself by not supporting older cards?
"Three of four 3DMark03 demos use Pixel Shader 1.4 which was introduced with DirectX8.1 and isn't natively supported by NVIDIA cards"
Support for PS1.4 is a requirement of DX9, so if the GF FX is a DX9 card then it supports PS1.4, and your claim is therefore bogus. If it doesn't support PS1.4, then it's not a real DX9 card.
But the key point is...it's only a fucking benchmark. Who cares anyways. Just another reason to never trust benchmark programs. I don't care how well a card performs on a benchmark since I don't PLAY benchmarks.
You'll have that sometimes...
PS1.4 isn't natively supported by *most* nvidia cards. The spec for PS2.0 is such that it's all-encompassing. If you support PS2.0 you support PS 1.4 and PS 1.1. If you support PS1.4 you support PS 1.1, etc.
So this is how it should look, properly:
- Game 1: no shaders at all, only static T&L (DX7-class effects, given comparatively little weighting in overall score)
- Game 2: vertex shader 1.1 and pixel shader 1.4 (natively supported by GFFX, ATI Radeon 8500 and above)
- Game 3: vertex shader 1.1 and pixel shader 1.4 (natively supported by GFFX, ATI Radeon 8500 and above)
- Game 4: vertex shader 2.0 and pixel shader 1.4+2.0 (DX9 cards only, Radeon 9x00 and GFFX)
Nvidia's lack of support for PS1.4 is their own design choice, and now they have to live with it. The GF4 was released after DX8.1 came out, which contained the PS1.4 spec, but they chose not to support it. ATI Radeon 8500 and above have no problem with this because they supported DX8.1 from the getgo, but nvidia did not change and continued their 8.0 support. As was previously mentioned in the article, nvidia was participating in the developer's beta until Dec 2002, well into the development period for 3dm03 and a month after they paper launched the GFFX, so they knew what was going on with the benchmark for a long time beforehand and didn't change their stance for a while. Presumably, as a beta member up until Dec 2002 if they didn't like the choice of PS 1.4 in extensive use, then they could've said something earlier.
The key to regarding 3dm03 is it's goal as a forward-looking benchmark. Both DX8 games and DX9 games are currently in development, and many DX7 games are still in existence (remember, HL2 doesn't require anything above a DX6 card), so in this respect 3DM03 is still fair in its test design.
Rewriting shaders behind an application's back in a way that changes the output under non-controlled circumstances is absolutely, positively wrong and indefensible.
Rewriting a shader so that it does exactly the same thing, but in a more efficient way, is generally acceptable compiler optimization, but there is a range of defensibility from completely generic instruction scheduling that helps almost everyone, to exact shader comparisons that only help one specific application. Full shader comparisons are morally grungy, but not deeply evil.
The significant issue that clouds current ATI / Nvidia comparisons is fragment shader precision. Nvidia can work at 12 bit integer, 16 bit float, and 32 bit float. ATI works only at 24 bit float. There isn't actually a mode where they can be exactly compared. DX9 and ARB_fragment_program assume 32 bit float operation, and ATI just converts everything to 24 bit. For just about any given set of operations, the Nvidia card operating at 16 bit float will be faster than the ATI, while the Nvidia operating at 32 bit float will be slower. When DOOM runs the NV30 specific fragment shader, it is faster than the ATI, while if they both run the ARB2 shader, the ATI is faster.
When the output goes to a normal 32 bit framebuffer, as all current tests do, it is possible for Nvidia to analyze data flow from textures, constants, and attributes, and change many 32 bit operations to 16 or even 12 bit operations with absolutely no loss of quality or functionality. This is completely acceptable, and will benefit all applications, but will almost certainly induce hard to find bugs in the shader compiler. You can really go overboard with this -- if you wanted every last possible precision savings, you would need to examine texture dimensions and track vertex buffer data ranges for each shader binding. That would be a really poor architectural decision, but benchmark pressure pushes vendors to such lengths if they avoid outright cheating. If really aggressive compiler optimizations are implemented, I hope they include a hint or pragma for "debug mode" that skips all the optimizations.
John Carmack