New DX10 Benchmarks Do More Bad than Good

NIMBY writes "An interesting editorial over at PC Perspective looks at the changing status between modern game developers and companies like AMD and NVIDIA that depend on their work to show off their products. Recently, both AMD and NVIDIA separately helped in releasing DX10 benchmarks based on upcoming games that show the other hardware vendor in a negative light. But what went on behind the scenes? Can any collaboration these companies use actually be trusted by reviewers and the public to base a purchasing decision on? The author thinks the one source of resolution to this is have honest game developers take a stance for the gamer."

Comments

The answer

[beavis88]

Can any collaboration these companies use actually be trusted by reviewers and the public to base a purchasing decision on?

No. There is some room for an "Unless..." argument, but frankly, "reviews" like this are so biased that no sane person should knowingly take them into account while evaluating a purchase. Unless (hah!) it's as a strike against the companies doing it. But you're screwed on both sides, there, so...

Re:John Carmack

[peragrin]

Use OpenGL.

quit already with 'optimized' drivers

[BattleTroll]

I'm getting tired of the back and forth between AMD and Nvidia. Drop the whole 'optimized' drivers crap and give us cards that work great out of the box. This entire trend of releasing per-game tweaked drivers is just hurting consumers. I shouldn't have to wait for Nvidia to tweak their drivers to get the best performance out of one of their cards. I shouldn't have to download new drivers every time a new games comes out. The whole reason you create your cards based on a known standard is to avoid this mess.

Stop fucking around and do it right the first time.

How hard is that?

Re:quit already with 'optimized' drivers

[LWATCDR]

Do it right the first time? Spoken like someone that has never written software.
Every time you write a piece of code you improve it you may find a new way of doing something.
Also as more and more programs come out that use the driver they people that write them will gain a better understanding of how they are used. That will help them optimize a code path. It could be as simple as selecting which branch tends to be used more and making that the default path.
The something is used the more performance you can get out of it.

Driving people to consoles

[SuperKendall]

It's exactly aspects of PC gaming like this that drove me to consoles. Then they can do all the per-game tweaking they like, it's not me doing the work.

Shows how bad DX-10 really is

[Nom du Keyboard]

To me, this just goes to show what a bad standard/interface DX10 really is. Looks to me like if you make calls to it in one way, ATI shines, but call it another way and its all Nvidia -- yet both cards+drivers allegedly comply with the standard. It sounds like trying to compare floating point benchmarks on AMD Athlon versus Intel Core 2. Depending on how you arrange the numbers and call the various floating point extensions can make all the difference.

And there's no indication here if someone is using corked drivers that favor one game over the other.

What I'd like to see is a benchmark rundown of each function in DX10, along with some realistic estimate of how much each function is called in normal game play. If different games favor different functions, then say so. Only then might I have some idea of how the two graphics powerhouses measure up against each other.

And if you have some reasonable way of testing common sequences of calls, show that as well.

Re:Shows how bad DX-10 really is

[KillerCow]

What I'd like to see is a benchmark rundown of each function in DX10, along with some realistic estimate of how much each function is called in normal game play. If different games favor different functions, then say so. Only then might I have some idea of how the two graphics powerhouses measure up against each other.

... or you could just benchmark the card running popular retail games.

Re:Shows how bad DX-10 really is

[Z34107]

Looks to me like if you make calls to it in one way, ATI shines, but call it another way and its all Nvidia -- yet both cards+drivers allegedly comply with the standard

Both cards do do the same thing.

Just that some engineer at nVidia thought of a genius way to, say, handle antialiasing, whereas some ATI engineer came up with a genius way to do T&L. (Just pulling these examples out of my Canada, but you get the idea.) Point is, chips designed by different people at different companies will perform differently.

It shouldn't be surprising that when two cards take different routes to the same pixel-shaded destination, that one card gets there faster.

The problem comes in when nVidia and ATI loan out "consultants" to help game development companies code. Sure, people who built the videocard your game will be running on from the ground up can have some pretty useful ideas, but then you end up with a big green "nVidia" logo on your game's splash screen and better performance on nVidia cards than ATI.

Does it really matter?

[Xest]

I'm not sure benchmarks really matter. It's not as if either of the cards are so bad that you're getting screwed by buying one instead of the other.

I've been using dedicated graphics cards since my old 3dfx Orchid Righteous 3D, since then I've had various ATI/nVidia cards and I've never been in the situation where I've thought "damn I wish I bought the other company's card".

I used to be someone that thought it was great to get 3 more fps than the other guy but when I came to realise that whilst I got 3 more fps in one game, and he got 5 more fps in another game that was OpenGL instead of DirectX or whatever. It became obvious that it's not as clear cut as one card is better than the other in terms of frame rates, it depends on the graphics API, the driver releases, the OS, the other hardware in the system, the game settings and on and on. Personally I prefer nVidia, but that's only because they have better developer support and I've had a better experience with the quality of their drivers over ATI's, image quality, features and frames per second has never once been an issue for me and I'm sure this is true for all but those people who think that getting an extra 3 more fps in game X actually makes the blindest bit of difference in the world.

Is it the DX 10 code or vista?

[Joe The Dragon]

vista will look very bad if that dx 10 for XP hack comes out and it turns out to be faster.

I don't see a conspiracy here.

[shaitand]

The simplest explanation doesn't require any malice on the part of the video card manufacturers. If the developers and engineers develop the cards and drivers to optimize the features they believe the most important for performance, it stands to reason they will think those same features the most important when collaborating on a benchmark program. Magically, the benchmarks will score heavily in favor of the features that camp optimized their hardware and drivers for.

Since graphics technology is actually a fairly complex field and the design philosophies of these two companies are different, the other companies cards/drivers will be optimized for what THEY feel are the real performance metrics and therefore they won't test as well on those benchmarks.

All of this can happen without anyone doing anything but coding and designing in the manner they believe to be the best balance of technology and practicality.