Slashdot Mirror
New DX10 Benchmarks Do More Bad than Good
NIMBY writes "An interesting editorial over at PC Perspective looks at the changing status between modern game developers and companies like AMD and NVIDIA that depend on their work to show off their products. Recently, both AMD and NVIDIA separately helped in releasing DX10 benchmarks based on upcoming games that show the other hardware vendor in a negative light. But what went on behind the scenes? Can any collaboration these companies use actually be trusted by reviewers and the public to base a purchasing decision on? The author thinks the one source of resolution to this is have honest game developers take a stance for the gamer."
What happens when a better way than the "known standard" comes around. Are we supposed to wait for some updated standard then updated hardware for that standard but by then don't you think some part of that standard will be obsolete?
In this case, DX10 (well, strictly D3D10) *is* the standard you're talking about. Waiting for a better way would involve waiting for D3D11 or similar. That's not what the OP's talking about.
Yes, there are some stark exceptions where a different driver can have substantial impact, but this is often the game developer's fault as much as the hardware developer.
I don't know about games, but I remember NVidia being caught cheating at 3DMark a couple of years ago. They released a driver that deliberately cut corners when it detected that that benchmark was being run, massively improving the framerate.
That's not so easy with games, but quite often optimisations can be made when you code for a specific case that can't (or shouldn't, for performance reasons) be made when coding more generally. I wouldn't put it past either vendor to tweak their drivers for say Half Life 3 at the expense of other, less hyped titles.
It's official. Most of you are morons.
You say "Drop the whole 'optimized' drivers crap", then in the next sentance you say " I shouldn't have to wait for Nvidia to tweak their drivers to get the best performance...". You're contradicting yourself. Obviously you want the better performance that can be gained through the tweaking.
Driver manufacturers try and get the generic code paths as fast as they can, but they can always make the driver a little bit faster by applying some domain-specific knowledge. If they know that a particular game has a particular hot path, they can optimize that path. Maybe the optimization that they use wouldn't make sense for the general case, but they know it will work in that particular case.
Sure it would be nice to have a card that was great in everything, but there will always be a way to make it just a little faster for that one special case....and we're back to the current scenario.
Actually, it already exists.
Twice.
You can benchmark in existing, released games. Those are the real application and as such the performances observed are relevant. The fact that their current driver is optimized or not does not matter, this is the current status.
Of course, as the author talks about pre-release, that does not apply here. The author is quite surprised that *BETA* code of incomplete demos runs better on the hardware of the company that helped the game developer. No foul play here.
In the same vein, you have 3dmark. Those are benchmarks developed in collaboration with ALL the players in the industry: AMD, Nvidia, Intel... They all have plenty of time to optimize their drivers and find bugs or non-optimized path. As this is a synthetic benchmark, results are less relevant.
So, in summary, the author complains that beta code is non optimized for all hardware.
A stupid complaint while the solution already exists: benchmark with known benchmark and final code.
They made the specifications slicker and more exact, but did't add any new breakthrough features. Not contradicting at all really.
It's like "they reduced the needs for annual service on the car by 10%, but didn't add a turbocharger".