Slashdot Mirror
New DX10 Benchmarks Do More Bad than Good
NIMBY writes "An interesting editorial over at PC Perspective looks at the changing status between modern game developers and companies like AMD and NVIDIA that depend on their work to show off their products. Recently, both AMD and NVIDIA separately helped in releasing DX10 benchmarks based on upcoming games that show the other hardware vendor in a negative light. But what went on behind the scenes? Can any collaboration these companies use actually be trusted by reviewers and the public to base a purchasing decision on? The author thinks the one source of resolution to this is have honest game developers take a stance for the gamer."
John Carmack used to be pretty good at cutting through the marketing crap and telling it like it was. Let's ask him.
occultae nullus est respectus musicae - originally a Greek proverb
Sorry I bit my tounge and I cant pronounce sing properly. What was the Author singing for anyway, shouldn't he have just written it down?
Do not try to read the dupe, thats impossible. Instead, only try to realize the truth
What truth?
There is no dupe
Can any collaboration these companies use actually be trusted by reviewers and the public to base a purchasing decision on?
No. There is some room for an "Unless..." argument, but frankly, "reviews" like this are so biased that no sane person should knowingly take them into account while evaluating a purchase. Unless (hah!) it's as a strike against the companies doing it. But you're screwed on both sides, there, so...
"...have honest game developers take a stance for the gamer."
Yeah, because you'd never hear a hack developer blame all the problems on the hardware, right?
I'd just like to say, "I already knew that".
Under the influence of Post-Cyberpunk Gonzo Journalism
DX10 or for the uninformed, Derendering eXtraction (10 megapixels/second) is a standard benchmark for measuring the performance of GPUs or Gradient Pixilization Units. Pretty much this is what the video card companies all base their prices on with price being directly related to how many pixels can be gradiated per unit (usually about 30 cents per pixel/ounce).
I'm getting tired of the back and forth between AMD and Nvidia. Drop the whole 'optimized' drivers crap and give us cards that work great out of the box. This entire trend of releasing per-game tweaked drivers is just hurting consumers. I shouldn't have to wait for Nvidia to tweak their drivers to get the best performance out of one of their cards. I shouldn't have to download new drivers every time a new games comes out. The whole reason you create your cards based on a known standard is to avoid this mess.
Stop fucking around and do it right the first time.
How hard is that?
i guess this may take away the "poor drivers" back door, eh?
An interesting editorial over at PC Perspective looks at the changing status between modern game developers and companies like ATI and NVIDIA that depend on their work to show off their products. Recently, both ATI and NVIDIA separately helped in releasing DX10 benchmarks based on upcoming games that show the other hardware vendor in a negative light. But what went on behind the scenes? Can any collaboration these companies use actually be trusted by reviewers and the public to base a purchasing decision on? The author things the one source of resolution to this is have honest game developers take a stance for the gamer."
"The author thinks the one source of resolution to this is have honest game developers take a stance for the gamer."
2048x1536 is the ONLY resolution.
How much is your data worth? Back it up now.
It's exactly aspects of PC gaming like this that drove me to consoles. Then they can do all the per-game tweaking they like, it's not me doing the work.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
And there's no indication here if someone is using corked drivers that favor one game over the other.
What I'd like to see is a benchmark rundown of each function in DX10, along with some realistic estimate of how much each function is called in normal game play. If different games favor different functions, then say so. Only then might I have some idea of how the two graphics powerhouses measure up against each other.
And if you have some reasonable way of testing common sequences of calls, show that as well.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
The author things the one source of resolution to this is have honest game developers take a stance for the gamer.
.98 gallons, or broadcasters who aren't sure what frequency they're transmitting on, or any of the much more universal issues that NIST exists to address.
Good idea. We should have politicians to take a stand for the voter. Or criminals to take a stand for the victims. Let's demand that the problems take care of themselves, and then we can go back to not paying attention.
The solution is people paying attention and voting with their wallets. Obviously this is never going to happen -- people have more important things to worry about, and they're not going to stop buying video cards -- so the next best thing might be to establish a division of NIST which focuses strictly on benchmarking hardware. Of course, that's probably not going to happen either, since it's really not that big of a problem, like gas stations trying to sell
At any rate, most of us figured out a long time ago that 2 extra FPS != better game, and 2 less != unplayable unless you're only getting 2FPS to begin with.
https://www.eff.org/https-everywhere
Looking for an honest man with his lamp.
If you want an honest answer, then you get AMD, Intel, nvidia, ATI, and whoever else has an interest in the outcome, and you have them all agree on some kind of standard test.
And, yes, there's a snowball's chance in hell that would happen, but if you want as close to an honest result as you will ever get, that is the only way to do it -- have them fight it out together rather than independently try to "game" the result. Oh, then they'll tweak things to fit the benchmarks, but it will be a little better.
They all know that one or the other's strategy favours particular types of tests, and skew the tests in a way that will make their solution look substantially better. If they can't agree on a more balanced standard together, then trust NONE of the results, regardless of who is making the claim.
I miss the days of arguing about whether the Diamond3d or the STB drivers were the best and whether the 8mb sli or 12mb sli was the better price to performance ratio
I'm not sure benchmarks really matter. It's not as if either of the cards are so bad that you're getting screwed by buying one instead of the other.
I've been using dedicated graphics cards since my old 3dfx Orchid Righteous 3D, since then I've had various ATI/nVidia cards and I've never been in the situation where I've thought "damn I wish I bought the other company's card".
I used to be someone that thought it was great to get 3 more fps than the other guy but when I came to realise that whilst I got 3 more fps in one game, and he got 5 more fps in another game that was OpenGL instead of DirectX or whatever. It became obvious that it's not as clear cut as one card is better than the other in terms of frame rates, it depends on the graphics API, the driver releases, the OS, the other hardware in the system, the game settings and on and on. Personally I prefer nVidia, but that's only because they have better developer support and I've had a better experience with the quality of their drivers over ATI's, image quality, features and frames per second has never once been an issue for me and I'm sure this is true for all but those people who think that getting an extra 3 more fps in game X actually makes the blindest bit of difference in the world.
vista will look very bad if that dx 10 for XP hack comes out and it turns out to be faster.
This post has a greater abundance of both poor grammar and spelling than I can recall in any post that I've read in at least the previous year. The grammar is so poor that one or two sentences are nearly unintelligible. Nice proofreading and editing, there, Mr. ScuttleMonkey.
DX10 runs only on Vista. I'm sure this article will be of great interest to the three Vista gamers out there.
Just wait for Intel's Larrybee GPU. That's when the fun will start. Those guys are doing in software what Nvidia and ATI are killing themselves to do in hardware. That will be interesting because they really claim to do well with software, which is unbelievably more flexible and easy to develop. (Although there is good reason to believe they have hardware texture filtering on the device...) The Intel DX10 pipeline is being done by one small company rather than a legion of hardware engineers...
"The bragging rights of being able to claim you can run your game at 150fps while other plebeans can only run at 140fps is just that - bragging rights. There is no practical effect on game play until framerates drop below 30fps."
At those framerates, isn't latency more important than fps? Higher fps need not imply lower latency, just as higher MHz does not imply more flops.
"honest game developers taking a stance for the gamer" would mean they would have to fess up that their game runs like crap on either vendor's cards. Which would mean lower sales.
Wait until the benchmarks come out showing its 20% slower(Note this is a guestimate, don't get upset... yet).
As my third year computer design lecture loved saying: There are lies; then there are damn Lies; and then there are benchmarks.
The simplest explanation doesn't require any malice on the part of the video card manufacturers. If the developers and engineers develop the cards and drivers to optimize the features they believe the most important for performance, it stands to reason they will think those same features the most important when collaborating on a benchmark program. Magically, the benchmarks will score heavily in favor of the features that camp optimized their hardware and drivers for.
Since graphics technology is actually a fairly complex field and the design philosophies of these two companies are different, the other companies cards/drivers will be optimized for what THEY feel are the real performance metrics and therefore they won't test as well on those benchmarks.
All of this can happen without anyone doing anything but coding and designing in the manner they believe to be the best balance of technology and practicality.
Doom 3 and Quake 4 both have native linux clients available.
Yes, for the real gamer, there are a lot of things to consider before doing a test on new ideas. Such as what could be seen on the different category of gamers. The benchmarks could hardly do a lot, but how about giving it a try?
Why is game support a driver problem? I have written a few drivers myself so I understand what they do. What I don't understand is why every time a new game comes out, ATI and Nvidia need to provide support for that game in their driver? If they are putting game specific code in the drivers, shouldn't that code be part of the game? And if their driver blows and doesn't use the hardware correctly, shouldn't they fix it for all games, not just one? Can somebody explain this to me please.
Obviously you will never want to trust any for profit corporations homegrown benchmarks. This is how it has ALWAYS been, no reason it pretending this is some new happening between Nvidia and ATI. Benchmarks have been skewed since day one. It's exactly the same as when corporations release favorable studies.
If you remember this is EXACTLY how many vendors used to market AMD and Cyrix chips. They would post the ONE benchmarks these chips could beat and Intel in and say 30% MORE PERFORMANCE. Nothing new folks.