Slashdot Mirror
NVidia Accused of Inflating Benchmarks
Junky191 writes "With the NVidia GeForce FX 5900 recently released, this new high-end card seems to beat out ATI's 9800 pro, yet things are not as they appear. NVidia seems to be cheating on their drivers, inflating benchmark scores by cutting corners and causing scenes to be rendered improperly. Check out the ExtremeTech test results (especially their screenshots of garbled frames)."
Isn't this SOP for the entire video card industry? Every few years someone gets caught targeting some aspect of performance to the prevailing benchmarks. I guess that's what happens when people wax on about "my video card does 45300 fps in quake and yours only does 45292, your card sucks, my experience is soooo much better". For a while now it's been the ultimate hype driven market wrt hardware.
The problem is that people are buying cards based on these silly synthetic benchmarks. When performance in one arbitrary set of tests is so important to sales, naturally you're going to see drivers tailored to improving performance in those tests.
Of course, if Nvidia's drivers were released under the GPL, none of the mud from this would stick as they could just point to the source code and say "look, no tricks". As it is, we just get a nasty combination of the murky world of benchmarks and the murky world of modern 3D graphics.
ATi does make better hardware but their software (drivers) are terrible and not very well supported.
that is a old accusation - that had a kernel of truth 24 months ago, but Ive used ati cards for years, and they have gone rock solid since forums like this just started to accept that schlock as 100% truth.
Bottom line: dont believe the hype. this is just *not* true.
Yeah, but they all do it, and it isn't strictly video board manufacturers either. That '80 GB' hard drive you just bought isn't 80 GB, it's (depending on the manufacturer) either a 80,000,000,000 byte hard drive or a 80,000 MB hard drive...either way it isn't by any stretch of imagination 80 GB. That Ultra DMA 133 hard drive, BTW, can't really do a sustained 133 MB/s transfer rate either, that's the burst speed and you'll probably NEVER actually achieve that transfer rate in actual use. That 20" CRT you just bought isn't 20", it's 19.2" inches of viewable area. A 333 MHZ FSB isn't 333 MHZ, it's 332-point-something mhz, and even then it isn't really 333 MHZ because it's really like 166 mhz and doubled because DDR memory allows you to read and write on the high and low side of the clock. That 2400 DPI scanner you just bought is only 2400 DPI with software interpolation. Your 56K modem can really only do 53K due the FCC regulations requiring them to disable the 56K transfer rate. The list goes on.
My journal has hot
Oh, c'mon. Benckmark fudging has been an on-going tradition in the computer field. When I was doing computer testing for InfoWorld, I found some people in a vendor's organization would try to overclock computers so they would do better in the automated benchmarks. ZD Labs found some people who "played" the BAPco graphics benchmarks to earn better scores by detecting a benchmark was running and cutting corners.
<Obligatory-Microsoft-bash>
One of the early players was Microsoft, with its C compiler. I have it from a source in Microsoft that when the Byte C-compiler benchmarks figures were published in the early 1980s Microsoft didn't like being back of the pack. "It would take six months to fix the optimizer right." It would take two weeks, though, to put in recognizers for the common benchmarks of the time and insert hand-optimized "canned code" to better their score.</Obligatory-Microsoft-bash>
Microsoft wasn't the only one. How about a certain three-letter company who fudged their software? You have multiple right answers to this one. :)
When the SPECmark people first formed their benchmark committee, they knew of these practices and so they made the decision that SPECmarks were to be based on real programs, with known input and output, and the output was checked for correct answers before the execution times would be used.
And now you know why reputable testing organizations who use artifical workloads check their work with real applications: to catch the cheaters.
Let me reiterate an earlier comment by Alan Partridge: it's idiots who think that a less-than-one-percent difference in performance is significant. (Whether you the shoe fits you is something you have to decide for yourself.) What benchmark articles don't tell you is the spread of results they obtain through multiple testing cycles. When I was doing benchmark testing at InfoWorld, it was common for me to see trial-to-trial spreads of three percent in CPU benchmarks, and broader spreads than that with hard-disk benchmarks. Editors were unwilling to admit to readers that results were collected that formed a "cloud" -- they wanted a SINGLE number to put in print. ("Don't confuse the reader with facts, I want to make the point and move on.") I see that in the years since I was doing this full-time that editors are still insisting on "keep it simple" even when it's wrong.
Another observation: when I would trace back hardware and software that was played with, the response from upper management was universally astonishment. They would fall over backwards to ensure we got a production piece of equipment. To some extent, I believed their protestations, especially when bearded during their visits to our Labs. One computer company (name withheld to protect the long-dead guilty) was amazed when we took them into the lab and opened up their box. We pointed out that someone had poured White-Out over the crystal can, and that when we carefully removed the layer of gunk the crystal was 20% faster than usual. Talk about over-clocking!
So when someone says "Nvidia is guilty of lying" I say "prove it", further saying that you have to show with positive proof that the benchmark fudging was authorized by top management. I can't tell from the article, but I suspect someone pulled a fast one, and soon will be joining the very long high-technology bread line.
Pray the benchmarkers will always check their work.
And remember, the best benchmark is YOUR application.
Really? Do you write benchmarks?
I used to write benchmarks. It was very common to include worst-case patterns in benchmark tests to try to find corner cases -- the same sort of things that QA people do to try to find errors. For example, given your example of a floating-point unit: I would include basic operations that would have 1-bits sprinkled throughout the computation. If Intel's QA people would have done this with the Pentium, they would have discovered the un-programmed quadrant of the divide look-up table long before the chip was committed to production.
Why do we benchmark people do this? Because we are amazed (and amused) at what we catch. Hard disk benchmarks that catch disk drives that can't handle certain data patterns well at all, even to the point of completely being unable to read back what we just wrote. My personal favorite: how about modems from big-name companies that drop data when stressed to their fullest?
The SPECmark group recognizes that the wrong answer is always bad, so they insist that in their benchmarks the unit under test get the right answer before they even talk of timing. This is from canned data, of course, not "generating random scenes." The problem with using random data is that you don't know if the results are right with random data -- or at least that you get the results you've gotten on other testbeds.
Besides, how is the software supposed to know how the scene was rendered? Read back the graphics planes and try to interpret the image for "correctness"? First, is this possible with today's graphics cards, and, second, is it feasible to try? Picture analysis is an art unto itself, and I suspect that being able to check rendering adds a whole 'nuther dimension to the problem. I won't say it can't be done, but I will say that it would be expensive.
For FPUs, it's easy: have a test vector with lots of test cases. Make sure you include as many corner cases as you can conceive. When you make a test run, mix up the test cases so that you don't execute them in the same order every pass. (This will catch problems in vector FPU implementations.) Check those results!
Now, if you will tell me how to extend that philosophy to graphic cards, we will have something.
What you are describing isn't benchmarking, it's stress testing.
e r-won't-matter.
Benchmarks are meant to predict performance. While it is essential to check the validity of the answer (wrong answers can be computed infinitely fast), the role of a benchmark isn't to check never-seen-in-practice cases or so-rarely-seen-in-practice-that-running-100x-slow
That reminds me of the "graphic benchmark" used by some Mac websites that compares Quickdraw/Quartz performance when creating 10k windows. Guess what, Quartz is slower, because Quartz windows are a lot more powerful/heavyweight than Quickdraw ones. But who gives a fuck, how often do you need to create 10k windows in a hurry ? No one, apart from those OS 9 zealots who are looking for ways to bash OS X. A realistic benchmark may to check to at most 10s of windows, but the conclusion would probably be that the difference in speed isn't observable by humans.
A good benchmark can only be judged by comparing its execution profile against what users will run. If it's not reflecting the reality, it's not an appropriate prediction of the performance for the user. And it's not a binary property. While Spec is by definition perfect for anyone that only runs Spec, it is known and accepted to be imperfect at anything else, and a completely useless predictor in some cases (as in very low statistical correlation between Spec scores and speed at running Foo). It's just a "best effort" suite of tests for workstation applications. I'm talking SpecINT / SpecFP here, other Spec benchmarks exist because (gasp!) SpecINT/FP don't cover the whole computing spectrum.
You also don't seem to have much of a clue about how processors are really tested. Guess what, the processors people do all that you describe and more, much more. All day long on many, many samples, for months on end, in good/bad conditions (thermal, electrical). It's just that no test suite can catch all the problems, so defects will always slip by. _Always_, even if the logic is formally proven correct, since processors aren't mathematical entities but subject to electrical / manufacturing variations. Even if no problem exists today on a given CPU, take a hundred of them from various batches, power-cycle them a few million times, run them for a few years in marginal conditions and check again.
Video performance from my Radeon 7500 under Linux (using the ATI optimized drivers for XFree86 4.3) is not nearly as good as the ATI-provided drivers under Windows 2000. I think ATI gives the type of ingredients to the Linux driver developers, but the quantity of those ingredients it keeps to themselves.
nVidia could really follow along this same philosophy, instead of hearing the massive complaints from their oft-buggy video driver.
Ayup
One of the first courses in all college business curriculums I've seen is "Business Statistics" (BA154 here at GRCC.).
The course focuses on making decisions based on statistics. In the second week of class, we learned what a standard deviation was, and we never stopped using it throughout the semester.
But perhaps ignorance would explain business tactics of the 90's.
What's this Submit thingy do?
I gather you read it differently?
First, faster video cards are not designed to make you a better gamer, they are designed to make your gaming experience better. If they are not doing that for you, then you're not playing the games that need the improvement, and you don't need the card. Which, I'm sure, is true for a lot of people out there.
On the other hand, ATI sold over 1 million Radeon 9700s in first few months of it being out, so there are definitely a lot of people out there who do need and want the best card the money can buy.
So, that gets us to your question of whether nvdia cheating really makes a difference. Obviously, it doesn't make a difference to you, because you don't want the buy any of the high-end cards in the first place. It should be obvious in the same way, though, that it does make a big difference to somebody who will buy a high end card.
If 9800 and FX5900 have the same price, and speed is what you're after (and it should be, since you're buying these cards), then you want to buy the faster one. The only way to figure out which one is faster is to check the benchmark results (unless you buy both and try them tyourself). If one of the companies cheated in a benchmark, they have tricked you into thinking that you're buying a faster card, while you're really buying a slower one.
Imagine you're picking between two equally expensive cars, and you want to buy the faster of the two. One claims to do 0-60 in 5s, and the other claims to do it in 3s. You'll go ahead and buy the latter one, only to learn later that they were testing the car going downhill while the other was accelerating on level ground! I think enraged would only begin to describe your reaction to that.
Never underestimate the bandwidth of a 747 filled with CD-ROMs.