Slashdot Mirror
Truth in Advertising?
PerformanceEng wonders: "I work as an engineer for a large technology company in the U.S., and have been privy to what I find a interesting practice. It's well known that marketing data sheets often paint the best picture of a product while leaving the devil in the details. I've come to expect this, and when I am evaluating technology, I always have a skeptic's eye for claims made by the sales and marketing folks.
However, I've also witnessed our product go into test labs (usually for the purposes of running a series of tests for a 'bake off' in a trade publication). Not uncommon is the attempt to 'tune' the configuration of the device under test to perform in the best light (not unlike tuning your car to pass emissions tests). I have seen it go as far as exploiting weaknesses in the test that, if the test operator discovered, would be considered bad faith. To the other engineers: Are you aware of this kind of practice at your company? To the IT professionals: How much faith do you put in these sorts of publications and their 'bake offs'? To everyone: When does spin doctoring cross the line and become false advertising?"
It wasn't an uncommon practice for video card makers to tweak their drivers to perform better on benchmarks.
Build it, and they will come^Hplain.
You're comparing how products perform under a specific test that you have devised. (which ideally, is similar to your production environment).
Tuning can have a dramatic difference in performance, and unless you're familiar with all of the products involved, it's impossible to get the best performance out of each one.
The original poster is talking about where one of the systems has been modified so it is not a default install, and specifically customized before being sent to the testor, so that they will perform better. (like with ATI's Quake 'optimization').
As another example, there were some folks trying to get higher rankings in SETI@home, who would return bogus results -- as that was faster than actually performing the calculations. If someone knows that the results won't be checked for accuracy (or can't), and only for time, they can boost their rankings dramatically.
Build it, and they will come^Hplain.
we had a bake-off on one of our products...i was called in when the results didnt meet what management were expecting. after adding cpus(!) to the product under test (without the knowledge of the tester) we finally got a result which beat our competition. this was in 2001. i was later first on the chopping block in 2002 after i noted at a meeting that we should not try to publicize the results too much since it might backfire. the VP who canned me noted that if we got results we should publicize them as much as possible and i was an "impediment to future marketing capaigns"). i got an above average severance package tho so i guess they paid me off to leave quietly. ironically HP's results got beaten by IBM which simply threw money at the problem 4 months later and won.
Just because CR is unbiased doesn't mean that their tests aren't subject to the type of "tweaking" that the original poster describes. If the methodology of the testing that CR uses is known by the manufacturers, then they can design their products to do well at the test. Hopefully this would have the effect of being an indicator of the overall quality of the product, but as we know, this isn't always the case.
As a hypothetical, let's say that CR judges crash-worthiness of a car using a 35 mph head on collison test. Car manufacturers which know this are going to optimize the structural integrity of the car to hold up well under this test at the expense of other types of crashs (side impact crashs, say). Another car may not perform as well in the head on test, but it may be safer over a entire universe of possible crashes. However, because it is not optimized for the CR crash test, it won't get as high a rating.
Lest you think I am putting stuff out of my butt, this situation actually occurred with respect to the Insurance Institute for Highway Safety. Up until a few years ago, cars were generally crash tested using the head on methodology. However, the IIHS decided to start using an offset crash methodology since was more likely to occur in real life. They found the results from the offset crashes did not necessarily match the results from the head on crashes. Cars that did well in the head on tests did not do as well in the offset crash tests. Obviously manufacturers had optimized crash worthiness for the test and not for overall safety.
So where does the blame lie? I would say it lies both with the testers and the manufacturers. The testers are to blame for coming up with a test that doesn't necessarily reflect real life. Meanwhile car makers are to blame for designing products to "beat the test" rather than to be safe overall.
I think the same is true in the case of the original poster. His company isn't doing anything illegal; if the tests can be beaten so easily, then what good are they? In fact, one could argue that his company is helping in the sense that they are revealing the test's shortcoming. However, I find it hard to believe that their underlying motives are altruistic. I would guess that their motivation for tweaking their system is to beat the test for their own gain, and not for some higher moral purpose. So in a sense they are violating the spirit of the competition, in my opinion, even if what they are doing isn't wrong in the legal sense.
------
www.moneybythenumbers.com
Until he retired, my uncle was head of their paint testing laboratory, and this is exactly what he did. He would, for example, test a paint's opacity by applying a coat directly to an unprimed test pattern. He used to drive the paint companies nuts -- but when he said a paint will cover in a single coat that's exactly what a consumer could expect.
And the brethren went away edified.