Truth in Advertising?

PerformanceEng wonders: "I work as an engineer for a large technology company in the U.S., and have been privy to what I find a interesting practice. It's well known that marketing data sheets often paint the best picture of a product while leaving the devil in the details. I've come to expect this, and when I am evaluating technology, I always have a skeptic's eye for claims made by the sales and marketing folks. However, I've also witnessed our product go into test labs (usually for the purposes of running a series of tests for a 'bake off' in a trade publication). Not uncommon is the attempt to 'tune' the configuration of the device under test to perform in the best light (not unlike tuning your car to pass emissions tests). I have seen it go as far as exploiting weaknesses in the test that, if the test operator discovered, would be considered bad faith. To the other engineers: Are you aware of this kind of practice at your company? To the IT professionals: How much faith do you put in these sorts of publications and their 'bake offs'? To everyone: When does spin doctoring cross the line and become false advertising?"

Comments

Video drivers

[Anonymous Coward]

It wasn't an uncommon practice for video card makers to tweak their drivers to perform better on benchmarks.

Re:Consumer Reports pays cash

[humblecoder]

Just because CR is unbiased doesn't mean that their tests aren't subject to the type of "tweaking" that the original poster describes. If the methodology of the testing that CR uses is known by the manufacturers, then they can design their products to do well at the test. Hopefully this would have the effect of being an indicator of the overall quality of the product, but as we know, this isn't always the case.

As a hypothetical, let's say that CR judges crash-worthiness of a car using a 35 mph head on collison test. Car manufacturers which know this are going to optimize the structural integrity of the car to hold up well under this test at the expense of other types of crashs (side impact crashs, say). Another car may not perform as well in the head on test, but it may be safer over a entire universe of possible crashes. However, because it is not optimized for the CR crash test, it won't get as high a rating.

Lest you think I am putting stuff out of my butt, this situation actually occurred with respect to the Insurance Institute for Highway Safety. Up until a few years ago, cars were generally crash tested using the head on methodology. However, the IIHS decided to start using an offset crash methodology since was more likely to occur in real life. They found the results from the offset crashes did not necessarily match the results from the head on crashes. Cars that did well in the head on tests did not do as well in the offset crash tests. Obviously manufacturers had optimized crash worthiness for the test and not for overall safety.

So where does the blame lie? I would say it lies both with the testers and the manufacturers. The testers are to blame for coming up with a test that doesn't necessarily reflect real life. Meanwhile car makers are to blame for designing products to "beat the test" rather than to be safe overall.

I think the same is true in the case of the original poster. His company isn't doing anything illegal; if the tests can be beaten so easily, then what good are they? In fact, one could argue that his company is helping in the sense that they are revealing the test's shortcoming. However, I find it hard to believe that their underlying motives are altruistic. I would guess that their motivation for tweaking their system is to beat the test for their own gain, and not for some higher moral purpose. So in a sense they are violating the spirit of the competition, in my opinion, even if what they are doing isn't wrong in the legal sense.