Truth in Advertising?

PerformanceEng wonders: "I work as an engineer for a large technology company in the U.S., and have been privy to what I find a interesting practice. It's well known that marketing data sheets often paint the best picture of a product while leaving the devil in the details. I've come to expect this, and when I am evaluating technology, I always have a skeptic's eye for claims made by the sales and marketing folks. However, I've also witnessed our product go into test labs (usually for the purposes of running a series of tests for a 'bake off' in a trade publication). Not uncommon is the attempt to 'tune' the configuration of the device under test to perform in the best light (not unlike tuning your car to pass emissions tests). I have seen it go as far as exploiting weaknesses in the test that, if the test operator discovered, would be considered bad faith. To the other engineers: Are you aware of this kind of practice at your company? To the IT professionals: How much faith do you put in these sorts of publications and their 'bake offs'? To everyone: When does spin doctoring cross the line and become false advertising?"

Comments

Video drivers

[Anonymous Coward]

It wasn't an uncommon practice for video card makers to tweak their drivers to perform better on benchmarks.

Consumer Reports pays cash

[LostCluster]

Consumer Reports is a such a respected publication because they have strict standards for the products they test. They don't accept items from the product makers, they go out into the marketplace and buy their test subjects using cash whenever possible. (Up until a while ago they even bought cars with cash, until they realized that car dealers began recognizing them as the only people who paid cash for cars, and the IRS requirement of reporting large cash transactions got in their way too.) As a result, their tests are immune to any tweaking...

It'd be nice if the tech publications could afford to do this, because at times they start to resemble the video game websites set up by kids who do it only to get prerelease copies of games for free under the guise of reviewing them. Such kids always have to write glowing reviews of everything they get because as soon as they post a negative review their stream of free stuff grinds to a halt.

Bottom line is that there's a foolproof way of preventing tampering in any review, but it costs money. Any review that involves accepting free stuff compromises the integrity from the start.

Re:Consumer Reports pays cash

[humblecoder]

Just because CR is unbiased doesn't mean that their tests aren't subject to the type of "tweaking" that the original poster describes. If the methodology of the testing that CR uses is known by the manufacturers, then they can design their products to do well at the test. Hopefully this would have the effect of being an indicator of the overall quality of the product, but as we know, this isn't always the case.

As a hypothetical, let's say that CR judges crash-worthiness of a car using a 35 mph head on collison test. Car manufacturers which know this are going to optimize the structural integrity of the car to hold up well under this test at the expense of other types of crashs (side impact crashs, say). Another car may not perform as well in the head on test, but it may be safer over a entire universe of possible crashes. However, because it is not optimized for the CR crash test, it won't get as high a rating.

Lest you think I am putting stuff out of my butt, this situation actually occurred with respect to the Insurance Institute for Highway Safety. Up until a few years ago, cars were generally crash tested using the head on methodology. However, the IIHS decided to start using an offset crash methodology since was more likely to occur in real life. They found the results from the offset crashes did not necessarily match the results from the head on crashes. Cars that did well in the head on tests did not do as well in the offset crash tests. Obviously manufacturers had optimized crash worthiness for the test and not for overall safety.

So where does the blame lie? I would say it lies both with the testers and the manufacturers. The testers are to blame for coming up with a test that doesn't necessarily reflect real life. Meanwhile car makers are to blame for designing products to "beat the test" rather than to be safe overall.

I think the same is true in the case of the original poster. His company isn't doing anything illegal; if the tests can be beaten so easily, then what good are they? In fact, one could argue that his company is helping in the sense that they are revealing the test's shortcoming. However, I find it hard to believe that their underlying motives are altruistic. I would guess that their motivation for tweaking their system is to beat the test for their own gain, and not for some higher moral purpose. So in a sense they are violating the spirit of the competition, in my opinion, even if what they are doing isn't wrong in the legal sense.

Re:Consumer Reports pays cash

[Anonymous Coward]

Problem is, CR's testing procedures often are god-awful. Case in point: CR's almost incomprehensible ignorance of how computers work, resulting in an anti-Mac bias that borders on the laughable. This most famously reared its ugly head when they ranked the brand-new PowerPC computers as very very slow, not realizing they had put the wrong (68040-only) software on them. Now in a recent issue they ranked the Macs on the bottom of the heap even though they played the whole thing up with teaser ad copy about how Macs are the only virus-free, zombie-free, difficult-to-hack boxes they test. It apperars that CR only looks at hardware (and that only from a PC perspective) and does not consider software at all.

Another good one. CR downgraded the Protege5 wagon, despite it having as good or better gas mileage, much better reliability, and MUCH better handling and breaking (a sport suspension). Oh, and it was cheaper too, and unquestionably better looking. Why didn't CR like it? Solely because its competition (PT Cruiser / Vibe / Matrix / Imprezza ) was higher up and had a cushier ride, like an SUV. So while the rest of the car trade ranked the P5 at the top, CR complained that it didn't feel enough like a Surburban.

Talk about losing your liberal moorings.

Re:Peer review

[krautcanman]

To answer your question of false advertising, I would say keep to the standard that most of us scientists do: Specifically, peer review and ensure that your results can be duplicated by said peers. If results cannot be duplicated, then it is false advertising.

Even science has a problem of touting the best data and "leaving the devil in the details." Research is driven by money just as much as industry. If you're not producing good results, you won't get funding.

Truth - Advertising?

[IamGarageGuy 2]

As a person who worked in the advertising business side, I can say wholeheartedly that truth in advertising is a complete misnomer. The whole concept of advertising rejects the idea of truth. I don't sound bitter do I?

Re:Truth - Advertising?

[gad_zuki!]

>I don't sound bitter do I?

No, you sound practical. Advertising usually affects the reptilian part of brains, preying on our patriotism (truck ads), vanity (gyms, makeup), greed (everything), etc. Its shameful there aren't controls on corporate "free speech" as McDonalds and others hire child psychologists to craft effective ads for their unhealthy products.

This is the golden age for ads. They're everything. Every webpage, above the urinal, people aren't very skeptical and have disposable incomes, the art of creating a working fad/meme is getting perfected, celebrities are manufactured from scratch, etc. And this is what people want.

The problem is two-fold. People, in general, need to take a good look at their consumerism and corps need controls on what they can and can't say. I'd like to see informative ads telling me cost, MPG, etc but a typical car ad is all mom, america, and apple pie stuff.

Similiar post over at nerdfilter today. The video is hilarious and worth watching.

Bill Hicks (RIP) said it all..

[little_fluffy_clouds]

"By the way if anyone here is in advertising or marketing... kill yourself. No, no, no it's just a little thought. I'm just trying to plant seeds. Maybe one day, they'll take root - I don't know. You try, you do what you can. Kill yourself.

Seriously though, if you are, do. Aaah, no really, there's no rationalisation for what you do and you are Satan's little helpers, Okay - kill yourself - seriously. You are the ruiner of all things good, seriously. No this is not a joke, you're going, "there's going to be a joke coming," there's no fucking joke coming. You are Satan's spawn filling the world with bile and garbage. You are fucked and you are fucking us. Kill yourself. It's the only way to save your fucking soul, kill yourself. Planting seeds.

I know all the marketing people are going, "he's doing a joke... there's no joke here whatsoever. Suck a tail-pipe, fucking hang yourself, borrow a gun from a Yank friend - I don't care how you do it. Rid the world of your evil fucking machinations. I know what all the marketing people are thinking right now too, "Oh, you know what Bill's doing, he's going for that anti-marketing dollar. That's a good market, he's very smart." Oh man, I am not doing that. You fucking evil scumbags! "Ooh, you know what Bill's doing now, he's going for the righteous indignation dollar. That's a big dollar. A lot of people are feeling that indignation. We've done research - huge market. He's doing a good thing." Godammit, I'm not doing that, you scum-bags!

Quit putting a godamm dollar sign on every fucking thing on this planet!

"Ooh, the anger dollar. Huge. Huge in times of recession. Giant market, Bill's very bright to do that." God, I'm just caught in a fucking web! "Ooh the trapped dollar, big dollar, huge dollar. Good market - look at our research. We see that many people feel trapped. If we play to that and then separate them into the trapped dollar..." How do you live like that? And I bet you sleep like fucking babies at night, don't you?"

[We miss you, Bill.]

Analyze, analyze, analyze

[rackhamh]

This isn't just a phenomenon in the IT arena. Have a look at medical journals some time... You have to be VERY careful when putting stock in the findings of studies -- the first thing to check is who *funded* the study.

I think it's just a fact of life: everybody wants their product to be seen in the best light, and to sell well (in the case of commodities or services).

That's why Amazon.com has reader reviews, sites like epinions.com exist, and Slashdot has moderator points. It's also why there are hardware review sites -- we can't just trust the manufacturer's PR now, can we?

So, people may be inherentely biased and often untruthful, but with proper monitoring (read: community involvement), the truth will out.

Consumer audio

[theLOUDroom]

To everyone: When does spin doctoring cross the line and become false advertising?"

In the consumer audio market, that's when.

From over-unity speakers (200W watts output from a 10W wall-wart), to "better-sounding" fiber optic cable, no claim seems too outrageous or fraudulent for a great many consumer audio manufacturers.

As an engineer who loves audio, it drives me nuts to see the bullshit that is constantly perpetrated in that market.

I'm sure there are tons of slashdotters who can post examples of incredibly unprofessional and possibly fraudulent specmanship in this arena.

I worked in a test lab...

[aiken_d]

...for about 5 years in the mid to late 90's. I started doing the testing on basic network equipment and graduated over time to oversee the testing methodology for every product comparison we ran.

I can tell you that, if the testers themselves are competent, it's a moot point. For instance, when testing server hardware by using a database application, I always insisted that the databases be identical and configured as identically as possible. Normal stumbling blocks were issues with stock disk sizes, but we always ensured that RAID configurations were as similar as possible within the realm of reason.

Testing is an art form. It requires a thorough and repeatable plan as well as a good bit of knowledge about real world usage of equipment and software (would it be realistic to enable a non-battery backed write cache on a raid controller in a database application?)

I can say that many, many vendors attempted to put one over on us. And it's entirely possible that I missed some of them, and they benefitted because of it. However, in general, professional test procedures should expose and nullify any sort of vendor tweakage of equipment or software.

Key principles for good testing:
- Set any basic configuration to manufacturer's public recommendations

- Don't let vendor representatives touch anything. If they need to send someone into the lab, allow them to recommend changes, and document all of those for later review / revocation

- If third party hardware/software is involved in a test, use the third party as a sounding board. If you're testing a layer 3 switch using streaming media, talk to the streaming media provider about realistic stream rates and usage patterns.

- If at all possible, wipe and reload vendor equipment and software. You should be looking at the setup process anyway, so that helps the test as well as helping to prevent shenanigans.

In short, good test procedures prevent, or at least mitigate, the kind of abuse in question. And, as consimers of reviews and tests, it's in all of our best interests to get educated and develop opinions about the competence, thoroughness, and honesty of any souce.

Cheers
-b

Benchmarking Tuning: Just as bad as Karma Whoring

[Nova Express]

The sort of people who would tune their software for a specific benchmark are the same sort of people who would karma whore here on Slashdot by throwing off-topic lines with guaranteed Slashdot appeal.

And you know who else hates that type of benchmarking whoring? Linus Torvalds, that's who! Linus would never stoop to such a thing, because Linus is a great guy!

And you know who else would never do it? Apple Computer, the people who make the greatest computers in the world! They would never stoop to rigging benchmarks!

Or karma whoring.

"Yes, None, Ten years ago"

[dbarclay10]

Yes, I've seen this sort of thing at other places I work. It's inherently dishonest. It's justified via a) claiming that it'll help sales (dubious), and b) claiming that everybody knows that they're bullshit anyways. Note that the two justifications are mutually exclusive. Doesn't stop them from using them though.

No, I trust none of these "bakeoffs". Or any other IT advertising for that matter. There isn't a single mainstream IT rag which is even marginally trustworthy. Go ahead and, instead of reading just the bakeoff that you're looking for, read an article about something you already know about (through hands-on experience with all the primary alternatives, including a FOSS alternative if it's software and there is a FOSS alternative). Note how much stuff they get wrong, how shallow the article is, and how it almost reads like an advertisement. The same is true for cars too, largely, at least from what I've read. I can't comment on other industries since I'm not particularly familiar with their trade press. Note, however, that I still don't trust them at all - I expect they're just as bad. It's just that I don't make enough decisions relating to those industries' products to warrant reading the trade press - instead I go to the store and carefully examine the alternatives.

This sort of thing crossed the line into fake advertising at least a decade ago. Companies routinely make absurd claims and get away with it. There's just no political interest in enforcing it. At best they'll include fine print in their ad. If it's a print ad, maybe you'll be able to read it. It's been a while since I've seen an ad with fine print whose fine print didn't take up at least 10 lines of extremely small type. Television ads are a joke, it's impossible to read the fine print at broadcast resolution, regardless of the size of your TV, and it typically takes up a whole screen.

What can we do about it? Elect governments with some spine. These sorts of advertisements will continue to be successful so long as people are poorly-educated, and people will continue to be poorly-educated unless there is a strong collective agreement in place that says "yes, everybody needs some minimum level of education, otherwise they're prone to manipulation and our society is controlled by those who control the media or the other forms of information dissemination." It's funny, isn't it, how political campaigns in the US almost exclusively take the form of commercials? (Except for the "debates", which are a joke to everybody outside the country.)

Note that when the US was founded, everybody who advocated democracy made sure to point out that the requirements for democracy included an educated public, free speech, and free press. People have totally forgotten the education bit and the press bit. (A government-controlled press is no more effective at disseminating important information than a press controlled by an aristocracy - corporate or otherwise.)

Re:I worked for HP....

[interstellar_donkey]

It's funny how that works. HP has somehow managed to go from one of the leading producers of quality printers, for example, to one of the many cheapo vendors.

Remember when a Laserjet 4 was the printer to have?

Or for that matter, remember when Diamond multimendia was the producers of graphics cards?

A company that overstates claims typically is a company that is cutting costs while sliding on their brand name. I wonder how many solid names in the industry have to go down the drain before they realize it's probably not a good idea, in the long run, to overstate the quality or performance of your products.

Heck, I can remember a time when Compaq actually made good computers.