Slashdot Mirror
Truth in Advertising?
PerformanceEng wonders: "I work as an engineer for a large technology company in the U.S., and have been privy to what I find a interesting practice. It's well known that marketing data sheets often paint the best picture of a product while leaving the devil in the details. I've come to expect this, and when I am evaluating technology, I always have a skeptic's eye for claims made by the sales and marketing folks.
However, I've also witnessed our product go into test labs (usually for the purposes of running a series of tests for a 'bake off' in a trade publication). Not uncommon is the attempt to 'tune' the configuration of the device under test to perform in the best light (not unlike tuning your car to pass emissions tests). I have seen it go as far as exploiting weaknesses in the test that, if the test operator discovered, would be considered bad faith. To the other engineers: Are you aware of this kind of practice at your company? To the IT professionals: How much faith do you put in these sorts of publications and their 'bake offs'? To everyone: When does spin doctoring cross the line and become false advertising?"
It wasn't an uncommon practice for video card makers to tweak their drivers to perform better on benchmarks.
I've also witnessed our product go into test labs (usually for the purposes of running a series of tests for a 'bake off' in a trade publication). Not uncommon is the attempt to 'tune' the configuration of the device under test to perform in the best light (not unlike tuning your car to pass emissions tests). I have seen it go as far as exploiting weaknesses in the test that, if the test operator discovered, would be considered bad faith.
:-) Seriously though, this has been the whole problem with "benchmarks" like SPEC and others that ultimately results in pissing matches between manufacturers saying "my product is faster than yours" which for 99% of the users out there means nothing. In fact, even for that 1% of us where it does make a difference, specific optimizations to ones code or algorithms typically will get you more performance. So, what it really comes down to is how productive is the product + environment + task that you are assigning to the platform.
Oh, you work for Intel then.
To answer your question of false advertising, I would say keep to the standard that most of us scientists do: Specifically, peer review and ensure that your results can be duplicated by said peers. If results cannot be duplicated, then it is false advertising.
Visit Jonesblog and say hello.
Consumer Reports is a such a respected publication because they have strict standards for the products they test. They don't accept items from the product makers, they go out into the marketplace and buy their test subjects using cash whenever possible. (Up until a while ago they even bought cars with cash, until they realized that car dealers began recognizing them as the only people who paid cash for cars, and the IRS requirement of reporting large cash transactions got in their way too.) As a result, their tests are immune to any tweaking...
It'd be nice if the tech publications could afford to do this, because at times they start to resemble the video game websites set up by kids who do it only to get prerelease copies of games for free under the guise of reviewing them. Such kids always have to write glowing reviews of everything they get because as soon as they post a negative review their stream of free stuff grinds to a halt.
Bottom line is that there's a foolproof way of preventing tampering in any review, but it costs money. Any review that involves accepting free stuff compromises the integrity from the start.
As a person who worked in the advertising business side, I can say wholeheartedly that truth in advertising is a complete misnomer. The whole concept of advertising rejects the idea of truth. I don't sound bitter do I?
Stay tuned for new sig...
"By the way if anyone here is in advertising or marketing... kill yourself. No, no, no it's just a little thought. I'm just trying to plant seeds. Maybe one day, they'll take root - I don't know. You try, you do what you can. Kill yourself.
Seriously though, if you are, do. Aaah, no really, there's no rationalisation for what you do and you are Satan's little helpers, Okay - kill yourself - seriously. You are the ruiner of all things good, seriously. No this is not a joke, you're going, "there's going to be a joke coming," there's no fucking joke coming. You are Satan's spawn filling the world with bile and garbage. You are fucked and you are fucking us. Kill yourself. It's the only way to save your fucking soul, kill yourself. Planting seeds.
I know all the marketing people are going, "he's doing a joke... there's no joke here whatsoever. Suck a tail-pipe, fucking hang yourself, borrow a gun from a Yank friend - I don't care how you do it. Rid the world of your evil fucking machinations. I know what all the marketing people are thinking right now too, "Oh, you know what Bill's doing, he's going for that anti-marketing dollar. That's a good market, he's very smart." Oh man, I am not doing that. You fucking evil scumbags! "Ooh, you know what Bill's doing now, he's going for the righteous indignation dollar. That's a big dollar. A lot of people are feeling that indignation. We've done research - huge market. He's doing a good thing." Godammit, I'm not doing that, you scum-bags!
Quit putting a godamm dollar sign on every fucking thing on this planet!
"Ooh, the anger dollar. Huge. Huge in times of recession. Giant market, Bill's very bright to do that." God, I'm just caught in a fucking web! "Ooh the trapped dollar, big dollar, huge dollar. Good market - look at our research. We see that many people feel trapped. If we play to that and then separate them into the trapped dollar..." How do you live like that? And I bet you sleep like fucking babies at night, don't you?"
[We miss you, Bill.]
What were the skies like when you were young?
This isn't just a phenomenon in the IT arena. Have a look at medical journals some time... You have to be VERY careful when putting stock in the findings of studies -- the first thing to check is who *funded* the study.
I think it's just a fact of life: everybody wants their product to be seen in the best light, and to sell well (in the case of commodities or services).
That's why Amazon.com has reader reviews, sites like epinions.com exist, and Slashdot has moderator points. It's also why there are hardware review sites -- we can't just trust the manufacturer's PR now, can we?
So, people may be inherentely biased and often untruthful, but with proper monitoring (read: community involvement), the truth will out.
To everyone: When does spin doctoring cross the line and become false advertising?"
In the consumer audio market, that's when.
From over-unity speakers (200W watts output from a 10W wall-wart), to "better-sounding" fiber optic cable, no claim seems too outrageous or fraudulent for a great many consumer audio manufacturers.
As an engineer who loves audio, it drives me nuts to see the bullshit that is constantly perpetrated in that market.
I'm sure there are tons of slashdotters who can post examples of incredibly unprofessional and possibly fraudulent specmanship in this arena.
Life is too short to proofread.
Build it, and they will come^Hplain.
...for about 5 years in the mid to late 90's. I started doing the testing on basic network equipment and graduated over time to oversee the testing methodology for every product comparison we ran.
I can tell you that, if the testers themselves are competent, it's a moot point. For instance, when testing server hardware by using a database application, I always insisted that the databases be identical and configured as identically as possible. Normal stumbling blocks were issues with stock disk sizes, but we always ensured that RAID configurations were as similar as possible within the realm of reason.
Testing is an art form. It requires a thorough and repeatable plan as well as a good bit of knowledge about real world usage of equipment and software (would it be realistic to enable a non-battery backed write cache on a raid controller in a database application?)
I can say that many, many vendors attempted to put one over on us. And it's entirely possible that I missed some of them, and they benefitted because of it. However, in general, professional test procedures should expose and nullify any sort of vendor tweakage of equipment or software.
Key principles for good testing:
- Set any basic configuration to manufacturer's public recommendations
- Don't let vendor representatives touch anything. If they need to send someone into the lab, allow them to recommend changes, and document all of those for later review / revocation
- If third party hardware/software is involved in a test, use the third party as a sounding board. If you're testing a layer 3 switch using streaming media, talk to the streaming media provider about realistic stream rates and usage patterns.
- If at all possible, wipe and reload vendor equipment and software. You should be looking at the setup process anyway, so that helps the test as well as helping to prevent shenanigans.
In short, good test procedures prevent, or at least mitigate, the kind of abuse in question. And, as consimers of reviews and tests, it's in all of our best interests to get educated and develop opinions about the competence, thoroughness, and honesty of any souce.
Cheers
-b
If I wanted a sig I would have filled in that stupid box.
Reminds me of the strips where Wally had to impersonate the demo.
When the image got fuzzy, they tried a razor.
In the pay per click world of google adwords (those text ads you see when you search) I advertise a free service. But since this free service is bundled with other nonfree services I put the prices on the ad itself.
So although they may be looking for something free, I don't pay for the click unless they know they're going to pay *something*, the visitor is better informed, and I get a higher conversion rate from the qualified traffic.
So although this may not be on the exact topic of yours, I submit that honesty in advertising works, especially when you pay for performance.
You're comparing how products perform under a specific test that you have devised. (which ideally, is similar to your production environment).
Tuning can have a dramatic difference in performance, and unless you're familiar with all of the products involved, it's impossible to get the best performance out of each one.
The original poster is talking about where one of the systems has been modified so it is not a default install, and specifically customized before being sent to the testor, so that they will perform better. (like with ATI's Quake 'optimization').
As another example, there were some folks trying to get higher rankings in SETI@home, who would return bogus results -- as that was faster than actually performing the calculations. If someone knows that the results won't be checked for accuracy (or can't), and only for time, they can boost their rankings dramatically.
Build it, and they will come^Hplain.
Marketing materials do not set out the faults of the product. This is not the role of marketing. Marketing aims to connect buyers to sellers. Providing information about faults does not help to make that connection. Also, many of the "tests" cited by marketers are labeled with titles such as, "Customer Success Story". This should be a clue that the material will not detail unsuccessful characteristics of the product.
Finally, marketers in most companies are not technical experts. They have to rely on the information provided by engineers and programmers. Many companies avoid ever telling the marketing department anything negative. As a result, in many cases, marketers aren't lying when they make claims -- they're explaining what they were told. Many of these marketers, especially the ones writing up collateral, are junior, new to the company, or even working on contract, so they don't have the depth of knowledge to tell that they've been given misleading information. Other people in the company sometimes lie to the marketers. It's not always black and white. (Not that all marketers tell the truth, of course.)
-- SYS 64738 --
And you know who else hates that type of benchmarking whoring? Linus Torvalds, that's who! Linus would never stoop to such a thing, because Linus is a great guy!
And you know who else would never do it? Apple Computer, the people who make the greatest computers in the world! They would never stoop to rigging benchmarks!
Or karma whoring.
Lawrence Person (lawrencepersonh@gmailh.com (remove all "h"s to mail)
http://www.lawrenceperson.com/
we had a bake-off on one of our products...i was called in when the results didnt meet what management were expecting. after adding cpus(!) to the product under test (without the knowledge of the tester) we finally got a result which beat our competition. this was in 2001. i was later first on the chopping block in 2002 after i noted at a meeting that we should not try to publicize the results too much since it might backfire. the VP who canned me noted that if we got results we should publicize them as much as possible and i was an "impediment to future marketing capaigns"). i got an above average severance package tho so i guess they paid me off to leave quietly. ironically HP's results got beaten by IBM which simply threw money at the problem 4 months later and won.
Yes, I've seen this sort of thing at other places I work. It's inherently dishonest. It's justified via a) claiming that it'll help sales (dubious), and b) claiming that everybody knows that they're bullshit anyways. Note that the two justifications are mutually exclusive. Doesn't stop them from using them though.
No, I trust none of these "bakeoffs". Or any other IT advertising for that matter. There isn't a single mainstream IT rag which is even marginally trustworthy. Go ahead and, instead of reading just the bakeoff that you're looking for, read an article about something you already know about (through hands-on experience with all the primary alternatives, including a FOSS alternative if it's software and there is a FOSS alternative). Note how much stuff they get wrong, how shallow the article is, and how it almost reads like an advertisement. The same is true for cars too, largely, at least from what I've read. I can't comment on other industries since I'm not particularly familiar with their trade press. Note, however, that I still don't trust them at all - I expect they're just as bad. It's just that I don't make enough decisions relating to those industries' products to warrant reading the trade press - instead I go to the store and carefully examine the alternatives.
This sort of thing crossed the line into fake advertising at least a decade ago. Companies routinely make absurd claims and get away with it. There's just no political interest in enforcing it. At best they'll include fine print in their ad. If it's a print ad, maybe you'll be able to read it. It's been a while since I've seen an ad with fine print whose fine print didn't take up at least 10 lines of extremely small type. Television ads are a joke, it's impossible to read the fine print at broadcast resolution, regardless of the size of your TV, and it typically takes up a whole screen.
What can we do about it? Elect governments with some spine. These sorts of advertisements will continue to be successful so long as people are poorly-educated, and people will continue to be poorly-educated unless there is a strong collective agreement in place that says "yes, everybody needs some minimum level of education, otherwise they're prone to manipulation and our society is controlled by those who control the media or the other forms of information dissemination." It's funny, isn't it, how political campaigns in the US almost exclusively take the form of commercials? (Except for the "debates", which are a joke to everybody outside the country.)
Note that when the US was founded, everybody who advocated democracy made sure to point out that the requirements for democracy included an educated public, free speech, and free press. People have totally forgotten the education bit and the press bit. (A government-controlled press is no more effective at disseminating important information than a press controlled by an aristocracy - corporate or otherwise.)
Barclay family motto:
Aut agere aut mori.
(Either action or death.)
When does spin doctoring cross the line and become false advertising?
It's quite a simple answer - misleading or misrepresenting anything whasoever is falsehood. There's not really any grey area, proposing the existence of such is a socially acceptable way of making the lie pallatable or discusable.
People generally have the common sense to know themselves if they're lying or not, but mainly prefer to not worry about it. The problem is that we live in a societies based on and that thrives on lies. Liars often win in a consumerist culture, because lies are usually selling people their own dumb desires right back to them.
The real issue is whether it is actually acceptable to lie. All politicians without exception lie and muddy the water, advertisers and PR people lie so much perhaps they don't even notice anymore. The alternative is too unpalatable to a mindless and uneducated society who want everyone to do their dirty work for them,
Most Americans would rather think that their army for instance is well equipped with modern and state of the art equipment. We like to think that our governments care about every soldier as we do our friends and family. Regardless of who's in power - the government is not a benevolent father who loves each and every one of us and watches down on us like a proud patriarch.
The reality is that dumb kids lives are cheaper than good equipment (regardless of who you vote for and who's in power). Another dead kid in Iraq isn't really top priority, unlike keeping the Whitehouse furniture and art restored. People don't like to admit that some dumb grunt isn't worth as much as a nice piece of Louis XIV furniture, so people pretend to care when in fact they don't terribly much.
The holy grail of technology is no different - the utopia of consumerist culture is just to tempting to refuse new technology for it's own sake. Nobody wants to know that the latest thing isn't all that good - hell most people don't really have an actual use for their computers as they're lives and work are usually fairly inconquential. We want to eat the dream of technology and time saving devices even though deep down we know that it's all make believe, and we don't really have anything to do with all our saved spare time anyway.
Si tacuisses philosophus mansisses. If you had kept quiet, you would have remained a philosopher.