Slashdot Mirror
Truth in Advertising?
PerformanceEng wonders: "I work as an engineer for a large technology company in the U.S., and have been privy to what I find a interesting practice. It's well known that marketing data sheets often paint the best picture of a product while leaving the devil in the details. I've come to expect this, and when I am evaluating technology, I always have a skeptic's eye for claims made by the sales and marketing folks.
However, I've also witnessed our product go into test labs (usually for the purposes of running a series of tests for a 'bake off' in a trade publication). Not uncommon is the attempt to 'tune' the configuration of the device under test to perform in the best light (not unlike tuning your car to pass emissions tests). I have seen it go as far as exploiting weaknesses in the test that, if the test operator discovered, would be considered bad faith. To the other engineers: Are you aware of this kind of practice at your company? To the IT professionals: How much faith do you put in these sorts of publications and their 'bake offs'? To everyone: When does spin doctoring cross the line and become false advertising?"
I work in developing web applications. When choosing server technologies I have learned to conduct my own bake off using the application that will be deployed. Each application is unique. Comparing your custom app to a published bake off is usually an apples to oranges comparison.
Consumer Reports is a such a respected publication because they have strict standards for the products they test. They don't accept items from the product makers, they go out into the marketplace and buy their test subjects using cash whenever possible. (Up until a while ago they even bought cars with cash, until they realized that car dealers began recognizing them as the only people who paid cash for cars, and the IRS requirement of reporting large cash transactions got in their way too.) As a result, their tests are immune to any tweaking...
It'd be nice if the tech publications could afford to do this, because at times they start to resemble the video game websites set up by kids who do it only to get prerelease copies of games for free under the guise of reviewing them. Such kids always have to write glowing reviews of everything they get because as soon as they post a negative review their stream of free stuff grinds to a halt.
Bottom line is that there's a foolproof way of preventing tampering in any review, but it costs money. Any review that involves accepting free stuff compromises the integrity from the start.
Similar to the game journalism post earlier today, if you want honest impressions, you're going to get them from your buddy saying the stuff rocks/sucks than from any sort of review/preview/ad.
To everyone: When does spin doctoring cross the line and become false advertising?"
In the consumer audio market, that's when.
From over-unity speakers (200W watts output from a 10W wall-wart), to "better-sounding" fiber optic cable, no claim seems too outrageous or fraudulent for a great many consumer audio manufacturers.
As an engineer who loves audio, it drives me nuts to see the bullshit that is constantly perpetrated in that market.
I'm sure there are tons of slashdotters who can post examples of incredibly unprofessional and possibly fraudulent specmanship in this arena.
Life is too short to proofread.
But the fancy numbers aren't for me, they're for PHBs that like to see lots of impressive numbers -- after all, the other product has them so if this one doesn't...
Looking at computer specs lately I'm beginning to think the principal point of them is to bulk out the specs -- make it look like it has lots of features, and the actual content of the specs is irrelevant.
Sun Microsystems (SUNW) shrugged off accusations today of unfairly reporting test scores for the beta version of one of its Java compilers.
Pendragon Software yesterday said that Sun, using Pendragon's CaffeineMark benchmarking tool, inaccurately inflated the test results of the Solaris 2.6 just-in-time Java compiler by optimizing the compiler specifically for that test. Solaris is Sun's version of the Unix operating system.
Sun responded by calling such optimization standard practice.
"The idea is that you want people to optimize for the benchmark," said Brian Croll, director of marketing for Sun's Solaris products. "We'll do everything in our power to do really well on all the benchmarks we get our hands on."
A benchmark is a battery of tests that gauges the speed and performance of software running in various configurations. Several developers have created Java benchmarks; CaffeineMark, which Croll called "the best benchmark we've got," is available free off the Web.
But how much optimization is fair play? Pendragon president Ivan Phillips contended Sun inflated the test results of the Solaris 2.6 just-in-time compiler by lifting code from CaffeineMark and inserting it into the compiler.
"The logic test is contained in the 'logicatom.class' file, and almost 50 percent of that file appeared in the compiler," he said. "The probability that this code made its way there accidentally is infinitesimal."
Reusing such a large chunk of specific code risks diverting too much of the compiler's resources, resulting in lower performance once the compiler is deployed in the real world, Phillips added.
Croll denied that Sun used CaffeineMark code but said the company "optimized around it." It will be difficult to determine who is correct, given that the beta compiler in question is no longer available. Croll stressed that the compiler is designed to perform well on a benchmark because that's what determines good real-world performance.
"If certain things happen frequently in a benchmark, you want to make sure you handle them well," he said. "If it turns out the benchmark doesn't truly represent true application performance, you need to evolve the benchmark."
The charges come at a time when Sun and Microsoft are entangled in tit-for-tat lawsuits over Microsoft's use of Java in its Internet Explorer 4.0 browser.
In an October 20 press release, Sun bragged that Solaris had the "world's fastest Java performance" and ran Java applications 50 percent faster than rival operating system Windows NT. After taking issue with Sun's test results, Pendragon said it asked Sun to retract its claims and remove the compiler from its Web site.
Sun removed the entire JDK 1.1.4 for Solaris on October 29 because the beta evaluation period ended, according to Croll. The company didn't take down the press release or rescind its claims, however, and Phillips responded yesterday by publishing his accusations.
Pendragon doesn't usually double-check testers' CaffeineMark scores. But when it saw Sun's results--the Solaris compiler hit a score of 1.4 million on the "logic" test, while the previous high for that test was 22,000--the software firm decided to investigate, fearing that CaffeineMark contained a bug.
If Sun indeed took deliberate steps to skew its results, Phillips was surprised at the lack of subtlety.
"If a company really wanted to conceal what they were doing, they could do a better job," he said.
Take a look and feel free: http://www.PieMenu.com
...for about 5 years in the mid to late 90's. I started doing the testing on basic network equipment and graduated over time to oversee the testing methodology for every product comparison we ran.
I can tell you that, if the testers themselves are competent, it's a moot point. For instance, when testing server hardware by using a database application, I always insisted that the databases be identical and configured as identically as possible. Normal stumbling blocks were issues with stock disk sizes, but we always ensured that RAID configurations were as similar as possible within the realm of reason.
Testing is an art form. It requires a thorough and repeatable plan as well as a good bit of knowledge about real world usage of equipment and software (would it be realistic to enable a non-battery backed write cache on a raid controller in a database application?)
I can say that many, many vendors attempted to put one over on us. And it's entirely possible that I missed some of them, and they benefitted because of it. However, in general, professional test procedures should expose and nullify any sort of vendor tweakage of equipment or software.
Key principles for good testing:
- Set any basic configuration to manufacturer's public recommendations
- Don't let vendor representatives touch anything. If they need to send someone into the lab, allow them to recommend changes, and document all of those for later review / revocation
- If third party hardware/software is involved in a test, use the third party as a sounding board. If you're testing a layer 3 switch using streaming media, talk to the streaming media provider about realistic stream rates and usage patterns.
- If at all possible, wipe and reload vendor equipment and software. You should be looking at the setup process anyway, so that helps the test as well as helping to prevent shenanigans.
In short, good test procedures prevent, or at least mitigate, the kind of abuse in question. And, as consimers of reviews and tests, it's in all of our best interests to get educated and develop opinions about the competence, thoroughness, and honesty of any souce.
Cheers
-b
If I wanted a sig I would have filled in that stupid box.
Absolutely none, I rely solely on product packaging.
Seriously though, I hold the belief that all sales and marketing folk are born liars and will never change. I purchase solely on word of mouth (from people I trust) and my past experience with a particular brand/manufacturer. I am the person that advertisers hate because I sit in front of the TV and explain to my wife exactly which mind fucts the advertiser is utilizing. Sales and Marketing (S&M how ironic) folk are beneath lawyers, politicians and criminals in my book.
(B) + (D) + (B) + (D) = (K) + (&)
In the pay per click world of google adwords (those text ads you see when you search) I advertise a free service. But since this free service is bundled with other nonfree services I put the prices on the ad itself.
So although they may be looking for something free, I don't pay for the click unless they know they're going to pay *something*, the visitor is better informed, and I get a higher conversion rate from the qualified traffic.
So although this may not be on the exact topic of yours, I submit that honesty in advertising works, especially when you pay for performance.
What are you, a Sharper Image salesman or something?
You probably shouldn't click this.
In my experience, the lies don't stop with the advertising. My bosses are both salesmen. The only thing they do at the small company that I work at is sell our software, and they'll tell anyone whatever it is they want to hear so that they'll buy it. But I've noticed that this is definately not the end of their dishonesty. They treat their staff, me included, just like a buyer. They promise us stuff like compensation for working weekends, etc, but then just like our software, they fail to deliver.
I have a huge problem with salespeople and advertisements specifically because of my bosses. IMHO everyone who works in sales is nothing but a glorified 419 scam artist. Politics really aren't any different either.
That is why I like open source so much. Almost everything is free, so there's no reason to lie.
No one cares what your captcha was
Houston TX, USA
More often we become aware of it when the competitor does it.
About 20 years ago there were a series of "shootouts" between Novell, Microsoft, and 3COM, to see which network OS was faster. That was when I was literated to the fact that tweaking parameters can make a HUGE difference in test results. If you have even more control, you can even tweak the tests. We used to have to supply "debunking" documents that explained how the competing companies got the results they published. Sometimes it was hard to reproduce their numbers, even tweaking our own sofware in the worst ways.
These days a lot of journalists try to maintain a neutral position. They go to great lengths to be fair, and document even tiny things that might give one product a slight edge over another. It gives them more credibility to those of us that have been through these product wars.It's funny how that works. HP has somehow managed to go from one of the leading producers of quality printers, for example, to one of the many cheapo vendors.
Remember when a Laserjet 4 was the printer to have?
Or for that matter, remember when Diamond multimendia was the producers of graphics cards?
A company that overstates claims typically is a company that is cutting costs while sliding on their brand name. I wonder how many solid names in the industry have to go down the drain before they realize it's probably not a good idea, in the long run, to overstate the quality or performance of your products.
Heck, I can remember a time when Compaq actually made good computers.
The Internet is generally stupid
When does spin doctoring cross the line and become false advertising?
It's quite a simple answer - misleading or misrepresenting anything whasoever is falsehood. There's not really any grey area, proposing the existence of such is a socially acceptable way of making the lie pallatable or discusable.
People generally have the common sense to know themselves if they're lying or not, but mainly prefer to not worry about it. The problem is that we live in a societies based on and that thrives on lies. Liars often win in a consumerist culture, because lies are usually selling people their own dumb desires right back to them.
The real issue is whether it is actually acceptable to lie. All politicians without exception lie and muddy the water, advertisers and PR people lie so much perhaps they don't even notice anymore. The alternative is too unpalatable to a mindless and uneducated society who want everyone to do their dirty work for them,
Most Americans would rather think that their army for instance is well equipped with modern and state of the art equipment. We like to think that our governments care about every soldier as we do our friends and family. Regardless of who's in power - the government is not a benevolent father who loves each and every one of us and watches down on us like a proud patriarch.
The reality is that dumb kids lives are cheaper than good equipment (regardless of who you vote for and who's in power). Another dead kid in Iraq isn't really top priority, unlike keeping the Whitehouse furniture and art restored. People don't like to admit that some dumb grunt isn't worth as much as a nice piece of Louis XIV furniture, so people pretend to care when in fact they don't terribly much.
The holy grail of technology is no different - the utopia of consumerist culture is just to tempting to refuse new technology for it's own sake. Nobody wants to know that the latest thing isn't all that good - hell most people don't really have an actual use for their computers as they're lives and work are usually fairly inconquential. We want to eat the dream of technology and time saving devices even though deep down we know that it's all make believe, and we don't really have anything to do with all our saved spare time anyway.
Si tacuisses philosophus mansisses. If you had kept quiet, you would have remained a philosopher.
Another one: just look at the old Dhrystone benchmark and all of the over-the-top "optimizations" that were used to get better compiler/processor results. The SPEC organization, created in a direct attempt to deal with this very kind of problem, still must update its bendhmarks regularly in order to deal with loopholes (and changing technology in general). A good example was when a particular benchmark (matrix300; ref is 2/3 the way down) was defeated because it took no input. That made it possible in principle to collapse the entire program to a constant, and at some point, somebody did. That last link also gives a good description of why initially good benchmarks go stale.
And while those two examples are old enough to show that this has been going on a long time, there are plenty of examples far older.
Benchmarking has always been an arms race.
"You do a commercial, and you're off the artistic rollcall forever. Everything you say is suspect, and is like a turd falling into my drink." - Bill Hicks
/^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$/i
Consumer Reports is an interesting case. I like their reviews in the sense that they *are* unbiased by ads and the rest as you have noted. Unfortunately, CR reviews so many product categories that they are not particularly well informed about many of the technologies involved. Their car reviews, for example, weigh reliability so high that someone with a preference for performance might not get much out of their reviews. Their computer reviews are similarly shallow, coming from a middle-of-the-road user standpoint. Interesting and informative, yes, but perhaps not very meaningful for a prospective geek user. It would be great, IMHO, if there were a magazine with the neutrality and consumer-focus of CR but with a more technical bent and a more limited scope of products covered. Say, for example, only cars, only electronics, etc.
" Advertising usually affects the reptilian part of brains, preying on our patriotism (truck ads), "
This reminds me of something I saw the other day. There was a big old truck (F350) relatively new stopped at a light in front of me. The bed was empty except for a full size american flag that was tied to a broomstick which was attached to rear corner of the truck.
It's hard to know how long that flag had been there but it was in horrible shape. It was dirty and wet and the leading edge has been torn to shreds by the wind.
When I was in the military we were thought to respect the flag. We were told that you don't fly the flag in the rain, if you were going to fly the flag at night it had to be lit, the flag could never touch the ground, it had to be spotlessly clean and of course it could not be torn.
This bozo in front of me probably thought he was a patriotic god fearing american when in reality he could not be disgracing the flag more.
It made me mad, disgusted, and amused all at the same time. The idea that this guy would probably thinks of himself as a patriot but would beat the shit out of (or shoot) anybody who told him not to treat the flag that way.
Anyway the point is that you can convince anybody of anything. This guy bought it hook, line and sinker. He is probably driving his truck, listening to toby keith and thinking himself one fine american all the while his flag continues to disintegrate in the wind.
evil is as evil does
Even if someone came up with a pair of headphones that had an S/PDIF or AES/EBU interface, it would still have to have a DAC and an analog transducer, because my ears are not digital nor will they ever be.
Actually, you could theoretically make actual digital headphones if you could get a solenoid to move back and forth at a few GHz (for decent fidelity). Then you'd probably need to place some sort of acoustic low pass filter between the transducer and your ears* but it is possible.
To be fair, I didn't really think you could have "digital" headphones either, until you said it couldn't be done. Then I had to ask myself "Is that really true?"
* To some extent our senses act as "low pass filters" by virture of their response time. They sort of integrate incident power over a time period. A sort of human "time constant".
Life is too short to proofread.
This post got me thinking of a movie from the early 90's (?) called Crazy People. An advertising executive decides to write advertisements truthfully. Really quite funny...The movie was so far from reality though, it was sad.
One of my many jobs is participating in vendor selection for my company ([sarcasm]it's a beautiful committee process...[/sarcasm]).
Last year we had a certain computer company (IANAL, so the name is intentionally missing) come in and give a sales presentation on why we should dump our existing vendor and go with them.
For the most part, they had our existing vendor beat from a price point. But we had been burned by previous computer vendors...made all of the mistakes...and knew exactly what we wanted (and, frankly, had made our existing vendor comply with our requirements over the period of 4 years that we dealt with them)
We image all of our PCs, we have specialized software for ensuring that everything is up to a baseline and that our environment is as predictable as possible. We needed hardware that would be easily inventoried, and *consistent, long-term, globally available configurations.* There were several other requirements we laid out and prior to the "sales pitch" meeting, we supplied this vendor with these "absolute requirements."
Of course, we received a 45 minute long power point presentation that basically regurgitated back to us everything we told them were our requirements. (lesson learned: it's better not to give the marketing guys the game plan. They tend to be more honest when they don't have time to power-point the lies and instead have to provide answers off-the-cuff).
It's a running joke on our team because if we took the entire content of their presentation and crossed out every word in each bullet point that represented some sort of "promise", we'd be left with about four words repeated over and over for 20 slides..."The" "a" "and" and "but".
I don't trust *anything* from any marketing or sales rep. After testing this vendor's products and talking to friends of mine who's companies had used this vendor in the past, we knew they weren't going to live up to their promises.
From day one, the information they gave us about getting loaner PCs for testing was sold to us as "far more flexible" than it turned out to be, and this poor customer service was going on *while* we were evaluating this company to determine if we should sign the contract!
Unfortunately, as the story goes, our opinions were appreciated, but the decision to choose this company was made anyway.
Myself and another coworker were noted as objecting to the switch in our final meeting minutes. Of course, that meant nothing except for a future "I told you so." And there was nobody left to say "I told you so" to because in the end, we were the ones left having to compensate for these broken promises.
Never forget: Caveat Emptor.
"God is dead!" - Nietzsche
"Nietzsche is dead!" - God
Unless you're approaching some quantum-mechanics related limit to were the solenoid can only switch states on at the same rate as your sample rate, then I would say that the act of vibrating the solenoid is your DAC in action.
A one-bit dynamic range with a rediculously high sample-range is close to that fuzzy gray line between analog and digital.