Slashdot Mirror
Are Review Units Better Than Store Versions?
Anonymous Howard writes "Every now and then you hear about hardware manufacturers optimizing their hardware for certain tests or games to make their hardware look superior. I was surprised to hear of a new controversy brewing over reviewer units sent to hardware reviewers. This article claims that Samsung is sending LCD monitors with a contrast ratio of 700:1 when the consumer version of the same monitor has a contrast ratio of 450:1. Various sites list different specs for the same model, so it's somewhat confusing to know for sure which is correct. I don't doubt this happens, but I'm surprised that it would be this blatant. Has anyone heard of other stories of manufacturers being deceptive so that they could get better reviews?"
Well, you know how some radeon graphics cards can be 'unlocked' and some can't? I'll give you one guess at which I bet radeon sent to all the reviewers.
When anger rises, think of the consequences.
Confucius (551 BC - 479 BC)
a lot of these units are sent to reviews before you can even buy the product in the store. with no actual consumer version to compare it against, everyone just pretty much accepts the results as long as they are within reason. by the time people are actually buying the products, reviewers have moved on to newer products.
car companies used to do this all the time. they would send a 'ringer' to the review magazines. you would then get your car, put it on a dyno or take to a track and not be able to match the numbers.
just one of those buyer beware things.
-Shadow
A review is supposed to be done on a random sample anyway.
Consumer reports had the right idea, that is why they have been so successful.
Like in "here's a free expensive item for review that you get to keep. We'll be watching the review to see if you get anything else to review? Oh, it's still happening, but sending the reviewer a item that isn't the same as the crap they intend to sell you and me is just a little added insurance.
You can pretty much see this in a lot of reviews that are written too. The only reviews that merit much trust are the independent ones where the reviewer actually went out and got an off-the-shelf item to review; but this is an almost dead pratice. No only does the reviewer not get neat fre stuff then, but his review may be months after the reviews by the company shills come out, and he ends up with the same crap you and I get rather than the free good working versions.
I'm an American. I love this country and the freedoms that we used to have.
Yes, companies will send their best stuff to reviewers, but there is a push from the opposite direction: they want reviewers to get their stuff early. In the computer world, this means that reviewers often get essentially prototypes. I've found that "first test" reviews of CPU's get processors that are worse than what the consumer will buy once production ramps up, because by then, many bugs get ironed out. AMD chips overclock much better later into the production process compared to the "for review only" samples. That's just one example.
And why aren't the reviewers performing their tests with retail purchased equipment for integrity sake anyway?
Have the companies send them a voucher or 'special id' that lets them get X product from any retail outlet. Then the hardware reviewers can pickup there stuff at Radio Shack, Future Shop, the local pc store or anywhere else.. and that way this negates any way of them 'upgrading' there products just for reviews.
No, this is
Cherries are one thing -- but did your employers really send out demo models with fundamentally different capacities than in the specs? I'm surprised this is the first time a reviewer noticed that, say, the 250 cc motorcycle he was reviewing looked suspiciously like a 600.
What I'm listening to now on Pandora...
The reason why is that the expense and volume of computer hardware would require the site to be an instant success.
Think about it...a good graphics card roundup should review cards from all of the companies that make a card based on a particular chipset. If there are 8 companies making that card, at $200+ retail each, that's $1600+ per review.
Of course, I don't think it's a bad idea. Just one that will take a little bit of ingenuity. A good method MIGHT be to sell advertising space not to hardware manufacturers, but to hardware SELLERS, something Anand (and Tom, who often has to buy his hardware since bad reviews have branded him unfavorable) definitely does. There's no shortage of hardware sellers, so if you piss off one there's four more right behind him.
Of course, you can also bolster your costs by RESELLING by auction the hardware you've tested.
Hey freaks: now you're ju
...are its usage bias of the products they review -- they assume that every product they review is made to satisfy some everyman need, and products built for speciality audiences or specifically designed to do one thing REALLY well and 3 other things just OK get dinged badly in reviews, even though the one thing they do really well may make the difference (eg, 50" HDTV with shitty speakers but FANTASTIC PQ).
This would be easily mitigated if the product reviews were serious articles with a lot more commentary and fact included in them, rather than a half-dozen photos, 4 paragraphs and a ranking table. An entire issue of CR is about 40-50 pages, with photos and tables taking up half the space.
If they reviewed the same amount of stuff every month but the magazine was another 50-100 pages of actual written content discussing the products, the review process, etc, the apparent everyman bias wouldn't be as bothersome AND the magazine would last me longer than the taxi and takeoff of an airplane flight.
Do you think if Intel sends toms hardware a new CPU to review, they don't test the hell out of it to make sure they send the best one possible - the one that can be overclocked the furthest?
Wouldn't you, if in Intels shoes?
I was burned by this when I bought the first run of the Asus P4S8X motherboard. Review sites like toms were talking about this board being the cats ass, stable as a rock.
However, the first runs of the retail version were garbage.
They had a different clock gen, a different stepping on the 648 northbridge chip, and were unstable at stock settings - myself and tons of folks were pissed. The released a bios update "fix" that underclocked the system, running DDR333 at 266 (though the bios said 333 benchmarks confirmed the real performance).
Anyways, the point is, the super mega ultra board that was reviewed was NOT what was shipped to customers. It was as simple as comparing photos in the reviews to the board to see that.
To me, the fault is the reviewers. They should go down to best buy and grab the same retail box I would.
To me, getting free booty to review == paid endorsement. I lend them as much credibility as I do John Stamos' on 1010-987.
I don't need no instructions to know how to rock!!!!
You are, of course, correct, but people have trouble balancing the "pay for integrity" concept against "information (particularly on the web) should be free" concept.
GreyPoopon
--
Why is it I can write insightful comments but can't come up with a clever signature?
That doesn't invalidate their methodology of buying review units off the shelf, though. Good magazines learn this lesson early. Car and Driver (which *ahem* does have good sports car reviews) has written about an early radar detector review in which they discovered one of the manufacturers had sent their outer shell with the inner workings of a different manfacturer's (much higher-scoring) device inside. They can't reasonably buy every car they test, but they can buy their radar detectors of the shelf. So now they do.
... the larger the sample size, even if it is from a self-selected sample, the better the data ...
No, the more random and representative the sample, the better the data. It is quite possible to have a very large sample size, but one that is still extremely skewed and non-representative. One of the most striking examples of this was the 1936 Literary Digest poll which had a sample size of 2,000,000 respondents and incorrectly predicted that Alf Landon would defeat Franklin Roosevelt in that year's presidental election by 57 to 43 percent. Of course, FDR defeated Landon, as correctly predicted by a George Gallup poll with a much smaller, but much more random and representative, sample of 300,000.
One obvious example of the unreliability of the self-reported reliability data in Consumer Reports is the often widely varying results from identical sibling models made on the same assembly line differing only in "brand engineering" of labels, grills, etc. This is most acute when one model wears a foreign make (e.g., Mitsubishi or Toyota) and another one wears a domestic make (e.g., Chrysler or Chevrolet).
How in the world would they pay for publishing Consumer Reports if they didn't charge you for the magazine? Maybe they could lower the price a bit for members who fill out the survey, but I'm not really sure I like that idea either.
It's nice to know that the data comes from people who also read the magazine, and are interested enough in the data to report it without any sort of incentive. I would be more worried about getting real data if any sort of incentives were offered.
For those of you who are not familiar with Consumer Reports, they do not accept any sort of advertising. Members (suscribers) pay the cost of testing and publication. People who subscribe think that this leads to less bias in their coverage.