Slashdot Mirror
Are Review Units Better Than Store Versions?
Anonymous Howard writes "Every now and then you hear about hardware manufacturers optimizing their hardware for certain tests or games to make their hardware look superior. I was surprised to hear of a new controversy brewing over reviewer units sent to hardware reviewers. This article claims that Samsung is sending LCD monitors with a contrast ratio of 700:1 when the consumer version of the same monitor has a contrast ratio of 450:1. Various sites list different specs for the same model, so it's somewhat confusing to know for sure which is correct. I don't doubt this happens, but I'm surprised that it would be this blatant. Has anyone heard of other stories of manufacturers being deceptive so that they could get better reviews?"
Bah. They had the same model number on two different models. That's a lie.
Is the difference between an outright lie and a deception really that important here anyway? They were expected to send the same product real consumers get. They didn't. That's enough to condemn them in my book, whether there's an outright lie there or not.
I was reading a review of a Subrau WRX on caranddriver.com and it mentioned that their original review unit was able to post a 0-60 time of about 5.4 seconds. The requested another car from Subaru for a "long-term" test drive, which for them is about 60,000 miles over the span of about two years. This long-term car was equipped very similarly to the previous model, yet, it was never able do below a 5.9 0-60 time.
Now, ever car enthusiast knows that 0-60 times and such the like are subject to various conditions, but that's a pretty large inequity in the difference between the two cars. They said they must have just gotten a lucky hot car, but I believe that perhaps they got a cherry that didn't have to last as long as the car on the long-term test. If they were only going to have the car for a few weeks, then it didn't matter if it was as reliable as a longterm car, so they upped a few things and gave it to them for review. Same thing with the monitors, I guess. Since its just for the flat panel review, they might as well spice it up. These companies base a ton of business on "independent" reviews, so I suppose its worth it to fix the results.
Just because they don't accept advertising, and they buy units off the shelf, doesn't mean the CR review is a de-facto "better review," or is definitely less biased than a trade magazine review. Based on my experience, they represent the worst possible example of "sound bite product reviews," and they rarely give me truly useful data. (For examples, read one of their loudspeaker, audio receiver, or sports car reviews. These are generally valuable only for the pictures.)
CR tries to distill down all sorts of subtle performance parameters into a box score that ranks easily against competitive products, and in the meantime, miss the value of those parameters. Quite honestly, I'd be surprised if CR could accurately determine if they had a cherry LCD display or not, given the "rounding error" of their review/comparison methodology.
Tim
No, they do not. Cars that must be stock for racing are stripped down and rebuilt with individual parts off the line that most closely match spec.
This is far more certain than testing every vehicle to find the "good" ones, which will never, ever, be quite as good as one assembled specifically to be good.
Even well heeled amatuers with access to a dealer's or distributer's parts bins do this. Hence classes like Star Mazda and Legends where the motor can only be touched by an official builder and has a seal affixed to it to prevent tampering.
This practice was first started in the 60's by the official Austrian Formula Vee team ( a class where every engine part must be absolutely box stock). Jochen Rindt simply ran away from the international field with a perfectly legal engine whose parts had all been individually cherry picked.
But the engine was completely stock.
KFG
I review computer hardware/software for a daily newspaper, and I can tell you that, in many cases, the quality of the hardware I get is less than what you'd see on store shelves. This is because reviewers often get pre-production units, which are essentially lacking some of the bells/whistles and fit/finish of production-line products. Whenever possible, I try to insist on production models, but that's not always possible. It's even tougher for dead-tree magazines, which work on lead times of months, to get full production units.
That said, there's no way for reviewers to know with pre-production units whether they are getting what will eventually be on the shelf - and it may not be a case of the manufacturer trying to get away with something. A processor in a pre-production unit may be faster, or an LCD screen have a greater contrast ratio, than what ends up at retail, but the reason often is that design changes are made at the last minute related to cost or part availability. In fact, sometimes the product may be less powerful in pre-production than what is finally delivered to buyers. This was particularly true in the days of falling RAM prices - I'd get review PCs with 128 MB of RAM, and when they shipped they'd have 256 MB.
C'mon, baby, kiss The King.
Cost. Most review sites, especially in the PC industry, don't have much cash to purchase the products themselves. Instead they rely on vendors sending their hardware to them for free.
:)
It definitely isn't an objective model, but one that allows multiple people to review the same product. Ultimately, you have to make two really shaky assumptions in such a model.
1. The vendor is sending the reviewer a consumer level product (or nearly so).
2. The reviewer is objective and honest enough to verify the capabilities match that of the consumer available product and disclose where the product came from.
For the most part the model works. Point 1 is completely out of the consumer's control and cannot even be known to the consumer without point 2. As a result, there is only one thing that consumers can do, and that is learn to spot the honest reviewers from the frauds, fanboys, and sponsored reviewers. For example, Anandtech and Tom's Hardware Guide do a pretty good job. They clearly indicate where the hardware comes from, identify any differences between it and the shipping hardware, and do their homework as best they can to prevent getting duped. Contrast this with many reviewers who seem to be simply paying lip service to the vendors so they can get quoted in an advert and continue to get free hardware to play with. Researching the product also means researching the reviewer if you don't wanna get burned. It's like taking advice from the Gartner Group without seeing who paid for their latest study.
"The words of the prophets are written on the Slashdot walls."
I thought the usual game was for these kinds of retailers to sell model numbers that don't exist anywhere else, but have the same specs as a commonly available one, so that they can "honestly" deny any compettitive pricing claims. ;-)
Having worked at best buy, in the audio department, I can tell you that this is a fact. We sold products that other people did not sell; in fact, it was fairly rare to find products from well known manufacturers that had the same model number at Circut City and at Best Buy. It was done so that we could a.) say that we had exclusive products, and b.) say that we weren't doing competative pricing, but the much bigger one was 3.) when the models had different features, some people want the one from CC, some want the one from best buy. If they had Identical features, people would just go the place with the lower price, but because one may have an extra S-video input or what-have-you, they're willing to shell out the extra $20 for the extra stuff. The trick is for the corporation to get the model that looks more desireable to consumers.
But, yeah, the main reason for the similar-but-different model numbers for similar-but-different products is to keep people from being able to compare identical products, and thus, simply wait for the sale. It's perfectly legal, and when you think about it, pretty smart.
~Will
sig?