Slashdot Mirror
Are Review Units Better Than Store Versions?
Anonymous Howard writes "Every now and then you hear about hardware manufacturers optimizing their hardware for certain tests or games to make their hardware look superior. I was surprised to hear of a new controversy brewing over reviewer units sent to hardware reviewers. This article claims that Samsung is sending LCD monitors with a contrast ratio of 700:1 when the consumer version of the same monitor has a contrast ratio of 450:1. Various sites list different specs for the same model, so it's somewhat confusing to know for sure which is correct. I don't doubt this happens, but I'm surprised that it would be this blatant. Has anyone heard of other stories of manufacturers being deceptive so that they could get better reviews?"
Alright, I understand that this is false advertising, because the reviewed product is different from the actual product, but don't both products accurately describe the contrast? Like...the reviewed products are 700 to 1, and the consumer ones are 450 to 1...but aren't they both labelled as that? I think this would fall into one of those "check before you buy" categories...one of those common sense things maybe. As long as both products clearly indicate what their specs are, there is deception, but no actual lies.
The anti-salmon
I was reading a review of a Subrau WRX on caranddriver.com and it mentioned that their original review unit was able to post a 0-60 time of about 5.4 seconds. The requested another car from Subaru for a "long-term" test drive, which for them is about 60,000 miles over the span of about two years. This long-term car was equipped very similarly to the previous model, yet, it was never able do below a 5.9 0-60 time.
Now, ever car enthusiast knows that 0-60 times and such the like are subject to various conditions, but that's a pretty large inequity in the difference between the two cars. They said they must have just gotten a lucky hot car, but I believe that perhaps they got a cherry that didn't have to last as long as the car on the long-term test. If they were only going to have the car for a few weeks, then it didn't matter if it was as reliable as a longterm car, so they upped a few things and gave it to them for review. Same thing with the monitors, I guess. Since its just for the flat panel review, they might as well spice it up. These companies base a ton of business on "independent" reviews, so I suppose its worth it to fix the results.
Has anyone heard of other stories of manufacturers being deceptive so that they could get better reviews?"
Well, how about this?
Search for the word 'Canada' to get to the falsification bit. Yes, this is a very old example, and no, it's not computer-related, but it still seems pretty relevant.
"Slashdot - News and Chat Sites Deviant". (Click "homepage" link above for details).
Just because they don't accept advertising, and they buy units off the shelf, doesn't mean the CR review is a de-facto "better review," or is definitely less biased than a trade magazine review. Based on my experience, they represent the worst possible example of "sound bite product reviews," and they rarely give me truly useful data. (For examples, read one of their loudspeaker, audio receiver, or sports car reviews. These are generally valuable only for the pictures.)
CR tries to distill down all sorts of subtle performance parameters into a box score that ranks easily against competitive products, and in the meantime, miss the value of those parameters. Quite honestly, I'd be surprised if CR could accurately determine if they had a cherry LCD display or not, given the "rounding error" of their review/comparison methodology.
Tim
No, they do not. Cars that must be stock for racing are stripped down and rebuilt with individual parts off the line that most closely match spec.
This is far more certain than testing every vehicle to find the "good" ones, which will never, ever, be quite as good as one assembled specifically to be good.
Even well heeled amatuers with access to a dealer's or distributer's parts bins do this. Hence classes like Star Mazda and Legends where the motor can only be touched by an official builder and has a seal affixed to it to prevent tampering.
This practice was first started in the 60's by the official Austrian Formula Vee team ( a class where every engine part must be absolutely box stock). Jochen Rindt simply ran away from the international field with a perfectly legal engine whose parts had all been individually cherry picked.
But the engine was completely stock.
KFG
I review computer hardware/software for a daily newspaper, and I can tell you that, in many cases, the quality of the hardware I get is less than what you'd see on store shelves. This is because reviewers often get pre-production units, which are essentially lacking some of the bells/whistles and fit/finish of production-line products. Whenever possible, I try to insist on production models, but that's not always possible. It's even tougher for dead-tree magazines, which work on lead times of months, to get full production units.
That said, there's no way for reviewers to know with pre-production units whether they are getting what will eventually be on the shelf - and it may not be a case of the manufacturer trying to get away with something. A processor in a pre-production unit may be faster, or an LCD screen have a greater contrast ratio, than what ends up at retail, but the reason often is that design changes are made at the last minute related to cost or part availability. In fact, sometimes the product may be less powerful in pre-production than what is finally delivered to buyers. This was particularly true in the days of falling RAM prices - I'd get review PCs with 128 MB of RAM, and when they shipped they'd have 256 MB.
C'mon, baby, kiss The King.
Cost. Most review sites, especially in the PC industry, don't have much cash to purchase the products themselves. Instead they rely on vendors sending their hardware to them for free.
:)
It definitely isn't an objective model, but one that allows multiple people to review the same product. Ultimately, you have to make two really shaky assumptions in such a model.
1. The vendor is sending the reviewer a consumer level product (or nearly so).
2. The reviewer is objective and honest enough to verify the capabilities match that of the consumer available product and disclose where the product came from.
For the most part the model works. Point 1 is completely out of the consumer's control and cannot even be known to the consumer without point 2. As a result, there is only one thing that consumers can do, and that is learn to spot the honest reviewers from the frauds, fanboys, and sponsored reviewers. For example, Anandtech and Tom's Hardware Guide do a pretty good job. They clearly indicate where the hardware comes from, identify any differences between it and the shipping hardware, and do their homework as best they can to prevent getting duped. Contrast this with many reviewers who seem to be simply paying lip service to the vendors so they can get quoted in an advert and continue to get free hardware to play with. Researching the product also means researching the reviewer if you don't wanna get burned. It's like taking advice from the Gartner Group without seeing who paid for their latest study.
"The words of the prophets are written on the Slashdot walls."
I thought the usual game was for these kinds of retailers to sell model numbers that don't exist anywhere else, but have the same specs as a commonly available one, so that they can "honestly" deny any compettitive pricing claims. ;-)
Having worked at best buy, in the audio department, I can tell you that this is a fact. We sold products that other people did not sell; in fact, it was fairly rare to find products from well known manufacturers that had the same model number at Circut City and at Best Buy. It was done so that we could a.) say that we had exclusive products, and b.) say that we weren't doing competative pricing, but the much bigger one was 3.) when the models had different features, some people want the one from CC, some want the one from best buy. If they had Identical features, people would just go the place with the lower price, but because one may have an extra S-video input or what-have-you, they're willing to shell out the extra $20 for the extra stuff. The trick is for the corporation to get the model that looks more desireable to consumers.
But, yeah, the main reason for the similar-but-different model numbers for similar-but-different products is to keep people from being able to compare identical products, and thus, simply wait for the sale. It's perfectly legal, and when you think about it, pretty smart.
~Will
sig?
To be specific, it's not "unlocking" as such, but some 9500 cards can be upgraded to a 9700 by changing the BIOS.
They both are based on the same R300 chip; apparently the 9500 is a 9700 with only half the pixel shaders enabled (four vs. eight). The PS is the most complex -- and difficult to manufacture -- part of the chip, so ATI smartly anticipated that they'll get chips with some of the PS units broken but everything else working, and designed the architecture so that they can easily (via BIOS) "halve" these into 9500 versions. So far so good. But apparently they didn't get enough of these half-broken chips to satisfy the demand for the 9500 (what with the superb bang for the buck!) and had to use the 100% functional "9700 class" chips for the 9500 boards. These and only these individuals can be upgraded! (From 9500 to 9700.)
But note that ATI *never* advertised this upgrading as an option. Why the hell would they -- it has already eaten into the 9700 sales. And IIRC it was never discovered nor discussed in the (shipping time) reviews, but somewhat later.
It's another matter that only the 9500 "Pro" cards have the 256-bit memory bus. The 9500 "non-Pro" cards have the 128-bit bus, so you'll never get 9700 class performance out of it. (9700 Pro and non-Pro are both 256-bit -- there the difference is only clock speeds.)
(This has been widely discussed on the more tech oriented gaming and 3D forums.)
I think another reason they may be doing this more is to stop bogus returns. What do I mean by that? Simple, buying on the internet, picking one up at BestBuy the same day, and returning the one you got via internet to Best Buy. I did that a few times with movies, but nowadays Best Buy is pretty competitive with online prices. That was probably even a worse problem with audio and video stuff. Heck, for video stuff I can't even find some of the same models between two different Best Buys!
An example of this is with movies and CD's now. Some of them (like Indiana Jones with the bonus DVD) have different UPC codes than the ones you get online.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
I gave up completely on CR after reading their reviews of Saturn cars. Now I know Saturns aren't for everyone, but they have some good qualities that most people don't dispute (their safety record, for one). Anyway, after looking at CR's table of parameters that ranked the Saturn I was interested in above average in almost every category, I read the few paragraphs of text which went on to make the car out to be one of the worst models ever tested. The disconnect between the measurables and the opinion basically told me that the review was worthless.
I heartily agree with this. Consumer Reports reviews ae pretty close to useless in many areas. Yes, they tell you which product they liked better. But they never seem to give you any of the numbers or any information on how they decided which product was better. They will tell you that one receiver performs better than the others, but they don't tell you how they tested it or exactly what the performance edge actually is. In areas of technology, I almost always disagree with them, especially things that I have extensive experience with.
About the only thing that is remotely useful in most CS reviews is the reliability data they have for some things - but they only have this for old products, so it is less usefull when I'm looking at buying something new.
In general, I think that CS has the right idea in insisting on purchasing all of the products they review - this gets rid of a lot of potential problems with cherries and such. However, they never seem to say anything about their testing parameters or methodology, so I feel like I can never really trust what they say (of course, disagreeing with most of the reviews they have on products I am familiar with contributes a lot to this).
My chief complaint is the lack of breadth in the product reviews. They don't review every model, or even every (or even most) competing products.
So the review is incomplete, at least to me.
For example, in the (aged) review on vaccuum cleaners, I beleive that I have purchased one of the better (if not the best) models on the market. Yet CR has never reviewed any machine from the company.
Also, they do not update their product ratings. If they reviewed a series of toasters in 1997, and that's the latest review, that's all the information you get and you have to go elsewhere.
Bottom line: They'd have far more of my money (and confidence) if they reviewed more competing products, and kept those reviews up to date.
If all you have is a hammer, everything looks like a nail.