Slashdot Mirror
Are Review Units Better Than Store Versions?
Anonymous Howard writes "Every now and then you hear about hardware manufacturers optimizing their hardware for certain tests or games to make their hardware look superior. I was surprised to hear of a new controversy brewing over reviewer units sent to hardware reviewers. This article claims that Samsung is sending LCD monitors with a contrast ratio of 700:1 when the consumer version of the same monitor has a contrast ratio of 450:1. Various sites list different specs for the same model, so it's somewhat confusing to know for sure which is correct. I don't doubt this happens, but I'm surprised that it would be this blatant. Has anyone heard of other stories of manufacturers being deceptive so that they could get better reviews?"
Review units are free and Store units are...well...not free. That would sway my opinion.
Well, you know how some radeon graphics cards can be 'unlocked' and some can't? I'll give you one guess at which I bet radeon sent to all the reviewers.
When anger rises, think of the consequences.
Confucius (551 BC - 479 BC)
Han shoots first. I think it's different in the retail version.
I heard Mandrake only sent copies of 9.2 to reviewers who didn't have LG CDROMS.
What ever happened to the ancient art of bribing the reviewer?
(Score:-1, Wrong)
Alright, I understand that this is false advertising, because the reviewed product is different from the actual product, but don't both products accurately describe the contrast? Like...the reviewed products are 700 to 1, and the consumer ones are 450 to 1...but aren't they both labelled as that? I think this would fall into one of those "check before you buy" categories...one of those common sense things maybe. As long as both products clearly indicate what their specs are, there is deception, but no actual lies.
The anti-salmon
Has anyone heard of other stories of manufacturers being deceptive so that they could get better reviews?
Yes. Thank you for asking.
Twin or more? ITA
Apache/Spring/La
buy their stuff off the shelf to use in reviews. Otherwise companies will send the cherries to reviewers.
I worked for a couple of electronic manufacturers that had a standard operating policy to do this very thing.
Take the cheese to sickbay, the doctor should see it as soon as possible - B'Elanna Torres, "Learning Curve"
But only by the fact that Samsung have never sent me any such thing.
Dammit, I got into this business for the corruption. But do I get over-spec high-dollar hardware, automobiles or prostitutes? No, I do not. It's a bloody swindle, I tell you.
Look, Samsung. 20 inch diagonal, 1600 by 1200, 700:1 contrast ratio, 16ms response time. Is that too much to ask?
Delivery address provided on application. Favourable review guaranteed.
Has anyone heard of other stories of manufacturers being deceptive so that they could get better reviews?
Quack II, anyone?
"They redundantly repeated themselves over and over again incessantly without end ad infinitum" -- ibid.
This is a crummy thing for the companies to do but it also makes you wonder about the reliability of reviewing companies. Like how he stated that designtechnica prided itself on reviewing retail products, but then never explained why they were using a review unit, and after noticing the discrepency did a lot of talking but still did not bother to pick up a shelf unit and test it, to see if it was true. Most of the hardware reviewers seem really flakey to me, more fan boys than reliable testing labs.
Unfair tweeking is part of the reason why Consumer Reports never accepts review units from companies, but rather buys them from retail stores, just like anyone else would. The other reason is that receiving free stuff creates a potential conflict of interest which is why they also do not have any advertizing in their magazine or their website. This means that you won't have reviews out before products are released, and operating this way is more expensive, relying on subscribers to run, but it is worth it. I don't always agree with CR's subjective descriptions of products (cars especially), but the hard numbers they provide are the most usefull I have found, and have saved me plenty of money.
I really wish that there was some site equally trustworthy in the computing world. For providing informative analysies there are usefull sites (I have always been impressed with anandtech). But for reviewing components, I have yet to find one I trust.
has the same model number as other retailers, but a lower price. If you look at the Bestbuy HW vs the other retailers, the best Buy HW actually is missing some 'components/functionality'.
Take a look real hard at that stereo reciever before you buy it....
a lot of these units are sent to reviews before you can even buy the product in the store. with no actual consumer version to compare it against, everyone just pretty much accepts the results as long as they are within reason. by the time people are actually buying the products, reviewers have moved on to newer products.
car companies used to do this all the time. they would send a 'ringer' to the review magazines. you would then get your car, put it on a dyno or take to a track and not be able to match the numbers.
just one of those buyer beware things.
-Shadow
I used to work for a company that produced benchmarks. We often found the reviewer machines had little extras (like more memory), more cache on the harddisk, or evil hacks (like the no-error-correction jumper) on harddisks. Sometimes they would even go as far as putting in a different processor and hope it was overlooked.
More often than not you could catch this stuff and even the playing field when reviewing hardware.
The video card hardware vendors were even more creative.
A while back, I bought a 17" samsung monitor which had 102kHz vertical refresh listed in the online "review" of the reseller, but upon closer examination, I discovered that it was, in fact, only 96kHz, so I informed them of this.
What they told me was quite strange at the time, they said their review unit had a different refresh rate and that they checked with Samsung, but that there was no definite answer as to how this could have happened. All in all, they gave me a 19" for free for the trouble (which they apparently had no part of.)
This happened in Toronto, Canada in 1998.
It is good to know SlashDot picks up on such small things.
AFAIK, Consumer Reports does not take ANY units from manufacturers because there's always the chance they'll be sent a "ringer" unit that is better than the store bought models. It would seem that this is very much case in point.
Review sites that take donated hardware and advertizing from those same hardware vendors should always be held somewhat suspect until you verify the quality through another source. Few sites are willing to give a bad item "both barrels" because they would be essentially slashing their own throat/revenue stream.
Like in "here's a free expensive item for review that you get to keep. We'll be watching the review to see if you get anything else to review? Oh, it's still happening, but sending the reviewer a item that isn't the same as the crap they intend to sell you and me is just a little added insurance.
You can pretty much see this in a lot of reviews that are written too. The only reviews that merit much trust are the independent ones where the reviewer actually went out and got an off-the-shelf item to review; but this is an almost dead pratice. No only does the reviewer not get neat fre stuff then, but his review may be months after the reviews by the company shills come out, and he ends up with the same crap you and I get rather than the free good working versions.
I'm an American. I love this country and the freedoms that we used to have.
Yes, companies will send their best stuff to reviewers, but there is a push from the opposite direction: they want reviewers to get their stuff early. In the computer world, this means that reviewers often get essentially prototypes. I've found that "first test" reviews of CPU's get processors that are worse than what the consumer will buy once production ramps up, because by then, many bugs get ironed out. AMD chips overclock much better later into the production process compared to the "for review only" samples. That's just one example.
Don't you remember the Win98 demo by Bill Gates, when it crashed? ;-) They didn't use any speciall things then! Oh.. sorry, they used their best OS at that time...
Evolution of Language Through The Ages: 6000 BC : ungh, grrf, booga 2000 AD : grep, awk, sed
I was reading a review of a Subrau WRX on caranddriver.com and it mentioned that their original review unit was able to post a 0-60 time of about 5.4 seconds. The requested another car from Subaru for a "long-term" test drive, which for them is about 60,000 miles over the span of about two years. This long-term car was equipped very similarly to the previous model, yet, it was never able do below a 5.9 0-60 time.
Now, ever car enthusiast knows that 0-60 times and such the like are subject to various conditions, but that's a pretty large inequity in the difference between the two cars. They said they must have just gotten a lucky hot car, but I believe that perhaps they got a cherry that didn't have to last as long as the car on the long-term test. If they were only going to have the car for a few weeks, then it didn't matter if it was as reliable as a longterm car, so they upped a few things and gave it to them for review. Same thing with the monitors, I guess. Since its just for the flat panel review, they might as well spice it up. These companies base a ton of business on "independent" reviews, so I suppose its worth it to fix the results.
And there's even an advertisment campaign for Gainward's line of graphics cards that specifically pokes at this concept, and doing so for quite some time ... here's just an example, and a Google search turns up many more results of this advertising campaign and the resulting products from it ...
...
Perhaps I'm just overly cynical, but I tend to trust reviews where the reviewer went out and purchased an off-the-shelf retail copy of X rather than those where the company sent something. Of course, this is hard to do in print publications, because of the time-lag that magazines run through (ie, two months after it's released on the shelves, they have a review of it), but I see no reason (aside from money, which is a big reason) that online reviewers can't do things such as this. I also tend to look towards user-reviews and give those a pretty good weigh-in when I'm making a purchase decision. This is the first instance that I can recall where products are blatently better when given to reviewers than those that are store-bought, but I get the feeling that it's been done in the past.
The above paragraph reflects what I do for my personal buying choices and should in no way construe that that's the optimal/correct/whatever way for large corporations/organizations/whatever to buy-in-bulk
topreacher@signature.slashdot.org 1% rm -rf sig
Has anyone heard of other stories of manufacturers being deceptive so that they could get better reviews?"
Well, how about this?
Search for the word 'Canada' to get to the falsification bit. Yes, this is a very old example, and no, it's not computer-related, but it still seems pretty relevant.
"Slashdot - News and Chat Sites Deviant". (Click "homepage" link above for details).
Have the companies send them a voucher or 'special id' that lets them get X product from any retail outlet. Then the hardware reviewers can pickup there stuff at Radio Shack, Future Shop, the local pc store or anywhere else.. and that way this negates any way of them 'upgrading' there products just for reviews.
No, this is
Every once in a while i read the Auto comparisons on Consumer Reports to get myself all worked up.
Consumer Reports will not accept donations of vehicles or products from manufacturers or vendors just for this reason. They will discreetly send someone out 'under cover' to go acquire the products in an "off the lot" or "off the shelf" state.
This is good, and commendable.
However, i see a lot of times they will end up mis-matching the cars and trucks they compare. Usually it is simply a matter of trim levels on similar classed models. This *will* have an impact on the final outcome. Obviously it's difficult to do things *exactly*.
Less often, but still wrongly, they will compare vehicles from incompatible classes- things like Buick Century vs. E-class Mercedes vs. Toyota Camry. Or the classic truck comparisons with the 3/4 ton, V8 powered Dodge and Chevy fullsize trucks, against a V6 F150, against V6 Toyota Tundra and Nissan.
Consumer Reports might do this to other product reviews too, but i only pay attention to their auto ads for `entertainment'.
I guess that no matter what, *any* test can be flawed.
do() || do_not();
No, they do not. Cars that must be stock for racing are stripped down and rebuilt with individual parts off the line that most closely match spec.
This is far more certain than testing every vehicle to find the "good" ones, which will never, ever, be quite as good as one assembled specifically to be good.
Even well heeled amatuers with access to a dealer's or distributer's parts bins do this. Hence classes like Star Mazda and Legends where the motor can only be touched by an official builder and has a seal affixed to it to prevent tampering.
This practice was first started in the 60's by the official Austrian Formula Vee team ( a class where every engine part must be absolutely box stock). Jochen Rindt simply ran away from the international field with a perfectly legal engine whose parts had all been individually cherry picked.
But the engine was completely stock.
KFG
I review computer hardware/software for a daily newspaper, and I can tell you that, in many cases, the quality of the hardware I get is less than what you'd see on store shelves. This is because reviewers often get pre-production units, which are essentially lacking some of the bells/whistles and fit/finish of production-line products. Whenever possible, I try to insist on production models, but that's not always possible. It's even tougher for dead-tree magazines, which work on lead times of months, to get full production units.
That said, there's no way for reviewers to know with pre-production units whether they are getting what will eventually be on the shelf - and it may not be a case of the manufacturer trying to get away with something. A processor in a pre-production unit may be faster, or an LCD screen have a greater contrast ratio, than what ends up at retail, but the reason often is that design changes are made at the last minute related to cost or part availability. In fact, sometimes the product may be less powerful in pre-production than what is finally delivered to buyers. This was particularly true in the days of falling RAM prices - I'd get review PCs with 128 MB of RAM, and when they shipped they'd have 256 MB.
C'mon, baby, kiss The King.
The early Samsung 955DF was a perfect 19" flat-black screen CRT with .20mm dotpitch. The control panel was a rectangle in the center that when pushed, slowly slid diagonally down to reveal the control buttons. Very slick.
Early Samsung 955DF
Now the "Samsung 955DF" has controls on the front, the screens are much more reflective and oily-looking, and black appears grey even when the brightness is all the way down. More recent Samsung 955DF
Two things I saw when I was writing product reviews. First, it was very common for the manufacturer to test the review unit before shipping it. It was uncommon to get hardware that hadn't been opened and resealed. Second, preproduction units often had different specs than the production models. Usually, known differences were noted, though.
A lot of the manufacturer reps and pr reps I worked with would hand-select or pre-screen review units, but I never ran into any where I thought I was being given something better than what would ship just to get a better review.
http://drteknikal.blogspot.com/
...are its usage bias of the products they review -- they assume that every product they review is made to satisfy some everyman need, and products built for speciality audiences or specifically designed to do one thing REALLY well and 3 other things just OK get dinged badly in reviews, even though the one thing they do really well may make the difference (eg, 50" HDTV with shitty speakers but FANTASTIC PQ).
This would be easily mitigated if the product reviews were serious articles with a lot more commentary and fact included in them, rather than a half-dozen photos, 4 paragraphs and a ranking table. An entire issue of CR is about 40-50 pages, with photos and tables taking up half the space.
If they reviewed the same amount of stuff every month but the magazine was another 50-100 pages of actual written content discussing the products, the review process, etc, the apparent everyman bias wouldn't be as bothersome AND the magazine would last me longer than the taxi and takeoff of an airplane flight.
Do you think if Intel sends toms hardware a new CPU to review, they don't test the hell out of it to make sure they send the best one possible - the one that can be overclocked the furthest?
Wouldn't you, if in Intels shoes?
I was burned by this when I bought the first run of the Asus P4S8X motherboard. Review sites like toms were talking about this board being the cats ass, stable as a rock.
However, the first runs of the retail version were garbage.
They had a different clock gen, a different stepping on the 648 northbridge chip, and were unstable at stock settings - myself and tons of folks were pissed. The released a bios update "fix" that underclocked the system, running DDR333 at 266 (though the bios said 333 benchmarks confirmed the real performance).
Anyways, the point is, the super mega ultra board that was reviewed was NOT what was shipped to customers. It was as simple as comparing photos in the reviews to the board to see that.
To me, the fault is the reviewers. They should go down to best buy and grab the same retail box I would.
To me, getting free booty to review == paid endorsement. I lend them as much credibility as I do John Stamos' on 1010-987.
I don't need no instructions to know how to rock!!!!
In the mid-1990's my company did quite a bit of high-profile graphics card benchmarking. Because we focused on testing chips just going into production we'd usually get alpha and beta revs of boards from the well-known board vendors at the time. It was routine for the boards that were sent for testing to have custom BIOS that set the clocks on the hardware well above specifications.
For vendors that did this chronically we switched to getting boards through other channels -- but we needed the hardware as soon as it was released, so we'd usually have pending orders with the retail arm of a board manufacturer. They got wise to this and started doing the same thing with retail boards being sent to us.
Then we switched to straw buyers. Since there were only a few preorders made in my state (AZ) they started doing it to all boards destined here, which was pretty entertaining. We'd wait a month and buy the board from a storefront and it'd be clocked 10-20% slower.
I won't even begin to go into what we saw happen with drivers...
The software industry is RIFE with such abuses.
In a previous job, my employer had a special team of people called "Product Managers" - but their job was to go visit magazine reviewers, ensure that they got top of the line grade A technical support during the review process, including onsite support, and coded patches directly from the developer's desktop to the reviewer's. Additionally, there was wining and dining, and talk of strippers and lapdances (though I never witnessed that). In that sense, what was reviewed in no way bore any resemblance to the shring-wrapped package some poor sucker paid $699 for.
I'm no longer working in that sector, but for my 10 years, the practice was commonplace. Which is why I never read reviews.
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
I think another reason they may be doing this more is to stop bogus returns. What do I mean by that? Simple, buying on the internet, picking one up at BestBuy the same day, and returning the one you got via internet to Best Buy. I did that a few times with movies, but nowadays Best Buy is pretty competitive with online prices. That was probably even a worse problem with audio and video stuff. Heck, for video stuff I can't even find some of the same models between two different Best Buys!
An example of this is with movies and CD's now. Some of them (like Indiana Jones with the bonus DVD) have different UPC codes than the ones you get online.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
I recieved a review copy of XP, but when I opened the fedex package, there was an unopened Mac OS X box with a piece of masking tape next to the "X" with "P" written on it with black sharpie pen...
:)
Needless to say, I now love Windows
We apologise for the fault in this post. Those responsible have been sacked. -- Signed RICHARD M. NIXON
Everyone here is ranting on about "Yeah, I've seen it done before in cars, video cards, etc. Only one post brought up the point that it may be a simple mistake and a little more digging would have clarified the situation. Note this:
The contrast specs on the Samsung USA site show the following:
172T - 700:1
173T - 450:1
The specs on the Samsung Canada site say:
172T - 500:1
173T - 700:1
Perhaps he got a Canadian unit although I don't know why they would be any different.
If you don't want to repeat the past, stop living in it.
The last company I worked for engineered a benchmark test before the company's new RAID SCSI host card was completed.
The card was benchmarked against a RAID Ultra SCSI LVD card from a German competitor named ICP Vortec. Even though the new card was in development, they expected it to beat the Vortec. It didn't...until they removed most of the Vortec's components and replaced them with parts from a much slower (and inexpensive) card that wasn't even considered a serious competitor. These benchmarks were used in advertisements, entire phrases stating the superiority of the product were co-opped from press releases by lazy magazine columnists (which is sadly common in most journalism nowadays), and sent to companies requesting further information. On a somewhat related note, a claim on the product sheets for the aforementioned product line that the product had been Windows certified ended up costing the company a $10K fine from MS and had to be destroyed.
I guess that ultimately it doesn't matter. The owner sold the company to the largest competitor.
Fuzzbuster gets busted by Escort and Car And Driver Mag for stuffing Escort guts inside of the Fuzzbuster supplied for radar detector review (1979):
:)
http://www.valentine1.com/lab/MikesLabRpt5.asp
While you're there, please check out a Valentine One. Mike Valentine makes by far the best detector on the planet, and he's a heck of a nice guy!
--
Out of order? Fuck! Even in the future nothing works! - Dark Helmet (Rick Moranis) "Spaceballs"