On the Integrity of Hardware Review Sites

leathered writes "Charlie Demerjian of The Inquirer has posted an interesting article on the integrity of hardware review sites. Apparently the benefits of running such a site go far beyond advertising revenue with a fair amount of 'sweeteners' from the hardware manufacturers to say the least. All is not lost as Charlie informs us that there are a small number are flying the flag for trustworthy reviews, but the question of which sites we can trust remains." I like Daniel Rutter's (of Dan's Data) policy best.

Comments

Trust? On the net?

[winkydink]

I don't trust anybody. I'll read multiple reviews and, if available, end-user experiences as well before making a serious buy decision.

But, then again, how do you know I'm not just making this up? :)

Integrity?

[schild]

I'm not so worried about the integrity of hardware sites so much as software. But they fall under the same categories. Those that shill and whore themselves out the most get the most goodies and are loved by companies. Those that are hard on product and serious with reviews tend to be ignored. As in, just not taken seriously. The entire press outlet/developer relationship is as corrupt as 1930's Manhattan. Integrity is not a word that should be used anywhere near it. At least hardware sites SEEM to be giving real benchmarks.

Re:Integrity?

[kannibal_klown]

Hardware benchmarks aren't what bother me, it's stability that matters to me.

The first PC I built from scratch was in college (my previous computers were either my parents or a small-shop one I bought). I hit the hardware sites and many claimed that a particular ASUS board was great and rock solid. I was naive and took the 3 or 4 sites word at it and bought it. I started having MAJOR problems and later found out it was the Via chipset on the board. Forums were FLOODED with the exact same complaints. I eventually had to replace the board.

Since then I don't buy hardware until I've searched forums for personal experiences with said product. I'll still look at some review sites but I now take what they say with a grain of salt.

Re:Identification of sites he's accusing?

[PaisteUser]

The [H]ardOCP review made clear that dual core chips were weak for gaming. Any idea which sites he's fingering?

Since the article he wrote was released today, I took a look at both Tom's Hardware and AnandTech. Both had "previews" of the Dual Core Intel chips...both major sites. That's just an educated guess, but seems correct.

Im not sure if he understands multithreading

[0kComputer]

On multitasking and multithreaded apps, they will shine like the sun, but how many of these are there? How many times do you encode a movie while typing a document, zipping your C drive, doing some heavy CFD work all while listening to a few MP3s?

Correct me if im wrong but isn't multithreading/multitasking pretty damn important considering all the background tasks/services that are needed just to keep an OS running?

Let's list them then folks...

[HerculesMO]

The one site I like, though the reviews are few and far between, is Ars.Technica. Only reason, is because they BUY THEIR OWN HARDWARE :)

Anybody have any sites that they feel are bad or good (with respect to this article)? Please list a few reasons too, few examples if you can -- it makes it nice to see if these points are driven home over time by reading the reviews on different sites :)

This is not journalism

[maynard]

These are accusations of bribery and conflict(s) of interest against unspecified hardware review sites. Without naming the organization(s) and specific instances of bribery, with resulting proof, this is just irresponsible hand waving. Journalists are supposed to print facts. It's important to realize the distinction between whether bribery is taking place (possibly so), and whether this article in question backs up the assertion of bribery with documentation and quotable sources on the record. It does not. IMO: This article does NOT deserve this level of publicity, nor did it deserve publication. --M

Re:This is not journalism

[Lord_Dweomer]

I respectfully disagree. If they are not naming review sites, then nobody is being singled out. All the article does is serve to increase public awareness of the fact that on the internet, people may have a motive to not tell you the whole truth, or to sugar coat it.

This will encourage people to check with multiple sources, talk to people, and make product decisions for themselves, which is possibly one of the most important skills one can learn for surfing the internet.

That's the truth

[Anonymous Coward]

We (voodoo networks) are in the process of starting up a review site for backend systems for server rooms, closets, etc. The reasons that drove us to do this are simple - it's almost impossible to find objective reviews of non-consumer gear, and when we do find them they are pretty much copies of manufacturers flyers.

The odd thing so far has been the reaction (or lack thereof) from manufacturers. I've lost track of the number of PR reps I've spoken with who asked my how much a listing or review cost. Most where surprised when I told them that serverroomstuff.com won't be charging for reviews - they are objective.

We are planning on accepting advertising, but the plan for that is simple - your ad dollars won't buy you a good review if your product doesn't stand up to it's claims, period.

Crux of the problem

[onyxruby]

Crux of the problem is fairly simple. If you want to get the newest shiniest hardware to review and get your review out right away you can only do so with pre-release hardware. The only way to get that is to play ball with the hardware co's. Why do you think so many sites were mum about the dual athlon mp boards until one day you had a couple dozen reviews all of a sudden? That's the day the NDA's all expired.

Without this hardware you have to buy your hardware yourself. Not only is this expensive, but by the time your review is out the ad value of your review of bleeding edge hardware is kaput. Unfortunately, these are the ones that do the most honest and best reviews. Pre-release hardware is often picked out from a large selection to make sure that the review site gets a good "sample". These reviews are also the least profitable for the review sites to do. It's a nasty catch 22.

epinions

[unk1911]

epinions.com remains a good across-the-board review site, not just for hardware. good detailed specs appear next to user writeups. plus you can make money by writing reviews, which i have done.

--
http://unk1911.blogspot.com

"I take goods you send me for free & review"?

[sulli]

Consumers Union has the right policy:

We accept no advertising, and buy any products we test on the open market. We are not beholden to any commercial interest.

I'm sorry, some guy who writes reviews, even ostensibly fair ones, in exchange for free product can't stand up to this.

Re:Trust? On the net?

[ShaniaTwain]

But, then again, how do you know I'm not just making this up?

..because you were moded up?

you have the support of your peers, I trust you. what do you want me to buy?

what if everyone's in on it?

[SuperBanana]

I'll read multiple reviews and, if available, end-user experiences as well before making a serious buy decision.

What good is reading multiple reviews if they're all crap? What good are end-user experiences posted on the net, if companies are posting fake reviews, which they are?

News flash- even if they're not getting "payola" (let's call it what it is- bribe money/gear), they're controlled quite effectively by hardware companies because everyone wants to be the first site with a review of Hot Product X to drive hits to their site to earn advertising revenue. Write something bad about a product, and that company will drop you to the bottom of the list.

Let's not forget that most of these guys litter their sites with advertisements for the very product they are reviewing, too. Bob's Extreme Hardware isn't going to be very happy if young Johnny says the PC case Bob just stocked is crap- and he's going to tell young Johhny that.

Why is any of this a surprise to any reasonably intelligent individual?

Re:Trust? On the net?

[badasscat]

I don't trust anybody.

I hope that includes Charlie Demerjian. This jumped out at me:

"here is the truth, if you are going to multitask and do and do anything that tasks both of the CPUs, one of those is going to be a game."

Bullshit. This drives me crazy on hardware sites, this supposition that the only reason anyone could ever want high performance in their PC is to play games.

At home, I use Photoshop, Illustrator and Flash, I run Windows Media Center for SDTV and DVD viewing, I do video encoding using various tools (Windows Media Encoder, Dr. Divx, and others). I often do all these things at the same time on the same PC, with a hacked version of Media Center that lets me log in remotely at the same time another account has the TV going.

I studiously avoid playing games on this system, because I'm asking it to do quite enough already - I've got another system that I play games on. But I would love a dual-core CPU for this thing, as it would help me out a lot.

Graphics professionals, photographers, multimedia content producers and other high-end users are, surprise surprise, a real market, and they spend even more money than gamers do. I don't see why that's so hard to grasp. I read the specific preview of Intel's dual-core CPU's that Charlie's talking about in his comment up there and I actually found it a refreshing change to find some real-world benchmarks that were not strictly based on playing Doom 3.

That said, I'm sure there is payola going on in the industry. But I worry more about the small sites that seem to give positive reviews to every single component they get sent for free than I do about sites that realize non-gamers are a legitimate group of users that require their own set of benchmarks.

Re:Two Words

[Anonymous Coward]

Two more. Bull shit. As someone who used to work as a technical contact within a hardware marketing team, I can personally guarantee that Tom's Hardware demanded sweeteners, more than any other review site we worked with. And they were arrogant jackasses to boot.

Article is Flamebait

[Phatboy]

I'm sorry, but he states he was angry while writing this and you can tell. It's the epitome of bad journalism - he doesn't have any proof, nor make specific allegations, and those he does make seem insubstantial. For example, he flames a site that did "a review in full, or at least in half", which is quite probably Anandtech, the article of which makes it quite clear that it is only the first part.

His only criticisms of the review are that it was an exclusive (which the article makes clear) and that it doesn't cover gaming (although it is only the first part of the review). He himself admits that gaming is not the point of these chips, so why does he feel that Anandtech should have to focus on gaming is the first part of their article? Indeed, in their second part they do cover gaming and conclude that you should buy an Athlon 64 if you mainly play single-threaded games, a fact that would be obvious to anyone who regularly reads any hardware site.

I can't claim that the hardware review sites are all without bias, but compared to mainstream news, hardware reviews are some of the hardest to bias given the ease of doing standardised, repeatable benchmarks.

viperlair did a good writeup

[enrico_suave]

viperlair did a good write up on what i like to call hardware review site payola a while back.

e.

Re:Trust? On the net?

[Desert Raven]

this supposition that the only reason anyone could ever want high performance in their PC is to play games.

Agreed.

I don't play computer games, I don't even *own* any. None of my systems have ever had a game installed on them (yes, this includes solitare). All of my systems are used for work, and non-work related research. Yet, over half of my systems are dual-cpu. I multi-task, a lot. I often have graphics filters chewing away, or CDs burning, while I'm doing something else. On my servers, I want to know that if I need to compile something, archive something or do some other processor-intensive task, that there's enough processor power left to continue with the server's normal tasks.

I read reviews, but they account for less than 20% of the weight I give to my purchasing decisions. The only reviews I will pay close attention to are the *bad* reviews. They're so rare that one has to believe the product must have been truly hideous.

Re:sponsorship

[cheinonen]

First, I recall him getting a BMW convertable back then, not a Porsche. Second, that article definately takes a stab at Anand when it talks about how one site came up with a way to benchmark when people are running multiple apps at once (which Anand did), but then attacks them for not benching games. Had the person read Anand's blog on the site to keep up with what he was working on, he could have seen that Anand lacked the time to get all the benchmarks done over the weekend, did the ones he thought most important, and the gaming results went up yesterday or today.

I think it was well assumed that games would show no advantage on standard benchmarks since they don't spawn multiple threads, and every site that ran conventional benchmarks showed that on Monday. Today, however, Anand has those results, and then also tests games with thinks like Norton Antivirus and MS Antispyware running in the background, which many of us do have running when we game. On those, the dual core showed better performance.

Anand is the one site I always trust, it's too bad the article didn't have the guts to come out and name who they thought was being bought.

Re:Two Words

[SavannahLion]

Tom's Hardware was the primary reason I stopped reading his site... er....

Let me clarify, A long time ago, during the BH6 heydays, I used to read Tom's Hardware almost religiously. I scarfed down every article with near fever, trying to stay on top of the latest hardware reviews. Then I got stupid and started dating, so I ignored Ol' Tom for a few years. When my other half and I broke up, I started digging into the hardware scene again, so I went back to Tom's to get up to speed.

A few articles into it, I realized the, "feel," of the articles changed. There didn't seem to be as much useful, practical, information in them anymore. They didn't have any interesting opinions on the hardware being reviewed. In fact, I don't think I found an article that was harshly negative in any way. Nor did I find any articles with useful grit-in-your-teeth information. Initially, I thought the, "neutralism," being presented in the articles were because of the close tolerances of the hardware being reviewed. I mean overclocking a 333MHz CPU to 450MHz had greater net results than overclocking the more modern CPU's now. But I put my faith in Tom, because it's supposed to be the best. After all, his site does say:

Tom's Hardware Guide readers have come to rely on the site for unbiased and authoritative articles on the products....

So who was I to argue with that?

I just can't put my finger on what, exactly, Tom is missing. As near as I can figure, Tom's Hardware reviews read too much like brochures. It's just enough to try and get a person interested. But whatever it is, it's enough to prevent me from utilizing his site for anything more than keeping track of the latest hardware. Nowadays, if I want real hard and honest opinions, I just hop on over to my favorite forum/BBS/IRC/whatever and sort through the flamefests to get a feel for a particular piece of hardwares viability.

The good, the bad, and the ugly

[Sivar]

Slashdotters are far more likely than the average joe to recognize a good review from a bad one.

Please post your good/bad/ugly (badly written but probably not corrupt) website opinions here.

I have been following hardware reviews since about 1995, have written a few myself, and am fairly knowledgeable on the inner-workings of hardware and of English, so I have become fairly critical of reviews. Here's my brief list:

The good (Quality, non-biased, well-written reviews with reproduceable results and non-marketfluff technical explanations):

- Ace's Hardware (on the odd occasion they review something)
- StorageReview.com (I am a volunteer moderator on the forums there, so I may be considered biased, but I doubt anyone can point out a website with better hard drive reviews)
- xbitlabs.com
- Anandtech (usually, Anand usually writes well but some of the staff writes rather awkwardly, and some of their reviews, like that of the Addtronics W8500 case (and most of their other case reviews) are staggeringly awful)
- Ars Technica, though I wouldn't classify them as a hardware review site.
- Maybe HardOCP. I don't read them much, but the odd occasion I have read them didn't set off any BS detectors. Perhaps a bit unprofessional.
- Maybe DansData.com. I rarely read them, but I have always been impressed with what I read.

The bad:
- tomshardware.com (These guys were my favorite review site years and years ago, but their quality and either integrity or competence (or both) dropped staggeringly years ago. Their hard drive reviews in particular are awful beyond belief, though that may be because I am more familiar with that type of review than others).
-

The ugly:
- Just about any review site whose URL contains the word "Game." These websites tend to be full of reviewers that seem to be random average highschool kids with no real understanding of what they are reviewing, how to write a review, or how to test or rate the product they are reviewing. The word "arbitrary" comes to mind. Note that I do not put Gamespot.com in this category (even considering their owner).
- CNet. I can't tell whether the reviewers are incompetent or bribed, but the reviews (usually, but not always) are quite lame.

Any opinions on other sites?

Re:Trust? On the net?

[Ohreally_factor]

I used to work for a company that did training in digital video technology. I was a trainer, sent around the country to train people on Final Cut Pro. I was also a producer on several tutorial CD Roms, and did product reviews on the company website.

We also did marketing CDs for several equipment manufacturers, although our name never appeared in any credits. That part of the business was hush hush. We weren't to disclose those relationships.

We got lots of free equipment from these manufacturers, and others that wanted to be reviewed. We were specifically told to push certain products at the seminars and classes, and to give good reviews to "our" manufacturers products. We were also supposed to push product on the web site's forums, when people asked for advice. We never returned anything. In fact, the boss would give product to us employees as bonuses. I just checked the website, and they're still doing the same thing.

I openly questioned the ethics of this, and this and other factors led the boss to conclude that I wasn't a "team player", so I was let go. Honestly, I was glad, because there was a lot of other bullshit at that company as well. For a while, though, I was able to rationalize this because the products we were pushing were good products (for example, Canon DV cameras or Miller tripods). Still, when something is wrong, you know it's wrong, no matter how you try to justify it. If it wasn't wrong, why did we have to hide our business relationship with Canon?

Since this is pretty much par for the course, I think that if a magazine or web site really wanted to present unbiased, ethical reviews, they should state somewhere that they return the equipment, that their editorial is separate from advertising, and also disclose if the product manufacturer is an advertiser.

Of course, there still could be abuses, but when someone puts a policy in writing, it's a little harder to wiggle around basic ethics.