Tom's Hardware To Cardmakers : Game Over

Merlynnus writes "Today, Tom's Hardware has announced that they're tired of being jerked around by the video card manufacturers and are going to publish reviews on their own schedule rather than one dictated by ATI or NVidia. From the site: "Reviews of cards with pre-launch buggy drivers and yet-to-come updates do no service to the user. We explain why we will no longer play that game."

Comments

who's fault is this?

[fireduck]

Tom's were the people putting the reviews out early using buggy drivers. They're the only ones to blame.

Nvidia and ATI are all about getting as much free publicity as possible. If slipping a card to Tom's gets a pre-release review that breathlessly points out how it gets 3.5 fps more at 1600x with FSAA turned on, the manufactorer's are going to do it. You can't really fault them for trying to one up the competition.

However, the article seems like TH are blaming the manufacturer's for this. Take this quote for example: "Maybe everyone has lost sight of the fact that all the analysis that is done is supposed to serve the consumer, the interested reader and to help them make informed decisions." The only people who seem to have forgotten this are the reviewers at TH and other sites that push to get their exclusive first reviews out before the product or any other website beats them to it.

And then, there's this doozy in the opening paragraph: "It's always the same, with no time to do thorough testing, constant last minute changes, and a rush to print. If we were cynical, we'd have to think that the companies involved do not want us to have time to do thorough testing." Are the companies actually giving them the cards and saying "you have 40 minutes to post a review?", or is it more likely TH is given the card (maybe around the same time a competing site is given a card), and TH is just being quick and sloppy to get the first review out?

No one is forcing them to publish reviews of products they've hardly tested except themselves. While it's good that they seem to recognize the symptom (shoddy reviews), it's be nice if they recognized the underlying problem (they want to be first to benchmark) rather than pawn it off on the manufacturers (who are just out to get free publicity).

Re:who's fault is this?

[fireduck]

again, it boils down to sites wanting to be first, even if first means 1 minute after midnight when the NDA expires.

The whole idea that we need a pre-release review is stupid. Every top of the line video card and CPU is going to be slower than the next top of the line product within a few months of release. That's practically a given. (so a pre-release review on a not-yet-ready for ready for primetime product is just going to qualitatively confirm this.) And its also a given that software is continually tweaked up until release, and even after release. So, if the review sites really wanted to do a service to their readers (rather than a service to their advertisers and their own pockets), they would wait until the product is finalized, and the shipping drivers are locked. Even if this means waiting until the product is shipping.

And why are they even reviewing products in beta-test stage (as pointed out in the 2nd paragraph)? Particularly in a quantative sense. It's absolutely fine to review the beta for World of Warcraft and say "generally, its loads of fun". But it's no where near fine to review nVidia's beta BFGPU and say it's 4.36x faster than ATIs Wonderblunder. That's just shoddy reporting of test results that are going to change depending on which beta version they're using.

The 4 points that they list in the article are fairly obvious. 1) We shouldn't rush a review because they tell us to. 2) we shouldn't review stuff that isn't finished. 3) we shouldn't review stuff that isn't going to be available for several months. 4) we (the collective review site community) shouldn't whore ourselves to all put out first day reviews simultaneously.

While it's nice that they spell it out, why haven't they been practicing this since day 1? Where is the mea cupla? Nice plain English that says "We chose to be a PR machine for nVidia/ATI and have released shoddy reviews in the past. We're sorry and this is how we're making it up to you..." The whole article pawns responsibiilty for the problem off to the manufacturers, in essence saying they are forcing review sites to publish reviews quickly. No, they aren't. Its the review sites who have become too accustomed to nVidia/ATI's nipple too actually adhere to some sort of journalistic principle that are to blame.

If the fear of not being first trumps being right, then perhaps these websites shouldn't be in the review business to begin with.

Good for Tom's

[Derkec]

I tend to go visit Tom's when I'm looking to purchase new hardware. Do I really care if a review comes out a week later than the other sites? Hell no. I won't be reading it for a month anyway.

I hope they are able to pull this off without losing readership. I think they can cede the first review of a product and go for providing the best review. That's strong marketting.

Expectations for reviewers

[0x0d0a]

I moved from Tom's Hardware to AnandTech, but I think that you're doing Tom a disservice here. While I agree that Tom's isn't perfect, that doesn't make his point here any less valid. The "preview culture" and hype does customers no good and gives them a good deal of hype.

As a matter of fact, I would expect a good, reputable product review site to follow the following policies:

1) No non-shipping products. This is a *review* not an advertisement (i.e. "preview"). What good is a review two days earlier if it's incorrect and misleading? If it isn't shipping, it's a "preview" and should be called that (and those of us looking for hard product data can skip it).

2) Absolutely no reviewing based on promises. If a driver is buggy and the product crashes, then the product crashes. Calling the vendor and saying "Hi, this is Mike's Review Center and we'd like to know whether you plan to fix this issue soon so that we know what to tell our readers" is fruitless. Of *course* they will tell you that they're working on the problem and will have it fixed in the moment. You are reviewing the customer experience. If a customer can go out to a store and buy the product and it crashes, then that product has problems that need to be mentioned. If there is an existing patch, then it is certainly reasonable to apply the patch (and mention to readers that you had problems that had to be corrected with a patch). However, reference to what "NVidia says" or "Adobe says" has no place in a review.

3) Use a sane resolution for ratings. Frequently reviewers use an absolutely insane resolution for their product ratings, such as a 1-20 scale (often 1-10 with half-point increments) or even a 1-100 scale). There is absolutely no way that they have the ability to make that kind of call. Opinions are already subjective, and it's a good bet that the reviewer is unable to (a) consistently give the same rating within a 10% range and (b) consistently give the same rating as other reviewers within even a 30% range. Eliminate this kind of false data. I could see using a "Very impressed", "Somewhat impressed", "unimpressed", "Somewhat disappointed", "Very disappointed" scale.

4) Eliminate the concept of a "combined rating". Frequently, products (especially games) are reviewed based on a scale where they are given ratings in several areas. "Graphics", "Audio", "Sound", and so forth. There is no reasonable mathematical way to make an effective "combined" rating of these scores. If you want to have a category for "Overall opinion", that's great -- but don't try to make it a mechanical synthesis of the other scores. I remember when themes.org tried doing an essentially mechanical score, with very little leeway given to the reviewer -- it was disappointing. If a user cares most about reviewers having good "overall opinion" of a game, great -- let them search based on that criteria. If a reviewer really likes or dislikes a game, even though it has fantastic/really bad sound, they shouldn't be restricted from panning/extolling the game.

5) Expect reviewers to note what percentage of the game they played/how many hours they played. If a reviewer played half the game and gave a review (not unreasonable for a large game, in my opinion -- in the case of a strategy game with long campaigns, 50% of the levels is probably far overkill). If they played the whole game, great. If they played five levels at the beginning and skipped to the last level to play that, they should mention it. There's no secrets involved, but I think that it is important to let people have honest data on what portions of the games were included in the review.