Slashdot Mirror
Tom's Hardware To Cardmakers : Game Over
Merlynnus writes "Today, Tom's Hardware has announced that they're tired of being jerked around by the video card manufacturers and are going to publish reviews on their own schedule rather than one dictated by ATI or NVidia. From the site: "Reviews of cards with pre-launch buggy drivers and yet-to-come updates do no service to the user. We explain why we will no longer play that game."
A lot of that message, especially the end, sounds like it's directed more towards other review sites than their to their general readers. Maybe they're hoping others will follow their precedent.
They're a large enough review site that they could hold out by themselves, but I doubt it will make any real difference unless some of the others adopt a similar position.
Tom's were the people putting the reviews out early using buggy drivers. They're the only ones to blame.
Nvidia and ATI are all about getting as much free publicity as possible. If slipping a card to Tom's gets a pre-release review that breathlessly points out how it gets 3.5 fps more at 1600x with FSAA turned on, the manufactorer's are going to do it. You can't really fault them for trying to one up the competition.
However, the article seems like TH are blaming the manufacturer's for this. Take this quote for example: "Maybe everyone has lost sight of the fact that all the analysis that is done is supposed to serve the consumer, the interested reader and to help them make informed decisions." The only people who seem to have forgotten this are the reviewers at TH and other sites that push to get their exclusive first reviews out before the product or any other website beats them to it.
And then, there's this doozy in the opening paragraph: "It's always the same, with no time to do thorough testing, constant last minute changes, and a rush to print. If we were cynical, we'd have to think that the companies involved do not want us to have time to do thorough testing." Are the companies actually giving them the cards and saying "you have 40 minutes to post a review?", or is it more likely TH is given the card (maybe around the same time a competing site is given a card), and TH is just being quick and sloppy to get the first review out?
No one is forcing them to publish reviews of products they've hardly tested except themselves. While it's good that they seem to recognize the symptom (shoddy reviews), it's be nice if they recognized the underlying problem (they want to be first to benchmark) rather than pawn it off on the manufacturers (who are just out to get free publicity).
I don't think its really necessary to save Tom's bandwidth, Tom is a massive site, and I'm sure they are quite used to being slashdotted.
You saving him bandwidth can only stop people going to the website and seeing thier adverts (which is how they pay the bills). Despite what you may or may not think about banners etc, you are not helping them at all, you are hurting them.
I stopped reading Tom's a LONG time ago. He loves mudslinging, both with manufacturers and with other enthusiast sites. I'm close to giving HardOCP the heave-ho as well. I want INFORMATION, not grousing about how this company did so-and-so wrong, or what site is faking their benchmarks. I can figure that stuff out for myself.
but didn't that guy sound a little self-important? Sure the readers deserve a good review, but it sounds like this guy went and publicly demonized NVIDIA and ATI because he's under a lot of pressure. To me that seems a little immature given that many people in those 'large, faceless companies' probably think they're locked in the 'battle of their lives...'
To me the guy seems to think his job is really that important, when it's not... it's just a review, and if he wanted to clarify by saying that they sent it to him last-minute, then he should say so. He's getting a free card (at least for a little while) out of it, I wouldn't complain too much...
I tend to go visit Tom's when I'm looking to purchase new hardware. Do I really care if a review comes out a week later than the other sites? Hell no. I won't be reading it for a month anyway.
I hope they are able to pull this off without losing readership. I think they can cede the first review of a product and go for providing the best review. That's strong marketting.
If they really wanted to be objective the review sites would buy their own products retail with no indication that it is for a review site. This is what Consumer Reports does. The goes for software reviews as well. Yes, this requires a different business model than review sites are using now, but right now they are not objective; they are a part of the business they are covering.
I moved from Tom's Hardware to AnandTech, but I think that you're doing Tom a disservice here. While I agree that Tom's isn't perfect, that doesn't make his point here any less valid. The "preview culture" and hype does customers no good and gives them a good deal of hype.
As a matter of fact, I would expect a good, reputable product review site to follow the following policies:
1) No non-shipping products. This is a *review* not an advertisement (i.e. "preview"). What good is a review two days earlier if it's incorrect and misleading? If it isn't shipping, it's a "preview" and should be called that (and those of us looking for hard product data can skip it).
2) Absolutely no reviewing based on promises. If a driver is buggy and the product crashes, then the product crashes. Calling the vendor and saying "Hi, this is Mike's Review Center and we'd like to know whether you plan to fix this issue soon so that we know what to tell our readers" is fruitless. Of *course* they will tell you that they're working on the problem and will have it fixed in the moment. You are reviewing the customer experience. If a customer can go out to a store and buy the product and it crashes, then that product has problems that need to be mentioned. If there is an existing patch, then it is certainly reasonable to apply the patch (and mention to readers that you had problems that had to be corrected with a patch). However, reference to what "NVidia says" or "Adobe says" has no place in a review.
3) Use a sane resolution for ratings. Frequently reviewers use an absolutely insane resolution for their product ratings, such as a 1-20 scale (often 1-10 with half-point increments) or even a 1-100 scale). There is absolutely no way that they have the ability to make that kind of call. Opinions are already subjective, and it's a good bet that the reviewer is unable to (a) consistently give the same rating within a 10% range and (b) consistently give the same rating as other reviewers within even a 30% range. Eliminate this kind of false data. I could see using a "Very impressed", "Somewhat impressed", "unimpressed", "Somewhat disappointed", "Very disappointed" scale.
4) Eliminate the concept of a "combined rating". Frequently, products (especially games) are reviewed based on a scale where they are given ratings in several areas. "Graphics", "Audio", "Sound", and so forth. There is no reasonable mathematical way to make an effective "combined" rating of these scores. If you want to have a category for "Overall opinion", that's great -- but don't try to make it a mechanical synthesis of the other scores. I remember when themes.org tried doing an essentially mechanical score, with very little leeway given to the reviewer -- it was disappointing. If a user cares most about reviewers having good "overall opinion" of a game, great -- let them search based on that criteria. If a reviewer really likes or dislikes a game, even though it has fantastic/really bad sound, they shouldn't be restricted from panning/extolling the game.
5) Expect reviewers to note what percentage of the game they played/how many hours they played. If a reviewer played half the game and gave a review (not unreasonable for a large game, in my opinion -- in the case of a strategy game with long campaigns, 50% of the levels is probably far overkill). If they played the whole game, great. If they played five levels at the beginning and skipped to the last level to play that, they should mention it. There's no secrets involved, but I think that it is important to let people have honest data on what portions of the games were included in the review.
May we never see th
There are always strings attached. "We'll give you this pre-release card if you promise to get the review out on the 12th.".
If they insult the manufacturers, or blow them off, they won't get an early card next time. There are only a few available anyway.
What would happen if TH had to buy a card off the shelf? They'd have late reviews. Would they still get readers? Depends if the readers want a quick review of a pre-release card, or an indepth review that can be trusted. Both readers exist.