Review Scores the "Least Important Factor" When Buying Games

A recent report from a games industry analyst suggests that among a number of factors leading to the purchase of a video game — such as price, graphics and word of mouth — the game's aggregated review score is the least important measure. Analyst Doug Creutz said, "We believe that while Metacritic scores may be correlated to game quality and word of mouth, and thus somewhat predictive of title performance, they are unlikely in and of themselves to drive or undermine the success of a game. We note this, in part, because of persistent rumors that some game developers have been jawboning game reviewers into giving their games higher critical review scores. We believe the publishers are better served by spending their time on the development process than by 'grade-grubbing' after the fact."

Comments

Let me be the first to say...

[thijsh]

maybe it's the other way around... You only have to buy a sucky game *once* based on a raving review to *never* trust those reviews again. While your friends can comment properly on the game without some obscure metric like '8/10 overall'.

Re:Let me be the first to say...

[Forge]

This phenomenon is not unique to Games. I watched a TV Rerun of the Matrix last night and remembered that some Newspaper reviews were very harsh on it. That I would watch it again after all these years and loosing count of how many times I have seen it suggests that the reviewer has standards incompatible with my own. That is not the worse case however.

There was the Mag Innovision letter to the editor after it's 17" monitor received the worst ranking in a roundup of 17" monitors. The Editor's choice award went to a Gateway 2000 monitor. The point of the complaint letter? "This is the same monitor, we just print different labels on the ones we ship to Gateway 2000." Or words to that effect.

So as a general rule I have very little use for published reviews of any product. Word of mouth, and personal trials work best. Also it's good to know what advertisers are allowed to lie about.

"This POS is the best on the market" -: Allowable lie.
"This overpriced crap is great value for money" -: Acceptable lie.
"This 500GB drive holds more data than 750GB of data without using compression" -: dangerous ground.

Does anyone really believe the scores ?

[DCFC]

If a magazine or website is really scoring out of 10 or out of 100, then we ought to see some 1's and 2's.
But we don't do we ?

The researchers would find more utility in measuring the correlation between ad spend and score.

Anyone think these two variables don't correlate strongly ?

Re:Does anyone really believe the scores ?

[clickclickdrone]

>But we don't do we ?
There is a reason for that. It's a lot of hard work and cost bringing a product to market and generally, the real dogs are killed long before they hit the shelves. I've been reviewing hardware/software for 20 odd years now and I can only remember giving a score of less than 4 a handful of times. Equally, 9 & 10 is rare (for me). The vast majority of stuff is 'good enough' and merits 7 or 8 out of 10. TBH, I get really frustrated by constantly dishing out 7s and 8s and the few times something has turned up for review that's truly bad, I'm been delighted as it gives me a chance to have a real opinion.

Re:Does anyone really believe the scores ?

[RogueyWon]

The problem is that you need a scale that encompasses everything from "hideously bad" to "sublimely good", and very, very few commercially released games these days actually fall into the former category. Sure, the usual anti-modern-gaming crowd here on slashdot likes to decry the latest overhyped blockbuster as "worst game ever", but in reality, pretty much every such game is "mediocre" at worst, and actually reasonably good fun if considered in isolation, on its own merits. It's not really fair to score a game down for being overhyped - only to review the game in front of you.

Genuinely bad games with genuinely low review scores do exist. Even if you look at IGN, who are generally felt to "score high", you can use the review filters to find plenty of games with scores of 3.0/10 or less. These are mostly clustered on the PC, Wii, PS2 and handhelds - platforms with relatively low development costs prone to low-quality shovelware (which is by no means to decry all titles for those systems as low quality). However, the development costs for high-end games these days are such that you really can't afford to let an absolute stinker go out the door. This does make the odd rare exception that slips through, such as Lair, all the more deliciously awful.

So yes, it's not a big conspiracy that you tend to get a clustering of review scores around the 7-9/10 mark. It's just a fair reflection of the overall quality of most modern big-budget games. Reader reviews, on the other hand, often tend to be callibrated to a less objective scale, and to take more account of factors such as the degree to which the game had been hyped (and to the kind of emotive factors that the console wars stir up), leading to a wider variation.

You do, of course, get the occasional game where the "professional" review scores seem a bit out of whack. Modern Warfare 2 felt like a bit of an example of this to me; I could have seen it as an 8/10 kind of game, but I suspect that review scores above that are being hype driven.

Ultimately, I find that the best way to use reviews isn't to go off meta-critic rankings or composite scores. It's to find a review site whose tastes generally accord with my own and use this as a rough guide. I already know in advance broadly which games I'm interested in. If I read the review, I use it as a guide-post and look for issues mentioned that are of particular importance to me. If a review flags that a game has an overly restrictive save-system, then I won't buy it even if the score is good, because I hate repeating content I've already passed unnecessarily. If a review criticises and marks down a game for not including online play, however, I won't let that deter me; it's not usually a huge issue for me, as aside from WoW, I'm primarily a singleplayer gamer.

Re:Does anyone really believe the scores ?

[Inda]

Does anyone buy games without trying them first? For me, if there is no demo, there is no sale.

I haven't even read a review in 20 years.

Re:Does anyone really believe the scores ?

[LatencyKills]

I've been doing game reviews online for about a decade now, and as I look back over my reviews I find the ratings pretty much hit the full spectrum from 95% (Bioshock) all the way down to 14% (Dukes of Hazard - Racing for Home). I'm also a constant consumer of game reviews for games that I buy that I don't review. I think for all reviews that have a high degree of opinion (movies, books, videogames) it is important to find a couple of reviewers who feel like you do and stick with them. It's clear that specifically to the videogame realm there is a high degree of sellout (I won't name names) but here's a hint - avoid a review for a game that has a banner ad for that game on the same page as the review.

Trust

[Aceticon]

It all boils down to trust (or more specifically lack of it) on the Game review sites.

I'm not overly surprised that people don't base their buying decisions primarily on the review scores from game-sites. In most sites I've seen one or all of the following:

• Creeping scores: games coming out now get higher average scores than games that came out 2 or 3 years ago
• No mention of the bugs: a game might be full to the brim with bugs on release day and yet there is no mention of it or a passing reference to "the version we tested had some problems but this is a pre-release version and they should be solved before release" (not!)
• Hype: game sites are the main culprits in creating/maintaining hype on often undeserving games. For example, before release Spore was being hyped to death by most game sites as a grand, revolutionary game - as it turns out, it wasn't even that much of a fun game and after release the hype-machine went suddenly quiet
• Shallow reviews on just the beginning of the game: a lot of reviews sound like the person doing it just played the game for a couple of hours and then wrote the review. Plenty of games out there become pointless and boring after a couple of hours playing them, and yet that's often not mentioned in the game reviews
• No mention of intrusive DRM: often enough the games come out with crazy phone-home, only works if Internet is always on, DVD-Writer-breaking DRM which installs a rootkit in our machine and yet not a peep about it in the main gaming sites. I suppose some might mention it if the activation process involved "chop your grandmother into little pieces and send them to the following address ..." but must would not

Personally, I usually wait a while after the game is out and then go check user reviews. If your discount the "100%, great thing since sliced bread" ones (which come from fanboys) you'll usually be able to get a good picture of all the above mentioned points that the game sites miss (bugs, long-term (re)playability, intrusive DRM, hands-on-beyond-hype experience)

Except you just illiustrate the problem

[Moraelin]

Except you just illustrate the problem: something that's just "good enough" (which really just means "mediocre") gets an 8 out of 10. I'm sorry, but in a perfectly linear scale, "mediocre" would mean a 5. That's the kind of a number you could punch in a formula and get a correlation or anything else.

Plus, if it were just a case of a honest review and the bad ones being already cancelled, the results would look much like the right half of a bell curve. You know, the curve with the below average ones removed. For virtually any sitze out there, it doesn't. It looks like a bell curve centered on 8 or 9, and which pretty much starts at 6 or 7. Sorry, that's not a case of the bad ones being already removed, that's a clear sign of an offset scale. It's what you get when the occasional "something that's truly bad" means you get to give a 5 or a 6, not a 1 or 2.

And then there is the occasional reviewer whose curve looks like two spikes. The kind who churns 90% to 99% scores all year long, and then occasionally picks up some 10 year old freeware game so he can give _something_ a 5% score and fix his street cred. Or publishes a yearly smack-talking "top 10 worst games of all time" -- conveniently all 20 years old and from publishers which are no longer in business -- just to show that he's that unbiased and can give a low score too.

But again, that's not being unbiased and fair at all, it's just trying to compensate one crap (or dishonest) job with another one skewed in the other direction. If it were a real fair and unbiased and non-skewed job, you'd get one bell curve centered in the right place, not two spikes centered near the extremes of the scale.