Slashdot Mirror
MetaFuture Talks Review Inflation
MetaFuture, a game journalism analysis site, has recently refocused on review scores from the big gaming sites. The author takes an interesting approach, taking a look at Gamespot's review spread and IGN's tendencies. Unsurprisingly, both sites tend towards the 7 to 9 range, making it debatable whether their numbers are actually useful. The site's eventual goal is to normalize the review scores from the major sites, and actually make them useful. From the article: "Games will still get an average score from all contributing reviews. But a site's contribution to that average will depend on that site's own individual normal curve-- with the immediate left and right of the bell's tip signifying three stars on a scale of one to five. Watch the drama as the biggest sweethearts see their 8.4 score for Gun and Car IV get pegged as three stars." This is the reason Slashdot videogame reviews don't have numbers anymore.
I gave up on game reviews when I worked at a game company and my boss explained his frustration with the review industry to me. He hated having to use dishonest means to catch the reviewers' attention - in particular, photoshopped preview game stills - but he said this was the only way to compete for those exceedingly rare "Game of the Month" etc. spots. He showed me a six page "exclusive preview" by a competitor and explained the only reason it was that big (as opposed to a small paragraph somewhere) was that the competitor had moved the preview session to Florida and paid the expenses of half a dozen editors for an extended weekend. How much sense does Florida make, when you're a European software developer and neither the game not the magazine are released in the English language? Now editors don't need to allow these tactics to work, but they evidently did. And worst of all, the magazine was the most respected (perceived fair) one in that market. Others were way more blatant about their corruption and deception of readers. The one that eventually made ours Game of the Month did so because we'd agreed to give considerable amounts of merchandise to readers who participated in a lottery.
When I want opinion on a game, I turn to a friend with a good taste in games, a fat pipe, and lots of time.
blow your mind already
A friend of mine is a reviewer for a gaming site that does their reviews on a [1,10] scale. Apparently the editors tell the writers to rate them on a [1,5] scale, then they just tack on a bonus 5 points to whatever the writers say.
In addition, game reviewers tend to review the types of games they enjoy playing. If a review company has a guy on staff that really likes fighting games, they'll likely give him the fighting games to review. This kind of thing likely tends to skew the results as well depending on the quality compared to other games within the same type.
Bear in mind that the games industry is not static. Not only are the technologies and concepts used in games and development constantly evolving and improving, but the budgets and resources being thrown at them far outstrip those seen 20, 10, even 5 years ago.
Inevitably this causes the leading edge of the games market to progress faster than our sense of cynicism and ennui, so we are more and more impressed with each new release. That's what ultimately gives a game a great score - how impressive is it? Scores for big releases will tend more and more towards the higher end (especially if we try to rate games comparitively with other fairly recent releases).
The term review inflation is surprisingly accurate in that regard but, while steady economic inflation is not a problem, it is troublesome in reviews where there is a fixed range of possible scores. The dollar can become worth 1% of its original value and there's no issue - as long as it's in line with market growth and currency values in other economies - becuase its value is defined by the market-perception of its value and has no technical limits. The star-rating system in reviews is inherently fixed, so inflation is a bugger.
Allowing the rating-system to inflate freely would get around this problem, but then you're looking at free-flux exchange-rates between reviewers and the issue of fitting 210 stars into a box-out and, frankly, the only sensible answer is for peeps to grow up and take everything they read with a pinch of salt. Hey, it's a valuable life-lesson. Learn to read, dammit.
Meta will eat itself
Isn't that a good thing? someone who doesn't give a shit about fighting games probably isn't going to notice the little subtleties of a new title. You want someone who knows all the various fighting games inside and out so he has a basis for comparison. If I'm out to buy a fighting game I want a reviewer to say "this game is good, but tekken is better". Then I know which fighting game is better. If the reviewer says "this game sucks, Civilization 4 is much better", how does that help me if I'm trying to decide which fighting game to buy?