MetaFuture Talks Review Inflation

MetaFuture, a game journalism analysis site, has recently refocused on review scores from the big gaming sites. The author takes an interesting approach, taking a look at Gamespot's review spread and IGN's tendencies. Unsurprisingly, both sites tend towards the 7 to 9 range, making it debatable whether their numbers are actually useful. The site's eventual goal is to normalize the review scores from the major sites, and actually make them useful. From the article: "Games will still get an average score from all contributing reviews. But a site's contribution to that average will depend on that site's own individual normal curve-- with the immediate left and right of the bell's tip signifying three stars on a scale of one to five. Watch the drama as the biggest sweethearts see their 8.4 score for Gun and Car IV get pegged as three stars." This is the reason Slashdot videogame reviews don't have numbers anymore.

Comments

Selection effects?

[JackBuckley]

I agree that fanboy magazine ratings should be viewed with scepticism, but I also worry about "normalizing" the reviews. There is an underlying assumption here that the population of games is symmetric in the distribution of quality. This may not be true, if, for example, games which are in the low quality tail are not released (game companies are strategic actors, right?) at a higher rate than games in the high quality side of the distribution. In addition, gaming magazines do not necessarily choose which games to review at random either--they either review interesting games with a higher probability of being of high quality (if you want to be kind to the industry) or else review in response to payola/swag (in which case it is the companies strategy which matters again.)

So, the question is, conditional on nonrandom selection of games to release and nonrandom selection of games to review, what should we expect the distribution of quality to look like? My guess is that this distribution is nonormal and is skewed with more observations in the higher quality tail. This does not necessarily mean, however, that the reviews are "fair," but it suggests that the question is more complicated than a simple "grade inflation" argument.

Note that I am also making an assumption that quality is judged in some sort of absolute terms, and not relative to the other games that are released. There are probably some other assumptions lurking in there as well. Just my $.02

Reviews are only useful when...

[mikeisme77]

Just looking at the end score in the review for ANYTHING is useless. The usefulness in a review is in reading the comments of the reviewer and understanding the reviewers preferences in games by looking at their reviews of other games you're interested in. The trick is to find a reviewer with similar thoughts on genres and such as your own, that way their review is relevant to you. The other trick is finding well thought out, well explained reviews--ones that tell you EXACTLY what the shortcomings and pros of the title were, this way you can decide if the shortcomings are shortcomings to you or if you just think the reviewer is being anal.

Who cares for hose numbers?

[Denial93]

I gave up on game reviews when I worked at a game company and my boss explained his frustration with the review industry to me. He hated having to use dishonest means to catch the reviewers' attention - in particular, photoshopped preview game stills - but he said this was the only way to compete for those exceedingly rare "Game of the Month" etc. spots. He showed me a six page "exclusive preview" by a competitor and explained the only reason it was that big (as opposed to a small paragraph somewhere) was that the competitor had moved the preview session to Florida and paid the expenses of half a dozen editors for an extended weekend. How much sense does Florida make, when you're a European software developer and neither the game not the magazine are released in the English language? Now editors don't need to allow these tactics to work, but they evidently did. And worst of all, the magazine was the most respected (perceived fair) one in that market. Others were way more blatant about their corruption and deception of readers. The one that eventually made ours Game of the Month did so because we'd agreed to give considerable amounts of merchandise to readers who participated in a lottery.

When I want opinion on a game, I turn to a friend with a good taste in games, a fat pipe, and lots of time.