Slashdot Mirror
MetaFuture Talks Review Inflation
MetaFuture, a game journalism analysis site, has recently refocused on review scores from the big gaming sites. The author takes an interesting approach, taking a look at Gamespot's review spread and IGN's tendencies. Unsurprisingly, both sites tend towards the 7 to 9 range, making it debatable whether their numbers are actually useful. The site's eventual goal is to normalize the review scores from the major sites, and actually make them useful. From the article: "Games will still get an average score from all contributing reviews. But a site's contribution to that average will depend on that site's own individual normal curve-- with the immediate left and right of the bell's tip signifying three stars on a scale of one to five. Watch the drama as the biggest sweethearts see their 8.4 score for Gun and Car IV get pegged as three stars." This is the reason Slashdot videogame reviews don't have numbers anymore.
I gave up on game reviews when I worked at a game company and my boss explained his frustration with the review industry to me. He hated having to use dishonest means to catch the reviewers' attention - in particular, photoshopped preview game stills - but he said this was the only way to compete for those exceedingly rare "Game of the Month" etc. spots. He showed me a six page "exclusive preview" by a competitor and explained the only reason it was that big (as opposed to a small paragraph somewhere) was that the competitor had moved the preview session to Florida and paid the expenses of half a dozen editors for an extended weekend. How much sense does Florida make, when you're a European software developer and neither the game not the magazine are released in the English language? Now editors don't need to allow these tactics to work, but they evidently did. And worst of all, the magazine was the most respected (perceived fair) one in that market. Others were way more blatant about their corruption and deception of readers. The one that eventually made ours Game of the Month did so because we'd agreed to give considerable amounts of merchandise to readers who participated in a lottery.
When I want opinion on a game, I turn to a friend with a good taste in games, a fat pipe, and lots of time.
blow your mind already
A friend of mine is a reviewer for a gaming site that does their reviews on a [1,10] scale. Apparently the editors tell the writers to rate them on a [1,5] scale, then they just tack on a bonus 5 points to whatever the writers say.
In addition, game reviewers tend to review the types of games they enjoy playing. If a review company has a guy on staff that really likes fighting games, they'll likely give him the fighting games to review. This kind of thing likely tends to skew the results as well depending on the quality compared to other games within the same type.
Bear in mind that the games industry is not static. Not only are the technologies and concepts used in games and development constantly evolving and improving, but the budgets and resources being thrown at them far outstrip those seen 20, 10, even 5 years ago.
Inevitably this causes the leading edge of the games market to progress faster than our sense of cynicism and ennui, so we are more and more impressed with each new release. That's what ultimately gives a game a great score - how impressive is it? Scores for big releases will tend more and more towards the higher end (especially if we try to rate games comparitively with other fairly recent releases).
The term review inflation is surprisingly accurate in that regard but, while steady economic inflation is not a problem, it is troublesome in reviews where there is a fixed range of possible scores. The dollar can become worth 1% of its original value and there's no issue - as long as it's in line with market growth and currency values in other economies - becuase its value is defined by the market-perception of its value and has no technical limits. The star-rating system in reviews is inherently fixed, so inflation is a bugger.
Allowing the rating-system to inflate freely would get around this problem, but then you're looking at free-flux exchange-rates between reviewers and the issue of fitting 210 stars into a box-out and, frankly, the only sensible answer is for peeps to grow up and take everything they read with a pinch of salt. Hey, it's a valuable life-lesson. Learn to read, dammit.
Meta will eat itself
Isn't that a good thing? someone who doesn't give a shit about fighting games probably isn't going to notice the little subtleties of a new title. You want someone who knows all the various fighting games inside and out so he has a basis for comparison. If I'm out to buy a fighting game I want a reviewer to say "this game is good, but tekken is better". Then I know which fighting game is better. If the reviewer says "this game sucks, Civilization 4 is much better", how does that help me if I'm trying to decide which fighting game to buy?
Guess I'd be in the same boat as you then. The last reviews I actually found really useful were done about 10 years ago in 2 different magazines. EGM had an OK system where four different people (most of whom were recurring) would given their thoughts on a game. GamePlayers, I think (it's been awhile), used to do a scale from 1-100% in addition to some very long and in-depth articles. Last time I picked up EGM, the "reviews" were much shorter (about 1 paragraph) and far less diverse; GamePlayers has been out of print for several years.
.5 ratings at the same time it replaced content with more ads and cut pages; if I haven't lapsed already (who knows?), I will next time it's up for renewal.
Nintendo Power, to which I still subscribe for some reason, has traditionally been more of a strategy magazine than a source of reviews. In the last several years, alot of that's gone out the window for a variety of reasons (no more 2D maps, less cheat codes, Internet walkthroughs, etc). The problem is that the reviews never expended to fill the void. GamePro dropped its
I haven't even started on the topic of multiplatform reviews either. 10 years ago, Earthworm Jim got 4 reviews (SNES, Genesis, GameBoy, GameGear). Today, it's always either PS2 or XBox/360 without a mention of the other consoles. It's bad enough being locked out of the sales market for having a GameCube but it's another thing being locked out of information on what actually does get released. And forget owning a handheld, it's a total crapshoot! Hey, I just remembered why I give NP $20 a year...because they actually cover GameCube and GameBoy.
So to sum it up, review inflation is terrible but I really couldn't care less because it doesn't affect my segment of the market at all and hasn't for a long time.