Slashdot Mirror
Ranking of Harshest, Kindest Game Reviewers
GameDailyBiz has an interesting feature up right now discussing which sites and magazines are harshest and kindest with their game reviews. Surprisingly the study shows that, overall, the multi-platform online sites aren't terribly biased, with some being somewhat nicer than the average and some somewhat harsher. Single console gaming magazines end up having the most forgiving coverage, with mainstream news sources being the harshest critics.
I think in many ways it's good to limit it in this manner. There is a significant amount of error in all these ratings, and they are trying to give a quantitative score from qualititive opinions. If one site gives Halo 2 a 9.5, and Half-Life 2 a 9.6, does that mean Half-Life 2 is than Halo 2? I think the scale gamespy has is very reasonable, and it gave both games 5 stars. There is no way numbers could tell you which of these games you'll like more, but they do reflect that they are both excellent games. If you want to know which one is better and for what reasons, you are just going to have to go read the review.
Single console gaming magazines end up having the most forgiving coverage
According to TFA, multiplatform magazines were the most forgiving. Single platform magazines were the second harshest, right behind mainstream media.
That said, I would like to see some more information about how Game Daily came up with these absolutely magical numbers. In particular, I'd like to see how all four of the categories of publications could score below average. Is there some secret "fansite" category that isn't shown in any of the graphs but is clandestinely throwing all of the numbers all out of whack?
Good reviewers describe their biases in reviews.
That's one of the main things I look for when reading a review. Descriptions of the game, storyline, etc, are relatively useless information to me in a review. I'm more interested in understanding how they relate to the game in terms of other games they've liked, or what they were personally thinking during their first impressions.
I find that the vast majority of reviews are pretty kind. The assumption seems to be that 50% is only reserved for bad games rather than games of only average quality. So on a 10 point rating scale we should expect most games to bell-curve out towards the middle. Most publications, however, tend to rank poor games that most people would not consider purchasing around 50-60% which in my mind constitutes being above average. These types of ratings end up going down to single percentage points to compare the many games clustered in the upper reaches.
I believe this may have come about due to the American educational system's common method of ranking student work wherein 70% is average and anything below 50% is ignored as being poor.
I used to go straight to gamespot, cause they have a history of being tough. Now, I can legitimately compare all scores at the same time with gamerankings.com.
What you're doing is what I use GR for too - seeing a list of all scores in one convenient place - but it gives me a chance to bring up a practice that a lot of people use GR for that I think is really bad, and that's simply looking at the average score for a game and taking that as a definitive rating. GR themselves encourage this by heavily featuring that number everywhere on their site (as the "Average Ratio" - the main qualitative ranking that they use).
A big problem with doing this is that GR's weighting for its scoring is completely arbitrary, which skews the averages. They used to have the opposite problem, which is that they'd count all reviews equally in the average, regardless of the size, reach, influence, or respectability of a site. Another issue is that many games appeal to specific sensibilities and genres and so may generate widely divergent views based on who's reviewing them, so simply taking an average of scores is just not informative.
They seem to pick and choose now which sites get included in the average (there's a disclaimer saying only "sites in bold" are used to calculate the average), but I can't see any real pattern to the sites they select, and in any case, they're still including about 90% of all the sites out there, and every site they do include gets an "equal vote". Which means a huge site like Gamespot, with a history of reliable reviews and editors with many years of experience in the business, gets only as much weighting as a site like My Gamer or eToyChest (which I've honestly never even heard of).
I just don't think this is really giving people any more accurate of an idea of how good a game is than just picking one single site you like and sticking with it. Here is a perfect example of what I'm talking about. Rez received scores ranging from the 50% to 100% - that's a huge spread, which tells you that people either loved this game or they just didn't get it. Yet the final average is an 82%. What does that average tell you? It tells you nothing, because depending on what you're looking for in a game, your tastes vs. the reviewers' tastes, and your personality vs. the game designers' personalities, you might love or you might hate this game. It's unlikely (from reactions I've seen, and from my own experience) that you're going to think this is just a "good" but unremarkable game, as the average score would suggest. Just because an average "smooths out" the highs and lows you might see in various reviews doesn't mean that average ends up being an accurate measure of how good the game really is. And there are a lot of questionable sites included in that average to begin with, so it's doubly faulty.
The same is true to one extent or another with pretty much every game, except maybe those that are almost universally praised and those that are almost universally panned. In those cases, maybe you can trust the averages a little more. But most games don't fit that description, where every review site pretty much falls in line with the general consensus. Most games have a fairly wide range of opinions for and against them and the average score isn't going to tell you anything.
Anyway, so I use GR to give me quick links to all the available reviews for games I'm interested in, but I basically ignore the average ratio. I can tell at a quick glance of the review list if most sites are in agreement or not; if they are, then I don't even need to see the average, and if they're not, then the average is pretty pointless anyway.