Slashdot Mirror

← Back to Stories (view on slashdot.org)

Ranking of Harshest, Kindest Game Reviewers

Posted by Zonk on from the which-side-of-their-bread-is-the-butter-on? dept.

GameDailyBiz has an interesting feature up right now discussing which sites and magazines are harshest and kindest with their game reviews. Surprisingly the study shows that, overall, the multi-platform online sites aren't terribly biased, with some being somewhat nicer than the average and some somewhat harsher. Single console gaming magazines end up having the most forgiving coverage, with mainstream news sources being the harshest critics.

11 of 23 comments (clear)

  1. Surprize! by MSG · · Score: 4, Funny

    ...some being somewhat nicer than the average and some somewhat harsher

    No way! In a group of sources, some are above average and some are below? Who could have guessed?

    1. Re:Surprize! by Yorrike · · Score: 4, Funny

      That's a bit mean.

      --

      Looks can be deceiving. Or CAN they?

  2. Scales? by Gothic_Walrus · · Score: 2, Interesting
    Maybe it's just me, but I think that the "harshness" is indelibly linked to the scoring system.

    Look at GameSpy. Their overall scores are out of five stars, with the possibility of half stars in the score. The overall score that a game can get is severely restricted, and as long as it's decent, it's almost guaranteed to get seven stars or higher - a score that most of us would consider to be "good."

    On the other hand, look at the mainstream media. Papers like the USA Today and the Detroit Free Press grade games on a scale of one to four. This is even more restrictive...but not in a good way. A game can only get one of two positive scores - a three or a four - and it's usually got to be bloody amazing to reach the four star level. Most of the games that I see get revieved in papers tend to get scores of two or three stars. That's not terrible, but I know that most of the people I know tend to think of a 3/4 as being much worse than a 7/10. Maybe it just seems harsher. I'm not sure why that is, but observations that I've made over the years seem to back that up.

    The scoring systems vary from site to site and from one kind of media to another, and that could be enough to make up the difference.

    --
    Goo goo g'joob.
    1. Re:Scales? by SamNmaX · · Score: 4, Insightful
      Look at GameSpy. Their overall scores are out of five stars, with the possibility of half stars in the score. The overall score that a game can get is severely restricted, and as long as it's decent, it's almost guaranteed to get seven stars or higher - a score that most of us would consider to be "good."

      I think in many ways it's good to limit it in this manner. There is a significant amount of error in all these ratings, and they are trying to give a quantitative score from qualititive opinions. If one site gives Halo 2 a 9.5, and Half-Life 2 a 9.6, does that mean Half-Life 2 is than Halo 2? I think the scale gamespy has is very reasonable, and it gave both games 5 stars. There is no way numbers could tell you which of these games you'll like more, but they do reflect that they are both excellent games. If you want to know which one is better and for what reasons, you are just going to have to go read the review.

    2. Re:Scales? by superpulpsicle · · Score: 2, Informative

      Go to www.gamerankings.com. It's the premiere shortcut to every freaking review site there is.

      I used to go straight to gamespot, cause they have a history of being tough. Now, I can legitimately compare all scores at the same time with gamerankings.com. They throw out magazine scores sometimes too, just not the review.

    3. Re:Scales? by badasscat · · Score: 2, Insightful

      I used to go straight to gamespot, cause they have a history of being tough. Now, I can legitimately compare all scores at the same time with gamerankings.com.

      What you're doing is what I use GR for too - seeing a list of all scores in one convenient place - but it gives me a chance to bring up a practice that a lot of people use GR for that I think is really bad, and that's simply looking at the average score for a game and taking that as a definitive rating. GR themselves encourage this by heavily featuring that number everywhere on their site (as the "Average Ratio" - the main qualitative ranking that they use).

      A big problem with doing this is that GR's weighting for its scoring is completely arbitrary, which skews the averages. They used to have the opposite problem, which is that they'd count all reviews equally in the average, regardless of the size, reach, influence, or respectability of a site. Another issue is that many games appeal to specific sensibilities and genres and so may generate widely divergent views based on who's reviewing them, so simply taking an average of scores is just not informative.

      They seem to pick and choose now which sites get included in the average (there's a disclaimer saying only "sites in bold" are used to calculate the average), but I can't see any real pattern to the sites they select, and in any case, they're still including about 90% of all the sites out there, and every site they do include gets an "equal vote". Which means a huge site like Gamespot, with a history of reliable reviews and editors with many years of experience in the business, gets only as much weighting as a site like My Gamer or eToyChest (which I've honestly never even heard of).

      I just don't think this is really giving people any more accurate of an idea of how good a game is than just picking one single site you like and sticking with it. Here is a perfect example of what I'm talking about. Rez received scores ranging from the 50% to 100% - that's a huge spread, which tells you that people either loved this game or they just didn't get it. Yet the final average is an 82%. What does that average tell you? It tells you nothing, because depending on what you're looking for in a game, your tastes vs. the reviewers' tastes, and your personality vs. the game designers' personalities, you might love or you might hate this game. It's unlikely (from reactions I've seen, and from my own experience) that you're going to think this is just a "good" but unremarkable game, as the average score would suggest. Just because an average "smooths out" the highs and lows you might see in various reviews doesn't mean that average ends up being an accurate measure of how good the game really is. And there are a lot of questionable sites included in that average to begin with, so it's doubly faulty.

      The same is true to one extent or another with pretty much every game, except maybe those that are almost universally praised and those that are almost universally panned. In those cases, maybe you can trust the averages a little more. But most games don't fit that description, where every review site pretty much falls in line with the general consensus. Most games have a fairly wide range of opinions for and against them and the average score isn't going to tell you anything.

      Anyway, so I use GR to give me quick links to all the available reviews for games I'm interested in, but I basically ignore the average ratio. I can tell at a quick glance of the review list if most sites are in agreement or not; if they are, then I don't even need to see the average, and if they're not, then the average is pretty pointless anyway.

  3. Damn lies about statistics by Bastian · · Score: 2, Insightful

    Single console gaming magazines end up having the most forgiving coverage

    According to TFA, multiplatform magazines were the most forgiving. Single platform magazines were the second harshest, right behind mainstream media.

    That said, I would like to see some more information about how Game Daily came up with these absolutely magical numbers. In particular, I'd like to see how all four of the categories of publications could score below average. Is there some secret "fansite" category that isn't shown in any of the graphs but is clandestinely throwing all of the numbers all out of whack?

    1. Re:Damn lies about statistics by Relic+of+the+Future · · Score: 2, Interesting
      "In particular, I'd like to see how all four of the categories of publications could score below average."

      If you RTFA closely, you'll see that they're comparing it against GameRankings.com. So, yes, it IS the secret, hiden, fansite category that's throwing off the numbers.

      --
      Those who fail to understand communication protocols, are doomed to repeat them over port 80.
  4. Bah by Dr.+Photo · · Score: 3, Funny

    The meta-review is pedestrian, at best. 2 out of 5 stars.

  5. Re:Read the article earlier... by vhold · · Score: 2, Insightful

    Good reviewers describe their biases in reviews.

    That's one of the main things I look for when reading a review. Descriptions of the game, storyline, etc, are relatively useless information to me in a review. I'm more interested in understanding how they relate to the game in terms of other games they've liked, or what they were personally thinking during their first impressions.

  6. Most reviews are too kind by Belgand · · Score: 4, Insightful

    I find that the vast majority of reviews are pretty kind. The assumption seems to be that 50% is only reserved for bad games rather than games of only average quality. So on a 10 point rating scale we should expect most games to bell-curve out towards the middle. Most publications, however, tend to rank poor games that most people would not consider purchasing around 50-60% which in my mind constitutes being above average. These types of ratings end up going down to single percentage points to compare the many games clustered in the upper reaches.

    I believe this may have come about due to the American educational system's common method of ranking student work wherein 70% is average and anything below 50% is ignored as being poor.