Are Review Scores Pointless?

donniebaseball23 writes: With Eurogamer being the latest popular video games site to ditch review scores, some are discussing just how valuable assigning a score to a game actually is these days. It really depends on whom you ask. "I've always disliked the notion of scores on something as abstract and subjective as games," says Vlambeer co-founder Rami Ismail. From the press side, though, former GameSpot editor Justin Calvert still believes in scores. "I've been basing my own game-purchasing decision on reviews ever since I picked up the first issue of Zzap! 64 magazine in the UK almost 30 years ago," he says, while admitting that YouTube is certainly changing the landscape today: "There's something very appealing about watching a game being played and knowing that the footage hasn't been edited in a way that might misrepresent the experience."

Comments

Meta scores and user's meta scores

[Deffexor]

I've found Metacritic to be a good aggregator of scores, but more importantly, the "users" scores (and reviews) tend to be more reliable in terms of not being overly critical of games that are generally pretty good, but don't meet the expectations of "hard core" gamers.

Re:Meta scores and user's meta scores

[jclarker6]

Totally agree with this. And taking it a step further one could say any single score on it's own is not that reliable, when taken in aggregate the cream definitely rises to the top

Re:Meta scores and user's meta scores

[houghi]

Scores by themselves are useless in many cases. I once was heavily involved in a customer service survey. It was basically "From 1 to 10, how do you like the service." What we noticed what that Nordic countries gave a complete different number compared to Mediteranian countries.

First they thought it was because the service was much better in some countries compared to others. Looking into it and asking customers we found nothing.

We then started asking a second question: "What service did you expect." and then measure the difference. So if you expect a 6 and you get a 6 it is much better then expecting a 9 and getting a 7. So a 6 can be better then a 7.

The issue was that the first time we did not have a base to start from. In school-tests the base is pretty easy. 100% is perfect without any errors.

Compare it to Americans and Enlish where one would say "Wow, this is AMAZING. It is the best I have ever seen." and the other would say "It's not bad." (I jope you get what I mean.)

When I look at at scres for movies, restaurants, books or whatever, I read the comments to know WHAT they thought about it.

Re: Meta scores and user's meta scores

[Anubis IV]

I'd go even further than that and say that it depends on the type of scale being used as well.

When it comes to user reviews, if the reviews are thumbs up or down, I'll do the same as you and read the thumbs down reviews first, since it's easier to filter out the extreme reviewers and get a sense for the common issues. If it's a 5-point scale, I'll read through the 2s and 4s, since those reviews can give you a quick understanding of the pros and cons for the product, without nearly the level of overstatement that you'll need to filter through in the 1s and 5s. And I don't even bother reading reviews based on 10-point scales, since the way that everyday users grade on a 10-point scale is arbitrary to the point of uselessness (e.g. some people treat it like a 5-point scale with better granularity, while others treat it like an academic scale).

Re:Meta scores and user's meta scores

[Dutch Gun]

the "users" scores (and reviews) tend to be more reliable in terms of not being overly critical of games that are generally pretty good

In my experience, users are very extreme in their assigned scores. If they enjoy the game, they assign it a 100 (ZOMG, Best Game EVAR!!!). If they didn't enjoy the game for some reason, it rates a 0 (WTF?! Worst !@#$!@ game of ALL TIME!!!). There are often relatively few scores in the middle. Also, user ratings will often pick up on issues that the press doesn't touch, though, which is a good thing. For instance, when a company introduces intrusive DRM, or if an online-only game has a very bad launch, users will flood the systems with very low scores, where professional scores would not have touched (or perhaps seen) these issues.

Generally speaking, if a game gets universal praise, there's probably something worthwhile about it, at least to many people. If it generally gets horrid scores, you know that there's something seriously wrong. No, review scores aren't pointless at all. If you want to get the details, then read the actual review, and you can find out if you agree with those specific points or not.

Eurogamer isn't really dropping the score, incidentally. They're just moving to a "four star" system ("Avoid", "No Recommendation", "Recommended", and "Essential"). In truth, I think that's probably a more honest way of scoring, because it's sort of silly to try to rank a different games based on a one or two percentage points of difference, which is probably completely arbitrary. For instance, what's the difference between a game that ranks 90% on metacritic and one that ranks 89% ? Answer: one more high profile review gave it a five out of five stars instead of four out of five stars. This also avoids the problem of having to try to rank very different genres against each other, or try to convey what a particular score "means" (there's almost always a chart along with the score). In a sense, giving it one of four rankings is sort of cutting out the score as a middleman.

Also, honestly, I sort of wonder if dropping numeric ratings is a way for gaming sites to give themselves an "out" with publishers, who may apply pressure if their review scores are too low. I've heard of bonus and such being tied to metacritic review scores, which is a pretty nasty thing to do to your employees, IMO. Also, I'm guessing websites don't care to have their review simply aggregated by metacritic into a single, unified score.

Game reviews have always been broken

[Piata]

Almost no games get below 40, while any game that doesn't get 80 or more is considerd a failure. Then you have people giving games 3 out of 5 stars which translates to a score of 60, which skews things even more. Plus tent pole games like CoD can be executed extremely well but offer nothing new so how do you review that? There are games with low interaction (point and click) or high interaction (RTS). How do you compare one against the other? Good reviews are also often given despite massive bugs, incomplete games being released or week 1 launch disasters (like Diablo III).

It's issues like that which make me understand the no score review trend.

Steam User Score beats traditional scores

[Holammer]

The Steam User Score is currently my most trusted metric for how good a game is, something which is considered "overwhelmingly positive" with a couple thousand user reviews is usually a worthy purchase.
For non-steam users, imagine Metacritic except you can only submit your score/review if you own the actual game and it's either thumbs up or down.