Game Reviews are Broken?

Kotaku is running an opinion piece looking at the process of reviewing videogames, and comes to the conclusion that the whole system is entirely broken. Author Mark Wilson takes potshots at the concept of assigning a numerical valuation to a game, and the emphasis on product reviews rather than content reviews. "If there is no such thing as a perfect game, when why the hell are you scoring out of 100? It's not just PC Gamer that thinks this way--most publications, even those who do give out 'perfect' scores, do so begrudgingly. It's as if the developer has somehow cheated and broken their system. The movie reviewers solved this problem a long time ago. That's why most adopted a simpler rating system in which a 4-star movie didn't imply 'perfection' but supreme excellence. In most cases, games are penalized through being divided by a sum that they can never possibly reach."

Comments

Not New

[quanticle]

Game publishers, consumers, and even the reviewers themselves have been going on about the shortcomings of the current system for quite a while now. Yet we never see any alternatives being proposed. I say to the article writer, "Yes, I agree that the current system sucks. But what is your alternative?"

Re:Not New

[exploder]

You didn't even need to RTFA to see the alternative; it's right there in the summary. A simpler four- or five-star rating system, like what's used for movies, restaurants, hotels, etc.

Re:Not New

[Aladrin]

That's the -same thing-. It's a numerical score. We already have that, just with a different number.

You see, there's no such thing as a perfect restaurant, either, but there -are- 5-star restaurants. That's 5/5 on the game scoring chart. Or 10/10. Or 100/100. It's all the same thing.

Just like a restaurant, 10/10 for a gaming score simply means 'it's as good as could possibly be expected' or 'it's better than expected, even when you expect excellence.' Nobody claims it means perfection anymore since that is humanly impossible to create.

Re:Not New

[Fozzyuw]

If there is no such thing as a perfect game, when why the hell are you scoring out of 100? [...]The movie reviewers solved this problem a long time ago. That's why most adopted a simpler rating system in which a 4-star movie

Am I the only one who finds that comment just odd? While I can agree with you that the system is broken because there's no such thing at a "zero" rated game, but I do not see the difference between 100 points and 4 stars, besides it being simply "divided by 25". Then, of course, that's not good enough, so they start assigning 0.5 stars.

Also, I never considered "100/100" to be perfection, but as "supreme excellence" as noted. After, WTF is the difference between "supreme excellence" and "perfection"? Someone is just trying to argue semantics.

Of course, I don't even like the "four star" or 100 point numerical system. When I ask/tell people about a movie I simply say "Is it worth seeing in the theater?", "Is it worth a theater matinée?", "Is it a rental?" or "not worth your time, period".

In this sense, I saw the "Number 23" in the theater and I recommended that it was worth seeing in the theater. While "28 Weeks Later" was easily worth waiting for a rental (despite being a fan of "28 Days Later" and zombie films in general).

In that regards, I would say games should be rated as "buy it!", "rent it/demo it!", "stay away". (rent for console / Demo for PC games). Guitar Hero games are "buy it" games while something like Zelda:Twilight Princesses might be a "rent it" kind of game (I bought it, I'm a fan of Zelda, but still feel money better spent on a rental. I would not have given it 100/100 as some reviews did). World of Warcraft? "Buy it".

Of course, such a system needs a context. I'm not going to tell a FPS fan to buy an RPG, it's in the context of RPG fans.

Cheers,
Fozzy

Maybe

[Mr_Silver]

Although he has a point, we humans love to compare and if you don't give us any metric by which to do that, then we don't feel like anything has been achieved.

"SuperGame is really good" is meaningless to me. What I want to know is, is it any better than GreatGame? If the reviewer gives a score for both then I can understand which he/she feels is better and by what margin. Since I've played GreatGame (and assuming I trust the reviewer), then I can set some sort of expectation of what SuperGame will be like.

Personally, I use Metacritic which aggregates a number of reviews. Again, it's not perfect, but when it gets a 75 or above score, I can be reasonably certain that I'm not getting a dud game. It might not be my type of game, but if it is, then it shouldn't be disappointing.

Re:Maybe

[MooseMuffin]

Sites like Metacritic are part of the problem. It's these review aggregate sites that force any writer who wants their review to be read to give out a numeric score, or a score that can be converted into a numeric score. Everyone would learn more about the game they're researching if game reviews had no score and you just read about what the reviewer liked and didn't like. But people are lazy and would much rather see that game A is 13 points better than game B. What is 13 points? Does that mean it has better graphics, or a better story? Its nothing, its meaningless, and it should go.

Re:Maybe

[xtracto]

Personally, I use Metacritic which aggregates a number of reviews. Again, it's not perfect, but when it gets a 75 or above score, I can be reasonably certain that I'm not getting a dud game. It might not be my type of game, but if it is, then it shouldn't be disappointing..

Yup, I find metacritic quite useful (I think gamerankings is another site among those lines), however I think that a service similar to what Criticker provides for films would be good. In criticker you put a score to serveral movies you have watched to create a profile and then, the system gives other films you have not scored a score which YOU would most likely give to them (according to some function).

So far, it has worked quite right for me, and the problem I have seen with game rankings is that there are lots of games which have really high scors (like Wii Zelda or paper mario) but after playing them I did not like them...

Now, talking about giving scores, I remember that more than 10 years ago, I used to buy one of those gaming magazines (IGN or GamePro, I do not remember) where the reviewers were usually very critical of the games and the "scores" were just a "good", "bad", "very good" qualifier, as they once properly said that if you gave a game like Super Mario Bros 3 a 100 score then, what scores would you have to give to say Super Mario World?

The best review

[SuiteSisterMary]

The best review has no score. Simply somebody playing the dang game, and talking about what they like, what they don't like, what they'd improve, what really bothered them, what really excited them.

Find a reviewer with a decent command of the language, and who likes the sorts of games you like, and you're good to go.

Re:The best review

[quanticle]

This is why I like the Zero Punctuation reviews so much. Yahtzee has a decent command of the language, goes through all of the good and bad parts of the games, and gives a quick conclusion stating his opinion of the thing.

Re:The best review

[moderatorrater]

I disagree. For good games, a good review is definitely more than the score, but when I check out a much-hyped game and see a score of 50/100, that's a big deal to me. That means I don't even have to worry about reading the review, and in some cases that's the best review there is. During the era of 3000 RTS games a month, it was nice to be able to sort the good from the bad with a glance. It's also nice to be able to see the high and low points of a series, to be able to see that most people find the mechanics of FFVIII and FFIX lacking so that I know, if i'm going to start the series, start it somewhere else.

As TFA states, the review industry is necessary because of the large amount of games coming out and the large proportion of crap that inhabits it. If something is crap from end to end and at least four different people agree, then there's no need for me to look further.

Exactly...

[rwven]

These very thoughts came to my mind when I was reading all the Halo3 Reviews... When a game has so-so single player and awesome multiplayer...how does that get the game loads of perfect scores? A perfect game wouldn't need to make up for areas of lacking ANYWHERE. That aside, even the multiplay, while fun, is far from perfect. Halo3 was, and is, a great package but it's nowhere near a "perfect" game. I'm not just picking on Halo3 here either. HL2 for instance was a phenomenal game....but to call it "perfect" (like so many reviewers did) is just naive and downright inaccurate.

Re:Exactly...

[king-manic]

"3/5 or 7/10" I could understand, or "3/5 or 5/10", since going from a ten-point scale to a five-point scale is going to cause you to lose resolution. But how do you get from 3/5 to 8/10 ? my excuse could be that most 10 point systems seem to be 5 point systems that add 5 to them. Even the most wretched game rarely get less then 5/10. While 5 point systems more frequently give 2/5 or even 1/5. However the truth was sloppy math.

Re:Exactly...

[Hotawa Hawk-eye]

The last perfect score I remember seeing was for Soul Caliber on the Dreamcast. Gotta say, it is perfect. The only possible quibble I can come up with is that they could have had maybe a dozen voice samples for the announcer to cycle through to avoid repetition. That's about all I can fault it on! Fucking 10/10, absolutely.

You forgot three words at the end of this sentence of your comment. "Fucking 10/10, absolutely, in my opinion." I played Soul Caliber for Dreamcast a few times at a friend's place. I don't like playing fighting games all that much -- I prefer role-playing games -- so I'd probably rate it as a 6 or 7 on a scale of 1 to 10. Now give me Final Fantasy Tactics, or Castlevania: Symphony of the Night -- in my opinion, those go to 11. Your mileage may vary.

Frankly, I'd like to see game reviews cover the whole real line -- from -infinity to +infinity, with the endpoints unattainable. Games start out with a neutral score of 0. Each reviewer can add or subtract points from a game's rating, but they have to give a reason. For instance, I might say "-10 points, since I'm not a fan of fighting games" for Soul Caliber. You might assign that same reason "+10 points, I am a fan of fighting games". Over time, the list of 'modifiers' ('this game is a fighting game', 'this is an RPG', 'too short', 'too long', etc.) for a game's score would become standardized, as would the scores each reviewer assigns to each modifier. People reading those reviews could decide what value they wanted to assign to each modifier and so would be able to translate the scores from the reviewer's scale to their own.

Re:Exactly...

[Sigma 7]

Over time, the list of 'modifiers' ('this game is a fighting game', 'this is an RPG', 'too short', 'too long', etc.) for a game's score would become standardized, as would the scores each reviewer assigns to each modifier. People reading those reviews could decide what value they wanted to assign to each modifier and so would be able to translate the scores from the reviewer's scale to their own. After being disappointed with games that were rated rather high, this was exactly what I contemplated - I identified specific flaws in games that were already released and see if other games reproduce these flaws. Here's a few examples based on various genres:

• First Person Shooters (Doom-style): Hard to find flaws, but the most common one is slow weapon switching based on interface (e.g. in SiN, if you cycle into the Sniper Rifle by the wheel, you are disabled for ~2 seconds.) Various ways to fix this, best is to abort the current weapon transition as soon as it's obvious the player wasn't choosing that weapon.
• Realtime Tactical Simulations: On completion of an order (e.g. some unit dies), units stop still in their tracks. (Correct behaviour is shown in Tiberian Sun - units keep move to the general location where the target died.)
• Computer Run Adventure Programs (also known as CRPGs):You need massive amounts of grinding just to advance past one obstacle. Correct behaviour requires a smoother flow by not emulating the first edition of Dragon Warrior - although Moria/Angband family of roguelikes seems to be an exception.
• Fighting games: "Unresponsive" controls. Usually this affects PC games where certain keypress patterns don't get recognized by the keyboard (e.g Ctrl+Left+Up registers as Left+Ctrl) but can also mean the game not recognizing actions when they should be valid (e.g. pressing jump one frame early prevents it from registering).
• Puzzle games:No ability to track progress. Giving out passwords is minimally acceptable (especially in the console era), but modern games are expected to show which ones are completed (and how well they're completed as well.

This list doesn't cover the generics, where you don't make Easy this difficult.

The benefit of assuming perfect and stripping away points for known flaw patterns is that you can properly assess how well games stand up to others. It can also allow ratings to be "depreciated" in the same way that other assets do as new flaws get discovered. The disadvantage is that you need to have a lot of experience reviewing and playing games to know and recognize flaw patterns.

Reviews have been fundamentally broken for years

[anss123]

Reviews have been fundamentally broken for years and years. Ten years ago Gamepro gave Bubsy 3D an impossibly high score of 3.5 out of 5 - a score comparable with Screamer 2, tempest, Cruis'n USA and other playable games. Playing Bubsy is about as enjoyable as stabbing your eyes out, it's a turd among turds. Incidentally there was a full page add for, you guessed it, Bubsy 3D in that very issue.

Problem is that these magazines are at the mercy at the games they review. They need to get exclusives, interviews, previews and adds to stay in the game. They are therefore very reluctant to give out bad scores to games from well known publishers.

Once upon a time there was a magazine (Amiga Power?) that did just this, said things as they were, and they found themselves cahoots by devs like Team 17, etc, for simply stating their actual opinions.

How about a survey:

[Bob-taro]

On a scale of 1 to 10, how bad do you think the 10 point rating system is.

Re:Does it matter?

[geminidomino]

There's a stigmata though when actually attaching a number to a review. Umm... ouch.

When game rags claim to put their blood and sweat into the magazine, I never knew they meant it so LITERALLY.

Re:Reviews are bought

[UbuntuDupe]

The solution would be to have a "Consumer Reports" of gaming where people unaffiliated with the publishers, buy the game at a retail outlet like anyone else at launch, play it, write the review, and then do this consistently for all games that are released.

The disadvantages:

-You wouldn't see the review until after launch. (Probably a week for some games.)
-It doesn't seem to have a viable revenue model, unless someone knows a counterexample?

You're not grading a math test.

[Swifti]

It seems as though the software entertainment press has been so affected by the grade school method of grading that it's pretty much in-grained in their minds that 90-100 (A) is Excellent, 80-90 (B) is Good, 70-80 (C) is Average, 60-70 (D), and 0-50 (F) is Awful which actually isn't the fundamental problem.

The problem is that reviewers don't take into account of the reason why grade school has this stratified curve; It's the curve you get when students are graded based on the percent of quantifiable problems they can either get right or wrong. I'm assuming the average amount of students can get about 75% of problems right on a math test which is why that's considered average. However, there are no quantifiable measurements you can make with artistic mediums like video games so it doesn't make sense at all to have a grading scale based on a scale of five, ten, or a hundred when most of the marks are going to be pivoted around the 75 average. You're not going to grade an essay based on the ratio of correct/in-correct questions so why are we grading video games this way?

The best way to get out of this 100-scale ditch is to rate on a scale of A-B-C-D-F where you're still communicating the idea of the 100 scale, but you get rid of any implications of quantifiable measurements. Plus, you get rid of the superfluous 0-60 range of scores.