Game Reviews are Broken?

Kotaku is running an opinion piece looking at the process of reviewing videogames, and comes to the conclusion that the whole system is entirely broken. Author Mark Wilson takes potshots at the concept of assigning a numerical valuation to a game, and the emphasis on product reviews rather than content reviews. "If there is no such thing as a perfect game, when why the hell are you scoring out of 100? It's not just PC Gamer that thinks this way--most publications, even those who do give out 'perfect' scores, do so begrudgingly. It's as if the developer has somehow cheated and broken their system. The movie reviewers solved this problem a long time ago. That's why most adopted a simpler rating system in which a 4-star movie didn't imply 'perfection' but supreme excellence. In most cases, games are penalized through being divided by a sum that they can never possibly reach."

Comments

Re:Not New

[Fozzyuw]

If there is no such thing as a perfect game, when why the hell are you scoring out of 100? [...]The movie reviewers solved this problem a long time ago. That's why most adopted a simpler rating system in which a 4-star movie

Am I the only one who finds that comment just odd? While I can agree with you that the system is broken because there's no such thing at a "zero" rated game, but I do not see the difference between 100 points and 4 stars, besides it being simply "divided by 25". Then, of course, that's not good enough, so they start assigning 0.5 stars.

Also, I never considered "100/100" to be perfection, but as "supreme excellence" as noted. After, WTF is the difference between "supreme excellence" and "perfection"? Someone is just trying to argue semantics.

Of course, I don't even like the "four star" or 100 point numerical system. When I ask/tell people about a movie I simply say "Is it worth seeing in the theater?", "Is it worth a theater matinée?", "Is it a rental?" or "not worth your time, period".

In this sense, I saw the "Number 23" in the theater and I recommended that it was worth seeing in the theater. While "28 Weeks Later" was easily worth waiting for a rental (despite being a fan of "28 Days Later" and zombie films in general).

In that regards, I would say games should be rated as "buy it!", "rent it/demo it!", "stay away". (rent for console / Demo for PC games). Guitar Hero games are "buy it" games while something like Zelda:Twilight Princesses might be a "rent it" kind of game (I bought it, I'm a fan of Zelda, but still feel money better spent on a rental. I would not have given it 100/100 as some reviews did). World of Warcraft? "Buy it".

Of course, such a system needs a context. I'm not going to tell a FPS fan to buy an RPG, it's in the context of RPG fans.

Cheers,
Fozzy

Re:Reviews are bought

[UbuntuDupe]

The solution would be to have a "Consumer Reports" of gaming where people unaffiliated with the publishers, buy the game at a retail outlet like anyone else at launch, play it, write the review, and then do this consistently for all games that are released.

The disadvantages:

-You wouldn't see the review until after launch. (Probably a week for some games.)
-It doesn't seem to have a viable revenue model, unless someone knows a counterexample?

Re:Exactly...

[Hotawa Hawk-eye]

The last perfect score I remember seeing was for Soul Caliber on the Dreamcast. Gotta say, it is perfect. The only possible quibble I can come up with is that they could have had maybe a dozen voice samples for the announcer to cycle through to avoid repetition. That's about all I can fault it on! Fucking 10/10, absolutely.

You forgot three words at the end of this sentence of your comment. "Fucking 10/10, absolutely, in my opinion." I played Soul Caliber for Dreamcast a few times at a friend's place. I don't like playing fighting games all that much -- I prefer role-playing games -- so I'd probably rate it as a 6 or 7 on a scale of 1 to 10. Now give me Final Fantasy Tactics, or Castlevania: Symphony of the Night -- in my opinion, those go to 11. Your mileage may vary.

Frankly, I'd like to see game reviews cover the whole real line -- from -infinity to +infinity, with the endpoints unattainable. Games start out with a neutral score of 0. Each reviewer can add or subtract points from a game's rating, but they have to give a reason. For instance, I might say "-10 points, since I'm not a fan of fighting games" for Soul Caliber. You might assign that same reason "+10 points, I am a fan of fighting games". Over time, the list of 'modifiers' ('this game is a fighting game', 'this is an RPG', 'too short', 'too long', etc.) for a game's score would become standardized, as would the scores each reviewer assigns to each modifier. People reading those reviews could decide what value they wanted to assign to each modifier and so would be able to translate the scores from the reviewer's scale to their own.

Re:Maybe

[xtracto]

Personally, I use Metacritic which aggregates a number of reviews. Again, it's not perfect, but when it gets a 75 or above score, I can be reasonably certain that I'm not getting a dud game. It might not be my type of game, but if it is, then it shouldn't be disappointing..

Yup, I find metacritic quite useful (I think gamerankings is another site among those lines), however I think that a service similar to what Criticker provides for films would be good. In criticker you put a score to serveral movies you have watched to create a profile and then, the system gives other films you have not scored a score which YOU would most likely give to them (according to some function).

So far, it has worked quite right for me, and the problem I have seen with game rankings is that there are lots of games which have really high scors (like Wii Zelda or paper mario) but after playing them I did not like them...

Now, talking about giving scores, I remember that more than 10 years ago, I used to buy one of those gaming magazines (IGN or GamePro, I do not remember) where the reviewers were usually very critical of the games and the "scores" were just a "good", "bad", "very good" qualifier, as they once properly said that if you gave a game like Super Mario Bros 3 a 100 score then, what scores would you have to give to say Super Mario World?

You're not grading a math test.

[Swifti]

It seems as though the software entertainment press has been so affected by the grade school method of grading that it's pretty much in-grained in their minds that 90-100 (A) is Excellent, 80-90 (B) is Good, 70-80 (C) is Average, 60-70 (D), and 0-50 (F) is Awful which actually isn't the fundamental problem.

The problem is that reviewers don't take into account of the reason why grade school has this stratified curve; It's the curve you get when students are graded based on the percent of quantifiable problems they can either get right or wrong. I'm assuming the average amount of students can get about 75% of problems right on a math test which is why that's considered average. However, there are no quantifiable measurements you can make with artistic mediums like video games so it doesn't make sense at all to have a grading scale based on a scale of five, ten, or a hundred when most of the marks are going to be pivoted around the 75 average. You're not going to grade an essay based on the ratio of correct/in-correct questions so why are we grading video games this way?

The best way to get out of this 100-scale ditch is to rate on a scale of A-B-C-D-F where you're still communicating the idea of the 100 scale, but you get rid of any implications of quantifiable measurements. Plus, you get rid of the superfluous 0-60 range of scores.

Re:Exactly...

[Sigma 7]

Over time, the list of 'modifiers' ('this game is a fighting game', 'this is an RPG', 'too short', 'too long', etc.) for a game's score would become standardized, as would the scores each reviewer assigns to each modifier. People reading those reviews could decide what value they wanted to assign to each modifier and so would be able to translate the scores from the reviewer's scale to their own. After being disappointed with games that were rated rather high, this was exactly what I contemplated - I identified specific flaws in games that were already released and see if other games reproduce these flaws. Here's a few examples based on various genres:

• First Person Shooters (Doom-style): Hard to find flaws, but the most common one is slow weapon switching based on interface (e.g. in SiN, if you cycle into the Sniper Rifle by the wheel, you are disabled for ~2 seconds.) Various ways to fix this, best is to abort the current weapon transition as soon as it's obvious the player wasn't choosing that weapon.
• Realtime Tactical Simulations: On completion of an order (e.g. some unit dies), units stop still in their tracks. (Correct behaviour is shown in Tiberian Sun - units keep move to the general location where the target died.)
• Computer Run Adventure Programs (also known as CRPGs):You need massive amounts of grinding just to advance past one obstacle. Correct behaviour requires a smoother flow by not emulating the first edition of Dragon Warrior - although Moria/Angband family of roguelikes seems to be an exception.
• Fighting games: "Unresponsive" controls. Usually this affects PC games where certain keypress patterns don't get recognized by the keyboard (e.g Ctrl+Left+Up registers as Left+Ctrl) but can also mean the game not recognizing actions when they should be valid (e.g. pressing jump one frame early prevents it from registering).
• Puzzle games:No ability to track progress. Giving out passwords is minimally acceptable (especially in the console era), but modern games are expected to show which ones are completed (and how well they're completed as well.

This list doesn't cover the generics, where you don't make Easy this difficult.

The benefit of assuming perfect and stripping away points for known flaw patterns is that you can properly assess how well games stand up to others. It can also allow ratings to be "depreciated" in the same way that other assets do as new flaws get discovered. The disadvantage is that you need to have a lot of experience reviewing and playing games to know and recognize flaw patterns.