Slashdot Mirror
Videogame Reviews - Playing With Numbers?
Thanks to NTSC-uk for its editorial discussing the possible confusion in using numbered rating schemes for videogame reviews. The author rhapsodizes: "No number can possibly capture the striking vision of the sun setting over Hyrule Field or the ingenious brilliance of Metal Gear Solid's interactive references to reality", before going on to conclude: "Treated as numbers with a defined value, they will always be looked down upon as having deficiencies. Yet when you read them as you would a word and open it up to your own interpretation, they begin to fully deliver the explanatory potential that is locked within." Do you think numbered ratings have an important place at the end of game reviews?
I think this person has missed the point of number ratings; they don't puppourt to sum up the entire game in a digit.
Hint: That's what those weird-looking "paragraphs of descriptive text" are for.
What makes a man want to be a mouse? (Python's Flying Circus)
We should do away with the star system for hotels (how can one number possibly describe a whole hotel?). We should do away with film genres (how can one word describe a whole film?). We should do away with races (how can one word describe a whole person?). We should do away with ages (how can one number describe the experiences of a whole lifetime?).
What a stupid article. It's not like the whole review is being replaced with a number. It's something to quickly glance at to see if it's a load of shit or a really good game. If there's a review of a game I've never heard of before, and it's got 2/10, then I'm not going to waste my time reading it.
I think that in the reviews, the writers comments and experiences are the most important aspect. Rating with numbers or percents is dangerous, because it seems to be a rule, that all games are rated between 80% to 100% and if any game receives any lower rating, it is automatically labelled as a bad game even if the game is billiant and the lower rating is given only by techincal reasons (bugs etc.). Also, it's tempting to compare numbers between different reviews even if there isn't any common rule set between different gaming magazines for giving these ratings (so the comparing is actually pointless). Numerical ratings are too subjective to be taken as a meter for the quality of the game (idea, storyline) itself.
I give the article a 62.
Everytime you look at porn a devil gets their horns.
We have ten or eleven values in a 10 point system, yet 7 often means average quality and anything below that is bad. Great games always get 9 or 10. That leaves the numbers 0-6 for bad, 7-8 for more average and 9-10 for great games. That means we can differentiate between 7 flavours of bad games but only 2-3 (depending on your interpretation of 8) different levels of great. However, as the article mentions, people aren't interested in bad or lower-average games, so you'd need one, maybe two different descriptions (bad and REALLY bad) for those games. I mean, who cares whether Teletubbies Adventure is better or worse than Jar Jar's Fun Games? People care for comparisons like Zelda vs. Metroid (to name a close example), which might get problematic if you consider both games a ten (yet most people would agree that one is better than the other, but not on a scale that would justify a full point in a ten point system). Sure, you can use a percentage score or other concepts to split the range up further, but then you still have 60%-70% nearly useless possible ratings. Also, such fine differentiation (especially single percents) often isn't possible and numbers are assigned within a certain range with some arbitrartity (93, 94 or 95? Roll a dice).
Note that average game here doesn't refer to statistical average, but perceived average.
The purpose of the summary ratings is not to let you pick "the best" item, it is to draw your attention to the outliers. For example, if a game (or car or whatever) from a company you have never heard of receives a really high overall rating, you might take the time to look at it. And if a game (or car or whatever) that you just assumed was good and were going to buy based on its brand receives a really low rating, you might check the review and maybe investigate alternatives.
I think that number scores are important for what they are: A distillation of opinion. That's all it is, and all it should be treated as. If you want the justification of the opinion, then that is what the review is for. The way I think about is like this: A random art critic can say 'Van Gogh is the greatest artist, ever.', and that is like a numeric opinion. If I want to know why he thinks that, then I'll read further. If I hate Van Gogh, then I'll be curious about how his opinion could be so different to mine, and if I love him, well, to be honest I probably won't care too much to read a gushing review of the man's work.
I think that the review method in Zzap was the best I have ever seen. They would give it a % rating for several categories and an overall rating, then if they thought it was worth buying by most people, it got a Silver medal. If you didn't like that type of game then maybe you still wouldn't like it, but the very best games got a Gold medal. These ones should be at least looked at by everyone. The best bit though, was the boxes that had the other members of staff commenting on the game. This meant that you got three or four different opinions on each game. That made their review method the best in my opinion.
Just take a hint from /., and instead of giving each game a discriptive number, give it a word as well - if it works for /. comments, it'll work for anything!
-1, bandwagoniser, for example, or +5, mind-bending.
Number ratings work ok when they are an average of many people opinions. The reason is people tend to give a 10 or a 0 (tend to gravitate towards the edges of the system.) When you average out a bunch of those then the rating will start to fall somewhere inbetween and have meaning. Some examples of this would be amazon (which applies this to products) and the bloated yak which applies it to websites.
Someone needs to start a rollup review site that takes the video game reviews posted everywhere else and averages them out. There are a million monkeys on the internet so its probably already been done.
Pocket PC Games
The greatest thing about number scores in reviews is that they let reviewers get their entire message out, even in a confined space. For instance, a lot of the magazines reviews for Prince of Persia started with a small blurb about how the reviewer absolutely LOVED the game, but the rest of it was usually dominated by the warnings that the reviewer felt were very important for every player to know: the camera has some problems, the fighting can be repetitive at some points, the difficulty is very uneven, it's as short as some GBA games, etc. Taken as a whole, the review is very negative; 80% of it is a list of complaints. But taken as a whole and with a score attached, the reviewer is allowed to use his very limited writing space to give the reader a head's up about the game's short-comings while still stressing how wonderful the game really is.
Magazine reviews don't often have the luxury of including a "bottom line" sentence like this one, let alone one that's in a separate paragraph (like this one!), so the number score really helps them sum up their view on a certain game without forcing their opinions of a more obscure game off the magazine's review section and into the ass-end of their website. It allows them to cover both the super-hyped AAA games like Metal Gear Solid: The Twin Snakes or Splinter Cell: Pandora Tomorrow and mostly unknown cult hits like Disgaea in the same issue without charging more than $6 an issue.
On a separate note, it also helps when comparing games. If you generally agree with a reviewer and follow them over a period of time, in the same way that many people follow Roger Ebert's movie column and TV show, then you can use their scores to compare different games that they've reviewed. For instance, if you loved Devil May Cry and Shinobi and a certain reviewer gave them both a 9.0, then it's worth looking at the scores to see how that reviewer sizes them up against a new game that you want to buy, especially something similar like Crimson Sea 2 or Samurai Warriors. He could really like those games and find very little to complain about when reviewing them, but still give them a 7.0, meaning that there's nothing wrong with them, but that they're not really top tier games, either.
And really, it's not as if having a number there hurts the review in some way.
I can remember GameRankings.com, there was another one (by GSI, IIRC) whose name I don't recall.
...you'll love www.gametab.com . I'm not affiliated, it's just something I wish I had done when I thought of it. :)
It even links to the reviews, so you can read through them all. The numbers help, so you don't have to waste too much time reading about really, really low rated games.
Forget about the scores of posts on /. - why trust anyone else to review stuff for you, let alone someone whose name you can see on article, and moderators are *anonymous*.
Also, I don't understand why some women should be deemed more or less attractive than others, because that is extremely subjective.
I don't read restaurant reviews, because if the place has rats, I want to find out about it first hand.
Numbers are really bad descriptors of a monetary situation as well; the numbers for the American financial situation aren't all that good, but hey, Bush says everything's cool, and that's what the number's can't describe.
--
Okay, I went a little overboard, but still.
However, it is a proven fact that people rating pretty much anything (academic performance, produce quality, car safety) become wildly inconsistent when rating on a scale above 3-point-something.
Percentage as game review score is not only flawed, it's supremely stupid. A 50% rated game in this system is most likely garbage of the highest order, while 100% score is routinely saved for Duke Nukem Forever, the final game, that is to be brought unto us by Jesus on the Day of Reckoning.
I think 1-10 scale, with 10 actually given to games which satisfy with supreme quality and/or innovation would be the most acceptable system. 1-5 would be better scientifically, but it just doesn't allow for enough "color" for a game magazine.
A discerning gamer should never base their game impression (and ultimate decide on whether to buy... or sadly, pirate) solely on these singular values because they abstract away all the qualitative properties of a game. That said, *-star ratings and final numbers /10 or /100 or percentages are all there to give a very quick and summative value on a product.
As someone mentioned earlier, many people want a general impression of what they're about to read. Personally, I like how sites like Gamespot and Gamespy throw the rating right up front, whereas a place like Firingsquad with its insightful yet girthy reviews requires navigating through a drop-down list to check out the "final verdict." I suspect most would rather spend time reading and learning about a "4-star" game than a "1-star" one.
Of course, that leads to the perceived notion that there is some grand quantitative scale when you see something like 79 and 81 / 100. Is the 81 game really better than the 79 reviewed on another site? Ultimately it's up to the reader. It's sometimes good to have bias -- if you're a hardcore genre or platform player, you may be more inclined to accept the given idiosyncrasies (i.e. directed linear levels vs. free-roaming, checkpoint saves vs. save anywhere, etc.).
These are ordinal values at heart, and should not be compared at interval levels.
Now with respect to that article, the author makes a good point about reinforcing the qualitative, descriptive muses of the reviewer. However, it's often preferential to give different abstraction levels of your information to pull in a greater volume of readers. The rating/percentage is a good start. It's doubtful that many readers will engage a lengthy game review (no matter how elegantly written) without having a hint of the final mark. Why read eight pages if it's a really crusty game? Conversely, why do that with a game that's already known from other sources to be great? Just a quick check to verify assumptions, and you're off to go get it. Game reviewers are not supposed to write elaborate and astounding essays for which its effect will fail if abstracted into a single value. They are supposed to aid in (and perhaps entertain) the decision to acquire a game for which the player will ultimately decide whether or not it is of good or sufficient quality.
It's necessary to have and utilize both a summative value and a qualitative review. Relying exclusively on a single value leads to game misconceptions, while a written piece alone cannot realistically convey your information to all but the committed (or bored) readers.
-Victor Chow (Elder_MMHS)
As someone who writes for a videogame website, we had this exact same debate. Numbers are ultimately meaningless and arbitrary, for all the reasons that the original poster said.
However, we use them anyway. Why? Because as much as we hate to admit it, there are a LOT of people out there who simply WON'T READ THE GODDAMNED REVIEW. In order to do most games justice, you have to write a great deal, and these Ritalin cases simply can't sit still long enough to read it all.
I experimented with this briefly, and the result was a flood of e-mails from people asking, "So, what did you THINK of the game?" No matter how plain I made my opinion, or how basic my vocabulary was, it was clear that they simply weren't READING the review.
We have our point ratings there basically as a way to shut those people up.
However I think we're running into trouble now, because at many points in the past milestone games have received, with hindsight, what we can now see as slightly overinflated scores which have owed more to their groundbreaking new effects than actual gameplay. Perusing the scores of early PS2 games can easily confirm this, as the typical average score has come down (I particularly hate when games get marked down because they're only marginal improvements over predecessors). I'm sceptical as to whether games are actually going to look that much more impressive in the next generation, and contention in the industry is squeezing the poorer software houses and titles out, which should lead to an overall level of quality increase.
This will (hopefully, IMO) lead to production quality being a virtual given, and allow scores to more accurately represent how good the game is, with 5/10 being average, and worth getting for fans of the genre (like I believe how most Final Fantasy games should score!). For example, if you were a big fan of a certain band, then you'd probably buy their CD even if a selection of music magazines gave them 4/10, as you know you want to hear it anyway; however if Mario 128 came out and received a 4/10 average you'd definitely have to think carefully about getting it.
You take every the game review and convert the score as a percentile (4/5 stars = 80%; 9/10 = 90% etc), then average all the available scores.
If a game can get 150 reviews and can average >95% then you know it's probably a pretty damn good title. Only about a dozen games have achieved 95% average ratings since the website started: Zelda:OOT, Halo, MGS2, GTA III, Metroid Prime are some examples and they represent some of the best titles those platforms have to offer.
Generally
>90% = excellent
>80% = good
>70% = mediocre
anything else is probably pretty poor.
No single reviewer with a bone to pickhas undue influence to the overall ranking
I completely agreed with this editorial. It's not much of a surprise, then, that I stopped reading CGW when they started adding scores to their reviews, something which they said they would NEVER DO. Not that I'm bitter.
Condemnant quod non intellegunt.
As has been mentioned by many of the previous posts, numbers can be beneficial for a number of reasons. However, I can see two major downfalls to using numbers: one on the part of the readers and one on the part of the reviewers.
With rating numbers, readers are allowed to skip over the actual review, ignoring details, and comparing games solely on the basis of how one's number stacks up to another's number. Of course this isn't a "bad" thing. Perhaps that's all the information the reader wants or needs. The reason I label this a downfall is only because the reader may skip over some vital information that the reviewer was trying to get across.
The other thing is that with a large range of values (0 - 10 with half values), what's the difference between a 7 and a 7.5? Moreover, what do the numbers really "mean?" On a 5 point scale, one could interpret the scores as Very Poor, Poor, Average, Great, and Very Great. But on a larger scale, what does it mean to have a Very Great, a Very Very Great, and a Very Very Very Very Very...(and so on) score?
It's well known that people will interpret a score of 5 out of 10 to be "poor" rather than "average," and it's quite natural for people to review something closer to 7 or 8 if they think a game is just "good" rather than giving it a score of 5 or 6, even though that would be more accurate. This causes ambiguity and may not be the clearest indication of what the reviewer intended. Though someone may want to rate a 4 out of 5 game a little higher than another 4 out of 5 game, the difference should be made in the actual review, rather than relying on a potentially ambiguous 7 out of 10 vs. 7.5 out of 10.
Of course, if a publication did make use of a 5 point scale, this takes us back to the intention of this article: Is that enough to really indicate how poor/good a game is? Not by itself, no, but it's a clearer indication of how a game stacks up overall, and with the accompanying review, a smaller range for the ratings is far clearer than a larger range without as much potential for downfalls.
I'm guilty of this myself. I see a rating, I know very roughly what the review would say, and I move on to something else. My curiosity has been satisfied.
Of course, ratings are handy. Even a casual viewer immediately knows, in essence, what the writer thinks. (Or, sometimes, what the writer's editors think. A game's rating may have many fingerprints on it.)
But, taken by itself, a number contains no degrees. No shades of gray. No hidden depths at all. Ratings only make games (or whatever is being rated) more similar; a "6" for shooter is a "6" for an RTS is a "6" for an adventure.
I suppose the convenience makes game ratings are inevitable; they're used for virtually everything else in popular culture (save for books), so why not games?
But it still feels like a battle lost in the war between instant gratification on one side and content and context on the other. People already are reading less than they once did, and this is just another small thing that to discourage them.
But one suggestion: I'd like to see ratings at the end of reviews, rather than at the top. That way, the rating is still available, but the reader now has to "earn" it by paging through the review. And if he's paging through a review, he might as well read it.
That can't be bad, right?
Peter
Now, I'm not a regualar reader of magazines in general, and this is the only full issue of PC Gamer I've ever even looked at. (It's the 10 year anniversary edition, my girlfriend grabbed it for me because she flipped through the pages and saw some nostalgia she knew I'd like.)
Anyway, they really do seem to have a sensical numbering system. on their key, 0-39% is labelled "Don't bother." That seems right. I don't want to bother with below-average games, right? The labels then go up from there: "Tolerable" at 40%, "Merely okay" at 50%, "Above average" at 60%, then "Good," "Excellent," and "Editors' choice."
Of course, the words can mean as little as the numbers, but here they really do seem to mean what they say. In this issue, they give ratings that fall into every one of those ranges--except Editors' choice. In seventeen total reviews, they give nine reviews below 60%.
Of course, they have decent full-text reviews, and a nice summary of the main points at the end in the score box.
The system itself isn't flawed...just most of the people who use it are.
Reminds me of communism!
I've always thought that percentages were too detailed to be useful. Is a 76% better than a 74%? Not necessarily, but the classification that it falls into the seventies does give information for a basis of comparison.
Perhaps games should be put into one of four categories: junk, OK, good, and great. MGS would fall into the great category, as would Prince of Persia and Halo. AoM Titans and Empires: DMW would fall into good. Moo3 would be categorized "OK", and Big Motha Truckers would be junk. That way, you can discriminate between those games that should achieve legendary status, those games that are fun to play, those games that might be fun to play but probably not, and those games that don't deserve the CD they were printed on.
That way, players can make up their own minds based upon the review and gameplay whether or not they would enjoy the experience, without giving too much rein to the biases of the reviewer. But those without the patience could easily know what is quality worth giving a look, and what should be avoided.
The ______ Agenda
or the ingenious brilliance of Metal Gear Solid's interactive references to reality
Ahem. Ingenius, my ass.
Having tried to play MGS and MGS2, I'd have to agree that the MGS creators are a bunch of idiots - and that the games themselves are just really, really stupid.
but only sometimes. The way that Nintendo Power used to do it was great. It was out of 100. No game was ever given a 100, therefore making good games have a lower score and not only having 20-30 numbers to rate a good game. Some rating systems are very bad though. Nintendo Power's new system, which is out of 5 stars, are horrible. Great games like Zelda get a 5, but games that arn't great, like Mary-Kate and Ashley gameboy games still get a 5.
He went like one that hath been stunned, And is of sense forlorn : A sadder and a wiser man, He rose the morrow morn.
This article gets my lowest rating ever, seven thumbs up!
Perhaps they should replace the number/star rating with a screenshot. If the reviewer really liked the game, they could put a screenshot of one of the best elements of the game. Conversely, they would put up a screenshot of the worst part of the game, in their opinion. Obviously, as with the number/star rating system, this wouldn't replace the review itself, only provide an easy to access summary of the reviewer's opinion.
That areticle is a load of hokey. He could have used one sentence to explain his point rather than wax poetic about sunsets. 2.5 out of ten. At most. (That said, if you don't like game ratings... got a better idea, mister? Maybe we'll rate games based using words: for some, Firebrand is a higher score than Petunia, but if your game is Donut, it really breaks free of the mold established by low-class Tanktank games.) (I got it! We'll rate games based on the Sunsets in Hyrule score. For instance, I think Halo merits five sunsets in Hyrule, but Grand Theft Auto only three. That way, we can open up the true enigmatic corbonatory flommoxing of the explanatory potential inherent in Metal Gear Solid's twice-removed data streams of love...)
"They say so much, but they never tell you if it's any good."
"Damn TV, you've ruined my imagination, just like you've ruined my ability to -- to, um...uh...oh well."
The beauty of this is that the job of sorting out a huge variety of games (and other things) can be handed right over to a computer, which can access huge databases of this information and collect meaningful results. As of yet, computers cannot read reviews and understand what elements of a game might make it good if it were described in writing.
What good does this do? Well, it saves the prospective game buyer a lot of time and effort; he can easily pull up a game that has a general reputation to be good. Even without a computer to examine the values for him, he can even find this information out at a single glance. Delving further, if a game looks good then he will take more into consideration.
As was said, you really do need to have an in-depth review if you are going to make a final decision, but a number allows a person to get to that stage quickly and painlessly without suffering the tedium of sifting through a big pile of titles whose quality he has no clue about--not even an arbitrary clue. Numbers may be more arbitrary, but they're enough of a clue to use as a jumping-off point.
There are holes in this way of working; people might potentially miss out on a game that they would actually love, even if the reviewer didn't like it that much. The pros outweigh the cons, however, and personal experience has shown me that many of the best games I have played were indeed widely reputed as good (and therefore scored as such more often). This saves me time, and as long as I can find a few good games to keep me happy I don't have to worry about the others that slip by, even if they are super fantastic.
The only thing that keeps the number system from being perfect is that different people have different ideas on what good game is. If we all fealt the same way about all of them then there would be no titles "slipping by". So, in order to make the number system more reliable, one has to be personally matched to a specific reviewer. If you can find a like-minded person who will rate games in a manner that you would, then you'll have infinitely more seccuss than with any random critic.
We might not have the time to personally try every game in existence, but we do have time check out games that are reputed to be good, find reviewers and scorers that tend to think the same way we do, and even try a few random games "just for the heck of it" (just in case we might find one of those elusive masterpieces). This is what people do in the real world, and when you look at the system there doesn't really leave that much to be desired; all of the bases are adequately covered. Granted, they aren't perfectly, but to a degree that is satisfactory for everyone.
Taking all of this into account, the number system serves as one of the basises for a larger system in place that has not failed for me yet, and I'm probably not missing out on much because of it. If I am, it's only to a negligible degree. ;)
that just told you how much the reviewer has been bribed. One star for T-shirts or free games, upto five stars for cash or promises for tickets to expos.
In Copenhagen we have a Burger King that came into media focus two summers ago.
Why?
They had their traywagons of burgermeat standing in their back yard, in the sun. With rat poison around the meat patties.
People still go there, all the same.