Slashdot Mirror
Do Game Review Scores Matter?
jasoncart writes "This piece on Ferrago discusses the use of review scores and the significance they play in gamers purchasing decisions. Ultimately, according to columnist Ben Parfitt, review scores are pointless." From the article: "Few things wind me up more than when what appeared to be a well-balanced and thoughtful gaming discussion descends into a successive barrage of review scores and Gamerankings ratings."
I think that what you should do is read the scores that you want to read but dont believe them to the letter. I read a couple of gaming magazines, i look at the score they give some games and take those into consideration when buying games. I dont put all my faith in those scores but i at least see if other people liked the game and how much. Sometimes they blow certain parts of a game way outta proportion and ive found those kinda features. So i read multiple reviews from different sources of the same game. If all the sources give a decent review, then that also adds to my consideration of the game. So i think you should read them, if you want to, and if you do, take them with a grain of salt and read up on the games and the reviewers both.
"Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive." - C.S. Lewis
When game reviewers on the big sites stop writing like robots, maybe [the scores] will stop mattering.
The best judgement of a game (because of the above), is to see anomalies in gameranking.com listings. If IGN or Gamespot or Gamespy or EGM give a review that seems to be an anomaly, ignore it. It's a bought review.
Always read the review giving a game the lowest score. Or stop reading reviews with scores altogether. The latter works well for me.
schild
editor, f13.net
a system meant to reduce all that is to be said about a game to a single-digit number somehow leaves the smaller nuances unsaid.
stunning.
Until I see a game developed by an AAA game developing company rated at a 1 or a 0, their ratings hold no power in my book. If they think all high-budget games are "excellent" to the point of having 9.x ratings out of 10, then I certainly don't trust their opinion.
Jay | http://oldos.org
A:Yes, but not as much as advertising budgets.
Slashdot: News for Nerds, Stuff that matters only to them
Scores do have a place in reviews, I use them as a guide to which games I should look into further and which games I should not waste my time on. It is of course best to look at at least 3-4 reviews of a game you are considering buying and see if you can get a demo before you put down your hard earned dosh.
Professional reviews are useful for a very general overview of a game, but usually it is easier to find important details at places like GameFaqs. People posting on their own without a profit motive are more likely to mention that a game is really short or overly linear, for example. Of course the noise to signal ratio is very high, but the information is there for people with a little patience.
-- Microsoft is the most expensive commodity operating system and office suite vendor in the marketplace.
Sites like game rankings give you the review numbers from a number of sites. Given that you get a fairly good idea of where a game sits. For films movies does the same for films. Given that and active reviewers on the site gives you a fair idea of what is good.
Knowing your reviewers is the other way to get good information. If you regularly read a particular reviewer you'll get a good idea about what they like and what they don't like. This is easier with films than games, but still possible.
Reviews are definitely subjective, but are still a useful way to make your money and time go further and if a bit of thought is used are well worth looking at.
The article points out the flawed logic inherent in the system of averaging random scores assigned on a 0-10 scale. Is it possible for a game with an 8.1 average rating to be better than a 9.1 average rating? Yes. Everone is entitled to their own opinions. The author cites an example of where in his opinion a game with an 8.1 rating is more enjoyable than a game with a 9.1. Apparently the author was chastised for expressing his opinion. This is a downside to averaging: it can lead to groupthink.
But what can we do to combat groupthink? Consider the following simple ranking systems:
At first glance it appears that any one of these systems would work adequately if used consistenly and then averaged for at least 30 reviewers. The average scores should then in theory be meaningful, right? Well unfortunately we have to note the key words there: 'used consistently'. If the reviewers cannot agree on a format, then you have to reduce it to the lowest common denominator. Similarly, many reviewers would simply ignore the 'recommended' option in favor of the extremes. This suggests that perhaps the best option is to average the binary review score.
But wait! What if the system gets flooded with artificial reviewers? This happened in recent memory when Sony admitted inventing fictitious reviewers to gush about the movie "A Knight's Tale". What if those artificial reviewers get included in the average? That is a serious problem, but it's easily addressed with moderation! Examine each reviewer's track record before adding them to the mix. And then pull any reviewer that is consistently out of touch with reality.
Recommendation: Find a bunch of games you like and a bunch of games you dislike. To be thorough, you want to find at least 30 in each category. Search out critics that agree with your tastes for at least 2/3 of the titles. Average the opinions of these critics when a new release comes out. If the result comes out at least 2/3 (0.67), then you'll probably like the game.
Addendum: For better results, you can assign weights to certain critics and then perform a weighted average. For example, you might observe that critic A agrees with you 90% of the time, while critic B only agrees 80% and critic C agrees 70%. In this example,if only C dislikes the game, then your result will be greater than 2/3 (favorable); however, if A dislikes it, then the result will be less than 2/3 (unfavorable). Keep in mind that to be statistically meaningful, you need to have at least 30 reviewers, and also remember that if you get burned by a critic, you can always mod him down. In fact, you could in theory set up a dynamic system that continuously adjusts the weights of reviewers based upon how well they match your opinions.
A note on resolution: If you're able to get tristate or better "resolution" in your reviewers, more power to you! In fact, I encourage this. However, on a practical note I think it will be difficult to find enough reviewers with a high enough common denominator. Of course, this does not prevent you from assigning special weights to the differing rating systems used by various reviewers. Be creative! Invent your own system. :)
Pipe dream: It's my personal pipedream to have a website where everybody can register their opinions on various topics. Each person could then seek out (or be matched to) other individuals with similar tastes. People with less time to devote to reviewing things would defer their opinions to others. Eventually this would trickle up to a small set of individuals making recommendation
It's all about who's reviewing it and how much you trust them.
I have a few sources that I trust pretty well. But even *those* it's not what score they give it. It's what they say.
Brian
Here is a quote from the article.
Games should always be considered on their individual merits, on the qualities that they offer and the accomplishments they boast. This can never be distilled into a percentage or ranking out of ten. Hold games up to examination and this evidently becomes the case. For instance, when I reviewed San Andreas I gave it a 90%+ review score. I would not on the other hand award as high a score to something like Castle of Shikigami 2 on the Gamecube though personally I feel it is the better game. It would score lower because it is less technically accomplished, far smaller in scope and offers far less variety. I still prefer it however because what it does it does extremely well and when push comes to shove I would rather play it than San Andreas. That's not to say I think it's more accomplished - I simply prefer it.
I'd ask the writer of the article this: why the hell did you rate GTA: San Andreas better? This IS the problem with these scores. GTA gets a better score simply because the conventional wisdom says it is a more accomplished game, and NOT because the reviewer actually likes it better. He admits it in the article for all to see. Hype = high scores, and even someone who is writing an article about how the scores don't work is swayed by it.
This is how a game like Katamari Damacy gets lost in the Half-Life 2s and Halo 2s of the world. Conventional wisdom says that a strange Japanese game with no real storyline, blocky graphics, and simple gameplay is not as "accomplished" as a sci-fi FPS. The $20 price tag alone almost screams "inferior game." But an expensive price, polished graphics, long development cycle, sweeping advertising campaign, and a big booth at E3 are not what makes a good game.
As a teacher, I frequently deal with student complaints about why a particular paper was a "C" or "B" paper. "Because," I'll answer them, "your paper was merely average. It fulfilled the requirements of the assignment, but did nothing more."
Similarly, game scores seem to evoke this feeling among fans of particular games. Anything below an 8/10 is perceived as "crap."
In reality, I own games that I would rate as a 6/10 which are still enjoyable. These games may be merely average, but if certain aspects are present, they can still be anjoyable. "Buffy The Vampire Slayer: Chaos Bleeds" would fall into that category. The game received in the 6.5/10 range all over, and it's a score I would agree with. The camera is lousy, and the controls are inferior to the original in almost all respects. Despite this, the story is entertaining, the voice acting is pretty good (with the exception of the knock-off Willow), and the subject matter is entertaining to me. It is a 6.5 game, and I don't believe anything to the contrary, but it's still entertaining.
Dead or Alive 3 is another great example. It's probably a 7/10 game. The graphics are beautiful, yes, but the game wasn't really substantial change from DOA2. Weakening the counters improved the battle system, but the new characters were universally dull (except for Hitomi), and the game was otherwise nothing more than the second. It doesn't really deserve anything spectacular as far as scores are concerned, but it's a favorite with my friends and I when we get together at my place.
EGM was one of the few magazines I discovered that was willing to make this stand. a 5/10 game was AVERAGE. You might enjoy it if it had a particular point that really appealed to you. If you were a huge RPG fan, a 6/10 RPG would be worth buying if you'd already finished the last three 8/10 games. The 6/10 was not crap. Games at 3/10 and below were crap. And a game had to be spectacular to get into the 9 range. Unfortunately, people don't seem to be willing to accept that scale; everything needs to be between a 6 and 10. The problem is that it just dilutes the actually worthwhile games. Gamepro was notorious for this. They gave straight 4.5/5 and 5/5 to Starfox64. The game was good, but it was not worthy of that level of score. When compared to something that truly was, it served to make the worthy game's scores "lesser."
Do scores matter? In EGM's case, I'd certainly agree. Back when I still kept up with that sort of thing for professional reasons (I was an assistant manager at a game store), they were generally pretty trustworthy. In a case like Gamepro's, which unfortunately seems to be more the standard than the exception, it makes the scores completely inconsequential. At that point, I learn to just ignore the score and read betweeen the lines of the reviewer's euphamisms.
What? I lost you after Score:3, Insightful.
While MOO III was a botched game, you'd have to purposely try to get one worse than this.
http://www.netjak.com/review.php/537
Netjak.com independent reviews of domestic & import video ga
Woeful name for a publication, however, I perceive their reviews as being extremely honest and balanced. They won't hesitate to pan a game, and I definitely don't get the feeling that they are giving a bia$ed apprai$al. Their coverage is detailed and offers a refreshing maturity compared with most of the magazines that I've read out there.
Check it out :
Computer Games Online
Cheers,
Kafka