Slashdot Mirror
Investigating Bias In Videogame Review Sites
jvm writes "We've all read comments that some videogame sites are allegedly biased for
or against some games, consoles, or companies. So, Curmudgeon Gamer has investigated whether bias can be seen in the review scores
over several games on each console. The review sites in question are GameSpot, GameSpy, and IGN, each of whom are compared to the game
review averages on GameRankings.
Additionally, a selection of review scores for crossplatform games are
examined. While solid conclusions are difficult to draw and improvements can
admittedly be made, perhaps people will find these results interesting to examine and
discuss."
The spreadsheet I used for my computations and graphs can be downloaded here. It's in OpenOffice format.
Nice to see people use opensource software, he could of released the data in excel.
I have to say, from the game reviewer's standpoint, that it's hard to give a game a bad rating.
1) You have advertisers. Typically, these people keep your site alive. They're expecting you to review their game.
2) You like games.
3) You get tons of swag, press passes, the royal treatment at trade shows, and a ridiculous amount of geek cred.
4) Did I mention that you like games?
5) You get to mention Sanya Thomas' butt in casual conversation. (see?)
6) It's a freaking game. What's not to like about a game? (well, unless it's Final Fantasy XI -- which I refuse to admit is actually a game)
Mystery@Warcry.com
The question is there no bias in this investigation of bias in video game sites?
Unless you were just born with the knowledge you'd have in 35 years but had no opinions, all reviews are gonna be a little biased, i mean cmon, reviewing games arent a mathematical thing, its giving your opinion.
You forgot the Editor... the position where the one with the Big Thumb can thump a candidate review back to draft status for adjustments. The EIC is responsible overall for a publication, and can be 'coached' to tell his team how to play.
It's mildly interesting to see that there may be an anti-GameCube bias in some reviews. However, what I would much rather see is the same methodology applied to other factors, like who the publisher is and who is advertising with a specific site. I'd never heard anything about specific-console bias, but I've darn well heard an earful about EA getting a free-ride on steaming piles like Earth & Beyond. I'd like to see whether the numbers back that up.
Bite the hand.
The stupidly named GamesAreFun is so weighted towards Nintendo its not funny. One of their no talent writers sometimes posts here - blindly defending Nintendo as if they can do no wrong. It would be quite funny if it wasn't so pathetic.
Example - two of their recent "articles" were -
1. Why online console gaming sucks
2. Why I don't want extra features on my console
Well guess what console doesn't have any decent support for Online games and no DVD playback feature?
Simoniker please don't ever link to them again
because you aren't doing anyone a favour.
If I buy a $50 game based on your review then I can never trust your opinion again.
Regular readers should notice that you never give bad reviews.
If you can't be honest then you are *just* advertising not "relying on advertising".
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
This article just highlights to me the benefit of meta review sites such as Game Rankings and Meta Critic. User review scores also help counter any bias in reviews. What I'd like to see is a comparison of official magazine review scores compared to the average. OXM would probably turn out to be far higher than the average - especially for Microsoft published games.
All in all, the Internet is a powerful equaliser in these matters.
Curmudgeon Gamer's examination is certainly an interesting one, but what it complete neglected was the trend of increasing stupidity in gaming reviews.
Perhaps the slow but steady rise of ludology means that we may yet see a rise in the quality of game reviews, but by and large the current level of intelligence in most game reviews is as low as the number of female companions most of the game reviewers have (and I mean that collectively). Reviews inevitably follow the "Background, Cool Trivia About Game, Graphics, Controls, Bad, Good" template. The size of the font of the scores at the of end review keeps getting bigger, as does the size of the screenshots. Basically, we're left with an industry wide standard of two paragraph GamePro-esque reviews that generally boil down to, "It rocks!" or "It sucks!" EGM's recent redesign is a fine example of this. There is little in the way of anaylsis, or the game's relation to the industry at large, or (heaven forbid) society itself. I think there are sites on the net that are the exception, such as GameCritics.com, and ironically, Penny-Arcade. Likewise, I think either Computer Games or CGW (I always get them mixed up) is making a conscious move in the other direction. Kudos to them, and shame on the rest of the reviewers. I suppose that online is, strangely enough, more "textual" than the magazines due to the lack of space confinement. But the actual content is about as low.
What I'd like to see is a complete abolishment of scores all together in a magazine or website. It would mean, yes, you'd actually have to read the review. That's not to say there's no place for the blurb-y Gamepro-style of reviews with a big thumbs up or thumbs down at the end of the review. There obviously is, as it's quite handy for a quick reassurance when you're in the line at EB almost ready to buy the game.
But there needs to be more peer reviewing, more intelligent discoursing, and more analysis ala ludology in the gaming industry. Not everything is the next best thing since, well, the last game the person reviewed. But some things most definitely are, and it's these that need extra attention. Not, contrary to popular belief, more screenshots.
my only problem with the article is that it is based on USER REVIEWS (the term he keeps using), and not the SITE REVIEWS, when the article was stated to be about whether or the SITES are biased and not if their READERS were biased. This also doesn't take into account the people that have no lives and will sit their and submit bad Xbox/PS2/GC game reviews on gamerankings without even owning the system, just to show their fanboyism.
Sorry I'm posting as Anonymous Coward, I just don't feel like setting up an account right now. I will post my e-mail though, nickerboy@aol.com
that can be drawn is that game players are harsher than the professional critics. no duh. visit any official or psuedo official forum for a game, and chances are you'll encounter a rather vocal group complaining about problem X. never mind that 95% of the players don't encounter problem X, but the 5% that do, really want to shout about it. I imagine the same thing would happen with reviews. One is much more likely to bitch about a game that was less than satisfying than praise one that was enjoyable (as you get satisfaction from the game in the latter case, and are satisfied by bitching about it in the former). So the scores are biased in the downward direction. that isn't too surprising.
I forgot to mention Play magazine. I don't particularly always agree with their reviews; I think that Gunvalkyrie is difficult not because of content but because of poor control design, and so on and so forth. I feel that at times Play tries too hard to be "hardcore." But Play magazine provides a niche, instead of trying to be *THE* magazine for gamers as [insert game magazine name here] is so fond of doing. You have to admire that.
Likewise, I thought the reviews in the defunct NextGen were absolutely brilliant writing. I need to be careful in saying this, because they were notorious for their doublepaged screenshots, but they really did see games as an integral part of society. However, that's what we need in the gaming review industry: great, creative writing. It needs to be an art unto itself. I see a long but straight line between the rock music reviewing of the late sixties, early seventies, and the gaming industry reviewing of today. Rock music was coming of age as the game industry is very much doing today; it's growing beyond the size of its own knickers, so to speak. Likewise, both industries are highly release driven, and every new release in either rock or in gaming was the "next big thing." There's a general feeling that reviewers are "cool" now.
That said, every game reviewer, print or otherwise, should have to watch Almost Famous, and they should walk away with the words, "Be honest, and be ruthless" etched in their minds coupled with the insistence that "We are not cool. We are the uncool." If game reviewing wants to grow up, the reviewers themselves need to realize that the game designers and publishers need to hate reviewers. The reviewers need to be the enemy. That passion, combined with the belief that one's own writing and reviewing can be artistic in nature, is enough to revolutionize the gaming media. Will it happen? That's up to them.
Your logical connections seem to be missing!
I always love reading articles with statistics. I'm glad that this article was written.
The first thing to keep in mind is that the writer predominately chose games which generally had received high review scores. So, in essence this article was really about whether there is a review bias among highly-rated games. I think this point is important because it means that each site is generally debating exactly how "very good" each of these games are.
Which brings us to looking at the rating systems of these sites -- can you tell the difference between a 7.0 and 8.0 game? The author should have translated the results for each site back into the units that the site employed.
I don't read a lot of gaming reviews (since the only consoles I own are an Atari 2600 and NES) so unfortunally I'm willfully ignornant about how these sites are set up. Still, if a hypothetical site only gives out ratings in terms of zero to five stars, then each unit is worth 20 points (using the author's scale). So, if we're talking about 9 point discrepancy, we're still speaking about less than half a star on that site's rating system. The author converted everything into a 100 point system, but failed to convert back for the conclusion. I think this would be relevant too.
The one thing I was disappointed was that he ended up looking at the discrepancy from the average score for each site. I would rather he considered the Standard Deviation from the mean instead of simply averaging the deviations together. This way, the results aren't so skewed should one particular game have an enourmous deviation.
Yes, the CSV file is now online so I could do this and perhaps I will just for my own curiousity.
In a similar vein, one could perform chi-square tests with the results between each of the sites to give a better measure of whether or not a bias appeared.
indeed..
Good stuff. My one concern is that by choosing only top rated games, he's built-in a flattening effect that will reduce any appearance of bias. To be a "top" game, by necessity the numbers are going to average high, which means less variation.
An "average" rate game is more likely to have a wider variety of scores. Examining games with average ratings (around 60-80%) would probably yield a better picture of any bias (if existant).
While I'm at a loss to explain just why GameRankings users score the way they do, it is interesting that their opinions of games are consistently lower than the average of all game review sites recorded on GameRankings. Furthermore, the Xbox is by no means alone in this respect; GameRankings users are similarly critical of the top twenty games on the PS2 and GameCube.
It should be noted that the user scores from GameRankings are going to be well below the average for all platforms. As as avid user on GR, I know how fanbois attempt to affect the user rating on games on the systems they do not like with the user voting system (and, conversely, attempt to raise the user rating on even shit games on their system of choice by voting "10" on every game). Many of the votes are from second and third accounts used just to vote against games on the other systems and for games on their system of choice.
For example, I'll show the top game from each system and how many total votes vs how many of those votes were for a "1".
GameCube:
Metroid Prime (User Avg. 8.3)
Total votes: 1969
Total "1" votes: 241 (12.2% of total votes)
PS2:
Metal Gear Solid 2: Sons of Liberty (User Avg. 8.6)
Total votes: 1573
Total "1" votes: 73 (4.6% of total votes)
Xbox:
Halo (User Avg. 8.7)
Total votes: 2350
Total "1" votes: 186 (7.9% of total votes)
Notice that the PS2 game actually had the lowest percentage of "1" votes (4.6%), while the GC game had the highest (12.2%), even though the GC game had less total votes than the Xbox game it still had a higher percentage of "1" votes. This trend seems to fluxuate between the GC and the Xbox for the highest "1" vote percentage on games.
One of the things those of us on the message board wanted to do was only make votes count towards user average if a review went with it, in order to keep fanboi votes to a minimum effect on the total user average....but that never happened. And now that Scott sold GR to C|Net, we doubt it will ever happen.
Thursdae