Slashdot Mirror
Game Reviews Don't Matter, Study Finds
Next Generation has an article up looking at a report from SIG, on the correlation between game reviews and sales. Their findings indicate that, while reviews obviously do have some effect on games sold, there just isn't that much of a correlation. From the article: "He said he doubted that publishers and PRs would stop caring about review scores, especially as they matter a lot with consumers who compare games from the same sub-genre — say, basketball games. But he said that, as with last year's report, the report's findings are unlikely to be popular. 'We received a lot of attention but the stats do not lie,' he said."
I tend to read about games through, for example, their website, previews, beta-tester forums, etc. I make a decision about whether or not a game is worth my money on my own, go out and buy it, play it, then think "hmm...that was [fun/stupid]. I wonder what other people think about it." Then and only then do I go and read reviews.
Anyone else think the comments just weren't rendering right before they turned off ABP and saw ads?
n/t
Seems to me that for every major game release out there, there's tons of pre-release hype generated by both the publisher, and the websites/magazines that end up reviewing the game. Screenshots, videos, interviews, all that serves to hype up the game. That hype often ends up being the reason for which people choose to buy the particular game. The whole pre-ordering system makes it even more common. Many people buy the game on the day it comes out, without paying too much attention to the reviews (many of which come later).
This is a sig. It is appended to the end of comments I post.
Especially when rubbish like Enter the Matrix sells tons of carts regardless of actual review scores.
There's a hardcore segment out there that scrutinizes every scrap of information, for sure, but most games are still purchased by Grandma and Grandpa for little Joey because of the title, genre, or franchise, not because of an aggregate score somewhere in cyberspace.
popularity is usually based on marketing, not on reviews.
sucky games (dirge of cerberus) do well with good marketing (commercials, ads, et. al.). good games with little/no marketing don't usually do nearly as well (not that they don't do well at all, just that they lose some of their potential).
a large enough portion of the market doesn't read reviews and bases their purchases off of the "coolness" factor of the game, instead of the quality. if a commercial or ad can make the game look cool, then they're all over it.
'We received a lot of attention but the stats do not lie,' he said."
What was that saying again? Something about lies, damn lies, and statistics?
DYWYPI?
A lower than expected or higher than expected review from a major game site, can generate hundreds of posts on Evil Avatar, or Gamespy, or bluesnews. I think that the need for reviews are not as great as they used to be, before the internet. I am sure that a lot of people get an idea of what a game is about, in previews. The secret to reading reviews, is to find a reviewer that likes the same games you do. :-)
The interesting thing here is that the study only looked at reviews versus sales.
It said nothing about reviews (written by magazines, etc) versus consumer opinions (or user reviews). It also said nothing about consumer opinions versus sales.
In my experience, reviews written by reviewers generally align pretty well with what consumers think of the game, while sales have little to do with either. In short, sales seem to have little to do with how good a game actually is. Sales seem to correlate more with things like movie and cartoon and brand tie-ins to a game, distribution methods, price point, and other such factors. All of this is really no big surprise, since the game industry has always successfully relied on churning out such drivel and it has obviously worked pretty well for them most of the time.
A more interesting thing to study would be what percentage of sales are purchases made by people who know nothing about games and won't be playing the games themselves... such as parents and grandparents choosing games as gifts for kids, etc. I bet they make up more than 50% of sales.
Remember when the Atari era went bust and the bottom of the video games market completely dropped out? My theory is that it was because the industry stopped creating any good-quality games, having realized from experience that they could just produce well-branded crap and rely on all those gullible non-gamer sales. I think the problem is that when the market floods with crap, the gamers (who ultimately receive those games from the purchasers) completely lose interest in games and stop asking their parents to buy them more. So then the purchasers stop buying completely.
In other words, a sufficient minority of titles must continue to be of good quality for the industry to sustain itself, but once that sufficient minority is met, the rest can be crap and the industry can thrive off the crap. The industry then foolishly thinks all it needs to produce is the crap, which kills demand completely, which kills the whole industry.
Moderator hint: a comment is neither "Flamebait" nor "Troll" if it is true.
Game reviews do matter - its called word of mouth. Terms such as "clickfest", "pushover" and "tedious" are negative aspects of a game that you should avoid.
If you want a list of things that a good review should look for, all you have to do is find lists of Cliches and reviews that make note of them. There are similar lists for strategy and action games - but common components among all such lists involve being railroaded through events outside of the players control (e.g. is captured by 3 units after taking out 2000 soldiers), or events that are obvious enough to be traps but the player is forced to go through them to advance the plot.
Newsflash: Can't predict the future based on one criteria.
Hell, the article doesn't even say that there isn't a correlation between good reviews and good games, just that it's not a reliable predictor of sales. Well duh. Maybe things like number of consoles in the market for that game, or marketing, or whether its a sequal, or the price, or whether it's released in May or during the holiday season, might all play a role.
We expect that the same game with good reviews will perform better than that same game with poor reviews. The article confirms that expectation, while trying to sound like it's conclusion is surprising. It's not.
Some people will buy a Pokemon game no matter how bad the review is. This is obvious. Doing a study that confirms it doesn't change that it's obvious and your study is just an excuse to fill some pages under the guise of 'news'.
paintball
Let me tell you a story. There was a game I had been long anticipating, a game I was so psyched about. I was, I must admit, young and naive, I had never played a truly bad game before, I didn't think 'bad' games existed, other then the Edutainment games. I got the game the day it came out. The game was Superman 64. I have never bought a game (other then sequels and launch titles) without reading a review ever since.
I don't buy many games (mostly I rent because most games are worth the $$$). If I REALLY want a game (New Mario, Guitar Hero) I'll just buy it ("Damn the torpedoes, full speed ahead!"). But if I want a game but I'm not that set, I'll wait for the reviews. If the review seems positive (no major flaws, still sounds fun) then I tend to buy the game. If not (bugs, doesn't sound fun, etc) then I don't.
I don't buy games just because of reviews though.
For the record, I get most of my reviews from X-Play (which I watch all new episodes of). I also check specific games on GameSpot and IGN.
A demo is FAR more likely to change my opinion of a game than a review, but I like having reviews around to see what is happening with games I don't tend to play (sports games) and to save me from buying a piece of junk that won't be that fun.
However, I can tell you that my little sister and her friends (all about 14) don't read reviews at all. They are just mindless consumers. "I liked spider man/the hulk, I'll get that game". "That football game looks like it might be fun". Mostly we rent (so many games are terrible) but they don't even consider reviews before renting (not their money, I guess).
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
which is what the study concludes, and what should be pretty obvious if you read lots of game reviews. Game ratings are basically random numbers between 8 and 10, and where it falls in that range seems to be largely divorced from the content of the review. How many times have you read a review that said something like "the gameplay was fun for the first few levels, but quickly became monotonous and boring" but gave the game a 9.5, or one that said "despite a few minor flaws, this game is all around a lot of fun" and gave it an 8? When I'm out looking for a game, I think I'm going to weigh "monotonous gameplay" a lot more than "Overall Score: 9, Excellent!".
I can understand them using the game rating, as it's the only obvious number you can apply to a game review and do correlations with. However just having a number doesn't mean it actually represents something, and I'm not surprised that game sales don't correlate well with a number that is basically pulled from the reviewer's ass.
The enemies of Democracy are
Perhaps I'm in the minority, but I always check the reviews before buying a game, and that silly little number does impact me. I am much more likely to buy a game with a score above 9 from the magazine and a similar user score than a game that scores below 9 in both categories. I tend to rely more on the player review numbers though, and ultimately use them to make my final decision. Actual player comments are also very nice. The whiners and the fanboys are easy to filter out, and a lot of people leave good comments about core game features that will have an impact on me. If I see the same complaints or cheers about an aspect of the game from several people, I tend to take those opinions very seriously. Likewise if I read a dozen comments about the game crashing and two comments about how the game never crashes and is awesome, then I'm going to assume the game has some issues and I might wait for the first round of patches. (I guess I'm mostly referring to gamespot, because it happened to be the game site that I found first however long ago and I don't feel like following multiple sites)
Here's an embarassing thing to admit: if I've been looking forward to a game and it gets an 8.9, I actually feel disappointed and I might take a few weeks to make a decision. But if that game gets a 9.0, I'm much more inclined to run out and grab it. Yeah, it's dumb. It's just like I know that $49.99 is only a penny away from $50, but it feels a lot more like $40.
One thing I realized as I was typing this is that while I'll look at the points the reviewer gives a game, I rarely ever read a professional review. They're simply too wordy and glossy. I'd rather just read some player comments about how "it sucks ass that you can't ride the donkey in multiplayer mode" than to try to discern these tidbits from the professional review.
Sales are, at least in my experience, based on three factors:
Name of Game/Company
Hype
Popularity of genre
In that order.
There's quite a lot of really awesome games out there that didn't get any attention. Why? They lack at least 2 of the three factors.
Imagine something is labeled "Command & Conquer Final Chapter". Will it sell? You bet it will! It could be the last crappy ripoff, a repackaged C&C1 with new graphics for example (no, you don't need to tell me "it's already been done", I know, thank you very much). It will sell. It has the name. Imagine Halo 3. Quake 5. NHL 2006. Diablo 3. Does/would/will it sell? Yes. Why? Name.
Same with "games to movies". Fortunately, at least some movie studios do care these days who buys the license for the name to slap on a game. It used to be a surefire way to say a game sucks donkey bits if it had the name of a current movie. But they always were bestsellers. Even the crappiest of the lot.
Then there is hype. DNF will sell. No matter when it comes out and no matter if it is up to par. You could offer an empty box and it would sell. Other games have the "new feature" hype, whether that feature actually matters or not. Max Payne was hyped as "revolutionary" in the genre of shooters for its bullet time and the actually rendered bullets. Whether it really was, I leave to the reader.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Why not look at something like http://www.metacritic.com/games/ which pools reviews from a bunch of sources?
My other sig is a
and their opinion is no more valid, or even less valid as I find that any game that is genuinely crap from a major advertisement purchaser doesn't get the shafting it deserves, yet a game of similar quality from a no-name developer/publisher usually gets raked over the coals.
You need to look at factors such as advertising dollars when looking at a review objectively, as well. As an example, a friend of mine was managing a gaming store when one of the game company reps that was showing him their latest catalogue got a call on his mobile from one of the local gaming mags. The guy from the magazine informed him of their latest review of one of their games, and the score was not too flattering, something like 5/10. The rep said "Nah, pull the review then", which caused the mag to back-pedal and make a deal to give the game 7/10 but use the same text of the review. Often you see big release games which don't live up to expectations in the public, one obvious example is Dues Ex 2. That game was an absolute turkey yet when it was released I saw several reviews calling it a classic with higher than 9/10 scores.
Blessed are the 1337, for they shall pwn the earth.
And yes, completists like me, especially of some well-known franchises like Might and Magic, C&C, will always buy the sequels no matter how horrible the reviews are. In many cases, even no matter how horrible the actual games are. Even if some of them are buggy and unplayable, they'll always be there on my shelf.
One of the posts here said DNF will sell pretty well because of coverage. Heck, given its extreme coverage, if it actually comes out, I predict it will be one of the best-selling games of all time. (Many will play it, and those who don't, will still buy it for their collection.)
How many copies of a game get traded in as the days/weeks/months pass since the release date. As that is more of an indicator of wether the game sucks or is worth holding onto, IMO. Few copies of the game in the 2nd hand section gets my attention and gets me asking questions to the EB staff.
My buying habits are erratic, sometime I go by reviews, or word of mouth, or a continuing franchise that I've bought previous versions of. But wether I keep hold of a game or trade it in is based on wether I enjoyed playing the game.
See my art -> http://herbevore.deviantart.com