Slashdot Mirror
Game Reviews Don't Matter, Study Finds
Next Generation has an article up looking at a report from SIG, on the correlation between game reviews and sales. Their findings indicate that, while reviews obviously do have some effect on games sold, there just isn't that much of a correlation. From the article: "He said he doubted that publishers and PRs would stop caring about review scores, especially as they matter a lot with consumers who compare games from the same sub-genre — say, basketball games. But he said that, as with last year's report, the report's findings are unlikely to be popular. 'We received a lot of attention but the stats do not lie,' he said."
I tend to read about games through, for example, their website, previews, beta-tester forums, etc. I make a decision about whether or not a game is worth my money on my own, go out and buy it, play it, then think "hmm...that was [fun/stupid]. I wonder what other people think about it." Then and only then do I go and read reviews.
Anyone else think the comments just weren't rendering right before they turned off ABP and saw ads?
n/t
...I tend to identify quality with specific companies. For example, anything Nintendo cranks out is worth a look, especially in their long-running franchises. Anything Valve Software releases is in my Steam list purchased the day of release, if not preloaded before then.
Of course for those games that don't have this advantage, the best way is simply based on word-of-mouth and also watching the game being played.
Seems to me that for every major game release out there, there's tons of pre-release hype generated by both the publisher, and the websites/magazines that end up reviewing the game. Screenshots, videos, interviews, all that serves to hype up the game. That hype often ends up being the reason for which people choose to buy the particular game. The whole pre-ordering system makes it even more common. Many people buy the game on the day it comes out, without paying too much attention to the reviews (many of which come later).
This is a sig. It is appended to the end of comments I post.
In Soviet Russia, games review YOU.
Only Old North Koreans read game reviews.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
was a great game with perfect lock on and great multplayer action
If you don't have it, too bad
Especially when rubbish like Enter the Matrix sells tons of carts regardless of actual review scores.
There's a hardcore segment out there that scrutinizes every scrap of information, for sure, but most games are still purchased by Grandma and Grandpa for little Joey because of the title, genre, or franchise, not because of an aggregate score somewhere in cyberspace.
popularity is usually based on marketing, not on reviews.
sucky games (dirge of cerberus) do well with good marketing (commercials, ads, et. al.). good games with little/no marketing don't usually do nearly as well (not that they don't do well at all, just that they lose some of their potential).
a large enough portion of the market doesn't read reviews and bases their purchases off of the "coolness" factor of the game, instead of the quality. if a commercial or ad can make the game look cool, then they're all over it.
'We received a lot of attention but the stats do not lie,' he said."
What was that saying again? Something about lies, damn lies, and statistics?
DYWYPI?
A lower than expected or higher than expected review from a major game site, can generate hundreds of posts on Evil Avatar, or Gamespy, or bluesnews. I think that the need for reviews are not as great as they used to be, before the internet. I am sure that a lot of people get an idea of what a game is about, in previews. The secret to reading reviews, is to find a reviewer that likes the same games you do. :-)
The interesting thing here is that the study only looked at reviews versus sales.
It said nothing about reviews (written by magazines, etc) versus consumer opinions (or user reviews). It also said nothing about consumer opinions versus sales.
In my experience, reviews written by reviewers generally align pretty well with what consumers think of the game, while sales have little to do with either. In short, sales seem to have little to do with how good a game actually is. Sales seem to correlate more with things like movie and cartoon and brand tie-ins to a game, distribution methods, price point, and other such factors. All of this is really no big surprise, since the game industry has always successfully relied on churning out such drivel and it has obviously worked pretty well for them most of the time.
A more interesting thing to study would be what percentage of sales are purchases made by people who know nothing about games and won't be playing the games themselves... such as parents and grandparents choosing games as gifts for kids, etc. I bet they make up more than 50% of sales.
Remember when the Atari era went bust and the bottom of the video games market completely dropped out? My theory is that it was because the industry stopped creating any good-quality games, having realized from experience that they could just produce well-branded crap and rely on all those gullible non-gamer sales. I think the problem is that when the market floods with crap, the gamers (who ultimately receive those games from the purchasers) completely lose interest in games and stop asking their parents to buy them more. So then the purchasers stop buying completely.
In other words, a sufficient minority of titles must continue to be of good quality for the industry to sustain itself, but once that sufficient minority is met, the rest can be crap and the industry can thrive off the crap. The industry then foolishly thinks all it needs to produce is the crap, which kills demand completely, which kills the whole industry.
Moderator hint: a comment is neither "Flamebait" nor "Troll" if it is true.
Game reviews do matter - its called word of mouth. Terms such as "clickfest", "pushover" and "tedious" are negative aspects of a game that you should avoid.
If you want a list of things that a good review should look for, all you have to do is find lists of Cliches and reviews that make note of them. There are similar lists for strategy and action games - but common components among all such lists involve being railroaded through events outside of the players control (e.g. is captured by 3 units after taking out 2000 soldiers), or events that are obvious enough to be traps but the player is forced to go through them to advance the plot.
At least in my experience, past success of a franchise (as mentoned in the article) is probably a better indicator of whether I will buy a game... I usually only read reviews if I'm for some reason undecided about a game. I do not buy that many games (having a job seems to interfere with my ability to play too many of them), so when I do it's usually based on something other than someone else's opinion.
I think these days you have two groups of people
1) buy games the day they come out because they have for some reason been anticipating it's release
2) wait until they can rent the game or play someone else's copy to see if they like it.
In both cases, I can see reviews playing some role ("I was waiting for Civ IV to come out, and man, the reviews say it's awesome so I HAVE to buy it" or "I wonder what games I've never played are good enough to bother renting"), but I think negative reviews probably carry more weight than positive ones in either case.
I just wish I had read a review of Madden 2007 for the 360. Worst implementation of Madden ever. I'd rather play the old copy I had for my Apple IIc+...
This space intentionally left blank
Newsflash: Can't predict the future based on one criteria.
Hell, the article doesn't even say that there isn't a correlation between good reviews and good games, just that it's not a reliable predictor of sales. Well duh. Maybe things like number of consoles in the market for that game, or marketing, or whether its a sequal, or the price, or whether it's released in May or during the holiday season, might all play a role.
We expect that the same game with good reviews will perform better than that same game with poor reviews. The article confirms that expectation, while trying to sound like it's conclusion is surprising. It's not.
Some people will buy a Pokemon game no matter how bad the review is. This is obvious. Doing a study that confirms it doesn't change that it's obvious and your study is just an excuse to fill some pages under the guise of 'news'.
paintball
Let me tell you a story. There was a game I had been long anticipating, a game I was so psyched about. I was, I must admit, young and naive, I had never played a truly bad game before, I didn't think 'bad' games existed, other then the Edutainment games. I got the game the day it came out. The game was Superman 64. I have never bought a game (other then sequels and launch titles) without reading a review ever since.
I don't buy many games (mostly I rent because most games are worth the $$$). If I REALLY want a game (New Mario, Guitar Hero) I'll just buy it ("Damn the torpedoes, full speed ahead!"). But if I want a game but I'm not that set, I'll wait for the reviews. If the review seems positive (no major flaws, still sounds fun) then I tend to buy the game. If not (bugs, doesn't sound fun, etc) then I don't.
I don't buy games just because of reviews though.
For the record, I get most of my reviews from X-Play (which I watch all new episodes of). I also check specific games on GameSpot and IGN.
A demo is FAR more likely to change my opinion of a game than a review, but I like having reviews around to see what is happening with games I don't tend to play (sports games) and to save me from buying a piece of junk that won't be that fun.
However, I can tell you that my little sister and her friends (all about 14) don't read reviews at all. They are just mindless consumers. "I liked spider man/the hulk, I'll get that game". "That football game looks like it might be fun". Mostly we rent (so many games are terrible) but they don't even consider reviews before renting (not their money, I guess).
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
Im old enough to remember both Siskel and Ebert totally throttling Star Wars, reviewers cannot review without bias nor should they be expected to. Everyone has different tastes and will always use what they like, are familiar with and are comfortable with as their benchmark. A madden fan for instance isnt going to review a non madden football game as highly simply because the controls and feel of the game are different. Games that are concidered complete crap to many will still find a market that loves them. My son for instance cant get enough of the Fantastic Four game that came out this past year...its complete junk but he gets to play "thing" and bash stuff so he thinks its great. The bottom line is there is no such thing as a barometer for everyones tastes, we all like different things and will judge them accordingly.
For about 2-3 years now i've been waiting for reviews of games to come out (it only takes ~3 days), because i've had some bad experiences. I used to buy the Official PS1/2 Magazine, and loosely base my purchases on what i saw there. Then i started getting good games that were too short, or crappy long games, or both (of bad), or just insatisfactory in their whole.
These days i usually read reviews on IGN, GameSpy, Gamespot and a copuple more here and there, and specifically look for comments regarding length, replayability/extra content, character control quality (and this part greatly concerns me when i think of the Wii) and whatever gets some aditional attention by the reviewer.
However there are some games that i must get no matter what (or almost), because i'm either a huge fan of the series or because reviews and previews bring the game to my attention for some particular reason. For instances, while i know that all Resident Evils (Biohazards) have tanklike character controls (except RE4), i am too fanatical about them not to get them (all for the GameCube), especially since i either did not get their PS# versions or had some old/cheesy pirate copies for the PS1. Games that pop up out of nowhere are Fear Effect, Assassin's Creed, Theseis (canceled?), Alan Wake, God of War, Okami, and some more.
Resident Evil, Silent Hill, Metal Gear, Tomb Raider (yeah i know), Final Fantasy (yeah i know too), Project Zero (Fatal Frame) and a few more are quintessencial for my survival. Some studios also have my undivided attention, like Remedy (no Alan Wake for the PS3, shame on yous), Take2 or Activision.
And by the way, anyone knows of any good gaming websites? While i read IGN and GameSpot, i know these have some reputation stains (is owned by FOX Noowbs and GSpot apparently has biased reviews, based on advertising bought from them.
I have to admit I was skeptical at first, but it is true. The actual report shows graphings and such on games and to be honest games just don't bust out. Great games with high ranks do sell more, but not every game with high ranks will sell a lot, and some will undersell the curve.
There's many more factors, the biggest being name brand recognition or movie tie in. Cars can be as bad as hell but people will buy it for their kids. The flip flop is the company name. I'll buy most anything from nintendo over something from EA because I find nintendo games fun and EA has a good amount of medicore "name brands". Enter the matrix sold well because of the name, while shadows of colleussus floundered because no one knew it.
Beyond that though I think the biggest flaw with the study is they skip the one thing that reviews follow... the previews. I can get hyped on a game or not by just the preview about the game. Just cause hyped me but then I played the demo and it was awful, if there was no demo I might have picked it up. If a magazine talks about features in a game that starts buzz on message boards and such. If the magazine talks about ideas developers have they can help gamers get a good vision of the game.
All this stuff is done monthes before the game is out, each step creates name recognition. Sequels help this, advertising and so on. The simple fact that the report misses is that even if the game is panned or praised what's going to get the game's sales is the fact that people know about the game, and there's a reason to get it. However at the same time no one wants to produce a crap game, so rankings will remain important to the industry.
And if you can't find the article, the games that just flat out smash the curve at least on the PC is GTA and Madden, no suprise.
While I'd like to see more detailed data, the conclusion seems plausible enough. But I'm curious about the long term effect of bad reviews on a franchise. For example, games like Driv3r and Enter the Matrix may have sold really well despite bad reviews, but if a game is bad people aren't going to buy the sequel, the frachise has been badly tarnished, and I don't think Driver 4 or Path of Neo sold that well (I may be wrong though). Tomb Raider is another franchise that used to be really huge but got tarnished by bad sequels. Acclaim Entertainment killed themselves with too many bad games.
Sorry last line is "at least on the PS2"
Ebert loved Star Wars.
which is what the study concludes, and what should be pretty obvious if you read lots of game reviews. Game ratings are basically random numbers between 8 and 10, and where it falls in that range seems to be largely divorced from the content of the review. How many times have you read a review that said something like "the gameplay was fun for the first few levels, but quickly became monotonous and boring" but gave the game a 9.5, or one that said "despite a few minor flaws, this game is all around a lot of fun" and gave it an 8? When I'm out looking for a game, I think I'm going to weigh "monotonous gameplay" a lot more than "Overall Score: 9, Excellent!".
I can understand them using the game rating, as it's the only obvious number you can apply to a game review and do correlations with. However just having a number doesn't mean it actually represents something, and I'm not surprised that game sales don't correlate well with a number that is basically pulled from the reviewer's ass.
The enemies of Democracy are
Heck, Ebert gave three and a half stars to Episode One. The man has been a sucker for Star Wars from day one.
I have seen the future, and it is inconvenient.
Reviews are nice, but the best way to decide what's good is to try them first. Download them via BitTorrent and play them for a while first.
Yup, that's what I do before I "buy" any of my games. Yarrr!
Property is theft.
I, and many people, had the exact same experience with the most recent Harry Potter game (Goblet of Fire). The first three games were great. I bought the 4th game without thinking about it, assuming it too would be good. But it was godawful. On some review sites, the consumer reviews were 200 to 1 thinking the game sucked.
The most oft-made comment on those review sites? "Never again will I buy a game without first reading the reviews."
There is an old US saying: "Fool me once, shame on you. Fool me twice, shame on me."
Never again. It only takes being burned once.
Perhaps I'm in the minority, but I always check the reviews before buying a game, and that silly little number does impact me. I am much more likely to buy a game with a score above 9 from the magazine and a similar user score than a game that scores below 9 in both categories. I tend to rely more on the player review numbers though, and ultimately use them to make my final decision. Actual player comments are also very nice. The whiners and the fanboys are easy to filter out, and a lot of people leave good comments about core game features that will have an impact on me. If I see the same complaints or cheers about an aspect of the game from several people, I tend to take those opinions very seriously. Likewise if I read a dozen comments about the game crashing and two comments about how the game never crashes and is awesome, then I'm going to assume the game has some issues and I might wait for the first round of patches. (I guess I'm mostly referring to gamespot, because it happened to be the game site that I found first however long ago and I don't feel like following multiple sites)
Here's an embarassing thing to admit: if I've been looking forward to a game and it gets an 8.9, I actually feel disappointed and I might take a few weeks to make a decision. But if that game gets a 9.0, I'm much more inclined to run out and grab it. Yeah, it's dumb. It's just like I know that $49.99 is only a penny away from $50, but it feels a lot more like $40.
One thing I realized as I was typing this is that while I'll look at the points the reviewer gives a game, I rarely ever read a professional review. They're simply too wordy and glossy. I'd rather just read some player comments about how "it sucks ass that you can't ride the donkey in multiplayer mode" than to try to discern these tidbits from the professional review.
Sales are, at least in my experience, based on three factors:
Name of Game/Company
Hype
Popularity of genre
In that order.
There's quite a lot of really awesome games out there that didn't get any attention. Why? They lack at least 2 of the three factors.
Imagine something is labeled "Command & Conquer Final Chapter". Will it sell? You bet it will! It could be the last crappy ripoff, a repackaged C&C1 with new graphics for example (no, you don't need to tell me "it's already been done", I know, thank you very much). It will sell. It has the name. Imagine Halo 3. Quake 5. NHL 2006. Diablo 3. Does/would/will it sell? Yes. Why? Name.
Same with "games to movies". Fortunately, at least some movie studios do care these days who buys the license for the name to slap on a game. It used to be a surefire way to say a game sucks donkey bits if it had the name of a current movie. But they always were bestsellers. Even the crappiest of the lot.
Then there is hype. DNF will sell. No matter when it comes out and no matter if it is up to par. You could offer an empty box and it would sell. Other games have the "new feature" hype, whether that feature actually matters or not. Max Payne was hyped as "revolutionary" in the genre of shooters for its bullet time and the actually rendered bullets. Whether it really was, I leave to the reader.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Of course a football game not based on the Madden engine is going to play completely differently. If it's not Madden, then by EA's exclusive license, it's not NFL, and if it's not NFL, it's more likely association football than American football.
More recently, Ebert game Napoleon Dynamite like 1.5 out of 4 stars. I think pretty much everyone agrees that the movie is better than that. From reading the review, it seemed to me that Ebert basically missed the entire point of it.
Comment of the year
That actually proves the point even more...my kids love Napoleon Dynamite but I think 1 1/2 stars was being generous. I remember leaving the theatre thinking that they sure dont make teen movies like they used to. Different strokes I guess...
I play a lot of games. I've found that the website I look to for reviews has only let me down twice in several years and hundreds of games/reviews.
Do scores influence sales? I'm not the one who did the study.
I have found though that the game reviews that I read, including their scores, influence my purchases to a very great extent.
Well, if game reviews were estimating sales, this might make sense.
However, game reviews review the enjoyment and playability of the game. Further, they are often framed, for example: "I give enchant arms a 10/10 if you like these types of RPG's, otherwise, avoid.".
This "study" sucks.
Most good games are sold from word of mouth but I find it absolutely scary when crap like 50 Cent: Bulletproof become a million seller while critically praised games like Eternal Darkness are left on shelves.
My other sig is a
and their opinion is no more valid, or even less valid as I find that any game that is genuinely crap from a major advertisement purchaser doesn't get the shafting it deserves, yet a game of similar quality from a no-name developer/publisher usually gets raked over the coals.
The thing about Napoleon Dynamite is that it was supposed to my type of movie. I love quirky independent films with an odd sense of humor. When I finally saw Napoleon Dyanamite, I couldn't understand what was so great about it. Granted there were some scenes that I thought were funny, but hearing "Vote for Pedro" and "Idiot!" over and over by people who normally don't like those types of films seemed only to annoy.
"Our opponent is an alien starship packed with atomic bombs. We have a protractor."
You need to look at factors such as advertising dollars when looking at a review objectively, as well. As an example, a friend of mine was managing a gaming store when one of the game company reps that was showing him their latest catalogue got a call on his mobile from one of the local gaming mags. The guy from the magazine informed him of their latest review of one of their games, and the score was not too flattering, something like 5/10. The rep said "Nah, pull the review then", which caused the mag to back-pedal and make a deal to give the game 7/10 but use the same text of the review. Often you see big release games which don't live up to expectations in the public, one obvious example is Dues Ex 2. That game was an absolute turkey yet when it was released I saw several reviews calling it a classic with higher than 9/10 scores.
Blessed are the 1337, for they shall pwn the earth.
with console games, this is by far the best way to decide if a game is worthy of purchase. if i rent a game and can see myself spending an additional month playing it at an hour/day average, then i'll buy it (eg gran turismo 3/4, final fantasy anything).
pc games are a slightly different animal. there was a game that was recently released that i never would have discovered had the pirating community not brought it to my attention. needless to say, i have forgotten the name because it didn't interest me. this just goes to show that piracy can give plenty of publicity. in fact, if there is no demo available for something that looks neat, i will download the whole thing; upon playing it, i'll buy it... if it's up to my standards: warhammer 40k is superb, titan quest has promise, quake wars shows promise, etc. i WILL NOT, however, ever buy a game the first month it's out, i made that mistake with warcraft 1 in high school; now i wait at least a few weeks to see if the price drops $10-20. don't misunderstand me, i believe that if i really like a game, the company should see something from me, but when my income is that of a sub-poverty college student...
that being said, i stand firmly behind not buying a game, or any expensive software, until trying it out. i think that software companies would realize the intelligence in offering trials of their products; if i were to purchase a game for $40-60 and it sucked... i wouldn't even look at products from that company ever again. had i played the demo, then things would be different.
And yes, completists like me, especially of some well-known franchises like Might and Magic, C&C, will always buy the sequels no matter how horrible the reviews are. In many cases, even no matter how horrible the actual games are. Even if some of them are buggy and unplayable, they'll always be there on my shelf.
One of the posts here said DNF will sell pretty well because of coverage. Heck, given its extreme coverage, if it actually comes out, I predict it will be one of the best-selling games of all time. (Many will play it, and those who don't, will still buy it for their collection.)
I hear you. Just replace "Superman 64" with "Tomb Raider: Angel of Darkness".
The first and last game I ever pre-ordered. Horribly scarred for life. And just for perspective, I'm 35 years old.
How many copies of a game get traded in as the days/weeks/months pass since the release date. As that is more of an indicator of wether the game sucks or is worth holding onto, IMO. Few copies of the game in the 2nd hand section gets my attention and gets me asking questions to the EB staff.
My buying habits are erratic, sometime I go by reviews, or word of mouth, or a continuing franchise that I've bought previous versions of. But wether I keep hold of a game or trade it in is based on wether I enjoyed playing the game.
See my art -> http://herbevore.deviantart.com
This is the most important reason why a game sells. Buy and download will change this and reviews will become more important, from pro or players. Our last racing game has the exact same Metacritic score as NFS Most Wanted and we have not reached the 20K units sold in North America.
I'm always suspicious on game review-sites that they enhance the rating of games from money-spending advertisers.
Smoke me a kipper, I'll be back for breakfast!
Ditto here, only the game was Phantasy Star 2 for the Genesis. Ugh.
Circumcision is child abuse.
Game reviews don't affect sales, so there's really no point in writing them.
Sincerely,
Sucky Game Producers, Inc.
Modern copyright is theft of culture from everyone and it retards the progress of the useful arts and sciences.
I'm sure everyone would agree that Half Life was a terrible game. I seem to remember several examples in there of what you describe - if anything, despite the silliness, it enhanced the cinematic feel and the coolness factor of the title.
That was my problem with it as well. It would have made a great short, but it evidently has quite a following..i was modded down simply for stating that I didnt care for it :)
anyone can play a game, get scammed or fragged because of lagging, then go write a review saying it's the worst game in the history of gaming. I like the other type of review that nuetrally describes how the game works, its features, its performance (glitches and speed), and any major problems with it. Reading any other type of review that's completely opinionated one way or another just doesn't tell the full story. However, nothing beats game demos.
Is it just me or is it not going to upgrade to Vista in here?
What, pray tell, was the point of that movie? It was mildly amusing at points, but I don't see why it gathered such a rabid fanbase.
There were many-a Gamecube and DS game that I didn't buy (yet anyway) solely because of a (or many) sub-par review(s)... My standards are pretty high and although Nintendo has (and still does) put out tons of games I love, they put out around as many that just don't seem to cut it for me.
I wish people would pay more attention to reviews. That way game companies would think twice before releasing unfinished crap. Luckily, I waited for the review of Lego Star Wars II for DS. IGN saved me $35!
Publishers can release these unfinished products and then don't have to support them or fix them. If I buy a product from Walmart that is broken or missing pieces, I simply take it back. But with games, you can't do that b/c they are afraid of piracy.
They can screw us out of our hard-earned money, but if I copy some of their games, then they want to complain about me ripping them off? There needs to be some actions we can take for companies releasing stuff that is obviously defective.
You need to watch out on sequals when some new company comes into the picture. No matter how much you love some game or some series, and no matter how fantastic the new sequel sounds during the development process, don't get suckered into pre-ordering when someone new is running the development.
One gigantic red flag is if they ever say anything to the general effect that the new release will appeal to a "Wider Audience". That is the kiss-of-death for a sequal to beloved game. The more you love a game, the more you should be deathly afraid when you hear anything like that. It almost certainly means that they are abandoning the existing fans that made the game so sucessful in the first place, exterminating the very elements that made existing fans love it so much.
-
- - You can't take something off the Internet! That's like trying to take pee out of a swimming pool.