Slashdot Mirror
Review Scores the "Least Important Factor" When Buying Games
A recent report from a games industry analyst suggests that among a number of factors leading to the purchase of a video game — such as price, graphics and word of mouth — the game's aggregated review score is the least important measure. Analyst Doug Creutz said, "We believe that while Metacritic scores may be correlated to game quality and word of mouth, and thus somewhat predictive of title performance, they are unlikely in and of themselves to drive or undermine the success of a game. We note this, in part, because of persistent rumors that some game developers have been jawboning game reviewers into giving their games higher critical review scores. We believe the publishers are better served by spending their time on the development process than by 'grade-grubbing' after the fact."
maybe it's the other way around... You only have to buy a sucky game *once* based on a raving review to *never* trust those reviews again. While your friends can comment properly on the game without some obscure metric like '8/10 overall'.
If a magazine or website is really scoring out of 10 or out of 100, then we ought to see some 1's and 2's.
But we don't do we ?
The researchers would find more utility in measuring the correlation between ad spend and score.
Anyone think these two variables don't correlate strongly ?
Dominic Connor,Quant Headhunter
And why? Because the grade-grubbing means that as of about 10-15 years ago, reviews are nothing more than adverts, and ratings are nothing more than auctions to the highest bidder.
I've *never* bought any game because of a review. Not even back when they were a bit more honest (e.g. in the Spectrum days, it was very common to see sub-50% and even sub-10% scores of games, some of them were even immortalised in things like a "crap games collection"). Game preference is completely subjective and neither words nor pictures can convey how a game operates.
But it's not just games that suffer from the problem - I know someone who buys cameras, cars, all manner of electrical goods etc. on the basis of the Which? review. I have seriously watched them buy something that costs a month's wages just because the Which? magazine said it was the best, only for them to discover that all the things *I* said about the brand / device / features etc. were true and it was useless to them. What was even more annoying is that they asked my advice every time about PC's and electrical goods, then completely ignored it, bought what the Which? review recommended, then complained and expected me to provide support for the thing they just bought.
I read reviews as entertainment. If I want to know about a game, I might read the review of it to pass the time and introduce me to the *suggested* features that it may have. But I would never use them as a basis for a purchase... that's why you let other dummies buy it first and then hear first-hand from them after a month if they are still playing it and enjoying it.
When I personally buy a game, I look at gamefaqs user reviews instead of Metacritic. When looking at the main page of a game on gamefaqs, the first two averaged review numbers are exceptionally useful to me. They seem to give a very strong feel of what the general reaction to a game is - anything under a 7 is probably not worth my money. Also, user reviewers seem to me to play the games more thoroughly than someone who does reviews for a job, and game depth/replayability is a big point for me. Although, if I think about it, I generally buy games for Nintendo DS - price is pretty uniform and graphics can only get so good. In order to look up the game at all I had to have heard about it from my friends or some sites, so my experience doesn't really contradict his research at all.
So, while there certainly is correlation between the review score and purchase numbers, there's very limited (if any) causation? With the immense integrity the game-review professionals command, who'da thunk it?
It all boils down to trust (or more specifically lack of it) on the Game review sites.
I'm not overly surprised that people don't base their buying decisions primarily on the review scores from game-sites. In most sites I've seen one or all of the following:
Personally, I usually wait a while after the game is out and then go check user reviews. If your discount the "100%, great thing since sliced bread" ones (which come from fanboys) you'll usually be able to get a good picture of all the above mentioned points that the game sites miss (bugs, long-term (re)playability, intrusive DRM, hands-on-beyond-hype experience)
If you ask people if they are willing to pay more for quality 90% will answer yes. However when the moment supreme is there to purchase for example a new notebook 80% will go for the cheapest and don't care about long term stuff like quality. I think there is a good chance this survey works the same; People SAY they are not influenced by reviews because 'Hey, I'm an original, I don't let anybody influence me'.
Luckily with games, there are free demos available on every major torrent site.
Having said that, I do realise that this applies less to console owners, who are in a more difficult position because they generally can't test games before purchasing it, meaning they will have to live with a lower signal-to-noise ratio. (But then, they were the ones who chose to invest in a closed platform.)
Anyway, I'm fairly happy that most games are available for free testing (I'm usually not really in a rush to get any particular game), because - looking back - I can't really say that I found very many games that would've been worth my money if I had bought them (not even when they were sold at half 'list' price).. For the last couple of years the list would pretty much be limited to Portal, EU3, World in Conflict, Vampire: Bloodlines and Civ4 (and Arkham Asylum was OK too, just not at the current prices). Not a very long list, I might note.
In all, I would suggest people don't get consoles, as too much bargaining power is taken away from you in getting one, and too many games just aren't worth wasting money on.
I read both media and user reviews, though mainly for games I'm thinking of buying because they're on sale on Steam or the like. If the reviews aren't good, I'm less likely to buy the game. The other way though, review causing me to buy a game, is less likely, simply because I don't tend to read reviews if I don't yet have an interest in the game.
I was surprised that advertising visuals are the next most important factor after genre, playing a previous version and price.
Depending on how much the parent company spends in ads on the site/magazine, the score will be inflated higher. The few times that an editor or reviewer really did stand up and score a bad game as such, they were immediately fired by the advertising department. There was a big scandal a few years back when the editor actually spoke out against one such firing... I am too tired to look it up myself right now.
Anyway, reviews anymore from the "gaming press" are total garbage due to this mechanic. The ads in the magazine are more important to the company than the reviews themselves. When was the last time you saw an EA game get a 1 out of 10.... And trust me, there are many deserving candidates, like the yearly sports rehash which change nothing in the game, just which player is on which team. Or Race Driver Grid, or Darkar 2009, or Rally Stars.... The magazines would just not post a review of a game when it gets bad because they don't want to potentially lose their ads from the publisher...
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
I have three tiers of deciding on purchase of the game.
1. I read the review about what the game contains. I thoroughly ignore any "positive personal thoughts" about the game as marketing fluff. The negative ones do add to the value of the review but aren't all that important. I just read what is the concept of the game, and whether it is anything original, with potential - a good idea. If the review talks loads about graphics and sound and development time and prior franchise, even in total superlatives, it means the game is junk. A reviewer would concentrate on the really good points if it had any.
2.I check some Internet fora to see what people complain about. If there is a number of complaints about the same thing, it may turn me away again. The thing being "awful execution of the wonderful idea" is one of possible choices.
3. Then I grab the game off a torrent. After I'm through with it, I look back at how it felt. The only deciding factor is "I enjoyed it". Yeah, I enjoyed Stalker: Clear Sky, despite hopeless story, dull ending and reuse of content. I enjoyed Oblivion despite being dumbed down to knees level of Morrowind.
If the game passes the three tiers of classification, I buy it.
45 5F E1 04 22 CA 29 C4 93 3F 95 05 2B 79 2A B2
I read the professional reviews only to get a detailed overview of what to expect. The buying decision is made after reading both the most positive and most negative user reviews - this usually gives you a fairly accurate idea of what to expect.
In the past I have bought too many games on hype alone, only to be disappointed because some aspect of the game was not (as good) as I expected. By reading the positive and negative user reviews you can be almost certain to learn about the quirkiest details.
I used to write Amazon reviews - you know, the bit before the buyers reviews but after the manufacturers descriptions? I was impressed when I signed up that made it clear I could slate a product if it really wasn't any good. Their only stipulation was that I should suggest another product on their site that was know to be better. Seemed fair enough to me. I stopped some years ago but if that policy is still in effect, it owuld add some weight to their value IMO. This was the .co.uk - the .com had diffferent reviews and possibly different criteria.
I want a list of atrocities done in your name - Recoil
I buy games when I've enjoyed playing the demo. If there isn't a demo available, I don't buy it.
Game world not designed to allow for demo-style play? Rubbish. You can sandbox an area of a GTA map, limit Dragon Age: Origins to one town, make level caps to prevent access to higher level play... It's just laziness.
Finally had enough. Come see us over at https://soylentnews.org/
LOL reviews are one of the most important factors I base my purchases on. And here in the UK I go by Edge Magazine which I have been reading for 16 years.
I can trust their reviews. Anything they give a 9/10 is a must buy (if you like the genre) and the rare 10/10 are no brainers. There have only been a handful of 10/10 in the magazines long history, including one this month: Bayonetta.
Edge Magazine: http://www.edge-online.com/magazine
Owning a Wii, there are a bunch of really fantastic games and a whole lot of dross (though this is true to an extent on all platforms the Wii seems to suffer more than most). Without reviews, there would be no way to sift through all the chaff and find the good titles. Though the fact that terrible software continues to sell by the bucket load on the Wii suggests that the average joe probably doesn't read reviews at all.
Since it has been disclosed that good reviews are being bought (you are allowed to conduct a test early if you guarantee at least a score of X), you cannot rely on reviews in any way. Currently the publisher try to sell any game as much as possible on the first three days before the "real" reviews hit.
Since the publishers are so keen on getting 8/10 reviews, lets replace the stars with a scoring system that just gives them more of what they want.
A good game gets one (8/10).
A mediocre game gets 3: (8/10)(8/10)(8/10)
And a terrible game gets a whopping 10: (8/10)(8/10)(8/10)(8/10)(8/10)(8/10)(8/10)(8/10)(8/10)(8/10) ... the publishers get what they want, and anyone with a calculator to hand knows what we really mean
Because at Lake Wobegone Software Publishing, all the secretaries are efficient, all the managers intelligent, and all the developers are above average.
--
BMO
Be extra suspicious of games that embargo reviews, or allow just a handful of "exclusive" reviews to break the embargo. More often than not those reviews have been paid for in one way or another. Just like with other kinds of media there is usually a very good reason that publishers don't want you to know upfront what a game is like - because the product sucks.
These days, I usually assume that if a game gets released, it's at least not half bad. At least on console side, there's some quality control in place. If the game gets released with really, really, really damning flaws, I expect every reviewer to whine about it. Fortunately, here the reviewers are at least trying to keep up to some journalistic standard and aren't lying all the time.
I usually just read the reviews to see a few things: 1) did the game impress the reviewer whom I know is a fan of the genre and has seen many comparable examples? 2) just how much bullshit regarding the pre-release hype did the reviewer actually uncover, and would this affect my own expectations? Are the promised features that I was excited about still there? 3) Are there any big problems with the game that I should be aware of?
Score is ultimately an useless metric that depends on too many things. If our local prominent game mag gives the game 70 or up, it's usually a sign that the game is going to be at least somewhat fun and worth trying.
XBLArcade and PSN do have demos of games.
Unless by demo you mean, pirate.
Non impediti ratione cogitationus.
I never read reviews as reviews.
A review to me contains two or three things: advertising for the game, some more details about the gameplay that might be missing from the full-page graphic laden ad or TV spot, and possible it might compare the game to relevant reference points (other games, other relevant media, etc.). If I want an opinion beyond those bits of factual information I will look elsewhere - within days of a game being release there will be many opinions out there to pick from. Admittedly you have to assess each as there will be a mix of astro-turfers and particular-company-haters-who-don't-even-know-the-product-they-are-bashing but if you can find a good active discussion or two you can usually get a good gauge of the state of play.
You also have to remember that the person writing the review isn't you. Your opinions may differ greatly when you actually get hold of the game, so try to read the facts upon which the opinion is based more than the opinion itself. Have a look at Zero Punctuation's reviews - if nothing else he rants entertainingly (IMO), and while the reviews are slanted towards the negative (intentionally so) he will mix in what good points he finds. For instance the BioShock review which if you don't pay attention at the beginning (where he lists the games major good points) you'd mistake a "good and very pretty, but not close to the hype" review fro a complete slating. When he does say something nice you know he means it (as being nasty is what gets him his viewers and therefore his paycheck). Of course I disagree with some of his views, because as stated above he is not me - I liked DeadSpace a lot more than he did (the trick being not to expect too much depth in what is essentially an interactive action flick) but didn't much like PainKiller when I tried it on a friends machine (though a lot of that is based on "what it my kind of game" and "what mood I was in" as much as the game itself.
In summary: you are never going to get a true impression of how much you will like a game from any one review or collection of reviews, so stop trying. "Out of 10" and similar scores are even less (far, far, far less) meaningful.
Caveat: I buy at most on or two of major games most years, and sometimes those are last year's games or earlier (which are now at little as 25% original full price) and occasionally pay for a good indie "casual" distraction, so I'm not really the industries key target audience.
Except you just illustrate the problem: something that's just "good enough" (which really just means "mediocre") gets an 8 out of 10. I'm sorry, but in a perfectly linear scale, "mediocre" would mean a 5. That's the kind of a number you could punch in a formula and get a correlation or anything else.
Plus, if it were just a case of a honest review and the bad ones being already cancelled, the results would look much like the right half of a bell curve. You know, the curve with the below average ones removed. For virtually any sitze out there, it doesn't. It looks like a bell curve centered on 8 or 9, and which pretty much starts at 6 or 7. Sorry, that's not a case of the bad ones being already removed, that's a clear sign of an offset scale. It's what you get when the occasional "something that's truly bad" means you get to give a 5 or a 6, not a 1 or 2.
And then there is the occasional reviewer whose curve looks like two spikes. The kind who churns 90% to 99% scores all year long, and then occasionally picks up some 10 year old freeware game so he can give _something_ a 5% score and fix his street cred. Or publishes a yearly smack-talking "top 10 worst games of all time" -- conveniently all 20 years old and from publishers which are no longer in business -- just to show that he's that unbiased and can give a low score too.
But again, that's not being unbiased and fair at all, it's just trying to compensate one crap (or dishonest) job with another one skewed in the other direction. If it were a real fair and unbiased and non-skewed job, you'd get one bell curve centered in the right place, not two spikes centered near the extremes of the scale.
A polar bear is a cartesian bear after a coordinate transform.
It did imply that they never even bought the games that were worth something.
Everybody seems to think I'm lazy I don't mind, I think they're crazy
Only a minority of gamers read reviews, get fact before buy. Theres a big group of people that just buy a game based on the box. And this groups is probably the bigger.
I am tempted to say that very few people that buy games are gamers (!). I mean, gamers as people that have gaming as his hobby.
-Woof woof woof!
If a magazine or website is really scoring out of 10 or out of 100, then we ought to see some 1's and 2's. But we don't do we ?
My wife and I were having this same discussion the other day. I was going through some reviews of games that just came out, comparing them to older games in the series. When I spotted one and mentioned the poor review, my wife asked what was the score. "6 out of 10". She was confused that a bad game got such a high score.
I guess I've been reading these reviews for so long, I didn't think of it anymore. 10/10 is awesome, 9/10 is great, 8/10 is good, 7/10 is okay, 6/10 is poor, 5/10 and lower is terrible.
"But when was the last time you saw a 5/10?" I honestly didn't know. Even the big-name movie tie-ins that we all know to be awful will somehow manage to score "6.5". I actually had to go look up some reviews to find lower than "6" - but they are out there.
I've started to view the "out of 10" or "out of 100" scores like the old A-F grading system we used in school. A is 9/10 or 10/10 ("A+"), B is 8/10, C is 7/10, D is 6/10 ... F is 5/10 or lower. It's not ideal to view games this way, but it makes sense of the review scores.
Nonsense. There are plenty of good games that are worth your money on game consoles, and it's not hard to make out what's good judging from word of mouth/keyboard and reading reviews (fuck the scores, read the review!).
That it's a closed platform doesn't matter at all to the consumers. We don't have to upgrade our hardware, deal with technical issues, and we don't get invasive DRM.
Having said that, there are free demos for many games on PSN and Xbox Live!.
We are a smart bunch, but so dumb some times. I say the average I.Q is what 130+ and yet we are so egotistical that we think advertisement does not work on us.
Of course the dumb rating system will not work on us, but they get us other ways.
I will use the iPhone as the best example of this, we ignore the silly ads. you know the lame ones that are comply unrealistic. They get to us through nerd speak,specs, and its almost like we get off on it. Then we got out and buy it. Knowing the limitations and the bad points.
No mater how bad because it is cool tech. Since everyone in the tech crow has one, the dumb adverts now work on the average customer.
Gaming is still a young market they will grow up, but for now this still works.
Compare two reviews one for P.C games and one for console games and the focus be will completely different, yet they are selling the same thing.
Adverts and reviews are make for the target audience that will most likely buy the product.
You can tell how well a market is doing by it’s adverts, Razors are still sold to you like your twelve and it still works.
Sent form my iPhone.
They are probably the MOST important factor, aside from series which I love dearly, eg Halo, Metal Gear, Gran Turismo. Before I get any game, I always check the reviews on two or three different sites. I even shop for games that way. PS2 -> RPG -> Browse All -> Sort by Review Score.
return ((score - (score_max/2)) * 2);
50 = 0.
60 = 20.
75 = 50.
95 = 90.
And so on.
As an avid gamer with multiple platforms (360, DS, PC) im usually buying 3-6 games per month depending on releases.
I use the metacritic websites aggregated score to get a feel for what people are thinking.
if i see something compelling i check for video footage to see if i like the art direction, usually on gametrailers or similar site.
Then i read some reviews OR i speak to people i know have played it or have worked on it.
Lastly if its a sequel that i have played before i replay it or use my memories of it to decided wether or not to buy.
tl;dr
my decision on buying games is based on
10% - aggregated review scores
30% - video footage
40% - word of mouth/in depth review
20% - previous history with game franchise or developer
I would give everything i own for a little bit more.
Not to mention the renting option - and a few places I've seen will generally discount a game by the price of the rental if you subsequently buy it.
Lucky you. Where I live, video game renting has been outlawed. :(
Actually, the vast majority of the bugs I've encountered in various games were just plain old fashioned bugs in their code, and had nothing whatsoever to do with the drivers or hardware configuration. They were script bugs (e.g., a dialog option remaining active when it should be gone), pathing bugs, collision or physics bugs, balance problems, AI bugs, interface problems, the occasional race condition, memory leaks, etc.
E.g., if you think that any hardware or drivers could stop WoW from having bugged enemies that evade every single attack, I have some logging rights in Sahara to sell.
Even when installing a different version of the drivers solved anything, it is often just a case of moving the timings a bit so the race condition hits you at a different time, or some call with wrong parameters that only incidentally doesn't crash a particular driver version, or various such. In short, still program bugs, rather than actual driver bugs.
That's really what annoys me about the PC. Too often that variability in drivers or hardware is used just as a blanket excuse to do a half-arsed QA job before releasing, or as a blanket excuse for reviewers to not mention the bugs.
A polar bear is a cartesian bear after a coordinate transform.
I don't know which of the two is more sad:
1. People don't bother informing themselves about the quality of games they might purchase.
2. Game reviewers are sellouts.
... if its obviously a piece of crap, the reviewer isn't even going to bother with it.
I think 6/10 is the low limit for anything to get reviewed.
We just released a game that has been getting some really nice reviews, and have spent very little on promotion (we don't have any money). Everybody has different tastes. Perhaps your tastes just happen to clash with those of the reviewers you read.
So you'll grab a torrent of a hacked up game with viruses, play it all the way through all the while bitching on the forums that it runs like crap, righteously deny the devs payment, and go on to your next theft.
Then you'll bitch because nobody develops for the PC platform anymore.
As a dev and a pc gamer, all I have to say to you is "Fuck off and die, you fucking thief."
It's pathetic that you'll commit a felony for 60 bucks.
Except the only current-gen system where piracy is impractical is the PS3. The 360, the Wii, the DS, and the PSP are all exceptionally pirate-friendly. Sorry, free-trial friendly. I also notice that the 'free trial' of MW2 was available for 360 quite a while before the PC version was available, and as for Assassin's Creed 2... Well these aren't games I'm interested in, but getting the 'free trials' for any games on these platforms is no more an issue than it is for PC.
Metacritic is great for finding out the shortcomings in a game. Just read the review snippets below 80%.
I seem to remember us hearing about "research" like this every six months or so. Hasn't it been done to death? The majority of people buying products of *any* kind, not just games, don't bother reading reviews. It's simply not a significant factor. X-Blades was rated terribly against any other game that came out at that time, yet we got it on Goozex.com once the price came down a bit and have a blast with it. You know why it was panned? Because it wasn't one of the "zomfg big game hype of the yearz!"
This doesn't even get into the fact that most review sites I've seen through metacritic never even bother to really play the titles anymore. I started to notice IGN, 1up, and the other "big time" review sites were starting to complain about odd things...like "no discernable way to change difficulty level in Fallout 3" in I believe IGN's review (might've been TeamXBox, but it was a big site). Right on the MAIN MENU under OPTIONS and even during the game you can at any time click "Options -> Difficulty" and change to suit you. You cannot tell me with a straight face that a professional reviewer of games who's done dozens, if not hundreds of reviews beforehand possibly missed something that simple that's been with us since the PS2 days. There are quite a few other examples where it's clear the reviewer simply did *not* play through the game (like how no reviewers bothered mentioned the utter split-screen FAIL on Borderlands with the menus in-game. Play it and you'll find out *immediately* what I'm talking about, they aren't resized so you can't navigate menus in-game).
People buy what their friends are going to buy, what hype tells them to buy, and what the box art tells them to buy. Just like movies, books, cars, houses, and everything else. Personally I think the whole review system is broken at the moment (working on my own, just need to get off my ass and buy a site for it) but even if it wasn't this just isn't news and hasn't been for some time.
"Just a fox, a whisper."
Game demos are readily available for consoles since the time of PS1; over a decade.
Furthermore, the signal to noise ratio was actually kept high also, among other things, closed market (for devs) - with barriers of entry it was sensible to try harder. It was also sensible to focus more on gameplay, since you knew the rough limits in GFX. Most importantly there were strong forces at work - console manufacturers - that promoted development of very good games.
And...console games have generally better resale value, so stumbling on something not very good wasn't that big of a problem.
But hey, keep going if you want to feel more elitist, more wise than those stupid folks who were also buying consoles...
One that hath name thou can not otter
Sadly this is the sorry state of the game review factions out there.....anyone see any similarities between gaming reviews and movie reviews? Soon the tired old numeric scores will be gone and everything will be two thumbs up from two guys who couldn't even make it past the first race in the Nintendo classic Excitebike. And shame on all the reviewers out there who hand out good reviews in exchange for advertising money or invitations to studios and events...I know this because I used to work for a site that adopted these very same tactics when writing up a review. Needless to say I was out of there shortly after I found out, and yes they are on Metacritic. RULE OF THUMB.....RENT FIRST.....BUY LATER....because buying a sixty dollar piece of crap is worse than renting a ten dollar piece of crap