Slashdot Mirror
Review Scores the "Least Important Factor" When Buying Games
A recent report from a games industry analyst suggests that among a number of factors leading to the purchase of a video game — such as price, graphics and word of mouth — the game's aggregated review score is the least important measure. Analyst Doug Creutz said, "We believe that while Metacritic scores may be correlated to game quality and word of mouth, and thus somewhat predictive of title performance, they are unlikely in and of themselves to drive or undermine the success of a game. We note this, in part, because of persistent rumors that some game developers have been jawboning game reviewers into giving their games higher critical review scores. We believe the publishers are better served by spending their time on the development process than by 'grade-grubbing' after the fact."
maybe it's the other way around... You only have to buy a sucky game *once* based on a raving review to *never* trust those reviews again. While your friends can comment properly on the game without some obscure metric like '8/10 overall'.
If a magazine or website is really scoring out of 10 or out of 100, then we ought to see some 1's and 2's.
But we don't do we ?
The researchers would find more utility in measuring the correlation between ad spend and score.
Anyone think these two variables don't correlate strongly ?
Dominic Connor,Quant Headhunter
When I personally buy a game, I look at gamefaqs user reviews instead of Metacritic. When looking at the main page of a game on gamefaqs, the first two averaged review numbers are exceptionally useful to me. They seem to give a very strong feel of what the general reaction to a game is - anything under a 7 is probably not worth my money. Also, user reviewers seem to me to play the games more thoroughly than someone who does reviews for a job, and game depth/replayability is a big point for me. Although, if I think about it, I generally buy games for Nintendo DS - price is pretty uniform and graphics can only get so good. In order to look up the game at all I had to have heard about it from my friends or some sites, so my experience doesn't really contradict his research at all.
So, while there certainly is correlation between the review score and purchase numbers, there's very limited (if any) causation? With the immense integrity the game-review professionals command, who'da thunk it?
It all boils down to trust (or more specifically lack of it) on the Game review sites.
I'm not overly surprised that people don't base their buying decisions primarily on the review scores from game-sites. In most sites I've seen one or all of the following:
Personally, I usually wait a while after the game is out and then go check user reviews. If your discount the "100%, great thing since sliced bread" ones (which come from fanboys) you'll usually be able to get a good picture of all the above mentioned points that the game sites miss (bugs, long-term (re)playability, intrusive DRM, hands-on-beyond-hype experience)
If you ask people if they are willing to pay more for quality 90% will answer yes. However when the moment supreme is there to purchase for example a new notebook 80% will go for the cheapest and don't care about long term stuff like quality. I think there is a good chance this survey works the same; People SAY they are not influenced by reviews because 'Hey, I'm an original, I don't let anybody influence me'.
Luckily with games, there are free demos available on every major torrent site.
Having said that, I do realise that this applies less to console owners, who are in a more difficult position because they generally can't test games before purchasing it, meaning they will have to live with a lower signal-to-noise ratio. (But then, they were the ones who chose to invest in a closed platform.)
Anyway, I'm fairly happy that most games are available for free testing (I'm usually not really in a rush to get any particular game), because - looking back - I can't really say that I found very many games that would've been worth my money if I had bought them (not even when they were sold at half 'list' price).. For the last couple of years the list would pretty much be limited to Portal, EU3, World in Conflict, Vampire: Bloodlines and Civ4 (and Arkham Asylum was OK too, just not at the current prices). Not a very long list, I might note.
In all, I would suggest people don't get consoles, as too much bargaining power is taken away from you in getting one, and too many games just aren't worth wasting money on.
Depending on how much the parent company spends in ads on the site/magazine, the score will be inflated higher. The few times that an editor or reviewer really did stand up and score a bad game as such, they were immediately fired by the advertising department. There was a big scandal a few years back when the editor actually spoke out against one such firing... I am too tired to look it up myself right now.
Anyway, reviews anymore from the "gaming press" are total garbage due to this mechanic. The ads in the magazine are more important to the company than the reviews themselves. When was the last time you saw an EA game get a 1 out of 10.... And trust me, there are many deserving candidates, like the yearly sports rehash which change nothing in the game, just which player is on which team. Or Race Driver Grid, or Darkar 2009, or Rally Stars.... The magazines would just not post a review of a game when it gets bad because they don't want to potentially lose their ads from the publisher...
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
I have three tiers of deciding on purchase of the game.
1. I read the review about what the game contains. I thoroughly ignore any "positive personal thoughts" about the game as marketing fluff. The negative ones do add to the value of the review but aren't all that important. I just read what is the concept of the game, and whether it is anything original, with potential - a good idea. If the review talks loads about graphics and sound and development time and prior franchise, even in total superlatives, it means the game is junk. A reviewer would concentrate on the really good points if it had any.
2.I check some Internet fora to see what people complain about. If there is a number of complaints about the same thing, it may turn me away again. The thing being "awful execution of the wonderful idea" is one of possible choices.
3. Then I grab the game off a torrent. After I'm through with it, I look back at how it felt. The only deciding factor is "I enjoyed it". Yeah, I enjoyed Stalker: Clear Sky, despite hopeless story, dull ending and reuse of content. I enjoyed Oblivion despite being dumbed down to knees level of Morrowind.
If the game passes the three tiers of classification, I buy it.
45 5F E1 04 22 CA 29 C4 93 3F 95 05 2B 79 2A B2
I used to write Amazon reviews - you know, the bit before the buyers reviews but after the manufacturers descriptions? I was impressed when I signed up that made it clear I could slate a product if it really wasn't any good. Their only stipulation was that I should suggest another product on their site that was know to be better. Seemed fair enough to me. I stopped some years ago but if that policy is still in effect, it owuld add some weight to their value IMO. This was the .co.uk - the .com had diffferent reviews and possibly different criteria.
I want a list of atrocities done in your name - Recoil
I buy games when I've enjoyed playing the demo. If there isn't a demo available, I don't buy it.
Game world not designed to allow for demo-style play? Rubbish. You can sandbox an area of a GTA map, limit Dragon Age: Origins to one town, make level caps to prevent access to higher level play... It's just laziness.
Finally had enough. Come see us over at https://soylentnews.org/
Since the publishers are so keen on getting 8/10 reviews, lets replace the stars with a scoring system that just gives them more of what they want.
A good game gets one (8/10).
A mediocre game gets 3: (8/10)(8/10)(8/10)
And a terrible game gets a whopping 10: (8/10)(8/10)(8/10)(8/10)(8/10)(8/10)(8/10)(8/10)(8/10)(8/10) ... the publishers get what they want, and anyone with a calculator to hand knows what we really mean
Be extra suspicious of games that embargo reviews, or allow just a handful of "exclusive" reviews to break the embargo. More often than not those reviews have been paid for in one way or another. Just like with other kinds of media there is usually a very good reason that publishers don't want you to know upfront what a game is like - because the product sucks.
Except you just illustrate the problem: something that's just "good enough" (which really just means "mediocre") gets an 8 out of 10. I'm sorry, but in a perfectly linear scale, "mediocre" would mean a 5. That's the kind of a number you could punch in a formula and get a correlation or anything else.
Plus, if it were just a case of a honest review and the bad ones being already cancelled, the results would look much like the right half of a bell curve. You know, the curve with the below average ones removed. For virtually any sitze out there, it doesn't. It looks like a bell curve centered on 8 or 9, and which pretty much starts at 6 or 7. Sorry, that's not a case of the bad ones being already removed, that's a clear sign of an offset scale. It's what you get when the occasional "something that's truly bad" means you get to give a 5 or a 6, not a 1 or 2.
And then there is the occasional reviewer whose curve looks like two spikes. The kind who churns 90% to 99% scores all year long, and then occasionally picks up some 10 year old freeware game so he can give _something_ a 5% score and fix his street cred. Or publishes a yearly smack-talking "top 10 worst games of all time" -- conveniently all 20 years old and from publishers which are no longer in business -- just to show that he's that unbiased and can give a low score too.
But again, that's not being unbiased and fair at all, it's just trying to compensate one crap (or dishonest) job with another one skewed in the other direction. If it were a real fair and unbiased and non-skewed job, you'd get one bell curve centered in the right place, not two spikes centered near the extremes of the scale.
A polar bear is a cartesian bear after a coordinate transform.
If a magazine or website is really scoring out of 10 or out of 100, then we ought to see some 1's and 2's. But we don't do we ?
My wife and I were having this same discussion the other day. I was going through some reviews of games that just came out, comparing them to older games in the series. When I spotted one and mentioned the poor review, my wife asked what was the score. "6 out of 10". She was confused that a bad game got such a high score.
I guess I've been reading these reviews for so long, I didn't think of it anymore. 10/10 is awesome, 9/10 is great, 8/10 is good, 7/10 is okay, 6/10 is poor, 5/10 and lower is terrible.
"But when was the last time you saw a 5/10?" I honestly didn't know. Even the big-name movie tie-ins that we all know to be awful will somehow manage to score "6.5". I actually had to go look up some reviews to find lower than "6" - but they are out there.
I've started to view the "out of 10" or "out of 100" scores like the old A-F grading system we used in school. A is 9/10 or 10/10 ("A+"), B is 8/10, C is 7/10, D is 6/10 ... F is 5/10 or lower. It's not ideal to view games this way, but it makes sense of the review scores.