Defying Review Aggregation

Logiksan writes "With the growing number of review aggregation sites like GameRankings and RottenTomatoes, it's becoming increasingly harder for individual game critics to be heard. GameDAILY Biz took a stab at the issue at came up with 5 aggregation-defiant tactics designed to help make reviews relevant again. Among their list of ideas is to destroy the typical review grading curve. The article states, 'If, for instance, a publication could establish a 10 point scale in which reviews were based upon purchase value and average games scored only a 3 or a 4, the higher scores would certainly become far more important. The lower scores would give the publication instant credibility as 'discerning gamers' and would free up the top scores (5-10) to show a more full range of differentiation for the top-tier titles gamers care about most.'"

Comments

why?

[free space]

If I make good reviews, and they get aggregated with other reviews, why is this a bad thing that has to be 'defied'?
Isn't the aggregation service something the readers persumably want?

And why assume that the reader will only look at the score from your review? perhaps the reader is actually interested in the detailed information you provide and click on a link to your site from the aggregation site? And then the aggregation site will actually benfit the reviewer, not go against him.

Re:why?

[Daetrin]

And why assume that the reader will only look at the score from your review? perhaps the reader is actually interested in the detailed information you provide and click on a link to your site from the aggregation site? And then the aggregation site will actually benfit the reviewer, not go against him.

When i'm thinking about going to a somewhat iffy movie i'll check out RottenTomatoes and see what aggregate score it got. Then i'll look over the blurbs, keeping an eye out for any reviewers i tend to agree with. And if the movie has a generally negative rating i'll check out the negative reviews to see _why_ they didn't like it.

For example i was trying to decide whether to go see Underworld Evolution, at the time i checked RottenTomatoes there were only a couple reviews up and almost all rotten. However i noticed the following blurb: "Stylish, but doesn't really re-imagine the vampire genre, cluttering the screen with creatures that no longer shock or surprise us. It's also a bit too long to pack a mean punch."

Okay, i liked the first cause it had lots of werewolves and vampires beating up on each other and a cute girl wearing cool outfits, and i actually like longer movies. So the review was negative but for reasons that completly didn't apply to me, in fact the only part that really interested me was the one positive comment, that it was "stylish." So i went and saw it and was happy with the results. It was a fun werewolves and vampires beat up on each other movie with a cute girl in it.

Presumably if game reviewers write well thought out reviews that explain _why_ they liked or disliked a game then the same process will happen there once the situation settles down. People will look at the aggregate scores, but they'll also look at what their favorite viewers have to say and they'll look at why the negative reviewers didn't like it. Some people complained about Shadow of the Collossus being too short, but given that i don't have as much time as i like to play video games that's not really a huge concern to me. I go the game, enjoyed beating up the first four or so Colossi (while feeling sorry for them at the same time of course) and then got distracted with other games i got for christmas. Clearly that particular criticisim wasn't relevant to me but knowing what the criticism was was important.

Aggregation not the problem

[yotto]

I don't read game reviews any more, but not because of access to aggregation sites. I don't read them any more because they are generally paid ads for the game or fanboys who just go on and on about how great it is that Joanna Dark's boobs are made from so many polygons.

Re:Aggregation not the problem

[SgtFajita]

Yeah, game reviews are quickly becoming irrelevant to quality. It's to the point where most gamers consider an aggregate score lower than 8/10 to be a "bad game." In their defense, it seems most reviews consider 5/10 to be their minimum score. So a 7/10 is only average, which (like other mass media) is pretty much crap.

I'd be interested in a website with reviews that only focus on the bad parts of the game. Advertisers, demos, and promotions be damned: list off the major problems, along with the little annoying things that can detract from enjoying the game fully, and I'll have a much better idea whether I should bother to play it. The good stuff is covered by pretty much every other site, so tell me about the lack of control, horrible camera, unintuitive menu, or repetitive gameplay so I don't waste my money/time playing it if any of those problems are dealbreakers for me.

From a former zine game editor....

[MBraynard]

A lot of the smaller guys have trouble being heard - or ever heard of. The aggregator includes your zine's reviews with all the big boys. GameRankings is a big opportunity for free advertising.

If you do a good job, readers may check out your review for a particular game (some people read all reviews) and decide to go directly to you from then on.

Gamerankings is also great because they allow their readers to also rate the reviews themselves. So if you are a good writer, your reviews will stand above the mediocre ones when people look for reviews on a particular game.

Maybe Gamedaily got tired of seeing so many of it's reviews receive one star or less?

If I was really interested in starting a new zine from scratch, I would LOVE to get my reviews into the aggregator, try to accumulate high rankings for my reviews, and the traffic would increase and so would the willingness of devs to send me demo copies and put my quotes on their game boxes and I would also be able to get more advert revenue. It's a win-win-win all around.

Is this a problem?

[BertieBaggio]

Now, I used to read reviews in magazines. I even had a subscription to Edge for a while, and used to like the longer reviews (the pieces on San Andreas from there and other mags come to mind). But now I tend to rely more on peer reviews - and by that I mean people I actually know. Combine that with selectively trying demos and I reckon I have a system that does pretty well (for me anyway).

Of course, if I see a game discussed in depth on the world wide webby and with a great community, I will more than likely give it a shot. Notable games from this category are x/netrek; nethack; and HardWar, to name but 3.

So who is suffering? Well, [formerly?] well-paid magazine or ezine professional game reviewers. And yes, that is a shame to some extent. I do enjoy reading the long reviews -- I appreciate when someone puts an effort in -- but I would rather spend the subscription money elsewhere, say on a new graphics card to take advantage of these great games I should be reading reviews of.

So how do you recapture my (and everyone else's) attention? From TFA:

Ditch The Scores - Makes the review stand on the merit of its content. Good idea - as long as the content is good.

Focus on MegaReviews - Yup. El Goog prvoed that more is less; so write less reviews, but make them longer.

Trumpet Your Own Credibility - Personality can be interesting, but a fine line lies between 'trumpeting credibility' and 'arrogant gits that I won't be rading again'.

Aggregate Reviews on Your Own - Can't beat them, so join them? Might work.

Crunch The Curve - If people are looking for a quick point score, they are looking for something that conforms to their expectations. Giving lower scores will ultimately damage credibility and turn off readers, even if your intentions are noble. Here's a better idea - lose the scores altogether. Give people a well-written indepth review. The ones that are looking for a point score won't read if you give it 4/10 as opposed to 7/10 anyway.

my .02

Finally!

[Tom]

Been about time. When I stopped regularily reading gaming mags 5 or so years ago, 70% (or 7/10 or whatever the scale was) was a pretty mediocre result, and it's only got worse. The same magazines that once celebrated the first ever game to score over 90%, ever - are now giving out 90% every month or so.

It's all marketing pressure. 100% does not mean "perfect" anymore. Problems that would've dumped the score 5, 10 points are shrugged off by many testers as "will certainly be fixed by the next patch" or "but it's still better than elsewhere".

There is one online review page that still writes very criticial and sometimes harsh reviews, where the stuff that rates 94% in your average mag (which only by coincidence has a two-page ad by the same company that month...) - well, that overhyped crap gets its 47% or whatever it's really worth once you remove the "big names" and the photoshopped screenshots.

I just wish for the life of me I could remember the URL. I lost my bookmarks once, and that was the only game review site worth having a link.

I also remember what some of those tester dudes said when flat out confronted with the fact that they only ever seem to review 70%+ games. They said "we don't want to waste precious magazine space with the mediocre games".
Sounds believeable. Except that the rations are still way overblown. I like "The Movies", for example, but it's not a 9/10 game. Civ4 - great game, but 9.7/10? If you can only imagine 3% missing to absolute perfection then you have damn little imagination. And so on, and so forth.

Really, an honest rating should either allow > 100%, or say "90% if you do everything in the best way that I can imagine. Points above that only if you found better ways to do it, and because you had a few years to work on it that's not an unrealistic expectation".

Re:Finally!

[Red Flayer]

"Really, an honest rating should either allow > 100%, or say "90% if you do everything in the best way that I can imagine. Points above that only if you found better ways to do it, and because you had a few years to work on it that's not an unrealistic expectation"."

This is where game reviewers could learn something from an established idustry that faced the same problem decades ago -- fine wines.

Robert Parker and other wine critics theoretically can assign values up to 100. But a 98 is a once-in-a-lifetime wine. A 90 is excellent (truly excellent). An 80 is fair.

Sure, most wines you'll see in the store are 85 pts & higher, but Joe BasementCask's '04 grenache isn't making it to market -- only the good, commercial stuff is.

But you know what? My wife wouldn't like a 92-pt cab, no matter how good everyone else thinks it is. Just like some people won't like CivIV, no matter how closely it approaches the ideal for that type of game.

My main point is, games should be reviewed on a bell curve -- like you say, scoring above 90 should be VERY difficult. And people need to understand that just because everyone who likes FPSs rated a game 9.2, doesn't mean they'll like the game that much, if they only play RTS games.

why the need for a numerical rating?

[usrusr]

while the suggestion to increase the actually used "dynamic range" of the scoring system used is certainly a step into the right direction, why does he stick to the an overall numerical rating system at all?

i could imagine very good reviews that would group the usual categories into pairs of that are contradicting each other or at least are quite opposite to each other, like "this game focuses on replayability" vs "this game focuses on an intense story", "focuses on technical aspects" vs "focuses on other qualities", or just distribute a _fixed_ number of points on the various categories, to describe the game, give it a rough position in an n-dimensional matrix, not rate it.

a serious review should never pretend to be able give ratings with more precision than something along the lines of "you will love it"/"you will like it"/"you might enjoy it"/"pain" (all assuming you generally enjoy the genre).

"game x is 3.5% better than game y" is pure bullshit and a sure sign of a review that is actually nothing more than comparing technical specs and skipping through the game with cheat codes to provide some screenshots.

and yeah, pissing off game publishers with honest ratings is a really bad idea in a business that mostly depends on hyping up "exclusive" previews to lure customers.

the threat of losing "early screenshot" benefits is probably an even bigger threat than the whole dependency on ad money from publishers, not only because it looks less like bribery but also because customers lost due to lack of exciting "next gen" cover stories will affect the ad-income from all companies, not only the one in question.

Game Reviews: Obsolete?

[Sage Gaspar]

I think reviews, themselves, are becoming outmoded. I can go to a message board and get opinions from hundreds of other players that I know share similar interests with me. They know what I'm going to care about in a video game more than some random shlomo.

Video game magazines tend to be targeted at a different audience than me, and I'm not entirely sure what that audience is: maybe younger gamers, or those who aren't quite as involved. You get the opinions of one or two people, max, and possibly short scores from a handful of others that I don't know from Fatal1ty (Adam for the less hardk0re :P).

Finally, before I plunk down money on a game I try the demo. Even a lot of console games now have computer versions with demos. They might lag behind, but I usually stay a year or two behind the console trend because the pricing on their games are ridiculous (some computer games are guilty of this as well, but they tend to be more reasonable).

Bizarre.

[Retroneous]

My site relaunches on Friday 27th and we decided to "drop the scores" a couple of months ago.

They're pointless in the current climate. Personally, I decided to take this route because I was sick of seeing referral traffic from forums where folks had come along to the site, read the score (not the review text) and then promptly headed back to their favourite forum to trash us.

"Why did he give that 9/10? It sucks! Its not as good as !" etc., etc.

The alternative? Just give them the review. Then they don't have to ask WHY a reviewer who's spent a good while playing a game and a good while writing about it has given the score that he has, just because they're too damn lazy to read the review text. We have a short summing-up at the end of the review, so folks can read that if they really are adverse to checking out the rest of it.

All I know, is that if I've personally taken time out to visit a website to check out a review, I'm going to READ it. Not everyone is like that. Some folks do just want a number in place of valid reasoning and qualification of the opinion in question. Good luck to them. I guess they're the guys that return things to stores because they "didn't do what they expected."

Or the guys that stood in front of a 4ft tall black-on-white sign pasted on my local game store's window that said "We have 4 Xbox 360's and they are all reserved for preorders." on launch day, for three hours. Before walking up to the cashier when the doors opened and saying "Do you have any spare Xbox 360's? I didn't manage to preorder one. No? Are you sure? There are none left? Can you check?"