Slashdot Mirror
Defying Review Aggregation
Logiksan writes "With the growing number of review aggregation sites like GameRankings and RottenTomatoes, it's becoming increasingly harder for individual game critics to be heard. GameDAILY Biz took a stab at the issue at came up with 5 aggregation-defiant tactics designed to help make reviews relevant again. Among their list of ideas is to destroy the typical review grading curve. The article states, 'If, for instance, a publication could establish a 10 point scale in which reviews were based upon purchase value and average games scored only a 3 or a 4, the higher scores would certainly become far more important. The lower scores would give the publication instant credibility as 'discerning gamers' and would free up the top scores (5-10) to show a more full range of differentiation for the top-tier titles gamers care about most.'"
these guys ought to work together!
If I make good reviews, and they get aggregated with other reviews, why is this a bad thing that has to be 'defied'?
Isn't the aggregation service something the readers persumably want?
And why assume that the reader will only look at the score from your review? perhaps the reader is actually interested in the detailed information you provide and click on a link to your site from the aggregation site? And then the aggregation site will actually benfit the reviewer, not go against him.
I would rather see "5 payola-defiant tactics designed to help make reviews relevant again". Maybe if so many game review sites weren't sucking at the game publishers' tits aggregation wouldn't matter...
I don't read game reviews any more, but not because of access to aggregation sites. I don't read them any more because they are generally paid ads for the game or fanboys who just go on and on about how great it is that Joanna Dark's boobs are made from so many polygons.
Pulp Audio Weekly - Geek News and Reviews
Making the average score lower, but still out of 10 shouldn't make much of a difference. What if magazines/reviewers just decided to increase their maximum score without upping the aggregate average? ...but this volume goes to 11, see?
What else can happen when an unstoppable force collides with an immovable object?
If you do a good job, readers may check out your review for a particular game (some people read all reviews) and decide to go directly to you from then on.
Gamerankings is also great because they allow their readers to also rate the reviews themselves. So if you are a good writer, your reviews will stand above the mediocre ones when people look for reviews on a particular game.
Maybe Gamedaily got tired of seeing so many of it's reviews receive one star or less?
If I was really interested in starting a new zine from scratch, I would LOVE to get my reviews into the aggregator, try to accumulate high rankings for my reviews, and the traffic would increase and so would the willingness of devs to send me demo copies and put my quotes on their game boxes and I would also be able to get more advert revenue. It's a win-win-win all around.
Now, I used to read reviews in magazines. I even had a subscription to Edge for a while, and used to like the longer reviews (the pieces on San Andreas from there and other mags come to mind). But now I tend to rely more on peer reviews - and by that I mean people I actually know. Combine that with selectively trying demos and I reckon I have a system that does pretty well (for me anyway).
Of course, if I see a game discussed in depth on the world wide webby and with a great community, I will more than likely give it a shot. Notable games from this category are x/netrek; nethack; and HardWar, to name but 3.
So who is suffering? Well, [formerly?] well-paid magazine or ezine professional game reviewers. And yes, that is a shame to some extent. I do enjoy reading the long reviews -- I appreciate when someone puts an effort in -- but I would rather spend the subscription money elsewhere, say on a new graphics card to take advantage of these great games I should be reading reviews of.
So how do you recapture my (and everyone else's) attention? From TFA:
Ditch The Scores - Makes the review stand on the merit of its content. Good idea - as long as the content is good.
Focus on MegaReviews - Yup. El Goog prvoed that more is less; so write less reviews, but make them longer.
Trumpet Your Own Credibility - Personality can be interesting, but a fine line lies between 'trumpeting credibility' and 'arrogant gits that I won't be rading again'.
Aggregate Reviews on Your Own - Can't beat them, so join them? Might work.
Crunch The Curve - If people are looking for a quick point score, they are looking for something that conforms to their expectations. Giving lower scores will ultimately damage credibility and turn off readers, even if your intentions are noble. Here's a better idea - lose the scores altogether. Give people a well-written indepth review. The ones that are looking for a point score won't read if you give it 4/10 as opposed to 7/10 anyway.
my .02
If all you have is a grenade, pretty soon every problem looks like a foxhole -- MightyYar
Been about time. When I stopped regularily reading gaming mags 5 or so years ago, 70% (or 7/10 or whatever the scale was) was a pretty mediocre result, and it's only got worse. The same magazines that once celebrated the first ever game to score over 90%, ever - are now giving out 90% every month or so.
It's all marketing pressure. 100% does not mean "perfect" anymore. Problems that would've dumped the score 5, 10 points are shrugged off by many testers as "will certainly be fixed by the next patch" or "but it's still better than elsewhere".
There is one online review page that still writes very criticial and sometimes harsh reviews, where the stuff that rates 94% in your average mag (which only by coincidence has a two-page ad by the same company that month...) - well, that overhyped crap gets its 47% or whatever it's really worth once you remove the "big names" and the photoshopped screenshots.
I just wish for the life of me I could remember the URL. I lost my bookmarks once, and that was the only game review site worth having a link.
I also remember what some of those tester dudes said when flat out confronted with the fact that they only ever seem to review 70%+ games. They said "we don't want to waste precious magazine space with the mediocre games".
Sounds believeable. Except that the rations are still way overblown. I like "The Movies", for example, but it's not a 9/10 game. Civ4 - great game, but 9.7/10? If you can only imagine 3% missing to absolute perfection then you have damn little imagination. And so on, and so forth.
Really, an honest rating should either allow > 100%, or say "90% if you do everything in the best way that I can imagine. Points above that only if you found better ways to do it, and because you had a few years to work on it that's not an unrealistic expectation".
Assorted stuff I do sometimes: Lemuria.org
while the suggestion to increase the actually used "dynamic range" of the scoring system used is certainly a step into the right direction, why does he stick to the an overall numerical rating system at all?
i could imagine very good reviews that would group the usual categories into pairs of that are contradicting each other or at least are quite opposite to each other, like "this game focuses on replayability" vs "this game focuses on an intense story", "focuses on technical aspects" vs "focuses on other qualities", or just distribute a _fixed_ number of points on the various categories, to describe the game, give it a rough position in an n-dimensional matrix, not rate it.
a serious review should never pretend to be able give ratings with more precision than something along the lines of "you will love it"/"you will like it"/"you might enjoy it"/"pain" (all assuming you generally enjoy the genre).
"game x is 3.5% better than game y" is pure bullshit and a sure sign of a review that is actually nothing more than comparing technical specs and skipping through the game with cheat codes to provide some screenshots.
and yeah, pissing off game publishers with honest ratings is a really bad idea in a business that mostly depends on hyping up "exclusive" previews to lure customers.
the threat of losing "early screenshot" benefits is probably an even bigger threat than the whole dependency on ad money from publishers, not only because it looks less like bribery but also because customers lost due to lack of exciting "next gen" cover stories will affect the ad-income from all companies, not only the one in question.
[i have an opinion and i am not afraid to use it]
So who's left? People who don't use an aggregator in the first place. And what's he imploring them to do? Not use an aggregator. Seems kind of pointless, doesn't it...?
Besides, I think aggregators are great. Especially well-stewarded ones like Metacritic (which was strangely dropped from the submission text while the other two aggregators weren't). With intelligently weighted averages you get a useful snapshot of opinion, plus the option to check out as many individual reviews as you like to get a fuller picture. I've bought a number of games that had mediocre averages because the text of the linked reviews made it clear that they were something I'd appreciate, just as I've avoided a number of games with high averages where the text of the reviews made it clear that they wouldn't be suited to my taste.
And yes, having your site included in an aggregator's round-up can only be a good thing -- it'll grant immediate exposure to your work and let readers decide for themselves how much clout to give your reviews vs. the weighted average in the future. This stands out even more when you review more obscure titles that don't get covered by all the big outlets, so you become one of a much smaller group of reviews.
I say aggregators are win-win for both readers and review sites.
I think reviews, themselves, are becoming outmoded. I can go to a message board and get opinions from hundreds of other players that I know share similar interests with me. They know what I'm going to care about in a video game more than some random shlomo.
:P).
Video game magazines tend to be targeted at a different audience than me, and I'm not entirely sure what that audience is: maybe younger gamers, or those who aren't quite as involved. You get the opinions of one or two people, max, and possibly short scores from a handful of others that I don't know from Fatal1ty (Adam for the less hardk0re
Finally, before I plunk down money on a game I try the demo. Even a lot of console games now have computer versions with demos. They might lag behind, but I usually stay a year or two behind the console trend because the pricing on their games are ridiculous (some computer games are guilty of this as well, but they tend to be more reasonable).
There are numerous review sites that are mad efor independant reviewers. A few include: http://gamefaqs.com/GameFAQs, and my personal website http://coreoblivion.gamersmack.com/The Core: Oblivion (pardon the plug)
Why not just allow anyone to write reviews and give game scores? Then the readers moderate the reviews, and can set threshholds of what sort of modded reviews they want to see.
What's that? Trying this good idea from the rest of the world would put professional game reviewers out of business? Well then they should change their business model, perhaps move into those game-reviewing websites which'll likely have pretty decent ad revenues, or possibly a charge for membership.
Has this been done elsewhere or should I get started?
My site relaunches on Friday 27th and we decided to "drop the scores" a couple of months ago.
They're pointless in the current climate. Personally, I decided to take this route because I was sick of seeing referral traffic from forums where folks had come along to the site, read the score (not the review text) and then promptly headed back to their favourite forum to trash us.
"Why did he give that 9/10? It sucks! Its not as good as !" etc., etc.
The alternative? Just give them the review. Then they don't have to ask WHY a reviewer who's spent a good while playing a game and a good while writing about it has given the score that he has, just because they're too damn lazy to read the review text. We have a short summing-up at the end of the review, so folks can read that if they really are adverse to checking out the rest of it.
All I know, is that if I've personally taken time out to visit a website to check out a review, I'm going to READ it. Not everyone is like that. Some folks do just want a number in place of valid reasoning and qualification of the opinion in question. Good luck to them. I guess they're the guys that return things to stores because they "didn't do what they expected."
Or the guys that stood in front of a 4ft tall black-on-white sign pasted on my local game store's window that said "We have 4 Xbox 360's and they are all reserved for preorders." on launch day, for three hours. Before walking up to the cashier when the doors opened and saying "Do you have any spare Xbox 360's? I didn't manage to preorder one. No? Are you sure? There are none left? Can you check?"
My Mind Is Rewired. Is Yours?
is gamerevolution.com
Relatively good games get a B+ or A- on their list. The rest get C's, D's, and more importantly, a hilarious review of just how bad the game really is.
Gamerankings already changes a review's score to work on a 0-100 scale. If they know that an "average" score for a site's reviews is a 3-4 and that compares to a "7" on other sites, they'll just mark them down as a 7.
Besides, the whole thing smells of sour grapes to begin with.
This is easily accomplished doing something instead of averaging scores for different elements of game play. You work instead by multiplying them together. A came that score three perfect tens ends up with 1000 in this system, while 3 sevens gives you 343 for a score, and 3 eights gives you a score of 512. In a 4 star system, 5 scores of 4 give you 1024, while 5 scores of three stars gives you 243 as total. 5 score of 3.5 stars gives you 525 and change.
"It is a greater offense to steal men's labor, than their clothes"
http://www.gamefaqs.com/ and http://www.gametrailers.com/.
The point for me is that the score doesn't matter one little bit. What is important is *why* they gave it that score and/or why they didn't.
For example the reviewer might mark a game down because it was too hard. For me that would be a plus. They might say it is fun but too short, which also might appeal to me at certain times.
What is also important for me is to see some actual gameplay. I can often pick up more about a game from a 2 minute video review than a 10 page article. When you can see things like camera, frame rate and so on it really helps you get a handle on what you are in for.
Write reviews which are more than just a description of the game with some suitable adjectives and a score attached. "Amazing level design, great graphics, 10/10" and "Horrible level design, losy graphics, 1/10" aren't really interesting to read.
No kidding!!! What do you say at this point?
Sounds to me like the games industry is bitching because they make it that less effective and more expensive to bribe up a handful of review scores using advertising revenue or whatever.
Edge magazine regularly give out 5's for average games and only give out a ten once every year or two. Perhaps it's not on the reviewers radar because it's a UK publication. http://www.edge-online.co.uk/archives/edge/index.p hp
Take one of the few quality gaming magazines that still exists, say Edge, and compare it to an internet review site, say GamePro. Edge's reviews are well written, interesting and provided a good overview of the game's quality; the score is almost irrelevant. GamePro's "Fun Factor" is utterly arbitrary, biased towards the upper end of the scale and given pride of place on the page. Meanwhile their reviews are lacking in detail and obsess over utterly unimportant details.
A will happily base purchasing decisions on an Edge review; all GamePro reviews tell me is that their reviewers should find another job.
I was thinking about writing game (and maybe other) reviews for my newly created site, and one of my ideas was to either ditch the score system completely, or make it very simple, almost binary. Assigning 0/1 didn't seem fair though, why should a game which barely redeems itself get the same score as a GOTY-class title? Right now I think giving out average/recommended/highly recommended/second coming of Christ ratings would give a clear idea of what my feelings were, without getting into the whole "give this game .5 more because i say so!" thing.
Now another thing is the length of the reviews. Short but to the point? Lengthy but deep? I'm-Neal-Stephenson-length essays with interesting, but not completely relevant bits thrown in? I'm not sure, but that's ok because I'm not writing the reviews right now. In fact, I should be studying for the two finals I have tomorrow and the day after that. I'll probably start with writing shorter reviews for the games I've played a while ago, expanding them when (if) I replay them. For newer games I could probably write longer stuff while all the experience is fresh in my memory.
What do you guys think?
I personally like a video review and I get most of mine from X-Play on G4. I know they are whores but they are honest whores. Nothing captures the essence of ultimate spiderman like the world's rudest Stan Lee impersonator. More importantly I can see the gameplay and Sessler is really fond of demonstrating why a game sucks showing you the terrible camera or ai or whatever. Plus Morgan is pleasing to look at if not listen to.
I also get a few mags like gamepro and I read every review, but I rarely find one that stands up to an established publisher with cash.
OSGGFG - Open Source Gamers Guide to Free Games
I work for a progressive review site called http://www.thegamechair.com/ and would like to mention a couple points. First, we would love to be included on some of the major content aggregators. The major aggregator's sites function something like a Google search for reviews with the handy addition of a mathematical formula for determining the average score. I feel like they drive out far more traffic out to individual sites than they steal from them. I know for a fact that I regularly read 5-10 reviews of a game or film, using the links on metacritic as a starting point.
Also, I'd like to mention our approach at TheGameChair, which involves a little bit of everything. We've kept the scoring as an integral part of our review process, but we've broken up the reviews into pieces that more accurately reflects the way that we play games. Basically, we divide the amount of playtime required to make it through a game into three chunks, and write a separate review for each chunk of play. The scores can vary based on whether a particular section of the game plays better, or can be affected as tedium or boredom sets in. The entire length of the review process stretches out over a three week period; covering the average length of time it takes an ordinary person to complete a game.
Our final score for a game is based on an average over the three separate play session, so we do have a type of aggregation over a time frame built into the review. However, because of the sheer volume of review sites on the web, and the amazing speed at which the reviews are released, the only way to distinguish yourself from the aggregate crowd is to actually provide original, thought provoking or even funny content.
only one everything