Slashdot Mirror

← Back to Stories (view on slashdot.org)

Fixing Steam's User Rating Charts

Posted by Soulskill on from the hmm-same-distribution-as-professional-reviews dept.

lars_doucet writes: Steam's new search page lets you sort by "user rating," but the algorithm they're using is broken. For instance, a DLC pack with a single positive review appears above a major game with a 74% score and 15,000+ ratings.

The current "user rating" ranking system seems to divide everything into big semantic buckets ("Overwhelmingly Positive", "Positive", "Mixed", etc.), stack those in order, then sort each bucket's contents by the total number of reviews per game. Given that Steam reviews skew massively positive, (about half are "very positive" or higher), this is virtually indistinguishable from a standard "most popular" chart.

Luckily, there's a known solution to this problem — use statistical sampling to account for disparate numbers of user reviews, which gives "hidden gems" with statistically significant high positive ratings, but less popularity, a fighting chance against games that are already dominating the charts.

6 of 93 comments (clear)

  1. Re:Valve Time by Nemyst · · Score: 3, Interesting

    Valve have not shown a particular tendency towards using algorithms to fix things though - they generally just throw the work to their users, from tagging to rating to curating. While I agree that a fairer system to sum up reviews would be good for the consumer and good for smaller developers, I don't think it's what Valve are looking for. It's quite likely that showing more of the top sellers sells more copies of those games, which have dramatically higher sales volumes and prices than indies. That's a bigger cut for Valve.

    As much as people forget, Valve are not in this to make gaming a better place. They're there to make gobs of money, and have been rather successful at doing so thus far. Considering the sort of talent they hire and have hired in the past, if they truly wanted to fix things, they'd be fixed. If they're not, they either don't consider it important or have a reason for not fixing it.

  2. It's too bad... by Charliemopps · · Score: 4, Interesting

    It's really too bad the way Valve has screwed the pooch with Steam over the last few years. They literally had The gaming platform for PC all locked up. There was a time where I was desperately hoping they'd have an IPO so I could invest. But they tried to make the store so user friendly to Game controllers... a use case that may very well never become popular... that it's almost useless. Now, the only reason I think they are still relevant is because no one has bothered to try and challenge them. But I think they are 1 clever startup away from losing their position for good.

    There are games on steam to this day, that I cannot find... even using Google searches with the site:steampowered.com modifier. I have to go to the damned games external website and use their link to get to the thing I want to buy. I want to buy it and Steams own search doesn't bring it up because their search algorithm is so broken. I try to browse games and it limits what I can browse to a few dozen. Yet, when I go back 2 days later, its the same few dozen... why doesn't it just show me game after game until I've seen them all? There are over 4000 games on Steam!!

    And you know... I know what people are going to reply to me with... "You didn't click X!" or "You moron, you have to go to the blah page!" or whatever... I'm sure it's entirely my fault for not knowing how to do it right. But let me tell you something... the biggest moron on the planet can walk into Walmart and leave with less money. That's the key to their success. You cannot enter a Walmart and avoid seeing something you need to buy today. You don't have enough money? No worries there's a god damned bank on premises to give you a loan! It's easy to find something you like, it's easy to get it to the register and its easy to get it to your car.

    Why Valve? Why is it so god damned hard to give you my money?!?! I can go on the google play store and spend money with one damned finger! My 4yr old spent $20 on Angry birds slingshots before my wife locked her phone. He couldn't even figure out how to launch a game from your damned app!

    1. Re:It's too bad... by Anonymous Coward · · Score: 3, Interesting

      This I entirely agree with. The funny part of their whole new Discovery design? 99% of that could have been resolved with an Amazon-like system where you could simply get a list of "other people bought this" on all their store pages for games and allow one to readily go to the store page from near every game-related page (art, forums, etc). Of course to do that well would require having multiple tabs and perhaps even a like/dislike system per user to help sort things down--and there's Netflix to inspire on that front.

      Honestly, the only reason I have any Steam games at all is I bought them through bundles. Trying to use Steam to buy games directly is too much of a hassle that almost guarantees that Valve and developers who put games on Steam will never get full listed price from me (unless, like you said, I find the game outside of Steam and Steam's store page the only way to buy it).

  3. A much simpler method by PacoSuarez · · Score: 5, Interesting

    If the only two choices are positive/negative (or thumbs up/thumbs down or some other equivalent 0/1 scheme), here's a formula that should work fairly well:

    (n_positive + 1) / (n_positive + n_negative + 2)

    So a single positive review gives you a score of .6667, and a single negative review gives you .3333. For large numbers of reviews, the score quickly converges to the actual fraction. If you don't have any reviews, you are at .5000.

    The mathematical justification for this formula is that if you try to use a Bayesian approach to estimating the true probability of getting a positive review, and you start with a flat prior, this formula gives you the average of the posterior probability after observing the given number of positive and negative reviews. The full posterior distribution is a beta distribution with parameters alpha=n_positive+1 and beta=n_negative+1.

    This formula is often used when applying Monte Carlo techniques to the game of go. I believe a lot of programmers simply start the counters of wins and losses at 1 to avoid corner cases (like division by 0), and they accidentally use the correct formula.

  4. Re:Ethics can't be patched in by Luckyo · · Score: 4, Interesting

    The said comment is complete and utter bullshit. When he did his Guns of Icarus thing with other people, he always disclosed it in video description. After some desperate gamasutra folks (who notably have massive vested interest in sinking Bain, NerdCubed and a couple of other big youtube gaming commentators, because they have been massively eating into their audience numbers) started whining that disclosure in video description was not enough (according to the laws, yes it is), he added a short message to the beginning of every such video where he states any potential sources of interest he may have, down to having received a review copy.

    Notably, gamasutra itself does not do this, and has been central to the whole gamergate scandal which revolves exactly around this kind of acting, only with gamasutra folks not disclosing their conflicts of interest anywhere. Not in descriptions, not in topics, not in articles. Nowhere.

    Frankly, Bain goes to ridiculous lengths to disclose any potential conflict of interest he has nowadays, to the point of holding 3-hour talk marathons with developers (including CEO of the company that bought the sponsored content you talk about) and games media journalists partially on this topic and then puts those on his channel:
    https://www.youtube.com/watch?...

    If you really care about the issue, join #gamergate and go after people who actually do this crap.

  5. Re:Valve Time by Jupix · · Score: 4, Interesting

    What Steam reviews are actually filled with is information about the games... exactly what you should be interested in, as opposed to a score or a conclusion of some kind.

    The aggregate score in the style of "very positive" etc. can be useful in filtering out the genuinely terrible games, but outside of that, not so much. What's needed for decisionmaking is a lot of information, a search engine, and your own thinking. Steam provides descriptions, tags, and now reviews, and for me anyway it's been incredibly easy lately to figure out whether I want to buy a particular game, or at least investigate it closer elsewhere.

    Scores are almost completely worthless. Doesn't matter what kind they are (Metacritic, user review average, magazine review score). Steam has already done enough for the scoring system. What is there to fix? IMHO they should concentrate on important things like search, GUI and customer service, all of which are pretty terrible for 2014.