Slashdot Mirror
Fixing Steam's User Rating Charts
lars_doucet writes: Steam's new search page lets you sort by "user rating," but the algorithm they're using is broken. For instance, a DLC pack with a single positive review appears above a major game with a 74% score and 15,000+ ratings.
The current "user rating" ranking system seems to divide everything into big semantic buckets ("Overwhelmingly Positive", "Positive", "Mixed", etc.), stack those in order, then sort each bucket's contents by the total number of reviews per game. Given that Steam reviews skew massively positive, (about half are "very positive" or higher), this is virtually indistinguishable from a standard "most popular" chart.
Luckily, there's a known solution to this problem — use statistical sampling to account for disparate numbers of user reviews, which gives "hidden gems" with statistically significant high positive ratings, but less popularity, a fighting chance against games that are already dominating the charts.
The current "user rating" ranking system seems to divide everything into big semantic buckets ("Overwhelmingly Positive", "Positive", "Mixed", etc.), stack those in order, then sort each bucket's contents by the total number of reviews per game. Given that Steam reviews skew massively positive, (about half are "very positive" or higher), this is virtually indistinguishable from a standard "most popular" chart.
Luckily, there's a known solution to this problem — use statistical sampling to account for disparate numbers of user reviews, which gives "hidden gems" with statistically significant high positive ratings, but less popularity, a fighting chance against games that are already dominating the charts.
If the only two choices are positive/negative (or thumbs up/thumbs down or some other equivalent 0/1 scheme), here's a formula that should work fairly well:
(n_positive + 1) / (n_positive + n_negative + 2)
So a single positive review gives you a score of .6667, and a single negative review gives you .3333. For large numbers of reviews, the score quickly converges to the actual fraction. If you don't have any reviews, you are at .5000.
The mathematical justification for this formula is that if you try to use a Bayesian approach to estimating the true probability of getting a positive review, and you start with a flat prior, this formula gives you the average of the posterior probability after observing the given number of positive and negative reviews. The full posterior distribution is a beta distribution with parameters alpha=n_positive+1 and beta=n_negative+1.
This formula is often used when applying Monte Carlo techniques to the game of go. I believe a lot of programmers simply start the counters of wins and losses at 1 to avoid corner cases (like division by 0), and they accidentally use the correct formula.