Slashdot Mirror
Fixing Steam's User Rating Charts
lars_doucet writes: Steam's new search page lets you sort by "user rating," but the algorithm they're using is broken. For instance, a DLC pack with a single positive review appears above a major game with a 74% score and 15,000+ ratings.
The current "user rating" ranking system seems to divide everything into big semantic buckets ("Overwhelmingly Positive", "Positive", "Mixed", etc.), stack those in order, then sort each bucket's contents by the total number of reviews per game. Given that Steam reviews skew massively positive, (about half are "very positive" or higher), this is virtually indistinguishable from a standard "most popular" chart.
Luckily, there's a known solution to this problem — use statistical sampling to account for disparate numbers of user reviews, which gives "hidden gems" with statistically significant high positive ratings, but less popularity, a fighting chance against games that are already dominating the charts.
The current "user rating" ranking system seems to divide everything into big semantic buckets ("Overwhelmingly Positive", "Positive", "Mixed", etc.), stack those in order, then sort each bucket's contents by the total number of reviews per game. Given that Steam reviews skew massively positive, (about half are "very positive" or higher), this is virtually indistinguishable from a standard "most popular" chart.
Luckily, there's a known solution to this problem — use statistical sampling to account for disparate numbers of user reviews, which gives "hidden gems" with statistically significant high positive ratings, but less popularity, a fighting chance against games that are already dominating the charts.
As a developer with a title on Steam, I can say ratings are just 1 of many issues with the store.
I can't tell you how many support tickets I have to go through because steam can't properly launch or update the main binaries that are covered under their drm. Most of the time it's AV issues, but once in a blue moon it's steam borkin' out. (0.25-0.5% of sales)
In the backend specific game information can't be shown to specific users. So I'm locked out of my real time statistics because my publisher has other titles from other developers...
But the worst, the god awful worst, are games on Early Access that are pure shit or have been abandoned... They drag the entire system down and majorly screw over legitimate titles that are in Early Access. IMO Steam should have a purging system for these titles. Perhaps even give coupon codes to users who bought games that have been purged.
As for the rating system. It definitely needs to be weighted. But there should also be incentive to give a ratings (even if they give a review anything) and there should be at least a "maybe" option. The thumbs up/thumbs down system doesn't really do it for me. Specially since I have negative ratings on my project such as "Game needs a German translation. 5/10" :\