Why You Should Be Suspicious of Online Movie Ratings (fivethirtyeight.com)

An anonymous reader writes: Statistical news blog fivethirtyeight.com noticed some odd discrepancies in online movie ratings, which caused them to do some investigating. They found it was generally a bad idea to rely on such ratings, particularly from sites like Fandango. "When I focused on movies that had 308 or more user reviews, none of the 209 films had below a 3-star rating. Seventy-eight percent had a rating of 4 stars or higher." Further, "In a normal rounding system, a site would round to the nearest half-star — up or down. In the case of Ted 2 [which was displaying 4.5 stars], then, we'd expect the rating to be rounded down to 4 stars. But Fandango rounded the 'ratingValue' [4.1] up. I pulled the number of stars listed on the page of each film in our sample of 437 (with at least one user review), as well as the ratingValue listed on the page's source. And I found that Fandango doesn't round a rating down when we'd mathematically expect that ... Fandango.com's rounding methodology, even if it was just an innocent bug, is a good example of why you should be skeptical of online movie ratings, especially from companies selling you tickets."

Comments

Rotten Tomatoes I've suspected of Payola

[sandbagger]

A friend is in the movie biz and his reaction to any criticism of the recent Star Trek reboots is Rotten Tomatoes is an objective measure. I can forgive him the logical error because he's in the industry and the financials are more important to him than say to you or I. So aggregated movie reviews that drive customer purchases to him indicate success.

However, as far as I know, Rotten Tomatoes never publishes its weighting formula
And it's opened by a movie studio.

This seems to me perfect for abuse.

Re:Rotten Tomatoes I've suspected of Payola

[gstoddart]

A friend is in the movie biz and his reaction to any criticism of the recent Star Trek reboots is Rotten Tomatoes is an objective measure.

Well, in part you need to remember these reviews are done by the entire movie-going public, and not just nerds.

So, it is entirely possible that you disagree with the movie-going public. But I don't think that means the reviews aren't honest.

I'm pretty sure TFA even says that both Rotten Tomatoes and IMDB consistently come out pretty close to one another.

I trust both Rotten Tomatoes and IMDB, because I've actually seen movies which are terribly reviewed, and which should have been.

But assessing your opinion of the Trek reboots vs the general opinion of them isn't really the point.

Rotten Tomatoes

[BillCable]

Rotten Tomatoes is the gold standard for movie quality measurement. Accept no substitutes.

Seriously, if someone is relying on Fandango to tell them if a movie is any good, they deserve to watch dreck.

Re:Ummm .... duh?

[Solandri]

Nearly all ratings are voluntary, and so suffer from self-selection bias. The measured ratings for general interest movie like Shawshank Redemption are typically lower than a special interest movie like Dark Knight (or Harry Potter, or Twilight, or Lord of the Rings) which appeals to a dedicated fanbase. The latter typically have a lot of fans who rate it highly just because it appeals to their group. That is, they rate it according to more lenient standard than they rate other movies, or they flat-out stuff the ballot box to try to get others to see it, to exaggerate the size of their interest group in hopes of encouraging more such movies to be made.

This sort of bias is so endemic to online polling that it's hopeless to try to correct it. All you can do is keep it in mind when you see ratings, and decide that Dark Knight is probably really around a 8.7, not a 9.0. And Shawshank Redemption must be really, really good if it's holding onto the #1 spot despite not appealing to a specific demographic.

I've seen some sites attempt to correct for this by assuming any "real" sample will be gaussian (have a distribution which falls on a normal curve). If the votes something receives are skewed away from guassian (e.g. clustered towards the high end), the site attempts to correct for this by skewing the score down. No idea how accurate or reliable that is, but it is being done in some places.

If you expect such ratings to 100% match your own opinion, you have an over inflated sense of self importance. ;-)

Rather than try to come up with one, universal rating which is implicitly applicable to everyone, Netflix's approach is probably more sensible. Depending on the movies you watch and the ratings you give them, Netflix builds up a profile of your preferences. They try to match your profile with that of other people who watched similar movies and gave them similar ratings, then makes recommendations based on what those other people watched. So if you hated Dark Knight, then there's a good chance you're not really into movies based on comic bo^H^H^H^H^H^H^H^Hgraphic novels, and so will downrate them for you personally.

This does raise some privacy implications, but on the balance I believe this is the more sensible approach to ratings. Giving up some privacy to greatly increase the signal-to-noise ratio of things like movie recommendations may be worth it in some cases. This also mostly corrects for self-selection bias, assuming your self-selection can be accurately measured.