Why You Should Be Suspicious of Online Movie Ratings (fivethirtyeight.com)

An anonymous reader writes: Statistical news blog fivethirtyeight.com noticed some odd discrepancies in online movie ratings, which caused them to do some investigating. They found it was generally a bad idea to rely on such ratings, particularly from sites like Fandango. "When I focused on movies that had 308 or more user reviews, none of the 209 films had below a 3-star rating. Seventy-eight percent had a rating of 4 stars or higher." Further, "In a normal rounding system, a site would round to the nearest half-star — up or down. In the case of Ted 2 [which was displaying 4.5 stars], then, we'd expect the rating to be rounded down to 4 stars. But Fandango rounded the 'ratingValue' [4.1] up. I pulled the number of stars listed on the page of each film in our sample of 437 (with at least one user review), as well as the ratingValue listed on the page's source. And I found that Fandango doesn't round a rating down when we'd mathematically expect that ... Fandango.com's rounding methodology, even if it was just an innocent bug, is a good example of why you should be skeptical of online movie ratings, especially from companies selling you tickets."

Comments

Very suspicious....

[tonyyeb]

Ted 2 [which was displaying 4.5 stars]... Was that out of 100?!

Rotten Tomatoes I've suspected of Payola

[sandbagger]

A friend is in the movie biz and his reaction to any criticism of the recent Star Trek reboots is Rotten Tomatoes is an objective measure. I can forgive him the logical error because he's in the industry and the financials are more important to him than say to you or I. So aggregated movie reviews that drive customer purchases to him indicate success.

However, as far as I know, Rotten Tomatoes never publishes its weighting formula
And it's opened by a movie studio.

This seems to me perfect for abuse.

Re:Rotten Tomatoes I've suspected of Payola

[gstoddart]

A friend is in the movie biz and his reaction to any criticism of the recent Star Trek reboots is Rotten Tomatoes is an objective measure.

Well, in part you need to remember these reviews are done by the entire movie-going public, and not just nerds.

So, it is entirely possible that you disagree with the movie-going public. But I don't think that means the reviews aren't honest.

I'm pretty sure TFA even says that both Rotten Tomatoes and IMDB consistently come out pretty close to one another.

I trust both Rotten Tomatoes and IMDB, because I've actually seen movies which are terribly reviewed, and which should have been.

But assessing your opinion of the Trek reboots vs the general opinion of them isn't really the point.

Ummm .... duh?

[gstoddart]

You mean people on the intertubes selling us stuff might not be honest about the reviews of the stuff they're selling us?

IMDB, sure, I mostly trust them. Because a LOT of people review things on IMDB. Rotten tomatoes is an aggregator which includes a lot of sources. I mostly trust them to be independent and coming from real sources.

But, really, ANY review site directly owned by a company trying to sell you stuff should probably a) be required to state their affiliation, and b) assumed to be engaging in a little corporate driven puffery.

From the sounds of it, fandango (which I am admittedly not familiar with) is either more likely to give good reviews, or is deliberately skewing to better reviews to sell product.

Are they uniformly rating all movies better (in which case they're just generally bad at reviews or too easily pleased), or if movies from specific studios get pushed up (in which case it's probably getting into a grey area).

The problem with content on the internet is knowing who paid for it, and what other affiliations they have.

Don't most video game sites also just give overly good reviews, often based on a product they've barely seen or have been prohibited from giving bad reviews?

Re:Ummm .... duh?

[gstoddart]

Well, look closely at that:

Ratings: 9.0/10 from 1,515,627 users Metascore: 82/100
Reviews: 4,406 user | 636 critic | 39 from Metacritic.com

When 1.5 million people say they liked it, the rating is saying "lots of people liked this film"

Now, contrast that with Shawshank Redemption, which is currently rated #1 on IMDB:

Ratings: 9.3/10 from 1,539,960 users Metascore: 80/100
Reviews: 3,773 user | 192 critic | 19 from Metacritic.com

At least they tell you how they got there.

Like Dark Knight or not, it was a wildly popular movie, which brought a very well known graphic novel to the screen. It also got Heath Ledger an Oscar, if you place any value on that.

If you expect such ratings to 100% match your own opinion, you have an over inflated sense of self importance. ;-)

Re:Ummm .... duh?

[Solandri]

Nearly all ratings are voluntary, and so suffer from self-selection bias. The measured ratings for general interest movie like Shawshank Redemption are typically lower than a special interest movie like Dark Knight (or Harry Potter, or Twilight, or Lord of the Rings) which appeals to a dedicated fanbase. The latter typically have a lot of fans who rate it highly just because it appeals to their group. That is, they rate it according to more lenient standard than they rate other movies, or they flat-out stuff the ballot box to try to get others to see it, to exaggerate the size of their interest group in hopes of encouraging more such movies to be made.

This sort of bias is so endemic to online polling that it's hopeless to try to correct it. All you can do is keep it in mind when you see ratings, and decide that Dark Knight is probably really around a 8.7, not a 9.0. And Shawshank Redemption must be really, really good if it's holding onto the #1 spot despite not appealing to a specific demographic.

I've seen some sites attempt to correct for this by assuming any "real" sample will be gaussian (have a distribution which falls on a normal curve). If the votes something receives are skewed away from guassian (e.g. clustered towards the high end), the site attempts to correct for this by skewing the score down. No idea how accurate or reliable that is, but it is being done in some places.

If you expect such ratings to 100% match your own opinion, you have an over inflated sense of self importance. ;-)

Rather than try to come up with one, universal rating which is implicitly applicable to everyone, Netflix's approach is probably more sensible. Depending on the movies you watch and the ratings you give them, Netflix builds up a profile of your preferences. They try to match your profile with that of other people who watched similar movies and gave them similar ratings, then makes recommendations based on what those other people watched. So if you hated Dark Knight, then there's a good chance you're not really into movies based on comic bo^H^H^H^H^H^H^H^Hgraphic novels, and so will downrate them for you personally.

This does raise some privacy implications, but on the balance I believe this is the more sensible approach to ratings. Giving up some privacy to greatly increase the signal-to-noise ratio of things like movie recommendations may be worth it in some cases. This also mostly corrects for self-selection bias, assuming your self-selection can be accurately measured.

Re:No Fix

[U2xhc2hkb3QgU3Vja3M]

I always liked the movie Starship Troopers because I saw it for what it is, an action/sci-fi/parody of military thinking/military logic taken to the extremes. The actors do a good job in their roles, the special effects are top-notch (for the time it was made) and the tongue-in-cheek humour of the story made it clear that the director didn't agree with the point of view of the book's author.

Rotten Tomatoes

[BillCable]

Rotten Tomatoes is the gold standard for movie quality measurement. Accept no substitutes.

Seriously, if someone is relying on Fandango to tell them if a movie is any good, they deserve to watch dreck.