Slashdot Mirror

← Back to Stories (view on slashdot.org)

Why You Should Be Suspicious of Online Movie Ratings (fivethirtyeight.com)

Posted by Soulskill on from the all-movies-are-great-except-for-the-terrible-ones dept.

An anonymous reader writes: Statistical news blog fivethirtyeight.com noticed some odd discrepancies in online movie ratings, which caused them to do some investigating. They found it was generally a bad idea to rely on such ratings, particularly from sites like Fandango. "When I focused on movies that had 308 or more user reviews, none of the 209 films had below a 3-star rating. Seventy-eight percent had a rating of 4 stars or higher." Further, "In a normal rounding system, a site would round to the nearest half-star — up or down. In the case of Ted 2 [which was displaying 4.5 stars], then, we'd expect the rating to be rounded down to 4 stars. But Fandango rounded the 'ratingValue' [4.1] up. I pulled the number of stars listed on the page of each film in our sample of 437 (with at least one user review), as well as the ratingValue listed on the page's source. And I found that Fandango doesn't round a rating down when we'd mathematically expect that ... Fandango.com's rounding methodology, even if it was just an innocent bug, is a good example of why you should be skeptical of online movie ratings, especially from companies selling you tickets."

21 of 184 comments (clear)

  1. Very suspicious.... by tonyyeb · · Score: 5, Funny

    Ted 2 [which was displaying 4.5 stars]... Was that out of 100?!

    1. Re:Very suspicious.... by Anonymous Coward · · Score: 2, Interesting

      Also remember that the minimum you are allowed to give on most sites is 1 out of 5, meaning even if everyone hated the movie it would have a "1". In other words people think of the "out of 5 stars" as a system of zero stars to 5 stars but in reality it is one star to five stars.

      Example:
      One person hates the movie and gives it "1"
      One person love the movie and gives it "5"
      The average is "3", which visually is not the middle,
      Despite only 50% of people liking the movie it appears as if 60% like the movie
      Also, stars have a positive connotation. 3 "stars" to me does not seem to represent that half of the people in the world hate it.

      This affect also skews the results the worse the reviews. It is nearly impossible to have only 1 star show.

    2. Re:Very suspicious.... by Gr8Apes · · Score: 2

      Ted 2 was actually 9 half stars, because no one could figure out how to rate it 0.

      --
      The cesspool just got a check and balance.
    3. Re:Very suspicious.... by Anonymous Coward · · Score: 2, Interesting

      The biggest problem is that we're using an all-positive scale to capture positive and negative opinions. The results will never be interpreted correctly because that's a feature of the system. "How much did you like this movie: a little, somewhat, a good amount, quite a bit, or a lot?" It's Colbert's "Great President or the greatest President?" bit, only with the expectation of being taken seriously. The RT method at least addresses this, to some extent.

      But there's still a lot missing in terms of magnitude. Without standard deviation bars, the star ratings don't tell you a whole lot. A bland but tolerable movie might get straight 3s, while something with strong niche appeal might get the same average with mostly 1s and 5s. There's a lot more useful data in the voting, especially if you can determine patterns in voting behavior and group movies based on who feels strongly about them.

      And the final issue is the skewed distribution of samples. Presumably, the worst of the bunch never see the light of day. The movies that don't even deserve one star and would be liked by nobody should never be released into the wild. Some still are, but many more aren't. So of course most movies, especially the ones seen by the most people, have high ratings, as seen in the comparison plot in the article. It is essentially a letter grading system where a 3 equates to a C; once you get much lower, you flunk out.

      What matters isn't how many people, on average, don't hate a movie, it's who likes it and how their opinion relates to you. Expect ratings systems to get a lot more personal in the future, essentially becoming targeted advertising.

  2. Rotten Tomatoes I've suspected of Payola by sandbagger · · Score: 4, Interesting

    A friend is in the movie biz and his reaction to any criticism of the recent Star Trek reboots is Rotten Tomatoes is an objective measure. I can forgive him the logical error because he's in the industry and the financials are more important to him than say to you or I. So aggregated movie reviews that drive customer purchases to him indicate success.

    However, as far as I know, Rotten Tomatoes never publishes its weighting formula
    And it's opened by a movie studio.

    This seems to me perfect for abuse.

    --
    ---- The above post was generated by the Turing Institute. Maybe.
    1. Re:Rotten Tomatoes I've suspected of Payola by gstoddart · · Score: 5, Interesting

      A friend is in the movie biz and his reaction to any criticism of the recent Star Trek reboots is Rotten Tomatoes is an objective measure.

      Well, in part you need to remember these reviews are done by the entire movie-going public, and not just nerds.

      So, it is entirely possible that you disagree with the movie-going public. But I don't think that means the reviews aren't honest.

      I'm pretty sure TFA even says that both Rotten Tomatoes and IMDB consistently come out pretty close to one another.

      I trust both Rotten Tomatoes and IMDB, because I've actually seen movies which are terribly reviewed, and which should have been.

      But assessing your opinion of the Trek reboots vs the general opinion of them isn't really the point.

      --
      Lost at C:>. Found at C.
    2. Re:Rotten Tomatoes I've suspected of Payola by gstoddart · · Score: 2

      At the end of the day, a review is a professional opinion.

      There are tons of movies the critics loved, but the public hated. Likewise, there's a bunch the critics hated, but the public loved. Then you get movies like Gigli, which everybody hated, and which IMBD says i the 50th worst movie of all time.

      You take your tastes, combine it with the reviews of critics and any other sources, and decide if you want to see it or not. Then you figure out how you really thought about it.

      Critical review often has nothing to do with box office success.

      --
      Lost at C:>. Found at C.
    3. Re:Rotten Tomatoes I've suspected of Payola by Flavianoep · · Score: 2

      I wonder if the film critics had some kind of beef with the cast or crew and took it out on them in the reviews, as there are plenty of similarly poor 'Sci-Fi' genre films scoring in the 40-55% range, which seems more reasonable, but I would have expected at least better than 30%.

      More likely they had a beef with Scientology.

      --
      Linux is for people who don't mind RTFM.
    4. Re:Rotten Tomatoes I've suspected of Payola by Ecuador · · Score: 2

      You don't seem to understand how the tomatometer works. Instead of it being similar to a 1-10 or star rating that gives an average over review scores, it is simply what percentage of reviewers were positive instead of negative. I find it quite easy to believe that only 1 out of 10 professional critics found the movie good overall. Now, rottentomatoes also averages the score of the same reviewers and for After Earth it gets 3.8/10. I suspect when you were expecting better than 30%, you were thinking something like average score, i.e. the reviewers did not thing it was good, but did not punish it by giving it a 1 out of 10 score. And that is indeed the case, on average they gave it a 2/5 or 4/10.
      Although I don't blindly follow critics, I do look at rottentomatoes and it usually gives at least an indication of what to expect. Sometimes even reviewers themselves are polarized, but since you get review "blurbs" if you scroll down you quickly figure out if things that reviewers like/dislike are the things you like/dislike. IMDB is also a great score, since I more or less agree with it for quite a lot more movies than the ones I disagree with it, which is not something I can say about many other ratings. Also note that IMDB has a genre bias, so for example a comedy getting 6.5 might actually mean a decent comedy when a drama getting 6.5 is probably not good. But you learn to use it and it helps narrowing down what to watch next. It certainly does a helluva better job compared to the VHS covers which was all I had to go with back in the 80's at the movie rental store...

      --
      Violence is the last refuge of the incompetent. Polar Scope Align for iOS
    5. Re:Rotten Tomatoes I've suspected of Payola by Sumus+Semper+Una · · Score: 2

      I've always felt that the way many people think about movie reviews is generally flawed. They seem to go to imdb or rottentomatoes and say something like "Oh, movie X got high user reviews. It must be a good movie." To me, what they should really be saying is, "Movie X got high user reviews. People who go to this kind of movie generally liked it." I think a good example of how these are different is the movie Jurassic World. It got 70%+ from both audiences and critics on rottentomatoes and more than 7 average rating on imdb. That doesn't mean it's an ground-breakingly good movie. Quite the contrary. If you saw the original Jurassic Park, you already saw a movie with the same basic plot and movie-going experience. If you go to that movie expecting to see an objectively wonderful film, you are going to be disappointed. If, however, you already know that the plot is basically the same and you want to see a shallow disaster/CG flick with dinosaurs, you're going to get exactly what you wanted and be mostly pleased with the result.

      Reviews are ALWAYS subjective and always will be. If you went and saw a movie based purely on how many people liked it without reading any of the complaints/praise of the movie, the problem is with you. If you vociferously disagree with a movie's popularity and think it should be less popular, again the problem is with you. You're looking for objective measurements that don't exist.

  3. Ummm .... duh? by gstoddart · · Score: 4, Insightful

    You mean people on the intertubes selling us stuff might not be honest about the reviews of the stuff they're selling us?

    IMDB, sure, I mostly trust them. Because a LOT of people review things on IMDB. Rotten tomatoes is an aggregator which includes a lot of sources. I mostly trust them to be independent and coming from real sources.

    But, really, ANY review site directly owned by a company trying to sell you stuff should probably a) be required to state their affiliation, and b) assumed to be engaging in a little corporate driven puffery.

    From the sounds of it, fandango (which I am admittedly not familiar with) is either more likely to give good reviews, or is deliberately skewing to better reviews to sell product.

    Are they uniformly rating all movies better (in which case they're just generally bad at reviews or too easily pleased), or if movies from specific studios get pushed up (in which case it's probably getting into a grey area).

    The problem with content on the internet is knowing who paid for it, and what other affiliations they have.

    Don't most video game sites also just give overly good reviews, often based on a product they've barely seen or have been prohibited from giving bad reviews?

    --
    Lost at C:>. Found at C.
    1. Re:Ummm .... duh? by gstoddart · · Score: 4, Insightful

      Well, look closely at that:

      Ratings: 9.0/10 from 1,515,627 users Metascore: 82/100
      Reviews: 4,406 user | 636 critic | 39 from Metacritic.com

      When 1.5 million people say they liked it, the rating is saying "lots of people liked this film"

      Now, contrast that with Shawshank Redemption, which is currently rated #1 on IMDB:

      Ratings: 9.3/10 from 1,539,960 users Metascore: 80/100
      Reviews: 3,773 user | 192 critic | 19 from Metacritic.com

      At least they tell you how they got there.

      Like Dark Knight or not, it was a wildly popular movie, which brought a very well known graphic novel to the screen. It also got Heath Ledger an Oscar, if you place any value on that.

      If you expect such ratings to 100% match your own opinion, you have an over inflated sense of self importance. ;-)

      --
      Lost at C:>. Found at C.
    2. Re:Ummm .... duh? by dpidcoe · · Score: 2

      The same goes for almost everything, I can safely assume that everyone likes cheese but that is not true, but if we put a rating to that it may come up with a 9 out of 10 or maybe more. But that does not mean everybody should buy it and eat it, it just means a lot of people like it, you have to try it for yourself to see if YOU like it.

      This is why I find the actual written reviews on IMDB to be vastly more useful than a number. To continue your analogy, if the people who give cheese 9/10 go on to describe why they like it (taste/texture/etc.), someone who hates cheese could skim the reviews and realize "everyone gave cheese 9/10 doesn't mind that their food is literally a giant mass of dead bacteria, this probably isn't a food that I'll like"

      Though oddly enough, I find the inverse of the scenario much easier. i.e. 9/10 people say they love cheese and it tastes great but don't give much feedback other than that. However, the one in ten posts a rant about how disgusting it is because it's literally just mold and bacteria. Generally the rants are more honest and on point about what it actually was that made them hate it, meaning I'd read the review from the guy who hates cheese and go "makes sense, but I don't care what it's made of as long as it tastes good so in my case it's worth a try".

    3. Re:Ummm .... duh? by Solandri · · Score: 4, Interesting
      Nearly all ratings are voluntary, and so suffer from self-selection bias. The measured ratings for general interest movie like Shawshank Redemption are typically lower than a special interest movie like Dark Knight (or Harry Potter, or Twilight, or Lord of the Rings) which appeals to a dedicated fanbase. The latter typically have a lot of fans who rate it highly just because it appeals to their group. That is, they rate it according to more lenient standard than they rate other movies, or they flat-out stuff the ballot box to try to get others to see it, to exaggerate the size of their interest group in hopes of encouraging more such movies to be made.

      This sort of bias is so endemic to online polling that it's hopeless to try to correct it. All you can do is keep it in mind when you see ratings, and decide that Dark Knight is probably really around a 8.7, not a 9.0. And Shawshank Redemption must be really, really good if it's holding onto the #1 spot despite not appealing to a specific demographic.

      I've seen some sites attempt to correct for this by assuming any "real" sample will be gaussian (have a distribution which falls on a normal curve). If the votes something receives are skewed away from guassian (e.g. clustered towards the high end), the site attempts to correct for this by skewing the score down. No idea how accurate or reliable that is, but it is being done in some places.

      If you expect such ratings to 100% match your own opinion, you have an over inflated sense of self importance. ;-)

      Rather than try to come up with one, universal rating which is implicitly applicable to everyone, Netflix's approach is probably more sensible. Depending on the movies you watch and the ratings you give them, Netflix builds up a profile of your preferences. They try to match your profile with that of other people who watched similar movies and gave them similar ratings, then makes recommendations based on what those other people watched. So if you hated Dark Knight, then there's a good chance you're not really into movies based on comic bo^H^H^H^H^H^H^H^Hgraphic novels, and so will downrate them for you personally.

      This does raise some privacy implications, but on the balance I believe this is the more sensible approach to ratings. Giving up some privacy to greatly increase the signal-to-noise ratio of things like movie recommendations may be worth it in some cases. This also mostly corrects for self-selection bias, assuming your self-selection can be accurately measured.

  4. Read the actual reviews by Anonymous Coward · · Score: 3, Insightful

    This is why I always read a sampling of the actual reviews, rather than going purely off ratings. We all have different tastes. Sometimes the things a reviewer points out about a movie as why they hate it are the very things I enjoy.

  5. Re:No Fix by U2xhc2hkb3QgU3Vja3M · · Score: 4, Informative

    I always liked the movie Starship Troopers because I saw it for what it is, an action/sci-fi/parody of military thinking/military logic taken to the extremes. The actors do a good job in their roles, the special effects are top-notch (for the time it was made) and the tongue-in-cheek humour of the story made it clear that the director didn't agree with the point of view of the book's author.

  6. Rotten Tomatoes by BillCable · · Score: 4, Interesting

    Rotten Tomatoes is the gold standard for movie quality measurement. Accept no substitutes.

    Seriously, if someone is relying on Fandango to tell them if a movie is any good, they deserve to watch dreck.

  7. This is why people follow particular reviewers... by xxxJonBoyxxx · · Score: 2

    This is why people follow particular reviewers, like Siskel OR Ebert back in the day.

  8. Some mod(s) will hate this, but.... by fyngyrz · · Score: 3, Insightful

    The reason I don't trust online movie ratings is because Dr. Who is popular. And Dr. Who is not something I can tolerate. In short, a large part of the populace's taste is not my taste, so those stars... meaningless to me.

    And sure enough, there are movies I loved that got poor ratings, and movies I thought were utter tripe that got high rankings.

    Same thing goes for Silkel and Ebert and that class of professional opinionators. Their taste is not my taste. So they can't be trusted by me.

    With this in mind, a site's questionable rounding of 4.1 to 4.5... not even on the radar.

    --
    I've fallen off your lawn, and I can't get up.
    1. Re:Some mod(s) will hate this, but.... by mwvdlee · · Score: 2

      Probably the most reliably movie ratings are the recommendations given by sites selling DVD/BluRays, as they have a financial interest in actually providing accurate results.
      Sites paid for by movie advertising have a financial interest in recommending whatever movie has the biggest advertising budget.

      --
      Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
  9. I Don't Trust Any Star Ratings by sudon't · · Score: 2

    I really don't trust any ratings done by "consumers" because the way I see people do ratings is, if they like it it, they give five stars, and if they don't like it, they give one star. Also, on seller sites such as eBay, or Discogs, it seems you're expected to give five stars to any seller who merely sends you the thing you ordered. Obviously, that leaves no room for a seller who goes above and beyond. If you give less then five stars, they'll flip out. I guess I can't blame them since that is the convention, now. Nevertheless, it makes the five-star rating system useless.
    Another surprise for me was when I found out the ratings on Netflix weren't generated by other viewers, but rather by Netflix guessing what I would think, based on my watching history.

    --
    -- sudon't

    Air-ride Equipped