Why You Should Be Suspicious of Online Movie Ratings (fivethirtyeight.com)
An anonymous reader writes: Statistical news blog fivethirtyeight.com noticed some odd discrepancies in online movie ratings, which caused them to do some investigating. They found it was generally a bad idea to rely on such ratings, particularly from sites like Fandango. "When I focused on movies that had 308 or more user reviews, none of the 209 films had below a 3-star rating. Seventy-eight percent had a rating of 4 stars or higher." Further, "In a normal rounding system, a site would round to the nearest half-star — up or down. In the case of Ted 2 [which was displaying 4.5 stars], then, we'd expect the rating to be rounded down to 4 stars. But Fandango rounded the 'ratingValue' [4.1] up. I pulled the number of stars listed on the page of each film in our sample of 437 (with at least one user review), as well as the ratingValue listed on the page's source. And I found that Fandango doesn't round a rating down when we'd mathematically expect that ... Fandango.com's rounding methodology, even if it was just an innocent bug, is a good example of why you should be skeptical of online movie ratings, especially from companies selling you tickets."
Ted 2 [which was displaying 4.5 stars]... Was that out of 100?!
A friend is in the movie biz and his reaction to any criticism of the recent Star Trek reboots is Rotten Tomatoes is an objective measure. I can forgive him the logical error because he's in the industry and the financials are more important to him than say to you or I. So aggregated movie reviews that drive customer purchases to him indicate success.
However, as far as I know, Rotten Tomatoes never publishes its weighting formula
And it's opened by a movie studio.
This seems to me perfect for abuse.
---- The above post was generated by the Turing Institute. Maybe.
You mean people on the intertubes selling us stuff might not be honest about the reviews of the stuff they're selling us?
IMDB, sure, I mostly trust them. Because a LOT of people review things on IMDB. Rotten tomatoes is an aggregator which includes a lot of sources. I mostly trust them to be independent and coming from real sources.
But, really, ANY review site directly owned by a company trying to sell you stuff should probably a) be required to state their affiliation, and b) assumed to be engaging in a little corporate driven puffery.
From the sounds of it, fandango (which I am admittedly not familiar with) is either more likely to give good reviews, or is deliberately skewing to better reviews to sell product.
Are they uniformly rating all movies better (in which case they're just generally bad at reviews or too easily pleased), or if movies from specific studios get pushed up (in which case it's probably getting into a grey area).
The problem with content on the internet is knowing who paid for it, and what other affiliations they have.
Don't most video game sites also just give overly good reviews, often based on a product they've barely seen or have been prohibited from giving bad reviews?
Lost at C:>. Found at C.
People really don't understand the extent of how much automated shilling is happening on major websites. Almost every single major company these days has easy access to software that will abuse online comment systems and online voting/rating systems to promote their product.
Go look at the major subs at reddit when any new movie is released and 90% of the comments will be blatant marketing garbage made by bots that upvote each other.
This is why I always read a sampling of the actual reviews, rather than going purely off ratings. We all have different tastes. Sometimes the things a reviewer points out about a movie as why they hate it are the very things I enjoy.
Is any rating on the internet not suspect? I sure don't trust any, especially if there's money involved.
So I can't trust everything I read on the Internet AND I have to apply critical thinking skills to determine whether a source of information, especially one possibly beholden to financial gain, is trustworthy?
The prible is not necessarily the ratings, but rather that the average rating (both mean and median) are artificially high, around 80 or so out of 100. This means pretty much any movie will appear to be good so someone not familiar with the formula. It's childish, and reminds me of when men rate women and just about every woman is a 9 or 10, when in reality those 10s would be excessively rare.
This also applies to video games and just about any rating online. And any site where people can log in the rate will have an innacurately high number of extreme ratings (1 and 10 stars) because people simply don't understand things like standard deviations or even averages.
Duh? Name me any situation (movies, software, music, cars, etc) where it isn't hysterically stupid (and to an obvious in-your-face degree) for reviewers (or their publisher) and sellers to have any sort of relationship.
Fandango reviews are for the kind of people who get their software from "app stores," i.e. pretty much the stupidest people that you will ever meet.
None of this means movie reviews are a bad idea, nor does it mean that buying tickets from a website are a bad idea. You just don't use the same fucking website for both things.
You knew this since you were 5 years old. You knew this when your parents or grandparents got a copy of "Consumer Reports" before shopping for anything expensive. The only time you don't know it, is when you make a conscious effort to become a stupider person (i.e. whenever a new iPhone or XBox comes out), presumably in the interests of world safety.
There is no fix for this. A good movie is subjective. Some times a bad movie is good when viewed through the right lens. A great movie might be reviewed poorly because the reviewer didn't understand some aspect, or might have had a bad day. Some people love sci fi, and hate romantic comedies, etc. Watch the trailer, compare that to your mental map of what the lead actors and directors have done in the past, and make the choice for yourself. If a friend says they hated/loved the movie, ask why. Add that to your mental map.
0.1 grams of trans fat equals 0 trans fat. Even 0.9! Because it's not >= 1, then, hey, it's still technically not a one.
If you see "The Asylum" in the credits, you know it's automatically a one-star movie, maybe two stars if they had a better budget.
I do have to say though, they did a really amazing job with Z Nation and I hope Netflix gets the second season soon.
Why You Should Be Suspicious of Online Movie Ratings
Because enjoyment is subjective.
systemd is Roko's Basilisk.
I just don't trust anything on the internet by default.
Be seeing you...
Crowd-sourcing opinion is a bad idea, unless you really want to like what other people like. If you have that kind of need for validation, then go ahead.
Better to find a handful of thoughtful reviewers whose opinions you trust.
You are welcome on my lawn.
I need approval from the herd before I am allowed to form an opinion.
"The story was great, the special effects were great and all the actors were great, but I hate actor XYZ so I'm rating this movie 1 star."
I noticed any movie with a score lower than say 6.8 is going to suck and not just because I or someone didn't like the style of movie, its just bad. On there you can usually tell the fake reviews when they are posted very early.
by TheSpoom (715771) Uncaring Linux user here. I have nothing to add to this but please continue. *munches popcorn*
Doh, which normal person would have a look at some rating of some foreigners before entering the cinema? These ratings are from raters for raters, so why do they take the effort to manipulate...
I lag
I am shocked! Shocked, i tell you!
Rotten Tomatoes is the gold standard for movie quality measurement. Accept no substitutes.
Seriously, if someone is relying on Fandango to tell them if a movie is any good, they deserve to watch dreck.
Ratings tell you - at best - how popular a movie is. Not how good it is. Same as the music charts. It has no correspondence to even sales figures, really. Measuring "sales this month" as opposed to "overall sales" is pretty much a nonsense if you think of it, if the person doesn't already own that product.
Ratings are about what's popular, what other people are liking now. For many, that's ALL they need to know because that's what they base their opinions on and then discuss with their peer group. For others, it means nothing at all.
I *GUARANTEE* you that if one of my school friends "recommends" a page to me, that I'll have no interest. I *GUARANTEE* you that the films that my friends or family like do not intersect in any meaningful or predictable way with my own personal preferences. It's just that simple.
However, I refuse to subscribe to the "You have to watch it first" kind of argument if that means parting with any money whatsoever. If I have to watch it to know whether or not I'll like it, I'm not going to pay to do so at the cinema on or DVD.
But it is possible, via trailers, reviews, ratings, comments, research and the necessary pinch of salt, to work out whether a movie is one you'll enjoy. You can still be wrong, but you can get a good indication.
I think the same of Steam's review system, There are game on there I love that get slated and games I hate that get 10/10 and Mostly Positive reviews. It doesn't affect my decision to buy or not. Give me a demo, though, or a gameplay trailer and that will help sway me one way or the other.
I've wanted "Grey Goo" since I first saw it. They did a free weekend this weekend, so I installed it. I'm pretty Meh about it now. But I wouldn't have known that without playing it, and wouldn't have bought it based on the Mostly Positive reviews alone.
Prison Architect, on the other hand, I thought was going to be rubbish. And then I played it. And now I can't stop. And no doubt reviewers and other players think just the opposite.
But, sorry, movie ratings are worthless. Ratings are worthless and ratings of something that you have to pay to prove them wrong are even worse. I've never paid them any attention. At least with a bit of music you can tell if it's going to be good for free, legally, before you buy it.
This is why people follow particular reviewers, like Siskel OR Ebert back in the day.
...when I checked out the weekend movies. Several of them were sufficiently unknown to me that I checked the MetaCritic number.
It seemed quite a coincidence that several came out to an 81% rating. Which sounds to me a lot like a four-star rating that could be rounded up to a 4.5-star rating as described in the post. I guess our natural inclination to distrust the ratings of someone who is trying to sell us something may be correct in this instance.
Aside from the rounding thing i suspect this is more a combination of (reasonable) bias and people sucking at rating than some active attempt at deception. ("You have attributed conditions to villainy...")
If someone goes out and sees a movie or plays a game and thinks it's awesome they're much more likely to go online and rate it highly, possibly 4 stars but more likely 5 stars (because people tend to extremes, especially when feeling emotional.) If someone sees a movie or plays a game and thinks it sucks they're much more likely to go online and lambast it, giving it 1 or 2 stars. (I believe 2 stars is the popular option for people who want to appear as if they're giving it a fair shake.)
How many people go see a movie, decide it's kinda mediocre, and feel really compelled to rush online and tell everyone about how they really don't feel much one way or the other?
Compounding this is the fact that, despite all the jokes to the contrary, the average person isn't stupid. Most of us have some kind of idea what kind of movies we like and are able to make a fairly reasonable guess as to how much we will enjoy an upcoming movie. And most of us choose not to spend money going to a theatre to see a movie we don't like.
So for new movies, the ones for which the ratings have the most financial impact and are the most closely watched, most people go to see it because they expect to enjoy it, and most of the time they're right, so they give it high ratings. And if they're _wrong_, they're probably going to be pretty pissed about it and may give correspondingly bad ratings. There are probably also a lot of people who go to a movie expecting it to be decent but not outstanding, and their expectations of decent mediocrity are met. But they're not going to bother taking the time to submit a rating like that.
I would expect that the majority of mediocre ratings would come at least a year after a movie's been out, when people have the chance to catch it on TV or online or rent it for a lark. There are lower expectations and less investment and therefore people will give something random a try and be okay giving it a median review. And if it's an online system, especially one that will make recommendations based on your ratings, the threshold of effort is low enough that the promise of some (minuscule) reward is enough to provide the motivation to leave a mediocre rating.
This Space Intentionally Left Blank
Star Ratings Meaning according to XKCD
Not to dispute that a site like Fandango will lie for money, but for the data from the Netflix challenge several years ago - where they made available an anonymized sample of peoples' movie ratings - the mean was 3.8 (https://www.igvita.com/2006/10/29/dissecting-the-netflix-dataset/), not the 3.0 one might expect for a random distribution over the range 1 to 5.
Upon reflection, this makes sense as people don't watch movies randomly - they watch what they think might be good and avoid what they think will be bad. I know I had trouble thinking of a movie that I had watched that I would rate 1 (except for "The Master of Disguise" to which I took my daughter when she was very young).
Wait, you mean someone on teh intarwebz might LIE?
Say it isn't so!!
Just cruising through this digital world at 33 1/3 rpm...
Bambi sucks! It was all cartoony and shit and there was zero in it for scifi fans to like. They didn't even have a single car chase in the whole movie. I'd give zero stars, but they don't let you give less than one star.
1 Star - Bambi sucks.
A lot of rating systems use Likert scales which are useful when ratings are distributed across the extremes (symmetry), my guess is there are a lot of ratings in the middle of the scale (2 to 3 stars, etc) and fewer ratings at the bottom of the scale
Is there a "skewness" value for movie reviews?
No matter how the rounding is made, as long as it is monotonous, a 4.5 star will always have a better rating than a 4 star.
It is also obvious that ratings are relative. If 4-4.5 is average then so be it. Not very accurate but it gives you an idea.
The problem is who is behind the ratings. Shills are an obvious cause of bogus ratings, however, even honest reviews may be troublesome. A typical thing is that when looking at ratings, you are usually looking for the best. Conversely, reviewers tend to give 5 stars most of the time while punishing the movie they didn't like. As a result, you are more likely to find widely acceptable movies than really good ones. If you wondered why "The Shawshank Redemption" is on top of the IMDB 250 while you can think of plenty of better movies, you have your answer.
Better to find a handful of thoughtful reviewers whose opinions you trust.
Haven't found any yet, at least when it comes to movies. Seriously, I haven't found a reviewer yet that I agree with consistently. And frankly I don't really care about specific opinions. Basically I just want to know A) is the movie of generally good quality and well told and B) is the premise and story likely to interest me.
I've found Rotten Tomatoes to be an ok (though imperfect) proxy evaluation of general quality though it tends to overrate certain types of movies. For example Rotten Tomatoes tends to overrate animated movies for some reason. Animated movies (esp Disney/Pixar) that are merely good often get 90%+ ratings when a 75%-85% might be more appropriate. For instance Wall-E is a very good movie but no way in hell should it have a 96% - it just wasn't THAT good. 96% should be for stuff that wins academy awards for best picture. Frozen got 89% and even though it is wildly popular, I think its score is also far too high. It was an ok-good (60-75%) but not great movie. But I'm aware of that bias so I can adjust mentally for it.
But Rotten Tomatoes isn't good at telling me if *I* will like the movie. There are some great quality movies that I find boring and some admittedly second-rate movies that I very much enjoy. I walked out of the movie The Aviator for example because it bored the crap out of me even though I'll acknowledge it is a pretty well made film and probably is rated appropriately on Rotten Tomatoes.
FTA:
"You decided to check out 'Fantastic Four' ... Fandango users thought it was good! Over 7,000 people had reviewed it, and it had an average of 3 out of 5 stars. This is going to be a decent movie."
This article must be targeted at people who would see a 3/5 rating for a movie and conclude that the users "thought it was good". I would interpret a 3/5 from 7000 users (or a 6/10 on IMDB) as an indication that the movie probably sucks. No way would I pay theater prices to go see it based on a 3/5.
A big budget Hollywood action flick like FF is likely going to earn 3 stars just for the special effects. It will get a few high ratings from people who went to see it simply for that purpose. People would generally know what to expect in such a film and those that would rate it '0' or '1' probably wouldn't go see it in the first place. You therefore need to expect a little ratings inflation because you're dealing with people who paid to see it, not a random sample.
On a side note, the 5 star system really doesn't have enough granularity. How can any movies earn '0's or '5's? That's like saying a movie couldn't possibly be any worse or any better and no other movie could possibly be worse/better. That makes no sense.
Even if the acting was lame and the plot was full of holes, I might give a movie a 2/5 for a few bright spots, but I would consider that a very bad rating.
This is why people follow particular reviewers, like Siskel OR Ebert back in the day.
I only found Ebert helpful because I almost always seemed to disagree with the guy. Nothing against him personally but whenever I watched the show I routinely found myself having a very different opinion if I had also seen the movie.
.
All online ratings and reviews are played with and are dubious, at best.
There are some good ratings/reviews, such as this review of a book of random numbers.
But, for the most part, online reviews should be viewed with a good deal of skepticism.
Anything with more than 2 options is too much granularity for a movie rating system.
The only reviews I trust: http://www.mrcranky.com/
His review of The Tigger Movie is pure gold!
Why You Should Be Suspicious of Online...
Shh! Stop. Say no more. Right there with you.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
the sky is blue and water is wet. More at 11.
If it ain't broke, don't fix it.
... that the law of averages does still function, that the internet is mostly wrong, that I hate everybody who isn't me, and that its best to just make my own opinions about things.
Rotten Tomatoes has been a subsidiary of Flixster since 2010, which was in turn purchased by Warner Bros. in 2011.
But does Warner Bros. Pictures have any more influence over Rotten Tomatoes than it has over CNN?
I liked Ebert so much, I stopped watching movies altogether and only read his reviews of them!
Now I have to read wikipedia. Oh well.
"First they came for the slanderers and i said nothing."
The article isn't questioning the reviews themselves at all. It questioned Fandango's rounding method, which presents higher averages than other sites would with the same review ratings.
I've seen the same thing happen with product reviews, a ton of "its a great product, buy it" reviews, and a bunch of "its terrible, piece of crap" reviews, but very few reviews in the middle with actual details. Presumably the positive ones are from the seller or a PR firm for the seller and the negative reviews are from competitors. The fact that it may be catching up to movie reviews shouldn't be surprising. What I do find surprising is that this isn't prosecuted as some kind of fraud or false advertising. I guess the appropriate agencies are too busy doing "important" things, like going after firework manufactures because some drunk/high moron started a sentence with "watch this" and a lighter.
There, fixed it...
The reason I don't trust online movie ratings is because Dr. Who is popular. And Dr. Who is not something I can tolerate. In short, a large part of the populace's taste is not my taste, so those stars... meaningless to me.
And sure enough, there are movies I loved that got poor ratings, and movies I thought were utter tripe that got high rankings.
Same thing goes for Silkel and Ebert and that class of professional opinionators. Their taste is not my taste. So they can't be trusted by me.
With this in mind, a site's questionable rounding of 4.1 to 4.5... not even on the radar.
I've fallen off your lawn, and I can't get up.
I really don't trust any ratings done by "consumers" because the way I see people do ratings is, if they like it it, they give five stars, and if they don't like it, they give one star. Also, on seller sites such as eBay, or Discogs, it seems you're expected to give five stars to any seller who merely sends you the thing you ordered. Obviously, that leaves no room for a seller who goes above and beyond. If you give less then five stars, they'll flip out. I guess I can't blame them since that is the convention, now. Nevertheless, it makes the five-star rating system useless.
Another surprise for me was when I found out the ratings on Netflix weren't generated by other viewers, but rather by Netflix guessing what I would think, based on my watching history.
-- sudon't
Air-ride Equipped
I always used to laugh out loud when I saw a quote in the paper about a film and it was from Travers. Pretty much that told me to avoid the film.
You can tell he's the critic of last resort that studios call upon when they have a bomb on their hands.
On a side note, the 5 star system really doesn't have enough granularity.
Just how accurately can you determine a movie's quality? Take a list of 100 random movies. Now sort them from best to worst. Repeat the process a few more times. Now aggregate the data to make a comprehensive ordered list of all movies. You'll end up with inconclusive groupings that are always above or below other groups and a few outliers that can go either way at the boundaries. Nothing is gained from having significantly greater precision than the accuracy of the measurement. In fact, it can be misleading, leading you to believe that the result is more accurate than it really is. 2 or 3 levels is probably sufficient for something like movie ratings, maybe up to 10 if you can force yourself to be very objective. More than that though is getting into the noise.
Movie reviews/ratings should be treated the same way as videogame reviews/ratings, with a grain of salt. You only need to skim the limited amount of specific details present in the reviews to form your own opinion while completely ignoring whatever score the reviewer gave. Too many reviews, positive or negative, are tilted by the author's personal history with the medium or genre and will never match your own.
-==- Buy a Mac and leave me alone!
I've been with Netflix since 2006 and have watched and rated around 2000 movies so they have some idea of what I like. Most movies they list for me are below 3 stars and rarely will a film have over 4 stars. If Netflix shows me a movie with one star and I watch it anyway it's usually terrible. Not sure what kind of system they use but they did have a contest for people to come up with a rating algorithm many years ago. I really trust the Netflix rating when deciding what to rent.
It's 2015.
If people at this point don't understand that there are PAID individuals (by the thousands) who will fill forums with whatever you want, vote whatever you want on whatever online poll, and generally astroturfing the shit out of anything of any actual value whatsoever (and a fairly large amount of stuff that DOESN'T matter, as well), then they're so oblivious and stupid that they shouldn't be allowed to care for themselves, much less handle money or purchases.
-Styopa
"Why You Should Be Suspicious of Online Reviews"
FTFY
mark kermode and simon mayo
If you read the comments at imdb, they all say things like - this movie looks stupid im never going to pay to see it i would give it 0 stars if i could - ...
Meaning they rated it but never watched it.
... that all online rating systems are abused, right? Even Slashdot's comment system. There is no more perfect way to control people than to feed them the appropriate data, and online reviews make this trivial when people are willing to trust the word of complete strangers.
While, as is the case with any film critic, one is bound to disagree with at least some of the ratings a given critic gives to the movies he reviews, the golden skill of Ebert was that reading his reviews would tell you whether you would like the movie, not just whether he did. While I often agreed with his ratings, there were numerous times I didn't, and in those cases I knew that I'd rate the movie differently merely from reading the review, without having seen the film yet (and every time I did follow up by watching it, my expectations were confirmed).
"Politicians and diapers must be changed often, and for the same reason."
These broad statistics like average rating and stars are really a throwback to a time when processors were too slow for techniques like cooperative filtering , which in this context, essentially uses people to recommend movies and movies to recommend people back and forth for a more accurate estimator. The system is simultaneously able to find people that like the same kind of movies based on a common set of features and movies that have those features. The idea being that if you're a person that has preferences similar to someone that has rates a film highly then you will tend to also rate it highly. But the problem for simple rating sites is that these advanced techniques require you to establish an identity to differentiate yourself from "the average" and contribute by rating, which can be tedious while systems like Netflix can probably do more by inferring you rating by repeat viewing, tendency to scan, etc. However, once identity is established, then even fraudulent types of behavior can be dealt with. It's surprising that imdb, rotten tomatoes, fandango, etc cling to such inaccuracies, but I could find no reference to such techniques, though admittedly I didn't spend a long time looking.
.....that you didn't fake yourself.
What my neighbor likes or dislikes, may not necessarily coincide with my preferences. I have seen panned movies that were great, and highly rated movies sucked. If the ratings system in place now were less subjective, rather than objective (with the object being to accurately reflect quality of the acting, directing, writing (and storyline), and editing), I might lend some credence to a rating. Even so, ratings can mislead. How about replacing ratings with ACCURATE TRAILERS!
Self-importance and self-indulgence is the root of ALL evil.
Anything with more than 2 options is too much granularity for a movie rating system.
Three would work: thumbs up, thumbs down and meh. Which you could translate as worth seeing again, not worth seeing at all and OK to see once.
To have a right to do a thing is not at all the same as to be right in doing it
hollywoodbitchslap.com FTW. Comments are limited length and the site doesn't seem to be as active as it was a few years ago, but I still rely on it for reviews.
.
== WolfriderV6 == I'm willing to admit that *I just might* be wrong... Are you??
:) did not pass my litmus test. Four lions review, kermode called it when he said it was elegiac and filled with pathos. a sense of poignancy. Some people only see comedy in that movie, I also saw a great deal of sadness, :) if a review site doesn't see sadness in four lions, it's not for me :)
5/5: BEST. MOVIE. EVAR.
4/5: I liked it, and I wouldn't mind watching it again.
3/5: I liked it, but once was enough.
2/5: I didn't like it.
1/5: WORST. MOVIE. EVAR.