Slashdot Mirror

← Back to Stories (view on slashdot.org)

Netflix Is Looking To Pay Someone To Watch Netflix All Day

Posted by samzenpus on from the how-do-you-hire-that-which-has-no-life dept.

An anonymous reader writes with news about a dream job for binge-watching couch potatoes in the UK. Ploughing through your new favourite series on Netflix is something you probably enjoy doing after a working day, but what if it was your working day? You see, Netflix has a fancy recommendation engine that suggests movies and shows you might like based on your prior viewing habits. To do that successfully, it needs information from a special group of humans that goes beyond the basics like genre and user rating. "Taggers," as they're known, analyse Netflix content and feed the recommendation engine with more specific descriptors if, for example, a film is set in space or a cult classic. In short, these people get paid to watch TV all day, and Netflix is currently hiring a new tagger in the UK.

63 of 86 comments (clear)

  1. Seems excessive... by Valvar · · Score: 2

    Why not just let the users do the job? Cheaper, faster and easier...

    1. Re:Seems excessive... by Anonymous Coward · · Score: 1

      Because that has worked out so well for IMDB and TMDB. Try looking at their genres sometime, especially ones like "comedy" where if there is anything even vaguely humorous no matter how passing or unintentional the movie gets classed as a comedy.

    2. Re:Seems excessive... by Anonymous Coward · · Score: 5, Insightful

      Because they want shills to recommend shows nobody else wants to see so they'll get kickbacks from the clueless publishers.

    3. Re:Seems excessive... by Anonymous Coward · · Score: 1

      mechanical turk crossed with pagerank

    4. Re:Seems excessive... by K.+S.+Kyosuke · · Score: 4, Insightful

      I've heard about this thing called metamoderation. Rumor has it that it is already being used on some sites to weed out garbage user inputs...

      --
      Ezekiel 23:20
    5. Re:Seems excessive... by fuzzyfuzzyfungus · · Score: 4, Insightful

      Why not just let the users do the job? Cheaper, faster and easier...

      Generally, when somebody is paying for what it sounds like they could get for free, or even get paid for, there is good reason to suspect that the job description is either underplaying the exact level of difficulty and/or boredom involved, or that somebody has already learned the hard way that what they can get for free isn't exactly what they want.

      In this case, I'd be inclined to suspect that the job is closer to being a 'machine vision' substitute for stuff that machines can't yet see or which it wouldn't be cost-effective to have an expensive analyst cobble together a ruleset and then cheap labor check for mistakes when you could just have cheap labor classify it (eg. 'movies set in space' is probably something that you could achieve reasonable accuracy on, if you do some futzing with detecting starfields and common flavors of "rocket thruster jet of flame"; but you'd have your false positives and false negatives from things in space that happen mostly inside spacecraft, and things not in space that happen to involve looking at the sky more than usual, and so on).

      It's probably a hell of a grind, actually, given that (unlike, say, being a film critic or some film-studies culture critic type) Netflix is going to want everything ground through and tagged on a variety of parameters, not just the stuff you happen to be a geek about, or the stuff that's worth watching, or what have you. It wouldn't much surprise me if, for efficiency's sake, they have you monitoring more than one stream at a time, or working in faster-than-real time, or a combination. You can probably extract the data they want rather faster than you can enjoy the program, even if it is one you like.

    6. Re:Seems excessive... by NotDrWho · · Score: 3, Funny

      Because it's like my grandpa used to say about volunteers: "Sonny, you can't fire someone for doing a bad job if they're doing it for free. Also, don't trust the jews or coloreds with money."

      --
      SJW's don't eliminate discrimination. They just expropriate it for themselves.
    7. Re:Seems excessive... by Chris+Mattern · · Score: 1

      Why not just let the users do the job? Cheaper, faster and easier...

      And subject to massive trolling by malicious users...

    8. Re:Seems excessive... by SQLGuru · · Score: 4, Funny

      But it only works if you participate.......which might explain Slashdot.

    9. Re:Seems excessive... by fishybell · · Score: 1

      Trust me, it's a fad at best.

      --
      ><));>
    10. Re:Seems excessive... by geminidomino · · Score: 2

      As opposed to amazing incompetence of the "taggers" doing it now.

    11. Re:Seems excessive... by nabsltd · · Score: 2

      Because that has worked out so well for IMDB and TMDB. Try looking at their genres sometime, especially ones like "comedy" where if there is anything even vaguely humorous no matter how passing or unintentional the movie gets classed as a comedy.

      "Genre" isn't really a problem on IMDB, as users can't directly set that. I believe you are thinking of "plot keywords", which are really nothing but tags, and have become silly.

      How does a "loud shirt" have anything to do with the plot of the listed titles?

    12. Re:Seems excessive... by AvitarX · · Score: 1

      The thing is, if "loud shirt" is irrelevant, than the back-end recommendation algorithm can throw it out.

      There very well may be weird unexpected things that corralate to liking various movies, and they may not be causal at all, Netflix only needs correlation though.

      --
      Wow, sent an e-mail as suggested when clicking on "use classic" banner, and got a fast response that addressed my msg
    13. Re:Seems excessive... by gauauu · · Score: 3, Insightful

      Why not just let the users do the job? Cheaper, faster and easier...

      I recently read an article (I wish I could find it again) that describes how and why Netflix does this. Basically, they train their viewers to watch for many certain qualities and attributes of movies, which are then tagged and categorized to set up their recommendation and category systems.

      For example, they might use a few movies as a baseline for a ratings system so their viewer/ranker staff are on the same page ("on a scale of 1-10, how sweet and sappy is this movie? Does it have a strong female lead? Does it feature cute animals?"), then the viewers watch the film and fill out extensive and standardized tagging information about it, which they build their ratings from.

      The article describes it in much better detail, but it's clear that the level of standardization and depth in their tagging and categorizing is beyond what you'd be able to get from the general public.

    14. Re:Seems excessive... by vux984 · · Score: 4, Informative

      Generally, when somebody is paying for what it sounds like they could get for free, or even get paid for, there is good reason to suspect that the job description is either underplaying the exact level of difficulty and/or boredom involved, or that somebody has already learned the hard way that what they can get for free isn't exactly what they want.

      Bingo.

      You won't watch what you want. You probably won't have enough time to finish watching anything... 99% tagging accuracy for comedy, sci-fi, action, etc, etc, can be assessed within the first half.

      And I can't think of much that would need to see the whole movie to tag correctly, except for "twist ending".

      For TV series you'll probably just watch a few parts of a few random episodes, and then move on.

      Your notion that you'd do it watching multiple streams is quite likely too -- and sped up... probably even skipping... watch 5 minutes, skip 5... watch 5 ...

      Because as you say, your job is to tag movies, not critique them. You'll only spend as much time with a movie as you need to tag it accurately, and that is far less than the 90-150 minutes it would take to watch it from start to finish.

      As an aside, another "dream job" that is truly abysmal in practice is "video game tester".

    15. Re:Seems excessive... by netsavior · · Score: 1

      #cultclassic #sarcasm #possibleJapaneseInfluence #dirtyBibleQuotes

    16. Re:Seems excessive... by RivenAleem · · Score: 1

      It depends on whether you embrace it or not. There are some people for whom game tester is an amazing job. It also depends on the games being tested. There is a small clip of a demo at a con done by one of the game testers.

      http://imgur.com/gallery/5XLUt...

      When you are this good, even game testing can be fun, I guess.

    17. Re:Seems excessive... by Brulath · · Score: 1

      It might be fun for them to show off their ability at playing the game after the fact, but that doesn't change the reality of testing not living up to the dream job standard when you're actually doing the work part. It's not the worst job available, but it's not "dream job" material in the overwhelming majority of cases (though it might be a stepping stone to a dream job, sometimes).

    18. Re:Seems excessive... by Lord+Lemur · · Score: 1

      #meta

  2. highlighting UK/IE cultural specificities by OzPeter · · Score: 2

    highlighting UK/IE cultural specificities and taste preferences.

    Given the predilections of UK politicians, this could mean working with some weird shit. OTOH if you're from the UK/IE then you are probably already used to that weird shit.

    --
    I am Slashdot. Are you Slashdot as well?
    1. Re:highlighting UK/IE cultural specificities by Anonymous Coward · · Score: 1

      Just add a little weed and you'll be fine watching Netflix all day. Not that it destroys the motivation of all users, but it does make watching movies and TV more enjoyable.

  3. Don't get me wrong by Anonymous Coward · · Score: 5, Interesting

    I like spending the occasional 1 - 4 hours watching a few episodes in a row or maybe two movies, but doing that 8 hours a day / 5 days a week? Enjoyment soon turns into torture, hope they get paid good.

    1. Re:Don't get me wrong by Anonymous Coward · · Score: 1

      They get a free netflix subscription to use at home.

  4. Re:IMDB is full of descriptors by BorgDrone · · Score: 3, Informative

    Probably because of all the lawsuits that IMDb would file over that.

  5. Netflix rating engine sucks by uigin · · Score: 4, Insightful

    Given how poor the Netflix rating engine is surely their money'd be better spent hiring a programmer? I mean, how about not suggesting to me the movie I've just watched? (Low hanging fruit?)

    1. Re:Netflix rating engine sucks by swb · · Score: 3, Interesting

      I thought they had a big contest where it was a big deal to beat the then-current suggestion engine by 10% because the current engine was supposed to be so good.

      IMHO the bigger problem is that streaming has a huge amount of shit associated with it and they will suggest shit movies which makes it appear that the suggestion engine doesn't work.

      My guess at this point given all they do to hide/obfuscate how crummy their streaming catalog is they don't really care about the suggestion engine anymore.

    2. Re:Netflix rating engine sucks by Anubis+IV · · Score: 2

      Teams of researchers from around the globe competed for the $1,000,000 Netflix Prize way back in 2009, that would be awarded to the team that managed to improve the algorithm by even 10%. It took them the better part of a year to accomplish it, and you seem to think that a lone programmer can just get in there and knock out a lot of low-hanging fruit to substantially improve things?

      I don't deny that there's always room for improvement (such as the example you provided), but suggesting that it can all be fixed by "hiring a programmer" is a bit naive.

    3. Re:Netflix rating engine sucks by PRMan · · Score: 1

      Netflix's rating system is worse than ever. It recently said that I would like "Amber Alert" at a 4.8 out of 5. I thought, "Not likely", but I tried it anyway. I turned it off in 10 minutes and rated it a 1 (which for me means couldn't finish). How on earth did it think I (or anyone else) would like that horrible movie with ugly, stupid people screaming at each other the whole time?

      --
      Peter predicted that you would "deliberately forget" creation 2000 years ago...
    4. Re:Netflix rating engine sucks by PRMan · · Score: 1

      And back then, I found it to be very helpful and accurate. Now, not at all.

      --
      Peter predicted that you would "deliberately forget" creation 2000 years ago...
    5. Re:Netflix rating engine sucks by hawguy · · Score: 2

      Netflix's rating system is worse than ever. It recently said that I would like "Amber Alert" at a 4.8 out of 5. I thought, "Not likely", but I tried it anyway. I turned it off in 10 minutes and rated it a 1 (which for me means couldn't finish). How on earth did it think I (or anyone else) would like that horrible movie with ugly, stupid people screaming at each other the whole time?

      To be fair, even humans aren't always great at choosing what another human will like, based on some of the horrendous Christmas presents I've gotten from close family members over the years.

    6. Re:Netflix rating engine sucks by SydShamino · · Score: 2

      Sure, but it's so much worse now than it was then. I was trying to add old Doctor Who to my DVD queue. With each add it pops up other recommendations, but a lot of the time none of them were Doctor Who episodes!

      It seems to recommend obscure crap when I'm adding a popular/cult item, and it recommends Frozen or some other recent big budget thing when I'm adding older obscure stuff. I have to think their algorithms have been messed with by their marketing and suits to push things their distribution contracts require them to, not what their users actually want.

      --
      It doesn't hurt to be nice.
    7. Re:Netflix rating engine sucks by Hunter-Killer · · Score: 2

      Crummy selection pretty much nails it. If there were an infinite number of movies, the algorithm would work well. Consider the following scenario: You are one of 3000 subscribers that likes 18th century historical dramas. A documentary on royal intrigues is highly regarded by the 30 or so subscribers in your group that have seen it. Unfortunately, it won't be recommended to you because other subscribers ran out of movies long ago and now watch whatever is on the main page. Many of those 3k subscribers watched Ip Man because it looked tolerable, not because it had an intersection with your interests, but it'll be recommended anyway. Hidden gems are drowned out because the algorithm can't tell the difference between a movie you want to see and a movie you saw because you wanted to see something, anything that night.

    8. Re:Netflix rating engine sucks by swb · · Score: 1

      When Netflix was just a DVD service, keeping up with the star ratings of movies you had watched wasn't hard. You'd log into the web site to manage your queue anyway and clicking on the ratings was simple.

      Now so many people watch things via streaming that it's easy to not do it (and so many STBs make it difficult/awkward to rate anyway). Plus I'd bet that much of the streaming viewing is series where rating kind of falls apart because you might watch a single show for a couple of weeks and you lose opportunities to rate many titles since series have a single rating.

      It makes me wonder if the suggestion algorithm ever included the critical quality of the movie or if it just included the user ratings. If critical quality was never a factor, skewing the movie base with bad titles makes it seem less effective, especially to a user who may have already taken into account general critical reviews because they see Netflix just pushing bad direct to video titles.

      If users are spending more time watching series, not rating due to streaming changing their interactions with the rating system and the recommendation engine not taking into account movie quality it's even easier to see how recommendations become increasingly useless.

    9. Re:Netflix rating engine sucks by Lord+Crc · · Score: 1

      Well the problem is that when I rate a movie, Netflix has no idea why I rated the way I did. They don't know the context.

      I recently rated a movie 1 star, I didn't even finish it, because the story was just so horribly badly written. But I liked just about everything else, the plot itself was great, my kind of genre, cinematography was good, actors too. But that stupid story just killed it for me.

      How's Netflix going to figure out why I rated that a 1 without asking me? I think they should ask follow-up questions to get some of that context if I rate a movie very different from their prediction.

      But yes, if I've recently watched a movie... don't recommend it to me again for some time. That'd be a good start.

    10. Re:Netflix rating engine sucks by omfgnosis · · Score: 2

      How's Netflix going to figure out why I rated that a 1 without asking me?

      This isn't really hard, in the abstract. They just have to have much better metadata about the content, and then an ever-deeper analysis of relative ratings can follow from that. Inference of context will never be perfect, but then again neither will a questionnaire (even if people voluntarily devote their time to answering it) which could recursively be subject to the same criticism that it lacks context. Unless Netflix (or any similar service) deeply understands its content, its recommendations will always be lacking.

      The reason online retailers can do relatively better is that a given product often has quite a lot of metadata that can be reasoned about, and it's often relatively easy to model in context of a given product's domain. The kind of qualities people discuss about content is generally much more vague and superficial in comparison.

      For instance, Netflix is often confused into believing I have any interest in genre. It might be better at predicting my taste if there were a deeper wealth of data on the kinds of qualities I care about in the content I do like, but it's generally pretty self-evident that they don't. They use coincidence of ratings across users to approximate this, but it's all very hand-wavy and often leads to confusing (if unsurprising) results. Nothing is a substitute for a deeper (currently, at least, human) analysis of the content.

  6. I seriously doubt this is leisure watching by Wycliffe · · Score: 3, Insightful

    There is very little reason that you would need to watch an entire movie to tag it properly.
    If nothing else you would probably be watching the movie in fast-forward.
    The movie itself does a pretty good job of doing a summary. Amazon turk or the netflix
    feedback would be a decent way to get short feedback from people who have actually seen
    the movie. My guess is that this position is more of a "scan the movie really quick" type job
    and/or taking user generated data and creating proper tags from it. You are not going to
    get to watch movies for 8 hours a day and only report on those 4-6 movies.

    1. Re:I seriously doubt this is leisure watching by ZombieBraintrust · · Score: 1

      I'm not sure. If the person is tagging it for cus words then they would need listen to the whole thing. (or at least up to the point where people start cussing)

    2. Re:I seriously doubt this is leisure watching by wisnoskij · · Score: 1

      Why eve watch it. I watch movies all the time and still have substantial trouble figure out what genre to store it in. Just go to wikipedia or imdb where groups of people collaborate to figure out which genre to label it as.

      --
      Troll is not a replacement for I disagree.
    3. Re:I seriously doubt this is leisure watching by geminidomino · · Score: 1

      I can think of only 1 movie that saves the Precision F-strike for the end of the movie (ST: Generations).

    4. Re:I seriously doubt this is leisure watching by markkezner · · Score: 1

      Ever actually watch the closed captions? The data isn't as accurate or consistent as you may hope.

      --
      Dangerous, sexy, turing complete: Femme Bots
    5. Re:I seriously doubt this is leisure watching by s0nicfreak · · Score: 1

      How do you know the accuracy if you aren't listening while you watch the closed captions? (Or ARE you listening while watching the closed captions, in the hope of teaching the two month old to read or some such?)

  7. Hopefully Netflix won't turn to Mechanical Turk by CRCulver · · Score: 2

    ...where they will offer $1 an hour to watch a whole day worth of content. American users will then be puzzled by tags of third-world origin such as "man-wins-ten-lakh-dollars" and "woman-removes-petticoats".

  8. Not to be that guy but... by mwfischer · · Score: 1

    ..why not crawl all these movie review sites and use some super algorithm to sort these things.

    1. Re:Not to be that guy but... by fuzzyfuzzyfungus · · Score: 1

      There's still a fairly big gap between the interpretive capabilities of the neural networks we manufacture with unskilled labor and anything the computer scientists and computational linguists have been able to achieve.

      For very, very, large datasets, that's not terribly relevant because you have no choice; but for comparatively constrained ones(like Netflix's catalog), this makes throwing meat at the problem rather more attractive...

  9. will they be on the hook for the data overages by Joe_Dragon · · Score: 1

    will they be on the hook for the data overages if they pay at or near min wage? as being forced to pay for added data may pull you under the min wage.

  10. Re:IMDB is full of descriptors by Anubis+IV · · Score: 3, Insightful

    You do realize that IMDb is a type of wiki, right? The tags are user-submitted. They're good for some stuff, but probably not so useful for the sorts of things Netflix likely needs them for. Besides which, IMDb is owned by Amazon, so there's likely all sorts of legal issues in using its data for their service.

  11. It like getting paid for coding... by apol · · Score: 1

    Look great at first sight... but only at first sight.

  12. Screen time by jeffmflanagan · · Score: 1

    It would be nice if these viewers could log actor's screen time so that could be used in ranking when searching for actors. I did a search for Steve Coogan, and he was only in the first search result "In The Loop" for about a minute. Peter Capaldi, the next Doctor, was good in it though, so it was worth watching.

  13. Ahhhhh! by Anonymous Coward · · Score: 1

    Because they want shills to recommend shows nobody else wants to see so they'll get kickbacks from the clueless publishers.

    THAT explains it!

    After having gone through all of the old decent series up on Netlfix, I just see crap. And they recommend to me crap. Just because I watched Star Trek doesn't mean I want to watch some B movie horror flick set on some asteroid or something or some stupid thing about ghosts.

    Now, I'm pretty much paying $8/month for the next series of Sherlock, Doctor Who, and other BBC stuff.

    If Amazon lowered their prices, it would be worth ditching Netflix and just buying from Amazon. But as it is, there isn't that much I am interested in.

    1. Re:Ahhhhh! by HornWumpus · · Score: 1

      You didn't watch those Star Trek episodes...you prefer the court drama and western ones?

      --
      John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
  14. Netflix doesn't get me... by MindPrison · · Score: 1

    You see, Netflix has a fancy recommendation engine that suggests movies and shows you might like based on your prior viewing habits

    Fancy schmansy...explain this:

    When I got Netflix, I watched all the episodes of Breaking Bad, the two seasons of House Of Cards, Batman and a heap of 80s movies, and guess what it tells me?!

    Top 10 recommendations for MindPrison: Dora the explorer, Go-Diego-Go, Lazytown, The Backyardigans...oh I'm not even going to go on, when I browsed for more results, it even came up with top Chick-flicks to watch. Yep, they need humans instead of an algorithm.

    --
    What this world is coming to - is for you and me to decide.
    1. Re:Netflix doesn't get me... by CryptDemon · · Score: 1

      It does this to me all the time. Oh you liked Arrested Development and Breaking Bad? We think you should watch Hoarders and Pawn Stars even though our suggestion engine thinks you'll rate it less than 2 fucking stars.

    2. Re:Netflix doesn't get me... by MindPrison · · Score: 1

      It knows you surf /b/ looking for Rule 34 of the items it recommended?

      Ah, of course - who in their right mind would want Rule 34 of Bryan Cranston and god forbid...Robin Wright, right?!

      --
      What this world is coming to - is for you and me to decide.
    3. Re:Netflix doesn't get me... by vux984 · · Score: 1

      Top 10 recommendations for MindPrison: Dora the explorer, Go-Diego-Go, Lazytown, The Backyardigans

      Are you sure your kids didn't do a Bob the Builder marathon when you weren't looking? :p

      The recommendation engine has its flaws, but your example is a bit unbelievable. The stuff it recommends me are pretty hit and miss too... but I can usually see where its coming from.

      In some cases, its based on cast ... suppose hypothetically you watch a lot of low brow redneck comedy including Larry the Cable guy stuff and it comes back suggesting Disney's Cars and the Tooth Fairy 2.

  15. Horrible inefficency by iamacat · · Score: 1

    Use a speech recognition engine or any available subtitles to automatically classify movies based on language and keywords. THEN have humans review challenging segments of text and, when necessary, video. May not be as fun, but much more efficient than purely manual process.

  16. Why all the focus on Recommendations? by zippthorne · · Score: 1

    Search is abysmal. Forget recommendations and work on improving search. Sometimes I don't need a recommendation because I know what I'm looking for - something in a particular category (or intersection of categories) that I haven't seen yet.

    --
    Can you be Even More Awesome?!
  17. what a horrid job by asmkm22 · · Score: 1

    Considering the ratio of decent to crappy movies is something like 1:50, I can't imagine this being a very good job. There has to be something like 200 hours of Asylum "mockbusters" alone. I stopped subscribing after watching the handful of quality TV shows and the very rare somewhat-newish-release movies dried up. At this point, I can honestly say they couldn't pay me enough to watch Netflix.

  18. ESPN Has been doing this for years... by Anonymous Coward · · Score: 1

    I remember listening to a "This American Life" story about people who work for sports broadcasters who do the exact same thing. All they do all day is watch sports footage and if a segment has a dropped ball they tag the video clip with "dropped ball" and any other relevant tags (the players involved, the stadium it happened at, ect...).

    They do this so for example if anyone in the broadcast department needs to show all the dropped balls a specific player made during his career for a segment they are doing they could just search "'Jo Somebody' 'Dropped Ball'" and get every video clip evolving that player and a dropped ball.

  19. Robots by robstout · · Score: 1

    They probably don't allow you to make a couple of robots out of spare parts so you can mock the films you are ebing forced to watch, to retain your sanity.

  20. Re:Netflix rating engine is excellent. by urbanriot · · Score: 1

    I've found the recommendation system an excellent experience. I rate many of the movies I enjoy and all the suggestions they give me are precisely the kinds of movies I love, either enjoyed them in the past or enjoyed them after Netflix recommended them to me.

    After a few years of using Netflix, it feels like the system knows me to well...

  21. My daughter by roc97007 · · Score: 1

    She does it anyway. Might as well get paid to do it.

    --
    Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
  22. Re:Toe taggers by roc97007 · · Score: 1

    > hire taggers

    "Buffy, season n, episoe m, good foot shot of Buffy in sandals. Season 5, episode m, Dawn notes Glory has nice feet. Season x, episode y, Tara sitting on the bed, left foot bottom visible, quite nice..."

    Can someone explain this to me? I see paparazzi articles talking about "toe cleavage", (toe cleavage? Seriously?) and googling any moderately attractive famous person inevitably yields galleries geared towards said celeb in bare feet. Apparently nothing else matters except that the feet are bare.

    When did feet become a thing?

    Ok, I said that wrong. In a country of 300 million, *anything* is a thing for some subset of the population. I guess I meant, when did this foot fetish thing become a common thing?

    --
    Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
  23. Couch potatoes re-unite by AbhishekDeyDas · · Score: 1

    Couch potatoes of the world re-unite!