Slashdot Mirror

← Back to Stories (view on slashdot.org)

Netflix Is Looking To Pay Someone To Watch Netflix All Day

Posted by samzenpus on from the how-do-you-hire-that-which-has-no-life dept.

An anonymous reader writes with news about a dream job for binge-watching couch potatoes in the UK. Ploughing through your new favourite series on Netflix is something you probably enjoy doing after a working day, but what if it was your working day? You see, Netflix has a fancy recommendation engine that suggests movies and shows you might like based on your prior viewing habits. To do that successfully, it needs information from a special group of humans that goes beyond the basics like genre and user rating. "Taggers," as they're known, analyse Netflix content and feed the recommendation engine with more specific descriptors if, for example, a film is set in space or a cult classic. In short, these people get paid to watch TV all day, and Netflix is currently hiring a new tagger in the UK.

23 of 86 comments (clear)

  1. Seems excessive... by Valvar · · Score: 2

    Why not just let the users do the job? Cheaper, faster and easier...

    1. Re:Seems excessive... by Anonymous Coward · · Score: 5, Insightful

      Because they want shills to recommend shows nobody else wants to see so they'll get kickbacks from the clueless publishers.

    2. Re:Seems excessive... by K.+S.+Kyosuke · · Score: 4, Insightful

      I've heard about this thing called metamoderation. Rumor has it that it is already being used on some sites to weed out garbage user inputs...

      --
      Ezekiel 23:20
    3. Re:Seems excessive... by fuzzyfuzzyfungus · · Score: 4, Insightful

      Why not just let the users do the job? Cheaper, faster and easier...

      Generally, when somebody is paying for what it sounds like they could get for free, or even get paid for, there is good reason to suspect that the job description is either underplaying the exact level of difficulty and/or boredom involved, or that somebody has already learned the hard way that what they can get for free isn't exactly what they want.

      In this case, I'd be inclined to suspect that the job is closer to being a 'machine vision' substitute for stuff that machines can't yet see or which it wouldn't be cost-effective to have an expensive analyst cobble together a ruleset and then cheap labor check for mistakes when you could just have cheap labor classify it (eg. 'movies set in space' is probably something that you could achieve reasonable accuracy on, if you do some futzing with detecting starfields and common flavors of "rocket thruster jet of flame"; but you'd have your false positives and false negatives from things in space that happen mostly inside spacecraft, and things not in space that happen to involve looking at the sky more than usual, and so on).

      It's probably a hell of a grind, actually, given that (unlike, say, being a film critic or some film-studies culture critic type) Netflix is going to want everything ground through and tagged on a variety of parameters, not just the stuff you happen to be a geek about, or the stuff that's worth watching, or what have you. It wouldn't much surprise me if, for efficiency's sake, they have you monitoring more than one stream at a time, or working in faster-than-real time, or a combination. You can probably extract the data they want rather faster than you can enjoy the program, even if it is one you like.

    4. Re:Seems excessive... by NotDrWho · · Score: 3, Funny

      Because it's like my grandpa used to say about volunteers: "Sonny, you can't fire someone for doing a bad job if they're doing it for free. Also, don't trust the jews or coloreds with money."

      --
      SJW's don't eliminate discrimination. They just expropriate it for themselves.
    5. Re:Seems excessive... by SQLGuru · · Score: 4, Funny

      But it only works if you participate.......which might explain Slashdot.

    6. Re:Seems excessive... by geminidomino · · Score: 2

      As opposed to amazing incompetence of the "taggers" doing it now.

    7. Re:Seems excessive... by nabsltd · · Score: 2

      Because that has worked out so well for IMDB and TMDB. Try looking at their genres sometime, especially ones like "comedy" where if there is anything even vaguely humorous no matter how passing or unintentional the movie gets classed as a comedy.

      "Genre" isn't really a problem on IMDB, as users can't directly set that. I believe you are thinking of "plot keywords", which are really nothing but tags, and have become silly.

      How does a "loud shirt" have anything to do with the plot of the listed titles?

    8. Re:Seems excessive... by gauauu · · Score: 3, Insightful

      Why not just let the users do the job? Cheaper, faster and easier...

      I recently read an article (I wish I could find it again) that describes how and why Netflix does this. Basically, they train their viewers to watch for many certain qualities and attributes of movies, which are then tagged and categorized to set up their recommendation and category systems.

      For example, they might use a few movies as a baseline for a ratings system so their viewer/ranker staff are on the same page ("on a scale of 1-10, how sweet and sappy is this movie? Does it have a strong female lead? Does it feature cute animals?"), then the viewers watch the film and fill out extensive and standardized tagging information about it, which they build their ratings from.

      The article describes it in much better detail, but it's clear that the level of standardization and depth in their tagging and categorizing is beyond what you'd be able to get from the general public.

    9. Re:Seems excessive... by vux984 · · Score: 4, Informative

      Generally, when somebody is paying for what it sounds like they could get for free, or even get paid for, there is good reason to suspect that the job description is either underplaying the exact level of difficulty and/or boredom involved, or that somebody has already learned the hard way that what they can get for free isn't exactly what they want.

      Bingo.

      You won't watch what you want. You probably won't have enough time to finish watching anything... 99% tagging accuracy for comedy, sci-fi, action, etc, etc, can be assessed within the first half.

      And I can't think of much that would need to see the whole movie to tag correctly, except for "twist ending".

      For TV series you'll probably just watch a few parts of a few random episodes, and then move on.

      Your notion that you'd do it watching multiple streams is quite likely too -- and sped up... probably even skipping... watch 5 minutes, skip 5... watch 5 ...

      Because as you say, your job is to tag movies, not critique them. You'll only spend as much time with a movie as you need to tag it accurately, and that is far less than the 90-150 minutes it would take to watch it from start to finish.

      As an aside, another "dream job" that is truly abysmal in practice is "video game tester".

  2. highlighting UK/IE cultural specificities by OzPeter · · Score: 2

    highlighting UK/IE cultural specificities and taste preferences.

    Given the predilections of UK politicians, this could mean working with some weird shit. OTOH if you're from the UK/IE then you are probably already used to that weird shit.

    --
    I am Slashdot. Are you Slashdot as well?
  3. Don't get me wrong by Anonymous Coward · · Score: 5, Interesting

    I like spending the occasional 1 - 4 hours watching a few episodes in a row or maybe two movies, but doing that 8 hours a day / 5 days a week? Enjoyment soon turns into torture, hope they get paid good.

  4. Re:IMDB is full of descriptors by BorgDrone · · Score: 3, Informative

    Probably because of all the lawsuits that IMDb would file over that.

  5. Netflix rating engine sucks by uigin · · Score: 4, Insightful

    Given how poor the Netflix rating engine is surely their money'd be better spent hiring a programmer? I mean, how about not suggesting to me the movie I've just watched? (Low hanging fruit?)

    1. Re:Netflix rating engine sucks by swb · · Score: 3, Interesting

      I thought they had a big contest where it was a big deal to beat the then-current suggestion engine by 10% because the current engine was supposed to be so good.

      IMHO the bigger problem is that streaming has a huge amount of shit associated with it and they will suggest shit movies which makes it appear that the suggestion engine doesn't work.

      My guess at this point given all they do to hide/obfuscate how crummy their streaming catalog is they don't really care about the suggestion engine anymore.

    2. Re:Netflix rating engine sucks by Anubis+IV · · Score: 2

      Teams of researchers from around the globe competed for the $1,000,000 Netflix Prize way back in 2009, that would be awarded to the team that managed to improve the algorithm by even 10%. It took them the better part of a year to accomplish it, and you seem to think that a lone programmer can just get in there and knock out a lot of low-hanging fruit to substantially improve things?

      I don't deny that there's always room for improvement (such as the example you provided), but suggesting that it can all be fixed by "hiring a programmer" is a bit naive.

    3. Re:Netflix rating engine sucks by hawguy · · Score: 2

      Netflix's rating system is worse than ever. It recently said that I would like "Amber Alert" at a 4.8 out of 5. I thought, "Not likely", but I tried it anyway. I turned it off in 10 minutes and rated it a 1 (which for me means couldn't finish). How on earth did it think I (or anyone else) would like that horrible movie with ugly, stupid people screaming at each other the whole time?

      To be fair, even humans aren't always great at choosing what another human will like, based on some of the horrendous Christmas presents I've gotten from close family members over the years.

    4. Re:Netflix rating engine sucks by SydShamino · · Score: 2

      Sure, but it's so much worse now than it was then. I was trying to add old Doctor Who to my DVD queue. With each add it pops up other recommendations, but a lot of the time none of them were Doctor Who episodes!

      It seems to recommend obscure crap when I'm adding a popular/cult item, and it recommends Frozen or some other recent big budget thing when I'm adding older obscure stuff. I have to think their algorithms have been messed with by their marketing and suits to push things their distribution contracts require them to, not what their users actually want.

      --
      It doesn't hurt to be nice.
    5. Re:Netflix rating engine sucks by Hunter-Killer · · Score: 2

      Crummy selection pretty much nails it. If there were an infinite number of movies, the algorithm would work well. Consider the following scenario: You are one of 3000 subscribers that likes 18th century historical dramas. A documentary on royal intrigues is highly regarded by the 30 or so subscribers in your group that have seen it. Unfortunately, it won't be recommended to you because other subscribers ran out of movies long ago and now watch whatever is on the main page. Many of those 3k subscribers watched Ip Man because it looked tolerable, not because it had an intersection with your interests, but it'll be recommended anyway. Hidden gems are drowned out because the algorithm can't tell the difference between a movie you want to see and a movie you saw because you wanted to see something, anything that night.

    6. Re:Netflix rating engine sucks by omfgnosis · · Score: 2

      How's Netflix going to figure out why I rated that a 1 without asking me?

      This isn't really hard, in the abstract. They just have to have much better metadata about the content, and then an ever-deeper analysis of relative ratings can follow from that. Inference of context will never be perfect, but then again neither will a questionnaire (even if people voluntarily devote their time to answering it) which could recursively be subject to the same criticism that it lacks context. Unless Netflix (or any similar service) deeply understands its content, its recommendations will always be lacking.

      The reason online retailers can do relatively better is that a given product often has quite a lot of metadata that can be reasoned about, and it's often relatively easy to model in context of a given product's domain. The kind of qualities people discuss about content is generally much more vague and superficial in comparison.

      For instance, Netflix is often confused into believing I have any interest in genre. It might be better at predicting my taste if there were a deeper wealth of data on the kinds of qualities I care about in the content I do like, but it's generally pretty self-evident that they don't. They use coincidence of ratings across users to approximate this, but it's all very hand-wavy and often leads to confusing (if unsurprising) results. Nothing is a substitute for a deeper (currently, at least, human) analysis of the content.

  6. I seriously doubt this is leisure watching by Wycliffe · · Score: 3, Insightful

    There is very little reason that you would need to watch an entire movie to tag it properly.
    If nothing else you would probably be watching the movie in fast-forward.
    The movie itself does a pretty good job of doing a summary. Amazon turk or the netflix
    feedback would be a decent way to get short feedback from people who have actually seen
    the movie. My guess is that this position is more of a "scan the movie really quick" type job
    and/or taking user generated data and creating proper tags from it. You are not going to
    get to watch movies for 8 hours a day and only report on those 4-6 movies.

  7. Hopefully Netflix won't turn to Mechanical Turk by CRCulver · · Score: 2

    ...where they will offer $1 an hour to watch a whole day worth of content. American users will then be puzzled by tags of third-world origin such as "man-wins-ten-lakh-dollars" and "woman-removes-petticoats".

  8. Re:IMDB is full of descriptors by Anubis+IV · · Score: 3, Insightful

    You do realize that IMDb is a type of wiki, right? The tags are user-submitted. They're good for some stuff, but probably not so useful for the sorts of things Netflix likely needs them for. Besides which, IMDb is owned by Amazon, so there's likely all sorts of legal issues in using its data for their service.