Netflix Prize Sequel Cancelled Over Privacy Concerns

← Back to Stories (view on slashdot.org)

Netflix Prize Sequel Cancelled Over Privacy Concerns

Posted by Soulskill on Friday March 12, 2010 @07:47AM from the beware-the-evil-agorithims dept.

An anonymous reader writes "Netflix just announced that they have cancelled the sequel to the Netflix Prize, which was promised last year. Netflix made the choice after they were sued over privacy concerns. The prize involves releasing large amounts of data about users' movie preferences, which raised concerns from the Federal Trade Commission and a lawsuit from KamberLaw LLC. Netflix's Neil Hunt said, 'We have reached an understanding with the FTC and have settled the lawsuit with plaintiffs. The resolution to both matters involves certain parameters for how we use Netflix data in any future research programs.'"

52 of 65 comments (clear)

Min score:

Reason:

Sort:

It's time to play... Name That Person! by LostCluster · 2010-03-12 07:47 · Score: 4, Interesting

From the linked to previous story...

If a data set reveals a person's ZIP code, birthdate and gender, there's an 87 percent chance that the person can be uniquely identified
Why does Netflix need to release something as precise as a birthday in order to make movie recommendations? I mean, TV ratings are done in demographic groups. Couldn't Netflix get away by just stating a birth year?
1. Re:It's time to play... Name That Person! by Anonymous Coward · 2010-03-12 08:29 · Score: 1, Funny
  
  When given a chance, I give my email address as billg@microsoft.com as well.
  They do a back trace on the IP - they find out from the sender what was the IP of the person who gave them that email address and then Microsoft uses their considerable power and influence to find out who you really are. Then, they sign you up for subscriptions to "The Advocate", "Gay Pornography Lovers Journal", and other gay magazines.
  You haven't gotten them yet?
2. Re:It's time to play... Name That Person! by Anonymous Coward · 2010-03-12 08:33 · Score: 2, Funny
  
  He was just wondering why he was suddenly started getting a second copy for free...
3. Re:It's time to play... Name That Person! by FunkyELF · 2010-03-12 08:54 · Score: 2, Insightful
  
  Why does Netflix need to release something as precise as a birthday in order to make movie recommendations? I mean, TV ratings are done in demographic groups. Couldn't Netflix get away by just stating a birth year?
  I was born on Dec 4th so I am partial to movies about Pearl Harbor.
  I'm sure people born on or around Dec 25th feel differently about Christmas movies than others.
  You comb through enough data you'll find patterns. Who knows.
4. Re:It's time to play... Name That Person! by jimbolauski · 2010-03-12 08:58 · Score: 1
  
  Because there is a higher probability that a girl will rent 16 candles on her 16th birthday.
  
  --
  Knowledge = Power
  P= W/t
  t=Money
  Money = Work/Knowledge so the less you know the more you make
5. Re:It's time to play... Name That Person! by OnlyJedi · 2010-03-12 09:14 · Score: 1
  
  For that matter, why the person's zip code? I can understand that geography may play a role in people's tastes (coasts vs central America, urban vs. rural), but the full zip code is overkill. Just give the decade of birth and state of residence, and if necessary the size of the town/city the user is from, bucketing them into groups like 100-1000, 1000-10000, etc). That should give enough data (approximate age, geography, urban vs. rural vs. suburban) to the researchers without being specific enough to identify any one person.
6. Re:It's time to play... Name That Person! by zkp · 2010-03-12 09:17 · Score: 1
  
  In the original netflix competition the data they did not release birthdays, zip codes or gender. Every movie and user was given a unique (presumably random) id. Essentially the data you had to work with was a bunch of tuples: (movie id, user id, rating (1-5), date) Netflix claims they even added some random noise (changing the dates/ratings a little bit) to preserve anonymity. Turns out even this isn't enough to guarantee anonymity...it turns out you can cross reference this data with imdb to look for similar date/ratings patterns and re-identify a lot of the people. See the paper: "How to break the Anonymity of the Netflix Prize dataset" (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.100.3581&rep=rep1&type=pdf)
7. Re:It's time to play... Name That Person! by zkp · 2010-03-12 09:27 · Score: 2, Interesting
  
  Actually you could probably learn a lot about a person from how they rated certain movies...
  
  Did you watch Michael Moore's film "Capitalism: A Love Story"?
  
  Rating of 5 => You are most likely very liberal
  
  Rating of 1 => You are most likely very conservative
  
  Other things I could potentially learn about you: your religious beliefs, how much time you spend watching movies etc...
  
  Personally, it would not bother me if someone saw my ratings of the films (probably a 1 in my case) or if you knew that I am pretty conservative. The issue is a matter over control. I should be the one who has control over that information, unless netflix explicitly tells you that they give this information out to other customers.
8. Re:It's time to play... Name That Person! by zkp · 2010-03-12 09:41 · Score: 1
  
  In the original netflix competition the data they did not release birthdays, zip codes or gender. Every movie and user was given a unique (presumably random) id. Essentially the data you had to work with was a bunch of tuples:
  
  (movie id, user id, rating (1-5), date)
  
  Netflix claims they even added some random noise (changing the dates/ratings a little bit) to preserve anonymity. Turns out even this isn't enough to guarantee anonymity...you can cross reference this data with IMDB to look for similar date/ratings patterns and re-identify a lot of the people.
  
  See the paper: How to break the Anonymity of the Netflix Prize dataset
9. Re:It's time to play... Name That Person! by Splab · 2010-03-12 11:50 · Score: 1
  
  Because that would be pretty much useless. I'm from Denmark, within a radius of 15 miles from my home you will find some of the poorest people in Denmark and some of the richest, several 100 zip codes (Copenhagen), and taste varying from the finest art to hardcore rave.
  You need to be way more specific than what state the person lives in, ZIP + quater and year of birth might be "enough".
10. Re:It's time to play... Name That Person! by OrangeCatholic · 2010-03-12 20:47 · Score: 1
  
  I think what he's saying is that it would be sufficient to label a person as being from Copenhagen. Don't forget, the age and zip code are secondary classifications. The primary data is the list of movies they watch. So, you don't need the zip code to determine who enjoys the finest art and the most hardcore rave. They are already distinct.
11. Re:It's time to play... Name That Person! by OrangeCatholic · 2010-03-12 21:01 · Score: 1
  
  Are you sure about this? I briefly looked over the paper and found it unconvincing. The first half is mostly bold claims like:
  
  With 8 movie ratings (of which 2 may be completely wrong) and dates
  that may have a 3-day error, 96% of Netflix subscribers whose records
  have been released can be uniquely identified in the dataset.
  And then the second half is a tremendous amount of math. I didn't see any evidence that the authors actually de-anonymized a single person, let alone 96% of them.
  Besides, how do you download imdb's data in order to perform this massive cross-reference?
Sad, but no need to cancel the whole contest by pwnies · 2010-03-12 07:47 · Score: 4, Insightful

This is definitely a sad outcome to all of this. However, couldn't Netflix just update their EULA and/or have an opt-in for users who want to make the experience better?
1. Re:Sad, but no need to cancel the whole contest by LostCluster · 2010-03-12 07:57 · Score: 1
  
  Well, Netflix needs to at least start over because the data set they disclosed is the point of the issue here.
2. Re:Sad, but no need to cancel the whole contest by natehoy · 2010-03-12 09:11 · Score: 1
  
  That is possible, but they still can't do the study they want to do today, because the information they have collected up to the point people start checking the tickyboxes was collected without the permission to release it.
  If they implemented the opt-in this week, they'd start collecting data and they'd probably have some useful data NEXT year.
  However, that also leads to some interesting questions about data validity. A valid study uses truly random data from a full range of customers. The new study, if done, would use data from ONLY customers who are unconcerned about the privacy implications AND are engaged enough to actually take action to opt in. Based on that demographic, they would need to remove "The X Files", "Enemy of the State", and just about any movie or series that has the slightest hint of government conspiracy-ism in it, because no customers in the study would EVER rent them.
  Anyone informed enough to know there is an opt-in is probably going to be informed enough to know it won't really help them. Anyone uninformed of the issues would also not know about the opt-in. You might get a 10% opt-in rate if you're lucky, and most of those people incessantly post what movie they are watching on Twitter anyway, or maintain a Facebook "favorite movies" list, so you can get more accurate and more in-depth data from a larger audience with a lot less effort (grin).
  Just make a "Netflix" fan page on Facebook, and as soon as people join you get to see their favorite movies lists, and chances are you also get their approximate location and birthdate.
  Problem solved!
  
  --
  "This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
Wow by Pojut · 2010-03-12 07:49 · Score: 4, Insightful

I don't know if people are just paranoid or what, but they seem to be intent on protecting EVERYTHING nowadays. Next thing you know, people will get sued for asking whether you put the toilet paper roll facing away from the wall or towards it.
For the record, it's away from the wall, you savages.

--
Living With a Nerd
1. Re:Wow by LostCluster · 2010-03-12 07:52 · Score: 1
  
  The libraries here post signs saying that they'll strictly protect your privacy, but if there's a law enforcement person who cites the PATRIOT Act, they have to give it all away.
2. Re:Wow by Pojut · 2010-03-12 07:54 · Score: 1
  
  Not sure who's sig it is, but someone around here has it set to:
  "Knock Knock"
  "Who's there?"
  "Under the Patriot Act, we don't have to tell you."
  It would be awesome...if it wasn't so true |:/
  
  --
  Living With a Nerd
3. Re:Wow by Dachannien · 2010-03-12 08:03 · Score: 1
  
  There's really only one thing people get sued for: having money.
4. Re:Wow by fahrbot-bot · 2010-03-12 08:27 · Score: 2, Informative
  
  ...you put the toilet paper roll facing away from the wall or towards it. For the record, it's away from the wall, you savages.
  
  Assuming "away" means "over", then you must not have pets. A kitten can easily unspool a whole roll like that. Having it facing the wall (or "under") prevents that.
  Perhaps that makes me a savage, but at least I'm a neat savage. :-)
  
  --
  It must have been something you assimilated. . . .
5. Re:Wow by maxume · 2010-03-12 08:27 · Score: 2, Insightful
  
  Sometimes they get sued for having insurance.
  
  --
  Nerd rage is the funniest rage.
6. Re:Wow by Pojut · 2010-03-12 08:41 · Score: 1
  
  I have a toothy orange two year old cat named Fizzgigg (yes, after the toothy puffball from The Dark Crystal...it's a very appropriate name for this animal, I assure you). She leaves the toilet paper alone...her thing is hair ties.
  Toss a hair tie, and she'll chase after it...and bring it back to you. Over. And over. And Over. She plays freakin' fetch, lol
  
  --
  Living With a Nerd
7. Re:Wow by Aldenissin · 2010-03-12 08:48 · Score: 1
  
  That and the fact that when you use the double rolls and the over placement they tend to bump into the wall and the friction causes the paper to tear prematurely. So then you have to make a decision. Is that enough sheets, or do I want to try to stack it on top of what you already have, and then balance it while you try to get it behind you. (Sometimes there are multiple attempts stacked...) If it falls off in the process (you may not notice... what I have heard anyhow) you won't have enough paper for it to do it's job and protect your hand. Or you could of course use a little more and wrap it around what you already have and negate the stack approach, but by then you have often used as much as you would have had the paper not been torn and possibly clog the toilet.
  Yes, yes I know I have over analyzed this. But I buy double rolls to save money and time. I can't get the girlfriend to put it the way I like it, and so I either switch it, or carefully think about not ripping the paper. It has now become that I see the roll balanced on top of the last empty one because she knows I will likely turn it around, and she won't just do it the way I like. So now the situation has gone and got to be more trouble than it was worth! C'est_la_vie...
  
  --
  Like a city whose walls are broken down is a man who lacks self-control.
8. Re:Wow by Aldenissin · 2010-03-12 08:49 · Score: 1
  
  The girlfriend has a cat that will take your straw out of the cup (sometimes not knocking it over) and will bring it to your feet over and over playing fetch.
  
  --
  Like a city whose walls are broken down is a man who lacks self-control.
9. Re:Wow by Bakkster · 2010-03-12 08:52 · Score: 1
  
  There's a reason they call it a guily pleasure.
  
  --
  Write your representatives! Repeal the 2nd Law of Thermodynamics!
10. Re:Wow by jimbolauski · 2010-03-12 09:04 · Score: 1
  
  For the record, it's away from the wall, you savages.
  Your the savage I have a stream of cold water shoot up my ass followed by a nice breeze since there is no wiping I don't even have to wipe my hands. High 5 anyone.
  
  --
  Knowledge = Power
  P= W/t
  t=Money
  Money = Work/Knowledge so the less you know the more you make
11. Re:Wow by spleen_blender · 2010-03-12 09:21 · Score: 1
  
  Unless you have cats.
12. Re:Wow by martin-boundary · 2010-03-12 09:26 · Score: 1
  
  I don't know if people are just paranoid or what, but they seem to be intent on protecting EVERYTHING nowadays.
  
  That's GOOD. I'm not happy that the second Netflix prize is canceled, and that the data of the first prize has been removed from UCI, but protecting privacy is IMPORTANT.
  Anonymizing data (ie while keeping all other statistical properties intact) is a very hard problem. Solving it properly is a huge prize itself that would revolutionize Science, and there's no reason to blame the researchers who find the leaks in published datasets, or the people who sue for breach of privacy on that basis.
  Hiding the flaws under the carpet for expediency is not going to get us closer to that goal. On the contrary, if we want quality datasets made available to the public in the future, we should keep standards of anonymity as strict as possible and learn how to do it better.
13. Re:Wow by ArundelCastle · 2010-03-12 09:49 · Score: 1
  
  For the record, it's away from the wall, you savages.
  We have a kitten. Towards the wall is the only way to keep her from unrolling an entire spool.
  Unless you'd argue that forcibly restraining a kitten is the less-savage option. :) Other than TP, it's the safest and sunniest room in the house to leave her alone in.
  But maybe we're just paranoid.
14. Re:Wow by ArundelCastle · 2010-03-12 09:54 · Score: 1
  
  The libraries here post signs saying that they'll strictly protect your privacy, but if there's a law enforcement person who cites the PATRIOT Act, they have to give it all away.
  That is precisely why good library circulation software does not preserve that information indefinitely.
  If you need to be reminded that you already read a book or rented a movie 3 years ago, you could probably stand to see it again. The convenience factor is minimal compared to the (almost) worst case scenario of a federal investigation. (The worst case is of course when they decide to skip the investigation.)
15. Re:Wow by Weirsbaski · 2010-03-12 15:17 · Score: 1
  
  I don't know if people are just paranoid or what, but they seem to be intent on protecting EVERYTHING nowadays.
  It's a reaction because of organizations that insist on protecting nothing. The douchebag orgs kind of ruined it for everyone.
  
  --
  
  I am not a sig.
16. Re:Wow by adolf · 2010-03-13 01:23 · Score: 1
  
  Right. Hair ties are fragile, though, and chewy -- the cat (my cats, anyway) tend to bite them in two, and then they're no longer interesting.
  Try a milk ring sometime. You know, that little red/orange/yellow/blue ring under the cap of a gallon of milk. I've got a cat that will chase one of these around, independently, for several unbroken hours before he gets bored with it, which is really quite something since he's approximately bloody ancient these days.
  And: They're free with every gallon.
  (-1 way off-topic, but at least I didn't post this at +1.)
  
  --
  Kid-proof tablet..
17. Re:Wow by Uzuri · 2010-03-19 03:13 · Score: 1
  
  My cats must be smarter than your cats -- they're quite capable of emptying a whole roll no matter which way it's spooled. Then they open the cupboard and take all the stored rolls out and do them in as well.
  
  --
  I'm a she-slashdotter... but I make up for it by living with my folks.
Why does Netflix need to release birthday by KharmaWidow · 2010-03-12 07:58 · Score: 1

Maybe they are consulting the Zodiac astrology?
I have to agree by T+Murphy · 2010-03-12 08:09 · Score: 4, Funny

I bet there are thousands of guys out there scared to death that someone will find out they rented Twilight (for the girlfriend, honest!). I'd rather be known as a lawyer-happy jerk than a Twilight fan.

--
My webcomic
1. Re:I have to agree by Cytotoxic · 2010-03-12 08:43 · Score: 1
  
  Ok, I admit it. I rented Twilight. Not for my wife, either. I was just curious. You know.... an experimental phase.
  You know what I learned? I'm getting old. 'Cause that movie sucked hard. Holy crap, was that one boring, boring, boring movie. I can't really convey how terrible it was. And the way that I know I'm now officially getting old is that I can't even see how a pre-teen could find it entertaining. It didn't even really do a good job of conveying teen angst, or alienation, or much of anything. Just a lot of sullen looks and staring contests.
  Actually, the only way I can really describe it is as the bastard lovechild of MTV's "The Hills" fake reality show and "Buffy the Vampire Slayer". Only instead of somewhat interesting characters and lots of action in a teen-angsty world, you get "The Hills" style crap dialog with lots of long pauses and staring into the distance. Even the action was boring.
  I cannot for the life of me see how anyone old enough to have ever seen an actual movie before could have rated that thing anything other than a complete zero. It wasn't even good enough to be bad .. in fact it was kind of orthogonal to anything on the good/bad scale. It was just a big, lame, slow, turd that never went anywhere.
  There. I got that off my chest. Ok. I feel better now that I admitted it.
2. Re:I have to agree by Anonymous Coward · 2010-03-12 09:01 · Score: 1, Funny
  
  Just wondering, what's your zip code and birthdate?
3. Re:I have to agree by AdamTrace · 2010-03-12 09:11 · Score: 1
  
  My wife and I just rented it, too. Just to see.
  I stopped watching at the vampire softball scene. Ugh, so bad.
4. Re:I have to agree by natehoy · 2010-03-12 09:17 · Score: 1
  
  No, no, this is Slashdot, it's OK to vent. Tell us what you REALLY think about it. You sound ambivalent.
  If it wasn't good enough to be bad, was it at least bad enough to be good? In a "Plan Nine From Outer Space" / Mystery Science Theater 3000 kind of way? Could Mike (or Joel) and the 'bots save this snoozefest and turn it into a laughfest, or at least make it chuckle-worthy?
  Or was it, as they say, just a large sack of suck?
  
  --
  "This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
5. Re:I have to agree by spxero · 2010-03-12 10:19 · Score: 1
  
  I made it to about minute 45 before I couldn't take anymore. If a movie can't get *somewhere* in 45 minutes, it's really just bad.
  There wasn't enough dialog for MST3K to work with. GP was right- it's just a long, boring staring contest. How this crap is even popular is beyond me. My wife's friend swore up and down that I need to read the books... no thanks. That's why I want to watch the movie- reading the book series for a couple weeks appeals even less than sitting through a 2hr+ movie.
  On XKCD suck scale ( http://xkcd.com/653/ ), it's somewhere around the Star Wars Holiday Special. But less entertaining.
Someone else would have to have by Shivetya · 2010-03-12 08:10 · Score: 1

a whole lot more on the people to identify them from those three pieces of information.
As in, you would need to know which house has the account, then guess among all the netflix in that zipcode.
I suppose you could stand outside, but I doubt that there is 87% chance unless you know other supposedly private information.
I don't see how exact birthdate matters, how would you confirm it when looking at people? Birthdate x/y/z lives at 144 Amazing avenue? Really? and what gave that away?

--
* Winners compare their achievements to their goals, losers compare theirs to that of others.
1. Re:Someone else would have to have by LostCluster · 2010-03-12 08:21 · Score: 3, Insightful
  
  Let's say you have a job. Your boss, by law, needs to see and retain a copy of either your drivers license or passport, either one of which identifies your birthdate, address (which contains your ZIP code) and gender.
  Now let's say he downloads this Netflix data... he's got enough info to be 84% sure that the only record that has your birthdate, gender. and ZIP code, is you. Worse yet, if he had your ZIP+4, the only way there'd be any ambiguity left is if you live with a twin.
2. Re:Someone else would have to have by Bakkster · 2010-03-12 08:51 · Score: 1
  
  And even if there are 2 matching accounts, a little bit of guesswork from information known about the person can remove the ambiguity. Look for foreign-language films if they are an immigrant, films or shows that appeal to their occupation (Dilbert cartoon or the IT Crowd for someone here, The Real World or Jersey Shore for someone at Abercrombie) or hobbies (musician, car person, etc), and boom, suddenly you know that the applicant watches softcore porn/disney shows/GLBT interest film.
  
  --
  Write your representatives! Repeal the 2nd Law of Thermodynamics!
3. Re:Someone else would have to have by winomonkey · 2010-03-12 08:53 · Score: 2, Informative
  I am curious which passport you have that includes your address. If I recall correctly, mine does not have an address in it. To verify, I looked to see what goes into a passport:
  
  Photograph
  Type [of document, which is "P" for "passport"]
  Code [of the issuing country, which is "USA" for "United States of America"]
  Passport No.
  Surname
  Given Name(s)
  Nationality [which is "United States of America"]
  Date of Birth
  Place of Birth (lists only the state followed by "USA" for those born in the United States)
  Sex
  Date of Issue
  Date of Expiration
  Authority
  Endorsements
  No address. A passport will not tell your employer where you live. At best, they will know which state you were born in.
4. Re:Someone else would have to have by Splab · 2010-03-12 11:45 · Score: 1
  
  So you have a job where your employer doesn't know where you live?
5. Re:Someone else would have to have by jim_v2000 · 2010-03-12 11:59 · Score: 1
  
  99% chance that your boss doesn't know enough to mine the data, and he wouldn't care what movies you've been watching even if he did know.
  
  --
  Don't take life so seriously. No one makes it out alive.
6. Re:Someone else would have to have by winomonkey · 2010-03-12 13:48 · Score: 1
  
  Actually, they do not know where I live. They need a mailing address, not a home address, and that is what they have. I use online systems for most of my regular documents from them, and use interoffice mail for most of the rest. There is a rare occasion where they send something to my mailing address (tax forms, etc). None of this really matters, anyways, as the Netflix account is in the wife-to-be's name.
7. Re:Someone else would have to have by retchdog · 2010-03-12 14:38 · Score: 2, Insightful
  
  For now. If this kind of data-leak became common, you can bet there would emerge specialist firms/consultants to do the mining, perhaps as a value-add onto a more general background check.
  Vigilance.
  
  --
  "They were pure niggers." – Noam Chomsky
Munge the privacy info by mrops · 2010-03-12 08:59 · Score: 1

Why don't they munge the privacy info, instead of saying a account belongs to John Smith, say it belongs to 1122113.
No one except netflix will know who exactly 1122113 is.
1. Re:Munge the privacy info by AdamTrace · 2010-03-12 09:10 · Score: 1
  
  This is exactly what they do.
  The problem is (and someone figured this out), John Smith might ALSO have a IMDB account. And if we compare the date and star ratings from his IMDB account with his Netflix accout, we can accurately figure out who 1122113 is. Some people actually did this.... that is, took the completely anonymous ratings, and used those ratings to match up with an unanonymous source, and figure out information.
  So Netflix is doing it right, I think. It's just that the Netflix data can be used IN COMBINATION with other sources to reveal identities.
2. Re:Munge the privacy info by zkp · 2010-03-12 09:36 · Score: 1
  
  See the paper "How to Break the Privacy of the Netflix Prize Dataset"
  
  I disagree that Netflix did it right. We live in a day and age where there are many sources of auxiliary information publicly available (IMDB, voter registration, etc...). Any attempt to preserve anonymity must take this into account. All to often companies leave out specific pieces of information (name, DOB, Zip Code etc...) and hope that what remains is anonymous. No one could ever identify me based on just a couple of movie ratings, right?
Re:It's time to play... Astrology! by b4dc0d3r · 2010-03-12 10:07 · Score: 1

My gf knows a lot about astrology, and believes it has an influences on your personality. A good recommendation engine could use this information along with everything else. Your birth year is significant, but it doesn't run 1 Jan to 25 Dec. I'm not sure if this is based on the Chinese calendar or if that's a separate data point... but in order to identify your sign and year, you need to know the exact date.
I don't lend astrology much credence, but if you're going to tell me it doesn't work I'd like some repeatable science on it, or at least download the dataset yourself and see if any patterns emerge before just saying it doesn't work. I haven't tried it myself, because I don't think I know enough about astrology to correlate significant data points. Of course, people lie about their birth date (I do, and at least one other in this thread does) so you're analyzing self-reported birth dates, not actual ones, making science cry.