Netflix Announces Second Data Mining Contest

John Snodgrass writes "Neil Hunt, Chief Product Officer at Netflix, has announced on the Netflix Prize Forums that they are planning to hold a new data mining competition. The second competition will have some twists and is expected to be shorter in duration. It will feature two grand prizes, to be awarded in a 6 and 18 month time frame. A previous competitor still active on the board has already dubbed it: 'The Sparse Matrix: Reordered' and 'The Sparse Matrix: Factorizations.'"

Comments

Re:Contests

[Smidge204]

> Why do tens or possibly hundreds of thousands of dollars worth of work just for the chance that you might get payed? It seems absurd.

Challenge and notoriety.

For that matter, just about everything you do has a chance of failure, so why do anything?
=Smidge=

Re:Contests

[TheRealMindChild]

Most of the time people want to tinker/play with concepts. They just need some sort of motivation to get them going. The chance that you might get payed is apparently enough for some people.

Re:Contests

[Helios1182]

Most of these are research groups that would publish their results and research anyway. This gives them a practical application and a chance for some fame and money -- the research still gets done and published.

Re:Contests

[Helios1182]

I'll add one more thing. Netflix has done the community a favor by providing a large dataset for testing algorithms. Data mining requires data. It requires more than just raw data. It is really difficult to know how well your algorithm works without data that has known answers to compare to. A good test dataset lets you compare your results to other results.

Human reaction machines. . .

[Fantastic Lad]

There's nothing at all wrong with studying how the human automatic processes work, but "Psychology for Prizes" does have a very Neil Stephenson feel to it.

The public eagerly jumping for the chance to teach corporate bodies how to better advertise to them seems a little preposterous. In a world where everybody's objective is openness and self-study for the betterment of humankind, this sort of thing would be laudable, but here it's a bald-faced attempt to fine-tune manipulation techniques.

What would be cool would be if Netflix, upon offering you a suggestion, would also explain what reasoning they used to offer that suggestion to you. Open-source advertising. If every billboard had an explanation of the psychology behind it, we could learn much more about ourselves. The amount of free will that we use every day versus automatic behavior can only increase when the illusion of free will is broken down and examined.

-FL

Re:Human reaction machines. . .

[Fantastic Lad]

There's always going to be an argument which makes a manipulative and self-serving action sound benign and cheerful. I remember watching a news piece about one of the top McDonald's CEO types heading over to Russia to try to establish the golden arches there. In a candid shot, he described McDonald's as a sort of angelic entity whose mission was to bring hungry children meat, bread and milk. I wondered if he had really convinced himself of that horseshit or if it was just a face he put on for others.

It's all about spin. The problem is that when profit is the primary motivator, then you cannot ever trust a seemingly friendly face put forth by a company. They don't want to be your friend. They want you to think that they are your friend in order that you might feel comfortable in giving them your trust, money, time and energy.

Now, if the Netflix guys are actually motivated not by profit, but by an over-riding love of film and the desire to share film with the world, then it's a whole different story. You do see this sometimes. I've known several owners of private shops who really love what they sell, but when you scale things up past a certain number of employees, even a founding love takes a back seat to the corporate need to grow profit share and absorb wealth. It's almost like a company only has a single soul which is shared by every participant in the company and thus gets stretched thin.

-FL

Re:Usefullness?

[sottitron]

I used to think I was unique in what rare movies and music I liked until I met someone who had almost an identical collection to me. On top of that, we both had some of the same clothes. The reason netflix researches these data mining techniques is because our tastes really do cluster into groups. For some it might be because they like DeNiro films and Spaghetti Westerns. For others it might be that they like two screen writers - though they never know it. The payoff for getting this right if you are Netflix is that if a customer gets recommendations and he/she use those to fill your queue, then they are much less likely to cancel netflix anytime soon.

Re:Contests

[Trepidity]

It allows the researchers to "cheat" a bit too via an argument by authority, which is not always good, but does at least make the researcher's job easier. A big issue in data mining is that it isn't purely a technical field, but one with both conceptual and technical issues. The over-arching goal is something like, "get useful and/or interesting information out of data". But what is "useful", what is "interesting", and how do we measure when we've gotten it or not? Usually you have to defend why your problem is the right one, why your metric is the right way to measure success on it, etc. Working on the Netflix competition lets you sidestep all that, because Netflix has already decreed exactly what the goal is, and what performance metric will be used to judge success at that goal, leaving only the technical problems.

Re:Contests

[blhack]

Why do tens or possibly hundreds of thousands of dollars worth of work just for the chance that you might get payed? It seems absurd.

Perhaps you have become lost on these internets.

I suggest trying this website.
or perhaps this one.

You will likely find them much more aligned with your interests than slashdot.

The Need for This Contest

[SEWilco]

The purpose of this contest is to figure out who won the previous contest.