Slashdot Mirror
Netflix Announces Second Data Mining Contest
John Snodgrass writes "Neil Hunt, Chief Product Officer at Netflix, has announced on the Netflix Prize Forums that they are planning to hold a new data mining competition. The second competition will have some twists and is expected to be shorter in duration. It will feature two grand prizes, to be awarded in a 6 and 18 month time frame. A previous competitor still active on the board has already dubbed it: 'The Sparse Matrix: Reordered' and 'The Sparse Matrix: Factorizations.'"
> Why do tens or possibly hundreds of thousands of dollars worth of work just for the chance that you might get payed? It seems absurd.
Challenge and notoriety.
For that matter, just about everything you do has a chance of failure, so why do anything?
=Smidge=
Most of the time people want to tinker/play with concepts. They just need some sort of motivation to get them going. The chance that you might get payed is apparently enough for some people.
"When life gives you lemons, don't make lemonade. Make life take the lemons back!" -- Cave Johnson
Most of these are research groups that would publish their results and research anyway. This gives them a practical application and a chance for some fame and money -- the research still gets done and published.
I'll add one more thing. Netflix has done the community a favor by providing a large dataset for testing algorithms. Data mining requires data. It requires more than just raw data. It is really difficult to know how well your algorithm works without data that has known answers to compare to. A good test dataset lets you compare your results to other results.
It's a neat contest, but I'm really not sure how it helps their customers at this point. We're talking about the decision-making process human beings go through to decide how they wish to be entertained. I favor some movies just because I enjoyed them as a whole, and their algorithm will not be able to be granular enough to figure out exactly why I like something. On the other hand, having a suggestion system is very helpful. I'd would be pleased to be reminded of a summer comedy that I perhaps forgot about. However, there just aren't enough movies out there that I would skip over one had it not been for their algorithm.
People who take on challenges just because they can? I do believe that they seem "weird" to you. It would seem that way to many lazy people who just want cash for their clunker and to be left aloe to play Halo.
Why is it so hard to only have politicians for a few years, then have them go away?
Obviously the odds do matter. At one end of the extreme is communism, where everybody gets the same regardless of what they accomplish. At the other extreme is "winner takes all," where somebody gets everything, even if they are only 0.01% better than the field. Between those extremes is quite a lot. Which is most motivating? I wouldn't claim to know, and I'm sure it depends on the situation, but it's an awfully important question to society. And I'm pretty sure neither extreme is the best answer (even the Olympics have silver and bronze).
There's nothing at all wrong with studying how the human automatic processes work, but "Psychology for Prizes" does have a very Neil Stephenson feel to it.
The public eagerly jumping for the chance to teach corporate bodies how to better advertise to them seems a little preposterous. In a world where everybody's objective is openness and self-study for the betterment of humankind, this sort of thing would be laudable, but here it's a bald-faced attempt to fine-tune manipulation techniques.
What would be cool would be if Netflix, upon offering you a suggestion, would also explain what reasoning they used to offer that suggestion to you. Open-source advertising. If every billboard had an explanation of the psychology behind it, we could learn much more about ourselves. The amount of free will that we use every day versus automatic behavior can only increase when the illusion of free will is broken down and examined.
-FL
It allows the researchers to "cheat" a bit too via an argument by authority, which is not always good, but does at least make the researcher's job easier. A big issue in data mining is that it isn't purely a technical field, but one with both conceptual and technical issues. The over-arching goal is something like, "get useful and/or interesting information out of data". But what is "useful", what is "interesting", and how do we measure when we've gotten it or not? Usually you have to defend why your problem is the right one, why your metric is the right way to measure success on it, etc. Working on the Netflix competition lets you sidestep all that, because Netflix has already decreed exactly what the goal is, and what performance metric will be used to judge success at that goal, leaving only the technical problems.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Why do tens or possibly hundreds of thousands of dollars worth of work just for the chance that you might get payed? It seems absurd.
Perhaps you have become lost on these internets.
I suggest trying this website.
or perhaps this one.
You will likely find them much more aligned with your interests than slashdot.
NewslilySocial News. No lolcats allowed.
The purpose of this contest is to figure out who won the previous contest.
to take advantage of hordes of unemployed technologists than to get them to provide months of free work?
Terrorists can attack freedom, but only Congress can destroy it.
I agree, but I still think it is still useful. I see three main requirements for data mining research: data, algorithms, evaluation criteria. (Note: I don't do data mining myself, but know many people in the field and have studied it some).
There are lots of algorithms, but they cannot be evaluated in a vacuum. In fact, the algorithms used tend to be highly customized and tuned for any specific problem. Really, the data, algorithms, and evaluation are a package deal.
Getting all of the necessary components is a lot of work, and I think having the data and evaluation criteria provided by an outside source has some merit. If each researcher defines "good" differently it is impossible to be objective in comparison. Also, how is having Netflix state the goal any worse than a research project leader stating the goal? If I say "predict tomorrow's weather based on the previous 3 days known weather patterns" I have given the project parameters.
Obviously Netflix isn't, and shouldn't be, the only one setting goals for the community; but I think this is win-win for everyone. Netflix gets a new algorithm, customers get better service, researchers get money and acknowledgment. It would be interesting to see other companies or agencies promote such work.
lazy people who just want cash for their clunker and to be left aloe to play Halo
Well, to be fair, aloe does help with the chaffing after 48 hours of non-stop Halo.
Yeah. Would you choose a neurosurgeon who pokes around people's brains in his spare time? I wouldn't.
Yeah, I agree with that. I probably shouldn't have used a pejorative-sounding word like "cheat", even in scare-quotes. I meant just that it lets the researcher get for free some of the things they'd usually have to argue for. From a researcher's perspective, this is a real win: there are many technically solid papers that get rejected from conferences because the reviewers thought the problem wasn't interesting enough ("maybe I believe you solved this, but why?"), or the metric wasn't the right one. Nobody's going to reject your Netflix-prize-related paper for those reasons.
It does even provide a good jumping-off point for questioning those assumptions, so I agree it's not a bad thing in any way for Netflix to be proposing goals like this. There have been papers, for example, that accept the basic ratings-prediction goal of the competition, but argue that the specific performance metric used doesn't capture the high-level goal that well.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Comment removed based on user account deletion
Comment removed based on user account deletion
> Netflix Announces Second Data Mining Contest
Oh thank god I've got another chance!
I was gonna solve the previous one challenge, but never quite got around to it.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
Seems like the only reason they keep coming up with such contests is their advertising value. Just my 2 cents.
Catalin Braescu
Ofaly.com
Definitions again. Allow me to clarify. . .
I tend to think that to call oneself a "consumer" is the result of a stupendous and multi-generational maneuver of marketing which reduces the human to the status of a mindless eating machine with no other virtues or qualities of significant value. Sadly, for the most part, this is an accurate state of affairs, but I choose to deviate from that model. I refuse to see my purpose in the world as being simply to desire and work relentlessly towards the acquisition of the colorful array of material goods presented to me by Walmart.
Life, I have discovered, is much, much bigger and far more fulfilling than that. Consequently, I have found endless ways to exist both comfortably and happily without the need to make or spend very much money at all. In this regard, I am a very bad consumer, however I strive to excel in the art of being a person. I have many more hours in the day than most to do with as I please, allowing me to explore the world to my satisfaction in countless fascinating ways. Further, I put my energies to good use; many people consider me a significant asset in their lives.
As I see it, those who accept the mantle of 'consumer' also accept by definition, a very narrow form of existence, --one which serves the world of corporations and governments and not of people and communities.
-FL