Slashdot Mirror
Netflix Prize Competitor Already Beats Netflix
Baldrson writes "Within the first week of the announcement of The Netflix Prize a team has already beaten Netflix's own movie recommendation algorithm. This is pretty impressive given the previously quoted researcher who said: 'You're competing with 15 years of really smart people banging away at the problem.' The team is WXYZConsulting.com apparently registered by a data mining professor named Yi Zhang. Congratulations are in order for Netflix and Prof. Zhang's team who are demonstrating, yet again, the power of prizes to accelerate progress."
the power of prizes to accelerate progress
Hmm...In that case, I'm offering $1000 USD to the person or group that can find me the perfect girlfriend!
I think this demonstrates how important "many eyeballs" are in problem solving. Intelligent people "who have been attacking the problem for 15 years" can still fail to see an "obvious" solution. I shudder at how many scientific fields probably have obvious solutions that aren't being found because only a small cadre of people have been exposed to the problem. I also shudder at people who artificially set up barriers to understanding their own fields, in order to protect their own egos. The attitude of "journal articles need to be cryptic or they must not be important" needs to go.
Apology to Ubuntu forum.
Looking in the competition rules, I was surpised to see that:
Residents of the province of Quebec in Canada are ineligible to participate. Residents of Cuba, Iran, Syria, North Korea, Myanmar (formerly Burma) and Sudan are also ineligible to participate.
Is Quebec the next target for regime change?
This shows that greed can be used in a positive way.
That is all I have to say, anyone else have anything to add?
what is the actual prize they will receive? You'd think within 30 seconds of looking at a site about a prize I'd know what the prize was.
Quite a few teams have beaten the Cinematch engine, but not by the required 10% for the prize. The submission is in error. They also haven't won the 1% Progress prize yet, but they're very very close.
The cesspool just got a check and balance.
... says "Por la boca muere el pez", which means something like "Fish die because of their mouths". We'll see if that research team can swim
When my Karma level reaches 0 I feel in piece with the Universe
although yes they have "beatten netflix", they haven't won the prize yet.
they have about a 1% improvement on the netflix algorithm, but the prize is for 10%. they are the frontrunner for the progress prize, though, being the people who are the closest to the mark after a year (i think).
on top of that, netflix has been doing improvements on their own code in the meantime, and its been looking like around a 1% improvement, also.
Sometimes one person with a different perspective on a problem can see something that a groups of "experts" had never thought of, or had discounted because they assumed it wouldn't work.
That's why a fresh perspective on a problem can be quite enlightening, and why I tend to go ask other programmers for their ideas/comments when I get stuck. I don't know everything, and I sometimes make stupid assumptions or forget to consider certain technquies. No group is immune from this.
Mainframe/UNIX Bit Twiddler and long time Windows/Linux Hobbyist.
The Theorem Theorem: If If, Then Then.
I have perfected the perfect movie recommendation mechanism. It's called a "friend."
I hold a patent on the idea, and I've copyrighted the statement "hey, I saw this movie you'd like."
Slashdot Burying Stories About Slashdot Media Owned
The team is WXYZConsulting.com apparently registered by a data mining professor named Yi Zhang.
Maybe they should run a contest to come up with a better business name? Something that doesn't sound like a fly-by-night operation or a variation of something already in the phone book.
Who really cares? The only area worth mentioning in Quebec is Montreal.
Sure, they're people, blah blah blah...but until they 'get over it' we still won't let them play our reigndeer games.
Remember, these *are* the same idiots that wanted to separate from the rest of thier country, but force the same country to provide them the majority of federal benefits they currently receive as a province.
Let the peppers suffer, I say.
If at first you don't succeed, try, try again. Then quit. No use being a damn fool about it.
Yeah yeah yeah, university coneheads fellating each other for the top prize of "person most lost in a universe of mathematical abstraction and most removed from reality". They should have been banging away at women instead of wasting 15 years of their lives for something that can be used to rate movies online....
What a cult university has become...
It would be nice to see the algorithm used, if only to see if it can be improved.
Help! I'm a slashdot refugee.
There is always someone/something better out there. I guess enough money can bring that out. I bet if they tried again in two months they would find something even better.
I will bend like a reed in the wind.
You know, I've never seen recommendation applications worth much of anything. Ringo was okay, until M$ turned it into Firefly, which died in 1999(?). It will be interesting if this turns out well, or if it turns out like TiVO, which in Patton Oswalt's words, is like "working with a retarded kid." "No, TiVo, NO! Westerns aren't cartoons! / But you like horsies! Liar!"
"Tu fui, ego eris" - Virgil
Progress!
Doesn't it make you feel good to know that our freedoms are protected by politicans, lawyers and journalists.
IMDB's recommendation system for movies is so bad it's funny in it's own right. You could look up a science fiction movie, and it would recommend a head cleaner cassette for a Betamax VCR.
Maybe it is just me, but a RMSE of .9430 is not better than a RMSE of .9419
It looks like everyone has just ignored the fact that netflix engine gave different scores for each section...
I think it also demonstrates how the oft-used mantra of "if it needs to be done, it will be done" doesn't always work without some incentive. One of the hurdles of OSS is that the only things that get worked on are the things that people want to work on. The love of developing software can only get you so far (and wow, has it gotten us far). But for some things to advance, it will need financial backing. It's a prickly problem for the OSS community.
My beliefs do not require that you agree with them.
Formal competitions regarding problems like these may or may not exist.
I think solving one of them (especially under computer science) would lead to significant employment opportunities.
Oh please. It took them years before they figured out how to handle multi-disk sets correctly. Yes, their people must be smart (designing a orders database that scales up to a rapidly growing customer base is not easy), but none of their smarts has been directed at customer-facing technology.
The shortcomings of Netflix recommendation system really have more to do with bureaucratic inertia than lack of technical insight. They started out as a simple online video store, so they designed their site with a "You must rent this movie!" model. When it became obvious that they had to do something really original to survive, they switched to the current business. When they did that, it no longer made sense to use the web site to hype individual movies — but they still do it. Which is why you get recommended a Kung Fu movie just because it was popular among some of the people who liked the Ivory-Marchant movie you also liked. Anybody could design a better recommendation system than that.
So, is this why all my movie recommendations are suddenly 'The Manturian Candiadate' ?
--Ne auderis delere orbem rigidum meum, non erravi pernicose!
I've been selling technology for almost two decades and one thing that I see over and over is that internal IT departments either a.) vastly overestimate their abilities b.) prevent introduction of outside techology providers for political reasons or c.) both. There are several companies where the CIO told me "oh, we're already building that in-house. it will be live next quarter" and years later they still have not successfully implemented that technology. Kudos to Netflix for acknowledging that somebody outside their company might be able to do it better. At most companies, the CIO would have never let this happen and/or the CEO wouldn't have the business sense to challenge internal assumptions.
WXYZConsulting.com registered to a Yi Zhang, eh? Probably co-founded it with Wilfred Xylem. Sounds fishy to me...
What do you mean they cut the power? How can they cut the power, man? They're animals!
Wow, it's almost as though you're saying a system with incentives works better than those without
That's why a fresh perspective on a problem can be quite enlightening, and why I tend to go ask other programmers for their ideas/comments when I get stuck. I don't know everything, and I sometimes make stupid assumptions or forget to consider certain technquies. No group is immune from this.
Bah, every time I ask other programmers for input, their ideas are always stupid, my ideas are much better...
I think I have about 300/500 recommendations that are Agatha Christie movies of one sort or another. Sure, recommend 4 or 5, but I'd like to see OTHER recommendations?!
The RMSE score (lower is better) currently posted by wxyzconsulting.com (0.9430) does indeed beat the CineMatch score (0.9514), which is almost good enough to qualify for the Progress Prize 2007 (0.9419 required), but not close to winning the Grand Prize (0.8563 required), so don't assume that this story means that the contest is over!
Deven
"Simple things should be simple, and complex things should be possible." - Alan Kay
My first thought on how to go about beating Netflix is to write a program that goes and gets more data. I read their rules, and I don't think this violates them. Mine all the movie review sites (The Onion AVClub, Ebert, etc.) Mine all the commentary for the film on IMDB, Amazon, etc. Mine the ratings by age, gender, etc. for the movie on IMDB. I haven't got time to start going into specifics, but use all this data to help associate what they're going to like.
It can't be done for this contest, but I'd suggest that Netflix should start getting more complete profiles on their users (optional, of course) so that they can start to collect more information about the individuals that liked or disliked each movie. They should be able to break down who liked the movie best by age, gender, region, ethnicity, religion, or whatever. This can give them more fine-grained sorting ability.
Can anyone tell me how to set my sig on Slashdot?
At most companies, the CIO would not let it happen because of the political fall-out that woudl ensue, not because they wouldn't recognize that other people have good ideas as well.
..
The fact that Netflix is allowing customer data out of their control (albiet sanitized data) is a major step that many company's would never take out of reasons not related to the technology at all.
And most CEO's don't challenge those internal assumptions not because of a lack of business sense, but again, because of political savy. The higher you go in a company, the greater the importance of poltiical acumen. It's not even mostly about business. It's mostly about political positioning.
Folks can argue all day long that it SHOULDN'T be about that, but in the real world it is. And frankly, good CEO's and good CIO's need to survive year to year too .
One important thing to keep in mind is how the Progress Prize works. The minimum for it is a 1% improvement over last years best score. However, the prize doesn't just go to the first to reach that 1%. It goes to the best algorithm that contest year that beats 1%. So if someone posts a 1% solution now and then in 11 months another posts a 2%, the 2% solution gets the prize.
Due to this, there's a big incentive NOT to post any results until near the end of the contest year, unless your results qualify you for the grand prize. Even then, there is a 30 day window in which other people (or you) can beat your results.
I'm tired of people not realizing that "Prizes" are really just Patronage in desguise. I'm not saying Patronage is a bad thing... far from it. But the idea that Prizes are somehow working shouldn't come as a surprise to anyone with knowledge of 15th century aristocracy.
Pay the people who do the work, don't get people to work for pay.
At least 2 teams have beaten the Cinematch score. Only one has gone past the Probe point.
The cesspool just got a check and balance.
...you might just win yourself a relationship so 'special' that 90% of all couples in America share. You'll buy her jewelry and allow her to spend your money on frivolous trifles, and she in turn will allow you to stick your penis in her vagina. When all else fails, resort to mediocrity!
Best. Webhost. Ever. Dreamhost.
So, I guess you are among the 1 in 3 people that actually has confidence in our leadership? How does it feel to know that 2 out of every 3 people think you are wrong?
I do not fear computers. I fear the lack of them. Isaac Asimov (1920 - 1992)
The nice thing about your post about Gerry was the "Consenting Adults" part, perhaps you are a homophobe and equate homosexuality between consenting adults on the same level of the actions of a pederast with youths under the age of consent?
If you don't vote, you don't matter, so don't waste your time telling me your opinion
uhhhhhhhh.....did you perhaps reply to the wrong story?
That's not patronage. These teams are supporting themselves until any of the prize conditions are met. And only one team will receive the grand prize (and a few may receive the progress prizes) - the rest will have mostly just have spent their own time and money and received no compensation in return. That qualifies it as a contest and a prize.
Ever heard of the Music Genome Project? It's used by a site www.pandora.com to drive a personalized internet radio station. I am extremely impressed at how well it picks new music that appeals to me.
I wonder if a similar project could do the same thing for movies...
According to Netflix, if you just use the global average to predict users ratings, you get a RMSE of about 1.05. Netflix, through years of work perfecting their algorithm Cinematch, can get about 0.947. (They have said that with all the additional information that they have about users and movies, in addition to Cinematch they are competitive with the grand prize level 0.8563.)
So, here's the results:
1.05 trivial algorithm
0.9474 Cinematch
0.9430 WXYZConsulting
0.86 or so Cinematch+other information available to Netflix
0.8563 $1,000,000 prize
So, WXYZconsulting is about 50% to the prize, and that was the easy 50%. It is still a long way to go before you see significant improvement. Furthermore, it isn't clear that anyone could ever get to the $1,000,000 prize. Why? People are fickle, and there must be some minimum level at which they are inconsistent.
There's a good presentation of the business case for Netflix improving this system at http://blog.recommenders06.com/?p=35 It comes down to the fact that new releases cost Netflix significantly more than older ones. They make more money, if their customers rent older movies.
Well ... the difference is that (I don't think) the Dems never claimed moral superiority or that homosexuality was somehow wrong. If two consenting guys want to bump uglies, fine ... knock yourselves out ... please don't send pictures ... but they shouldn't run around preaching how doing so is destroying America right after they get a chance to gargle with some Listerine.
Good people do not need laws to tell them to act responsibly, while bad people will find a way around the laws-Plato
Good! Maybe their recommendation system will start being useful. Currently it starts with nothing recommended and the more things you rate the more it recommends! How does this help me? Shouldn't it start by recommending all the movies and as I tell it what I like it narrows the number of recommendations.
I find being offended by me offensive.
1) Gerry Studds & consenting adult page
2) Barney Frank - was he going around spouting off about how he was religiously & morally upstanding and everyone should live like him?
3) The Democrats are merely pointing out the hypocrisy of the Republicans, since the Republicans entire platform was supposedly how the Democrats were doing all these bad things.
Oh, and last and not least, let's remember to "think of the children" for real this time, since it was an underage youth that was being harassed by a perverted pedophile, who was being protected by that morally superior group. (sick sick sick)
I don't like either party, I just like Republicans a whole lot less at the moment.
The cesspool just got a check and balance.
It's a pretty clear distinction. This is a prize.
{sticks pinky to mouth edge} Mwah... Mwah hah... Mwah ha ha ha haaaaaa!!!!!!!
1. Please define homophobe. I just want to make sure that we're meaning the same things by the terms that we use.
Adults on equal footing are free to do whatever they like in privacy. For what it's worth, regardless of sexual preference I tend to believe that most people live in sexual immorality. In a free society, it's my obligation to tolerate their behavior, as it is theirs to tolerate mine. We need not agree, we merely need to tolerate each other's views. And their right to hold their view is every bit as valid as mine.
2. Homosexuality aside, "between consenting adults" is relevant how? Despite the fact that legally the age of consent in Massachusetts was apparently 16, they were hardly on equal footing. At the time, Studds was 46 years old - 29 years the senior of his sex partner. The elder was a congressman, the younger was his employee. For what it's worth, Studds apparently said he demonstrated "a very serious error in judgement." because "it had been inappropriate to engage in a relationship with a subordinate"
But Herr Heisenberg, how does the electron know when I'm looking?
Just because one person is a hypocrite doesn't make the whole platform unstable.
For that matter - there are Democrats who think that abortion and homosexuality are wrong. Does that make them hypocrites? Does that make the platform of the Democrats invalid?
But Herr Heisenberg, how does the electron know when I'm looking?
They asked me what my Age and Sex are. Guess they already know my Location. Oh crap, POS GTG.
I tip my hats to wxyzconsulting, but I visited their website and it looks Bogus. Could they have been trying to avoid the slashdot effect. They say they are located in Los Gatos, CA. the same area Netflix is located (check Netflix's "job postings" page). Hmmm.... ( i hope they don't give the 1M away to themselves setup by a bogus company...that would do no good) ~= Chris =~
"Where the program goes, the freedom goes with it." -- rms
If a job's not worth doing, it's not worth doing right.
Looks like they already gave away the Grand Prize to "The Thought Gang." This just appeared on the site within the last hour.
And I just finished downloading the dataset... jesus.
# Erik
"The Thought Gang" has the highest ranking at this point but it doesn't meet the criteria of being 10% better than CineMatch's results. It looks likes anything better than .9474 goes into the progress category while any result better than .9419 will go into the grand prize category. This article submission is bit disappointing, especially with the title misleading one to believe the contest has been wrong.
Typo: "Won" not "Wrong"
Kudos to Netflix for acknowledging that somebody outside their company might be able to do it better.
Also known as Joy's Law: "Most of the bright people don't work for you - no matter who you are. You need a strategy that allows for innovation occurring elsewhere."
Actually, it's completely ontopic :)
molmod.com - computing tips from a molecular modeling
True but it works both ways. I've seen plenty of companies harmed by buying in expensive, specialist software that could easily have replaced themselves by leveraging general purpose tools and their own expertese. Speciality software packages are often steaming POS and many business problems are generic e.g. Change management, data entry or archiving.
---
Don't be a programmer-bureaucrat; someone who substitutes marketing buzzwords and software bloat for verifiable improvements.
Sounds more like they are offering to pay to own your IP.
"the power of prizes to accelerate progress"
In graphic design and advertising, holding contests to develop and choose suitable product is considered spec work. It is recognised as being bad for business, bad for the industries, and is discouraged by professional organisations.
The power of contests lies with one client, who has a lot of people work for nothing so the client can get their finished product on the cheap, with little or no risk to themselves. The client who uses contests is demonstrating a lack of commitment to project development, and everyone else suffers financially, which drags the entire industry down. I don't see how it is any different in developing technology.
RTFM; please, I beg you.