Slashdot Mirror

← Back to Stories (view on slashdot.org)

Netflix Prize Contest Ends, Down To the Wire

Posted by kdawson on from the nail-biter dept.

suraj.sun updates us on the Netflix Prize now that the competition has officially closed. We discussed the new leader with one day to go in the contest: The Ensemble, taking the lead from long-time leader BellKor's Pragmatic Chaos, the first contestant to submit an entry that broke the 10% barrier. In the contest's final day, BellKor re-took the lead with 20 minutes to go, then The Ensemble apparently pulled a Michael Phelps with 4 minutes to go, squeaking ahead by 0.01%. At least so the leaderboard claims — but those numbers are posted by the competing teams. The NY Times reports that an official winner will not be named until September — Netflix needs that much time to pore through the complex entries and read the code. Netflix contacted BellKor on Sunday to tell them the team remained in first place; The Ensemble has had no such notification.

78 of 100 comments (clear)

  1. I'll tell you how they did it by Anonymous Coward · · Score: 4, Funny

    They realized that all movies starring Matthew McConaughey and Kate Hudson were actually the same movie. The compression on that alone was enough.

    1. Re:I'll tell you how they did it by dyingtolive · · Score: 1

      Don't forget Adam Sandler movies.

      --
      Support the EFF and Creative Commons. The war is coming, and they're supporting you...
    2. Re:I'll tell you how they did it by gEvil+(beta) · · Score: 1

      Don't forget Adam Sandler movies.

      Hey now! Punch-Drunk Love is an amazing non-Adam Sandler Adam Sandler movie. But of course, it's the only one.

      --
      This guy's the limit!
    3. Re:I'll tell you how they did it by Knackered · · Score: 2, Funny

      Don't forget Adam Sandler movies.

      Damn you, I was trying to.

      --
      a.
    4. Re:I'll tell you how they did it by owlnation · · Score: 1

      Hey now! Punch-Drunk Love is an amazing non-Adam Sandler Adam Sandler movie. But of course, it's the only one.

      There's also "Reign Over Me"

    5. Re:I'll tell you how they did it by c_forq · · Score: 1

      Yeah, but that doesn't count because it is the exception that proves the rule.

      --
      Computers allow humans to make mistakes at the fastest speeds known, with the possible exception of tequila and handguns
  2. I am still waiting... by jeffmeden · · Score: 2, Insightful

    What they need to start is a contest to improve their incredibly lousy on-demand service, the Silverlight player is beyond terrible. All this effort (and money) over getting 10% more accurate guesses that the same guy who liked "Terminator" will like "Terminator 2" is nice and all, but it's a bit of a time waster don't you think?

    1. Re:I am still waiting... by dyingtolive · · Score: 2, Informative

      Its not about the algorithm for movie sorting. Imagine situations where having a 10% more accurate guess would actually count for something important. Now imagine licensing and patenting that algorithm and building revenue from that.

      --
      Support the EFF and Creative Commons. The war is coming, and they're supporting you...
    2. Re:I am still waiting... by 0100010001010011 · · Score: 1

      Except some of us also like movies that may not have had huge releases or made before we had internets...

      If it wasn't for the internet I wouldn't have heard of Equilibrium (which was pulled back because of 9/11).

      Right now I go through nzbmatrix and open the IMDB links and see what I really like and give it a shot.

      Go, The Man who Knew Too Little, Equilibrium, Dark City, etc are all movies I've seen that I hadn't heard of because I was either in Middle School or they had limited releases.

    3. Re:I am still waiting... by furrer · · Score: 1

      Upgrading to Silverlight 3 improved the on-demand image quality and fixed the annoying screen lock issue (at least for me on 64-bit Vista). Now if they could just get some more good movies on there...

    4. Re:I am still waiting... by lefiz · · Score: 1

      Apparently Netflix is stuck on this side of the service (the on-demand shizz) due to the arcane Hollywood studio system that has contracts with cable, premium stations, and others that lock up the movies for literally decades leaving new service providers like Netflix with no options. See discussion here: http://slate.com/id/2216328/

    5. Re:I am still waiting... by Anonymous Coward · · Score: 1, Interesting

      No joke.

      I was watching a show last night (Leverage, which Netflix suggested for me at a 4.6 stars and they were right, I love it.). 3-4 times per hour episode it would reset and tell me that it was adjusting things. Often, it never came back until I hit Refresh. I noticed that it would only buffer a small amount ahead (30 seconds) and then try to keep it there, calling an incredibly laggy site called something like controls.netflix.com or something like that. Once that happened, it had about a 25% chance of locking up completely.

      They seem to be trying to save bandwidth by only allowing your movie to get 30 seconds ahead, which is way too short.

    6. Re:I am still waiting... by peipas · · Score: 3, Interesting

      I, for one, think the Silverlight player is phenomenal.

      I have limited Internet options-- even though I'm living urban I am not close enough to a CO to get decent DSL speeds (the max Qwest offers is 1.5Mbps). Cable is not an option because my complex has a contract with the television provider who wired the buildings at construction, which is good for those who watch any TV since you get 50+ channels of cable television for free, but bad for Internet options.

      Long story short, my Internet connection has a very high bit error rate percentage because I am getting my DSL over Qwest's line but from an ISP (AT&T via Covad) willing to boost the artificial limit of 1.5Mbps Qwest imposes to 3Mbps, at the expense of a quality signal. This results in being able to truly realize the faster speeds, but also in having a very burst-y connection.

      I find the new Silverlight player to be far superior with its buffering saving the day, allowing me to watch Netflix streaming at maximum quality. The fact that the Silverlight player adjusts quality on the fly is outstanding as well-- when I first start streaming content it may look like shit at first but after a short time it is crystal clear, it realizing my connection can support the data load with a little buffering.

      By contrast, with the old player, even before I had this error-ridden Internet connection, I would find myself initiating an instant streaming session only to find the stupid player would decide my connection was slow and give me piss poor video quality. I would have to click the "Back to Browsing" button and reinitiate the streaming several times sometimes in order for it to give it to me in high quality.

      The new player also provides a great new feature when seeking through the content, where it will scroll past freezeframes of the content as you scroll forward or backward, which is perfect for skipping the intros for TV shows, for example.

      I only wish it would "back buffer" a little because currently when I rewind a little bit, rather than replaying it from memory it rebuffers altogether, as if I hadn't just watched those few seconds prior.

    7. Re:I am still waiting... by shawn(at)fsu · · Score: 1

      Speak for yourself. I use the silverlight service at home at it works great for me. I output from my computer to my LCD TV and it looks good to me. As a bonus since it's not flashed based I can full screen it on my TD and still do stuff on my monitor.

      --
      500 dollar reward for tip(s) leading to the arrest of the person(s) who stole my sig.
    8. Re:I am still waiting... by adolf · · Score: 4, Interesting

      I, for one, think that the Silverlight player is crap.

      I have a dual-head machine with a very nice 1600x1200 IPS-panel NEC LCD as my primary monitor, and a nice (but far lesser) 24" 1920x1080 TN-panel Asus LCD as the secondary.

      I want to pop up a Netflix show on the secondary monitor, full-screen, and continue to do stuff like read Slashdot on the other display. Silverlight has no problem putting good-quality video up, full-screen, on the second display -- but as soon as I click outside that window (ie, to browse Slashdot), it shrinks back down to windowed mode. My dual-head computer is therefore retarded into being effectively a single-head machine for the duration of the film, unless I either want to watch it in a little window or soak up a couple of cores worth of CPU power zooming in with Ctrl-+.

      Allegedly, if I had Media Center on my computer, this could be worked around. But with Netflix + Silverlight, it cannot be accomplished. Of course, this situation works fine if I'm playing a DVD on my own computer -- it just doesn't work with Netflix's streaming service.

      It is therefore retarded (in a very literal sense of the word).

      I'd like also to note that Flash seems to have the same difficulty, and that its behavior is similarly inexcusable and retarded.

      The best I can do, if I want to watch a film in my office and occasionally fuck around on the Web, is fire up my 4-year-old laptop and use that to browse with instead of my badass dualhead desktop rig. Which, also (and obviously) is retarded.

      I've complained to Microsoft Silverlight developers directly about this, and the best they ever say is something like "You're right. It is retarded. Maybe we'll fix it some day. *harumph*" while months/years pass by and it's still an issue.

      --
      Kid-proof tablet..
    9. Re:I am still waiting... by xmas2003 · · Score: 1

      Mod parent +999 ... this is a serious PIA and it would be nice if this was not the behavior.

      --
      Hulk SMASH Celiac Disease
    10. Re:I am still waiting... by cparker15 · · Score: 1

      And if it weren't for you, I wouldn't have heard of Equilibrium. Thanks!

      --
      Have you driven a fnord... lately?

      You must wait a little bit before using this resource; please try again later.

    11. Re:I am still waiting... by Ajaxamander · · Score: 1

      I heard Equilibrium had limited release because they blew through their budget on actors & effects, then couldn't afford marketing. Do you have a source?

      NB: I don't claim to have a source for *my* assertion, only that the movie is clearly NOT low-budget, but only ran for a short time in my local theaters and has one of the most cheaply-made DVD releases I think I've ever seen. (My buddy's brother made liner notes for it with a chapter listing because there was NOTHING in the box with the disc.)

      Also I heard a rumor to that effect. I'd love to know the real scoop either way.

    12. Re:I am still waiting... by exhilaration · · Score: 2, Informative

      I get this same behavior with every desktop video player I've used. The flaw is with Windows, not Netflix, not Silverlight, not VLC or any other video player.

    13. Re:I am still waiting... by Oldstench · · Score: 1

      Considering it's such a terrible, terrible movie, is missing out on Equilibrium such a bad thing?

    14. Re:I am still waiting... by JacksBrokenCode · · Score: 1

      Well they certainly didn't blow through it on wardrobe and props. They rocked motorcycle helmets and unbadged Cadillacs in that twisted dystopian future state...

    15. Re:I am still waiting... by omega_dk · · Score: 1

      It's user error if you're having the issue with VLC. Try moving the controller window to the other monitor.

      --
      Just because you don't like the truth, does not make it false.
    16. Re:I am still waiting... by Hurricane78 · · Score: 1

      Of course it is great, when all your know is Flash. Flash is one of the worst things that ever happened to internet video.
      I hope browser-video-integration (like in Firefox 3.5, but with all formats, like H.264, XviD, MP3, AC3, Matroska containers, etc.) will change this for the better, and kill off Silverlight too.

      Look at this (with Firefox 3.5), if you don't think it can kill Silverlight: http://people.mozilla.com/~prouget/demos/

      --
      Any sufficiently advanced intelligence is indistinguishable from stupidity.
    17. Re:I am still waiting... by adolf · · Score: 1

      You read poorly.

      I specified that it works fine with a DVD on my own computer. And that it works fine with Media Center, which is just another player (though rather more far-reaching than most).

      If it's possible with some applications but not others, then it is plainly an application problem and not a Windows problem.

      --
      Kid-proof tablet..
    18. Re:I am still waiting... by Tablizer · · Score: 1

      For all this bickering about whether Silverlight is any good, in my experience offering multiple viewers is the better way to go: each viewer will work better on different machines. If they have a choice of say 3 viewers, then one is bound to work.

      --
      Table-ized A.I.
  3. these guys are all improving each other's code by circletimessquare · · Score: 2, Interesting

    team a makes algorithm improvement b

    team c takes algorithm improvement b and makes algorithm improvement b(+d)

    team e takes algorithm improvement b(+d) and makes algorithm improvement b(+d)->f

    the guy who squeaked out the extra 0.01% did that on top of someone else's code that eked out 0.05%, etc., ad nauseum

    so how do you ascertain who won? all the teams won

    they should take the final prize money and try to fractionate each incremental improvement in the algorithm and proportionally dole out the money that aways. anything else is unfair

    --
    intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
    1. Re:these guys are all improving each other's code by AlXtreme · · Score: 5, Insightful

      so how do you ascertain who won? Netflix won

      There, fixed that for you.

      --
      This sig is intentionally left blank
    2. Re:these guys are all improving each other's code by johnsonav · · Score: 5, Insightful

      so how do you ascertain who won? all the teams won

      No, they didn't, at all.

      Any bozo can get 5% improvement. It's the last 5% that's tough. And, of that last 5%, the first 2.5% is cake, compared to the last 2.5%.

      they should take the final prize money and try to fractionate each incremental improvement in the algorithm and proportionally dole out the money that aways. anything else is unfair

      As someone who participated, but did not win, the first place team deserves the entire million (if not more). This was a race, and second place is the first loser.

      --
      ... and that's when the C.H.U.D.'s came at me.
    3. Re:these guys are all improving each other's code by flynt · · Score: 1

      The code is not shared unless they wanted it to be. You can't take the current leader's code and make an improvement unless the team gives you their code. This was not how the contest was run.

    4. Re:these guys are all improving each other's code by gurps_npc · · Score: 2, Informative

      Not exactly. Team A makes algorithm improvement Z Team B hears about Team A's work and creates something SIMILAR. They then fine tunes it to Algorithm Y Team C hears about A & B's work. They try and duplicate it, but can't. Instead they fine they come up with X, something they thought was Y but really wasn't. Team D hears about A,B and C. They find a way to combine Y and X, making W. It is clearly original, but is based on similar ideas for Z/Y and X.

      --
      excitingthingstodo.blogspot.com
    5. Re:these guys are all improving each other's code by Spy+Hunter · · Score: 5, Insightful

      According to the contest rules the winning algorithm will not be exclusive to Netflix but will be published to the public, so we all win.

      --
      main(c,r){for(r=32;r;) printf(++c>31?c=!r--,"\n":c<r?" ":~c&r?" `":" #");}
    6. Re:these guys are all improving each other's code by HeronBlademaster · · Score: 1

      Exactly.

      I do find it interesting that one of the winning teams (BellKor's Pragmatic Chaos) is a merger of three other top ten teams, and I wouldn't doubt it if you told me The Ensemble is a merged team as well (especially given its name). I certainly can't fault them for merging. Why risk winning nothing, when you can split a cool million with two other teams and have a much better chance of winning?

      What would be funny is if the last two teams merged at the last minute ^_^ (Not feasible, I'm sure, but it would be humorous.)

    7. Re:these guys are all improving each other's code by timeOday · · Score: 1

      BTW, "Ensemble" is a particularly cute name for a team in this contest since, in machine learning, ensemble methods are compositions of other machine learning algorithms to achieve (hopefully) higher classification algorithm than any of the component algorithms alone.

    8. Re:these guys are all improving each other's code by LandDolphin · · Score: 1

      Yes, The Ensemble is a merged team as well in an attempt to beat BellKor as the clock was winding down.

      --
      Spelling and Grammar errors have been added to this post for your enjoyment
    9. Re:these guys are all improving each other's code by swillden · · Score: 4, Insightful

      because we all need to provide movie suggestions to our millions of users?

      Because there are many, many organizations who can effectively use a high quality, freely-available automatic preference identification system. Some aspects of the winner are probably very specific to movies, but I'm sure most of the core, and the underlying ideas, are relevant to any sort of preference identification and clustering.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    10. Re:these guys are all improving each other's code by daveime · · Score: 1

      Unfortunately, this is not the case. Each year, the winner of the Progress Prize WERE obliged to publish details of their algorithms to date ... and it was very clear that all the other competitors jumped on these details.

      The contest was very clearly defined by these periods (sometimes of 11 months) where no real progress was made except by 1 or 2 top teams, and then 1 month right after publication of the latest ideas and algos, which caused everyone to catch up significantly by clearly copying and tweaking those published ideas.

      Now of course this is a kdawson edited article, so the bit about "the scores on the leaderboard being published by the teams themselves" is complete nonsense.

      However, the set on which the algos are evaluated is split into two subsets, the results of one IS published on the leaderboard, but the ultimate winner will be determined by their results on the unpublished half.

      It seems that Belkor et al HAVE done better on this unpublished half, and personally I feel they SHOULD be the winners as the Ensemble was johnny-come-lately collection of groups of mergers of individuals, and there is some debate over hundreds of dummy accounts that were created in the last days (possibly by members of the Ensemble, possibly not) to game the system and basically train against the Oracle (at least the half that results are published for).

      This is why I suspect Ensemble's result is overtrained on the published half, but hasn't done as well on the unpublished half.

      Bellkor deserve this for the simple fact they've been at the top of the leaderboard for about 98% of the whole three years the competition has been running, and for also having won both yearly Progress Prizes.

      And I also think Brandyn Webb deserves some credit for his SVD (singular value decomposition)work, which really helped everyone and spurred a massive leap in machine learning ideas and the general interest in the competition. I believe he does have a patent pending for his algorithms, might be interesting to see how this affects Netflix in future, as I think every competitor has implemented and used SVD in some form as part of their final results.

      Anyway, go Bellkor (yes, I'm biased too) :-)

    11. Re:these guys are all improving each other's code by DNS-and-BIND · · Score: 3, Interesting

      Uh, that's a pretty disgustingly American viewpoint of the issue. Can't we all agree that if you didn't come in first, then you can still be a winner? This has been taught in schools for a long time now, it still hasn't been internalized?

      --
      Shutting down free speech with violence isn't fighting fascism. It IS fascism!
    12. Re:these guys are all improving each other's code by ryturner · · Score: 1

      Uh, that's a pretty disgustingly American viewpoint of the issue. Can't we all agree that if you didn't come in first, then you can still be a winner? This has been taught in schools for a long time now, it still hasn't been internalized?

      No, this was a contest to see who could improve the algorithm the most. There can be only a single winner. If the contest was to improve the algorithm by x%, then there might be multiple winners. If you start calling everyone a "winner", you just cheapen the experience for the true winner.

    13. Re:these guys are all improving each other's code by johnsonav · · Score: 1

      Uh, that's a pretty disgustingly American viewpoint of the issue. Can't we all agree that if you didn't come in first, then you can still be a winner?

      No, we can't; not unless you want to redefine the word "winner" in such a way that it loses all meaning.

      That's not to say that the people who didn't win, don't get anything out of the experience. I learned a hell of a lot about recommender systems. But, I still did not "win". I'm okay with that.

      This has been taught in schools for a long time now, it still hasn't been internalized?

      And I hope it never will be.

      --
      ... and that's when the C.H.U.D.'s came at me.
    14. Re:these guys are all improving each other's code by quadrox · · Score: 1

      It's this childish view that makes people think they should take up tasks they are not suited for, boldly proclaim "facts" they don't know anything about and in general behave unresponsibly.

      Kids/People need to know that life isn't always fair, not everyone is a winner and not people just aren't equal (in an individual sense).

      Telling them this stupid crap just makes them have unrealistic expectations and gives them a sense of entitlement that won't do them any good and quite probably will cause problems during their entire life.

    15. Re:these guys are all improving each other's code by Son+of+Byrne · · Score: 1

      Actually, that's a pretty disgustingly realistic viewpoint of the issue. I'm sorry that you think Americans are competitive bastards who only care about winning, but you're wrong: if you don't come in first, then you lose. Pretty simple (even for a competitive bastard).

      Now, I agree that whether I win or lose (note there's only two categories there) is irrelevant when it comes to the experience. I can still enjoy a competition thoroughly and lose miserably.

      There have to be winners and there have to be losers. How dare you perpetuate the ridiculous notion that we should whitewash over a young person's obvious intellectual superiority with a healthy dose of "No one is any better than you."

      --
      I'd happily pay you Tuesday for a biopsy today!
    16. Re:these guys are all improving each other's code by quadrox · · Score: 1

      Yeah, that particular part was probably a bit harsh. But the GGP was just trying to make a point (there can only be one winner), not trying to belittle the efforts of the others. It was a rhetorical hyperbole - not an actual "insult".

      The GP may have tried to do the same, but I find his message far less clear. If I have misinterpreted his post, I apologise.

    17. Re:these guys are all improving each other's code by simplerThanPossible · · Score: 1

      If you ain't first, yer last!

    18. Re:these guys are all improving each other's code by bluefoxlucid · · Score: 1

      Why is it us intellectual superiors are the rich, powerful, successful people; but the people who are edging by are the happy ones with exciting lives and goals?

      --
      Support my political activism on Patreon.
    19. Re:these guys are all improving each other's code by johnsonav · · Score: 1

      Yeah, that particular part was probably a bit harsh.

      Harsh? I'm like the 300th loser. I'd love to be the first loser.

      --
      ... and that's when the C.H.U.D.'s came at me.
    20. Re:these guys are all improving each other's code by quadrox · · Score: 1

      it's all relative man... To anyone not involved a statement like yours seems a bit harsh.

      At least if they don't understand in what context you meant it. Which someone apparently did not.

    21. Re:these guys are all improving each other's code by sonamchauhan · · Score: 1

      > Unfortunately, this is not the case. Each year, the winner of the Progress Prize WERE obliged to
      > publish details of their algorithms to date ... and it was very clear that all the other competitors
      > jumped on these details.

      Unfortunately? :) What's wrong with it? They win money, and NetFlix get them to disclose the details.

      This is just great for humanity as a whole.

  4. Pulled a Michael Phelps? by basementman · · Score: 1

    So The Ensemble ripped bong with 4 minutes to go which gave them the creativity to squeak ahead by %0.01?

    --
    A Magic the Gathering Article and Forum Aggregator
    1. Re:Pulled a Michael Phelps? by Dahamma · · Score: 4, Funny

      No, they lost to a German wearing a polyurethane suit and then declared they wouldn't race any more until the suits are banned.

  5. !those numbers are posted by the competing teams by kilraid · · Score: 4, Informative

    Netflix calculates the score shown on the leaderboard from a set of rating predictions submitted by a team. The team does not, and will not, know the correct answers. For testing their algorithms, the teams use another dataset. The two datasets, part of the package made available to the competitors, are known as "qualifying" and "probe".

  6. Go Bellkor!!! by jaydonnell · · Score: 1

    Go Bellkor!!! Sorry, I'm biased :)

  7. Published scores irrelevant by Spy+Hunter · · Score: 5, Informative

    The reason BellKor is still first is that the published scores are irrelevant. The scores that matter for the prize are based on an unpublished data set known only to Netflix (to prevent people submitting answers that are optimized for the challenge data and work poorly on everything else). On this secret data set, BellKor's algorithm apparently performs better than The Ensemble's.

    --
    main(c,r){for(r=32;r;) printf(++c>31?c=!r--,"\n":c<r?" ":~c&r?" `":" #");}
    1. Re:Published scores irrelevant by currivan · · Score: 1

      I don't think this is quite clear. The teams submit their predictions on a qualifying set of about 1.8 million user/movie pairs. Half of these are used for the published leaderboard rankings, and the other half secretly scored for the actual prize. What the teams don't know is which user/movie ratings are in which half. The teams aren't submitting programs that Netflix runs though, they're submitting predictions on the whole quiz set.

  8. Re:What contest? by panoptical2 · · Score: 3, Informative

    The contest has been going on for several years straight, and /. has had several stories about it. The article takes knowledge of the contest as a given.

    See Wikipedia and Netflix's own site for details.

  9. Re:What contest? by CaseCrash · · Score: 5, Informative

    Where have you been?

    [2009-07-26]New Leader In Netflix Prize Race With One Day To Go
    [2009-07-26]Netflix Prize May Have Been Achieved
    [2007-11-27]Anonymity of Netflix Prize Dataset Broken
    [2007-11-14]Close but no Cigar for Netflix Recommender System
    [2006-10-02]Build a Better Netflix, Win a Million Dollars?
    [2008-11-22]Interest Still High In the Netflix Algorithm Competition
    [2009-10-09]Netflix Prize Competitor Already Beats Netflix

    etc..

    --
    No, that link you posted to a web comic we've all seen a hundred times is not "obligatory."
  10. Which leads me to wonder, what is by DRAGONWEEZEL · · Score: 1

    I don't use netflix, i'm a blockbuster guy cause we happen to have one close to our house. But What is 10% of zero? in all seriousness, what is their accuracy now? How is it determined?

    --
    How much is your data worth? Back it up now.
    1. Re:Which leads me to wonder, what is by MaskedSlacker · · Score: 1

      # of suggested matches people actually say they like / Total # of suggested matches.

      Was it really that hard to guess?

    2. Re:Which leads me to wonder, what is by DRAGONWEEZEL · · Score: 1

      I don't guess when it comes to statistics. Because there are usually a ton of variables and long algorythms. It's not usually just x/y. Often, it's more like X(a+b-sqrt(z)(2))/Y(.8(z^2)).

      That can only work for people who vote. It's a selection bias or something if I remember right from stats. For instance, I don't want to fill out an effing survey everytime I buy a product. I just want the thing in my hands as soon as possible so I can use it. When it's use is up, rentals get returned, disposables get recycled, and Blackbutte porter gets turned into Bud "Lite".

      So 1. that means that people have to rent the selection suggested, two actually have watched it when they return it, 3rd vote on how well they liked it. Is it a 1-10 scale? or a yes no?

      Come on this is /. I expected at least a little more detail.

      --
      How much is your data worth? Back it up now.
    3. Re:Which leads me to wonder, what is by MaskedSlacker · · Score: 1

      I don't guess when it comes to statistics.

      Fair enough.

      Because there are usually a ton of variables and long algorythms. It's not usually just x/y. Often, it's more like X(a+b-sqrt(z)(2))/Y(.8(z^2)).

      That can only work for people who vote. It's a selection bias or something if I remember right from stats.

      It isn't necessarily a selection bias. It can create one, but it doesn't have to. If "people who vote" are representative of netflix customers in general, there is no selection bias. But you are right that selection bias is a concern.

      For instance, I don't want to fill out an effing survey everytime I buy a product. I just want the thing in my hands as soon as possible so I can use it. When it's use is up, rentals get returned, disposables get recycled, and Blackbutte porter gets turned into Bud "Lite".

      So 1. that means that people have to rent the selection suggested, two actually have watched it when they return it, 3rd vote on how well they liked it. Is it a 1-10 scale? or a yes no?

      Come on this is /. I expected at least a little more detail.

      You must be new here.

    4. Re:Which leads me to wonder, what is by Wildclaw · · Score: 3, Informative

      They used root square mean for the competition.

      Basically, the difference between the guess and the real answer for each vote is squared giving a value between 0 and 16 (as the biggest error is 4 when you guess 5 on a vote that is 1 or vice versa). This is summed up for each vote in the test and then divided by the number of votes in the test. Finally you take the root of that.

      The winner score in the competition is around 0.855. Which is smaller than 0.9514*0.9 score. Where 0.9514 was the result scored by the netflix algorithm.

      I hope that explains everything.

  11. I'm just an aerospace engineer but.... by Shane112358 · · Score: 2, Interesting

    ...I'm sitting here wondering how stable these algorithms are over long periods of time. I'm assuming that the "practice" data set and the "test" data set are equal in terms of time distribution (date of movie release; date of review). But 10 years from now, 20 years from now, I see the RMSE numbers slowly drifting upwards as the algorithm was optimized to the 2000-2009 data set, not the 2000-2020 data set or tahe 2000-2030 data set. But this is not my area of expertise so I'm wondering what others have to say on this topic.

    1. Re:I'm just an aerospace engineer but.... by jdigital · · Score: 1

      Each year that the competition was running NetFlix awarded a progress prize to the best ranked team at the end of the year. Part of the requirements of winning this prize is publication of scientific papers describing key elements of their algorithms. BellKor's Yehuda Koren presented a paper at SIGKIDD in July describing the improvements they made to their algorithm to take advantage of predictable temporal dynamics of ratings. Check out the paper here

      From an article about the paper:

      While movies themselves stay the same, the humans who rate them are anything but static. As Koren puts it, "The way I rate movies today can be very different from how I rate them even tomorrow." To the frustration of Netflix Prize contenders, a four-star rating can mean "great" or merely "so-so" depending on the user's current mood or comparisons with other recently seen movies. Besides such erratic shifts in the rating scale, people's actual tastes tend to change over time -- as when someone tires of action films, for example, and develops a yen for screwball comedies. There's an overall rise in ratings over the years, as well: for various reasons, a typical movie's ratings become more favorable as the DVD ages.

      --
      :wq ~ ~ ~ ~ ~
    2. Re:I'm just an aerospace engineer but.... by im_thatoneguy · · Score: 1

      Ideally it wouldn't be make any difference.

      The idea behind the solution is that it can be applied to any set of data where people make a preference regardless of 'type'. That could be cars, politicians or slashdot comments.

      "Based on the way you previously have moderated slashdot comments and the way other moderators have previously ranked similarly to you we think these comments will be +5 insightful to you."

  12. And the winner is BellKor, in a quantum finish by Utopia+Tree · · Score: 1

    No fair! You changed the outcome by measuring it!

    1. Re:And the winner is BellKor, in a quantum finish by bluefoxlucid · · Score: 1

      This is the special olympics?

      --
      Support my political activism on Patreon.
  13. Comment removed by account_deleted · · Score: 2, Interesting

    Comment removed based on user account deletion

  14. Re:What contest? by romcabrera · · Score: 2, Insightful

    They have invented this nifty thing called "the Google", you know?

  15. Am I the only one...? by quadrox · · Score: 1

    Am I the only one on slashdot who refuses to ever install silverlight?

    Yes, I'm an anti MS fanboy, but after the things that happened during the OOXML ISO approval I just cannot support that company anymore. In any way whatsoever.

    If you too (speaking to the general audience here) feel this way about MS, I hope that you do not support them by installing silverlight or in any other way really.

  16. Re:What contest? by quadrox · · Score: 1

    Yes, and I remember reading about it, but it took me quite some time to remember what particular contest they were talking about. It wouldn't have hurt to either give a link to the description page or put in half a sentence about the contest goals.

  17. Re:What contest? by quadrox · · Score: 1

    Well, even though I rad slashdot more or less every day, all those stories have eloped me for some reason or other (vacation? boring?). I remember only the first announcement of the contest back then.

  18. Re:What contest? by hesiod · · Score: 1

    Yeah I had no idea what the hell this was about either.

  19. Secret formula by Alzheimers · · Score: 1

    If ((MovieReleaseDate > 2000) OR (MovieProducer = "Bay, Michael")) AND (CustomerAge > 20) Then
      Score = Score * .5

    If (MovieFABRating = "Nudity") AND (MovieLeadActor != "Cohen, Sasha Baron")
      Score = Score * 10

  20. Here's the Netflix thread where Koren leaks he won by currivan · · Score: 1

    http://www.netflixprize.com/community/viewtopic.php?id=1498

    "Thanks. In fact, this is a very happy day for us - our team is top contender for winning the Grand Prize, as we have a better Test score than The Ensemble. (Probably this is the first post revealing this in the forum smile)"

    Also, Yehuda Koren is at Yahoo now, not AT&T.

  21. Re:What contest? by david_thornley · · Score: 1

    I see you can get Slashdot articles from October 2009. Could you tell us who your ISP is? I could use a connection that fast.

    --
    "When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
  22. All Incremental Voted Systems by physburn · · Score: 1
    I was hoping one of these groups would invite some new and powerful algorithm for categorisation. Its a nearest match/ similarity problem, so any algorithm would be very useful else where. But they just seem to have taken existing many techniques and merged them, by combining the votes/scores from each algorithm, in a way thats probably fits just the movie matching problem and not anything else.

    ---

    Information Retrieval Feed @ Feed Distiller

  23. On Squaring by Tablizer · · Score: 1

    I always wondered why squaring is always used for distance measurements. It seems it would over-magnify the influence of non-matches and outliers compared to what the "real world" would want. I realize it makes the math easier, but having easier math and having the best answer may not be the same thing. I've asked math experts about this, but it seems it's an under-studied question.

    --
    Table-ized A.I.
    1. Re:On Squaring by Wildclaw · · Score: 1

      Well. You pretty much answered the reason why, just by saying "measuring distance". Root Mean Square is the formula for measuring distance in an n-dimensional euclidean space.

      But you are right to question if it is the best solution to some real world problems. Why should you represent a person's opions about movies, as a location in an n-dimensional euclidean space?

    2. Re:On Squaring by Tablizer · · Score: 1

      Perhaps I misunderstood the original message. I'm mainly thinking about regression formulas.

      --
      Table-ized A.I.