Augmenting Data Beats Better Algorithms

Posted by kdawson on Tuesday April 1, 2008 @07:10AM from the tell-it-to-the-dhs dept.

eldavojohn writes "A teacher is offering empirical evidence that when you're mining data, augmenting data is better than a better algorithm. He explains that he had teams in his class enter the Netflix challenge, and two teams went two different ways. One team used a better algorithm while the other harvested augmenting data on movies from the Internet Movie Database. And this team, which used a simpler algorithm, did much better — nearly as well as the best algorithm on the boards for the $1 million challenge. The teacher relates this back to Google's page ranking algorithm and presents a pretty convincing argument. What do you think? Will more data usually perform better than a better algorithm?"

2 of 179 comments (clear)

Min score:

Reason:

Sort:

Re:Is it just me that is surprised here? by gnick · 2008-04-01 07:27 · Score: 5, Informative

The netflix challenge is to arrive at a better algorithm with the supplied data. Actually, the rules explicitly allow supplementing the data set and Netflix points out that they explore external data sets as well.

--
He's getting rather old, but he's a good mouse.
Re:Heuristics?? by EvanED · 2008-04-01 07:33 · Score: 5, Informative

One would hope that the thing that calculates the heuristic is an algorithm. See wikipedia.