Slashdot Mirror
ESPN and TopCoder Run College Football Algorithm Challenge
Mike writes with a timely link to a story about the ESPN/TopCoder Winning Formula Challenge, a combination of fantasy football and competitive programming. The goal is to write an algorithm to predict the outcome of college football games using a collection of historical data provided by the tournament organizers. The season is broken up into 3-4 week chunks that are used to evaluate the results. Prizes will total $100,000.
I'm assuming the winner will be the algorithm that most closely predicted the correct outcome.
Unfortunately, this wouldn't consider whether or not the algorithm just got lucky. Like you said, considering only historical data isn't nearly enough.
Also, given enough algorithms that essentially pick a random outcome (read: many of these simply luck into good predictions), the actual best algorithm could be totally overlooked.
Well, what can you expect when combining ESPN and coding?
I just pooped your party.
The problems you point out are no doubt real, but what other way is there to determine the "best" algorithm? You can get as fancy with statistics and theory as you want, but in what sense could an algorithm that doesn't perform best on the real prediction be "the actual best algorithm"?
If someone truly came up with an algorithm worth it's salt for predicting football games. Why not just go to Las Vegas and make a lot more than $100,000. :P
I vaguely remember this being the reason I ignored Top Coder in the past - a good programmer picks the best tool for the job, and Top Coder only provides the option of using some of the worst tools for any job.
Fair enough, but in the "real world" you often don't get that option. I suppose if you want a pure computer science competition, you should absolutely get to choose your language. A good computer scientist will pick the right language. A good hired gun will work within the constraints given.
It's a shame that ESPN chose to do this through TopCoder, as TopCoder's general practices are poison for a machine learning contest. TopCoder chose to impose a gig memory limit and a nine minute runtime on any approach to this problem, which murders most machine learning tactics right out the door. It's a shame they didn't do this themselves on the NetFlix model, where contestants just submit predictions.
This contest isn't to get football predictions. It's to get football predictions under arbitrary ram and cpu caps. ESPN's staff wouldn't face such restrictions when using the work; there is literally no reason for this limitation to exist.
This contest's design precludes most modern approaches to machine learning to no appreciable benefit, and is therefore fundamentally flawed. ESPN is going to get seriously quality-limited results.
Very disappointing.
StoneCypher is Full of BS