SXSW: Nate Silver Discusses Data Bias, the Strangeness of Fame

Nerval's Lobster writes "Nate Silver feels a little odd about his fame. That's not to say that he hasn't worked to get to his enviable position. Thanks to his savvy with predictive models, and the huge readership platform provided by The New York Times hosting his FiveThirtyEight blog, he managed to forecast the most recent presidential election results in all 50 states. His accuracy transformed him into a rare breed: a statistician with a household name. But onstage at this year's SXSW conference, Silver termed his fame 'strange' and 'out of proportion,' and described his model as little more than averaging the state and national polls, spiced a bit with his algorithms. "It bothered me that this was such a big deal," he told the audience. In politics, he added, most of the statistical analysis being conducted simply isn't good, which lets someone like him stand out; same as in baseball, where he made his start in predictive modeling. In fields with better analytics, the competition for someone like him would be much fiercer. He also talked about, despite a flood of data (and the tools to analyze it) in the modern world, we still face huge problems when it comes to actually understanding and using that data. 'You have a gap between what we think we know and what we really know,' he said. 'We tend to be oversensitive to random fluctuations in the data and mistake the fluctuations for real relationships.'"

Comments

Science is rare

[Hatta]

Very, very little in this world actually happens because the data suggests its a good idea. People make decisions based on their comfort level, tradition, who their friends are, etc. Suggesting that we should listen to the data disempowers the powerful. It's 2013, and the principles of evidence based medicine were only developed 20 years ago, and are still not widely used in practice. We're going to have to wait centuries before evidence based public policy becomes the norm.

Re:Science is rare

[Hatta]

The thing is you can have mountains of data (which we do). The problem is asking the right questions.

The problem is that those in power dont want to ask the right questions because it will force their hand. e.g., we have mountains of data that show that Cannabis is less harmful than most over the counter drugs. Certainly less harmful than alcohol. So we 75,000 of us get together and ask the President a question. "Why can't we regulate Cannabis like alcohol?"

His response amounted to "Cannabis is a dangerous drug", and he never mentioned alcohol once. He didn't even bother dismissing the question, he failed to acknowledge its existence.

Why would he do that? It's obvious, if he actually considered the question, he would have been unable to come up with an answer that would be consistent with our current policy. He knows that, and he doesn't care that the policy is demonstrably harmful.

That, my friends, is the face of evil.

Optimism

[sjbe]

We're going to have to wait centuries before evidence based public policy becomes the norm.

I think you are being optimistic. Very optimistic.

Deadlocked into debt

[sjbe]

Deadlocking them produces the best feasible outcome.

Except on the periodic occasion when we need them to do actually something. You know, like not endlessly raise the national debt because they want to promise everything but don't want to have to tell the voters they have to actually pay for it someday.