SXSW: Nate Silver Discusses Data Bias, the Strangeness of Fame

Nerval's Lobster writes "Nate Silver feels a little odd about his fame. That's not to say that he hasn't worked to get to his enviable position. Thanks to his savvy with predictive models, and the huge readership platform provided by The New York Times hosting his FiveThirtyEight blog, he managed to forecast the most recent presidential election results in all 50 states. His accuracy transformed him into a rare breed: a statistician with a household name. But onstage at this year's SXSW conference, Silver termed his fame 'strange' and 'out of proportion,' and described his model as little more than averaging the state and national polls, spiced a bit with his algorithms. "It bothered me that this was such a big deal," he told the audience. In politics, he added, most of the statistical analysis being conducted simply isn't good, which lets someone like him stand out; same as in baseball, where he made his start in predictive modeling. In fields with better analytics, the competition for someone like him would be much fiercer. He also talked about, despite a flood of data (and the tools to analyze it) in the modern world, we still face huge problems when it comes to actually understanding and using that data. 'You have a gap between what we think we know and what we really know,' he said. 'We tend to be oversensitive to random fluctuations in the data and mistake the fluctuations for real relationships.'"

Comments

silver is honest

[SlippyToad]

I think Silver stands out because unlike too many modern American politicians, he is interested in the facts, and not what bullshit he can use the data to support.

So it's not so much that he's done a fantastic job figuring all this out, it's just that he's fucking honest about the results unlike a certain perpetually-deluded political party I'm sick of naming.

Re:silver is honest

[Anonymous Coward]

It's not about political parties, it's about our disfunctional national media. They're so in love with the "close race" that they'll basically start making stuff up about how candidates are "really close" so just stay tuned to this story, etc. etc. If you look at the actual polling data, nothing about either the 2012 republican primary or general election was ever anywhere near as close as the media wanted you to think. It was refreshing to have Silver there to cut through the bullshit, both in 2012 and in 2008.

Re:silver is honest

[fuzzyfuzzyfungus]

I think Silver stands out because unlike too many modern American politicians, he is interested in the facts, and not what bullshit he can use the data to support.

So it's not so much that he's done a fantastic job figuring all this out, it's just that he's fucking honest about the results unlike a certain perpetually-deluded political party I'm sick of naming.

Arguably, it isn't really politicians who he differs from most meaningfully. Sure, there are a lot of politicians living in absurd contrafactual fantasy worlds; but that is(unfortunately) mostly a product of the fact that they are acting as representatives of people who do exactly the same thing... Pandering is a nonfactual enterprise in the sense that it may involve telling people the most insane lies, if that is what they want from you; but it is an eminently empirical exercise in the sense that you must constantly strive to better understand what people want to hear, so that you can better pander to them.

Where Silver, and his data-driven compatriots, really differ from the traditional is with the 'pundit' class. Pundits are selected pretty much entirely for their ability to tell emotionally compelling stories, with minimal reference to data, and provide marketable column inches and cable news minutes. The better ones, to their credit, are masterful in engaging audience emotions, weaving stories, and other affectively gripping flimflam. However, they tend to be somewhere between extraordinarily weak and overtly hostile to the idea that 'data' rather than 'feelings' can actually provide excellent information about the world, particularly if you use this crazy 'math' stuff that the nerds are always going on about.

Pundits make good TV(and, very conveniently, can offer viewers everything from lowbrow talk radio shouting matches to middlebrow 'public intellectual' posturing with little more than a change in tone and presence or absence of a thesaurus, unlike stat-heads who pretty fundamentally lean on nontrival math); but the kind of suck compared to statistical models.

Re:silver is honest

[WaywardGeek]

Thank God somebody finally decided to talk about TFA! Even though the poster didn't name the party that pisses him off, it's obvious what he meant, and that sent all the flamers over the edge. I can't recall such a long sequence of useless posts on slashdot, and that's really saying something!

Yes, Nate Silver called out the Pundits. Even I could have done that. He went further, and pointed out a significant discrepancy between the national and state polls. He made a gut call and pointed the finger of bias at the national polls, and based his model mostly on the state polls. That he was right isn't what impressed me. Most of the Democratic pundits were already screaming about national poll bias. What impressed me was that he had the guts to point that finger right at the most respected name in polling, Gallup. He totally put his reputation and career on the line. He explained in clear detail why Gallup was wrong, and by how much. He predicted how much Gallup would be wrong, and explained how Gallup would spin their errors as being due to Hurricane Sandy, all before Gallup's final errors. Faced with this onslaught from Nate, Gallup, the most experienced polling organization in the world, following the script Nate predicted point by point. Gallup is about as useful as used toilet paper. They're frankly stupid morons. So, thanks for letting us know, Nate!

Re:Deadlocked into debt

[HornWumpus]

Whenever they 'do something' they make that worse.

The Ds think they don't have a spending problem. The Rs think only the Ds have a spending problem.

That said: In a global 'economic war' deficit spending is the counter to currency pegs. The problem is the end game of that, leave both China and their customers fucked. That and exchange rate shifts that should be between the Yahn and all western currencies appear between the yahn/dollar and other western currencies.