Slashdot Mirror
SXSW: Nate Silver Discusses Data Bias, the Strangeness of Fame
Nerval's Lobster writes "Nate Silver feels a little odd about his fame. That's not to say that he hasn't worked to get to his enviable position. Thanks to his savvy with predictive models, and the huge readership platform provided by The New York Times hosting his FiveThirtyEight blog, he managed to forecast the most recent presidential election results in all 50 states. His accuracy transformed him into a rare breed: a statistician with a household name. But onstage at this year's SXSW conference, Silver termed his fame 'strange' and 'out of proportion,' and described his model as little more than averaging the state and national polls, spiced a bit with his algorithms. "It bothered me that this was such a big deal," he told the audience. In politics, he added, most of the statistical analysis being conducted simply isn't good, which lets someone like him stand out; same as in baseball, where he made his start in predictive modeling. In fields with better analytics, the competition for someone like him would be much fiercer. He also talked about, despite a flood of data (and the tools to analyze it) in the modern world, we still face huge problems when it comes to actually understanding and using that data. 'You have a gap between what we think we know and what we really know,' he said. 'We tend to be oversensitive to random fluctuations in the data and mistake the fluctuations for real relationships.'"
I think Silver stands out because unlike too many modern American politicians, he is interested in the facts, and not what bullshit he can use the data to support.
So it's not so much that he's done a fantastic job figuring all this out, it's just that he's fucking honest about the results unlike a certain perpetually-deluded political party I'm sick of naming.
One day I feel I'm ahead of the wheel / the next it's rolling over me / I can get back on / I can get back on
Very, very little in this world actually happens because the data suggests its a good idea. People make decisions based on their comfort level, tradition, who their friends are, etc. Suggesting that we should listen to the data disempowers the powerful. It's 2013, and the principles of evidence based medicine were only developed 20 years ago, and are still not widely used in practice. We're going to have to wait centuries before evidence based public policy becomes the norm.
Give me Classic Slashdot or give me death!
According to Wikipedia, "SXSW Film and Multimedia", now split in separate "SXSW Film" and "SXSW Interactive" started in 1994, seven years after the music festival did.
Everything is better with chainsaws.
We're going to have to wait centuries before evidence based public policy becomes the norm.
I think you are being optimistic. Very optimistic.
Deadlocking them produces the best feasible outcome.
Except on the periodic occasion when we need them to do actually something. You know, like not endlessly raise the national debt because they want to promise everything but don't want to have to tell the voters they have to actually pay for it someday.
Silver's rise to prominence most closely mirrors Google's. Find a good model and apply it to an area that has been underserved.
When they write the book on the history of the Information Age, it will be about how we learned to leverage analytics for the common good.
Both sides are not equally wrong, but that doesn't mean both sides are right. The Republicans are wrong 99% of the time. The Democrats are wrong 95% of the time. Why can't we field a candidate who's right even half the time?
Because "right" is for better or worse often a matter of opinion. There is no single objectively right answer for many questions. Think abortion or gun control. Lots of opinions on both sides but there is never going to be a single "right" answer. At best there might be a consensus but probably never a unanimous one. Even for questions where a single objectively right answer may theoretically exist, there often is insufficient data to figure out what that answer is. (for example what is the optimal tax rate)
What is evil is how to make generally good people collectively manifest "evil" deeds.
Which is exactly what I described in the above post. The War on Drug Users is the perfect example what Thoreau meant when he said "Law never made men a whit more just; and, by means of their respect for it, even the well-disposed are daily made the agents of injustice."
I think we're both right. It is part of human nature, but it's also evil. Jealousy and greed are also part of human nature, but they still make us do evil things. Good people learn to control those urges. Politicians apparently never learned to control theirs.
Give me Classic Slashdot or give me death!
The Ds think they don't have a spending problem. The Rs think only the Ds have a spending problem.
It's not solely a problem with either the Ds or the Rs really. It's a problem with the voters who elect them. Their disagreements usually are just a symptom of the problem. WE are the ones who demand all these services (medicare, defense, etc) but WE are the ones who vote people out of office who dare to suggest it will cost something and that we might have to pay taxes for them. WE are the the ones who refuse to acknowledge that we might not actually need 11 aircraft carrier battle groups or perhaps we might be ok with a bit less Medicare. Our leaders are to an alarming degree a reflection of our own dysfunction. It's easy to blame them but collectively if we want to point fingers the mirror is a good place to start.
Cathy O'Neil (Mathbabe) offers a well-argued criticism of Nate Silver when he stepped beyond his area of expertise in his recent, popular book, '"The Signal and the Noise: Why so many predictions fail – but some don’t"
http://mathbabe.org/2012/12/20/nate-silver-confuses-cause-and-effect-ends-up-defending-corruption/
According to Ms. O'Neil, Mr. Silver fails to recognize situations where bad models are deliberately used to game that system.