How To Import Raw Political Data For Crunching

Ed Pegg writes "For those that want to get their fingers stained red and blue with actual political data, resources beyond 538 and pollster can be accessed. In a blog item for Wolfram Research, Jeff Hamrick gives step by step details for how to import raw data from Mason-Dixon, Rasmussen, and Quinnipiac. Then he uses Mathematica to analyze the political data." Related: Slashdot developer Pudge presented at OSCON in July his own approach to gathering Washington-state polling data for analysis [PDF].

Comments

who would want....

[Anonymous Coward]

...to import bullshit?

Re:who would want....

[chill]

Even bullshit can be used for fertilizer.

Re:who would want....

[WTF Chuck]

The people who would want to import this kind of data are the same people that would prefer to analyze things for themselves, rather than rely on someone else's biased interpretation.

is that all we can crunch?

[jollyreaper]

How about importing raw political candidates for crunching instead? If sacred cows make the best hamburger, what would we make with politicians, chum?

Re:is that all we can crunch?

[CorSci81]

Well, if sacred cows are the best hamburger, I'm guessing whatever you made with politicians would be highly toxic and inedible.

Re:is that all we can crunch?

[Red Flayer]

How about importing raw political candidates for crunching instead?

That's just plain disgusting and in bad taste. You're kidding, right?

Politicians do not crunch when you eat them, unless you deep-fry them first. The proper way to serve them is thinly sliced on a bed of lettuce.

Re:is that all we can crunch?

[unitron]

I keep seeing "IANAL" in comments on the RIAA and wonder to myself, what does the RIAA have to do with sodomy?

You mean besides what the record companies do to the artists and consumers? :-)

Not sure I would want his methods

[damn_registrars]

Pudge presented at OSCON in July his own approach

Considering all of the hardcore conservative journal entries that Pudge has made in just the past few weeks, I'm not sure I want to know how he comes up with his conclusions.

Re:Not sure I would want his methods

[Nutria]

I'm not sure I want to know how he comes up with his conclusions.

That's not very intelligent. Knowing how your enemy processes his data, finding flaws in his analysis should be the first thing an anti-conservative does.

Re:Not sure I would want his methods

[damn_registrars]

Knowing how your enemy processes his data, finding flaws in his analysis

Well, based on the same journal entries that I pointed out earlier, it would seem reasonable to expect that he just takes feeds straight from conservative news sources and then "analyzes" by posting it here on slashdot as well as on his own website.

I would say there isn't really any "raw data analysis" involved. At least if there is, he isn't posting it here.

Although of course reality does have a known liberal bias. So he may choose to simply ignore reality on that basis.

Re:Not sure I would want his methods

[Nutria]

Although of course reality does have a known liberal bias.

This is marked +4 Insightful?

Re:Not sure I would want his methods

[Profane MuthaFucka]

That's complete bullshit. We didn't defeat Hitler by analyzing his data.

(ghost of Turing) Fuck off!

Re:Not sure I would want his methods

[edward2020]

Oh yeah?

Re:Not sure I would want his methods

[Profane MuthaFucka]

Ask yourself, why would the ghost of Turing tell me to fuck off in response to my comment? Jokes just aren't funny when you've got to explain them.

Re:Not sure I would want his methods

[Anonymous Coward]

It's okay, that one had no funny to lose anyway.

Re:Not sure I would want his methods

[pudge]

Obviously you don't really understand a few things.

First of all, despite what you said in a comment below, I do not read any conservative news sources, except when someone gives me a link to them. My main sources of news are CNN.com and PBS NewsHour. I used to regularly read National Review but I don't have time anymore and gave it up.

Second, in at least one of the journal entries you linked to, I defended Obama from what I thought was an unfair attack. And in many of the others, including the posts about Fred Walser and Anne Kornblut and Sarah Palin, what I called "lies" were reported by the mainstream media as such, too.

Third, and most important, however, is that none of this has anything at all to do with my presentation, which is about getting detailed information from government sources about the electorate, or from the voters themselves (about only themselves). That is: geo data for precincts and counties, canvassing data for precincts, individual voter demographic data, and candidate contribution data (in this case, by city).

I come to no conclusions with the data, I simply gather it and make it available in different forms. Any analysis I do is very cursory and obvious (such as, Senator Stevens gets most of her money from Arlington, where she lives, while Walser gets most of his from Monroe, where he lives and his wife is mayor). And some of the data I even got from the Democratic Party itself! ZOMG!

Oh right, I see, anyone who disagrees with you is stupid and evil and should be ignored. Noted for future reference.

Re:Not sure I would want his methods

[damn_registrars]

despite what you said in a comment below, I do not read any conservative news sources

Except that I didn't actually say that. Did you read my comment or are you just retaliating based on what you feel from reading it?

My main sources of news are CNN.com

You aren't actually trying to claim that CNN is a liberal, or even a non-conservative news source, are you? Anyone who has been paying attention to CNN can tell that they have made a hard swing to the right over the past few years. Take a look at how often they turn to Glenn Beck for commentary, or how often they stack the deck for Dr. Gupta to favor the for-profit health care system, even when they have to use incomplete data to do it. And those are just two obvious, recurring conservative elements in CNN. Soon CNN will stand for "Conservative News Network".

and PBS NewsHour

I haven't seen you cite PBS NewsHour in your journal entries anytime in the past month. Maybe you watch it or read it online, but I haven't seen you employ their data in your postings here.

anyone who disagrees with you is stupid and evil

I never called you stupid or evil. I pointed out that you are partisan and biased. Those are two very distinct sets of allegations. I disagree with over 90% of what you say in your journal, but not once did I call you stupid or evil, you added that yourself.

should be ignored

If I was ignoring you, then how would I know that you have written over a dozen highly conservative journal entries in just the past month?

Re:Not sure I would want his methods

[pudge]

despite what you said in a comment below, I do not read any conservative news sources

Except that I didn't actually say that.

Yes, you did. You wrote that "it would seem reasonable to expect that he just takes feeds straight from conservative news sources." This obviously implies that I access such sources on a regular basis. I do not.

My main sources of news are CNN.com

You aren't actually trying to claim that CNN is a liberal, or even a non-conservative news source, are you?

It absolutely is not a conservative news source. For every conservative on there, I can name two liberals. Gupta certainly is no conservative, though he has some free-market leanings. Neither is Lou Dobbs a conservative: though he agrees with conservatives on immigration, he agrees with liberals on trade. Glenn Beck is a conservative, but one of the worst advocates for conservativism on TV, and Anderson Cooper and Jack Cafferty and most of the anchors have a distinct liberal bent.

Besides, I never watch Glenn Beck, Anderson Cooper, Lou Dobbs, and the other pundits and talk shows if I can help it. I watch their straight news programming during the day sometimes.

I haven't seen you cite PBS NewsHour in your journal entries anytime in the past month.

Irrelevant, of course.

If I was ignoring you ...

You directly implied that what I presented should be ignored because of your (false) perception of my biases.

Re:Not sure I would want his methods

[Profane MuthaFucka]

Seems you're right. The moderators on Slashdot, infallable judges of teh funnay that they are, have not smiled on that post.

Re:Not sure I would want his methods

[Nutria]

We didn't defeat Hitler by analyzing his data.

Political battles are different than physical battles. Thus, there is different data to analyze.

In politics, one of the things that some of your operatives (should) monitor is what messages the opposition is spreading in each state, and why they think $THIS is successful and $THAT is not.

Likewise, in war, you try to break the enemy's cryptographic codes, send in spies to collect economic/manufacturing data, try to determine the "mood of the people", fly spy planes to determine where you want to bomb, how successful the bombing missions were, etc.

Re:Not sure I would want his methods

[Profane MuthaFucka]

OH, is that why the ghost of Turing told me to fuck off? Do you suppose that he's pissed that I disregarded his contributions to the cracking of the enigma codes?

Or do you suppose the joke just went over your head. Go back and read my comment again. The moderators didn't get it, but I am surprised that you didn't either.

Re:Not sure I would want his methods

[damn_registrars]

For every conservative on there, I can name two liberals

That is only true if you include the anchors and reporters, who are there to present the news, as opposed to the commentators like Beck and Gupta, who are there to provide commentary on the news. How many liberal commentators has CNN given their own shows to?

You directly implied that what I presented should be ignored because of your (false) perception of my biases.

You are stretching my statement to meet your own assumptions of me. I merely said I wasn't interested in your methods. I came to the conclusion that I am not interested in your methods by reading the journal entries that you have made here on slashdot. At no point did I advocate ignoring you. You inserted that yourself to try to skewer me, it appears.

Re:Not sure I would want his methods

[pudge]

For every conservative on there, I can name two liberals

That is only true if you include the anchors and reporters, who are there to present the news, as opposed to the commentators like Beck and Gupta, who are there to provide commentary on the news.

No, it's not. There are more liberal commentators on there than conservatives. But that said, anchors often DO provide commentary, and reporters sometimes do as well -- did you watch the debate? CNN reporter John King was providing his opinion on the debate as it happened (though I couldn't say what his particular political leanings are) -- and even when not providing commentary their biases affect their reporting.

How many liberal commentators has CNN given their own shows to?

On a weekday, the shows are Wolf Blitzer, Lou Dobbs, Campbell Brown, Larry King, Anderson Cooper. Wilf Blitzer, and even moreso his sidekick Jack Cafferty, are on the left. Lou Dobbs is a populist who straddles both sides of the fence. Campbell Brown seems to play it down the middle; I haven't detected a significant bias in her reporting over the years. Larry King doesn't count because no one cares about him. Anderson Cooper is on the left.

On Saturday: there's Gupta, who is not a conservative, even if he agrees with conservatives on some issues (similarly to Lou Dobbs). Indeed, he has donated money solely to Democrats. There's Tom Foreman, who seems to me to be on the left, but I don't watch him enough to know. There's Roland Martin, definitely on the left. Fareed Zakaria, definitely on the left.

So ... you were saying? I see not a SINGLE conservative with his own show on CNN, and several well-known liberals. (Glenn Beck's show is on Headline News.)

I merely said I wasn't interested in your methods. I came to the conclusion that I am not interested in your methods by reading the journal entries that you have made here on slashdot.

So you falsely attack my "methods" out of the blue, and then say you aren't interested in them. OK, that makes ... um, sense ...

Re:Not sure I would want his methods

[damn_registrars]

Wilf Blitzer, and even moreso his sidekick Jack Cafferty, are on the left.

That Wolf Blitzer. And he's an anchor, not a commentator. I'll give you Jack Cafferty as a liberal commentator, but Wolf Blitzer is not a commentator so he doesn't count in the list of "liberal commentators".

So that would make one

Lou Dobbs is a populist who straddles both sides of the fence

If he's a populist, then he must not be an evil liberal.

Campbell Brown seems to play it down the middle; I haven't detected a significant bias in her reporting over the years.

So then you admit she is neither a commentator nor an evil, evil liberal.

Larry King doesn't count

Larry King is also a reporter, still not a commentator. For that matter have you looked at his interview list? He's had numerous interviews in the past year that would never go to evil, evil, evil liberals.

Anderson Cooper is on the left.

Repeat after me:
Anderson Cooper is not a commentator. You can call him one, but that does not make him one. Anderson Cooper is a reporter. You were ranting about evil, evil, evil, evil liberal commentators. Cooper is not a commentator and therefore does not belong on your list.

there's Gupta, who is not a conservative, even if he agrees with conservatives on some issues

I have seen him agree with conservatives more than liberals. And being as he is called in for commentary on health-related issues, he needs to be examined in the light of the commentary he gives on the same. Which is conservative far more often than not. Remember when he came out with a bag of half-truths in response to Sicko?

Tom Foreman, who seems to me to be on the left, but I don't watch him enough to know

Tom Foreman is an anchor. Still not a commentator.

There's Roland Martin, definitely on the left

I'm not familiar with him, but I see in his background that he is a Christian reporter from Texas. I'll leave that one as a wash.

Fareed Zakaria, definitely on the left.

He is a commentator, though he is regarded as a centrist.

So really, you came up with one actual liberal commentator on CNN that we could verify. And how recently was he hired? And we're talking about someone who doesn't even have his own show outright.

In contrast, CNN made a huge deal out of hiring Glenn Beck. He has his own show every night. He often gets short spots in the morning news on both CNN and headline news to give conservative spin to the morning news while sitting in his studio to remind us he also does conservative radio.

So you falsely attack my "methods" out of the blue, and then say you aren't interested in them.

Did you read the title of the thread? It says Not sure I would want his methods. I never attacked said methods. You spun my statement into an attack on your methods. I was criticizing your extremely partisan journal entries.

Re:Not sure I would want his methods

[pudge]

That Wolf Blitzer.

That's a typo. But I just went through everyone who has a show this week. NONE of them were conservatives.

If [Dobb]'s a populist, then he must not be an evil liberal.

Correct. You asked how many were conservatives, and I went through everyone. He is not a conservative.

Larry King is also a reporter

No, he is not. He never reports. He is an interviewer, not a reporter.

He's had numerous interviews in the past year that would never go to evil, evil, evil liberals.

False.

Anderson Cooper is not a commentator.

Yes, in fact, he is. Unlike Blitzer who keeps his views close to the chest, Cooper, like Lou Dobbs, often injects his view into his "reporting." Cooper is a commentator.

You were ranting about evil, evil, evil, evil liberal commentators.

You are lying. I never did any such thing.

Let's go over what actually happened.

I said I get some of my news from CNN, when you said I took my news from right-wing sources. YOU incorrectly said that CNN is conservative. I said no, it has more liberals than conservatives. Then you tried to limit it to commentators. I never was talking about merely commentators, that was your invention, so then I went over the people actually on CNN.

I have seen him agree with conservatives more than liberals.

I have seen him agree with liberals more than conservatives.

Remember when he came out with a bag of half-truths in response to Sicko?

No. I remember him talking about Sicko, and being mostly correct.

There's Roland Martin, definitely on the left

I'm not familiar with him, but I see in his background that he is a Christian reporter from Texas. I'll leave that one as a wash.

Um. It's not. He is a huge Obama fan, and as a Christian he is on the Tony Campolo wing of the church: decidedly Democrat and Liberal. Just because you're ignorant of him doesn't mean you get to dismiss him as "a wash."

So really, you came up with one actual liberal commentator on CNN that we could verify.

False.

And we're talking about someone who doesn't even have his own show outright.

Martin and Cooper are liberal commentators who have their own shows. And while Cafferty does not have his own show, he is the only regularly featured commentator who has his own segment on the show, so obviously this tilts the whole show to the left. If they had a conservative to balance it out, then you could dismiss it.

In contrast, CNN made a huge deal out of hiring Glenn Beck.

Who does not have a show on CNN, so that's irrelevant.

He often gets short spots in the morning news on both CNN

So do many liberal commentators, so that's irrelevant.

So you falsely attack my "methods" out of the blue, and then say you aren't interested in them.

Did you read the title of the thread? It says Not sure I would want his methods.

Yes. And then you proceeded to falsely attack them, insinuating they were biased.

I never attacked said methods.

You're lying.

Fivethirtyeight.com

[ireallylovelinux]

For those people that don't want to install mathematica you could go to fivethirtyeight.com and get updated polls. I got this from the typewriter xkcd.

Mathematica Advertisment

[Myrddin Wyllt]

This doesn't seem like a serious forecasting project, just a bit of topical advertising fluff thrown together by a couple of Wolfram staffers. It might be fun to download and play with if you have Mathematica already, but it doesn't make me want to rush out and blow two and a half grand on propriety software.

Re:Mathematica Advertisment

[Anonymous Coward]

Particularly when R is free and, being specifically a statistics package, will probably do a better job than a general-purpose mathematics package.

Money *in* politics is more interesting

[TheGrapeApe]

If you want to import data from the *money* in politics, which I generally find more amusing/entertaining check out the Federal Election Commission's FTP site

Pudge's pick for November: Red Sox!

[Anonymous Coward]

By a suspiciously large margin.

election.princeton.edu

[jackchance]

If you want to see some poll meta-analysis that is pretty rigorous check out election.princeton.edu.

What kind of lies? Statistics, there you go then.

[Seedy2]

What I'd really like to see is if the data were replaced with purely random data (or as close as we can get) would the people "analyzing" it get any different results, really? Or would they keep finding the trends they are looking for, do you think?

Soylent Green is Politicians?

[billstewart]

You could make something more useful than chum, pal...

Re:What kind of lies? Statistics, there you go the

[Quiblle Spear]

As a student of mathematics, I say that it is much more difficult to manipulate data to legitimately suggest different results that most would suspect. As they say, the numbers never lie. Of course, the statisticians might lie. Or the statistics might lie, in the sense that there is always a chance of a biased study. But the numbers never imply causation, merely correlation.