Slashdot Mirror
Daily Kos Pollster Made Up Numbers
jamie found a story up on Daily Kos revealing that the polling firm they had contracted with for 18 months, Research 2000 or R2K, apparently made up or at least manually tweaked its polling results. The blog published a preliminary report by a team of statistics gurus (Mark Grebner, Michael Weissman, and Jonathan Weissman), and it is an exemplar of clarity and concision. The team reports, "We do not know exactly how the weekly R2K results were created, but we are confident they could not accurately describe random polls." Daily Kos will be filing a lawsuit against its former pollster. "For the past year and a half, Daily Kos has been featuring weekly poll results from the Research 2000 (R2K) organization. These polls were often praised for their 'transparency,' since they included detailed cross-tabs on sub-populations and a clear description of the random dialing technique. However, on June 6, 2010, FiveThirtyEight.com rated R2K as among the least accurate pollsters in predicting election results. Daily Kos then terminated the relationship. One of us (MG) wondered if odd patterns he had noticed in R2K's reports might be connected with R2K's mediocre track record, prompting our investigation of whether the reports could represent proper random polling. ... This posting is a careful initial report of our findings, not intended to be a full formal analysis but rather to alert people not to rely on R2K's results."
Nobody expects the Daily Kos to be accurate.
It would be like trusting Ann Coulter, Rush Limbaugh, Sean Hannity, or anything ever aired on "Air America" before it went bankrupt.
There's lies, damned lies, and then there's statistics.
The fraudulent process was discovered when 5 out of 5 dentists recommended the gum, for patients that chew gum.
Those who aren't used to phrases used with "political" centric organizations might mistake the title as saying someone who is on Daily Kos' payroll flubbed the numbers, rather than a company working on contract with them.
Living With a Nerd
there are Three sorts of Lies: Lies, damned Lies, Statistics. but then he did plead......Clemmency...
- Minutus cantorum, minutus balorum, minutus carborata descendum pantorum.
For me, the surprising part of this isn't so much that R2K made up poll results, but that the results actually were noticably less accurate than traditional polling, which I like to think of as representing a broad cross-section of people who still have landlines with no caller ID for some reason (or are desperate enough to talk to another human being that they'll answer their landline anyway).
Unlike the many Republican outfits which used partly- or wholly-fabricated polls by Strategic Vision, or the many media outlets which continue to use the horribly flawed Rasmussen polls to create eye-catching headlines, Kos immediately dumped the pollster, did an investigation, owned up to the errors publicly, and is now pursuing legal recourse.
This is exactly how you would expect an honest media organization (if one with a considerable political agenda) to behave. Too bad the mainstream media and those on the other side of the aisle don't seem to want to do the same.
Ha-ha
Everyone knows that 78.49% of statistics are made up.
You'd think they would spend Soros money more carefully.
Professional Politicians are not the solution, they ARE the problem.
I used to work a different major polling company, and I can assure you R2K is not alone in just making up numbers. Easily 80% of surveys that went through my region were completely falsified, and the remaining 20% rarely matched the demographic they were supposed to be answering for. Survey administrators have quotas, and then get paid extra for additional surveys past that, but there is basically nothing done to verify any of the surveys turned in, and everybody in the company knows it. Don't always trust what you read, especially not statistics.
For headline skimmers, this post would produced a completely inaccurate sense of what the article was all about... at length, the D.KOS are the ones who found out about this and are doing something something about it. That's good... but if you just read the headline, you'd come away thinking that D.KOS were the culprits instead.
Comment removed based on user account deletion
of course this will turn into a "bash the left" and a "bash the right" thread. when ideology isn't the point. polling is the idiocy in question
and the guy who manipulated the numbers is clearly an amateur. the way you do proper poll manipulation is LOAD THE QUESTION. you poll people with a question with the proper turn of phrase to lead them towards the answer you want. then, when you present the answers to the poll, you also cage the results in such a way to lead the audience in the way you want them to interpret the results
polling is fucking joke. all results from the left, or the right, is complete bullshit, and a waste of your time
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
If you mod me down, I shall become more powerful than you could possibly imagine.
Daily KOS employed bullshit artists for their polling data.
Commence turd polishing!
sometimes i'll look at my list of comments, and there will be an across the board "flamebait" or "offtopic" for 10 comments, across all commentary and different threads, all at the same time
it used to bother me, but it never really effects my karma, so now i take it as a form of demented compliment that someone should be so devoted to me (in hate, which is just the flip side of love. what we all really fear is being ignored)
the signal (the genuine mods) still drown out the noise (the troll modding), so slashdot still works for me. if the noise begins to drown out the signal though, indeed, slashdot is doomed
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
http://xkcd.com/756/
I think we should take a poll on how seriously we take poll numbers. I mean We already know how we vote in them
Nostalgia isn't what it used to be.
If you place a statistician's head in ice and his feet in boiling water, then on the average he is quite comfortable!
I've abandoned my search for truth; now I'm just looking for some useful delusions.
Because of course Fox News, World Net Daily, TownHall.com, and PowerLineBlog have never, ever, ever, passed on inaccurate numbers. This is the first time that the world has ever seen such a thing and clearly it is inherently a liberal problem, no doubt about that.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
false association is the best sort of poll manipulation
for example, if the poll was
"do you believe obama is the antichrist?"
of course this wouldn't poll very high. even those who oppose obama, the vast majority do so for genuine reasons of policy, not out of some demented partisan bloodsport (although unfortunately, the unthinking zombie partisan hordes seem to be on the rise in this country)
but if the poll was
"if obama were the antichrist, should he be impeached?"
and of course, this will poll high. when of course the question is completely loaded bullshit. if ANYONE were the antichrist, they should be voted out of office. but this is exactly this sort of false association that goes on all the time with polling questions. albeit a lot more subtle than this, and usually preying on people's preexisting prejudices and simplistic ideological understandings. it leads them to completely useless answers, but answers that can then be twisted when reported upon, with cable news HD graphics and bombastic audio cues and cinematic announcers with heavy gravitas and serious, low voices:
"in a shocking new poll, the idea of impeaching obama polls high among a wide sampling of americans"
polls are nothing but propaganda
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
if you just read the headline, you'd come away thinking that D.KOS were the culprits instead.
This is slashdot, after all. If they don't keep up their conservative credentials how will they continue to attract conservative advertising revenue?
Hell, it's already Tuesday afternoon in the US and this might be the first pro-conservative story they've run all week; in which case they have a lot of catching up to do in order to make their quota...
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
In Neal Stephenson's Cryptonomicon, there's a scene early in the book where the Allies are assembling the personnel for Station X (aka Bletchely Park). Statistician, turned Nazi codebreaker Lawrence Waterhouse, points out that his Nazi counterpart Rudy von Hacklheber, would notice something was amiss with the Allied personnel changes based the statistics of people being transfered to Bletchely Park, and then quickly deduce that the Allies are attempting to break the Enigma code. To camouflage the transfers, Waterhouse suggests creating ficticious personnel and have some of them transfered to Bletchely Park as well. However the military can't just make any random fake person, the fictious people must be statisitically drawn from a distribution that when added to distribution of real Bletchely Park personnel, the combined distribution is statistically insignificant (i.e. fail to reject the null hypothesis) than any other large military base.
If Research 2000 did what is suggested, they failed to taint the polls with the right kind of fake data, just like what the novel warned about.
Can anyone give a reasonable explanation for why this story is listed under "science.slashdot.org"? Clearly this is a political story and should have been placed as such.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
but one left leaning group acting does not support a trend on one side or another. While your political bias is obvious and you do believe what you do extrapolating it such as you do is not accurate.
What we see here in your response is others is acceptance of an issue because it supports the viewpoints of those responding favorably to it. Throw in the left/right condemnation of the mainstream media; itself a generic term that has different meaning to different viewers; and the whole comment comes off as a coffee sipping head nod.
There are very few polls conducted fairly, we have seen this in polls leading up to elections as well as exit polls. Having seen some exit pollsters at work I have seen one common theme, they ask people like themselves. As in, they look similar.
What this action by DailyKOS may do is spur groups on both sides to look at their data in new light. Not every group is blessed with good statistical oversight and many on both sides will discount examinations of their data because it the criticism is seen as leaning the wrong way. Right now with the level of political strife it is only natural for some groups to dig because of the level acrimony.
I know just as many on the right and left who latch on to polls and it never amazes me that they like pollsters who show results that agree with their political leaning and disparage polls that don't as slanted. Its human nature.
* Winners compare their achievements to their goals, losers compare theirs to that of others.
In Neal Stephenson's Cryptonomicon, there's a scene early in the book where the Allies are assembling the personnel for Station X (aka Bletchely Park). Statistician, turned Nazi codebreaker Lawrence Waterhouse, points out that his Nazi counterpart Rudy von Hacklheber, would notice something was amiss with the Allied personnel changes based the statistics of people being transfered to Bletchely Park, and then quickly deduce that the Allies are attempting to break the Enigma code. To camouflage the transfers, Waterhouse suggests creating ficticious personnel and have some of them transfered to Bletchely Park as well. However the military can't just make any random fake person, the fictious people must be statisitically drawn from a distribution that when added to distribution of real Bletchely Park personnel, the combined distribution is statistically insignificant (i.e. fail to reject the null hypothesis) than any other large military base.
If Research 2000 did what is suggested, they failed to taint the polls with the right kind of fake data, just like what the novel warned about.
Let the fools in Washington be suprised come election day, not trying to figure out how to leverage someone who is leading in the polls months before they get elected to an office.
I think polling itself is a travesty as some of the dumb idiot sheep following fools out there will tend to "vote for the winner" instead of voting for their ideals. Why not just stop all polling, the only polling we need is in primaries and in november.
I would really love to see this happen as it would cause people in the news to actually be suprised when the people they help pick by biased reporteing get their pants handed to them. Nothing is funnier than watching a media personality get a nice nasty surprise on election night...
Tsukasa: All I really want, is to be left alone...
Considering they call me up while I'm eating dinner, I find the average polling person to be mean.
Red Herring: "A fallacy of distraction, committed when a listener attempts to divert an arguer from his argument by introducing another topic."
Tu Quoque: "The assumption that because someone else has done a thing there is nothing wrong with doing it."
Ad Hominem: "Attacking not the idea, but the person putting forward the idea."
It's about polling and statistics, not politics.
Seriously, how do they even get polls to work now, given that people try to avoid them like the plague?
Information theory is life. The rest is just the KL divergence.
... but my fellow Tea Partiers complain pretty loudly, e.g., here:
here.
Paul B.
A website that has an openly biased point of view published data that backed up their point of view for a year and a half without looking at it critically? Why are we surprised at all? By that same token, why would anyone care what dailykos or any of those other rabble-rousing political bitchfest sites (from either side, a pox upon both their houses) have to say about anything?
Understanding is a three edged sword. Your side, their side, and the truth.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
Math outs a fraud. How is that not cool? It's like real life Mathnet!
Can anyone give a reasonable explanation for why this story is listed under "science.slashdot.org"?
Because the statistical analysis used to reveal the fraud is a good examples of how science works: take the raw data and squeeze it for internal and external inconsistencies.
Science is the policy that to be credible an idea must pass tests involving systematic observation and controlled experiment subject to as careful quantitative and qualitative analysis as the subject matter permits. This is a wonderful example of practical scientific technique applied to perfectly ordinary empirical data.
Labelling data "political" doesn't make it any less empirical. It just labels the labeller as someone who doesn't understand science.
Blasphemy is a human right. Blasphemophobia kills.
It's about polling and statistics, not politics.
Wrong. This is entirely political. Read the headline -
Daily Kos Pollster Made Up Numbers
Everyone knows the Daily Kos is a political news site with a liberal sway. If it was about the statistics then an appropriate headline would have been something like
Pollster made up numbers and sold them for money
Instead the headline as written carries a direct implication of Daily Kos intentionally handing out known bogus data.
This story is political, plain and simple.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Because the statistical analysis used to reveal the fraud is a good examples of how science works: take the raw data and squeeze it for internal and external inconsistencies.
While the analysis may be scientific, the story - especially the way it was posted here - is decidedly not. Even the headline itself wreaks of a political vendetta. While science should not carry a bias, it also should not be used to attempt to cloak one.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
n/t
Someone else found out the polls Markos has been putting on his site are wrong. He was forced to address this, and now he's trying to distance himself by suing his pollster. None of this would be happening if someone else hadn't exposed the bad polling he was putting on his site.
He doesn't deserve credit for that. It's a scandal, and now he's trying to save face. If you think he deserves credit, you're being biased. It's okay to be a Daily Kos reader yet also shake your head over this. It's more important to be objective than supportive.
I did actually single them out as completely making up numbers. They were so historically wrong, plus Daily KOS was also completely misinterpreting them. If you have three polls two of which routinely agree with each other with the standard deviation, and a third which consistently does not. Well, that third one is suspicious and not to be trusted, regardless of how much you like its results. Here's the thing Daily Kos doesn't freaking care, they have memories of goldfish. But to be fair, so does every other political blogger regardless of ideological bent. You always remember the times and ways in which you were correct, you never remember all the times you were wrong. Such a constant diet of politics is a death trap of the ego.
... can DailyKos, which promoted the polls to show how evil conservatives/Republicans are on a daily basis, be given a free pass and the right be ripped.
Did Markos Moulitsas ever release his book, American Taliban, in which he relies heavily on R2K to
"catalogue(s) the ways in which modern-day conservatives share the same agenda as radical Jihadists in the Islamic world. ...
found himself "making certain claims about Republicans that I didn't know if they could be backed up." Working backwards, Moulitsas set out to prove, via a Daily Kos/Research 2000 poll, that self-identified Republicans have much in common with the makeup-wearing, women-beating acolytes of Mullah Omar."
Don't worry, I'm sure the links in here that point to dailykos.com will be removed as quickly as those numerous General betray-us ones were.
Good Read
R2K, the research firm, has had it's lawyers send a cease-and-desist letter to the guys who found out about the less than credible polling results.
See fivethirtyeight.com for the actual letter.
Those who make up numbers.. ...and those who don’t get caught. ;)
Any sufficiently advanced intelligence is indistinguishable from stupidity.
Daily Kos frequently posts the results of several polling organizations, in addition to those they commissioned. Rasmussen almost always shows Republicans several points higher than the other polls.
Never let a lack of data get in the way of a good rant.
you'd see that a lot of mainstream media outlets use the same polling company...and none of them realized they were being defrauded. They're not some fly by night outfit, they had a good rep in the industry.
Never let a lack of data get in the way of a good rant.
The people who were investigating these anomalies were able to do so because The Daily Kos made sure that the crosstabs of all the poll data were made available when each poll was published.
So I *do* give them credit, because they made sure that the information behind the information being presented was available for people review in order to determine its accuracy. It sort of sounds like... I don't know... providing the source code along with the application and letting the users submit bug reports and patches.
Eviscerati.Org: All Hail the Eviscerati
http://www.dilbert.com/strips/comic/2008-12-10/
Is 1563649 a prime number?
Are you kidding?
I answer nearly ever pollster that calls me. Note that I tell telemarketers to fuck off, but pollsters can have 15-20 minutes of my day almost any time -- and if I'm busy, they'll call me back at whatever time I tell them I'll be free next.
I think it's important to have my opinion counted, especially when they're asking me what kind of food I want in my local supermarket. You know those nutritious, tasty, inexpensive frozen dinners we've been seeing more and more of? Yeah, you're welcome.
Do daemons dream of electric sleep()?
That $663.8B figure is the President's request for FY2010. Congress increased that amount to $680B, and that's now law. But the budget is still short for Iraq and Afghanistan, so there's another $33B supplemental pending for FY2010. (Although Representative Lowey trimmed that to about $30B, so let's go with that.) That brings the total to $710B. And that's just the Department of Defense. Defense-related spending outside the DoD budget is a minimum of $200B. Let's round that total down to a total of $900B. And there you have it: defense spending is about two thirds of discretionary federal spending. The previous poster is correct.
Given the fact that nearly every time I mention this problem I'm slapped with a troll mod, as well as some of the other issues mentioned here (most of which I've also seen) I suspect that somebody (or more than one somebody) with unlimited or nearly unlimited mod points has a stick up their sphincter about this issue. There's a theoretical foundation for understanding why somebody slaps Trolls down on stuff they don't agree with, but why on Earth would they slap them down so reliably on "fix the damn moderation system" posts? Only if they're an Slashdot Editor, I suspect.
If you mod me down, I shall become more powerful than you could possibly imagine.
Have you read the link that I have posted? Do you have anything better to say than calling people names?
Do not you realise that there is a rather large chunk of what one would call libertarians (or which I am one) willing to work with both Left and Right to solve the immediate problems that we are having right now, and try to resolve our differences later, maybe "agreeing to disagree", maybe just finding a better social order and using it to step up as species? ;)
I was offering the GP guy my hand, and now you seem to refuse to take it for pure ideological reasons, without stopping to think -- do you think it's right? What happened to 'question the authority?" thing, is it irrelevant now that Your Guy looks like he is holding the strings, and you are willing to support *whatever* he is up to?
Do you really support continuation of both wars (one somewhat declared, another one not), continuation of executive power special privileges, *new* PATRIOT act, absence of any move to de-criminalize recreational drugs, bailouts of huge corporations, making people pay through their nose for their health insurance, printing trillions of dollars of new money (if infusion of money into economy is *really* a solution, why not cancelling income tax for a year, instead of giving your buddies those trillions to "distribute", while taking a percentage off it? :) )?
Anyway... I think it was wrong of me to try to convert you (and yes, I did mean *you*, BMO!) Now play by play:
> A traitor scum liberal
What people used to be call "liberal" is now "libertarian", for the last century and half or so... What is not called "liberal" is more like a "socialist". And, if you have ever read my posting history, I come from *Soviet Russia*, really, so I know what happens under those rules, and I do not like it!
> douchebag going-to-hell-athist
Totally agnostic here, but when filling out forms and questionnaires (in Soviet Russia we used to call them "anckettes") offered by *your* government, I wrote in "Taoist" in religion section.
> communist
See above
> nazi socialist
Do you mean Nationalsozialismus, or the International Socialism? Do you think there is really that much difference between them?
> jew
Hmm, funny -- I thought it's the "left wing" that have problems with Jews (and take side of Palestinians), me, personally, have no opinion on that ancient struggle and would just leave people to resolve their difficulties themselves.... Agreed? ;)
> from Canada (or something)
Funny that I am typing this from Canada, but I am not here for political reasons, only job-related ones...
> as described by the incessant idiocy on the Right.
Do I still sound like an idiot? or do you want to find out how it is possible for me to have a consistent set of views without resorting to "Obama is always right!" thing? (Which would contradict the above-mentioned "question authority" thing, but it's apparently OK for you, not for me).
Willing to down-mode, or engage in real discussion?
Paul B.
A bunch of ideologues made some shit up? Wow. Color me astonished.
Did replied to BMO dude here: http://slashdot.org/comments.pl?sid=1702634&cid=32740516 -- as much as I hated it, maybe it still has a chance to change someone's mind, or at least crack it open just a little bit. However minuscule... :(
Good luck!
Paul B.
P.S. Since I like you, here is my e-mail: p-b;u=n*y'kATgmail.cammm :)
OK, can someone who knows something about statistics please explain this to me? After presenting the difference in margin in the "Attitude toward Obama among "Independents" and "Other"" table, the authors write
The actual average of the square of the weekly change in the difference between these reported margins was 1.475.
How do they arrive at this figure? The differences between the reported margins are, in order, -3, -3, -4, -2, +3, +3, +3, +3. Ergo, the weekly changes are 0, -1, +2, +5, 0, 0, 0. The squares of that are 0, 1, 4, 25, 0, 0, 0. And finally, the average of that is (1+4+25)/7 = ~4.285714...
How did they arrive at 1.475? Did I misunderstand something? I'm assuming that maybe they're using a different calculation, but I cannot figure out where I'm misinterpreting what they write. Worse, I cannot conceive of a calculation that would actually yield a value of 1.475 at all.
Help?
While I am deeply disappointed in Kos over this, to argue that both sides are just as bad in this regard is fatuous just on the face of it. Call me back when MSNBC argues in a court of law that it is their constitutional right to fire reporters for refusing to lie.
Most of the time that argument is used, it is to imply that ALL data that involves statistical analysis must be false (unless they happen to agree with whatever swill FOX News is pushing), and that we should only trust in anecdotal evidence provided it comes from a sufficiently extremist right wing blowhard.
Statistical analysis is the more reliable means of getting at the truth precisely because it is possible to uncover misleading and inappropriate things like what happened with this Kos pollster. That's why science relies on statistical analysis instead of "'cause Rush said so".
If someone provides you an answer that is the result of statistical analysis but refused to provide enough information for you to verify their conclusions (data, analysis methods used, etc.), then you can assume the conclusion is bad.
Thanks for providing evidence that there is intelligent life, on Earth. I appreciate it.
If you mod me down, I shall become more powerful than you could possibly imagine.