Math Indicates Pollster Is Forging Results

An anonymous reader writes "Nate Silver suggests the political pollster Strategic Vision is 'cooking the books. And whoever is doing so is doing a pretty sloppy job.' Silver crunched five years worth of their polling data, and found their reported results followed a suspicious pattern which traditionally suggests fraud. The five-year distribution of the numbers 'is not random. It's not close to random.' The polling firm had already been reprimanded by the American Association for Public Opinion Research for failing to disclose their methodology, though the firm argues they did comply with the organization's request. Their response to Silver's accusation? 'We have a call in to our attorney on this and fully intend to take action that will vindicate us.'"

Comments

major fcukup at slashdot

[postmortem]

a. you can't post
b. if you do manage to post, post goes to wrong topic!

Re:major fcukup at slashdot

[multisync]

Yeah, it's been like that off and on all day.

To those with mod points: use them on something worthwhile. Noting that your posts are turning up in the wrong topic is on topic. Modding postmortem's post Off Topic is a mis-use of your mod points.

Re:Why should I care?

[fuzzyfuzzyfungus]

From TFA, it looks like they handle a fair variety of sundry topics in American politics. Not a giant deal, I've certainly never heard of this particular outfit before; but I find it extraordinarily hard to believe that anything which increases the amount of false-but-plausible-looking noise in the world is a good thing.

On important topics such is more dangerous than on less important ones; but its mere existence makes the world a less knowable place either way. Either you have people believing false data, or you have people falling into the essentially nescient "all data are just source biases" position.

Re:Why should I care?

[Spy der Mann]

Pretend I know nothing about Pollster (which happens to be true). Why should I care whether they've faked results? By that, I mean: do they research options of favorite flavors of cotton candy, or public support for health care reform, or the best style of car, or...? In other words, do they do stuff that actually matters?

Faked polls = astroturfing.

Need I say more?

Horseshit

[Saint Stephen]

I call total, 100%, biased, fuck me up the ass horseshit on this inane accusation. Lies, damn lies, and statistics.

Ah ha!

[NoYob]

"[W]e categorically deny them and will refute them.

So, which category do they deny? The category of truth or the category of lying?

Re:Ah ha!

[etymxris]

Not sure if you're trying to make a pun, but "categorical" in this case means "without exception." For example, Kant talks about categorical and hypothetical imperatives. Categorical imperatives you do always without exception (such as never lying, according to Kant anyway). Hypothetical imperatives are what you do based on the situation (CPR is appropriate only when someone is not breathing, for example).

So what?

[msauve]

Polls show that 78.6% of all statistics are made up on the spot.

Re:So what?

[TheGreenNuke]

Source please? Also discuss the sample data and statistical analysis used to arrive at this conclusion. I'd also like to know the 95% confidence interval.

Re:Why should I care?

[interkin3tic]

First of all, I don't think "What do I care" is anything but flamebaiting. Who cares if you don't care?

Second, if they're the same "strategic vision" that the article is talking about, their webpage says
"Strategic Vision has worldwide experience developing tools to measure decision-making, human behavior, attitudes and perceptions. Its globally relevant, comprehensive theory of human behavior creates the most effective strategies addressing decision-making in product development and communications in the widest variety of fields, including automotive, customer service, government and politics, medicine and healthcare, organizational and jury, travel and leisure, food and beverages, and education." So they probably report on anything you will pay them to poll on, or rather, anything you will pay them to make a graph from nothing.

Their self-reported client list. Granted, they may have just made that list up as well.

Lastly, a quote in TFA by the company gives you plenty of reason to care:

[W]e categorically deny them and will refute them. We have a call into our attorney on this and fully intend to take action that will vindicate us...he has attempted to do severe damage to our reputation and what is he going to do when we disprove him just say I am sorry. That isn't enough at this point.

There you go: the company is mad about being uncovered and is doing the next step any stupid assholes do when their misdeeds come to light: sue in a vain attempt to keep the information from becoming well known. Therefore, -everyone- should know they're faking the results. I'm tempted to e-mail all their clients with a link to the article. If they go out of buisiness, maybe other shitty companies will finally realize you don't sue people who expose you as charlatans.

Bwhahahah, sometimes I say ridiculous things.

Re:Evolution in Action

[NoYob]

I've been experiencing weird things too with Slashdot and stories not loading and seeing things that don't make sense. I don't get it.

Anyway, back to the topic of Windows 98 being released today. I wonder if the Clinton Administration will continue with the anti trust investigations into M$.

Re:Why should I care?

[TeethWhitener]

In other words, do they do stuff that actually matters?

In a word, yes. Nate Silver manages the blog FiveThirtyEight and is well-known as a statistical analyst from the 2008 US election (among other things). Strategic Vision has released quite a few polls. In Silver's words,

...Strategic Vision's polls cover a wide array of topics: Presidential horse race numbers in any of a dozen or so states, senate and gubernatorial polling, primary polling, approval ratings of various kinds, polling on issues like the war in Iraq, and more abstract questions such as whether voters think that 'experience' or 'change' is the more important quality in a Presidential candidate.

So yes, this is pretty big news, should it turn out that Strategic Vision's behavior is in fact illicit. They're influential enough that news agencies may pick up their polling results. This is bad enough, but when you factor in the fact that polling results can be very effective propaganda in something like a presidential race, fraudulent polling can have significant consequences.

you can't believe anything anymore

[commodoresloat]

Agreed. Who is this Math guy anyway? Perhaps it's Math who faked the results, and Pollster is beyond reproach!

Re:Evolution in Action

[Anonymous Coward]

BREAKING NEWS:
The AP is reporting a major fuckup at Slashdot. The web site cannot even do the most basic task essential to its operation, allows readers to leave comments on articles. No comments were available from anyone employed by the web site. Phones rang and rang and rang. Several other Sourceforge properties had their numbers disconnected due to non-payment.

It is apparent no one in charge of the place gives even a sliver of a fuck, or even reads the front page after articles are posted, as it is 2009 and there are 50 fucking ways to notify the readership of the nature of the problem and the expected timeline for resolution. And that 50 is just from a fucking cell phone. If a person had an actual computer and an internet connection, even a netbook at a Starbucks, the number rises into the 1000s.

Long gone are the days when the popular geek web site devoted to technology actually worked. Long gone are the days when there were actual technical explanations of outages. Instead its more stories about politicians arguing over traffic ticket revenue posted as "Your Rights Online", iPhone slashvertisements, slashvertisements masquerading as book reviews, and links to people's blogs about blogs about news stories, and/or tweets about tweets about press conference summaries.

What's wrong with this data?

[klapaucjusz]

I'm not sure I understand what Silver is claiming about the data.

He shows that the distribution of second digits in the results of Pollster's polls doesn't follow a uniform distribution -- and from that he somehow deduces it's not random.

If you look at the figure in the second article, it looks to my untrained eyes like a gaussian curve with maximum around 8 -- since when are gaussians not random?

Re:What's wrong with this data?

[Johnny Loves Linux]

> since when are gaussians not random?

That's exactly the problem he's pointing out. The second digit should be a UNIFORM distribution if it came from real data. If the digits are gaussian that indicates that either

• there's some process accounting for a gaussian distribution that he doesn't know about (and he does consider that possibility) or
• the numbers are cooked by a human being who has a preference for 8's over other digits.

Re:What's wrong with this data?

[nixish]

When looking for fraud, Silver was not looking at the poll numbers but the raw data numbers themselves (essentially hundreds of thousands of numbers , if not millions). Out of all the raw numbers, when analyzed there should not be any distribution. But the numbers were slanted towards 6 & 8 suggesting (proving perhaps) tampering. There's plenty of sound theory in this. Just look it up.

Re:What's wrong with this data?

[wrook]

IANAS (I am not a statistician), but according to Wikipedia, Benford's law applies to the distribution of the first digit. It has a logarithmic distribution. This makes complete sense since the probability for certain numbers will be higher than others (i.e., in telephone bills, the 1 is probably much more likely since there are a lot of people with $100+ phone bills). But they are discussing the *2nd* digit. This should be uniform unless it's a very strange dataset.

improbable

[drDugan]

Reading TFA, Nate's analysis implies that there is a systematic bias toward some last digits in the overall poll percentages aggregated over many disparate topics.

What seems so improbable (to me) is that if someone really were grossly "cooking the books" like this - literally not doing the poll, or tallying any numbers at all, but instead simply reporting fake results for press ... is that they would be so stupid to make up the results manually instead of using a computer in some way. What, some guy in an office reading other polls and saying "gee I think the number will be 45%."

If this kind of bias really has been introduced by manually creating and publishing the results (as the analysis seems to imply), then it will be easy to track down and prove with further digging into the data, interviewing people who made the calls or took the data, etc. However, accepting such an explanation would requires a level of stupid on the part of the principals in this company that is so extreme that I find such a scenario an improbable explanation for the results presented.

Re:improbable

[cptdondo]

But here's the deal:

You do the poll. You have to; you can't just make up the numbers. Sooner or later someone would figure out you don't have a phone bank.

But the poll numbers come up as 46 for, 43 against, and the rest undecided.

Now you can't go and say, 98 for, 1 against, and 1 undecided; that's what the communists do and everyone knows they're lying.

But you report it as 47 for, 42 against, and the rest undecided. Now you've falsified your data, but you think in a way that's hard to catch. You bump the numbers one or two or three points in favor of your position.

However, I'm unconvinced that this is some sort of smoking gun; Silver needs to really run this sort of simplistic analysis on a lot of other polls and see if there in fact is a bias towards a 47 - 43 split with 10% undecided. That actually sounds about right for a lot of the polls I remember in the last election.

Re:improbable

[Silentknyght]

However, I'm unconvinced that this is some sort of smoking gun; Silver needs to really run this sort of simplistic analysis on a lot of other polls and see if there in fact is a bias towards a 47 - 43 split with 10% undecided. That actually sounds about right for a lot of the polls I remember in the last election.

If you read the TFA, Nate addresses this. He states that his data--SV LLC's polling results--are selected from a wide, wide, wide variety of topics, not just necessarily the highly divisive ones where there may be a relatively even split between two choices.

Moreover, (as Nate states) over enough data, even the effect of the undecided percentage on the trailing digit should be random.

Re:improbable

[internic]

I don't know, this sort of reminds me of a recent case of fraud in Physics. If a PhD physicist can make such a mistake, it doesn't seem totally unbelievable to me that a polling firm might. Also, you have to ask yourself if they ever actually expected their results to come under much scrutiny.

Handwaving math.

[Gorobei]

Nate Silver does great analysis at the first order multiple-linear-regression level -- he outperformed all the other polls/predictors in 2008 iirc.

He sucks at meta-analysis though, in that he just doesn't understand the math. His 2008 monte-carlo stuff gave good results, but was just a bad reinvention of averaging. His recent foray into analyzing stock returns was interesting but 0-information (i.e. useless.)

Now he's mentioning Benford's law, but playing with trailing digits. Then he handwaves a non-normal result with an appeal to "it looks wrong." Come on, give us some real math here!

That said, he's probably right, but he's given us no math to support his claim.

Re:Handwaving math.

[Artifakt]

Benford's law is sometimes called the First Digit law. It deals with cases where numbers are not equally probable, but rather lower integers are more common than higher ones. A good example of such a number is the first digit of street addresses. There are many short streets that only have a 100's block, and only a portion are long enough to also have a 200's block, fewer to have a 300's block, and so on, so the first digit is not equally likely to be, say, a 4 or a 7, rather there will be more fours than sevens. Some stock market numbers should fit Benford's law, and there are plenty of other cases with real world applications.

      However, the law in extended form does work for second or higher digits, or cases where the most likely value for a digit is not 1. Take the IRS for example. Last year, the standard deduction for married filing jointly was an even $10,000. Many people didn't bother to itemize schedule A unless it got them at least a couple of hundred extra back. So there were many people who claimed $10,2XX on their itemized returns, a few less that claimed in the $10,3XX and so on. $10,0XX or $10,1XX values probably weren't the most common, because a lot of people probably didn't bother to gather all the records needed and do all the paperwork if they though it was only going to get them, say, an extra $27 or even $104.

      The IRS could, and probably does use Benford's law to look for number patterns that may indicate fraud, but for some of those numbers, it's the second or latter digit that they should start at. (They won't publicly discuss whether they have any sorting/flagging software that is Benford's law based. I suspect they do as it would be foolish not to take advantage of the math here, but I have absolutely no proof other than that I use some of the same math in a private role, and it's been damned useful a couple of times in spotting a client trying to get me involved with something shady, so it should work equally well for the government.).

      So, using Benford's law for second or other trailing digits is legitimate. I can't tell from the article whether Nate Silver is doing everything else correctly, but the extension to a particular trailing digit isn't itself a flaw, and I could come up with a good psychological argument whey humans might fudge the second digit by a point or two, but only when it isn't already an 8 or 9, so as not to make the 10's digit roll, so focusing on digit 2 could certainly be justified. (as could focusing on the second digit to the right of a decimal point for precision results, by much the same logic).

Re:Why should I care?

[multisync]

its mere existence makes the world a less knowable place either way

Well said.

I find it disturbing, too, that the media just reports the polling companies' results, without reporting things like what questions were asked, in what order, how the poll was conducted or who commissioned it, all of which can have a big effect on the results. A lot of "push polling" goes on, especially when the polls are commissioned by special interest groups, business associations, unions or political parties themselves.

I'm not in the US, so I don't know this polling company, but I've had a municipal, provincial and federal election in the past 12 months (with another possible federal election imminent) and I think polling and radio call in shows have a great deal of effect on people's opinions these days, more so than traditional newspaper and television newscasts.

If Strategic Vision was conducting fraudulent poles, I would be looking at their client list and going after whoever paid for them as well.

Re:Too many 7s and 8s?

[HornWumpus]

Take any data set and you'll find patterns that are statistically impossible.

Not if you understand statistics.

Also note: If you understand statistics you would _never_ use the phrase 'statistically impossible'

Use stats, not laws

[unlametheweak]

Their response to Silver's accusation? 'We have a call in to our attorney on this and fully intend to take action that will vindicate us.'"

Generally, I would expect a logical course of action from an honest and transparent firm would be to hire a statistician to vindicate themselves. Lawyers don't make a reputable firm appear any less reputable.

Re:Use stats, not laws

[Frosty Piss]

Lawyers don't make a reputable firm appear any less reputable.

Lawyers don't make a reputable firm appear any more reputable.

Re:Too many 7s and 8s?

[blueskies]

If i take any data set (say one with a standard distribution), how many of those data sets would i have to sample on average before i found one that looked like the ones he is talking about? If the expected number of data sets i would have to look at is in the millions, you are correct in that i might find it in my first sample, but the chances are incredibly tiny.

Re:Too many 7s and 8s?

[evanbd]

Fortunately, there are corrections you can do for that. And he took a fairly normal statistical test on the numbers, which is equivalent to saying he didn't perform that many comparisons. To very rough approximation, you need to correct your p-value for all the less weird analyses you might have performed on the data instead. It's a bit hard to pin down an exact p-value for the analysis he did (the underlying data isn't expected to be flat; it's also not expected to be that bizarrely lumpy), but I promise that Nate Silver has an understanding of this issue (which you'd see, if you'd read the post).

Re:Why should I care?

[bfields]

if they're the same "strategic vision" that the article is talking about, their webpage says "Strategic Vision has worldwide experience developing tools to measure decision-making, human behavior, attitudes and perceptions....

Nope, you're looking at the webpage of a different company! See Nate's previous article:

Why would you pick the name "Strategic Vision, LLC" for your company when the name "Strategic Vision, Inc." was already in use by an extremely well regarded, San Diego-based research firm that has been in business for more than 30 years? Are you deliberately trying to confuse your potential clients and leverage Strategic Vision, Inc.'s much stronger brand name?

Re:Why should I care?

[plague911]

Strategic Vision is a Republican pollster. Meaning when a Republican politician wants a poll about a particular set of data they give Strategic Vision some money and they do a poll. This can be for either internal polling to give them and idea how the "battle" is going or for general consumption. And yes Strategic Vision is big enough to matter, but they are just the tip of the iceberg how misleading "R" pollsters

In general there are some Republican some Independent and some Democrat pollsters however all of their results are supposed to be scientific the idea is dose a poll for internal consumption really help if tells you that you are going to win easily on election day only to have to be a landslide against you?The answer is no.

The reason why this is dangerous is multi fold. 1) Due to the supposed scientific nature it has been used to make public policy decisions 2) It can influence peoples opinions. 3) It can influence a senator's or some other politicians choices while they are in power.

Here is a perfect example of this. A certain Republican senator from Maine is considering if she should support a public option, so she wants to see what the citizens of her state think about the topic. She hires Strategic Vision to do a poll for her. Strategic Vision comes back and says 60% of your state's citizens are against it. She gose "Wow I guess im not supporting that bill" In reality its 60% the other way. From this the senator decides to not support the bill and it dose not pass.

I will be as blunt as possible. I am accusing Rasmussen, Strategic Vision and other Republican pollsters of deliberately lying to the American people in order to alter the public debate. If you follow the math they have been consistently off for years. If you want to just look at the last election cycle Rasmussen etc all had the results a lot tighter than the results on election day. This could just be poor polling on their part but I will offer exhibit B

Since health care reform has been a topic in the news the difference between the several Republican pollsters and "everyone else" has been steadily growing. I firmly believe that the insurance industry has been paying these pollsters to lower their numbers for the democrats to push them to drop health care reform.

Yes the Democrats poll numbers have been sliding somewhat across the board. However if you look at the data from the Republican sources. They have the numbers significantly different than those of the "Independent and Democratic" pollsters.

Over all I want to say this "dishonest polling" helps no one. It may help push a certain agenda temporarily but It can also cause those who support it to loose elections..... Look at the results from 2008 the REPUBLICAN PARTY IS BEING MISLEAD BY ITS OWN POLLSTERS AND IT IS COSTING THEM ELECTIONS

Re:Why should I care?

[maxume]

NBC always reports on the NBC/Wall Street Journal poll. I think they commission it. They seem to do a decent job of describing how they do it:

http://online.wsj.com/article/SB124527518023424769.html

(that link works when clicked on from a Google search, but given that the WSJ has a mighty paywall, I don't know if it will work otherwise)

So maybe you need to talk about a more nuanced group than 'the media' (I wouldn't be particularly shocked if other major outfits were at least approximately as responsible).

Re:Why should I care?

[quantaman]

Second, if they're the same "strategic vision" that the article is talking about

They're not, from another helpful article from FiveThirtyEight

Why would you pick the name "Strategic Vision, LLC" for your company when the name "Strategic Vision, Inc." was already in use by an extremely well regarded, San Diego-based research firm that has been in business for more than 30 years? Are you deliberately trying to confuse your potential clients and leverage Strategic Vision, Inc.'s much stronger brand name?

You're looking at the page from the well regarded Strategic Vision, Inc. Funny that SV LLC seems to be so happy to sue Nate Silver, it would seem that SV Inc has a far stronger case against SV LLC.

Could be an interesting intersection of Trademark/Slander laws...

Re:Why should I care?

[MobileTatsu-NJG]

Pretend I know nothing about Pollster (which happens to be true). Why should I care whether they've faked results? By that, I mean: do they research options of favorite flavors of cotton candy, or public support for health care reform, or the best style of car, or...? In other words, do they do stuff that actually matters?

Faked polls = astroturfing.

Need I say more?

Well, you might need to explain what astroturfing is. Most people here think that astroturfing is when you are satisfied with a mass-market product.

Re:Not statistically significant

[plague911]

When you are making decisions based on public opinion and the differance between 52% and 48% makes the difference between whether you keep your elected position. Imagine what the difference would be between 60% and 40%. I'm not sure of the exact reasoning for these kinds of polls 20% seems to be about the stranded margin of error. I imagine it has some aspect of what the state of the art is in scientific statistic estimation theory is. In which case 57% to 20% difference would like using 1850's technology compared to the technology we will have in 2010. At the very least Strategic Vision is run by idiots. If not they are intentionally misleading the public.

Re:Why should I care?

[(startx)]

Except you've linked to the wrong company. Strategic Vision, Inc. is a well respected 30-year old polling firm in California. Strategic Vision, LLC is the shady 5-year old GOP shill corp with questionable poll results and no real office (or polling results allegedly). Careful with those links, you don't want to slander the wrong company here. I think SV Inc. may have a trademark case on their hands if their feeling litigeous.

Re:Why should I care?

[Brian Gordon]

If the vote is to reflect public opinion, people should vote their own opinion. They don't need to try to help the system by guessing the most popular option.

Re:Why should I care?

[schon]

I don't think "What do I care" is anything but flamebaiting. Who cares if you don't care?

According to a poll I just saw by Strategic Visions LLC, 68% of Americans care!

Re:Why should I care?

[Attack DAWWG]

They are a partisan, Republican-oriented polling company. They have gotten into trouble in the recent past for their questionable results.

Re:Not statistically significant

[ceoyoyo]

First, the example he gives where he looks at polls from ALL sources is an example of a plausible distribution of real results because, assuming the majority of pollsters are not cooking their data, the data should be dominated by randomness. He then looks at this particular pollster and finds a much greater disparity in trailing digit frequency. The question is, is it significant, or just chance?

Given the numbers, it's not particularly hard to figure out. You can calculate the likelihood of any particular result given a theoretical distribution using a G test of goodness of fit. Technically for numbers this small you could use an exact test but I don't know of a web version and I'm too lazy to write one up. But here's a description of, and an excel spreadsheet that performs, the G test of goodness of fit: http://udel.edu/~mcdonald/statgtestgof.html

Basically, you plug in the distribution you see and compare it with the one you expected. What you get is the probability of that distribution occurring by chance. So if we plug in the observed data for all the pollsters and assume equal likelihood for all trailing digits we get a p=0.006. Whoops, looks like our assumption isn't quite correct. As the blog author notes, the observed distribution is humped a little, favouring the middle numbers. He also gives a possible explanation. For giggles, the probability of the Strategic Vision results given equally probable trailing digits is absolutely microscopic: p=1.44x10^-17. Together those tell us that our assumption of equal digit distribution is probably not quite right, but the Strategic Vision data still looks mighty funny.

Okay, so assume instead that most pollsters aren't making up their numbers. Not that their numbers are necessarily accurate, but that they're at least not making them up off the top of their heads. So using the data from all pollsters as a template, how likely is the Strategic Vision distribution? That's a G test of independence: http://udel.edu/~mcdonald/statgtestind.html. We could use Fisher's exact test, but I can't find one that will do a 2x10 table.

Plugging in the data, we get G=43.068, d.f.=9, which gives p=2.09x10^-6. The blog author was actually a little careless when he said the chances of Strategic Vision's results are millions to one against. If you insist on the equal-probability theory then the odds are 70 quadrillion to one against Strategic Vision and 166 to one against the industry as a whole. Taking the more realistic approach that the industry average is a better representation of the actual probability, the odds against Strategic Vision's results are about half a million to one against. Not millions to one, but close enough.

Re:Evolution in Action

[Arthur Grumbine]

I know this might be slightly off-topic, but I think that the issues Slashdot has been having are due to an unexpected spike in traffic after they posted the story of how 3D Realms was switching over to Epic's Unreal Engine for the upcoming Duke Nukem Forever. I'm pretty stoked about this and am saving up to be able to afford a Voodoo2 - DNF is gonna be da bomb!!

Re:Why should I care?

[Anonymous Coward]

you are so naive

our democracies aren't founded on who's the best candidate, it is on the most popular.

voting isn't about getting the right person for the job, it is all so often trying to make sure the wrong person doesn't get the job... whomever that may be.

if you intend to vote against someone, it is often best to vote for someone that is otherwise popular... that's strategic voting.

Not voting is also strategic, in the sense that your vote won't help anyone but the most popular. It's good to know where your non-vote is going.

Re:Why should I care?

[dragonturtle69]

Pretty good on the explanation of the "who" was polled, but not the questions.

Just a silly example: "Are you in favor of decreasing the speed limit on Main Street to 5 MPH?" vs. "Are you in favor of saving cats and squirrels on Main Street?". I know silly example, but it is non-political and illustrates the point the the wording of the question, as well as the sequence of each question, contributes to determining the results of the poll. Even just the tone of voice can push someone in a direction. Think of a good salesperson.

I've not found a link, but I do recall this some years ago when Zogby started up, and was much more accurate than the other pollsters. They explained that their success was due to how openly they asked their questions, trying to word and order them so as to not provoke or create emotions or guide someone to an answer.

So, any poll without the questions and their order is of little value to me, other than infotainment.

Bothered Slightly

[mathimus1863]

I've been following Nate ever since the 2008 elections, and I've much enjoyed his analysis. Being a mathematician, I can spot BS math, but Nate usually does a decent job with no BS. But this article is has so many analytical gaps that I feel awkward supporting him this time, even though the article as a whole is convincing. To make such a bold claim as he is, I would've expected him to assess this more completely. He did no comparisons to other pollsters, and sampled data that is not IID (identically and independently distributed). i.e. if a boolean poll has 49% for one side (9) the other answer has to be 51% (1) The last digits (1 and 9) are completely dependent. Not all polls are boolean, but there will still be correlations, and many polls in the sample are boolean. Not only that, but he mis-applied the reference to Benford's Law. I know he knows what Benford's law is, because he's had multiple other posts about it, but got it dead wrong in this article.

I'm glad there is someone sufficiently mathematical to look for things like this and have a wide enough audience to be heard, but I wish he'd taken some more time to do look at more control groups and do some confidence intervals before sticking his head into a potential legal mess.

Re:Bothered Slightly

[ytm]

i.e. if a boolean poll has 49% for one side (9) the other answer has to be 51% (1) The last digits (1 and 9) are completely dependent.

He could just as well pick the lowest number of the two and check distribution of 0-5 digits. I have a bigger problem with his analysis, from TFA:

I did not include "non-response responses" like "other" or "undecided", nor did I include a tally for third-party candidates in races beteween the two major parties.

Given the dependence between the possible outcomes of the poll, I'm more curious about results with this data included.

Re:Why should I care?

[interkin3tic]

I hereby take back everything I said about Strategic Vision and reapply it to Strategic Vision, LLC, times two.

Re:Too many 7s and 8s?

[bidule]

Also note: If you understand statistics you would _never_ use the phrase 'statistically impossible'

If you understood thermodynamics, you'd know that 'statistically impossible' is why the world doesn't go crazy. Like sudden appearance of vacuum when you try to breathe or random melting of spoon when stirring your coffee.

Re:Why should I care?

[Discordantus]

It shouldn't (but probably will) be considered trolling to point out that the political section of their client list consists of the Republican Party, the Conservative Party (of England), The Department of Defense, the Whitehouse, and the State of California. That section hasn't changed in that last year, so I assume it's referring to not only the Republican governor of California, but also Dubbya's Whitehouse. Sounds like they get most, if not all, of their political business from conservative sources.

Re:Why should I care?

[zippthorne]

I firmly believe that the insurance industry has been paying these pollsters to lower their numbers for the democrats to push them to drop health care reform.

Yeah, you go ahead and cling to the belief that the insurance industry doesn't want the health care bill to go through. Why would they possibly look at 30 Million people who aren't buying their product and support a bill that will require everyone, by force of law, to buy their product?

I'd certainly like to see some numbers regarding who the insurance industry as a whole is contributing to.

Re:Why should I care?

[Just Some Guy]

First of all, I don't think "What do I care" is anything but flamebaiting. Who cares if you don't care?

I didn't say that I didn't (or wouldn't) care, but was asking why I should care. I thought I was fairly clearly about that. The story basically boiled down to "some group you've never heard of is falsifying data that you may or may not be interested in, but I didn't want to bother to explain any of this and would rather make every single reader figure it all out for themselves".

There you go: the company is mad about being uncovered and is doing the next step any stupid assholes do when their misdeeds come to light: sue in a vain attempt to keep the information from becoming well known. Therefore, -everyone- should know they're faking the results. I'm tempted to e-mail all their clients with a link to the article. If they go out of buisiness, maybe other shitty companies will finally realize you don't sue people who expose you as charlatans.

First, I don't have a dog is this hunt. I don't know who the accuser or target of accusation is, and certainly don't have opinions about either of them.

Playing devil's advocate, what if the accuser really was slandering the target? It's evident that you believe the accusation and want to get vigilante justice against them. In that case, what should they do? Keep quiet and leave the slander unanswered, or take out full-page ads to claim their innocence, or what?

But again, the story didn't get into any of that. It just said that a few people who aren't well-known here on Slashdot are throwing accusations and threats back and forth. If it were Linus accusing SCO of fibbing, then OK, I have the background and context to evaluate that information. I'd suspect the same hypothetical story in "CEO Magazine" would at least tell who the main actors are.

Re:Too many 7s and 8s?

[Artifakt]

Statistically Impossible may well have meaning. In Cosmology, various people at various times (Hawking, Guth, Dirac, and Einstein (1n the late 40's working with Minkowski and Godel), all found that they had to write a few pages on whether very improbable events were distinguishable from zero probability events before they could justify using some of their math. All were working on their own takes on the origin of the Cosmos problem at the time. Most of them decided that any event with a probability of less than 1 in the whole lifetime of the Cosmos was 'statistically impossible' and not just 'improbable'. Rosen later argued that it was better to phrase it in terms of less than 1 during that part of the cosmos's lifetime when entropy was low enough to allow other events of that same energetic magnitude to happen normally rather than the whole lifetime, and others have debated the point various ways, but it's still common to call some things statistically impossible when doing fundamental cosmology.
      Oh, and I need a new spoon.

Re:Not statistically significant

[Anonymous Coward]

I've just checked, and it's pretty easy to generate last digit distributions that look a great deal like the one shown for strategic vision. If you assume they poll over contentious issues (which are divided close to 50/50 in the population opinion) and that there are a small number of nonrespondents, then you get distributions that with lots of 49s and 48s ,and fewer 41s. My sample histograms even reproduce the spike at 0, and the peak at 7 or 8. This is 10 lines of code in python:

from pylab import *
mnvar = 2 # deviation from 50/50 for each question
nonresp = 3 # mean nonrespondents on each side
ssize = 10000 # number of questions
a1 = floor(normal(50, mnvar, [ssize/2])) # first group answers
a2 = 100-a1 # the second group, their opponents
a1 -= poisson(nonresp, [ssize/2]) # nonrespondents in the first group
a2 -= poisson(nonresp, [ssize/2]) # nonrespondents in the second group
a = concatenate([a1,a2]) # put them all together
hist(mod(a,10))

Obviously, I didn't choose any numbers by hand. It seems at least reasonable that pollsters might focus on questions that are close to evenly divided in the population. So, while there's no excuse for not publishing your methods, there is at least one innocent, and quite plausable, explanation for this distribution.

Re:Why should I care?

[plague911]

"Yeah, you go ahead and cling to the belief that the insurance industry doesn't want the health care bill to go through"

You are right the insurance industry would stand to gain massively by that proposal. That's exactly why the liberal sect of the democratic party has been fighting that provision.

I would like to point out that the insurance industry is being very pragmatic they have a two tier battle plan. They don't want the bill to pass however if it dose pass they want to have things like that put in

That provision was added to some of the bills to "tempt" republicans into voting for it as several Republicans have explicitly said they would like to see that included.

As far as "I'd certainly like to see some numbers regarding who the insurance industry as a whole is contributing to." The money has been flowing quite rapidly into the conservative arm of the democratic party. Ben Nelson, Mary Landrieu and Max Baucus have all goten heavy donations since this whole thing has started (from insurance companies). That is not to say that the republicans have not been getting a lot of money from the insurance companies. (That goes without saying) So to some it up Republicans are continuing to get good pay checks,(the usual) however some conservative democrats are now also getting paid for their services(Newish). Just for your info many progressives want political blood for this, Ben Nelson and Max Baucus and to a much lesser extent Mary Landrieu are the one thing that is standing in the way of progressives' holy grail. For that many of us want political revenge at any cost.

rural places need guns to protect from criminals

[circletimessquare]

the police are too far away. so we have a status quo here currently in the usa where hundreds of urban dwellers die every year from thugs with guns for the sake of a law which serves only the rural minority. but as the usa continues to urbanize further, and begins to equal european urban/rural ratios, political status quo will fall in line inevitably

and instead of HUNDREDS of urban dwellers dying every year for the sake of rural-friendly laws as we currently have, DOZENS of rural folks will die instead for the sake of urban friendly laws

inevitable. deal with it

"I am not FRINGE because I don't vote."

that's true. your SELF-DISENFRANCHIZED because you don't vote. your vote is your voice in your society. if you seek to not vote, you have willfully removed your own voice, you have chosen to be irrelvant. so why are you still fucking talking? you seek to not be a member of society. which is fine, drop out if you like: in which case, shut up and stop commenting on a society you freely choose not to belong to. if you want your opinion to be considered by us in this society, try to be a part of it by voting, and make your voice heard

but you don't get to drop out of society by your own choice and still think anything you say is relevant

if you want to be relevant, vote, and consider yourself to be a member of the same society as me. or don't, and, in logical coherence with that choice of yours, shut the fuck up

otherwise, there is absolutely zero for me to respect about anything you say, because by your own admission, you choose to not matter to me by not voting

oh you have your gun. awesome: why solve problems with voting when you can shoot, is that your point of view? fucking shizophrenic loser

Re:Why should I care?

[Orion Blastar]

Actual fraud in any form or sense should not be tolerated.

Many people made decisions based on those polls, including politicians. If the results are not random samples but where cherry picked, it could influence those politicians to support bills and policies that they think the public wants (Patriot Act, Warrentless Wiretapping, Waterboarding, Wars, etc) but in reality they might not actually want as a majority.

This applies for anything using statistics including scientific theories, the same fraud detecting method can be used on scientific theories to weed out the problems and fraud in science.

Executive Summary:

[Farmer Tim]

Web 2.0

Re:Why should I care?

[khchung]

I find it disturbing, too, that the media just reports the polling companies' results, without reporting things like what questions were asked, in what order, how the poll was conducted or who commissioned it, all of which can have a big effect on the results. A lot of "push polling" goes on, especially when the polls are commissioned by special interest groups, business associations, unions or political parties themselves.

tl,dr. (Too long, didn't read).

Unfortunately, for most of the world, this will be the response from most readers if the media took the time to report on the details of the poll.

Although, really, in the internet age, the media could have added a link so anyone interested could see the details of the poll. However, I suspect doing so would just expose to world how ignorant/lazy the reporters are, because you may find most poll results are either horribly slanted or extremely poorly designed (to the point that the poll was designed to mislead will be obvious).

For example, I recall seeing a newpaper headline saying ">80% of women has been sexually assaulted at least once". Surprised at this, I RTFA, and it turned out the "poll" was done by an NGO aimed at helping rape victims, and they "polled" 8 (eight) of their staff to get this result. My view of that newspaper (and reporters/editors in general) dropped a few notch after that.

Nothing new

[hawk]

I did a statistical analysis off the year 2000 "recount" almost 9 years ago, looking at the counties with "unusual" results.

There were six counties in which the changed votes didn't fit the normal bell curve, four benefiting Gore and two Bush.

Both of Bush's and one of Gore's had rules in which replacement ballots were made for idiot voters who used an X rather than filling the bubble, explaining them.

One of Gore's had machine problems in the recount and stuck with the original figures.

And then there were the two counties, which accounted for the lion's share of the "correction" from the recount.

One of them was 50 standard deviations out--so far out that it is less likely than winning the California Lottery every week for thirteen weeks running . . .

I wasn't the only one to notice the oddity, but the sad fact is that noone cares . . .

hawk

Re:Strategic Visions Inc. != Strategic Visions, LL

[mwvdlee]

I love how the ".biz" TLD is effectively the "evil bit".

Re:rural places need guns to protect from criminal

[Jarjarthejedi]

You really think that the only people who want guns legal are rural? And that the laws are "rural friendly" in that regards? I've got news for you, the vast majority of gun owners and enthusiasts are urban dwellers, and that isn't looking like it's going to change anymore now than it has in the past couple of decades.

In addition, you really think that the majority of murders with weapons wouldn't happen without weapons? People murdered each other before guns were invented, removing them might make a few cases go away but won't impact the vast majority of homicides.

Good luck voting to stop that bear from getting you by the way, I'm sure he'll listen to your excellently thought out democratic system of determining who he should eat next.

-Someone who owns no guns but isn't dumb enough to think guns are the root of problems humanity has had to deal with for centuries before the discovery of gunpowder.

Re:Why should I care?

[petermgreen]

Well, you might need to explain what astroturfing is
Astroturfing is where a special interest tries to create the impression of grassroots support. That may be through paying shills to post a lot on message boards with posts that support your position, it may be through dodgy polls or it may be through other means.

Re:Why should I care?

[selven]

Strategic voting is the worst thing that you can do to a democracy. It makes every political system fall into a two-party system, which (see: United States) becomes a de facto one-party system.

When will people learn?

[Migala77]

You don't need fraud to lie using statistics!

Re:Not statistically significant

[Rockoon]

First, the example he gives where he looks at polls from ALL sources is an example of a plausible distribution of real results because, assuming the majority of pollsters are not cooking their data, the data should be dominated by randomness.

Here is the thing. Did he begin with the theory that Strategic Vision was fraudulent, or did he begin with the theory that some pollsters were fraudulent?

After all, he was churning a lot of pollsters data.

Isnt it quite possible that he was simply mining his massive dataset for something, anything, that made any pollster look bad?

In short, how likely is it for one legitimate pollster out of many legitimate pollsters to have data that isn't quite normal (pun intended?)

Re:Not statistically significant

[ceoyoyo]

From his posting, he talked to SV about their refusal to reveal their methodology, then decided to test to see whether their results showed any suspicious bias. He was specifically testing SV and not searching for any pollster.

You're right, if he tested multiple pollsters then he'd have to correct for multiple comparisons. Even so, you'd expect results as bad as SV's about one time in half a million. There aren't that many major pollsters, so you could detect results skewed as badly as SV's to a high confidence level using a data mining technique.

Re:Why should I care?

[jltnol]

Because bad or false or misleading data is sometimes what people use to make a decision on. Kind of like the piling on theory.. I was going to vote for Mr. A, but since the polls show Mr. B is winning, I want to vote for the winner! Weird, sad, tragic, and very underhanded. But don't put it past folks to publish outright lies in an effort to sway the public.

Re:Not statistically significant

[Rockoon]

From his posting, he talked to SV about their refusal to reveal their methodology, then decided to test to see whether their results showed any suspicious bias. He was specifically testing SV and not searching for any pollster.

I suspect that refusal to reveal methodology is quite common, given that most are agenda-driven. Did he only speak to SV, or did he speak to lots of pollsters who refuse to reveal methodology?

Even so, you'd expect results as bad as SV's about one time in half a million. There aren't that many major pollsters, so you could detect results skewed as badly as SV's to a high confidence level using a data mining technique.

But there are LOTS of ways (infinite, really) to "test" data, so even if there are only 50 pollsters, you can still end up with millions of chances of finding arbitrary million-to-one outliers (where a lack of outliers would actually be suspicious!)

Is this second-digit test a common test for normal distribution, or is it an unusual method?

Re:Not statistically significant

[ceoyoyo]

Actually, I think most of the large polling organizations are pretty good about releasing their methodology. From the sound of it, this one is kind of an exception, and has taken a lot of flak for it.

You're certainly correct, if you go around comparing things long enough you're likely to get a false positive, unless you correct for multiple comparisons.

I've never actually seen any second-digit analysis before, but election and poll fraud isn't my field. I expect a lot of election monitoring would use similar techniques. A fair bit of work has been done looking at distributions of digits, including Benford's law and I believe work that shows that the fourth and on digits are actually uniformly distributed. There is also some psychological research looking at patterns in numbers that people tend to select. I seem to recall that if you ask a large group of people to pick a number between one and a hundred, a disproportionate number will pick either 32 or 36. It's a trick used by psychics - ask a large audience to pick a number between one and one hundred. Then ask whoever picked 32 to put up their hands. An impressive number of hands go up. Next pretend to be a little uncertain and say, wait, I'm also getting a strong signal on 36... and suddenly a bunch more hands go up. In a big audience it suddenly looks like most of the hands are up and now you can take their money.

In the real world...

[DragonWriter]

If the vote is to reflect public opinion, people should vote their own opinion. They don't need to try to help the system by guessing the most popular option.

Sure, in an unattainably perfect world with perfect election systems, this would be true. However, one most note that its impossible to have a single-winner voting system where more than two candidates stand for election where strategic voting is not rewarded if voting actually matters at all.

In the real world, strategic voting which takes into account the preferences and likely behavior of other voters, assuming it is based on accurate information, produces better results than blindly voting your own true preferences.

Even ignoring the incentives for strategic voting, though, there is a cost benefit analysis in pre-voting activities which effect the success of candidates and ballot propositions -- even if a person believes something is a good idea and plans to vote for it, they are far less likely to expend resources (whether by donations of money or of time and effort) if they feel that those resources are unlikely to make a difference in the outcome.

So, ultimately, there are good reasons why people's understanding of the popularity of a political idea or candidate affects their behavior regarding that idea or candidate.

Re:Why should I care?

[plague911]

A) I agree with much of what you said. I also did not accuse republican politicians of anything. I accused R pollsters and the insurance industry of working together to mislead everyone...including republican politicians.

B)I agree with you it is much better to look at trends. So my point was that although the trend has been downwards for the democrats. The slope for Rasmussen and Strategic vision and has been very different than the slope of the over all trend.

C) If you already agree that Strategic Vision tends to be overly biased is it that big if a leap to think a few else have been. I am honestly mostly concerned with Rasmussen who is one of the big names. He previously have been "close" to the norm but more recently he has gone on TV and publicly endorsed republican ideas and since than his polls have gone on to be more and more extreme.

"And I hate to tell you this ... but on the health care issue (since you brought it up), Strategic Vision is not the ONLY one showing that it's losing traction" Please don't think you corrected me. I already stated that in my first post my point was their results have been much different than the downward trend of other pollsters.

To your last comment about crooks. I cant disagree more. Yes their are crooks in both parties. Ill point out how i think very very lowly of Max Bacus and Ben Nelson right about now. But for the most part I think they are good hearted power hungry people. The reason to me why "R" get labled as corrupt more often is due to the fact they general have some of the richest industrial friends.

Re:rural places need guns to protect from criminal

[Brian Gordon]

I don't really have an opinion on gun control but I think this is wrong:

People murdered each other before guns were invented, removing them might make a few cases go away but won't impact the vast majority of homicides.

Premeditated murders maybe, but crime in general is greatly assisted by the availability of guns. The problem is that they're just so powerful. If you go into a bank with a knife and start waving it around and telling people to get on the ground they're just going to run away. But pull out a gun and everyone 10 meters around is going to obey every word because you can kill them instantly.

And people are defenseless against a gun but they can at least run or throw a chair or punch an attacker with a knife. And gun killings are easy and impersonal while with a knife the attacker has to struggle and get covered in blood and listen to screams or whatever.. much nastier

Swords are a problem I guess but they're impossible to carry concealed