Slashdot Mirror
Math Indicates Pollster Is Forging Results
An anonymous reader writes "Nate Silver suggests the political pollster Strategic Vision is 'cooking the books. And whoever is doing so is doing a pretty sloppy job.' Silver crunched five years worth of their polling data, and found their reported results followed a suspicious pattern which traditionally suggests fraud. The five-year distribution of the numbers 'is not random. It's not close to random.' The polling firm had already been reprimanded by the American Association for Public Opinion Research for failing to disclose their methodology, though the firm argues they did comply with the organization's request. Their response to Silver's accusation? 'We have a call in to our attorney on this and fully intend to take action that will vindicate us.'"
a. you can't post
b. if you do manage to post, post goes to wrong topic!
From TFA, it looks like they handle a fair variety of sundry topics in American politics. Not a giant deal, I've certainly never heard of this particular outfit before; but I find it extraordinarily hard to believe that anything which increases the amount of false-but-plausible-looking noise in the world is a good thing.
On important topics such is more dangerous than on less important ones; but its mere existence makes the world a less knowable place either way. Either you have people believing false data, or you have people falling into the essentially nescient "all data are just source biases" position.
Polls show that 78.6% of all statistics are made up on the spot.
"National Security is the chief cause of national insecurity." - Celine's First Law
First of all, I don't think "What do I care" is anything but flamebaiting. Who cares if you don't care?
Second, if they're the same "strategic vision" that the article is talking about, their webpage says
"Strategic Vision has worldwide experience developing tools to measure decision-making, human behavior, attitudes and perceptions. Its globally relevant, comprehensive theory of human behavior creates the most effective strategies addressing decision-making in product development and communications in the widest variety of fields, including automotive, customer service, government and politics, medicine and healthcare, organizational and jury, travel and leisure, food and beverages, and education." So they probably report on anything you will pay them to poll on, or rather, anything you will pay them to make a graph from nothing.
Their self-reported client list. Granted, they may have just made that list up as well.
Lastly, a quote in TFA by the company gives you plenty of reason to care:
There you go: the company is mad about being uncovered and is doing the next step any stupid assholes do when their misdeeds come to light: sue in a vain attempt to keep the information from becoming well known. Therefore, -everyone- should know they're faking the results. I'm tempted to e-mail all their clients with a link to the article. If they go out of buisiness, maybe other shitty companies will finally realize you don't sue people who expose you as charlatans.
Bwhahahah, sometimes I say ridiculous things.
Anyway, back to the topic of Windows 98 being released today. I wonder if the Clinton Administration will continue with the anti trust investigations into M$.
It's NOT me! It's the meds! I'm on 1000mg of Fukitol.
In other words, do they do stuff that actually matters?
In a word, yes. Nate Silver manages the blog FiveThirtyEight and is well-known as a statistical analyst from the 2008 US election (among other things). Strategic Vision has released quite a few polls. In Silver's words,
So yes, this is pretty big news, should it turn out that Strategic Vision's behavior is in fact illicit. They're influential enough that news agencies may pick up their polling results. This is bad enough, but when you factor in the fact that polling results can be very effective propaganda in something like a presidential race, fraudulent polling can have significant consequences.
Well said.
I find it disturbing, too, that the media just reports the polling companies' results, without reporting things like what questions were asked, in what order, how the poll was conducted or who commissioned it, all of which can have a big effect on the results. A lot of "push polling" goes on, especially when the polls are commissioned by special interest groups, business associations, unions or political parties themselves.
I'm not in the US, so I don't know this polling company, but I've had a municipal, provincial and federal election in the past 12 months (with another possible federal election imminent) and I think polling and radio call in shows have a great deal of effect on people's opinions these days, more so than traditional newspaper and television newscasts.
If Strategic Vision was conducting fraudulent poles, I would be looking at their client list and going after whoever paid for them as well.
I don't care why you're posting AC
Fortunately, there are corrections you can do for that. And he took a fairly normal statistical test on the numbers, which is equivalent to saying he didn't perform that many comparisons. To very rough approximation, you need to correct your p-value for all the less weird analyses you might have performed on the data instead. It's a bit hard to pin down an exact p-value for the analysis he did (the underlying data isn't expected to be flat; it's also not expected to be that bizarrely lumpy), but I promise that Nate Silver has an understanding of this issue (which you'd see, if you'd read the post).
Nope, you're looking at the webpage of a different company! See Nate's previous article:
Strategic Vision is a Republican pollster. Meaning when a Republican politician wants a poll about a particular set of data they give Strategic Vision some money and they do a poll. This can be for either internal polling to give them and idea how the "battle" is going or for general consumption. And yes Strategic Vision is big enough to matter, but they are just the tip of the iceberg how misleading "R" pollsters
In general there are some Republican some Independent and some Democrat pollsters however all of their results are supposed to be scientific the idea is dose a poll for internal consumption really help if tells you that you are going to win easily on election day only to have to be a landslide against you?The answer is no.
The reason why this is dangerous is multi fold. 1) Due to the supposed scientific nature it has been used to make public policy decisions 2) It can influence peoples opinions. 3) It can influence a senator's or some other politicians choices while they are in power.
Here is a perfect example of this. A certain Republican senator from Maine is considering if she should support a public option, so she wants to see what the citizens of her state think about the topic. She hires Strategic Vision to do a poll for her. Strategic Vision comes back and says 60% of your state's citizens are against it. She gose "Wow I guess im not supporting that bill" In reality its 60% the other way. From this the senator decides to not support the bill and it dose not pass.
I will be as blunt as possible. I am accusing Rasmussen, Strategic Vision and other Republican pollsters of deliberately lying to the American people in order to alter the public debate. If you follow the math they have been consistently off for years. If you want to just look at the last election cycle Rasmussen etc all had the results a lot tighter than the results on election day. This could just be poor polling on their part but I will offer exhibit B
Since health care reform has been a topic in the news the difference between the several Republican pollsters and "everyone else" has been steadily growing. I firmly believe that the insurance industry has been paying these pollsters to lower their numbers for the democrats to push them to drop health care reform.
Yes the Democrats poll numbers have been sliding somewhat across the board. However if you look at the data from the Republican sources. They have the numbers significantly different than those of the "Independent and Democratic" pollsters.
Over all I want to say this "dishonest polling" helps no one. It may help push a certain agenda temporarily but It can also cause those who support it to loose elections..... Look at the results from 2008 the REPUBLICAN PARTY IS BEING MISLEAD BY ITS OWN POLLSTERS AND IT IS COSTING THEM ELECTIONS
Second, if they're the same "strategic vision" that the article is talking about
They're not, from another helpful article from FiveThirtyEight
Why would you pick the name "Strategic Vision, LLC" for your company when the name "Strategic Vision, Inc." was already in use by an extremely well regarded, San Diego-based research firm that has been in business for more than 30 years? Are you deliberately trying to confuse your potential clients and leverage Strategic Vision, Inc.'s much stronger brand name?
You're looking at the page from the well regarded Strategic Vision, Inc. Funny that SV LLC seems to be so happy to sue Nate Silver, it would seem that SV Inc has a far stronger case against SV LLC.
Could be an interesting intersection of Trademark/Slander laws...
I stole this Sig
Except you've linked to the wrong company. Strategic Vision, Inc. is a well respected 30-year old polling firm in California. Strategic Vision, LLC is the shady 5-year old GOP shill corp with questionable poll results and no real office (or polling results allegedly). Careful with those links, you don't want to slander the wrong company here. I think SV Inc. may have a trademark case on their hands if their feeling litigeous.
They are a partisan, Republican-oriented polling company. They have gotten into trouble in the recent past for their questionable results.
First, the example he gives where he looks at polls from ALL sources is an example of a plausible distribution of real results because, assuming the majority of pollsters are not cooking their data, the data should be dominated by randomness. He then looks at this particular pollster and finds a much greater disparity in trailing digit frequency. The question is, is it significant, or just chance?
Given the numbers, it's not particularly hard to figure out. You can calculate the likelihood of any particular result given a theoretical distribution using a G test of goodness of fit. Technically for numbers this small you could use an exact test but I don't know of a web version and I'm too lazy to write one up. But here's a description of, and an excel spreadsheet that performs, the G test of goodness of fit: http://udel.edu/~mcdonald/statgtestgof.html
Basically, you plug in the distribution you see and compare it with the one you expected. What you get is the probability of that distribution occurring by chance. So if we plug in the observed data for all the pollsters and assume equal likelihood for all trailing digits we get a p=0.006. Whoops, looks like our assumption isn't quite correct. As the blog author notes, the observed distribution is humped a little, favouring the middle numbers. He also gives a possible explanation. For giggles, the probability of the Strategic Vision results given equally probable trailing digits is absolutely microscopic: p=1.44x10^-17. Together those tell us that our assumption of equal digit distribution is probably not quite right, but the Strategic Vision data still looks mighty funny.
Okay, so assume instead that most pollsters aren't making up their numbers. Not that their numbers are necessarily accurate, but that they're at least not making them up off the top of their heads. So using the data from all pollsters as a template, how likely is the Strategic Vision distribution? That's a G test of independence: http://udel.edu/~mcdonald/statgtestind.html. We could use Fisher's exact test, but I can't find one that will do a 2x10 table.
Plugging in the data, we get G=43.068, d.f.=9, which gives p=2.09x10^-6. The blog author was actually a little careless when he said the chances of Strategic Vision's results are millions to one against. If you insist on the equal-probability theory then the odds are 70 quadrillion to one against Strategic Vision and 166 to one against the industry as a whole. Taking the more realistic approach that the industry average is a better representation of the actual probability, the odds against Strategic Vision's results are about half a million to one against. Not millions to one, but close enough.
I hereby take back everything I said about Strategic Vision and reapply it to Strategic Vision, LLC, times two.
"Yeah, you go ahead and cling to the belief that the insurance industry doesn't want the health care bill to go through"
You are right the insurance industry would stand to gain massively by that proposal. That's exactly why the liberal sect of the democratic party has been fighting that provision.
I would like to point out that the insurance industry is being very pragmatic they have a two tier battle plan. They don't want the bill to pass however if it dose pass they want to have things like that put in
That provision was added to some of the bills to "tempt" republicans into voting for it as several Republicans have explicitly said they would like to see that included.
As far as "I'd certainly like to see some numbers regarding who the insurance industry as a whole is contributing to." The money has been flowing quite rapidly into the conservative arm of the democratic party. Ben Nelson, Mary Landrieu and Max Baucus have all goten heavy donations since this whole thing has started (from insurance companies). That is not to say that the republicans have not been getting a lot of money from the insurance companies. (That goes without saying) So to some it up Republicans are continuing to get good pay checks,(the usual) however some conservative democrats are now also getting paid for their services(Newish). Just for your info many progressives want political blood for this, Ben Nelson and Max Baucus and to a much lesser extent Mary Landrieu are the one thing that is standing in the way of progressives' holy grail. For that many of us want political revenge at any cost.
I did a statistical analysis off the year 2000 "recount" almost 9 years ago, looking at the counties with "unusual" results.
There were six counties in which the changed votes didn't fit the normal bell curve, four benefiting Gore and two Bush.
Both of Bush's and one of Gore's had rules in which replacement ballots were made for idiot voters who used an X rather than filling the bubble, explaining them.
One of Gore's had machine problems in the recount and stuck with the original figures.
And then there were the two counties, which accounted for the lion's share of the "correction" from the recount.
One of them was 50 standard deviations out--so far out that it is less likely than winning the California Lottery every week for thirteen weeks running . . .
I wasn't the only one to notice the oddity, but the sad fact is that noone cares . . .
hawk
Well, you might need to explain what astroturfing is
Astroturfing is where a special interest tries to create the impression of grassroots support. That may be through paying shills to post a lot on message boards with posts that support your position, it may be through dodgy polls or it may be through other means.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register