Many Surveys, About One In Five, May Contain Fraudulent Data (sciencemag.org)

sciencehabit writes: How often do people conducting surveys simply fabricate some or all of the data? Several high-profile cases of fraud over the past few years have shone a spotlight on that question, but the full scope of the problem has remained unknown. [Tuesday], at a meeting in Washington, D.C., a pair of well-known researchers presented a statistical test for detecting fabricated data in survey answers. When they applied it to more than 1000 public data sets from international surveys, a worrying picture emerged: About one in five of the surveys failed, indicating a high likelihood of fabricated data.

Comments

Gee, you don't suppose respondents lie?

[Anonymous Coward]

When I take most surveys I answer calculated to confound the test as much as possible, on the assumption that anyone could be doing so; and thus, to accelerate the process is to bring it to the attention of the researchers faster. The problem is that it's an inherently flawed method.

Re:Gee, you don't suppose respondents lie?

[ShanghaiBill]

When I take most surveys I answer calculated to confound the test as much as possible

If a few people did this randomly, then it wouldn't skew the results much. But it is not random. Liberals are more willing to participate in surveys, and more willing to answer honestly. Conservatives tend to be more cynical and calculating. Other factors skewing the results are that Democrats are more likely to be home, more likely to answer the phone, and more likely to participate in social media polls. Republican are more likely to let their phone calls roll over to voice mail, but also more likely to have more than one phone line. Professional pollsters try to take all these factors into account.

The statistical test in TFA won't fix the problem. It will just give the fakers another tool. Just make up some fake data, apply the test, and if it fails then manipulate the data and try again.

Re:Gee, you don't suppose respondents lie?

[codeAlDente]

Probably not the article you're talking about, but same conclusion, and a good read: http://prospect.org/article/ac...

Re:Gee, you don't suppose respondents lie?

[ShanghaiBill]

Can you link to a paper, etc., that calculates the adjustments for liberal vs. conservative survey-taking patterns?

Sorry, I have seen these issues mentioned several places, including Nate Silver's blog, but I have never seen them actually quantified.

I think the most famous skewed poll was in 1948. Phone surveys predicted a decisive win by Dewey, but instead Truman was re-elected. The reason was that back in 1948, households with lower incomes (and more likely to vote Democratic) often didn't have a phone.

Self referential?

[goombah99]

23.7 % of statistical analyses make up their statistics.

One in 5 surveys incompetently fraudulent

[Diss Champ]

It's not that only 1 in 5 surveys may contain fraudulent data, it is that the fraud is only incompetent enough to be caught by this method in 1 in 5 surveys.

I'm not actually surprised ...

[gstoddart]

Like it or not, a lot of public opinion polls are paid for by people who want to support a specific point.

Public opinion polls these days are as much PR and marketing as anything else.

Honestly, Pew makes money doing this stuff; honest player or not, they have a vested interest in keeping up the belief that their stuff is honest, unbiased, and accurate.

But I'm entirely willing to believe opinion polls are carefully crafted, or sneakily tweaked, to arrive at the conclusions they've been commissioned to a arrive at.

Re:I'm not actually surprised ...

[gstoddart]

Right, like the Gartner Magic Quadrant shit that most C-suite people drool over.

Or bond ratings agencies.

I suspect most people, except the people who cite those things, have long since assumed they're full of shit and the conclusions are paid for.

Why would you assume it's honest and objective information? Someone has to make money off it.

And if you don't like that, start your own foundation or think tank, and have them publish stuff to your liking.

Sorry, it's all PR and marketing. It sure as hell aint facts or accurate predictons.

May not be fraud - simply incompetence.

[gurps_npc]

There are a lot of ways you can screw with a study. For example, you get dramatically different answers by rephrasing the question.

If you ask: "Do you believe that mothers should be able to legally murder their babies within 2 months of the creation of life?" you get a very different answer than if you ask "Do you believe that women should have the legal right to abortion when the fetus can be demonstrated to show no brain activity more significant than that of a snail."

This might be intentional, or simple unconscious bias.

Re:only 1 in 5 fraudulent?

[penguinoid]

The rest are merely intentionally misleading. You can get just about any answer you want, without making up data, by carefully selecting your questions and your survey population.

Be careful...read what they actually said.

[sbaker]

What they ACTUALLY said was that in surveys conducted in the Western World - only 5% failed their test - but in developing countries - the number was 26% of faked surveys.

Then, they also say that the KIND of survey matters. Their approach is to say that if 85% of answers are identical between two or more respondents then the result is likely to be faked...but they recognize that (for example) in a health survey, all of the healthy people will answer identically to questions about how healthy they are. So that kind of survey is excluded.

So if the research is to be taken at face value, then in the Western world, one in twenty of *some* classes of survey are probably faked. But they looked at 1000 surveys to arrive at that number - we don't know what fraction of those came from the developing world. If all you're interested in is Western World surveys - then maybe the sample size is very small. Given that there are some classes of survey that are known to be excluded - is it possible that they included a few of "the wrong kind" in their sample.

All surveys have an error bar of a few percent - this is a survey about surveys.

I think the conclusion here is that you should ignore surveys carried out by dubious agencies in the developing world. I don't think you should conclude that surveys done by reputable agencies in the western world are unreliable.

Re:Be careful...read what they actually said.

[codeAlDente]

This is not a survey about surveys. This is a statistical test of the number of identical responses in a survey, and the likelihood that such uniformity could have resulted by chance. That probably indicates not just fraud, but incompetent fraud (or, to be fair, perhaps people in developing countries are just much more likely to answer independent questions in exactly the same way, and the researchers drastically underestimated the independence of these questions). Seems like grading math or physics homework though - it's a lot harder to demonstrate when the top students are copying answers from each other.

How many?

[JustAnotherOldGuy]

Many Surveys, About Eight In Three, May Contain Fraudulent Data

All surveys are not the same

[drew_kime]

"Do you prefer chocolate or vanilla?" is different from "Do you support Falun Gong?" Opinion surveys always have to account for the confounding factor that each respondent may be more likely to provide the socially acceptable answer than their true opinion. The stronger the social stigma associated with the question, the more likely this will be a problem.

This new test is a useful addition to the data analysis process, but doesn't "prove" anything. The challenge is how to refine the technique. If you want to eliminate "false positives" you would need some way to identify "true positives". And if we had a way to do that, we wouldn't need to do surveys.

Bottom line: Surveys don't prove anything. At best they point to interesting ideas for future study.

Half the marketing departments I've seen

[xxxJonBoyxxx]

Roughly half the marketing departments at companies I've worked for have used half-baked surveys to gather statistics so the company name and the statistic get repeated in the industry over and over again.

This often happens like this: "At (industry conference) this year, let's pass out a survey asking whether or not someone has every heard of a coworker getting hacked by (whatever threat our product purports to mitigate). Survey goes out to already half-paranoid people walking by, and the entire marketing and sales department fills one out that says 'yes I have'. A week later a press release goes out that says "(company) surveyed (# of people) IT managers and other attendees at (conference) and found that (high percentage) had direct knowledge of a coworker getting hacked by (threat)." Very often this stuff gets picked up by the press, bloggers and even other competitors, and the essentially made-up stat gets repeated and repeated until some people even think its true.

Examples:
- http://www.tripwire.com/compan...
- http://www.prnewswire.com/news...
- https://www.voltage.com/breach...

Re:Leading questions are also an issue

[EndlessNameless]

It had absolutely no hint of the city being fictional.

If they're willing to bomb a city and kill people without even knowing the specific reason for that bombing, their ignorance is truly dangerous to the world.

If they cannot immediately recall the "where" and "why" to justify homicide, they deserve to be embarrassed.

The only conceivable justification for their position is the lack of an option for indicating "unsure" or "no opinion". And I'd be shocked if a modern survey didn't include that.

Re:97% Consensus

[frovingslosh]

Hey, we thought that 1 in 5 sounded like a nice believable number. How dare you question it. We looked at at least three surveys to come up with this.