Slashdot Mirror
Many Surveys, About One In Five, May Contain Fraudulent Data (sciencemag.org)
sciencehabit writes: How often do people conducting surveys simply fabricate some or all of the data? Several high-profile cases of fraud over the past few years have shone a spotlight on that question, but the full scope of the problem has remained unknown. [Tuesday], at a meeting in Washington, D.C., a pair of well-known researchers presented a statistical test for detecting fabricated data in survey answers. When they applied it to more than 1000 public data sets from international surveys, a worrying picture emerged: About one in five of the surveys failed, indicating a high likelihood of fabricated data.
Well, we know one does for sure.
Our survey shows many surveys contain fraudulent data. And I am lying right now.
When I take most surveys I answer calculated to confound the test as much as possible, on the assumption that anyone could be doing so; and thus, to accelerate the process is to bring it to the attention of the researchers faster. The problem is that it's an inherently flawed method.
23.7 % of statistical analyses make up their statistics.
Some drink at the fountain of knowledge. Others just gargle.
flesh feet
I bet its higher than that.
"You can come up with statistics to prove anything, Kent. Forfty percent of all people know that."
It's not that only 1 in 5 surveys may contain fraudulent data, it is that the fraud is only incompetent enough to be caught by this method in 1 in 5 surveys.
Like it or not, a lot of public opinion polls are paid for by people who want to support a specific point.
Public opinion polls these days are as much PR and marketing as anything else.
Honestly, Pew makes money doing this stuff; honest player or not, they have a vested interest in keeping up the belief that their stuff is honest, unbiased, and accurate.
But I'm entirely willing to believe opinion polls are carefully crafted, or sneakily tweaked, to arrive at the conclusions they've been commissioned to a arrive at.
Lost at C:>. Found at C.
...a pair of well-known researchers presented a statistical test for detecting fabricated data in survey answers...
That sounds a little suspect...
If you want news from today, you have to come back tomorrow.
Obligatory:
Lies, Damn Lies, and Statistics
There's a 76.3% chance your statistic is true. And a 23.7% of a paradox at play.
If you ask: "Do you believe that mothers should be able to legally murder their babies within 2 months of the creation of life?" you get a very different answer than if you ask "Do you believe that women should have the legal right to abortion when the fetus can be demonstrated to show no brain activity more significant than that of a snail."
This might be intentional, or simple unconscious bias.
excitingthingstodo.blogspot.com
4chan is known to trolls lots of these things often with hilarious results.
So one in five studies might contain false data. Essentially what they are saying is that there's a 20% chance that they are lying about there being a 20% chance of them lying.
My sci-fi novel, Ghost Thief, is now available from Amazon.com.
What they ACTUALLY said was that in surveys conducted in the Western World - only 5% failed their test - but in developing countries - the number was 26% of faked surveys.
Then, they also say that the KIND of survey matters. Their approach is to say that if 85% of answers are identical between two or more respondents then the result is likely to be faked...but they recognize that (for example) in a health survey, all of the healthy people will answer identically to questions about how healthy they are. So that kind of survey is excluded.
So if the research is to be taken at face value, then in the Western world, one in twenty of *some* classes of survey are probably faked. But they looked at 1000 surveys to arrive at that number - we don't know what fraction of those came from the developing world. If all you're interested in is Western World surveys - then maybe the sample size is very small. Given that there are some classes of survey that are known to be excluded - is it possible that they included a few of "the wrong kind" in their sample.
All surveys have an error bar of a few percent - this is a survey about surveys.
I think the conclusion here is that you should ignore surveys carried out by dubious agencies in the developing world. I don't think you should conclude that surveys done by reputable agencies in the western world are unreliable.
www.sjbaker.org
A few years ago a survey said 2/3 of the Danes supports nuclear power. Two days later another survey said 1/3 supports it. A journalist then figured at least one of them is wrong and started digging. It turned out that they didn't answer the same question. 2/3 said yes when asked if nuclear power can be used to reduce worldwide CO2 emissions. 1/3 said yes to Denmark gaining nuclear power for environmental reasons. Despite not answering the same question, the press releases and following headlines made it look like they did.
There is way more to this than "not in my backyard" to this issue. For starters Denmark can't get rid of nuclear waste. Despite the politicians declaring it a nuclear free country, somebody got away with building 3 test reactors "for research purposes". Today there is 233 kg nuclear waste and it turns out that there is nowhere to get rid of it. Being islands of mud placed by the glacier during the ice age, there is no stable rocky underground to store it. With that in mind and the huge debate about it, I'm more surprised 1/3 even said yes to nuclear power. Main argument (in the media) is that the world leading country in wind energy shouldn't start focusing on something it has no experience in. From an engineering perspective, the high voltage network needs to be rebuild to handle a single source of production rather than distributed production, which would be at least as expensive as the powerplant itself. The list of reasons other than "not in my backyard" for saying yes to nuclear power elsewhere and not in Denmark seems almost endless.
Back on topic, US had a leading question not long ago. 30% of the republicans wants to bomb Agrabah, the city Aladdin lives in. It makes good headlines, but looking at the survey, it had 50 questions, 49 serious. Question 34 (or was it 32?) asked if Agrabah should be bombed or not. Based on wording and context, at least 30% read that as "should (middle east city pointed out by US military) be bombed?". It had absolutely no hint of the city being fictional.
...if the data from THIS survey was deliberately falsified to see if anybody actually checked the sources?
Darryl L. Pierce "What do you care what people think, Mr. Feynman?"
Would be easier to write fixed questions wouldn't it? Also harder for the less observant to spot. :D Surveys in my mind go into the category of statistics, which can lie in a number of ways. Surprised at this news.
"Imagination is more important than knowledge" - Einstein
Many Surveys, About Eight In Three, May Contain Fraudulent Data
Just cruising through this digital world at 33 1/3 rpm...
"Do you prefer chocolate or vanilla?" is different from "Do you support Falun Gong?" Opinion surveys always have to account for the confounding factor that each respondent may be more likely to provide the socially acceptable answer than their true opinion. The stronger the social stigma associated with the question, the more likely this will be a problem.
This new test is a useful addition to the data analysis process, but doesn't "prove" anything. The challenge is how to refine the technique. If you want to eliminate "false positives" you would need some way to identify "true positives". And if we had a way to do that, we wouldn't need to do surveys.
Bottom line: Surveys don't prove anything. At best they point to interesting ideas for future study.
Nope, no sig
Over half of all analysis of surveys is flawed.
I once had a girlfriend who was doing her PhD. She habitually 'nudged' her data co-ordinates closer to the line of best fit. Other data she merely fabricated.
The other 4 of 5 is just biased by wording the question in a carefull way or with an odd scale.
A survey showed that 1 in 5 surveys have fraudulent data. The other 4 surveys had different results.
Roughly half the marketing departments at companies I've worked for have used half-baked surveys to gather statistics so the company name and the statistic get repeated in the industry over and over again.
This often happens like this: "At (industry conference) this year, let's pass out a survey asking whether or not someone has every heard of a coworker getting hacked by (whatever threat our product purports to mitigate). Survey goes out to already half-paranoid people walking by, and the entire marketing and sales department fills one out that says 'yes I have'. A week later a press release goes out that says "(company) surveyed (# of people) IT managers and other attendees at (conference) and found that (high percentage) had direct knowledge of a coworker getting hacked by (threat)." Very often this stuff gets picked up by the press, bloggers and even other competitors, and the essentially made-up stat gets repeated and repeated until some people even think its true.
Examples:
- http://www.tripwire.com/compan...
- http://www.prnewswire.com/news...
- https://www.voltage.com/breach...
It works every time.
has more than single digit support. Corporations like Gallop support him and exaggerate his support.
When surveys ask for personal details that I don't want to supply, I often put crap data in if they don't offer a way to bypass it, such as a "Do not wish to state" option among the choices. For example, age, occupation, and income level.
Table-ized A.I.
How does this affect the outcome of every episode of FAMILY FEUD??? Maybe the Jones family won after all?
If telephones are outlawed, then only outlaws will have telephones.
...were technology surveys by Gartner that were sponsored by Microsoft?
Now any paper on sociology, psychology, education or law has to be destroyed and the tests/surveys repeated under strict supervision.
Political surveys, media surveys, advertising surveys ( "three in ten dentists use KY...") must all be suspected of being fraudulent.
By law, if they are using false information claims for advertising, then they are commiting a crime.
Media and politics is already distrusted so much that the owners/CEOs hide.....
Blame it on lawyers and the mentality of "how should this actually look? Not like this! I'll just fix it..."
Lack of integrity, honesty and common sense are contributing factors.
Who in the world takes a 100 question survey? Every survey I've been asked to take has been less than 10 questions.
There is no data that will ever meet your standards. You'll always find some reason to conclude that the data you're reviewing are incorrect or have been "tampered with" or whatever unless they don't show any warming trend or you can find some some way to illustrate that any warming trend is beyond human control (bonus points if you can show that it's temporary).
How many? Something like One on Five?
Or that third world totalitarian governments have succeeded it brainwashing their people, to the point that they all answer questions the same?
Of course, here in America we aren't brainwashed. Instead, they give Kardashians.
But I can say that on every porn site that asks, I can truthfully say that I was born on January the 1st, 1927.
Public opinion polls these days are as much PR and marketing as anything else.
Honestly, Pew makes money doing this stuff; honest player or not, they have a vested interest in keeping up the belief that their stuff is honest, unbiased, and accurate.
But I'm entirely willing to believe opinion polls are carefully crafted, or sneakily tweaked, to arrive at the conclusions they've been commissioned to a arrive at.
This is NOT AT ALL about public opinion polls which "people who want to support a specific point" would pay to skew in a certain direction NOR is it about polls that are designed (or "sneakily tweaked") to create certain results.
It's not even about confirmation bias by the pollsters or researchers leaching into the data.
This is about finding cases where a pollster would just sit down and fill out a survey after survey by themselves instead of going door to door.
I.e. Charging for a field survey, forging the data and then pocketing the money.
Further more, it is not about most kinds of "public" opinion polls.
The results show that what most people here would consider "public opinion polls" are actually mostly free of such fabrication.
It's that "other public" whose opinion polls are the issue.
I.e. The "developing world" public.
From TFA:
That made the results all the more worrying: Among 1008 surveys, their test flagged 17% as likely to contain a significant portion of fabricated data.
For surveys conducted in wealthy westernized nations, that figure drops to 5%, whereas for those done in the developing world it shoots up to 26%.
Surveying communities in the developing world often requires face-to-face interviews, going house-by-house in dangerous environments.
So one of the inevitable problems, Robbins says, is "curbstoning" where an interviewer sits on the curb and invents survey responsesâ"often duplicating answersâ"in order to avoid risk or save time.
Mit der Dummheit kämpfen Götter selbst vergebens
From TFA:
During her turn on the stage, Kennedy mounted an attack on the test's methodology. For example, she points out, it does not account for the number of questions on a survey, the number of respondents, nor other factors that can skew the results. She also takes exception to the 85% similarity threshold. "I would choose a different threshold depending on the population and the survey," she says. By putting a number on the extent of data fabrication across all surveys, "they took it too far," Kennedy says. Pew's rebuttal is now online.
Some at the meeting saw merit in both sides of the fight. Rather than overestimating data fabrication, the method of Kuriakose and Robbins "very likely underestimates the true extent of the problem," says Michael Spagat, an economist at Royal Holloway, University of London, who has investigated high-profile cases of possible data fabrication in war zones. Yet Kennedy's response impressed him, too. "I think the Pew paper is interesting and made some good points," he says. "Specifically, there isn't a hard and fast cutoff beyond which you know there is fabrication." Overall, however, Spagat remains very concerned about data fabrication in surveys. "Robbins and Kuriakose have uncovered a massive problem and the Pew paper doesn't change that."
Nothing was settled by the end, says the meeting's co-organizer Steven Koczela, president of the MassINC Polling Group in Boston and a previous survey research leader for the U.S. State Department. The case laid out by Kuriakose and Robbins "seems unassailable to me," he says, "but [Pew] are giving it their level best."
Mit der Dummheit kämpfen Götter selbst vergebens
I used to own a polling company with my wife Tracy Costin The people that sponser the survey's are always asking for more then is realisticly possible they will be making changes to survey questions right up to the last minute then they want a certain number of responsise you can only cal a certain number of people in a certain geographic area once use use up all your numbers and you can't get anyone else to answer questions what do you do? plus they are always in a hurry you have to complete the survey in a very short time. you can only call between certain hours. it's really a high pressure biz you get a lot of serveys that are incomplete it's really frustrating a little cut and pate of partial srveys can go a long way you don't have to make up data just good old cut and paste
http://Lenny.com
An online survey found that 92847639 people per day didn't read long numbers
...why only 4 out of 5 dentist approve of Crest White Strips
Mod me down, I shall become more off-topic than you could possibly imagine.