Slashdot Mirror
Study Shows One Third of All Studies Are Nonsense
SydShamino writes "CNN has a report on new research to confirm claims made in initial, well-publicized studies. According to the new study, about a third of all major studies from the last 15 years were subsequently shown to be inaccurate or overblown. The study abstract is available."
My apology in advance for being a MC-person, but if 1/3 of the studies are inaccurate, which means this study can be 1/3 inaccurate, does it mean that the actual inaccuracy is 1/3 * 2/3 = 2/9 of all major studies are inaccurate?
Rock that crushes, Paper & Scissors that don't matter.
What... are you guys trying to blow up our heads?
;)
I think it is possible this is the most amusingly ridiculous piece of "legitemate" news I've read in awhile...
Anyone got anything to beat it? Post it, I need to shock my brain a little more
MoM++ - A Classic Expanded - [Master of Magic 1.5]
http://mompp.sourceforge.net/
Since when is "inaccurate" or "overblown" nonsense? That's what science is: study something, make a theory, and just about dare someone else to prove it wrong, because that's what makes for a better theory.
[
This study will cause an infinite loop..PLEASE SHUTDOWN IMMEDIATELY. 0x381F
Zapp: Kiff we have a conundrum!
"There is no time, sir, at which ties do not matter," Jeeves, (Jeeves and the Impending Doom)
Because this is from the other 2/3rd studies talking about the other 1/3rd studies.
What does your Credit Report look like?
Well, it says 1/3 of the studies are inaccurate, so let us rank that on a percentage scale, say the study is inaccurate, we give them a 0 value.
... *GASP FOR AIR* ... okay... so you have 66% chance of that particular study being at least somewhat accurate...
... (rolls percentile dice)... OOOOHH! Man, rolled a 72, looks like I can't believe it.
Accurate studies, lets say 100 (I know nothing is 100% accurate, and I know most studies even if they are somewhat accurate probably don't exceed 70% probability even in the specific environments they are enacted in, but lets just be over-generous since this whole thing is rather ridiculous anyway)
Right?
Rats.
MoM++ - A Classic Expanded - [Master of Magic 1.5]
http://mompp.sourceforge.net/
Just because a study has inaccuracies doesn't mean the whole thing is nonsense.
Obligatory Simpsons quote:
"Oh, people can come up with statistics to prove anything, Kent. 14% of people know that."
According to a recent study involving 100 clones based on DNA fragments of Karl Popper, a statistically significant number of the clones agree that this is pretty goddamn good result, considering that that's how science is supposed to work.
You know - that silly process whereby you make a falsifiable claim, run an experiment, report your results, and encourage others to add to the store of scientific knowledge by attempting to falsify your original hypothesis?
The fact that they get away with it is a shame. It's even worse when they have an influence on government policy. Ugh.
Lots of people can't think of a good reason to do science, maths and statistics at school. Well, a bloody good reason is so you can prevent the wool being pulled over your eyes.
Deleted
all studies are biased and show exactly what the person who is doing
them want. much like stats, you can make anything prove anything
through working the #'s and asking the right questions
Can you prove this with a study? If not, I'll not believe you. I asked 100 people if they thought studies were overblown, and 33 of them said 'yes.' Where's
Pulp Audio Weekly - Geek News and Reviews
What a nice paradox this story presents. "I am lying."
The way the topic makes this sound, this is some sort of blatantly obvious study...
:)
... so it's kind of... ambiguous...
it isn't, really
It is about the effectiveness of interventions... if you skipped over it, its worth a perusal to a skim... at the very least... but it would seem to me that the whole thing has lead to almost no positive conclusion itself... with 44% of the experiments being replications and 24% unchallenged... the 66% really don't seem to have much value...
Ahhh, academic research... only there can you get paid well to tell us absolutely nothing...
MoM++ - A Classic Expanded - [Master of Magic 1.5]
http://mompp.sourceforge.net/
it was published three times?
four out of three people have problems with fractions.
According to the new study, about a third of all major studies from the last 15 years were subsequently shown to be inaccurate or overblown.
The actual figure turns out to have been 26.4% - much closer to 1/4 than 1/3.
I did my own study and it shows that one half of all studies are nonsense.
The people responsible for the previous study have been sacked.
Aaargh. Comments like this turn up in droves in every story that mentions statistics (in any light, whether good or bad) and ... wait for it ... they're always wrong. 100% of the time. I can state that with absolute certainty. No margin of error.
The fact is, yes, statistics can be misused. So can every other field of study. But used right, statistics are a tremendously powerful way to understand our world, and often reveal information that can't be obtained any other way. And believe me, nobody gets more peeved at statistics abuse than statisticians do.
But that's okay, pal. Just keep on making fun of things you don't understand. The smart people of the world will keep on working, keep doing things that make your and everyone else's life better, whether you know it or not.
The correlation between ignorance of statistics and using "correlation is not causation" as an argument is close to 1.
When companies can buy reports and studies to say whatever the fuck they want them to say (*cough*microsoft*cough*), of course they are going to be bullshit.
Who's surprised by this? Seriously.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
Even worse are the lazy journalists who report it. After a New York Times piece last week claimed bisexual males were "lying" based on results from a highly questionable study, I reminded their editors of this excellent piece Blinded by Science in Columbia Journalism Review.
This kind of sloppy reporting is perfect for lazy journalists-- it's a three-for-one deal. They get to break the news, and then later they have a second story when real experts point out the flaws, and a third when the people finally get discredited. More evidence of the shameful state of journalism in this country.
Evil sig is livE.
..and one third of the replies to this article will be lame jokes about this study being nonsense.
Drink 8 Glasses of Water a Day (does beer count?)
Depends. One would have to calculate the water content of beer versus the rate of dehydration produced by the alcohol content. Following through, one would conclude that 8 glasses of beer would fall short of the goal of 8 glasses of pure water, with the only recourse being to drink more beer.
This, kids, is a practical demonstration of how to make science work for you.
Shouldn't you be doing something useful?
Most obvious comment ever has been taken. On November 12, 2001 (61 days after the WTC attack in NYC) American Airlines flight 587 took off from JFK and promptly crashed into a Queens neighborhood. Obviously, most Americans suspected the worst. That day, I was watching CNN when one of these so-called "experts" (sic) came on and actually said in plain english:
"This is not a very good time for something like this to happen."
So my question is this: when is a good time for an airplane full of people to crash into a residential neighborhood? This guy should designate a day for us so we can make sure all the airlines and pilots know when the good day for crashing is. Morons.
No - the standard way to report statistics is with a 95% confidence interval (i.e. there is a 95% chance that the true value lies within the interval quoted). It's just a norm that has nothing to do with overblown results (at least not directly) and nothing to do with the study.
See subject
I feel like alot of posters are not understanding what the study is... this is probably because the abstract (or, if you have access, the actual article) is much more meaningful than the CNN report.
First, notwithstanding the many good jokes about a self-referrential study that will proven to be exaggerated, this study specifically checked whether highly cited clinical studies had claims that were later contradicted or softened due to other research. This study was not claiming that 1/3 of all scientific studies published were wrong in some way. It's worth noting that doing clinical research is very difficult, and that the error bars will always be quite large. It's also important to keep in mind that sometimes clinical research may be unduly influenced by financial pressures... and that clinical research undergoes very heavy scrutiny.
So having 1/3 of all clinical studies be later contradicted should not make us worry that clinical research is being done wrong. We should be happy that so much verification occurs, that any erroneous conclusions will (probably) be checked again. One line from the CNN article rings true:
Experts say the report is a reminder to doctors and patients that they should not put too much stock in a single study and understand that treatments often become obsolete with medical advances.
I think that should be the take-home message for the casual reader. Science is doing its job of verification, but people need to stop jumping to conclusions (or worse, changing their life habits) based on the results of a single study. The results need to be double-checked. The study may have been a fluke, or have flaws, or the data may have been manipulated. Whatever the reason, we should not trust single experiments, especially where human lives are at stake!
Having (partially) read the JAMA article, I think their result is sobering and useful. It really shows how intense the competition is in that field (which leads both to people making exagerrated claims, but also alot of pressure to dis-prove other's claims and get at the "right answer").
Connectivity - global media, the internet -- have created a winner-take-all world that drives both the creators of studies and the reporters of studies toward hyperbole. If someone wants their 15 minutes of fame, they need to do (or appear to do) something spectacular. When attention is a scarce resource (because of an explosion of applications/demands for attention), then it drives people toward excessive behavior in crafting and reporting the results of studies.
At the same time, I wonder if the long tail efect means that an increasing number of once-obscure, high-quality studies are being discovered, read, and used by an increasing number of people. Those that do create unbiased studies may not get much popular press, but they do become more widely read due to Google.
Ultimately, we seem to be floating in a rising tide of both good and bad studies. Perhaps the ratio of studies is being biased toward the bad (winner take all) but the ratio of impressions -- the numbers of times that good studies have been accessed -- has actually improved due to long-tail effects.
Two wrongs don't make a right, but three lefts do.
It's not that boggling when you consider 62% of all Insightful's are Overrated and 83% of all Funny's are Redundant.
-Rabbit
I think that this study is measuring what it says it does correctly(so isn't flawed by its own criterion), but it is measuring the wrong thing.
Think about it: they are measuring highly cited studies that get a "stronger" result than other subsequent studies. However, they simply say "stronger"(at least in the abstract). Whenever you measure something other than by a census, you take a sample. Therefore, as anyone familiar with elementary statistics should know, you have sampling variation. Researchers then usually appeal to the 'central limit theorem' to assume that the mean comes from a normal distribution, and then, knowing the distribution of the mean, make a statement like "We are 95% confident that x is between a and b". The later experiment will make a similar such claim.
Assume the experiment to measure x was performed correctly and identically both times, and there is no change in the effect with respect to time. Each time the experiment is performed, a different mean value of x will be obtained due to sampling error. Since both measurements of x come from an identically distributed population, by symmetry 50% of the time the earlier one will be "stronger" than the later one shows. However, not all highly cited studies are repeated after publication, and not all "me too" studies are published, hence the figure less than 50%.
Clearly, what they should be doing is comparing the confidence intervals, and looking for a statistically percentage of studies which do not overlap. This then would be relevant, as it would show us about non-sampling errors, rather than sampling errors, such as experiments designed or performed or interpreted incorrectly.
X-Has-Sig: yes
Correction: Study shows 66.66% chance that 1/3 of studies are nonsense.
I'm an American. I love this country and the freedoms that we used to have.
"when is a good time for an airplane full of people to crash into a residential neighborhood?"
When the passengers of that airplane are terrorists and that residential neighborhood is al-Qaeda's? or maybe when the passengers are kittens and that neighborhood is actually a big pile of yarn that coushins the fall. awwww, so cute.
-Rabbit
Actually, I believe the statistic was Forfty Percent.
Another one C/o Homer:
Facts are meaningless. You could use facts to prove anything that's even remotely true!
----- Concentrate on promoting more than demoting.
I can personally relate to this. While in AP Statistics, we practiced our mad stat skillz on the real world. We were encouraged to bring in newspaper clippings of studies and experiments and see if they were statistically sound. Most weren't. The most common form of bias in those studies were known as negative response. In this form of bias, only people with negative, strong feelings reply to the study question.
I am willing to bet that the CNN study is correct in it's assumption that most studies are incorrect.
Ordinary decent people are sick and tired of being told that ordinary decent people are fed up with being sick and tired. Well I'm certainly not, and I'm sick and tired of being told I am!
That which does not kill her only prolongs my agony.
The research was done for MEDICAL studies, not tech studies, or animal habitat studies, or psychological studies, or sociological studies... only medical studies. Nowhere in the title or the post's main body is this mentioned. This is very poor reporting of the news. It is misleading. The study also only measured studies from 1990 to 2003. That's 13 years not 15 years!
Word to the wise, don't trust the press at face value. Expect sources, preferably cited and available for you to review, and check your facts before you buy into whatever the press happens to be reporting today.
Being able to make up stats and get away with it is one of the nice things about having a PhD... most people do not have the qualifications nor the data necessary to expose the made-up nature.
This appears to be particularly frequent in more abstract (non-maths) sciences like environment. (I once had lectures on the topic where the speaker cited stats that did not match the notes and were inconsistent across presentations.)
So if a third of all studies are overblown, how do we know that this study isn't overblown or inaccurate? Hmmm?
This signature has Super Cow Powers
The main reason that journalism is "flawed" is because in the MSM or Main Stream News, either print or broadcast, the main focus is on making money. It's a "business" first. You will have to somehow make it be "news" first to get more accuracy and objectivity in reporting.
And then it could segue into something roughly analagous to the debates over for-sale closed source software and collaborative information-sharing free software. Could a news reading public be persuaded to actually become critical reporters and "share" news freely? Could it replace the expensive and established profit motive design of "news" as we know it today?
Some might say blogging is at least an attempt in that direction.
Thats a pretty good one, my favorite was this one from the CNN news ticker:
"Public split on whether Bush is a divider."
Just how do we know this study isnt one of the 1/3?
Phredd - "I have found people tend to take you far less seriously once you start waving your genitals at them..."
All that studies report is their results. They don't gaurantee other studies will find the same thing. If they did, there'd be no reason to have replications. This is a basic part of doing science.
And who's to say the replications aren't the ones that missed the mark?
1/3 right
1/3 wrong
1/3 we have no idea what the answer means.
That's what I was told to expect from research in my first semester of grad school. Not from reading it -- from DOING it.
They really should teach science in school. Not the just areas of science, but the subject of science itself.
"I may be synthetic, but I'm not stupid." -- Bishop 341-B
Okay, I know that everyone likes throwing out wisecracks about the headline, which was ever-so-cleverly chosen by the article submitter, but consider the article for a moment.
This is about the accuracy of clinical trial research. This is not about market research studies in the latest clothes fashions. Medicine is an extremely lucrative and risky field -- being associated with the group that pushes through the next Viagra can ensure that your family becomes the next Rockefellers. Your only opposition is the FDA (and the politicians that influence it, which are always hungry for money, which you have lots of).
There is a tremendous amount of pressure on pharmaceutical researchers to produce favorable results. Let's say that you're a new, idealistic researcher who runs some tests on a new drug that your employer wants to market. Your tests show that our drug produces an increased rate of cancer? Well, been nice having you work here...bye. Bob down the hall has consistently gotten us much better results to feed to the FDA for approval. We really don't know how or why he gets better results, but he's definitely the man we want on the job. Sure, maybe twenty years down the road there will be some complaining, but *we didn't know*...*we did all our due dilligence and somehow our results just wound up showing that our drug was okay*.
And even the more innocent "conclusive results" become suspect. A pharmaceutical doesn't want "inconclusive results", where "further tests are recommended". They have a bloody lifetime on the product ticking away, and a competition breathing down their neck. They want some scientist to sign off on this thing with a nice firm "Okay" or "Not Okay", or else what are they paying the guy for? He's not here to do ivory tower work -- he's here to serve the company, which is in the business of extracting savings from aging and achy baby boomers and subsidies paid for by their tax-paying children.
What is being said is that a full third of examined clinical trials were essentially horseshit. This is really not a laughing matter.
Any program relying on (nontrivial) preemptive multithreading will be buggy.
However, statistics are not determinative. This is a mistake I've heard from both laymen and experts. The fact that, according to what's known (and factored in to the calculation) an event is 99.999999999% likely to happen... well, that doesn't mean it will happen. Really, it doesn't matter how unlikely your statistics demonstrate something to be, it won't prevent the unlikely from happening.
In fact, it's demonstrable from statistical analysis that we should expect tremendously improbable events to happen quite often, and that the chances of the most probable outcomes to occur at every instance is an incredibly unlikely outcome as time stretches on.
So statistics are an interpretive tool, not an answer. Statistics alone cannot tell you what will happen, they can't tell you what has happened, and they certainly cannot tell you what should happen. And all these comments you're talking about, I think they come from a valid frustration borne from sloppy reporting telling us "scientists have discovered that 75% of" this and "they now know that 25% of" that outside of any meaningful context.
And what's the likelihood that all these percentages are correct? What's the margin of error, and what's the margin of error's margin of error? Certainly the people telling us these "facts" (reporters) have no idea.
"...innumerable global warming studies that the scientific community can't make up its mind on (for example)." - Bad example, climate scientists "know" the planet is warming and also why it is warming, but fossil fuel politics creates an enormous amount of FUD in an attempt to make you and me think the scientists are contradicting each other and basically haven't got a clue. A similar thing occured when medical scientists said tabacoo was bad for your health. Incredibly some of the same "researchers" who "proved" tabacco was harmless have also been involved in "proving" climate scientists are wrong.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
From what I gather, this statistic applies to scientific studies. Am I surprised that this happens? Not very. Most people believe that scientists are impartial observers of natural phenomena that follow the evidence where it leads. This is how Science is supposed to work, but it doesn't work like this all the time.
Why do scientists jeopardize there careers by falsifying evidence and studies? Because they are human. Some do this unintentionally by subconsciously ignoring evidence that is contrary to their preconceived hypothesis. Others do this intentionally. They do this because if their hypothesis is found wanting; they will lose prestige and/or funding. Or sometimes the evidence points to concepts that they are unable or unwilling to understand or acknowledge as true. Every scientist has preconceived beliefs on what they will discover in a given study; the good ones do not let these beliefs get in the way of the evidence.
History is littered with studies and discoveries that were later shown to be hoaxes. The Piltdown Man is a very famous one; the supposed early human fossil was created to promote a certain view of human evolution. Another recent hoax was the Chinese "feathered dinosaur" fossils that were heavily promoted by the National Geographic to be ancestral to birds (circa 1996). In fact, National Geographic created an evolutionary timeline based on these fossils and presented it as fact. Not only were these supposed fossils later exposed as elaborate hoaxes, it was revealed that National Geographic had a major lapses in scientific and journalistic ethics. First off, the fossils were stolen and smuggled out of China; secondly they were not verified as being authentic. It was shown that National Geographic knew this and still published the story. Why? Because National Geographic is a big proponent of the theropods to birds view of bird evolution, so any evidence that fit this preconceived belief was accepted and not questioned. If on the other hand the evidence had supported the tree-reptiles to birds view, the fossils would have been very critically examined and rejected.
Of course this is not limited just to archeology; many other branches of science suffer from the same problem. But I can somewhat sympathize with these researchers. Imagine arguing your whole life that your hypothesis is correct, all the evidence for the past 50 years points to your conclusion, and then suddenly new evidence pops up that invalidates your life's work. Would you not be tempted to suppress the evidence? Like I said, scientists are human just like us, and while I do not condone suppressing contrary evidence, I can understand why someone would do just that. On the upside, history shows that evidence cannot be suppressed indefinitely, it will always surface. It may take several decades, as was the case with the plate tectonics theory, but evidence cannot be silenced forever.
a quick study shows that one third of all slashdot headlines are nonsense.
And another third is inaccurate or overblown.
A: 3.215 ± 3%
. . . 95% of the time.
Free Software: Like love, it grows best when given away.
Presumably the expert was alluding to the fact that an incident such as this, so soon after WTC, could take on a greater significance than had it been looked at in isolation.
It could lead to an escalation in hysteria, racial hatred and so forth from otherwise-rational people who believe the end of the World is nigh, or something.
Case in point: Following the London attacks there was a bomb scare in Birmingham (about 120 miles North) which resulted in 20,000+ people being evacuated from the city centre. It turned out to be a firework. Taken in isolation I'm sure the response to something like this would be a lot more measured, but coming so soon after the events in London everyone (including the Police) reacted in a typical knee-jerk fashion.
I guess my point is that there is obviously never "a good time" for a plane to crash into a neighbourhood, but there is certainly a less dramatic time for it to happen.
I'm curious to know how Mr. Gribbin defines a religious nut.
While there are a large number of people who reject facts and reason due to their a priori commitment to a religious beliefs, there are a great number who do the same whose religion is science itself.
That is to say, preconceived notions and personal bias prevent many so-called scientists from acknowledging facts and realizing that their pet theories are baseless.
As an example, I offer Carl Sagan. Here was a man who made a nice living talking about extraterrestrial life. Is there ANY evidence of extraterrestrial life? Is there ANY science that supports it? After all, the best that the SETI institute has is Drake's equation which at best merely multiplies speculation upon speculation.
Is SETI science? Perhaps, but Sagan's beliefs and public discussions were based on fantasy and hope rather than fact.
My point is this: Bias appears in religion and in the name of science. Science has dirty hands, too.
Remember, power tends to corrupt, regardless of world view. I'd be willing to bet that a similar book could be written demonstrating horrible abuses of human rights where science was allowed to 'progress' unchecked by morality.
Finally, it is important to note that much of science has been advanced by people with strong religious convictions. Pascal, Pasteur, Lister, Knuth, Kelvin, Joule, Carver, Bacon, Boyle, and many many others. Strong religious conviction is NOT the antithesis of scientific advancement, as demonstrated by the legacy of those I listed above and I could list many more.
Respectfully,
Anomaly
But Herr Heisenberg, how does the electron know when I'm looking?
CNN is great, my favourite is there comment about the terrorists having "veteran suicide bombers".
If you're a suicide bomber long enough to become a veteran I think there's a good chance you're doing something wrong.