Slashdot Mirror
Why Most Published Research Findings Are False
Hugh Pickens writes "Researchers have found that the winner's curse may apply to the publication of scientific papers and that incorrect findings are more likely to end up in print than correct findings. Dr John Ioannidis bases his argument about incorrect research partly on a study of 49 papers on the effectiveness of medical interventions published in leading journals that had been cited by more than 1,000 other scientists, and his finding that, within only a few years, almost a third of the papers had been refuted by other studies. Ioannidis argues that scientific research is so difficult — the sample sizes must be big and the analysis rigorous — that most research may end up being wrong, and the 'hotter' the field, the greater the competition is, and the more likely that published research in top journals could be wrong. Another study earlier this year found that among the studies submitted to the FDA about the effectiveness of antidepressants, almost all of those with positive results were published, whereas very few of those with negative results saw print, although negative results are potentially just as informative as positive (if less exciting)."
It isn't so much a problem of peer review. Peer review has limitations of course, to thoroughly review an article, one would have to repeat the experiment, which most reviewers (for good reasons) do not do. Peer review is good for what it does, give feedback to the authors of the paper, and as you said, it does have a filtering effect.
The other issue is, a lot of papers aren't really worth much at all. Nature might get their share of interesting articles, but in smaller journals, a lot of research ends up being something like, "I had this idea, and I did a small little experiment to see if it was worth anything. Maybe it is." But of course, with a small little experiment, your chances of being wrong are greater: it's just an entry-point for someone else to maybe continue research in an interesting direction. And I have done peer review, FWIW (and if you trust random guys you meet on the internet).
Qxe4
A handful of glaciers are indeed growing. The vast majority are shrinking, and they are shrinking much more than the handful of anomalous ones are growing.
A handful of unusual data points in a complex system does not prove a trend. It's as if you were to argue, "Scientists *say* that cigarette smoking will damage your health. But I know one guy who smoked and lived to a ripe old age. Therefore, these `scientific' findings are clearly the result of some politically-motivated anti-tobacco conspiracy."
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood
Peer review doesn't always help. I've studied papers that have the most detailed, thoroughly tested research but end up relegated to some obscure journal because the people peer reviewing the topic don't agree with it. In one case, the pioneers of the field of siRNA lambasted a study which showed that short RNAs can enhance transcription, as well as negatively regulate it. Because this was so far outside of their model for how siRNAs work, they dismissed the work as nonsense, despite the paper showing five replicates of every experiment, and practically putting their entire work, step by step, into the supplementary materials. Papers get into good journals with far less than what I saw for this one, but peer review condemned it to obscurity. I think peer review works, but only if the people reviewing keep an open mind and don't get piqued if the findings disagree with their own views.
Amenacier
No, you are exactly right. His paper was really only intended for the field of population genetics and genetic epidemiology (his fields), where people have been using the standard p0.05 statistical cutoff as their metric for whether a given analysis is significant. So if you have 20 research groups analyze the same question (like is a mutation in gene X responsible for disease Y), according to that methodology by definition 1 of the 20 researchers will find a statistically significant result simply due to chance alone. This is old news and journals stopped accepting papers that *only* had that statistical analysis about 3-5 years ago. Almost without exception, they now require you to show some kind of biological verification (show mutant protein X actually is defective and has reduced activity) OR you can do a replication in a completely independent sample, which is unlikely (again 1/20) to be significant by chance. Unfortunately people have misinterpreted his paper and are applying his point to other fields like chemistry, astrophysics, or even areas of biology where it doesn't apply.
I read a bunch of dental papers recently, and discovered something rather disturbing. A good 90% or more of studies for dental procedures do NOT use any control group. They all say, "we did X and got the expected result." There is no checking whether the procedure is better than other procedures or even doing nothing at all.
Something to think about next time someone you know is told they need wisdom teeth extracted or some orthodontic appliance.