Why Most Published Research Findings Are False

Hugh Pickens writes "Researchers have found that the winner's curse may apply to the publication of scientific papers and that incorrect findings are more likely to end up in print than correct findings. Dr John Ioannidis bases his argument about incorrect research partly on a study of 49 papers on the effectiveness of medical interventions published in leading journals that had been cited by more than 1,000 other scientists, and his finding that, within only a few years, almost a third of the papers had been refuted by other studies. Ioannidis argues that scientific research is so difficult — the sample sizes must be big and the analysis rigorous — that most research may end up being wrong, and the 'hotter' the field, the greater the competition is, and the more likely that published research in top journals could be wrong. Another study earlier this year found that among the studies submitted to the FDA about the effectiveness of antidepressants, almost all of those with positive results were published, whereas very few of those with negative results saw print, although negative results are potentially just as informative as positive (if less exciting)."

Comments

Re:How universal is this.

[Mr. Slippery]

How about the recent stories which prove that glaciers in the north have been *growing*?

A handful of glaciers are indeed growing. The vast majority are shrinking, and they are shrinking much more than the handful of anomalous ones are growing.

A handful of unusual data points in a complex system does not prove a trend. It's as if you were to argue, "Scientists *say* that cigarette smoking will damage your health. But I know one guy who smoked and lived to a ripe old age. Therefore, these `scientific' findings are clearly the result of some politically-motivated anti-tobacco conspiracy."

Re:Peer review helps

[Amenacier]

Peer review doesn't always help. I've studied papers that have the most detailed, thoroughly tested research but end up relegated to some obscure journal because the people peer reviewing the topic don't agree with it. In one case, the pioneers of the field of siRNA lambasted a study which showed that short RNAs can enhance transcription, as well as negatively regulate it. Because this was so far outside of their model for how siRNAs work, they dismissed the work as nonsense, despite the paper showing five replicates of every experiment, and practically putting their entire work, step by step, into the supplementary materials. Papers get into good journals with far less than what I saw for this one, but peer review condemned it to obscurity. I think peer review works, but only if the people reviewing keep an open mind and don't get piqued if the findings disagree with their own views.