Slashdot Mirror
Why Most Published Research Findings Are False
Hugh Pickens writes "Researchers have found that the winner's curse may apply to the publication of scientific papers and that incorrect findings are more likely to end up in print than correct findings. Dr John Ioannidis bases his argument about incorrect research partly on a study of 49 papers on the effectiveness of medical interventions published in leading journals that had been cited by more than 1,000 other scientists, and his finding that, within only a few years, almost a third of the papers had been refuted by other studies. Ioannidis argues that scientific research is so difficult — the sample sizes must be big and the analysis rigorous — that most research may end up being wrong, and the 'hotter' the field, the greater the competition is, and the more likely that published research in top journals could be wrong. Another study earlier this year found that among the studies submitted to the FDA about the effectiveness of antidepressants, almost all of those with positive results were published, whereas very few of those with negative results saw print, although negative results are potentially just as informative as positive (if less exciting)."
Peer review no doubt helps to limit people who intentionally want to cause problems. Sokal's bullshit paper on quantum gravity (see The Sokal Hoax ) made it into print only through a non-peer-reviewed journal. While it is disturbing to think much published scholarship is unreliable, at least it isn't necessarily malicious.
How long until some researcher releases a study showing that Dr. Ioannidis' research findings are themselves wrong?
Prevent Windows piracy. Use Linux instead.
Title is wrong. It says that the FDA is corrupt. And that published papers take around 3years to get peer reviewed where the bad ones are removed. What a blatent attack on science generally. Sure paper publishing needs to be reviewed but 'most published research is false' is an outright LIE. 'Most published research' includes all of our basis of scientific knowledge. If most of our theories on biology were wrong really we realistically wouldnt have been able to move forwards into working with genes if we didnt know what a cell did.
At the risk of being modded down to oblivion, I am still curious to how this effects popular theories like global warming. We already has people claiming that the science is wrong and they are generally mocked and ignored because their works are published in major journals. Well, this story seems to indicate that publishing those claims will give them a larger change of it being incorrect.
Anyways, it seems that if you don't tow the line on climate change, there is no room for you anywhere. So where does this leave the accuracy of the claims in light of how common it seems that they can be wrong even when published in a respectable scientific journal. I know the IPCC looked at them, but they didn't validate any of the claims, they only looks at whether or not Humans were the cause (that was their charter and they acknowledged this in their reporting).
I would think that "Publish or Perish" must contribute to a lot of crappy papers getting published. Shovel it out the door, somebody else says it's wrong, write another grant for a study to verify that, shovel that one out the door, rinse, lather, repeat...
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
Those responsible for refuting the research of the people who have just been refuted, have been refuted.
Basic idea: high-profile journals want papers that are new and exciting. This means that scientists have an incentive to 1) rush their work, 2) choose fields that are popular, and 3) claim that their papers solve more than they actually do. This leads to sloppy, dishonest papers.
I'm not going to judge this paper - I haven't read it thoroughly - but to pair a title like "Why most published research findings are false" to a pretty well-known problem seems itself like an example of problem 3!
Even after reading the article, I'm still not sure if the authors are saying:
A) Given that research has been published, it is more likely to be false than not; or
B) Given that research is false, it is more likely to be published than is the case for true research.
I mean, it says:
So, (Wrong Articles)/(Total Articles) = >=0.5, right?
But the only figures I can find in the same article are:
So.. "most" is now "less than one third"?
I'm somewhat alarmed that The Economist lets people who don't seem to grasp basic statistics write their articles.
Antiquis temporibus, nati tibi similes in rupibus ventosissimis exponebantur ad necem.
How long until some researcher releases a study showing that Dr. Ioannidis' research findings are themselves wrong?
Who needs a study? Simply reading the article shows that he has fallen precisely into the trap that he is complaining about i.e. overstating his results. He forgets one very simple point: not all science is medicine/biology.
As a particle physicist I would strongly disagree with his conclusions, at least as applied to experimental particle physics. It is certainly true that some papers turn out to be wrong but this is rare and usually ends up as a 'big thing' in the field. Outside my field I'd be very surprised if the majority of physics or even chemistry papers turn out to be wrong (but I certainly not a chemist so this is just my impression).
As for medicine I can certainly see that they have a problem. Afterall how many times have we been told "don't eat X/do Y it is bad for you" only later to find out that actually it isn't half as bad as they thought and may even have benefits? Just because a lot of medical research is often flawed does not mean that all of science has the problem on the same scale.
So, Dr. Ioannidis either show us some data from chemistry, maths and physics or stop complaining that all of science has a problem on this scale. From where I stand your evidence points to a problem with bioscience/medical research only.
In order to test the Global Warming theory you need 2 carbon copies of 1900 earth
That wouldn't work, as the copies would be comprised mostly of carbon powder, so the geophysics would be completely different.
Indeed in my field (a sub-area of computer science) people are usually highly skeptical of any supposedly important new result in the field that was first published in one of the highly prestigious but generalist journals, like Nature or Science. These often end up being, if not outright wrong, at the very least seriously over-extending their claims or the importance of their claims, in a way that would never get them published in a specialized journal filled with an editorial board who were actually experts in the specific area in question.
This is only exacerbated by the fact that, because generalist publications know they don't have expertise in every specialized area on staff, they often ask the authors to suggest potential reviewers of their own papers. Of course, authors are likely to suggest reviewers who they think will like the paper, not the ones who would give it a grilling.
I think the interest of this particular study is not so much that a lot of science turns out to be wrong, but that a lot of the most prestigious publication venues turned out to be wrong more often.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Afterall how many times have we been told "don't eat X/do Y it is bad for you" only later to find out that actually it isn't half as bad as they thought and may even have benefits? Just because a lot of medical research is often flawed does not mean that all of science has the problem on the same scale.
The problem here is that the popular press always report the very latest 'finding' in what is a complex field. Yet we should know that not only in medicine, but in virtually all experimental sciences, a single paper is not sufficient to establish some new profound truth.
Dr Ioannidis' largest problem is that he thinks he has identified a problem. There isn't one. This is how science is supposed to work! We publish methodologies so that the work can be replicated by other teams. Some findings survive futher scrutiny, some don't. The "hotter" the field, the less you are going to rely on the latest single study, no?
So he's found 1/3 of studies were refuted, but later work. Great, they were refuted, what's the problem? And how do we move from that to the conclusion that "most" scientific papers (even outside the hotter fields of bio-medical research) are wrong. And what about looking at outcomes? The advances of medicine even in my lifetime are astounding, this is hardly the result of a system that isn't working!
Better to be despised for too anxious apprehensions, than ruined by too confident a security. --Edmund Burke