Slashdot Mirror
Scientist Says Most Scientific Papers Are Wrong
An anonymous reader writes "According to epidemiologist John Ioannidis, the majority of published scientific papers are wrong. If Ioannidis's own paper is right, a randomly chosen scientific paper has less than a 50% chance of being true. He also says that many papers may only be accurate measures of the prevailing bias among scientists. However, a senior editor of a scientific journal says that scientists are already aware of this: 'When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about.'"
Wow! Science can be wrong.
That is how the system works.
But just because these two scientists were wrong about the precise mechanics of evolution doesn't mean that they were wrong about how the data should be interpreted. The data shows that life has progressed to meet the demands of its environment. Survival of the fittest is correct, but there is no straight-line progression of lifeforms leading one from another as was supposed when these authors first penned their ideas.
Scientific ideas may come and go, but the data set just gets larger. That is why this guy can claim the others are wrong: he has a better data set.
"Rocky Rococo, at your cervix!"
Gee, i didn't know most of "IEEE transactions on Image Processing", "Journal of Algorithms" or "IEEE Transactions on Pattern Analysis and Machine Intelligence" were probably wrong.
Please be more specific next time. Thank you.
I concur, this has been pounded into my head since I started taking higher (college) level science classes. I think this sort of thing should be taught at an earlier age, that science is a methodology of trying to determine the root cause of an event or thing. No one has (at least any sane person) ever claimed that science is always right. The only thing science is trying to do is find the best possible answer to a given situation, and learn, to the best of our ability, and provide a sutable and resonable answer to that question. All in the name of giving ourselvs a base to work off of and further our knowlege on a given topic.
Whether anything anyone says is right or wrong, it's a matter of opinion first and foremost.
No, it's not.
Our biology does not provide us
Our biology provides us with excellent truth detectors: throughout most of primate evolution, if you were wrong about whether your food was poisonous or whether there was a lion hiding in the bushes, you didn't get to pass on your genes. You didn't get to debate social relativism with the lion before he made a tasty meal out of you.
Most of science is still ultimately about matters like that, matters that have good answers, at least in principle.
Some science has veered off course, however. Every major scientific discipline (physics, biology, chemistry, etc.) has subareas where people start conflating experimental facts with opinion, aesthetics, and prejudice.
So, scientific truth is not a matter of opinion, but a lot of what is published in science is not about scientific truth.
God's irrelevant to whether ID is true or not --- it's whether ID is falsifiable or not that's important.
Which, AFAICT, it isn't, so it's still not science. But let's at least be precise when slagging them off...
John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.
... you have to be careful.
OK, I'm going to go through these one by one.
First, small sample size is a problem. That's why you have error bars on your graphs - in fact, if you don't see the error bars, check the tables to see if the t size is big enough - many studies start with thousands of inputs to get only a handful of outputs - in biochemistry, you can have more than 10,000 PCRs of something made, only to result in 10-40 final structures in crystallography at the other end of the pipeline.
The study we're on is unusual in that it actually has sufficient numbers that the t sizes are big enough to ask many questions - but most have such small numbers that they could easily be wrong.
2. Poor study design - again, how you ask the question is important, as well as the conditions - so this may be true. I always check the holes in the logic as well as the basic logic - because those holes can lead to incorrect conclusions - and many popularized science articles don't bother checking for the holes in the logic. They do a quick summary saying "breast cancer is caused by too much salt in the diet" when the study really said "there is a high correlation among middle-aged women having first onset breast cancer if their diets are in the top range of salt intake" - but that could also mean they live in conditions where the high salt intake could be due to the other things in their environment that caused the breast cancer in the first place.
For example, you could say Romans got lead poisoning because they lived in cities, when it was actually the use of lead in their pipes, not the living in cities - although we don't know, as perhaps cities had lead particulates in food from airborne fallout from factories or burning certain things in their candles
3. Researcher bias - ok. Not going to argue that.
4. Selective reporting - see 2 for how this occurs.
But that doesn't mean a good high-quality peer reviewed scientific paper in a respected and well-juried paper is "inaccurate". There are a lot of journals out there, and different standards and quality levels.
-- Tigger warning: This post may contain tiggers! --
It's not just psychology that has so much publication list padding. During my PhD I was asked to do a detailed review of about ten papers in my area (process scheduling algorithms). They were all peer-reviewed and published in reputable journals by well-known researchers. IIRC:
- About half had very little novel content. Maybe one equation changed, a few different examples added
- Two or three had basic mathematical errors
- About half omitted details that were required to easily replicate their results or actually use their methods. I spent weeks piecing together what the authors meant from various clues scattered across appendices, tables and figures.
- Several had gaping holes in the method that were apparent to me, a first year PhD with no experience.
- All of them cherry-picked examples to show their methods in the best light, completely omitting any bad results.
The article is about "Published Research Findings". It doesn't specify that all the papers analyzed were from peer reviewed journals. There are a lot of non peer reviewed journals out there. Usually you only publish in those if you have a short paper, or one that's not extremely novel, or just not of great general interest. Many times researchers will publish in those journals when they can't get the paper published in a peer reviewed journal. I'm sure the percentage of false findings in those journals is much higher, and may have altered the ratio of found false papers significantly.
The paper stating that ~50% of scientific papers are false is published in the Public Library of Science (PLOS) Medicine. The paper only examined medical studies and not scientific papers on physics, chemistry, engineering science, (and mathematics).
While molecular biology papers can be prone to statistically insignicant, but factually stated conclusions, the biggest culprits are clinical studies and 'large-scale' analyses of data.
Good experiments are constructed to give a 'yes' or 'no' answer based on the presence or absence of evidence. The zeal of high-throughput studies and analysis have put more pressure on good statistical analysis. Unfortunately, statistical analysis requires math...which sometimes eludes doctors and biologists. Hence, the problem of missuing statistics and stating inadequately supported conclusions.
-Howard
Favorite
Stalin believed that Darwinian evolution was just a bouguoise concept. He believed in Lamarckian evolution and directed his agricultural ministry to ignore studies that supported Darwinain evolution. Their agricultural industry suffered and people went hungry in the process.
Ah yes, Lysenkoism. Science and ideology do not mix well. Although, to be fair to Stalin, the people went hungry more because of forced collectivization than because of Lysenko, although the pseudo-science didn't help matters any. Ideology shouldn't trump science, social or agricultural.
I have still to RTFP (put it on my list), but I did notice that this is an essay, and not a research article. As such, it probably did not go to peer review as it is more of a discussion piece.
Reality or nothing.
http://medicine.plosjournals.org/perlserv/?request =slideshow&type=table&doi=10.1371/journal.pmed.002 0124&id=4104this table seems to be the most interesting part of it all, showing what effort should be done to get a PPV (positive predictive value) above 50%. This is specifically aimed at clinical studies, BTW, people with anti-evolutionist feelings have nothing to see here ;)
Furthermore, as an essay, it might or might not be peer reviewed, didn't go into that. The study itself is probably not as crappy as you might think after reading the New Scientist link, because, as parent makes clear, it provides a modeling approach to assess articles in this field.
molmod.com - computing tips from a molecular modeling