Slashdot Mirror
Scientist Says Most Scientific Papers Are Wrong
An anonymous reader writes "According to epidemiologist John Ioannidis, the majority of published scientific papers are wrong. If Ioannidis's own paper is right, a randomly chosen scientific paper has less than a 50% chance of being true. He also says that many papers may only be accurate measures of the prevailing bias among scientists. However, a senior editor of a scientific journal says that scientists are already aware of this: 'When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about.'"
Great... watch the Creationist/Intelligent Design kooks run with this.
Trolling is a art,
Wow! Science can be wrong.
That is how the system works.
But just because these two scientists were wrong about the precise mechanics of evolution doesn't mean that they were wrong about how the data should be interpreted. The data shows that life has progressed to meet the demands of its environment. Survival of the fittest is correct, but there is no straight-line progression of lifeforms leading one from another as was supposed when these authors first penned their ideas.
Scientific ideas may come and go, but the data set just gets larger. That is why this guy can claim the others are wrong: he has a better data set.
"Rocky Rococo, at your cervix!"
Their is a 50% chance that that's not true.
"Science" is NOT the same as "fact" or "truth". It is a METHOD -- a PROCEDURE one follows in an attempt explain some event or phenomenon. It should hardly be surprising that "Scientific" papers are mostly wrong. There may be only one "right" or "correct" theory for a given phenomenon -- but there are countless wrong ones.
Wow, I thought Study Shows One Third of All Studies Are Nonsense is bad enough, who knows scientific papers are worse!
I patiently await the next article: "Research Shows Three-Quarters of All Researches Are Bullshit".
Rock that crushes, Paper & Scissors that don't matter.
I have found this to be sadly true. My coworkers are college professors who often publish papers on social trends using large datasets obtained from government records. I am frequently pointing out errors in their analysis (mainly that they simply don't look at their data, such as just because Jane Doe and John Doe have the SSN they are assumed to be the same person) but I'm generally ignored or told to fix it myself though that isn't my job. They are more interested in getting something published and don't want to have to retract something.
What I do see as harmful is the attitude towards bad papers. To many academics try to accumulate more and more published papers the same way that slash-dotters try to build up karma. I understand that having papers published can reflect well on someone, but we need more accountability. Journals need to create a more strict system for reviewing papers that are to be published to weed out more of the crap plain and simple. If the evidence does not reflect the claims throw it away. If the research was conducted on a population that was too small or specific for a grand generalized claim about the topic as a whole, throw it out.
I understand that you will always have people just trying to throw their names around, but this needs to be looked at from the grander perspective.
"When I read the literature, I'm not reading it to find proof like a textbook.
Sure there are probably many scientists that think of it this way. But the problem is that bad research (or a bad paper) rarely dies after being published. They are often cited as evidence for years to come in other papers until enough evidence to the contrary comes out to raise questions. Plus, you have crazy professors giving this bad research for their classes to read, and often they don't explain to their classes where research is possibly flawed--so we find ourselves training generations of new scientific minds that run around spouting out bad research. I understand that we all need to take research with a grain of salt when we read it, however bad scientists trying to become famous with their bad ideas or bad papers can be very detrimental to any field.
Comment removed based on user account deletion
But seriously, reviewers are biased and sloppy, as are the editors. The fact that reviews are blind means that they are also unaccountable, which fosters even more bias.
Journals take months or years to respond to a submision, and often as not they respond with a rejection so the submitter has to give up or start the whole process over with another journal. There are so many scandals that one could quote. The whole process seems more designed to support the status quo than to promote knowledge.
I have discussed this with many people in academia and they react not with logic, but with horror that I would dare to question a system that they view almost mystical reverence.
Test 1 2 3 4
If a submitted paper is scientifically unsound, it should be rejected.
If a scientific paper is useless to the readership, that publication should reject it and recommend a different journal.
If a paper is wrong and the reviewers KNOW IT then they should send it back for corrections.
If it's WRONG but the reviewers don't or more typically can't know it because it is novel, then publish it. The rightness or wrongness will be sorted out soon enough.
Ever heard of Isaac Newton? Turns out his theories were incomplete in some very fundamental ways, but his theories regarding the motion of objects were the best approximations we had for hundreds of years and are still very useful for macroscopic objects traveling way below c.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
"If we knew what we were doing, it wouldn't be research"
That's what my supervisor used to say to me when I got depressed about lack of progress.
Whether anything anyone says is right or wrong, it's a matter of opinion first and foremost.
No, it's not.
Our biology does not provide us
Our biology provides us with excellent truth detectors: throughout most of primate evolution, if you were wrong about whether your food was poisonous or whether there was a lion hiding in the bushes, you didn't get to pass on your genes. You didn't get to debate social relativism with the lion before he made a tasty meal out of you.
Most of science is still ultimately about matters like that, matters that have good answers, at least in principle.
Some science has veered off course, however. Every major scientific discipline (physics, biology, chemistry, etc.) has subareas where people start conflating experimental facts with opinion, aesthetics, and prejudice.
So, scientific truth is not a matter of opinion, but a lot of what is published in science is not about scientific truth.
> Does this mean that peer review fails as a method to filter out time-wasting, tree-killing dreck?
Peer review isn't a certification of correctness. It's just supposed to filter out the papers where the authors didn't do their homework. It can spot bad logic, use of outdated data, failure to consult important papers in the field, etc. But it can't tell us whether string theory is correct or not.
Sheesh, evil *and* a jerk. -- Jade
Therefore anything that anyone says is simply an opinion.
That's just your opinion.
Who ordered that?
I didn't read the article but I don't believe the conclusions of the summary. Maybe in epidemiology it is true but not in physics where usually the results are reproducible and I very rarely find papers that are just wrong. I might agree that most of the papers are not 100% right (small mistakes in formulas happen quite frequently) but it does not impair the usability of a paper.
However, peer review does not solve all the problems. Most of the research takes a lot of time and effort and referees just read the papers. They do not reproduce the experiments or calculations. So peer review can weed out only obviously bad papers but not papers that looks OK but are wrong.
Save the bandwidth. Don't use sigs!
If 50% are wrong, then 50% are right. So if I write a scientific paper, the chance of it being right is 1/2. And if I write the same paper, say, 8 times, the chance of it being right at least once is 255/256.
I think I'll write that paper on statistics.
The article is about "Published Research Findings". It doesn't specify that all the papers analyzed were from peer reviewed journals. There are a lot of non peer reviewed journals out there. Usually you only publish in those if you have a short paper, or one that's not extremely novel, or just not of great general interest. Many times researchers will publish in those journals when they can't get the paper published in a peer reviewed journal. I'm sure the percentage of false findings in those journals is much higher, and may have altered the ratio of found false papers significantly.
Science progresses when well thought out hypothothies based on a good data are replaced by more inciteful reasoning based on more complete data. Lamarck wasn't guilty of faulty reasoning. He just didn't have a complete enough data set.
But the article at hand, isn't talking about that kind of "wrong". He is talking about conclusions that can not be supported by the data presented. Either the reasoning is faulty or the data collection methods are so faulty that no meaningful conclusions can be drawn.
When a theory is proven wrong in the scientific sense, it is a good thing. We learn something new and that be the basis for further developments. But if a theory is proven "wrong" in the mechanical sense, we have no new insights, just a relief from further time wasting.
Well yes, for the scientific community it's common knowledge (at least IMHO) that these papers are HARD to prove wrong, most are assumed true, but then again, what I learned most about image processing was in these papers. They do contain very valuable information, and a lot of these are works based upon previous works. (This is how science is done right now).
Even when some of these papers could be wrong in their conclusions, or maybe one or two algorithm flaws, but it was papers like these (image processing, etc) that contributed to technology used today, like MPEG4 video.
My point is, unlike these which are done with scientific methodology, in *medical* "research papers" there's oh so much money at stake. I'm sure the article could have said "most medicine research papers are wrong", and I would have believed that.
But science is much more than medicine, and as a scientist, I find it an insult to stain the name of Science because of commercial vias in medical research.
Curiously, I googled for "bias in medical research" (with quotes) and here's the top result, of 426 search results:
Bias in Medical Research by Maria Spicer.
In contrast, googling for "bias in image processing research" (with quotes) yielded no results.
Of course, google is only a very statistical method for finding out whether something exists or not, but I think you get my point.
But what our scientist does not realize is that every time he makes a measurement, the Flying Spaghetti Monster is there changing the results with His Noodly Appendage.
Which explains why most scientific papers are wrong.
Who ordered that?
Did anyone actually RTFP? It's one of the most spurious pieces of "research" I've ever read. And with a biophysics degree, I have read quite a few. The author actually didn't investigate any actual papers, but he builds a mathematical model out of his own biases, statistical projections, and some back of the envelope computation. Even then, his conclusions are much less stringent than the submitter makes them out to be. He "proves" that under all his assumptions, half the research papers *might* be wrong, but shows not even statistical evidence that they are.
I think PLoS is peer reviewed, but that paper should never have survived peer review. Occasionally, bad papers slip through, even in the so called hard sciences. This one seems to be one of them. Since PLoS Medicine is pretty well respected for an open access publication, lets assume that this was a lark and more on.
But it makes me curious what the fraction of bad papers looks like in an open access publication like PloS versus a traditional journal like, say, Nature, The Lancet, or New England Journal of Medicine. One reservation people have about open access (or author pays) models was that since PLoS gets paid about $1500 from they authors, they might be accepting vanity papers, or don't triage as well as traditional journals. I don't think they are, but if this paper is any indication, PLoS might take a second look at their peer review process.
When you think about it, that's positively astounding. There are vastly more ways to be wrong than to be right. We've managed to get 50% right answers out of the myriad wrong answers. Pretty impressive!
It would be better still if it was more than 50%, but we can just apply the process repeatedly to push up our confidence (50%, 75%, 87.5%, etc.). A little more attention to statistics would help us raise the base rate above 50%.
the level of religous dogma in some camps.
The only question is, who decides which science is wrong? I doubt very seriously any big money areas will have a published high rate of error. After the high money science the next protected type would be whatever is en vogue for the time.
Scientific integrity took a big dive in the late 80s as special interest groups suddenly realized that marketing, confusion, and intimidation were far better at advancing agenda than honest science.
* Winners compare their achievements to their goals, losers compare theirs to that of others.
Skeptic had their take on it in the last issue. In a nutshell
"Seven Deadly Sins? I thought it was a to-do list!"
If science can be wrong, then why trust it?
It is the only objective process for assessing facts from fiction.
In other words, if the best a scientist can tell you today is that, he might be wrong tomorrow, why even bother listening to him?
No one is forcing you to listen. You ignore the information provided by science at your peril.
So you can use science for real things, like physics and design of military weapons and consumer goods, but the rest of it is so much speculative nonsense.
Quantum physics is speculative, but you don't seem to be throwing your computer out the window.
The consequences of guessing wrong about the origin of humanity are completely immaterial to most people's lives.
Dead wrong.
Stalin believed that Darwinian evolution was just a bouguoise concept. He believed in Lamarckian evolution and directed his agricultural ministry to ignore studies that supported Darwinain evolution. Their agricultural industry suffered and people went hungry in the process.
You can't show people evolving any more than someone else can show God making something.
I can show a progression of hominid fossils leading to homo sapien sapien. The Bible is silent about these fossils.
It's immaterial, unprovable, and so why fight over it?
It may be immaterial to you, but the theory is consistent with the evidence we possess. You may not choose to believe it, but that the only thing immaterial about this discussion.
Yeah you can roll out the eliptical argument that evolution is somehow necessary for medicine but most doctors are concerned with the human species, here and now, and now plants and people are related.
Why bother? You obviously believe that the scientific method works differently for investigations related to the origin of humanity than it does when applied to chemistry.
To wit, you can get a Chem E degree and still get into Med School.
You are correct.
Just don't whine to me when you have difficulty making sense of the data you gather without using evolutionary theory.
You will never amount to anything more than a glorified technician.
I can live with that.
"Rocky Rococo, at your cervix!"
If science can be wrong, then why trust it?
And to think you posted that with a device that is arguably high technology. Gee. It's a good thing those practical thinkers at Signetics and Intel didn't listen to those shifty eyed physicists.....
So you can use science for real things, like physics and design of military weapons and consumer goods, but the rest of it is so much speculative nonsense.
Newsflash. The same people that don't like evolution only like physics when it can be used to attack evolution. The rest of the time it gives rise to uncomfortable facts like the Earth being round and the Universe being billions of years old. They'll get around to rest of the so-called "useful" sciences once those pesky life sciences have been properly re-aligned. I'm also glad people like you don't decide what is "useful" in science. After a demonstration of electrical phenomena, the Queen of England asked Micheal Faraday what of what possible use was all this nattering about electricity. He replied, "Of what use is a newborn babe." Sheesh.
Science is all about being wrong. 99% of it is long painful slogging through mucky fields of sheer wrong and trivialities to find the occaisional nugget of right. I'm mildly amazed that Scientists Can Be Wrong is a subject of discussion. This is only a problem when people who don't have any idea how science works expect scientists to be some sort of infallible priesthood. It also doesn't help when the press seizes on new research that hasn't endured years of attacks and splashes it all over the place. It is as though Firebird 0.3 is headlined as the New Killer App. The press is the worst offender in this regard.
You can't show people evolving any more than someone else can show God making something. It's immaterial, unprovable, and so why fight over it?
You can show things that reproduce really fast evolving. It is quite easy with microorganisms and it isn't too awful bad with insects like fruit flies. It's a bit harder with some fast reproducing plants and an absolute pisser with anything that takes more than a week or two to reproduce. One can still do things like genome tracing and compare and contrast with currently living things that haven't changed in a long time. It is hard to show people evolving. It isn't all that hard to show the effects of evolution on people. Unless of course you live in the US.......
There's a famous quote by Box: "All models are wrong; some models are useful". That's what science is all about -- making models, which are useful until a better model comes along. So by definition, 100% of *all* scientific papers are wrong. But some are wrong in useful ways that inspire new generations of scientists to improve upon them.
Indeed it is. Which is precisely why our senses are so easily fooled: given stimuli that do not correspond to those seen in survival tasks in the EEA (Environment of Evolutionary Adaptedness, i.e. hunting and gathering on the African plains or whatever), our brains do not necessarily respond correctly.
Now explain why a creator would have built brains that are so subject to misdirection, geometric optical illusions, etc. Why would he/she/it have done so?
belief in evolution is self-defeating because on the supposition that evolution is responsible for our reasoning ability, we have no confidence that the deliverances of reason (i.e. the theory of evolution) correspond to reality.
And if a creator built our reasoning capabilities, how do you know that he/she/it programmed it to accurately reflect reality? We'd be seeing whatever he/she/it wanted us to see, for whatever reasons. You'll get less mileage out of this argument for creation than for evolution, even.
Pointing out that evolutionary theory itself can't guarantee the accuracy of our reasoning faculties (which is true) gets you absolutely nowhere because Creation mythology is significantly worse. Consider:
Prove the above statement wrong. You certainly can't invoke anything it says in the bible, because that -- or rather your memory of it -- was created six seconds ago as well. It says exactly what the creator wanted it to say, for reasons of his/her/its own.
As soon as you invoke a creator, falsifiability is utterly gone, your conclusions can be ANYTHING, and future argumentation is pretty much futile. Thus creation mythology serves primarily as a tool for a person to project their own emotional needs and desires into their own understanding of reality.
Fortunately, there are other ways to evaluate the accuracy of our reasoning capabilities than evolutionary theory or creation mythology. Sparing a couple thousand years of philosophy, I'll stick to the pragmatic argument: Reason seems to work. It gives us effective tools for functioning, ergo we're best off assuming that our intellect and reason is what it seems to be, and make use of it.
End note: Your sig links to a story about Antony Flew "converting to religion". You'll notice he's a self-described Deist: a philosophy that is in no way contradictory to any contemporary understanding of evolution, physics, or any other branch of the sciences. He explicitly states he doesn't believe any sort of revealed religion. How does this bolster any point in favor of creationism or any other branch of post-Enlightenment fundamentalist thought? The point is lost, because Flew explicitly still rejects all that.
I stole this sig from someone cleverer than me.
Universities have failed a lot of scientists in that a) those papers are the result of stupid tenure policies and b) universities often do little to promote their researchers.
Engineers often read papers to solve problems. When they know about them! (Google Scholar might fix this)
A worse problem than them often being wrong is that:
a) there is frequently no way to determine if a given paper is accurate, has mistakes, is partially accurate, is laughable, was accurate at one point but is outdated, etc etc. At least from an outsider's perspective.
b) there is no good way to stay abreast of current interesting developments - hell, there's no way to see interesting things from 20 years ago easily! Again, this is from an interested outsider's perspective.
Once or twice a year I have the luxury of spending a week or two in an engineering library for the express purpose of finding out new and interesting things in my field. I'm SHOCKED at the amount of material that is being duplicated (often badly) in industry, material that is inaccurate or poor quality, and VERY GOOD material that never sees the light of day again.
..don't panic
"That statement makes no sense whatsoever. So a nation has to believe in evolution to feed its people? For lack of better terminology at the time, that's stupid."
Except the grandparent didn't say that. He said that that Stalin couldn't feed his people because he directed his ministry to ignore studies that supported Darwinain evolution, which meant his country used inferior methods. The grandparent did not in any way conclude that a nation has to believe in evolution to feed its people. He merely pointed out that researching the origins of humans has valuable and real benefits beyond knowing the truth, and that to ignore such information can be detrimental to society.
This guy are sick.