Slashdot Mirror
Scientist Says Most Scientific Papers Are Wrong
An anonymous reader writes "According to epidemiologist John Ioannidis, the majority of published scientific papers are wrong. If Ioannidis's own paper is right, a randomly chosen scientific paper has less than a 50% chance of being true. He also says that many papers may only be accurate measures of the prevailing bias among scientists. However, a senior editor of a scientific journal says that scientists are already aware of this: 'When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about.'"
Wow! Science can be wrong.
That is how the system works.
But just because these two scientists were wrong about the precise mechanics of evolution doesn't mean that they were wrong about how the data should be interpreted. The data shows that life has progressed to meet the demands of its environment. Survival of the fittest is correct, but there is no straight-line progression of lifeforms leading one from another as was supposed when these authors first penned their ideas.
Scientific ideas may come and go, but the data set just gets larger. That is why this guy can claim the others are wrong: he has a better data set.
"Rocky Rococo, at your cervix!"
Their is a 50% chance that that's not true.
"Science" is NOT the same as "fact" or "truth". It is a METHOD -- a PROCEDURE one follows in an attempt explain some event or phenomenon. It should hardly be surprising that "Scientific" papers are mostly wrong. There may be only one "right" or "correct" theory for a given phenomenon -- but there are countless wrong ones.
Wow, I thought Study Shows One Third of All Studies Are Nonsense is bad enough, who knows scientific papers are worse!
I patiently await the next article: "Research Shows Three-Quarters of All Researches Are Bullshit".
Rock that crushes, Paper & Scissors that don't matter.
"See? Scientists don't know what they're doing! All your answers are in Teh Bile-Balllllllll! Praise JEEEEEEE-zussssssssss!"
Yes, but unlike religious dogma, scientific theories are meant to be falsifiable.
Unless someone in the ID camp is willing to admit that God is falsifiable, their theory will not be considered science.
"Rocky Rococo, at your cervix!"
What I do see as harmful is the attitude towards bad papers. To many academics try to accumulate more and more published papers the same way that slash-dotters try to build up karma. I understand that having papers published can reflect well on someone, but we need more accountability. Journals need to create a more strict system for reviewing papers that are to be published to weed out more of the crap plain and simple. If the evidence does not reflect the claims throw it away. If the research was conducted on a population that was too small or specific for a grand generalized claim about the topic as a whole, throw it out.
I understand that you will always have people just trying to throw their names around, but this needs to be looked at from the grander perspective.
"When I read the literature, I'm not reading it to find proof like a textbook.
Sure there are probably many scientists that think of it this way. But the problem is that bad research (or a bad paper) rarely dies after being published. They are often cited as evidence for years to come in other papers until enough evidence to the contrary comes out to raise questions. Plus, you have crazy professors giving this bad research for their classes to read, and often they don't explain to their classes where research is possibly flawed--so we find ourselves training generations of new scientific minds that run around spouting out bad research. I understand that we all need to take research with a grain of salt when we read it, however bad scientists trying to become famous with their bad ideas or bad papers can be very detrimental to any field.
Comment removed based on user account deletion
If a submitted paper is scientifically unsound, it should be rejected.
If a scientific paper is useless to the readership, that publication should reject it and recommend a different journal.
If a paper is wrong and the reviewers KNOW IT then they should send it back for corrections.
If it's WRONG but the reviewers don't or more typically can't know it because it is novel, then publish it. The rightness or wrongness will be sorted out soon enough.
Ever heard of Isaac Newton? Turns out his theories were incomplete in some very fundamental ways, but his theories regarding the motion of objects were the best approximations we had for hundreds of years and are still very useful for macroscopic objects traveling way below c.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
"If we knew what we were doing, it wouldn't be research"
That's what my supervisor used to say to me when I got depressed about lack of progress.
Whether anything anyone says is right or wrong, it's a matter of opinion first and foremost.
No, it's not.
Our biology does not provide us
Our biology provides us with excellent truth detectors: throughout most of primate evolution, if you were wrong about whether your food was poisonous or whether there was a lion hiding in the bushes, you didn't get to pass on your genes. You didn't get to debate social relativism with the lion before he made a tasty meal out of you.
Most of science is still ultimately about matters like that, matters that have good answers, at least in principle.
Some science has veered off course, however. Every major scientific discipline (physics, biology, chemistry, etc.) has subareas where people start conflating experimental facts with opinion, aesthetics, and prejudice.
So, scientific truth is not a matter of opinion, but a lot of what is published in science is not about scientific truth.
Therefore anything that anyone says is simply an opinion.
That's just your opinion.
Who ordered that?
I didn't read the article but I don't believe the conclusions of the summary. Maybe in epidemiology it is true but not in physics where usually the results are reproducible and I very rarely find papers that are just wrong. I might agree that most of the papers are not 100% right (small mistakes in formulas happen quite frequently) but it does not impair the usability of a paper.
However, peer review does not solve all the problems. Most of the research takes a lot of time and effort and referees just read the papers. They do not reproduce the experiments or calculations. So peer review can weed out only obviously bad papers but not papers that looks OK but are wrong.
Save the bandwidth. Don't use sigs!
If 50% are wrong, then 50% are right. So if I write a scientific paper, the chance of it being right is 1/2. And if I write the same paper, say, 8 times, the chance of it being right at least once is 255/256.
I think I'll write that paper on statistics.
Your missing the point: ID-ers are claiming their theory is falsifiable. Which made others come up with equally falsifiable theories :)_ Activating_Meatballs
e.g. Flying Spaghetti Monsterism
http://en.wikipedia.org/wiki/Spaghetti_%26_Pulsar
Science progresses when well thought out hypothothies based on a good data are replaced by more inciteful reasoning based on more complete data. Lamarck wasn't guilty of faulty reasoning. He just didn't have a complete enough data set.
But the article at hand, isn't talking about that kind of "wrong". He is talking about conclusions that can not be supported by the data presented. Either the reasoning is faulty or the data collection methods are so faulty that no meaningful conclusions can be drawn.
When a theory is proven wrong in the scientific sense, it is a good thing. We learn something new and that be the basis for further developments. But if a theory is proven "wrong" in the mechanical sense, we have no new insights, just a relief from further time wasting.
I don't dispute for a moment that intelligence can create new forms of life.
I do dispute the scientific validity of the claim that ONLY intelligence can create new forms of life.
Your straw man is now on fire.
Why yes, I AM a rocket scientist!
Did anyone actually RTFP? It's one of the most spurious pieces of "research" I've ever read. And with a biophysics degree, I have read quite a few. The author actually didn't investigate any actual papers, but he builds a mathematical model out of his own biases, statistical projections, and some back of the envelope computation. Even then, his conclusions are much less stringent than the submitter makes them out to be. He "proves" that under all his assumptions, half the research papers *might* be wrong, but shows not even statistical evidence that they are.
I think PLoS is peer reviewed, but that paper should never have survived peer review. Occasionally, bad papers slip through, even in the so called hard sciences. This one seems to be one of them. Since PLoS Medicine is pretty well respected for an open access publication, lets assume that this was a lark and more on.
But it makes me curious what the fraction of bad papers looks like in an open access publication like PloS versus a traditional journal like, say, Nature, The Lancet, or New England Journal of Medicine. One reservation people have about open access (or author pays) models was that since PLoS gets paid about $1500 from they authors, they might be accepting vanity papers, or don't triage as well as traditional journals. I don't think they are, but if this paper is any indication, PLoS might take a second look at their peer review process.
the level of religous dogma in some camps.
The only question is, who decides which science is wrong? I doubt very seriously any big money areas will have a published high rate of error. After the high money science the next protected type would be whatever is en vogue for the time.
Scientific integrity took a big dive in the late 80s as special interest groups suddenly realized that marketing, confusion, and intimidation were far better at advancing agenda than honest science.
* Winners compare their achievements to their goals, losers compare theirs to that of others.
After reading the summary, I was under the impression that this artical was about a high rate misleading papers published by scientists.
Based on all of the anti-creationalist comments I thought maybe I had misread the summary, so I looked at the article itself.
Not once did I see mention of the universe's creation in the summary or in the linked artical, in fact the example stated was "such as whether a particular gene influences a particular disease."
It seems to me that lately a lot of comments on slashdot have been trying to start a witch hunt for advocates of ID. Can we please knock it off and stop screaming wolf every time some thing that is related to science is mentioned on slashdot.
Skeptic had their take on it in the last issue. In a nutshell
"Seven Deadly Sins? I thought it was a to-do list!"
If science can be wrong, then why trust it?
It is the only objective process for assessing facts from fiction.
In other words, if the best a scientist can tell you today is that, he might be wrong tomorrow, why even bother listening to him?
No one is forcing you to listen. You ignore the information provided by science at your peril.
So you can use science for real things, like physics and design of military weapons and consumer goods, but the rest of it is so much speculative nonsense.
Quantum physics is speculative, but you don't seem to be throwing your computer out the window.
The consequences of guessing wrong about the origin of humanity are completely immaterial to most people's lives.
Dead wrong.
Stalin believed that Darwinian evolution was just a bouguoise concept. He believed in Lamarckian evolution and directed his agricultural ministry to ignore studies that supported Darwinain evolution. Their agricultural industry suffered and people went hungry in the process.
You can't show people evolving any more than someone else can show God making something.
I can show a progression of hominid fossils leading to homo sapien sapien. The Bible is silent about these fossils.
It's immaterial, unprovable, and so why fight over it?
It may be immaterial to you, but the theory is consistent with the evidence we possess. You may not choose to believe it, but that the only thing immaterial about this discussion.
Yeah you can roll out the eliptical argument that evolution is somehow necessary for medicine but most doctors are concerned with the human species, here and now, and now plants and people are related.
Why bother? You obviously believe that the scientific method works differently for investigations related to the origin of humanity than it does when applied to chemistry.
To wit, you can get a Chem E degree and still get into Med School.
You are correct.
Just don't whine to me when you have difficulty making sense of the data you gather without using evolutionary theory.
You will never amount to anything more than a glorified technician.
I can live with that.
"Rocky Rococo, at your cervix!"
It seems to me that lately a lot of comments on slashdot have been trying to start a witch hunt for advocates of ID. Can we please knock it off and stop screaming wolf every time some thing that is related to science is mentioned on slashdot.
You realize you're talking to zealots, right? I don't think "please" is going to cover it.
Universities have failed a lot of scientists in that a) those papers are the result of stupid tenure policies and b) universities often do little to promote their researchers.
Engineers often read papers to solve problems. When they know about them! (Google Scholar might fix this)
A worse problem than them often being wrong is that:
a) there is frequently no way to determine if a given paper is accurate, has mistakes, is partially accurate, is laughable, was accurate at one point but is outdated, etc etc. At least from an outsider's perspective.
b) there is no good way to stay abreast of current interesting developments - hell, there's no way to see interesting things from 20 years ago easily! Again, this is from an interested outsider's perspective.
Once or twice a year I have the luxury of spending a week or two in an engineering library for the express purpose of finding out new and interesting things in my field. I'm SHOCKED at the amount of material that is being duplicated (often badly) in industry, material that is inaccurate or poor quality, and VERY GOOD material that never sees the light of day again.
..don't panic