Slashdot Mirror
Scientist Says Most Scientific Papers Are Wrong
An anonymous reader writes "According to epidemiologist John Ioannidis, the majority of published scientific papers are wrong. If Ioannidis's own paper is right, a randomly chosen scientific paper has less than a 50% chance of being true. He also says that many papers may only be accurate measures of the prevailing bias among scientists. However, a senior editor of a scientific journal says that scientists are already aware of this: 'When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about.'"
So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about.
:-)
Whether anything anyone says is right or wrong, it's a matter of opinion first and foremost. Our biology does not provide us with the ability to view things from a topological perspecitve that extends beyond our fixed notion of an asseblage point; our peepers, the mind's eye, whatever you will call it. Therefore anything that anyone says is simply an opinion. It could be an opinion based on data, or facts long believed, or based on stimulus, but it doesn't prove truth at all and it never will.
The day our species extends beyond the fabric of time, space and matter, is the day we will know something closer to the truth. Not beforehand.
Oh we might get lucky! Let's face it... what are the odds that this planet would exist and support life? I think we won the lottery, really.
I can see where Ionnidis is coming from because I'm wrong half the time. I might be wrong about this comment too, but I'm going with my gut feeling!
It is however, the duty of humanity, to reach. In reaching we may fall, or fail, but it is the effort that rewards and replenishes, not the result.
That's why I like Slashdot... it's the ideas and the thought provoking commentary. Don't mod this funny -- I'm being serious.
Story A is posted, 200 people weigh in on everything from the topic of the story, to the troll of the month. The moderation puts most of the good ideas provided in the first five minutes up to a very high score, and the rest of the comments are buried.
So I don't look at the scores as much as I look at the titles. If the title is thought provoking, I read the comment and reflect on someone else's point of view, which is usually highly debatable.
The dangers of knowledge trigger emotional distress in human beings.
I have found this to be sadly true. My coworkers are college professors who often publish papers on social trends using large datasets obtained from government records. I am frequently pointing out errors in their analysis (mainly that they simply don't look at their data, such as just because Jane Doe and John Doe have the SSN they are assumed to be the same person) but I'm generally ignored or told to fix it myself though that isn't my job. They are more interested in getting something published and don't want to have to retract something.
So true! One of the biggest reasons scientists publish formal papers is so that other scientists can study and attempt to corrobotate -- or disprove -- the results of the paper.
I know of a whole company based on a bad paper. Some type of "fast blood analyzer". After a number of bad pre-production starts, yelling matches between software and hardware people, firings, suings, quitings, it was finally determined that whole premmis (from a founder's scientific paper) was false.
-- www.globaltics.net
Political discussion for a new world
> Does this mean that peer review fails as a method to filter out time-wasting, tree-killing dreck?
Peer review isn't a certification of correctness. It's just supposed to filter out the papers where the authors didn't do their homework. It can spot bad logic, use of outdated data, failure to consult important papers in the field, etc. But it can't tell us whether string theory is correct or not.
Sheesh, evil *and* a jerk. -- Jade
"All models are wrong; some are useful" -George Box
I'd say it's more like 100% of scientific papers are wrong, it's merely a question of the limitations of the model.
And, as the article mentions (but doesn't go into detail on), this is why reproducibility is so important.
Let's say a scientist comes up with a new idea, does the research, and publishes a result. OK, assuming we buy the article's premise, there's a 0.5 chance s/he's made a mistake. Now another scientist and then a third duplicate the experiment and get the same result. The odds that the original proposition is in error drops to 0.25, then to 0.125. The odds are now 8:1 the result is valid.
See cold fusion for an example of the converse.
What you say is very true. I am working towards my Ph.D. right now in Physics, and I encounter this every day.
I think both you and the poster hit on something very important here. First is that we (as people who are reading the research papers) are not looking for proof, etc. of something. When I grab the latest paper on a topic I work on, I am not going to read it and say, "Oh. They found x which contradicts with what I am seeing. They must be right." Instead, I am looking at their models, results, and the like to see what HAS been done, what the outcome was, and if they have any problems. That way, I can address these issues in papers I write and talks I give. It also gives me something to compare my results to.
Your cousin is facing what I am facing, as well. I think people in the community understand perfectly well where she is -- that she has found interesting results thus far and needs to work more on it to get deeper understanding/whatever. To the outside world though... "it's just a theory" or "it's just preliminary" (phrases people love to throw around) drowns out the important stuff she's doing.
The second thing, that you specifically hit on, is that we need to eliminate options when we're working in an area. The pressure to produce sexy, nightly-news-ready results keeps us from doing that. Of course, I'm biased. Hah.
This doesn't mean all of this work has been pointless. People do studies, report their results (which are sometimes wrong)... but we know what we're doing, and we know what to take with a grain of salt and what not do.
Mike.
Mmmm......sacrelicious.
Well yes, for the scientific community it's common knowledge (at least IMHO) that these papers are HARD to prove wrong, most are assumed true, but then again, what I learned most about image processing was in these papers. They do contain very valuable information, and a lot of these are works based upon previous works. (This is how science is done right now).
Even when some of these papers could be wrong in their conclusions, or maybe one or two algorithm flaws, but it was papers like these (image processing, etc) that contributed to technology used today, like MPEG4 video.
My point is, unlike these which are done with scientific methodology, in *medical* "research papers" there's oh so much money at stake. I'm sure the article could have said "most medicine research papers are wrong", and I would have believed that.
But science is much more than medicine, and as a scientist, I find it an insult to stain the name of Science because of commercial vias in medical research.
Curiously, I googled for "bias in medical research" (with quotes) and here's the top result, of 426 search results:
Bias in Medical Research by Maria Spicer.
In contrast, googling for "bias in image processing research" (with quotes) yielded no results.
Of course, google is only a very statistical method for finding out whether something exists or not, but I think you get my point.
Science is not about finding the truth. It has nothing to do with the truth; people who look to it for truth misunderstand it. Science, and the scientific method, are based on one thing: reproducible effect. I do X, Y, and Z, and T results. If this can be confirmed, reproduced independently, you might have something scientifically useful.
Notice what this does not say: X, Y, and Z are "true"; Z is "true"; X, Y, and Z cause T. Nor does it state the meaning of X, Y, Z, or T. Nor does it say why, in the presence of X, Y, and Z, T occurs. These are irrelevant. The only thing science does, the only thing it is capable of, is one thing: testing if, in the presence of X, Y, and Z, we repeatedly get T. For most things, that's all that matters. This is the scientific method.
Thus it is that science is, quite literally, magic. Look over most fictional magic systems. We have things like "if we say this spell, this thing happens." "If we write these symbols, this thing happens." "If I visualize this thing in my mind, this thing happens." "If a mix a pinch of this and a hair of that, this thing happens." Because it's reproducible, it's useful. The mechanic does not matter: only reproducible effect matters. If waving ones hands and saying a phrase were to be followed consistently by a minor explosion, it would be just as scientific as mixing two chemicals to produce the same effect.
It doesn't matter why. Theories get revised consistently to fit the facts, to document reproducible effects. If phlostigen and ether were accurate and useful models, the fact they have been discarded for more useful models does not matter: science isn't about truth. It is about reproducible effects.
This is why not clinging to pet theories (yes, this includes everyone's favorite: natural evolution) is important: the theories do not matter. One should never fit facts to a theory. One should create a theory to fit the facts.
This is what makes science useful.
Don't think of it as a flame---it's more like an argument that does 3d6 fire damage
Actually, I'm one of those kooky folks who believes in God and Creation, but my response to the idea was rather more like 'That can't possibly be true...' My second reaction was that his conclusion is plausible, challenging, and disturbing. I can accept the idea that 50% or more of all conclusions reached through scientific inquiry will be refuted or replaced by more precise studies reaching more accurate conclusions. I can accept that idea because the history of science appears to bear the pattern out, e.g., the Newtonian->Einsteinian->Quantum Mechanics progression, but the idea that 50% of all 'scientifically' reached conclusions are both Bad and Wrong is a little scary. I wonder how much bother one has to endure in order to get a chance at accuracy.
Huh? What is then the driver of evolution? Gerbils? I guess getting a PhD in 2003 counts as a long time ago...
Human genome = 3 billion base pairs = 6 GBit. Windows + Office = 20 Gbit. Which is more impressive?
Skeptic had their take on it in the last issue. In a nutshell
"Seven Deadly Sins? I thought it was a to-do list!"
- Nipples for men
- Why is the tailbone a series of fused, degenerate vertebra instead of a single solid bone process, if it's so well needed and "designed"?
- Why do dormant tail genes sometimes activate and create an actual, if distorted, tail? Why are those genes in there?
- Why use vertebra at all for a vertical animal? They're wonderful for a horizontal one, but terrible for a vertical one. This is why your arms and legs are long bones rather than vertebra.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
Yes, I agree with your first statement, though, it would be hard to believe in purpose without something to create the purpose, wouldn't it? By that, I mean I understand someone could rationally conclude the idea that the complex system we live in didn't just...happen out of raw chance but there would have to be some kind of initiating actor, wouldn't there? Actually, what you describe would fit the "pure" creationist belief.
Sure, the vast majority of stuff in Biblical Archaeology Review (if it's still published...) would be things like fingind some old well in a place which matches a story in the Bible or whatnot. Discovery Channel has shows about this type of thing periodically. They had one about a theory that Egypt had 2 competing Pharaohs at one time and, if true, that pulled the timeline of some Biblical stories into perfect mesh with Archaeological timeline evidence which was previously believed to conclusivley discount the Biblical story. Even so, what happens when they find evidence which would fit stories of supernatural activity? At some point physical discoveries or "decoded" manuscripts, etc. would match non-human actors in the Bible. That's all I was trying to bring up. BAR is (or was, I don't know if it's in print now) was a true scholarly journal. It had an obvious preference for what was published but was most certainly along the "physical evidence" as opposed to dogmatic mental constructs line.
I don't think your statement about secular archaeologists fits what I was trying to say. I understand the comment but, by it's nature, a secular denial viewpoint (for lack of a better word) wouldn't have analogy to the Bible truth viewpoint of Christian archaeology (for lack of a better term.) IOW, BAR would be making the case through physical evidence that the stories in the Bible are actual recorded events. The more that is "proven", the less "fictional" the Bible would be. There just wouldn't be an analogy for secular viewpoints. (I know what I'm trying to say, it might not have come out my fingertips just now. Try to read past my poor word choices.)
Sorry if it seemed I pulled things off topic. That wasn't the intent.
For every problem, there's going to be some socialist proposing a solution. That shouldn't be a factor when we ask whether or not the problem itself exists.
... Unless you're claiming that a worldwide socialist conspiracy is fudging scientific data, which is a whole 'nother story.
First things first; We need to figure out the facts of the situation, without regard to the consequences of those facts.
"Global Climate Change" is a scientific theory with good support.
Some of the reactions to those changes and anticipated changes might be described as "religious."
And incidentally, last century is replete with deaths from many forms of dictatorships gone out of control, both Communist and Facist. If you're refering to socialism within a functioning democracy when you tak about 'demonstratable tens of millions of deaths' then please let me know specifically what you're refering to. A little regulation of food and commerce seems to have increased health and prosperity rather than the opposite.
___
It's the end of my comment as I know it and I feel fine.
Indeed it is. Which is precisely why our senses are so easily fooled: given stimuli that do not correspond to those seen in survival tasks in the EEA (Environment of Evolutionary Adaptedness, i.e. hunting and gathering on the African plains or whatever), our brains do not necessarily respond correctly.
Now explain why a creator would have built brains that are so subject to misdirection, geometric optical illusions, etc. Why would he/she/it have done so?
belief in evolution is self-defeating because on the supposition that evolution is responsible for our reasoning ability, we have no confidence that the deliverances of reason (i.e. the theory of evolution) correspond to reality.
And if a creator built our reasoning capabilities, how do you know that he/she/it programmed it to accurately reflect reality? We'd be seeing whatever he/she/it wanted us to see, for whatever reasons. You'll get less mileage out of this argument for creation than for evolution, even.
Pointing out that evolutionary theory itself can't guarantee the accuracy of our reasoning faculties (which is true) gets you absolutely nowhere because Creation mythology is significantly worse. Consider:
Prove the above statement wrong. You certainly can't invoke anything it says in the bible, because that -- or rather your memory of it -- was created six seconds ago as well. It says exactly what the creator wanted it to say, for reasons of his/her/its own.
As soon as you invoke a creator, falsifiability is utterly gone, your conclusions can be ANYTHING, and future argumentation is pretty much futile. Thus creation mythology serves primarily as a tool for a person to project their own emotional needs and desires into their own understanding of reality.
Fortunately, there are other ways to evaluate the accuracy of our reasoning capabilities than evolutionary theory or creation mythology. Sparing a couple thousand years of philosophy, I'll stick to the pragmatic argument: Reason seems to work. It gives us effective tools for functioning, ergo we're best off assuming that our intellect and reason is what it seems to be, and make use of it.
End note: Your sig links to a story about Antony Flew "converting to religion". You'll notice he's a self-described Deist: a philosophy that is in no way contradictory to any contemporary understanding of evolution, physics, or any other branch of the sciences. He explicitly states he doesn't believe any sort of revealed religion. How does this bolster any point in favor of creationism or any other branch of post-Enlightenment fundamentalist thought? The point is lost, because Flew explicitly still rejects all that.
I stole this sig from someone cleverer than me.
"It seems to me that lately a lot of comments on slashdot have been trying to start a witch hunt for advocates of ID. Can we please knock it off and stop screaming wolf every time some thing that is related to science is mentioned on slashdot."
/. as much anymore is because I'm fed up with all comments about why religion sucks/creationism and ID is bad/Bush was wrong when they have no real relevance to the topic at hand. If you don't believe it, you still have to respect others for believing what they do. Insulting those who have a faith is just plain disrespectful.
/watches Karma disappear...
I was beginning to worry that I was the only one who thought this as well.
The main reason why I'm not reading
Slashdot needs a "Likely to turn into religious/political flamewar" category so I can ignore those stories.
ID could be a valid theory if the people who currently advocate ID had never heard about it.
These people know little about the workings of science, and merely took up ID as another weapon for their own agenda. In doing so they discredited both ID and the agenda they were trying to support.
As both a Christian and a scientifically minded person that makes me rather sad and a bit angry.
There just wouldn't be an analogy for secular viewpoints.
Oh yes there frikken would. It might be called something like Ancient Roman Archeological Review and it would be a bunch of archaeologists talking about archaeological evidence being used to describe events taking place in ancient Rome, just as Biblical Archaeological Review would discuss archaeological evidence for events taking place during the Bible.
Karma: Chevy Kavalierma.
I didn't say it lacked scientific merit, or that the observations of Evolution would be proved wrong. I AM saying that a to philosophy or world-view implied by the Evolution as it is currently understood is highly likely seen to be foolish in a few hundred years, just as people who believe in Determinism based on Newtonian physics were wrong.
Newtonian Physics is not wrong.
Determinism IS wrong.
And none of this means that it's a sure thing that the "philosophy inspired by belief in Evolution" won't be seen to be exactly the total truth.
I'm just saying, that based on past evidence, one should be highly skeptical of world-views based on theories, because *every* *single* *theory* mankind has posited here-to-date has been "refined" and the underlying worldview has been turned upside down.
Something's going to come along which is as "ridiculous" as string theory and charmed quarks would have seemed in the 1700s.
So, be careful of believing "science has told me there is no God." The science isn't wrong, but the philosophy is. Just the same way "science used to tell me that I can predict the entire state of the universe given a known initial condition amd the 3 basic laws."
... Galilean relativity, which was proven wrong by Einstein and friends.
...
Actually, strictly speaking, Einstein (and friends) didn't prove Galileo or anyone else wrong. That had already been done by others. Thus, precise measurements of the orbit of Mercury and turned up discrepancies with Newton's and others' laws of orbital mechanics. The Michaelson-Morley experiments produced the apparently-absurd result that light moved at the same speed relative to all observers, even if those observers were moving relative to each other or the light source. Etc.
What Einstein did was develop a new theoretic approach that could explain a number of these anomalies. It was then up to the scientific community to viciously attack Einstein's theories, and attempt to prove him wrong. They've been at this for a century now, and all of their tests so far have end up with results consistent with Einstein's theories, to within the error bounds of the measurements. In scientific circles, this constitutes "proof" that Einstein's theories are either correct, or are very close to correct.
Even then, the earlier theories hadn't really been proven wrong. Rather, they were shown to be merely good approximations. After all, if your instruments can measure something to 12 places, but Einstein's and Newton's equations predict a difference in the 20th place, you can't show either set of equations to be wrong. This is why those earlier "disproved" theories are still taught in science and engineering schools. Newton's equations are a lot simpler than Einstein's, and in situations where you can't measure the difference, you might as well use the simplest equations. You just have to be careful not to apply the simpler equations in situations where they aren't good enough.
But note that Einstein himself didn't disprove those earlier theories; that had been done by the others that found the anomalies. And Einstein didn't prove his own theories; that has been done by a century of tests by the entire scientific community. He did the really hard job: He came up with his wild new theories of a universe that behaved rather differently than anyone thought. But his theories were consistent with those strange observations. And his theories included equations that could be tested against the real universe. And his theories keep passing every test that anyone comes up with.
Now if we could get some other would-be scientists to present us with versions of their theories that can be tested against the real universe
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
The problem is not that many scientific theories are wrong, we all knew that. The problem is that a majority of published scientific papers are provably wrong at the time of publication, and the author should've known that it was wrong, but is too stupid and/or busy to publish a correct one.
Scientific papers are usually written by grad-students trying to earn a degree, and that is usually the only real purpose they will ever serve. The project I am working on now is a continuation of the work that was carried out by someone who now has a PhD. Nearest I can tell, one of the important equations he used was not appropriate for our equipment. It's just a +/- error, but it's a pretty big deal in terms of the data you get. He also made some rather inappropriate assumptions. A paper was published from his work.
People need to realize that "scientific journals" are simply catalogues of the work that has been done by various grad-students and do not necessary reflect reality. I'm not saying they're not useful, I'm just saying that they aren't often correct.
> Yes, I agree with your first statement, though, it would be hard to believe in purpose without something to create the purpose, wouldn't it?
Why so? Is purpose necessarily dictated by some external agency? Can't purpose-oriented creatures decide on their own purposes? Do people who believe in a false religion have any difficulty finding a purpose in life?
> By that, I mean I understand someone could rationally conclude the idea that the complex system we live in didn't just...happen out of raw chance but there would have to be some kind of initiating actor, wouldn't there?
I can't figure out what you're saying there, but just to make sure everyone understands, scientists don't think stuff just happens "out of raw chance". If they did, they wouldn't waste their time looking for explanations.
> BAR is (or was, I don't know if it's in print now) was a true scholarly journal.
Actually, for as long as I've known about it it has been more of the "science magazine" genre, far more like Scientific American than Nature.
That's not to say it isn't worth reading, but we need to be realistic about these things.
> I don't think your statement about secular archaeologists fits what I was trying to say. I understand the comment but, by it's nature, a secular denial viewpoint (for lack of a better word) wouldn't have analogy to the Bible truth viewpoint of Christian archaeology (for lack of a better term.)
So, should we have "Christian archaeology" and "Muslim archaeology" and "Shinto archaeology", like we once had "Deutsche physic"?
> IOW, BAR would be making the case through physical evidence that the stories in the Bible are actual recorded events.
What's the difference between BAR doing that and someone else doing that?
The investigation of the history of "the holy land" is certainly legitimate, and surely merits a rag focused on that topic, but shouldn't they be trying to "see what happened" rather than "make the case"?
(BTW, where did BAR come out on those archaeologists' claims that the united kingdom never existed?)
> The more that is "proven", the less "fictional" the Bible would be.
And the same applies to the Iliad and the Odyssey, right?
(Surely you realize that even if the bible is packed with facts, that wouldn't make the rest of it true.)
> There just wouldn't be an analogy for secular viewpoints.
So, BAR is apologetics that just happens to use archaeology for its vehicle, rather than an attempt at unbiased archaeology?
I'm sorry, but I'm really not sure what you're getting at.
Sheesh, evil *and* a jerk. -- Jade
Ok, as from someone who is in thie "business" of research, and papers "creation", you have to know, that there is no perfect idea, there is no perfect solution, there is no perfect paper. But this is not the goal, either. Conferences and conference papers are there to provide a ground for scientists to make their latest stuff public and let it be chewed and digested by others. It's after many iterations and discussions and quarrels sometimes, when one either gets to a point when the re- and re-corrected idea seems to work ok, or it turns out to be useless junk although it seemed like being good at first.
:P
I read many papers, I don't know numbers, but many dozens in a month. Usually I don't care if they seem good or bad, if they are correct or not. The ideas therein are what matter. Sometimes you get ideas on how to improve an old idea, sometimes you get new ideas from older papers. sometimes it's just nice to know what others are doing.
The matter is very much different when you have to review papers, but the seriousness of that review also depends very much on how much time you have, possible IRL problems, etc., but that's why there are >=3 reviewers+associate editor assigned to the paper at most of the serious journals.
Stating such things as a certain percent of all papers are crap is just crazy sh*t. It happens very seldom that I read a published (conference or journal) paper and I think it was useless. Anyway, if it would be true that would mean that this guy's paper is also half useless. You are free to choose which half
I am putting myself to the fullest possible use, which is all I can think that any conscious entity can ever hope to do.
Even in the journals that I regularly read (every issue, every year), I only read a relative handful of papers, germaine to my research. When my research topics evolve, I might go back and read different papers in the same issue. Maybe there are some scientists out there who read every paper in every issue of journal in their field, but they must read a hell of a lot faster than I do. I rely on Current Contents, automated lit searches, and other computer-based tools to sift through the flood of info. I also rely on my colleagues - they know what research I do, and I know what research they do. If I see something that might interest them, I forward it to them, and vice versa.
there is no good way to stay abreast of current interesting developments
I would respectfully point out that that's why the annual scientific conferences are useful. The research presented in the talks and the posters precedes that presented in the papers and book chapters, giving you a feel for what the latest interesting problems are. If all of a sudden there are three times as many posters on Probelm X at the 2005 conference than there were at the 2004, then that should tip you off that something is up. If they are all coming out of one institution, that should tell you something, too. I know IEEE and other engineering societies hold annual meetings; are they not as useful as, say, ASM?
Once or twice a year I have the luxury of spending a week or two in an engineering library for the express purpose of finding out new and interesting things in my field.
This means that you are necessarily reading the journals at least a month, perhaps as much as a year after they come out. I don't mean to flame, but I suggest that this is not a very good strategy for staying current. By the time something is published in a journal, a lot of people will have known about it for a year or more, right down to the experimental details.
As to how to tell good work from bad work, that's what collective and individual professional judgement is for. If the profession is divided, and your individual level of expertise in that particular area is inadequate to make a good judgement, that's when you ask a few of your colleagues, "So, did you see that presentation on Problem X from Dr. Smith at Big State University? It looks like he was directly contradicting Dr. Jones from Small Private University. What do you think?" If none of you can tell who's probably right, then you either wait for more data or go generate the data to decide the issue.
The man who does not read good books has no advantage over the man who cannot read them. - Mark Twain
Did anyone see the recent NOVA program about Rosalin Franklin. She was doing DNA research at the same time as Watson and Crick, but in a different lab in England.
It turns out that Watson used her data without her permission and without attribution. And he went on to seriously misrepresent her in the book that he later wrote on the discovery of DNA. In fact, Harvard, the original publisher, ended up not publishing the book because of the complaints about the way she was portrayed by many of the other people who were there and mentioned in the book.
Watson basically created a fictional account about the way that DNA was discovered. And the public at large drank it up.
Watson got the Nobel prize. Rosalin Franklin is hardly remembered.
The scientific community is as full of intrigue and back-stabing as any other human community. Well, maybe except for Slashdot.
Evolution is a fact. Darwinism is a joke.
Everything in the Universe is caused by something else, requiring a cause to exist.
One imperfect thing may be caused by another, but the causer needed to have been caused by something else (since it is imperfect and requires a cause).
If imperfect things exist, there must be a being who can cause them, but having the characteristic of not needing to be caused himself (and this being we call God).
If there were no God, nothing imperfect, requiring a cause, could exist.
But imperfect things exist, therefore God exists.