Slashdot Mirror
Scientist Says Most Scientific Papers Are Wrong
An anonymous reader writes "According to epidemiologist John Ioannidis, the majority of published scientific papers are wrong. If Ioannidis's own paper is right, a randomly chosen scientific paper has less than a 50% chance of being true. He also says that many papers may only be accurate measures of the prevailing bias among scientists. However, a senior editor of a scientific journal says that scientists are already aware of this: 'When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about.'"
I have found this to be sadly true. My coworkers are college professors who often publish papers on social trends using large datasets obtained from government records. I am frequently pointing out errors in their analysis (mainly that they simply don't look at their data, such as just because Jane Doe and John Doe have the SSN they are assumed to be the same person) but I'm generally ignored or told to fix it myself though that isn't my job. They are more interested in getting something published and don't want to have to retract something.
> Does this mean that peer review fails as a method to filter out time-wasting, tree-killing dreck?
Peer review isn't a certification of correctness. It's just supposed to filter out the papers where the authors didn't do their homework. It can spot bad logic, use of outdated data, failure to consult important papers in the field, etc. But it can't tell us whether string theory is correct or not.
Sheesh, evil *and* a jerk. -- Jade
And, as the article mentions (but doesn't go into detail on), this is why reproducibility is so important.
Let's say a scientist comes up with a new idea, does the research, and publishes a result. OK, assuming we buy the article's premise, there's a 0.5 chance s/he's made a mistake. Now another scientist and then a third duplicate the experiment and get the same result. The odds that the original proposition is in error drops to 0.25, then to 0.125. The odds are now 8:1 the result is valid.
See cold fusion for an example of the converse.
Well yes, for the scientific community it's common knowledge (at least IMHO) that these papers are HARD to prove wrong, most are assumed true, but then again, what I learned most about image processing was in these papers. They do contain very valuable information, and a lot of these are works based upon previous works. (This is how science is done right now).
Even when some of these papers could be wrong in their conclusions, or maybe one or two algorithm flaws, but it was papers like these (image processing, etc) that contributed to technology used today, like MPEG4 video.
My point is, unlike these which are done with scientific methodology, in *medical* "research papers" there's oh so much money at stake. I'm sure the article could have said "most medicine research papers are wrong", and I would have believed that.
But science is much more than medicine, and as a scientist, I find it an insult to stain the name of Science because of commercial vias in medical research.
Curiously, I googled for "bias in medical research" (with quotes) and here's the top result, of 426 search results:
Bias in Medical Research by Maria Spicer.
In contrast, googling for "bias in image processing research" (with quotes) yielded no results.
Of course, google is only a very statistical method for finding out whether something exists or not, but I think you get my point.
Huh? What is then the driver of evolution? Gerbils? I guess getting a PhD in 2003 counts as a long time ago...
Human genome = 3 billion base pairs = 6 GBit. Windows + Office = 20 Gbit. Which is more impressive?
Skeptic had their take on it in the last issue. In a nutshell
"Seven Deadly Sins? I thought it was a to-do list!"
Indeed it is. Which is precisely why our senses are so easily fooled: given stimuli that do not correspond to those seen in survival tasks in the EEA (Environment of Evolutionary Adaptedness, i.e. hunting and gathering on the African plains or whatever), our brains do not necessarily respond correctly.
Now explain why a creator would have built brains that are so subject to misdirection, geometric optical illusions, etc. Why would he/she/it have done so?
belief in evolution is self-defeating because on the supposition that evolution is responsible for our reasoning ability, we have no confidence that the deliverances of reason (i.e. the theory of evolution) correspond to reality.
And if a creator built our reasoning capabilities, how do you know that he/she/it programmed it to accurately reflect reality? We'd be seeing whatever he/she/it wanted us to see, for whatever reasons. You'll get less mileage out of this argument for creation than for evolution, even.
Pointing out that evolutionary theory itself can't guarantee the accuracy of our reasoning faculties (which is true) gets you absolutely nowhere because Creation mythology is significantly worse. Consider:
Prove the above statement wrong. You certainly can't invoke anything it says in the bible, because that -- or rather your memory of it -- was created six seconds ago as well. It says exactly what the creator wanted it to say, for reasons of his/her/its own.
As soon as you invoke a creator, falsifiability is utterly gone, your conclusions can be ANYTHING, and future argumentation is pretty much futile. Thus creation mythology serves primarily as a tool for a person to project their own emotional needs and desires into their own understanding of reality.
Fortunately, there are other ways to evaluate the accuracy of our reasoning capabilities than evolutionary theory or creation mythology. Sparing a couple thousand years of philosophy, I'll stick to the pragmatic argument: Reason seems to work. It gives us effective tools for functioning, ergo we're best off assuming that our intellect and reason is what it seems to be, and make use of it.
End note: Your sig links to a story about Antony Flew "converting to religion". You'll notice he's a self-described Deist: a philosophy that is in no way contradictory to any contemporary understanding of evolution, physics, or any other branch of the sciences. He explicitly states he doesn't believe any sort of revealed religion. How does this bolster any point in favor of creationism or any other branch of post-Enlightenment fundamentalist thought? The point is lost, because Flew explicitly still rejects all that.
I stole this sig from someone cleverer than me.
... Galilean relativity, which was proven wrong by Einstein and friends.
...
Actually, strictly speaking, Einstein (and friends) didn't prove Galileo or anyone else wrong. That had already been done by others. Thus, precise measurements of the orbit of Mercury and turned up discrepancies with Newton's and others' laws of orbital mechanics. The Michaelson-Morley experiments produced the apparently-absurd result that light moved at the same speed relative to all observers, even if those observers were moving relative to each other or the light source. Etc.
What Einstein did was develop a new theoretic approach that could explain a number of these anomalies. It was then up to the scientific community to viciously attack Einstein's theories, and attempt to prove him wrong. They've been at this for a century now, and all of their tests so far have end up with results consistent with Einstein's theories, to within the error bounds of the measurements. In scientific circles, this constitutes "proof" that Einstein's theories are either correct, or are very close to correct.
Even then, the earlier theories hadn't really been proven wrong. Rather, they were shown to be merely good approximations. After all, if your instruments can measure something to 12 places, but Einstein's and Newton's equations predict a difference in the 20th place, you can't show either set of equations to be wrong. This is why those earlier "disproved" theories are still taught in science and engineering schools. Newton's equations are a lot simpler than Einstein's, and in situations where you can't measure the difference, you might as well use the simplest equations. You just have to be careful not to apply the simpler equations in situations where they aren't good enough.
But note that Einstein himself didn't disprove those earlier theories; that had been done by the others that found the anomalies. And Einstein didn't prove his own theories; that has been done by a century of tests by the entire scientific community. He did the really hard job: He came up with his wild new theories of a universe that behaved rather differently than anyone thought. But his theories were consistent with those strange observations. And his theories included equations that could be tested against the real universe. And his theories keep passing every test that anyone comes up with.
Now if we could get some other would-be scientists to present us with versions of their theories that can be tested against the real universe
Those who do study history are doomed to stand helplessly by while everyone else repeats it.