Slashdot Mirror
Why Most Published Research Findings Are False
Hugh Pickens writes "Researchers have found that the winner's curse may apply to the publication of scientific papers and that incorrect findings are more likely to end up in print than correct findings. Dr John Ioannidis bases his argument about incorrect research partly on a study of 49 papers on the effectiveness of medical interventions published in leading journals that had been cited by more than 1,000 other scientists, and his finding that, within only a few years, almost a third of the papers had been refuted by other studies. Ioannidis argues that scientific research is so difficult — the sample sizes must be big and the analysis rigorous — that most research may end up being wrong, and the 'hotter' the field, the greater the competition is, and the more likely that published research in top journals could be wrong. Another study earlier this year found that among the studies submitted to the FDA about the effectiveness of antidepressants, almost all of those with positive results were published, whereas very few of those with negative results saw print, although negative results are potentially just as informative as positive (if less exciting)."
Peer review no doubt helps to limit people who intentionally want to cause problems. Sokal's bullshit paper on quantum gravity (see The Sokal Hoax ) made it into print only through a non-peer-reviewed journal. While it is disturbing to think much published scholarship is unreliable, at least it isn't necessarily malicious.
How long until some researcher releases a study showing that Dr. Ioannidis' research findings are themselves wrong?
Prevent Windows piracy. Use Linux instead.
Of course there is absolutely no chance that this particular piece of research is also wrong...
Title is wrong. It says that the FDA is corrupt. And that published papers take around 3years to get peer reviewed where the bad ones are removed. What a blatent attack on science generally. Sure paper publishing needs to be reviewed but 'most published research is false' is an outright LIE. 'Most published research' includes all of our basis of scientific knowledge. If most of our theories on biology were wrong really we realistically wouldnt have been able to move forwards into working with genes if we didnt know what a cell did.
At the risk of being modded down to oblivion, I am still curious to how this effects popular theories like global warming. We already has people claiming that the science is wrong and they are generally mocked and ignored because their works are published in major journals. Well, this story seems to indicate that publishing those claims will give them a larger change of it being incorrect.
Anyways, it seems that if you don't tow the line on climate change, there is no room for you anywhere. So where does this leave the accuracy of the claims in light of how common it seems that they can be wrong even when published in a respectable scientific journal. I know the IPCC looked at them, but they didn't validate any of the claims, they only looks at whether or not Humans were the cause (that was their charter and they acknowledged this in their reporting).
I would think that "Publish or Perish" must contribute to a lot of crappy papers getting published. Shovel it out the door, somebody else says it's wrong, write another grant for a study to verify that, shovel that one out the door, rinse, lather, repeat...
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
So... how do we know that John Ioannidis's research isn't false and likely to be refuted in a couple of years?
As the authors point out, published work is often wrong because journal editors know that sensational, wrong headlines garner much more information than mundane, correct headlines.
Their conclusion is that putting the articles on the internet would solve the problem.
Unfortunately, their solution is a like 10,000 spoons when all you need is a knife.
publication bias.
(founded 95,000,000 yrs ago, very space opera)
Those responsible for refuting the research of the people who have just been refuted, have been refuted.
Basic idea: high-profile journals want papers that are new and exciting. This means that scientists have an incentive to 1) rush their work, 2) choose fields that are popular, and 3) claim that their papers solve more than they actually do. This leads to sloppy, dishonest papers.
I'm not going to judge this paper - I haven't read it thoroughly - but to pair a title like "Why most published research findings are false" to a pretty well-known problem seems itself like an example of problem 3!
News flash! What works in one situation (or for one person) might not work so well in another. Too little research takes the context into account, particularly regarding any research that is human-related, and so it becomes easy to "disprove" prior findings.
Ironically, this Economist article did a small, unrepresentative study and claimed that it has broad significance over a much larger topic. Its authors published the most compelling explanation they could think of for their results, ignoring more mundane ones (like, for example, the fact that a 'groundbreaking' study in Nature is going to spur more research which may well refute portions of the original study, whereas a study published in a secondary journal may not).
Who's to say that the papers refuting this research are correct? It seems to be taken for granted that the dissenting papers are correct, and thus the original papers are wrong. It seems likely that the refuting papers may be wrong, or that there are complex situations in which both papers are correct (to differing degrees).
If one third of the papers are refuted, and this study concludes that "most" of the research is false, does that mean this study is part of the problem? No, i did not RTFA.
My prediction for this thread:
Several people will post about how this validates the TINY, TINY, TINY, number of scientists and LARGE number of completely uneducated "opinion" formers and MASSIVE number of people who think that "belief" is the same as fact.
What they will miss:
This article talks about how things are put out there then invalidated by SUBSEQUENT PUBLISHED RESEARCH, not about how there is a great conspiracy around something being "right" and everyone shouting down those who dare to disagree. Global Warming is something that has consistently been found to be happening and while certain bits have been revised due to subsequent research, most of that research has found that previously incorrect models were in fact too optimistic in their view.
This article doesn't strengthen your misguided, and uneducated, belief that Global Warming isn't real. When even the Republican candidate says its real then its time to let go and become part of the solution.
An Eye for an Eye will make the whole world blind - Gandhi
I really hate reading research papers. Many of them are cookie cutter papers to make it look like work was done - a bunch of text with a few graphs and lists thrown in, all followed by a massive list of references. The descriptions defy anyone to actually replicate the work. Theorems are proved on the basis of some other theorems in difficult-to-find references, but the logical steps (as you would see in a math textbook where theorems are proven in detail) are at best a few scattered mentions of the format "if x and y then z (and you better believe it because that's what happens when you do all the substitutions)."
In the past, people didn't have such a huge reference base so they could follow the logic, but now with computers, the Internet, and massive hard drives, papers ought to be much longer and more detailed. This would force researchers to not have the attitude of "because you are either studying for a Ph.D. or you have one, you should have the IQ to reverse engineer my logic".
When people start their education, they are told "show your work". Full credit is not given for just the final answer. Because there are more and more universities, institutes, students, problems, etc., people have less time to read about each project and follow every step of reasoning. It was necessary to keep papers short, and that wasn't such a bad thing when people took care to present well, but the system of trust can only persist as long as the trust is not broken. There are too many research areas now with their own little symbolisms and patterns of communications, as well as too many researchers who invent their own symbolisms and styles when they are unsure whether any standard exists. It's a Tower of Babel.
A solution in the computer age is quite simple. Computer storage is cheap enough to permit massive appendixes that give the details of derivations. The Internet can be used to distribute standard ways of expressing ideas and themes that are commonly found. The entire system needs to be more self enforcing by having papers widely available so that people can see what is the right way versus the wrong way. Then, the statistic of which papers have the most references can give a meaningful idea of which papers are the best.
When I see movies depicting life decades ago, I see that people presented themselves with greater complexity and attention to detail. Communications seem to be more bursty now, perhaps because everyone is trying to finish quicker with every objective. Often this leads to shallower thought though because there isn't time taken or given to ponder. So we may well be seeing "research" that just marginally advances a randomly selected result from someone else's papers, and that is the easy path to getting credit and "getting on with life". It's the whole attitude of "no one is going to care because there is so much going on and I'm just insignificant".
Computers can help here. If people want to achieve more signficance, they can produce more full-bodied writeups - this process itself forces them to think deeper and better, and if they have something worth telling, the world will find out.
Know your pads. One time pad: good for cryptography. Two timing pad: where to take your mistress.
Single malt scotch proven to be very effective in eliminating depression.
I now open the research to peer review of long-term efficacy studies.
Many are wrong. This is pretty obvious, and it's also why science works. Eventually, the wrong ones will be replaced with something less wrong, and so on.
SJW n. One who posts facts.
This is in a large way because the empirical method is flawed for humans. It requires complete rationality and unaffected views, two things you just aren't going to find in human beings. Even our currently taught scientific method promotes going into an experiment with expectations. The fact these experiments cost money, time, etc... (many times in the forms of grants that will be pulled if you don't show significant progress), results are often overblown or outright false. Its not that the lofty ideals of the scientific and empirical methods aren't impressive--simply that they are as currently impossible to reach as the idea of being a perfect Christian.
Ginga no Rekshiya Mata Each page.
that proves most published research findings are true.
"You'll get nothing, and you'll like it!"
Why is it that a large portion of scientific research today is garbage. Well one very powerful reason, money. I saw this firsthand working at a major university medical center on large scale behavior research projects. The outcomes of our studies directly effected existing and experimental drugs and the drug company representatives were right there alongside the researchers at all levels of the process. Professors received "gifts" and other unofficial incentives from them regularly. I saw at least one study where the results were out and out fabricated so that the results would support the effectiveness of a particular drug for treating a childhood psychiatric disorder. In other cases data was included after the fact or blanks were filled in by clinicians from memory. All practices that are highly unscientific. Many of these studies resulted in drugs and treatments for children that are in use today and based on research that is at best questionable and at worst fraudulent. When there is a profit motive behind science it becomes very difficult for it to remain true science and sadly that is the state of affairs in many fields today.
Strange... Let us see how this works out: 33% of papers are refuted in later-published papers. But of those later-published papers, 33% of those are refuted too. And of those, 33% are refuted... If my maths is right, then over time *all* research will have been refuted!? (I don't recall anybody refuting the general theory of relativity...) Luckily, this piece of science will then also be refuted - which then wipes out the basis of my argument... G'ah - this science stuff is complicated!
I agree, but not for the same reasons (although the author's reasoning sounds plausible for big-name journals).
I've found that peer reviewers very seldom give good critiques of the methodology. Rather, most of the comments appear to be on the scope of the paper - comments of the variety "You're doing X. Smith et al. has done Y (which is tangentially and usually very weakly related to X) yet you don't mention this." I suspect that this is because most reviewers don't know enough about the research methods being used to provide a thorough, useful, and accurate critique, but still need to write something, so they take shots at the scope instead, trying to draw on what they do know.
If the methodology is not being effectively critiqued, there is little to no selection for sound research. And if each point you bring up causes reviewers to demand you bring up five more, this introduces selective pressure towards papers that either say far too much or far too little.
I still think a Digg-style system would work well for paper publication. Any half-sound research would be available, but research that people find "useful" would naturally rise to the top.
So... Since this study is published, I can assume that it's wrong... but then, if this study is wrong, it's right... but then... Excuse me while I pass out -- If it weren't for my horse...
Scientific research is just that -- research. If it were as easy as doing a couple of experiments, revealing the "truth" and moving on to the next thing, we'd all be living around Alpha Centauri by now. But science is hard and therefore a lot of conclusions are naturally going to be wrong. If that weren't the case then we wouldn't even need any scientific journals -- all we'd need would be newspapers.
Remember the whole "theory of evolution" issue that the creationists keep harping on? "They call it a theory so it must not really be true?" We all know that evolution is just about as "true" as any science gets -- and yet surely there are some portions of the current body of knowledge about evolution that will one day be falsified by later research. That's not a bad thing.
Notable research that has since been thought to be flawed or insufficient: Newtonian physics. Niels Bohr's model of the atom. Gregor Mendel's research into genetics. Einstein's theory of general relativity. Koch's postulates for determining disease causation. Quantum mechanics. And so on.
Breakfast served all day!
Please replace "false" with "incorrect". The word "false" implies deliberate fraud, and while that undoubtedly happens the cited articles do not suggest that fraudulent papers are in the majority.
New information supplants old ideas and how does this process become sensible? I often run into manuals that are a few versions out of date on the web and they never get removed. Until somebody can come up with a way that everyone can share information in a common data base that is version controlled, the problem will persist. Wikis and Wikipedia are good ideas, but the concept needs some kind of extension so that information and its corrections are connected in some way.
The fact that government / companies might screw with the data for personal interest is a separate issue of letting an inherently corrupt process manage your information.
I suppose the responsibility falls to the end user to deal with the inherent problems in managing and verifying the data that they use for their purpose. I doubt that a complete, secure, common, valid data base ( or system ) could be devised when more than one person is involved.
Even after reading the article, I'm still not sure if the authors are saying:
A) Given that research has been published, it is more likely to be false than not; or
B) Given that research is false, it is more likely to be published than is the case for true research.
I mean, it says:
So, (Wrong Articles)/(Total Articles) = >=0.5, right?
But the only figures I can find in the same article are:
So.. "most" is now "less than one third"?
I'm somewhat alarmed that The Economist lets people who don't seem to grasp basic statistics write their articles.
Antiquis temporibus, nati tibi similes in rupibus ventosissimis exponebantur ad necem.
Well then this finding itself is more likely wrong than right, so we can just ignore it...
Yes I agree - this paper is not about "Most Published Papers" in Science. It is about published papers in the area of therapy effectiveness. Especially those where we do not have a good model. Thus of course about half should be wrong I would guess, as established by later studies. This is statistics in action. When you are looking for high correlations and selecting for the positive, you will will get false ones. As long as this paper's authors could find LATER PUBLISHED RESEARCH showing this stuff was wrong, that is the scientific method. In fact if, say, 98 and 4/100's of papers were shown to be right later, I would smell something in the woodpile.
The meat of the article is the bias about reporting negative results. This is not a secret.
In regard to something like climate research, really it doesn't apply. but if you take the premise, it would generally bolster the 1000's (+) of papers over the years that show consistent effects and generally put more shadow on the couple showing otherwise. It would mean the papers the doubters bring up are wrong with even more percentage, since these are the papers with no validated mechanisms and generally many defects which immediately get pointed out. That is we would expect some wrong or null correlations to pop-up, given this paper and shouldn't put much support to isolated work that is not buttoned down to the max.
How long until some researcher releases a study showing that Dr. Ioannidis' research findings are themselves wrong?
Who needs a study? Simply reading the article shows that he has fallen precisely into the trap that he is complaining about i.e. overstating his results. He forgets one very simple point: not all science is medicine/biology.
As a particle physicist I would strongly disagree with his conclusions, at least as applied to experimental particle physics. It is certainly true that some papers turn out to be wrong but this is rare and usually ends up as a 'big thing' in the field. Outside my field I'd be very surprised if the majority of physics or even chemistry papers turn out to be wrong (but I certainly not a chemist so this is just my impression).
As for medicine I can certainly see that they have a problem. Afterall how many times have we been told "don't eat X/do Y it is bad for you" only later to find out that actually it isn't half as bad as they thought and may even have benefits? Just because a lot of medical research is often flawed does not mean that all of science has the problem on the same scale.
So, Dr. Ioannidis either show us some data from chemistry, maths and physics or stop complaining that all of science has a problem on this scale. From where I stand your evidence points to a problem with bioscience/medical research only.
Well I suspect you didn't do any math.
If 1/3 = 33.33% is refuted and 1/3 of the remaining 2/3 and so on, then eventually 1/2 = 50% will be refuted.
The formula for this is p/(1-p) cf. http://en.wikipedia.org/wiki/Power_series
Hey don't blame me, IANAB
His study merely observes the obvious.
Scientific consensus in any field is reached by research, which is then published and subsequently challenged by other scientists.
At the point of initial research, the idea being tested is merely a hypothesis, otherwise known as "educated conjecture".
In the case of medicine, there are numerous uncontrollable variables which lead to a high degree of error in small case studies. (the placebo effect, environmental factors, the variable resilience of each patient, etc)
Global warming does not fall under this. It has been researched, and retested, and re-challenged numerous times. The resulting climate change predictions are presented in confidence intervals, and the usual results brought to policy makers are conservative figures from the lower third of that interval, which are still quite scary.
VLC FOR MAC IS DYING! IF YOU DEVELOP, PLEASE SAVE IT!!
The problem is not even necessarily that any of the studies themselves are flaws, just that only the "exciting" results see the light of day. If 100 groups perform the same study, you would expect 10 of them to reject the null hypothesis with 90% confidence even if it were true. Now if only those studies that reject are published, it looks like 10 studies all rejected the null hypothesis with 90% confidence. On the other hand a meta-study that took into account the work of all 100 groups would reveal no statistically significant results.
When did "almost a third..." become "most?"
Like missing car keys, the correct/best theory of [_____] is in the last place we look -- because once we find them, we stop looking!
"I can't imagine how things could get any worse!" (some guy) "That could just be failure of imaginatioÂn on your p
So, a paper claiming that small sample sizes lead to wrong conclusions is based on analysing a sample of only 49 other papers? The mind boggles with the self-applicability ...
what research findings were published to promote medicines in order to further the corporate profits (greed)...
Politics is Treachery, Religion is Brainwashing
Correct is that two thirds eventually will be refuted(like my first post is refuted) 1/3 + 2/9 + 4/27 = 1/3 *(1 + 2/3 + 4/9 + ..) = 1/3 * (2/3 / (1-2/3)) = 1/3 * 2 = 2/3 = 66.67%
Hey don't blame me, IANAB
This sounds like a research paper written by an MBA.
0) The scientific method is a little more complicated than "trying things and observing what happens".
1) Procedures are scientific, not explanations or conclusions.
2) In science, repeatability trumps everything, and ALL RESEARCH ULTIMATELY HAS TO PASS THE REPEATABILITY TEST. If your results cannot be repeated, something was wrong with it.
3) The commercial medical field in particular has a history of forwarding the first round of research results straight through the marketing department instead of treating it as opportunities for further investigation.
-G.
P.S. No, I didn't RTFA. ...I'm not going to, either. ...On principle! ...SCIENTIFIC principle! ...Not gonna explain what I mean, either! ... shoulda went with ////slashies!////, 'P.'s are hard to type!
P.P.S.
P.P.P.S.
P.P.P.P.S.
P.P.P.P.P.S.
P.P.P.P.P.P.S.
1/3 = Most?
Oh, the irony!
The MAFIAA is a bunch of mindless jerks who will be the first up against the wall when the revolution comes
a study of 49 papers ... almost a third of the papers had been refuted by other studies
"Almost 1/3 of 49" = "Most Published Research Findings"
Nice math, Economist. I suppose Dr. Ioannidis does have something to gain by publishing inaccurate journal articles, though...
Wow, headline grabbing, potential break-through scientific theories get a lot of scrutiny and citations from fellow scientists, and many of the theories fail the test of peer review. Daring theories are important for the further advances of science, but by definition most of them will fail.
--- guns don't kill people, people with guns kill people ---
and then shot.
It's important to identify a problem no matter what, but are any of these biases fixable? I would argue that some of them, specifically the bias toward positive results, is not fixable and is inherent to how science works.
To quote one of the articles "...negative results are potentially just as informative as positive results, if not as exciting." But negative results often require much much more verification than positive results, if they can be verified at all, and are limited in how much they can tell you. The antidepressant studies mentioned, a negative result, that the antidepressants did nothing, only tells you that in the patients tested, the doses tested did not give you a noticeable positive result. Publishing a negative result on that would have very limited conclusions. The next year, they could find that doubling the dose was actually effective, making the writeup of the earlier negative result pointless and even more trivial. A waste of time, plus then you've published saying your own product doesn't work.
Negative results get even more pointless in other fields. If someone does a mutagenesis screen for a particular defect in C. elegans, and doesn't find any mutants affecting that, it could be noteworthy, indicating that any genes affecting that process were so vital that when you took one away you didn't get a worm at all, or it could just be luck that genes affecting the process were never mutated, or the researcher didn't do it correctly, or all genes involved were redundant, or some combination. What conclusions could you draw from that? It would be a negative result that would be nigh impossible to tell anything from. Without any positive hits, you could go to the trouble of making sure you did it correctly, but you're not going to make sure every gene got hit at least once, that would be impossible.
In still other cases, a negative result is often retrospectively found to be the fault of the researcher. Who wants to publish something that is basically telling your peers how dumb you are?
There's also that it requires a lot of extra work to make sure it's a negative rather than a null result. Usually when I hit a negative result, my inclination is to see if I did it wrong by repeating the experiment if possible, if it comes up negative again I usually take a different approach, if that also gets a negative result I re-evaluate. I don't ever do all the other supporting experiments that would be needed to convince a reviewer it's a real negative result. If I use an RNAi construct to knock down a gene, and it doesn't do what I'm expecting or anything else interesting, I don't verify the gene is actually knocked down, since that's more effort that would probably be a waste. I'm definitely taking a risk that it's a real result, but it's hard to prove a negative and there's also less motivation to do so.
The limited ability to make positive conclusions about negative results also limits where they could be published. There is a journal for negative results, but a publication there is not something I personally would put on a CV.
So while it is interesting that a bias against negative results may be throwing us off, it's not very usefull knowledge, because I don't see us able to do anything about it.
Now, I do wonder how many of my fellow scientists have ever published a paper with an approach, which they invalidate with their own validation in the same paper. I have the feeling the peer review process puts to much emphasize on positive results. Author perception is that reviewers easily ignore the fact that negative results are also significant. Hence, it's not worthwhile to publish something that doesn't work. However, imho knowing why something doesn't work is just as important as knowing how it should work...
Because they're PUBLISHED! They're after getting funding and positive reviews of the findings, and if that means fibbing them then so be it.
Why is this news?
So the real news here is that Wikipedia is about as accurate as science journals.
Twinstiq, game news
(but I certainly not a chemist so this is just my impression).
Having done some research in theoretical chemistry, and read the (admittedly few) papers related to my work, I have so far only found one mistake, and it is disputably incorrect (a theoretician vs. experimentalist argument).
Astounding! Science, based on principles of falsification, experimentation and a culture of building on the work of others, continues to advance! Amazing! Each advance proves previous work wrong! For science to *really* hit the nail on the head, it clearly needs to stagnate, thus ceasing to refine old works.
http://melbournephilosophy.com/
If medical journals are publishing bad research, that's a problem with medicine. Real science doesn't take three years to peer review and real science is usually (iteratively) correct. Medical research is more similar to social science these days than physical science. The reliance on large samples of different people to smear out inconsistencies in data is a mark of poor understanding of the system they're studying. It's fine to use this method, but it needs to be differentiated from science in general.
If it's truly the case that most medical research is wrong, then medical doctors need to sit down with economists, psychologists and other social scientists who understand how to properly evaluate and weight human data.
This latter group are far more prone to different interpretation by different people.
Engineering is the art of compromise.
Alan Goldhammer, deputy vice president for regulatory affairs at the Pharmaceutical Research and Manufacturers of America, said the new study neglected to mention that industry and government had already taken steps to make clinical trial information more transparent. "This is all based on data from before 2004, and since then we've put to rest the myth that companies have anything to hide," he said.
All I can say to that is :
ROFL
LMAO
Hillarious!
"Most Published Research Findings Are False"
Does that apply to this published research finding too?
You seem to have NO idea at all about what you are saying. Global warming is based on very exact scientific studies, where "faith" is needed only in that one believes that there exists a reality around us that follows some self-consistent laws.
The studies that present exactly the effects that Dr. Ioannidis mentions are the opposing view, those that pretend to disprove the existence of global warming. Scientist, all over the world, are very strongly in agreement that global warming is an indisputable fact.
Let me get this straight. A scientist posts a paper saying that the most published papers are often quickly refuted. Scientist's paper ends up on slashdot. Paper becomes thoroughly published.
What comes next again?
While Ioannidis, the author of the original work that was discussed here may be correct that a large fraction of a very specific type of clinical research findings, are incorrect, there is no reason to believe (from his published work) that "most published research findings are false". Most of the ones he looked at were not reproduced, but they had well understood limitations. My papers, I can assure you, are only incorrect about 10% of the time.
Science in general isn't about "publishing what is right" but rather creating a network of accountability in the form of methods, ideas, data, procedures, etc. so others can try and reproduce and critique the results. Even if the published results are shown to be incorrect by other studies, this does not mean the system is broken. The scientific process is an iterative, self correcting, one. However, if after many years and many studies, a particular field fails to converge on an accepted baseline conclusions, there is a good chance something is wrong (you may even be doing pseudoscience).
i\hbar\dot{\psi}=\hat{H}\psi
Indeed in my field (a sub-area of computer science) people are usually highly skeptical of any supposedly important new result in the field that was first published in one of the highly prestigious but generalist journals, like Nature or Science. These often end up being, if not outright wrong, at the very least seriously over-extending their claims or the importance of their claims, in a way that would never get them published in a specialized journal filled with an editorial board who were actually experts in the specific area in question.
This is only exacerbated by the fact that, because generalist publications know they don't have expertise in every specialized area on staff, they often ask the authors to suggest potential reviewers of their own papers. Of course, authors are likely to suggest reviewers who they think will like the paper, not the ones who would give it a grilling.
I think the interest of this particular study is not so much that a lot of science turns out to be wrong, but that a lot of the most prestigious publication venues turned out to be wrong more often.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Ioannidis wrote a previous article, titled "Why Most Published Research Findings Are False." A very provocative title, one that practically begged the scientific community to read it just to accumulate a laundry list of holes in his argument. A good marketing maneuver. It may or may not be true that most published research findings are false, but the article certainly didn't demonstrate it. Within the context of the discourse he intiated, that would have to be viewed as a kind of willful stupidity, or perhaps marketing brilliance. After all, if the same journal received ten equally well argued articles with titles like "why most research is pretty good," they would still of course prefer to publish his. This truism seques nicely into the new article (on which he is not first author), which is titled, "Why Current Publication Practices May Distort Science." Use of the word "may" is quite helpful here. Does the article live up to its title? It may not be convincing, but it would be even less so without the word "may."
Afterall how many times have we been told "don't eat X/do Y it is bad for you" only later to find out that actually it isn't half as bad as they thought and may even have benefits? Just because a lot of medical research is often flawed does not mean that all of science has the problem on the same scale.
The problem here is that the popular press always report the very latest 'finding' in what is a complex field. Yet we should know that not only in medicine, but in virtually all experimental sciences, a single paper is not sufficient to establish some new profound truth.
Dr Ioannidis' largest problem is that he thinks he has identified a problem. There isn't one. This is how science is supposed to work! We publish methodologies so that the work can be replicated by other teams. Some findings survive futher scrutiny, some don't. The "hotter" the field, the less you are going to rely on the latest single study, no?
So he's found 1/3 of studies were refuted, but later work. Great, they were refuted, what's the problem? And how do we move from that to the conclusion that "most" scientific papers (even outside the hotter fields of bio-medical research) are wrong. And what about looking at outcomes? The advances of medicine even in my lifetime are astounding, this is hardly the result of a system that isn't working!
Better to be despised for too anxious apprehensions, than ruined by too confident a security. --Edmund Burke
Many aspects of astronomy is definitely in the marginal group. That's why the Hubble Constant has converged somewhat, but is still in a state of flux and debate.
Engineering is the art of compromise.
The issues on global warming that are in dispute are numerous... we keep hearing predictions from models that have not been proven to have any predictive powers, and they keep getting more alarmist, and the increasingly ridiculous claims that every example of bad weather is a function of global warming. The issue is the "hockey stick" part of the forward feedback loop... that's the claim that because events will create forward feedback, we will hit a point in a few years where it isn't preventable, because even if we never emitted another CO2 gas, the forward feedback would be self sustaining.
Most things in the universe have negative feedback... The issue with global warming is we know that the current models show this forward feedback, but we KNOW that the models are incomplete. Are we missing significant variables that would create a feedback loop? It seems reasonable to wonder if something will happen with the higher CO2 levels that will cause a negative affect on global warming.
The consequences of GW are dire, and it's a real concern. But the scientific credibility is NOT enhanced by the political advocates calling for the same policies that their fellow ideologues called for for different reasons before, the celebrities weighing in, or the silly exaggerated movies.
Theory of evolution has been tested and demonstrated in small areas with smaller organisms. Theory of evolution is also a concept more than a specific theory, making it easier to demonstrate pieces... yes, evolutionary biology shows the process of small organisms... not the same thing.
Increased CO2 -> increased heat, that's the easy claim
Increased heat -> increased CO2 and there is no way of stopping it is the stranger claim
Perhaps we'll see a spread of CO2 absorbing plants move out of tropical areas to other zones as temperatures change, who knows, but there are plenty of areas for negative feedback, and only time will tell.
...this research was published, right?
Who's to say that the papers refuting this research are correct? It seems to be taken for granted that the dissenting papers are correct, and thus the original papers are wrong. It seems likely that the refuting papers may be wrong, or that there are complex situations in which both papers are correct (to differing degrees).
Very true - but it doesn't matter - if the refuting papers are wrong, then there is still the same number of wrong papers out there...
Say goodbye to your karma and hello to /.'s new communist overlords....
Media that can be recorded and distributed can be recorded and distributed.
-kfg
It depends on how you count "most". Science can move forward as long as the really important things that everyone relies on are at least close to correct. Most of the actual results could still be wrong, or seriously off, and you could still get your job done.
In fact, regardless of what they might say in public, a lot of experimental scientists seem to operate under the assumption that most of the relevant published research is wrong, or at least far enough off to be useless. When you want to synthesize some molecule, you don't look up all the published data on mechanisms, piece it together using some first-principles logic, and then assume it's going to work. Usually, it doesn't. Sometimes this is because you messed up, but often it was because the stuff in the literature isn't entirely correct. The very basics are correct: our understanding of how chemical reactions work is not likely to be seriously off. But a lot of the details, like how two particular substances will react in the presence of a third in a particular set of circumstances, need to be taken with a bit of skepticism, especially if you're trying to rely on the claimed implications of an experiment (i.e. the general scientific conclusions it's supposed to imply), rather than literally replicating the exact experiment under the exact same conditions. You only reduce the skepticism a bit if lots and lots of papers have been published on the subject, so even if "most" are wrong, 10 saying the same thing still gives you a good chance the result is right.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Maybe science needs to be knocked down a few pegs in the public perception? As far back as the 50s and atomic power (and earlier) the public has been told that science will solve all their problems, has been built up as infallible by the media and by ourselves, so any signs of fallibility are instantly descended upon and torn apart by the public, held up as unacceptable when people in the field know that trial and error really is just how science goes.
Not sure exactly what I'm getting at here, except that we scientists are surely partly to blame for this and if we don't bother to try and fix people's understanding of how science works, well, not one else is going to care enough to bother.
You said, "So, Dr. Ioannidis either show us some data from chemistry, maths and physics or stop complaining that all of science has a problem on this scale."
I'm sympathetic to the direction you are going, but I don't agree completely.
The problem is due to being able to get extra money by exaggerating claims. The problem is in every area of science, in my experience. If there is no chance to get more money by exaggerating claims, then I agree, the problem seems minimal.
In computing, claims about "Artificial Intelligence" have been extremely exaggerated.
In physics, there are those who claim they may have found a method of cold nuclear fusion. Search for Sonofusion, for example, fusion that is caused by extremely intense ultrasonic sound. Some of those claims are exaggerated, or there are omissions of the limitations.
It's a lot easier to come up with a theory that doesn't match the data or produce data that doesn't match a theory than it is to get theory and data to match. Likewise, it's a lot easier to try to accomplish something and fail than it is to try to accomplish something and succeed.
For example, I have conducted experiments proving that the following things do not cure cancer: reading Slashdot, eating ice cream, playing World of Warcraft, watching The Muppet Show. But if I found one thing that did then I would be rich and famous.
So the reason that positive results dominate the literature is because those are the ones that are scarce and valuable. Publishing negative results would swamp reviewers, publishers, and readers in a mass of unusable anecdotes.
On the other hand, if a theory is already accepted or some agent is believed to work, then research showing negative results is new and valuable.
yes you can.
Numerous independent climatology models (we're talking virtually every accredited university and think tank on earth with enough resources) based on hundreds of thousands of years of data from geological, oceanographic, and ice core samples, run through supercomputers millions of times.
The only ones that "disagree" are tied to oil companies.
VLC FOR MAC IS DYING! IF YOU DEVELOP, PLEASE SAVE IT!!
I read a bunch of dental papers recently, and discovered something rather disturbing. A good 90% or more of studies for dental procedures do NOT use any control group. They all say, "we did X and got the expected result." There is no checking whether the procedure is better than other procedures or even doing nothing at all.
Something to think about next time someone you know is told they need wisdom teeth extracted or some orthodontic appliance.
Cleary, 100% of UNPUBLISHED papers (un-like those not in the study) are un-refuted, and do so with a 100% sucess rate!. (you can't refute it if you can't read it! ) ...so that must make them all right! So everyone should stop publishing! now. Infact lets ban all forms of written media....
Well, I don't get the impression it's unique to science. It's just run of the mill not questioning experts because of apathy/ignorance. Look at economists, at least when the economy was working and didn't affect joe plumber personally. The news commonly took their word as infallible. "Economists said today that housing prices would continue their climb. Quote 'What goes up continues going up basically forever, so invest in my mortgage company.'"
And they are always truncating what politicians say. If what they say can't be summed up in 10 words, it's likely to not be reported. Look at Al Gore and the "He said he created the internet!" The man clearly did not think he created the internet, nor did he say anything that mean "I made the internet." If I remember correctly, he was saying he helped fund the initiative that gave rise to the internet, it got taken out of context. So while it would be best if the public understood the process a little better and bothered to understand the full picture rather than just an oversimplified version of what they want to hear, it could be worse. As such, only creationists are really out there trying to oversimplify things to use them against us.
I think the easiest way to improve things though is to start early, get good science teachers for high school and earlier. Of course, I'm not volunteering, so...
Dr Ioannidis' largest problem is that he thinks he has identified a problem. There isn't one. This is how science is supposed to work! We publish methodologies so that the work can be replicated by other teams.
Yes the purpose of publishing is so others can reproduce your work...but if you have 66% of published results being found to be wrong you have a huge problem! The point of publishing is to allow for verification but you should expect that a good majority of published results are correct because it is assumed that some care and thought has gone into the work before it is published. Journals are not supposed to be a running brain dump of researchers in the field. Or, put another way, why is it that no other branch of science has such a huge error rate in its publications? I'm not ruling out that there is a very good reason for this but so far I have not heard it.
Please replace "false" with "incorrect". The word "false" implies deliberate fraud,
To anyone doing boolean logic, "false" implies "not true". And I'm sure you could drive any programmer bananas by disallowing these traditional terms (under threat of compiler warnings and calling the PC police) and requiring the politically correct versions "correct" and "incorrect" instead.
"Studies have shown that ice-cream is good for you.
I heard it on the radio this morning!
Full of calcium..."
One swallow does not a fellatrix make
I can't see what is so surprising here. Basically when you do research, you are groping in darkness - after all, it wouldn't really be worth doing if the results were already known, would it? Approaching a new problem is a bit like looking at the notorious elephant through a keyhole; different people will have different guesses as to what it is and most will be wrong, until at some point enough observations are made and you can construct a more complete picture.
When you publish scientific articles you don't claim that "THIS IS THE TRUTH" - you are merely putting forward your opinion and then somebody else comes along and says "No, because ...". And even the articles with the "wrong" results are valuable, because they tell us that this particular interpretation is not the right one. It can take a lot of false turns before you find the right way through a maze, and in fact it tells us something about the generally high quality of research that we are not seeing about 90% wrong results.
Look how effective it is on comments posted to slashdot!
Do you have any idea how sensitive these models are to their dozens of control variables? When each variable plays out over an entire interconnected planet, and the model runs for twenty or fifty years, even tiny changes are very powerful. Change one setting, such as the carbon sequestration rate of the ocean as a function of its thermocline, from (say) 0.0336 grams/square-meter/day to 0.0337, and suddenly the whole future changes. And so I wonder how sure you are about that 0.0336 setting, or all the other settings.
You are probably quick the criticize the Drake equation for the very same vulnerability.
FATMOUSE + YOU = FATMOUSE
Should I be reassured that (eventually) the scientific method works?
Or should I be really worried about the remaining two thirds?
At the bottom of the
Whether Dr. Ioannidis and others need to check other disciplines of science to claim that "most published research" depends heavily on how much publications other science disciplines (besides biology/medical sciences) contributed to the total. If the majority of the publications in top-tier journals are in biology/medical sciences, I don't see why the author's paper title is not valid. Of course I don't have the data.
How do we know his study is true if most studies are false?!
It means that 1) science is good at finding mistakes, and 2) we are learning a lot, as mistakes are the source of knowledge.
Solid evidence backs global warming and man made global warming. The idea of 'repression' of countering science is a lie, straight out.
;)
If you want to claim man made global warming doesn't exist you need evidence, and it has to be at least as solid as the other evidence to be take seriously. Sure making a claim against the popular wisdom is somewhat harder because people go through your data and methods more carefully, but if the rigor is there it will get through.
Being a very unpopular theory never stopped quantum mechanics.
The 'global warming is not man-made' crowd were the 'global warming does not exist' crowd 10+ years ago. They are in trouble their predictions and methods have failed so far and they have not come up with solid research or data compared to others in the field.
You don't need to claim global warming exists to get published/tenure, you need solid research. Reality has been know to have a severe biasing effect on competing theories, however
And stay away from the global cooling theory BS. A single paper, whose conclusions were later contradicted by its own author (because his calculations underestimated the warming effects of CO2), is not 'comparable' to the current theories. Scientists did not react to it in the same way the media does.
Negative results don't sell meds.
Big pharma is about selling; treating and curing are just a bonus.
It must have been something you assimilated. . . .
No, I'm not accounting for refuting papers being refuted.
Which kind of refutes my argument.
Actually following your idea, one could see it another way: According to the article, .. = 2/3 * (1+1/9+1/81 ..) = 2/3 + (1/9 / (8/9)) = 2/3 + 1/8 = 19/24 = about 79%
2/3 don't get refuted(if 1/3 is refuted). Of the remaining 1/3, again 1/3 of the refutations is refuted, but only 2/3 of these are not refuted in the next cycle. And presumably it goes on like this. This leaves us with:
2/3 + 2/27 + 2/243
But lets follow the other model proposed by Jorgensen, then let x = number of true papers, y= number of refuted papers, and dp be the number of completely new papers, then leaving any sanity checks aside
(A)dx = -1/3 x + dp + 1/3 y
(B)dy = 1/3 x - 1/3 y
Which for dp = 0 is in equilibrum for x=y (50:50). Now I'll just do a leap of faith and assume that the system will actually decay towards the equilibrum, then always 50%+p with p depending on dp:dx will be papers considered to be true.
This actually means that the 1/3rd mentioned in the study mentioned by slashdot is a sensible lower number bound or worst case, since a paper should have at least a 50% chance to fulfill a "random" yes or no statement.
Hey don't blame me, IANAB
Something like 99% of all species that have existed on the planet Earth are now extinct. So you take that statistical trick and publish a headline that screams "99% of All Species Will Go Extinct." Technically it's true. I mean, I think we can be pretty sure that someday 100% of species will go extinct, in the heat death of the universe, the big crunch, the end of the Matrix, etc.
Likewise, since all scientific knowledge is provisional, it's not shocking that the vast majority of scientific statements ever made turned out to be wrong. That's the whole point of science, isn't? That doesn't mean they weren't useful approximations at the time though. (See: Newton)
Build a man a fire, he's warm for one night. Set him on fire, and he's warm for the rest of his life.
I would just like to point out there is a reason a doctors office is named Medical PRACTICE. Because nothing in medicine is 100%.