Slashdot Mirror
Registered Clinical Trials Make Positive Findings Vanish
schwit1 writes: The requirement that medical researchers register in detail the methods they intend to use in their clinical trials, both to record their data as well as document their outcomes, caused a significant drop in trials producing positive results. From Nature: "The study found that in a sample of 55 large trials testing heart-disease treatments, 57% of those published before 2000 reported positive effects from the treatments. But that figure plunged to just 8% in studies that were conducted after 2000. Study author Veronica Irvin, a health scientist at Oregon State University in Corvallis, says this suggests that registering clinical studies is leading to more rigorous research. Writing on his NeuroLogica Blog, neurologist Steven Novella of Yale University in New Haven, Connecticut, called the study "encouraging" but also "a bit frightening" because it casts doubt on previous positive results."
In other words, before they were required to document their methods, research into new drugs or treatments would prove the success of those drugs or treatment more than half the time. Once they had to document their research methods, however, the drugs or treatments being tested almost never worked. The article also reveals a failure of the medical research community to confirm their earlier positive results. It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
In other words, before they were required to document their methods, research into new drugs or treatments would prove the success of those drugs or treatment more than half the time. Once they had to document their research methods, however, the drugs or treatments being tested almost never worked. The article also reveals a failure of the medical research community to confirm their earlier positive results. It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
The basic flaw is worse. They didn't just run one test, find the results they wanted and go with it. They ran a test with only an idea of what they wanted, then took all the results they got and picked out ones that were positive for conditions or treatments they could go with. It's like going into a test for a drug to treat heart attacks, finding that it doesn't do anything for heart attacks but does seem to lower cholesterol levels, and announcing that the trials of your new cholesterol medication were positive.
Having to declare up front what their goals are destroys the ability to cherry-pick like this. What we're seeing with the drop in positive results isn't so much the difference in clinical effectiveness of the drugs but the dragging into the spotlight of the pharma companies' ability to predict what their drugs will do and how well they'll do them. There's a very interesting blog here that covers a lot of this, and one conclusion that keeps coming up again and again is that medical biochemists and researchers don't really have a good way of predicting from lab results what a compound will do in a live human. It also highlights fairly often how the drug companies will keep pushing a drug through trials even though the results aren't encouraging. It's a common attitude in business and finance, that now that you've invested this much money in something you have to get some return out of it to justify the cost. It's also a common failing in gambling, the belief that now that you're in the hole you have to dig yourself out somehow. But in gambling, if you're holding a bad hand your best bet is to fold. Don't worry about how much you've already got in the pot, it's already lost. Fold and cut your losses before you throw any more money away. Drug companies are notoriously bad at making that decision to walk away. They're also notoriously bad at dealing with a field where there aren't many good rules you can follow to get results. MBAs like process and procedure and predictable results, and right now biochemical research is in a situation where the new stuff is all likely out in areas where there isn't a lot of research, there isn't a good map of the territory and you're going to be doing a lot of "poke it with a pointy stick and let's see what it does" work.
Correct. Even if you specify your subgroups beforehand in the experimental design, you still need to modify your interpretation of statistical significance (downward) to account for the consideration of multiple hypotheses. If you're going on a fishing expedition by identifying subgroups post hoc, then you ideally need to base this correction on the potentially large number of conceivable subgroups that are available to be drawn. It's very hard to achieve real significance under those circumstances. On the other hand, you might find a subgroup result suggestive and conduct a separate follow-on study to test it independently; that's perfectly legitimate.
@Gadget_Guy ...prove that such rigor isn't required of climate science...
>
One way to evaluate scientific hypotheses is to look at what the events they predict, then observe nature to see if the predicted events correspond to reality. In that sense, modern climate science is an epic fail because the "global warming" predicted by their models failed to happen. (Prompting the climate-alarmist "true believers" to switch to 'climate change' (so, up or down, can't lose))
http://www.americanthinker.com...
> ...most climate scientists have done a good job of keeping corporate interests away from their research...
BS. 'Big Oil' is a red-herring to divert attention away from 'Big Government', whose grants and funding tend to force researchers to become, in effect, lobbyists for political activism in order to 'pay the rent'.