Slashdot Mirror
Registered Clinical Trials Make Positive Findings Vanish
schwit1 writes: The requirement that medical researchers register in detail the methods they intend to use in their clinical trials, both to record their data as well as document their outcomes, caused a significant drop in trials producing positive results. From Nature: "The study found that in a sample of 55 large trials testing heart-disease treatments, 57% of those published before 2000 reported positive effects from the treatments. But that figure plunged to just 8% in studies that were conducted after 2000. Study author Veronica Irvin, a health scientist at Oregon State University in Corvallis, says this suggests that registering clinical studies is leading to more rigorous research. Writing on his NeuroLogica Blog, neurologist Steven Novella of Yale University in New Haven, Connecticut, called the study "encouraging" but also "a bit frightening" because it casts doubt on previous positive results."
In other words, before they were required to document their methods, research into new drugs or treatments would prove the success of those drugs or treatment more than half the time. Once they had to document their research methods, however, the drugs or treatments being tested almost never worked. The article also reveals a failure of the medical research community to confirm their earlier positive results. It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
In other words, before they were required to document their methods, research into new drugs or treatments would prove the success of those drugs or treatment more than half the time. Once they had to document their research methods, however, the drugs or treatments being tested almost never worked. The article also reveals a failure of the medical research community to confirm their earlier positive results. It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
Similar issues have shown up in other fields. Psychology has had serious faillures to replicate many major studies http://www.slate.com/articles/health_and_science/science/2014/07/replication_controversy_in_psychology_bullying_file_drawer_effect_blog_posts.html and when there have been attempts to replicate them they have often not gotten the same results. And there are very similar problems in education https://www.insidehighered.com/news/2014/08/14/almost-no-education-research-replicated-new-article-shows. Pre-registration of experiments is important, but it would also help a lot if there were journals dedicated to replication and also if academia took replication more seriously: I know people who are tenure track who haven't tried to replicate some studies because it doesn't look as good for tenure promotion to just replicate something rather than do something new. There are serious cultural issues that need to change.
No, it is a pretty accurate summary of what is happening. You focus post-hoc on where you got a good result. Say for example you want to test a new anti-diabetes drug. Does the drug work in the general population? Well, data doesn't support that. So then you look at subgroups. Does your data show success in say just men or just women? What about black men? Black women? White women? Etc. This isn't the only serious problem, sometimes one can choose which statistical tests to do or how to compensate for complicating factors. If you have enough choices you can make anything looks successful.
The basic flaw is worse. They didn't just run one test, find the results they wanted and go with it. They ran a test with only an idea of what they wanted, then took all the results they got and picked out ones that were positive for conditions or treatments they could go with. It's like going into a test for a drug to treat heart attacks, finding that it doesn't do anything for heart attacks but does seem to lower cholesterol levels, and announcing that the trials of your new cholesterol medication were positive.
Having to declare up front what their goals are destroys the ability to cherry-pick like this. What we're seeing with the drop in positive results isn't so much the difference in clinical effectiveness of the drugs but the dragging into the spotlight of the pharma companies' ability to predict what their drugs will do and how well they'll do them. There's a very interesting blog here that covers a lot of this, and one conclusion that keeps coming up again and again is that medical biochemists and researchers don't really have a good way of predicting from lab results what a compound will do in a live human. It also highlights fairly often how the drug companies will keep pushing a drug through trials even though the results aren't encouraging. It's a common attitude in business and finance, that now that you've invested this much money in something you have to get some return out of it to justify the cost. It's also a common failing in gambling, the belief that now that you're in the hole you have to dig yourself out somehow. But in gambling, if you're holding a bad hand your best bet is to fold. Don't worry about how much you've already got in the pot, it's already lost. Fold and cut your losses before you throw any more money away. Drug companies are notoriously bad at making that decision to walk away. They're also notoriously bad at dealing with a field where there aren't many good rules you can follow to get results. MBAs like process and procedure and predictable results, and right now biochemical research is in a situation where the new stuff is all likely out in areas where there isn't a lot of research, there isn't a good map of the territory and you're going to be doing a lot of "poke it with a pointy stick and let's see what it does" work.
Correct. Even if you specify your subgroups beforehand in the experimental design, you still need to modify your interpretation of statistical significance (downward) to account for the consideration of multiple hypotheses. If you're going on a fishing expedition by identifying subgroups post hoc, then you ideally need to base this correction on the potentially large number of conceivable subgroups that are available to be drawn. It's very hard to achieve real significance under those circumstances. On the other hand, you might find a subgroup result suggestive and conduct a separate follow-on study to test it independently; that's perfectly legitimate.
when did academia get taken over by idiots? now that the gatekeepers are dumb we're fucked and all the smart people just go make money.
Well, actually, academia wasn't taken over by idiots. It was taken over - infiltrated - by smart people who were much more interested in money, prestige and power than in scientific truth.
That's the unfortunate fact about American culture. The USA was founded on the belief that all people (well, all white males of a certain age with property, but that's a small detail) should be treated alike. No titles of royalty, nobility or gentry. No class system. No special distinctions or honours.
The result, which became obvious very early on, was a society in which the only value was money. And money, it turns out, corrodes everything that is honest, decent and worthwhile. Now that culture is flooding the rest of the world - although some nations have done their valiant best to build dykes to keep it out.
"As the sociologist Georg Simmel wrote over a century ago, if you make money the center of your value system, then finally you have no value system, because money is not a value".
– Morris Berman, “The Moral Order”, Counterpunch 8-10 February 2013. http://www.counterpunch.org/20...
I am sure that there are many other solipsists out there.
Similar issues have shown up in other fields.
Indeed. The biggest issue is statistical ignorance, but even people with a decent amount of training in stats can be fooled if they want to find a particular result. Anyhow, whenever things like this come out, everyone always thinks it's about scientists who manipulate data deliberately. While that happens, it's more often just researchers who "try things out" after collecting data and notice a pattern (unintentionally skewing things). If they have to declare methods and statistical tests beforehand, it's harder to make these errors.
A few months back, I happened upon a very useful guide to the problems in modern scientific publication, which can be found partly online here. I ended up buying the print edition, and the sheer number of examples of completely bogus research ending up being accepted in various scientific fields due to erroneous stats and various biases that creep into the publication process... well, it's just shocking. Seriously.
As the book notes, the other problem is that even finding these errors is incredibly time-consuming and labor-intensive. I specifically remember one case where a new oncology test was proposed by Duke researchers and seemed to have great results. This case eventually became so infamous that it was reported on in the popular media.
Anyhow, basically they had a couple independent statisticians analyze the work (where they found HUGE numbers of problems in mislabeled data, mistakes in analysis and basic computation, etc., which appear par for the course in many labs, if you believe the studies on this stuff in the book). Ultimately, estimates are that it took TWO THOUSANDS HOURS of work for these independent statisticians to complete their analysis and render a verdict.
And once they did this, the statisticians tried to publish it -- but major journals didn't want it. Groundbreaking results are much more interesting that tedious statistical analysis. The National Cancer Institute caught wind of the problems and initiated an independent review, which found no errors (probably because the review was done by cancer experts, not stats experts, and they hadn't been giving the stats analysis done by the other researchers).
The only reason any of this ever really got much attention is because one of the lead researchers was accused of falsifying some aspects of his resume, which led to people actually going back and questioning his papers.
The book is full of stories like this, though, as well as citations of analyses of how many journal articles in various fields suffer from serious statistical problems.
It's all really scary when you start realizing how much bogus research is out there... most of it completely unintentional, and most of it passing peer review because it follows the field's "standard methodologies."
You're kidding, right? How often do we have to chant the mantra that climate is not the same as weather. The climate models may suggest that we will see in increase in precipitation, but that doesn't mean that in one specific location that there will be more rain. Also, there are other localised factors with droughts.
None of what was in that article is enough to make the claims that all the models have failed. I also don't understand the point of the article. If 97% of the scientists agree that man has a hand in global warming, it doesn't mean that they agree on all the details. Nor does it mean that any disagreement within the community is proof that the whole thing is a big fat lie.
BS. 'Big Oil' is a red-herring to divert attention away from 'Big Government', whose grants and funding tend to force researchers to become, in effect, lobbyists for political activism in order to 'pay the rent'.
And that is even more BS. There is no proof that there is any "Big Government" that is attempting to control the scientific community, especially when 50% of those people in power are actively against the idea of climate change. Whenever you hear of political interference with the scientific process, how often is it some left-wing conspiracy to force the hand of scientists compared with conservatives attempting to shut down institutions that do research into climate change? Where is the evidence of this giant conspiracy, other than far-right pundits speculating as if it was fact?