Slashdot Mirror
Registered Clinical Trials Make Positive Findings Vanish
schwit1 writes: The requirement that medical researchers register in detail the methods they intend to use in their clinical trials, both to record their data as well as document their outcomes, caused a significant drop in trials producing positive results. From Nature: "The study found that in a sample of 55 large trials testing heart-disease treatments, 57% of those published before 2000 reported positive effects from the treatments. But that figure plunged to just 8% in studies that were conducted after 2000. Study author Veronica Irvin, a health scientist at Oregon State University in Corvallis, says this suggests that registering clinical studies is leading to more rigorous research. Writing on his NeuroLogica Blog, neurologist Steven Novella of Yale University in New Haven, Connecticut, called the study "encouraging" but also "a bit frightening" because it casts doubt on previous positive results."
In other words, before they were required to document their methods, research into new drugs or treatments would prove the success of those drugs or treatment more than half the time. Once they had to document their research methods, however, the drugs or treatments being tested almost never worked. The article also reveals a failure of the medical research community to confirm their earlier positive results. It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
In other words, before they were required to document their methods, research into new drugs or treatments would prove the success of those drugs or treatment more than half the time. Once they had to document their research methods, however, the drugs or treatments being tested almost never worked. The article also reveals a failure of the medical research community to confirm their earlier positive results. It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
Similar issues have shown up in other fields. Psychology has had serious faillures to replicate many major studies http://www.slate.com/articles/health_and_science/science/2014/07/replication_controversy_in_psychology_bullying_file_drawer_effect_blog_posts.html and when there have been attempts to replicate them they have often not gotten the same results. And there are very similar problems in education https://www.insidehighered.com/news/2014/08/14/almost-no-education-research-replicated-new-article-shows. Pre-registration of experiments is important, but it would also help a lot if there were journals dedicated to replication and also if academia took replication more seriously: I know people who are tenure track who haven't tried to replicate some studies because it doesn't look as good for tenure promotion to just replicate something rather than do something new. There are serious cultural issues that need to change.
It appears the medical research field has forgotten this basic tenet of science:
It's almost as if it isn't science at all, but rather advertising, where the target audience is a government agency that gives the company permission to transfer the product to their other advertising division who then advertise it to doctors and the public. What percentage of these clinical trials are trials of something not destined to produce wealth for an organization if the results of the trial are positive? When a wealth generating organization approves expenditure of the large amount of money to do a clinical trial, I'm sure it also adds extra management, or transfers management of the research so as to help the results of the trial be more profitable than just doing science ever could be. I'm thankful that this requirement to register these trials exists.
It's not even a good description of what was happening. Positive clinical trials were published, but if a company had invested in a drug then it would often do a dozen clinical trials. Of these, a few (due to normal statistical variation) would show that there was some correlation with whatever they were looking for, and those are the ones that they'd publish. It's similar to the stock market scam, where you create a dozen funds, trade them at random, and then cherry-pick the one that had done a lot better than the market, claim that it's due to your brilliance and then ask for investors (charging a hefty administration fee to each one).
I am TheRaven on Soylent News
Similar issues have shown up in other fields.
Indeed. The biggest issue is statistical ignorance, but even people with a decent amount of training in stats can be fooled if they want to find a particular result. Anyhow, whenever things like this come out, everyone always thinks it's about scientists who manipulate data deliberately. While that happens, it's more often just researchers who "try things out" after collecting data and notice a pattern (unintentionally skewing things). If they have to declare methods and statistical tests beforehand, it's harder to make these errors.
A few months back, I happened upon a very useful guide to the problems in modern scientific publication, which can be found partly online here. I ended up buying the print edition, and the sheer number of examples of completely bogus research ending up being accepted in various scientific fields due to erroneous stats and various biases that creep into the publication process... well, it's just shocking. Seriously.
As the book notes, the other problem is that even finding these errors is incredibly time-consuming and labor-intensive. I specifically remember one case where a new oncology test was proposed by Duke researchers and seemed to have great results. This case eventually became so infamous that it was reported on in the popular media.
Anyhow, basically they had a couple independent statisticians analyze the work (where they found HUGE numbers of problems in mislabeled data, mistakes in analysis and basic computation, etc., which appear par for the course in many labs, if you believe the studies on this stuff in the book). Ultimately, estimates are that it took TWO THOUSANDS HOURS of work for these independent statisticians to complete their analysis and render a verdict.
And once they did this, the statisticians tried to publish it -- but major journals didn't want it. Groundbreaking results are much more interesting that tedious statistical analysis. The National Cancer Institute caught wind of the problems and initiated an independent review, which found no errors (probably because the review was done by cancer experts, not stats experts, and they hadn't been giving the stats analysis done by the other researchers).
The only reason any of this ever really got much attention is because one of the lead researchers was accused of falsifying some aspects of his resume, which led to people actually going back and questioning his papers.
The book is full of stories like this, though, as well as citations of analyses of how many journal articles in various fields suffer from serious statistical problems.
It's all really scary when you start realizing how much bogus research is out there... most of it completely unintentional, and most of it passing peer review because it follows the field's "standard methodologies."
Americans are actually considerably less materialistic than other nations: http://www.ipsos-na.com/news-p...
Money to most Americans is a means to an end, a way to pursue the values and ideals they actually care about.
Well, Georg Simmel certainly got his wish, because about a decade after his death, his country made community, fairness, and equality the center of its value system, under the NSDAP party program. And it did it again two decades later in the GDR. And it was a resounding success: people indeed stopped caring about money, because they ended up caring mostly about not starving, not getting shot, and not getting gassed. Germany's tradition of academic philosophy that Simmal was part of bears a great deal of the blame.
And, of course, Simmel himself came from a well-off family and never had to worry about a lack of money limiting what he could do. His teachings are the philosophical equivalent of "let them eat cake".
You couldn't publish something like that in a major peer-reviewed medical journal today.
Anil Potti, Sheng Wang, and many others managed to do it just fine. Where drug companies get into trouble it is usually when they hire people at hospitals/companies/universities to run sites for a clinical trial. The contracts often create huge conflicts of interest that can skew the results of the trial.