Slashdot Mirror
Registered Clinical Trials Make Positive Findings Vanish
schwit1 writes: The requirement that medical researchers register in detail the methods they intend to use in their clinical trials, both to record their data as well as document their outcomes, caused a significant drop in trials producing positive results. From Nature: "The study found that in a sample of 55 large trials testing heart-disease treatments, 57% of those published before 2000 reported positive effects from the treatments. But that figure plunged to just 8% in studies that were conducted after 2000. Study author Veronica Irvin, a health scientist at Oregon State University in Corvallis, says this suggests that registering clinical studies is leading to more rigorous research. Writing on his NeuroLogica Blog, neurologist Steven Novella of Yale University in New Haven, Connecticut, called the study "encouraging" but also "a bit frightening" because it casts doubt on previous positive results."
In other words, before they were required to document their methods, research into new drugs or treatments would prove the success of those drugs or treatment more than half the time. Once they had to document their research methods, however, the drugs or treatments being tested almost never worked. The article also reveals a failure of the medical research community to confirm their earlier positive results. It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
In other words, before they were required to document their methods, research into new drugs or treatments would prove the success of those drugs or treatment more than half the time. Once they had to document their research methods, however, the drugs or treatments being tested almost never worked. The article also reveals a failure of the medical research community to confirm their earlier positive results. It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
Similar issues have shown up in other fields. Psychology has had serious faillures to replicate many major studies http://www.slate.com/articles/health_and_science/science/2014/07/replication_controversy_in_psychology_bullying_file_drawer_effect_blog_posts.html and when there have been attempts to replicate them they have often not gotten the same results. And there are very similar problems in education https://www.insidehighered.com/news/2014/08/14/almost-no-education-research-replicated-new-article-shows. Pre-registration of experiments is important, but it would also help a lot if there were journals dedicated to replication and also if academia took replication more seriously: I know people who are tenure track who haven't tried to replicate some studies because it doesn't look as good for tenure promotion to just replicate something rather than do something new. There are serious cultural issues that need to change.
No, it is a pretty accurate summary of what is happening. You focus post-hoc on where you got a good result. Say for example you want to test a new anti-diabetes drug. Does the drug work in the general population? Well, data doesn't support that. So then you look at subgroups. Does your data show success in say just men or just women? What about black men? Black women? White women? Etc. This isn't the only serious problem, sometimes one can choose which statistical tests to do or how to compensate for complicating factors. If you have enough choices you can make anything looks successful.
The basic flaw is worse. They didn't just run one test, find the results they wanted and go with it. They ran a test with only an idea of what they wanted, then took all the results they got and picked out ones that were positive for conditions or treatments they could go with. It's like going into a test for a drug to treat heart attacks, finding that it doesn't do anything for heart attacks but does seem to lower cholesterol levels, and announcing that the trials of your new cholesterol medication were positive.
Having to declare up front what their goals are destroys the ability to cherry-pick like this. What we're seeing with the drop in positive results isn't so much the difference in clinical effectiveness of the drugs but the dragging into the spotlight of the pharma companies' ability to predict what their drugs will do and how well they'll do them. There's a very interesting blog here that covers a lot of this, and one conclusion that keeps coming up again and again is that medical biochemists and researchers don't really have a good way of predicting from lab results what a compound will do in a live human. It also highlights fairly often how the drug companies will keep pushing a drug through trials even though the results aren't encouraging. It's a common attitude in business and finance, that now that you've invested this much money in something you have to get some return out of it to justify the cost. It's also a common failing in gambling, the belief that now that you're in the hole you have to dig yourself out somehow. But in gambling, if you're holding a bad hand your best bet is to fold. Don't worry about how much you've already got in the pot, it's already lost. Fold and cut your losses before you throw any more money away. Drug companies are notoriously bad at making that decision to walk away. They're also notoriously bad at dealing with a field where there aren't many good rules you can follow to get results. MBAs like process and procedure and predictable results, and right now biochemical research is in a situation where the new stuff is all likely out in areas where there isn't a lot of research, there isn't a good map of the territory and you're going to be doing a lot of "poke it with a pointy stick and let's see what it does" work.
...such rigor were required of Climate Science.
And now the onus is on your to back up what you say and prove that such rigor isn't required of climate science, or that the results are being significantly skewed because of their funding source. Just because what the science says does not match your gut instinct is not proof. Just because it could happen is not proof; we need evidence of inconvenient funding sources being omitted from research, or a meta study showing differing results depending on how the methodology is described.
In fact, I would go so far as to say that most climate scientists have done a good job of keeping corporate interests away from their research. If a scientist wanted to ensure future funding and be free of political interference, then they would take the easy path of downplaying global warming. No scientific research organisation has been defunded or disbanded because they gave evidence that contrary to man's impact on the environment.
It appears the medical research field has forgotten this basic tenet of science:
It's almost as if it isn't science at all, but rather advertising, where the target audience is a government agency that gives the company permission to transfer the product to their other advertising division who then advertise it to doctors and the public. What percentage of these clinical trials are trials of something not destined to produce wealth for an organization if the results of the trial are positive? When a wealth generating organization approves expenditure of the large amount of money to do a clinical trial, I'm sure it also adds extra management, or transfers management of the research so as to help the results of the trial be more profitable than just doing science ever could be. I'm thankful that this requirement to register these trials exists.
I want to show that all cars are red, so I go out and start counting cars. It turns out that not all cars are red, but by choosing every seventeenth car, except if the previous one had a number-plate ending with an odd number OR a number x such that the x-th fibonachi number has a 4 as the second digit, THEN all those cars are red! Dang, that's actually what I meant, forget that thing about all cars red. Publish!
Another serious issue is that it is hard publishing negative results. So even though every researcher is playing by all the rules, they might not know that 500 others have done the same trial before and failed, and this is just the 1 in 500 which accidentally gives a positive result.
Correct. Even if you specify your subgroups beforehand in the experimental design, you still need to modify your interpretation of statistical significance (downward) to account for the consideration of multiple hypotheses. If you're going on a fishing expedition by identifying subgroups post hoc, then you ideally need to base this correction on the potentially large number of conceivable subgroups that are available to be drawn. It's very hard to achieve real significance under those circumstances. On the other hand, you might find a subgroup result suggestive and conduct a separate follow-on study to test it independently; that's perfectly legitimate.
It's not even a good description of what was happening. Positive clinical trials were published, but if a company had invested in a drug then it would often do a dozen clinical trials. Of these, a few (due to normal statistical variation) would show that there was some correlation with whatever they were looking for, and those are the ones that they'd publish. It's similar to the stock market scam, where you create a dozen funds, trade them at random, and then cherry-pick the one that had done a lot better than the market, claim that it's due to your brilliance and then ask for investors (charging a hefty administration fee to each one).
I am TheRaven on Soylent News
This basic idea should be applied much more broadly, and given that most publishing is electronic now, all experiments should be published and evaluated on the validity of their methodology and how interesting or insightful the hypothesis being tested is, not their result.
How much money and time is wasted trying the same negative result experiments at different labs because they don't know that someone else has tried it and gotten a negative because it was not published?
'There was a derisive laugh from Alexandrov.
"Bloody argument," he asserted.
"What d'you mean 'bloody argument'?"
"Invent bloody argument, like this. Golfer hits ball. Ball lands on tuft of grass - so. Probability ball landed on tuft very small, very very small. Millions other tufts for ball to land on. Probability very small, very very very small. So golfer did not hit ball, but deliberately guided on tuft. Is bloody argument. Yes? Like Weichart's argument"'.
- "The Black Cloud" by Fred Hoyle
I am sure that there are many other solipsists out there.
The trouble is that when a pharmaceutical corporation carries out research on a new drug, it very much wants that drug to be found useful, safe, and fit for use. (Although it's in the corporation's interest that any really serious side-effects should be identified, as it would lose even more money if it marketed something that turned out to be a new thalidomide).
Unlike a proper scientific team, the corporation undertakes its research wanting a particular outcome. That alone is almost enough to guarantee that any results reported will be unreliable and unsafe. Research can only be reliable if it is done without concern for the outcome - that is, a given set of results is never regarded as "good" or "bad". It should all be just knowledge.
This is what comes of mingling pure scientific research with technology or engineering. In principle, science is the dispassionate pursuit of knowledge for its own sake. Technology and engineering are the application of scientific knowledge to accomplish something deemed useful. As far as possible, the two should be kept separate.
I am sure that there are many other solipsists out there.
@Gadget_Guy ...prove that such rigor isn't required of climate science...
>
One way to evaluate scientific hypotheses is to look at what the events they predict, then observe nature to see if the predicted events correspond to reality. In that sense, modern climate science is an epic fail because the "global warming" predicted by their models failed to happen. (Prompting the climate-alarmist "true believers" to switch to 'climate change' (so, up or down, can't lose))
http://www.americanthinker.com...
> ...most climate scientists have done a good job of keeping corporate interests away from their research...
BS. 'Big Oil' is a red-herring to divert attention away from 'Big Government', whose grants and funding tend to force researchers to become, in effect, lobbyists for political activism in order to 'pay the rent'.
when did academia get taken over by idiots? now that the gatekeepers are dumb we're fucked and all the smart people just go make money.
Well, actually, academia wasn't taken over by idiots. It was taken over - infiltrated - by smart people who were much more interested in money, prestige and power than in scientific truth.
That's the unfortunate fact about American culture. The USA was founded on the belief that all people (well, all white males of a certain age with property, but that's a small detail) should be treated alike. No titles of royalty, nobility or gentry. No class system. No special distinctions or honours.
The result, which became obvious very early on, was a society in which the only value was money. And money, it turns out, corrodes everything that is honest, decent and worthwhile. Now that culture is flooding the rest of the world - although some nations have done their valiant best to build dykes to keep it out.
"As the sociologist Georg Simmel wrote over a century ago, if you make money the center of your value system, then finally you have no value system, because money is not a value".
– Morris Berman, “The Moral Order”, Counterpunch 8-10 February 2013. http://www.counterpunch.org/20...
I am sure that there are many other solipsists out there.
Similar issues have shown up in other fields.
Indeed. The biggest issue is statistical ignorance, but even people with a decent amount of training in stats can be fooled if they want to find a particular result. Anyhow, whenever things like this come out, everyone always thinks it's about scientists who manipulate data deliberately. While that happens, it's more often just researchers who "try things out" after collecting data and notice a pattern (unintentionally skewing things). If they have to declare methods and statistical tests beforehand, it's harder to make these errors.
A few months back, I happened upon a very useful guide to the problems in modern scientific publication, which can be found partly online here. I ended up buying the print edition, and the sheer number of examples of completely bogus research ending up being accepted in various scientific fields due to erroneous stats and various biases that creep into the publication process... well, it's just shocking. Seriously.
As the book notes, the other problem is that even finding these errors is incredibly time-consuming and labor-intensive. I specifically remember one case where a new oncology test was proposed by Duke researchers and seemed to have great results. This case eventually became so infamous that it was reported on in the popular media.
Anyhow, basically they had a couple independent statisticians analyze the work (where they found HUGE numbers of problems in mislabeled data, mistakes in analysis and basic computation, etc., which appear par for the course in many labs, if you believe the studies on this stuff in the book). Ultimately, estimates are that it took TWO THOUSANDS HOURS of work for these independent statisticians to complete their analysis and render a verdict.
And once they did this, the statisticians tried to publish it -- but major journals didn't want it. Groundbreaking results are much more interesting that tedious statistical analysis. The National Cancer Institute caught wind of the problems and initiated an independent review, which found no errors (probably because the review was done by cancer experts, not stats experts, and they hadn't been giving the stats analysis done by the other researchers).
The only reason any of this ever really got much attention is because one of the lead researchers was accused of falsifying some aspects of his resume, which led to people actually going back and questioning his papers.
The book is full of stories like this, though, as well as citations of analyses of how many journal articles in various fields suffer from serious statistical problems.
It's all really scary when you start realizing how much bogus research is out there... most of it completely unintentional, and most of it passing peer review because it follows the field's "standard methodologies."
Where are the experimental controls? This brazenly silly meta study sports all the symptoms of a classic research pathology: confirmation bias.
It depends on what you're looking at.
Is this entirely independent research by respected labs? Or is this research in one particular area, by a specialised lab, with a particular sponsor? The latter is, we all know but can't prove easily, biased.
Is this a paper designed to do nothing more than back up a sponsor's advertising claims? Or is it something groundbreaking from a lab with few political ties, nothing to prove, and some serious science behind it.
It all changes the perspective of what a "scientist" is (i.e. someone who work not-for-profit to forward the cause of humanity, versus some guy in a lab coat with a PhD who's sold out?). As difficult as it is for us to distinguish, imagine what that means to the general public. Those people who invent terms for things for shampoo commercials that have fuck all to do with making your hair shiny, even if they sound like that, are held in the same regard as that guy doing genetic research to hunt down some elusive connection is an ultra-rare but devastating condition.
The problem is that there is nothing to distinguish the two, and both are technically "science", and thus can generate papers and be done by postdocs.
Sorry, but this is why I believe that referees on paper should be chosen specifically, why all connections should be documented IN THE PAPER (didn't declare who sponsors your lab? Bam, you're in the shit pile and your paper is forever disregarded), and why anything published should not be accepted until it's been confirmed - ideally by a bitter rival.
Too much shit is published nowadays as "science", commercial crap, ridiculous notions, unreviewed papers, basically anything from arxiv.org, etc. And the requirements aren't strict enough.
My girlfriend's a PhD in a medical field. Her contributions to a book were copied basically verbatim by someone else and published as another chapter in the very same book. It discredits her work, and that of the plagiarist, not to mention the primary author/editor. Things like asking people to reproduce their work under independent scrutiny are the only way to verify what's true, who did it, and how it was done so it can be repeated by future generations. We've lost all that in the last 20 or so years, even in things like Maths, etc. by having people referee their own friend's papers and other crap like that.
I also had this problem. And when I asked an PhD student about a thing that seems suspicious in his papers experiment setup he admitted that it's actually not as great as described.
Another big problem is research in development processes like Scrum, XP, Waterfall etc.. I argued quite a bit with my professor as all the experiments cited where like "take a bunch of students and let them work on a toy project, which never has changing requirements and is tiny in code size." Well this of course can't be used to make any conclusions of real software projects with real programmers, yet the results where generalized. This is just bad science. And we should call people out for it and definitely not publish this stuff. A better but still wrong approach was getting real programmers and let them work for _one_ day. This is outright ridiculous. What would be the result of this? The problems in software projects arise after _months_ not hours. Also people need time to get used to a style of working, which will at least take a couple of days.
Even worse the professor was really in awe with this experiment, as somebody actually used real developers. When I talked with him about this flaw he just waived his hands telling me, then no real experiment could be conducted. This is actually right, but what he meant was actually: this is better than nothing. I strongly disagree.
Americans are actually considerably less materialistic than other nations: http://www.ipsos-na.com/news-p...
Money to most Americans is a means to an end, a way to pursue the values and ideals they actually care about.
Well, Georg Simmel certainly got his wish, because about a decade after his death, his country made community, fairness, and equality the center of its value system, under the NSDAP party program. And it did it again two decades later in the GDR. And it was a resounding success: people indeed stopped caring about money, because they ended up caring mostly about not starving, not getting shot, and not getting gassed. Germany's tradition of academic philosophy that Simmal was part of bears a great deal of the blame.
And, of course, Simmel himself came from a well-off family and never had to worry about a lack of money limiting what he could do. His teachings are the philosophical equivalent of "let them eat cake".
Money grubbers are nice people next to power grubbers.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
Yes because climate scientists are all trying to get rich by pandering to the Government? THis is the most ridiculous argument against climate change. The only "scientists" with a demonstrable financial interest are the corporate shills denying the evidence. Take a look at the history of the campaign to end leaded additives in gasoline. The kind of corporate-funded "research" trying to discredit the voices sounding the alarm against lead in fuels sound eerily similar to what is going on in the climate change debate today.
You're kidding, right? How often do we have to chant the mantra that climate is not the same as weather. The climate models may suggest that we will see in increase in precipitation, but that doesn't mean that in one specific location that there will be more rain. Also, there are other localised factors with droughts.
None of what was in that article is enough to make the claims that all the models have failed. I also don't understand the point of the article. If 97% of the scientists agree that man has a hand in global warming, it doesn't mean that they agree on all the details. Nor does it mean that any disagreement within the community is proof that the whole thing is a big fat lie.
BS. 'Big Oil' is a red-herring to divert attention away from 'Big Government', whose grants and funding tend to force researchers to become, in effect, lobbyists for political activism in order to 'pay the rent'.
And that is even more BS. There is no proof that there is any "Big Government" that is attempting to control the scientific community, especially when 50% of those people in power are actively against the idea of climate change. Whenever you hear of political interference with the scientific process, how often is it some left-wing conspiracy to force the hand of scientists compared with conservatives attempting to shut down institutions that do research into climate change? Where is the evidence of this giant conspiracy, other than far-right pundits speculating as if it was fact?
because the "global warming" predicted by their models failed to happen
Just for kicks, look at this set of organizations that disagree with you.
For what you're saying to be true, not only every one of those organizations would have to be concealing data or faking models, every single organization on the planet would have to be in on it - - because science is science.
So the claim is that the entire global scientific community is suddenly full of shit, but only on this one subject (although of course the implication is that nothing affiliated with any university or mainstream research institute can be trusted, because scientist == liar in that world).
As always, all IMO. Insert "I think" everywhere grammatically possible.
This was explained in the journal xkcd.
https://xkcd.com/882/
Ok, it's not peer-reviewed. But it has a very high impact factor.
You couldn't publish something like that in a major peer-reviewed medical journal today.
Anil Potti, Sheng Wang, and many others managed to do it just fine. Where drug companies get into trouble it is usually when they hire people at hospitals/companies/universities to run sites for a clinical trial. The contracts often create huge conflicts of interest that can skew the results of the trial.