Slashdot Mirror
Registered Clinical Trials Make Positive Findings Vanish
schwit1 writes: The requirement that medical researchers register in detail the methods they intend to use in their clinical trials, both to record their data as well as document their outcomes, caused a significant drop in trials producing positive results. From Nature: "The study found that in a sample of 55 large trials testing heart-disease treatments, 57% of those published before 2000 reported positive effects from the treatments. But that figure plunged to just 8% in studies that were conducted after 2000. Study author Veronica Irvin, a health scientist at Oregon State University in Corvallis, says this suggests that registering clinical studies is leading to more rigorous research. Writing on his NeuroLogica Blog, neurologist Steven Novella of Yale University in New Haven, Connecticut, called the study "encouraging" but also "a bit frightening" because it casts doubt on previous positive results."
In other words, before they were required to document their methods, research into new drugs or treatments would prove the success of those drugs or treatment more than half the time. Once they had to document their research methods, however, the drugs or treatments being tested almost never worked. The article also reveals a failure of the medical research community to confirm their earlier positive results. It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
In other words, before they were required to document their methods, research into new drugs or treatments would prove the success of those drugs or treatment more than half the time. Once they had to document their research methods, however, the drugs or treatments being tested almost never worked. The article also reveals a failure of the medical research community to confirm their earlier positive results. It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
Uhh... that's not even wrong. It's either something that started out OK and has been edited down into nonsense, or it shows such a fundamental misunderstanding of how controlled experiments work that, well, it's not even wrong.
Similar issues have shown up in other fields. Psychology has had serious faillures to replicate many major studies http://www.slate.com/articles/health_and_science/science/2014/07/replication_controversy_in_psychology_bullying_file_drawer_effect_blog_posts.html and when there have been attempts to replicate them they have often not gotten the same results. And there are very similar problems in education https://www.insidehighered.com/news/2014/08/14/almost-no-education-research-replicated-new-article-shows. Pre-registration of experiments is important, but it would also help a lot if there were journals dedicated to replication and also if academia took replication more seriously: I know people who are tenure track who haven't tried to replicate some studies because it doesn't look as good for tenure promotion to just replicate something rather than do something new. There are serious cultural issues that need to change.
The basic flaw is worse. They didn't just run one test, find the results they wanted and go with it. They ran a test with only an idea of what they wanted, then took all the results they got and picked out ones that were positive for conditions or treatments they could go with. It's like going into a test for a drug to treat heart attacks, finding that it doesn't do anything for heart attacks but does seem to lower cholesterol levels, and announcing that the trials of your new cholesterol medication were positive.
Having to declare up front what their goals are destroys the ability to cherry-pick like this. What we're seeing with the drop in positive results isn't so much the difference in clinical effectiveness of the drugs but the dragging into the spotlight of the pharma companies' ability to predict what their drugs will do and how well they'll do them. There's a very interesting blog here that covers a lot of this, and one conclusion that keeps coming up again and again is that medical biochemists and researchers don't really have a good way of predicting from lab results what a compound will do in a live human. It also highlights fairly often how the drug companies will keep pushing a drug through trials even though the results aren't encouraging. It's a common attitude in business and finance, that now that you've invested this much money in something you have to get some return out of it to justify the cost. It's also a common failing in gambling, the belief that now that you're in the hole you have to dig yourself out somehow. But in gambling, if you're holding a bad hand your best bet is to fold. Don't worry about how much you've already got in the pot, it's already lost. Fold and cut your losses before you throw any more money away. Drug companies are notoriously bad at making that decision to walk away. They're also notoriously bad at dealing with a field where there aren't many good rules you can follow to get results. MBAs like process and procedure and predictable results, and right now biochemical research is in a situation where the new stuff is all likely out in areas where there isn't a lot of research, there isn't a good map of the territory and you're going to be doing a lot of "poke it with a pointy stick and let's see what it does" work.
Nobody "forgot" that. It's just that clinical trials often take years and involve thousands of people in the later stages, making them very expensive. Funding to replicate results is always lacking, particularly if it might disconfirm a positive finding - you're not going to convince a drug company to pay to have the effectiveness of their drug thrown into dispute.
It's the same in computer science. Many measurement methodologies are plain wrong or misleading. And in many cases, the source code of what people did is not available, so independent evaluation is not possible (someone also published a paper about that, where the author couldn't even get the code in many cases after explicitly asking for it, but I forgot the title). It's not just a problem of alpha sciences.
Donate free food here
...such rigor were required of Climate Science.
And now the onus is on your to back up what you say and prove that such rigor isn't required of climate science, or that the results are being significantly skewed because of their funding source. Just because what the science says does not match your gut instinct is not proof. Just because it could happen is not proof; we need evidence of inconvenient funding sources being omitted from research, or a meta study showing differing results depending on how the methodology is described.
In fact, I would go so far as to say that most climate scientists have done a good job of keeping corporate interests away from their research. If a scientist wanted to ensure future funding and be free of political interference, then they would take the easy path of downplaying global warming. No scientific research organisation has been defunded or disbanded because they gave evidence that contrary to man's impact on the environment.
...profit!
You'd find out we're in even deeper shit than even the alarmists realize.
no, no, no...
it's not about the 'basic tenet of science',
it's about the 'basic tenet of profit'.
It appears the medical research field has forgotten this basic tenet of science:
It's almost as if it isn't science at all, but rather advertising, where the target audience is a government agency that gives the company permission to transfer the product to their other advertising division who then advertise it to doctors and the public. What percentage of these clinical trials are trials of something not destined to produce wealth for an organization if the results of the trial are positive? When a wealth generating organization approves expenditure of the large amount of money to do a clinical trial, I'm sure it also adds extra management, or transfers management of the research so as to help the results of the trial be more profitable than just doing science ever could be. I'm thankful that this requirement to register these trials exists.
I want to show that all cars are red, so I go out and start counting cars. It turns out that not all cars are red, but by choosing every seventeenth car, except if the previous one had a number-plate ending with an odd number OR a number x such that the x-th fibonachi number has a 4 as the second digit, THEN all those cars are red! Dang, that's actually what I meant, forget that thing about all cars red. Publish!
Another serious issue is that it is hard publishing negative results. So even though every researcher is playing by all the rules, they might not know that 500 others have done the same trial before and failed, and this is just the 1 in 500 which accidentally gives a positive result.
I distinctly recall the friendly warnings of professors at the start of each course that picking data to reach a conclusion was not science. As such, non-science would be given the grade it deserved (zero) and recurrences would be viewed, at best, as an advertisement against a student's suitability for the profession or, worse, as grounds for academic dishonesty charges.
Maybe standards have dropped precipitously since my lab days or money really does have that corrosive of an effect on rigor.
This basic idea should be applied much more broadly, and given that most publishing is electronic now, all experiments should be published and evaluated on the validity of their methodology and how interesting or insightful the hypothesis being tested is, not their result.
How much money and time is wasted trying the same negative result experiments at different labs because they don't know that someone else has tried it and gotten a negative because it was not published?
In psychology, individuals are ... individual. If you can replicate the results exactly, then you must be doing something wrong.
In biology, everyone has a pair of kidneys that work the same. Some minor variations, but in virtually every study you should be able to replicate the results.
when did academia get taken over by idiots? now that the gatekeepers are dumb we're fucked and all the smart people just go make money.
Everything is going to kill you eventually. They just want to be the ones to make a dime off it.
@Gadget_Guy ...prove that such rigor isn't required of climate science...
>
One way to evaluate scientific hypotheses is to look at what the events they predict, then observe nature to see if the predicted events correspond to reality. In that sense, modern climate science is an epic fail because the "global warming" predicted by their models failed to happen. (Prompting the climate-alarmist "true believers" to switch to 'climate change' (so, up or down, can't lose))
http://www.americanthinker.com...
> ...most climate scientists have done a good job of keeping corporate interests away from their research...
BS. 'Big Oil' is a red-herring to divert attention away from 'Big Government', whose grants and funding tend to force researchers to become, in effect, lobbyists for political activism in order to 'pay the rent'.
This is what happens when they intermingle...
“He’s not deformed, he’s just drunk!”
In psychology, individuals are ... individual. If you can replicate the results exactly, then you must be doing something wrong.
This is confused. People are individuals but studies look at general trends. And those should remain the same. Humans being individuals doesn't make them immune to statistical analysis.
Similar issues have shown up in other fields.
Indeed. The biggest issue is statistical ignorance, but even people with a decent amount of training in stats can be fooled if they want to find a particular result. Anyhow, whenever things like this come out, everyone always thinks it's about scientists who manipulate data deliberately. While that happens, it's more often just researchers who "try things out" after collecting data and notice a pattern (unintentionally skewing things). If they have to declare methods and statistical tests beforehand, it's harder to make these errors.
A few months back, I happened upon a very useful guide to the problems in modern scientific publication, which can be found partly online here. I ended up buying the print edition, and the sheer number of examples of completely bogus research ending up being accepted in various scientific fields due to erroneous stats and various biases that creep into the publication process... well, it's just shocking. Seriously.
As the book notes, the other problem is that even finding these errors is incredibly time-consuming and labor-intensive. I specifically remember one case where a new oncology test was proposed by Duke researchers and seemed to have great results. This case eventually became so infamous that it was reported on in the popular media.
Anyhow, basically they had a couple independent statisticians analyze the work (where they found HUGE numbers of problems in mislabeled data, mistakes in analysis and basic computation, etc., which appear par for the course in many labs, if you believe the studies on this stuff in the book). Ultimately, estimates are that it took TWO THOUSANDS HOURS of work for these independent statisticians to complete their analysis and render a verdict.
And once they did this, the statisticians tried to publish it -- but major journals didn't want it. Groundbreaking results are much more interesting that tedious statistical analysis. The National Cancer Institute caught wind of the problems and initiated an independent review, which found no errors (probably because the review was done by cancer experts, not stats experts, and they hadn't been giving the stats analysis done by the other researchers).
The only reason any of this ever really got much attention is because one of the lead researchers was accused of falsifying some aspects of his resume, which led to people actually going back and questioning his papers.
The book is full of stories like this, though, as well as citations of analyses of how many journal articles in various fields suffer from serious statistical problems.
It's all really scary when you start realizing how much bogus research is out there... most of it completely unintentional, and most of it passing peer review because it follows the field's "standard methodologies."
You hit the nail on the head. By forcing researchers to declare, a priori, what they were looking for and how they were going to measure the effect they were looking for, the registry prevented cherry-picking. A researcher could not change his or her mind after the results came in, and choose another effect that had shown up in the data. Cherry picking is like staring at random noise and accidentally seeing patterns that aren't there.
Where are the experimental controls? This brazenly silly meta study sports all the symptoms of a classic research pathology: confirmation bias.
Exactly: If you have a bunch of random data, and want p that specific effect, in order to ensure that you aren't reacting to random noise.
Enjoy life! This is not a dress rehearsal.
I also had this problem. And when I asked an PhD student about a thing that seems suspicious in his papers experiment setup he admitted that it's actually not as great as described.
Another big problem is research in development processes like Scrum, XP, Waterfall etc.. I argued quite a bit with my professor as all the experiments cited where like "take a bunch of students and let them work on a toy project, which never has changing requirements and is tiny in code size." Well this of course can't be used to make any conclusions of real software projects with real programmers, yet the results where generalized. This is just bad science. And we should call people out for it and definitely not publish this stuff. A better but still wrong approach was getting real programmers and let them work for _one_ day. This is outright ridiculous. What would be the result of this? The problems in software projects arise after _months_ not hours. Also people need time to get used to a style of working, which will at least take a couple of days.
Even worse the professor was really in awe with this experiment, as somebody actually used real developers. When I talked with him about this flaw he just waived his hands telling me, then no real experiment could be conducted. This is actually right, but what he meant was actually: this is better than nothing. I strongly disagree.
That paper did get published in a smaller journal, or something, right?
So far as I'm concerned they hadn't been testing new drugs thoroughly enough, and people were getting hurt or dying because of it.
Are YOU using the TOOL, or is the TOOL using YOU? Think about it!
That's not sufficient. In many experiments, it is easy for multiple independent researchers to share incorrect assumptions or to make the same mistake.
Trusting scientific results is something that should take decades, many repeated experiments, and an understanding of how those results fit in with the rest of science.
... if they work really hard at curbing egos and huge financial incentives. As it is, quite a bit of medical "research" is utterly pathetic and counter-productive. Unfortunately, more and more of that is happening in other fields too. For example in CS, not a lot of real scientific progress has been made in the last 30 years, due to the same stupidity, arrogance and perverted incentives.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
"The problems in software projects arise after _months_ not hours."
I'm doing some game coding right now. I get problems cropping up every COMPILE. What the hell do you mean by MONTHS?
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
Uh, "not sure if serious"!?
Forget compile time bugs or errors in algorithm, think more towards project management, development processes, maintainability, impact of requirement changes or new feature requests to a large project after it's basically done etc.
Money is not a value, true - it is a measure.
That is why I believe funding of trials should not be done by pharmaceutical companies. It is a conflict of interest from beginning. Somethings by their very nature should be funded by taxes, because it is too important to simply let the organization with the biggest wallet win.
Exactly so. Money and its pursuit cannot replace the more human values such as honesty, courage, kindness, courtesy, imagination, charity, etc. There is something uniquely corrupting about money, in that when you start to think about it a lot you eventually become almost incapable of thinking about anything else. That's when you start sacrificing the good things of life for more money - and more, and more. If you are Larry Ellison, you want to be richer than Bill Gates. And so on.
I am sure that there are many other solipsists out there.
Any problem the compiler can find is easy.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
Problem is, I doubt the actual researchers could do anything about it. If I, as a researcher, decided tomorrow morning that I'd attempt to reproduce results from a bunch of other papers, I'd most likely lose all of my funding and would have no support from my university or research institution. You'd need a full top-down rework of science for something like that to pan out.
It's ironic too because we tend to waste a lot of time trying things that don't work, concluding that they don't work, and moving on to the next thing. If we'd published the negative finding, we'd ultimately save everyone else's time.
Money grubbers are nice people next to power grubbers.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
"look at what the events they predict,"
Yes, lets do that:
http://blog.hotwhopper.com/2013/07/denier-weirdness-collection-of-alarmist.html
Lets do a test of the models and see how well they do when given the actual emissions, rather than the projected emissions (plus the actual solar output, the actual volcanic output, and so on):
http://skepticalscience.com/2015-global-temps-in-line-climate-models.html
Hell, the oft-lambasted Hansen88 paper modelled the actual trends only 10% higher in trend than currently observed. And that paper used a model form the mid 1980's.
Meanwhile, as can be seen from the "contrarian" side, their models have failed not merely "doing poorly" but by being catastrophically wrong.
So, again, back to you to show that the climate science is broken because of a clique and preconception of results, for grant money or not.
Yes because climate scientists are all trying to get rich by pandering to the Government? THis is the most ridiculous argument against climate change. The only "scientists" with a demonstrable financial interest are the corporate shills denying the evidence. Take a look at the history of the campaign to end leaded additives in gasoline. The kind of corporate-funded "research" trying to discredit the voices sounding the alarm against lead in fuels sound eerily similar to what is going on in the climate change debate today.
You're kidding, right? How often do we have to chant the mantra that climate is not the same as weather. The climate models may suggest that we will see in increase in precipitation, but that doesn't mean that in one specific location that there will be more rain. Also, there are other localised factors with droughts.
None of what was in that article is enough to make the claims that all the models have failed. I also don't understand the point of the article. If 97% of the scientists agree that man has a hand in global warming, it doesn't mean that they agree on all the details. Nor does it mean that any disagreement within the community is proof that the whole thing is a big fat lie.
BS. 'Big Oil' is a red-herring to divert attention away from 'Big Government', whose grants and funding tend to force researchers to become, in effect, lobbyists for political activism in order to 'pay the rent'.
And that is even more BS. There is no proof that there is any "Big Government" that is attempting to control the scientific community, especially when 50% of those people in power are actively against the idea of climate change. Whenever you hear of political interference with the scientific process, how often is it some left-wing conspiracy to force the hand of scientists compared with conservatives attempting to shut down institutions that do research into climate change? Where is the evidence of this giant conspiracy, other than far-right pundits speculating as if it was fact?
Share link :)
Kunjungi Artikel kesukaan saya. baca ya artikel saya ini..
(Please visit and to translate my article favorite and read please..)
Belanja di Elevenia Gratis Voucher 1 Juta
Dan juga Belanja di Elevenia Gratis Voucher 1 Juta
7 Kebiasaan Sukses Ala Pengusaha Dunia
fitrarahim
Oli mempunyai peranan penting bagi sepeda motor. Nah, apa saja peranannya itu,,,.. baca selengkapnya
Oli Motor Terbaik – TotalHi-Perf
Unfortunately, not just social science.
Sent from my ASR33 using ASCII
The basic flaw is worse. They didn't just run one test, find the results they wanted and go with it. They ran a test with only an idea of what they wanted, then took all the results they got and picked out ones that were positive for conditions or treatments they could go with.
There's a name for this. It's called pseudoscience.
I stopped going to the doctor after my 4th year of grad school. Once you see what is going on it is too scary. I guess for broken arms, etc or some trauma that leaves you bleeding out it still makes sense to go. This is not a good situation.
Scientists have a job because the vast majority of funding for climate science comes from government or NGO organizations that only fund research that is looking to confirm human caused global warming er global climate change.
Which is why most of the criticisms come from older scientists with tenure and no longer trying to maintain a research lab so don't need funding or are simply retired.
because the "global warming" predicted by their models failed to happen
Just for kicks, look at this set of organizations that disagree with you.
For what you're saying to be true, not only every one of those organizations would have to be concealing data or faking models, every single organization on the planet would have to be in on it - - because science is science.
So the claim is that the entire global scientific community is suddenly full of shit, but only on this one subject (although of course the implication is that nothing affiliated with any university or mainstream research institute can be trusted, because scientist == liar in that world).
As always, all IMO. Insert "I think" everywhere grammatically possible.
Most of the tenured scientists agree that AGW is real, man made, and a problem today, and will get worse if we continue as we are doing.
Most of the criticism comes from people who have no expertise or knowledge of science at all.
So in very many ways you are wrong. The only way you are not wrong is that you have edited out any contrary information and therefore only know that you remember information that confirms your claim.
This was explained in the journal xkcd.
https://xkcd.com/882/
Ok, it's not peer-reviewed. But it has a very high impact factor.
Say you want to find out if certain candies act as a libido increaser.
You a do a study on 20 different candies. One of them seems to act like a weak form of viagra, at a 95% certainty that it isn't random chance. Except 95% is also known as one in 20 and you tested 20 candies...
If you didn't start out saying you were testing if M&M's were the viagra, you can claim that you have a positive effect. But if you were forced to specify the candy you were testing, you get negative results - and a hint to try another study.
In science this kind of thing happens often - only instead of testing 20 candies, they test one item against 20 different 'cures' (decrease weight, increases sensitivity to insulin, sleep aid, congestion aid, etc. etc.),
excitingthingstodo.blogspot.com
Of course AGW would be man made, that's what the 'A' stands for.
The thing I see enviro scientists disagreeing on, is not the existence of it, but the size of it, and what it actually means, rather than 30 different models that give different results.
focused on human randomized controlled trials that were funded by the US National Heart, Lung, and Blood Institute (NHLBI). The authors conclude that registration of trials seemed to be the dominant driver of the drastic change in study results. They found no evidence that the trend could be explained by shifting levels of industry sponsorship or by changes in trial methodologies.
Are you kidding? If a scientist could actually make a good case that the current model is wrong his career would be made and he would rapidly become so rich he wouldn't need government funding to continue his work. Nothing succeeds in science like upsetting the old paradigm. Ask Marshall and Warren (Nobel prize in Medicine).
This is great news, it shows that money paying for results spoils the process. When it's opened up for peer review, one leg of any review being the ability to repeat the experiment using the exact same conditions and expecting the exact same results, corporate bias in reporting results shows up like Fat Waldo in a herd of adele penguins.
The bias being brought about by the sponsored research simply dismissing negative results as experimental error rather than seeing it as legitimate reading as far as variance in their method allows. For instance, the efficacy of vaccines has long since been biased by virtue of the historical infection rate being ignored (which for most diseases, including polio, indicates the infection rate actually reducing - naturally - to almost nil *before* the vaccine hits the market. The given example also has a side effect brought about by the use of live but attenuated cultures that has been given a name intended to hide its origin: ALS. Together these show that polio vaccine is not only ineffective, it is also unnecessary and in fact demonstrably harmful).
Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel
Apparently not in climate research, instead you get blacklisted and have to publish under pseudonyms. What is the strength of the greenhouse effect? 33 C?
If there's one thing that is consistent is that the Face Painting Homers of AGW ignore inconsistencies and integrity issues in the data.
What other Science allows you to massage the very data you rely on to validate your theories?
Only at first sight.
Thinking a bit harder, the pharmaceutical companies could love your suggestion, because having you and me, the taxpayer, paying for all their clinical trials would result in just more profits for them. And they didn't have to go step-by-step, they can throw substance XYZ against the publicly funded trials, and lean back and wait.
There's more to it still: you can actually exploit the incertitude on the measurements you're using to categorize your subjects into subgroups, in a way to ensure your drug WILL report positive effects even if it has zero real effect. It's very well explained in this short article by Tom Naughton, complete with a numerical demonstration.
To put it shortly: you can design the subgroups' criterion in a way that overrepresents false positives and underrepresents the false-negatives that would otherwise counterbalance them.
Maybe we deserve this world ?
My belief is that until the same timeframe, longitudinal studies were both data intensive and privacy limited. With new privacy requirements from the study takers and the cost of data storage near zero, we are just picking the low hanging fruit yet. The soft sciences are easy pickings, but human health and more to my point, economic shibboleths are in BIG trouble with the combination of cheap tech and enough of us to make these meta musings worth the effort. On with the data taking, we have much to learn, and much to cast aside as ignorance yet.
admitting the obvious.
"Science" is not the high ideal people here tend to believe it is, the bulk of published stuff is garbage, scientists lie all the time, and most so-called "research" is never verified much less challenged.
Stop worshiping and start using your ability to think critically for a change.
Only every experiment that gets readings from instruments with known biases? What the fuck do you think scientific instruments are, oracles? You think they're made of unobtainium and starshine, wishes and wands? Lab instruments can be expensive and delicate and uniform.
Weather stations can't. Lab instruments can be operated in temperature- and humidity- and everyotherfuckingthingelse-controlled environments. Weather stations can't. They degrade, making the instruments behave differently. Conditions change, making the instruments behave differently.
But then, people do tend to judge others' characters by their own.
As always, all IMO. Insert "I think" everywhere grammatically possible.