Slashdot Mirror
Reanalysis of Clinical Trials Finds Misleading Results
sciencehabit writes: Clinical trials rarely get a second look — and when they do, their findings are not always what the authors originally reported. That's the conclusion of a new study (abstract), which compared how 37 studies that had been reanalyzed measured up to the original. In 13 cases, the reanalysis came to a different outcome — a finding that suggests many clinical trials may not be accurately reporting the effect of a new drug or intervention. Moreover, only five of the reanalyses were by an entirely different set of authors, which means they did not get a neutral relook.
In one of the trials, which examined the efficacy of the drug methotrexate in treating systemic sclerosis—an autoimmune disease that causes scarring of the skin and internal organs—the original researchers found the drug to be not much more effective than the placebo, as they reported in a 2001 paper. However, in a 2009 reanalysis of the same trial, another group of researchers including one of the original authors used Bayesian analysis, a statistical technique to overcome the shortcomings of small data sets that plague clinical trials of rare diseases such as sclerosis. The reanalysis found that the drug was, as it turned out, more effective than the placebo and had a good chance of benefiting sclerosis patients.
In one of the trials, which examined the efficacy of the drug methotrexate in treating systemic sclerosis—an autoimmune disease that causes scarring of the skin and internal organs—the original researchers found the drug to be not much more effective than the placebo, as they reported in a 2001 paper. However, in a 2009 reanalysis of the same trial, another group of researchers including one of the original authors used Bayesian analysis, a statistical technique to overcome the shortcomings of small data sets that plague clinical trials of rare diseases such as sclerosis. The reanalysis found that the drug was, as it turned out, more effective than the placebo and had a good chance of benefiting sclerosis patients.
Isn't this generally know as The Decline Effect? It's not just clinical trials, it applies to almost everything (to varying degrees). It's also been interpreted as The Half-Life of Knowledge.
Almost had me there article! Until you said the most evil words known to man... "statistical technique". AKA "bullshit"
Bayesian statistics is far from bullshit.
I suggest you read up on it.
You can do some really cool stuff with it.
Testing if a coin flip is fair.
Correct images.
Filter spam
They looked at reanalyses that had already been done for other reasons, rather than doing their own reanalyses on randomly selected trials. It occurs to me that these trials may have been subjected to reanalysis precisely *because* there were doubts about the initial analysis.
No, the GP is right. While BA gives you a probability distribution for the effectiveness, unless the effect is really strong (or you bad a really bad choice of priors), that distribution is going to be quite wide for a small data set. Such results are not proving that what you were testing was effective, but that there is a decent probability it might be effective given the knowledge you gain from the test, and that you should pursue a larger test. I've found it to be quite rare to have a BA result that strongly excludes a null hypothesis in a small scale test without having already been flagged as effective by simpler tests (i.e. the effects were so obvious, didn't require trying that hard to see).