Slashdot Mirror
Medical Journals Fight Burying of Inconvenient Research
A dozen leading medical journals have announced that they plan to refuse publication of clinical studies unless those studies were publicly registered ahead of time. Part of the intention is to prevent researchers from privately doing multiple studies and then selectively releasing for publication only those which yield favorable results. There are many other journals which have not signed on to this plan, however, and it remains to be seen what will happen. Personally, I'm surprised it's taken this long; as Karl Popper wrote, "what distinguishes the scientific approach and method from the prescientific approach is the method of attempted falsification."
Funny that a colleague and I were talking about this the other day. Clinical studies and general statistics in (peer-reviewed) medical journals generally use a "0.95 confidence interval".
What this means is that, if left to chance, the experiment/trial/study would be positive in only 1 in 20 times. Example: you give a bunch of sick people an experimental drug, and you give another bunch of sick people a placebo (or a known standard of treatment). The people on the experimental drug get better. Was it really because of the drug? Maybe it was just random chance --but if that chance is calculated to be only 1 in 20 or less, then we say, "Yeah, it's probably because of the drug, and not just chance."
The overwhelming majority of drug trials are corporate-funded. A company that's desperate enough to get its drug to market could easily fund 20 studies, and even if the drug were just placebo, chances are good that 1 in 20 of those studies would turn out positive. (Yes, yes, I'm just approximating.) Without the "negative results registry", you'd think that the drug was working.
Would companies be desperate enough to do this? You bet. I'm not saying that any particular company did this, but consider what happens to get a drug to market. Someone invents a molecule (typically a lab with 12 employess or something), gets bought by a biotech firm with 4 employees (they subcontract everything out), some other lab tests it on animals, some other firm develops a formulation (tablets, capsules, makes sure it doesn't melt inside the bottle or degrade, etc.), a big drug company buys the formulation (or the entire firm) and starts gearing up for clinical trials while submitting for FDA approval. This all takes about five years. What if it doesn't work out? As a backup, the Big Pharma Company also invests in about five backup compounds, and each of those compounds has five backup compounds. We're talking about, after ten years and researching thirty compounds, you might get ONE drug out to the market. (Btw, my wife is the project manager for a bunch of these drug research pipelines at one such Big Pharma company.) But, boy, will that drug make it big! What if the drug didn't really work? Well, let's make it look like it did! (I can see Big Pharma CEO's rationalizing this as "let's put it in the best light possible.")
Example of drug research being biased? Ever heard of celecoxib (Celebrex)? Wow, anti-inflammatory pain reliever that, unlike ASA (Aspirin) or ibuprofen (Advil, Motrin), does NOT irritate your stomach! No stomach bleeding (uncommon but serious side effect of ASA/ibuprofen)! They did the research and showed that people actually did (statistically) significantly better than ASA after six months. JAMA (Journal of the American Medical Association) published the study and even sang its praises in the editorial. They get it out and market it to all the physicians all over the place.
And then we find out that the study went on for more than 6 months. We find out that beyond 6 months, the people using Celebrex got WORSE, and deteriorated until at 12 months, they were no better than ASA users. Boy was the JAMA editor mad! (If I recall, he even publicly lambasted Searle for this in the New York Times.)
But you know what? It didn't matter. Celebrex was everywhere, on American TV ads, and people asked for it. Docs who don't really have time to delve into the medical literature already had established in their mind that "Celebrex is better". (My colleagues certainly continued to use it even when ASA was sufficient.) And the drug reps, who ooze snake oil from their skin pores, keep pushing it. One drug rep even questioned my choice of medication when I was getting it from our drug sample closet. I lit into her like you wouldn't believe. Is she the doctor or am I?? (whew, catharsis, feel better now)
So, yes, I think the companies are perfectly capable of doing this (stacking the studies). The benefits are just too great. I welcome the use of the clinical trial pre-registry.
404555974007725459910684486621289147856453481154 in hex is "You sank my Battleship?"
[GPG key in journal]
This nicely illustrates Lowe's point: what you're saying is widely believed, but is absolutely, utterly, entirely, absurdly false. (Except for the non-sequitur at the end about "credit" which I don't understand at all.)
And since when is an industry spokesman considered a reliable source of information..?
First, he's a chemist, not a "spokesman". Second, he (and I) do precisely the work you claim doesn't exist and might be thought to have something to say on the matter. But, if you want to limit your "reliable sources of information" to people who don't know what they're talking about, that's certainly your right.
What I'm listening to now on Pandora...
I've always wanted to start a journal called "Journal of Null Results" where people can publish research that's well done, but came up with a result that doesn't qualify as earthshaking or "sexy".
Publishing null results would help people avoid repeating things that have been done already, as well as help refine research to see if there is a positive result hidden in the null.
In the case of big medical studies it could provide a great source for data mining, where metastudies of a bunch of null results might suggest something that would be hard to see without a lot of overlapping data.