Medical Journals Fight Burying of Inconvenient Research

A dozen leading medical journals have announced that they plan to refuse publication of clinical studies unless those studies were publicly registered ahead of time. Part of the intention is to prevent researchers from privately doing multiple studies and then selectively releasing for publication only those which yield favorable results. There are many other journals which have not signed on to this plan, however, and it remains to be seen what will happen. Personally, I'm surprised it's taken this long; as Karl Popper wrote, "what distinguishes the scientific approach and method from the prescientific approach is the method of attempted falsification."

Comments

Great idea!

[keiferb]

Now, if only this could be extended to include IT studies. I wonder how many studies people run that show that their product isn't that hot.

On second thought, they're all funding their own studies, so it's probably a non-issue.

Finally

[hubs99]

The is great news. If the JAMA can do it then so should everyone else. In addition to the NIH proclaiming that they are hoping to force Researches to publish to public domain, research and reporting of research is moving in a positive direction.

Re:Finally

[wetlettuce]

Exactly, all research should be published whether it be good or bad and let the reviewers decide whether the results are valid and the methdology is good and is worth publishing. In that way the researchers will probably gain from the positive critism allowing them to do even better research the next time around.
If they don't know why things are failing, how can they improve?

Re:Finally

[bitingduck]

I've always wanted to start a journal called "Journal of Null Results" where people can publish research that's well done, but came up with a result that doesn't qualify as earthshaking or "sexy".

Publishing null results would help people avoid repeating things that have been done already, as well as help refine research to see if there is a positive result hidden in the null.

In the case of big medical studies it could provide a great source for data mining, where metastudies of a bunch of null results might suggest something that would be hard to see without a lot of overlapping data.

Re:Finally

[radtea]

Back in the day when I was still working directly in research as a pure physicist, I got a lot of null results. People used to tell me that old lie, "A null result is just as good as a positive result."

I got fed up with this after a while, and when other people got positive results, I started telling them, "Don't worry, a positive result is just as good as a null result." It got me a lot of funny looks.

Later, working as an applied physicist, where null results are far less common, I tried to include a section in every paper with the heading "Things that didn't work so well", to describe the stuff I'd tried that hadn't worked well. I think there should be more of this in the scientific literature, and as a reviewer have at times asked authors to explain why their paper only includes stuff that worked--what happened to the failures?

Publishing failures is important for lots of reasons. One is that facts are facts, and we shouldn't just publish the economically valuable ones. The other is that our duty as scientists is to help our colleagues avoid the mistakes we've made, and one important way to do that is to publish them.

--Tom

Re:Finally

[Anonymous Coward]

By the way, the NIH "buries" at least as much stuff as anybody. I was in an NIH-funded lab doing NIH work for nearly a decade, and there was a lot of good science that did not get published because it was a "negative" result. Even though nearly every such instance, if widely communicated, could have saved hundreds of thousands of dollars and months or years of efforts for other (NIH!) labs by preventing them from going down blind alleys, you will not get tenure and you will not get subsequent funding if you publish "negative" results. I've worked in industry and in academia, and, frankly, I think there is much, much more "burying" of information in academia, since typically academic labs build themselves around one or two themes to get their funding - if data doesn't "fit" the hypothesis, it can torpedo a lab. In pharma, there are so many projects going on at once that killing a particular project or hypothesis, while not without pain, is generally far, far less of a challenge to a lab's existence. The only thing that saves academic/NIH science is that it is likely that there is another lab somewhere in the world working in a similar enough area that it is hard (but not impossible) to propagate dubious hypotheses for more than a few years.

Meanwhile

[Otter]

Derek Lowe was just posting on this topic and on the general inability of the pharmaceutical industry to get basic facts to the public's attention.

Please note that despite Jamie's spin (and the Times') all studies have to be reported to the FDA. I'm not quite sure of all the differences between this new registry and the existing registration, but the idea that drug companies can perform trials in complete secrecy is as wrong as the idea that NIH-funded research does all (or even much) of the work of drug development -- a point Lowe addresses elsewhere.

Re:Meanwhile

[hung_himself]

as wrong as the idea that NIH-funded research does all (or even much) of the work of drug development

No they just do the basic research that results in the drug leads. The companies then do the expensive but scientifically easy trials and rake in all the money (and now it seems, the credit as well).

And since when is an industry spokesman considered a reliable source of information..?

Re:Meanwhile

[Otter]

No they just do the basic research that results in the drug leads. The companies then do the expensive but scientifically easy trials and rake in all the money (and now it seems, the credit as well).

This nicely illustrates Lowe's point: what you're saying is widely believed, but is absolutely, utterly, entirely, absurdly false. (Except for the non-sequitur at the end about "credit" which I don't understand at all.)

And since when is an industry spokesman considered a reliable source of information..?

First, he's a chemist, not a "spokesman". Second, he (and I) do precisely the work you claim doesn't exist and might be thought to have something to say on the matter. But, if you want to limit your "reliable sources of information" to people who don't know what they're talking about, that's certainly your right.

Re:Meanwhile

[CodeMonkey4Hire]

I'm pretty sure that the FDA has to keep silent unless they perceive that the drug in question is unnecessarily dangerous, in which case all they can do is pull it off the market. They can't force any of the studies to be released or disseminate their contents.

Re:Meanwhile

[DAldredge]

If you work at a major pharmco what do you think of so much of the yearly budget being spent on Ads not research?

Re:Meanwhile

[Anonymous Coward]

And since when is an industry spokesman considered a reliable source of information..?

First, he's not an "industry spokesman." He is a practicing Ph.D. medicinal chemist who also happens to be an entertaining and erudite blogger.

Second, it just drives me nuts when people who have absolutely NO IDEA what they are talking about somehow feel that it's appropriate to dismiss as shills those who have actual, hands-on, hard-earned KNOWLEDGE OF THE FIELD!

Somehow, ignorance is privileged.

Re:Meanwhile

[Anonymous Coward]

Sales pay the research bill. If you don't advertise, you don't sell. Just ask the owner of the pizza place down the street from you.

Partial solution

[crow]

So what stops someone who registers a study from still deciding not to publish the results if they aren't favorable to the funding drug company? It does mean that there is public information that a study was started, which can mean some pressure to publish.

And nothing stops a drug company from funding a bunch of studies that aren't registered, and then registering duplicate studies that they then expect to be most favorable. Of course, the registration process would then add expense and delay in getting out this sort of slanted results.

At least even with the current system, I expect the peer reviewed journals are much better than the sort of "studies" that get published regarding the computer industry (e.g., TCO of Windows vs. Linux).

Re:Partial solution

[KWTm]

And nothing stops a drug company from funding a bunch of studies that aren't registered, and then registering duplicate studies that they then expect to be most favorable.

The results of the studies are random, or contain randomness. If you fund a bunch of studies (say, 20 studies) and one of the shows positive results, you can't just duplicate the results. You'd have to duplicate the study, again asking a hospital or two to recruit 300 patients to test the drug, and this time the results might not be positive. You can't even recruit the same patients over, because the standard is to recruit the first consecutive 300 patients to walk in the door who fit the criteria (and consent to the study, of course) to prevent selection bias. Any other way of recruiting would raise red flags that any medical student could spot a mile away.

(For the record --yes, IAAD.)

Re:Partial solution

[tomthumbkop]

I think the point is that the company would do a study using methodology A to determine the efficacy of their drug. If the results were favorable, then they would register a study using methodology A and publish the results. If the results from using methodology A the first time were bad, then they would move on to mthodology B and give it a whirl. This way they could do many studies using different methodologies but only register and publish the favorable ones.

Re:Partial solution

[Ayaress]

When the article was talking about companies doing multiple studies and only publishing the best one, though, they were talking about this sort of thing. The companies weren't changing the experiment to make it work. Doing that would raise red flags - maybe not so obvious as creative methods for patient recruiting, but something that other doctors would likely catch. They're just doing the studies several times and publishing the best one (or just doing one big study and only using part of the data set for publishing). Doing a pre-run study as you suggest would help their odds, but would nowhere near assure them good resuls in the second, registered study.

Only 20 studies needed to prove placebo works

[KWTm]

Funny that a colleague and I were talking about this the other day. Clinical studies and general statistics in (peer-reviewed) medical journals generally use a "0.95 confidence interval".

What this means is that, if left to chance, the experiment/trial/study would be positive in only 1 in 20 times. Example: you give a bunch of sick people an experimental drug, and you give another bunch of sick people a placebo (or a known standard of treatment). The people on the experimental drug get better. Was it really because of the drug? Maybe it was just random chance --but if that chance is calculated to be only 1 in 20 or less, then we say, "Yeah, it's probably because of the drug, and not just chance."

The overwhelming majority of drug trials are corporate-funded. A company that's desperate enough to get its drug to market could easily fund 20 studies, and even if the drug were just placebo, chances are good that 1 in 20 of those studies would turn out positive. (Yes, yes, I'm just approximating.) Without the "negative results registry", you'd think that the drug was working.

Would companies be desperate enough to do this? You bet. I'm not saying that any particular company did this, but consider what happens to get a drug to market. Someone invents a molecule (typically a lab with 12 employess or something), gets bought by a biotech firm with 4 employees (they subcontract everything out), some other lab tests it on animals, some other firm develops a formulation (tablets, capsules, makes sure it doesn't melt inside the bottle or degrade, etc.), a big drug company buys the formulation (or the entire firm) and starts gearing up for clinical trials while submitting for FDA approval. This all takes about five years. What if it doesn't work out? As a backup, the Big Pharma Company also invests in about five backup compounds, and each of those compounds has five backup compounds. We're talking about, after ten years and researching thirty compounds, you might get ONE drug out to the market. (Btw, my wife is the project manager for a bunch of these drug research pipelines at one such Big Pharma company.) But, boy, will that drug make it big! What if the drug didn't really work? Well, let's make it look like it did! (I can see Big Pharma CEO's rationalizing this as "let's put it in the best light possible.")

Example of drug research being biased? Ever heard of celecoxib (Celebrex)? Wow, anti-inflammatory pain reliever that, unlike ASA (Aspirin) or ibuprofen (Advil, Motrin), does NOT irritate your stomach! No stomach bleeding (uncommon but serious side effect of ASA/ibuprofen)! They did the research and showed that people actually did (statistically) significantly better than ASA after six months. JAMA (Journal of the American Medical Association) published the study and even sang its praises in the editorial. They get it out and market it to all the physicians all over the place.

And then we find out that the study went on for more than 6 months. We find out that beyond 6 months, the people using Celebrex got WORSE, and deteriorated until at 12 months, they were no better than ASA users. Boy was the JAMA editor mad! (If I recall, he even publicly lambasted Searle for this in the New York Times.)

But you know what? It didn't matter. Celebrex was everywhere, on American TV ads, and people asked for it. Docs who don't really have time to delve into the medical literature already had established in their mind that "Celebrex is better". (My colleagues certainly continued to use it even when ASA was sufficient.) And the drug reps, who ooze snake oil from their skin pores, keep pushing it. One drug rep even questioned my choice of medication when I was getting it from our drug sample closet. I lit into her like you wouldn't believe. Is she the doctor or am I?? (whew, catharsis, feel better now)

So, yes, I think the companies are perfectly capable of doing this (stacking the studies). The benefits are just too great. I welcome the use of the clinical trial pre-registry.

Re:Only 20 studies needed to prove placebo works

[Nos.]

The problem with statistics is that they can often be made to say what you want. There was a recent advertisement around here that said 80% of accidents happen within a five mile radius of your home. The ad was pushing to use seatbelts even for a short trips. However, if 90% of driving is done within that same five mile radius, then your actually less likely to get in an accident then when your driving close to home. The message is still good, but the statistics may not actually support what the message is saying.

Re:Only 20 studies needed to prove placebo works

[Anonymous Coward]

So, yes, I think the companies are perfectly capable of doing this (stacking the studies). The benefits are just too great. I welcome the use of the clinical trial pre-registry.

New to Slashdot?

I welcome the use of the clinical trial pre-registry overlords.

Re:Only 20 studies needed to prove placebo works

[jannesha]

The overwhelming majority of drug trials are corporate-funded. A company that's desperate enough to get its drug to market could easily fund 20 studies, and even if the drug were just placebo, chances are good that 1 in 20 of those studies would turn out positive. ...

Would companies be desperate enough to do this? You bet.

I bet not. These studies can cost millions. Sure, BigPharma has BigMoney, but you're suggesting that they can easily soak a 2000% increase in their phase 4 clinical trial budget!? No way.

Re:Only 20 studies needed to prove placebo works

[Retric]

True but how about this you have money to run 1 - 4 studies. With 4 you get a 18.5% chance it works and little chance of a false negative.

1: drug works and you wasted a few mill but your about to make 100's of mill so not a big deal.
2: drug works and one study would have failed to detect that... HUGE WIN!
3: drug fails and random chance favor's you. Make back most / all your reasearch money.
4: drug fails on all study's... O well you lost all your money anyway.

Now I can easly see this being worth it.

Re:Only 20 studies needed to prove placebo works

[Anonymous Coward]

Respectfully, this idea of getting approval based on just the 1 in 20 chance results doesn't work.

You can't legally pick and choose which trial results you send to FDA when you apply for approval. You have to send everything.

You also can't legally conduct clinical studies with an unapproved new drug unless you inform FDA in advance, so they know about all the trials anyway. And when you do apply for approval, FDA actually comes and looks through your records. They don't just assume that you've given them everything you should have.

So you would have to be willing to break the relevant laws, and it would still be pretty hard to get away with. And if the FDA caught you trying it, officers of the company would likely go to jail.

Another point is that a single clinical trial showing evidence of efficacy at the 95% confidence level is normally not enough to get approval. The typical standard is two independent, randomized, controlled trials showing efficacy. There are some exceptions, but even there, one trial at 95% won't cut it by itself.

That said, I agree selective publication can still be a significant problem. I think that's particularly true for already approved drugs being studied for new indications (but a later comment suggests that's not what you mean.) I just wanted to clarify that you can't really get new drugs approved this way. (Since your wife works for a big pharma, she should be able to confirm this.)

Re:Only 20 studies needed to prove placebo works

[KWTm]

You can't legally pick and choose which trial results you send to FDA when you apply for approval. You have to send everything .

One word: Celebrex

(As I mentioned in my previous post, Searle obviously did not give everything to the FDA, who fast-tracked the approval.)

You are right that, realistically, you can't get people to use a drug based on a single trial; that's just an extreme example to show that even placebo can do well if you study it hard enough and allow publication bias. In real life, presumably there would be some efficacy to the drug, which can then be biased favourably with publication bias.

Re:Only 20 studies needed to prove placebo works

[Anonymous Coward]

First, I think you missed my main point.

I agree that selective publication has the potential to influence use of an approved drug. My point was that there's no way to get a new drug approved based on selective data submission, unless the company's officers are willing to take a substantial risk of going to jail.

Second, Searle did give the FDA all that data on Celebrex, but the whole mess took place after the drug was already approved.

Celebrex was originally approved in Dec 1998. At that time, FDA told Searle they could not make any claims that Celebrex was safer than other NSAIDs. Their clinical data at the time didn't justify such claims. Note that this was based on data submitted to FDA between June and Dec 1998.

In the studies you're referring to, altbough the original protocols date back the Jan 98, it wasn't until Sep 99 that all patients had been treated for at least 6 mos. The studies ended in Jan, 2000. The 6 month data was published in the 9/13/2000 issue of JAMA. All of the data was submitted to FDA and discussed by the Arthritis Advisory Committee on 2/7/01, because Searle wanted FDA approval to amend their labeling to say that Celebrex had lower risk of GI problems than other NSAIDs.

In fact, that is how the discrepancy between the full data and the JAMA article became public. One of the participants in the advisory committee meeting was a GI named Wolfe, who coauthored the favorable editorial accompanying the 2000 JAMA paper! Of course, his editorial was based on just the 6 mos data. When he saw the full data set at the AC mtg in '01, he was quite unhappy.

To boil it all down:

1) Drug companies cannot get unapproved drugs approved by doing lots of clinical studies, and only submitting the "good" ones.

2) Assuming they can get the drug approved, drug companies can potentially use selective publication to enhance perception & use of their drugs.

P.S. If you really want, I can provide links to support the above Celebrex info. It's almost all from FDA's website.

Sigh...

[hung_himself]

I never claimed that no research was done at the pharmaceuticals - they do have huge compound libraries - but it is all very applied not to say that it is trivial - it is just not my cup of tea. I also do not doubt that many drugs come out this work. I have no doubt however that this is an extremely small percentage of the overall "research" costs that the companies like to quote.

My point was that to claim that no drug development is done with public funding is misleading and mendacious - because while it is true that little research is directly aimed at developing drugs, the knowledge that does come out of NIH funded projects is the underpinning of novel drug discoveries. That's what we write in our grants anyways. However, if it really is mostly undirected testing of random compounds and modifications of existing working compounds then I take it all back.

Wouldn't u spend billions to prove placebo works?

[KWTm]

Would companies be desperate enough to do this? You bet.

I bet not. These studies can cost millions. Sure, BigPharma has BigMoney, but you're suggesting that they can easily soak a 2000% increase in their phase 4 clinical trial budget!? No way.

If you could take a placebo and fund 20 studies to show that it worked, wouldn't you? That's like saying, I can sink millions and billions into this research to prove that potato chips cure cancer. And I hold the patent on the potato chip. I gain the ability to market this directly to consumers ("scientific studies have shown that it works...") You know how big a market we're talking about? Anything under a billion in expenses is peanuts in comparison.

Btw, it's Phase 3. Phase 1 = safety (does it HARM people?); phase 2 = effectiveness (any sign that it works at all?); phase 3 = efficacy (does it work better than placebo / standard of care?); phase 4 = aftermarket (now that we're selling the drug, have we run into any problems?)

Methodology can't be substituted that easily

[KWTm]

If the results from using methodology A the first time were bad, then they would move on to mthodology B and give it a whirl. This way they could do many studies using different methodologies but only register and publish the favorable ones.

It's much harder to set your own study methodology (and still get the same respect). The family doc who makes an effort to do evidence-based medicine (as opposed to "this is so because my doddering old professor back in medical school said that his teacher said it was so") looks for several key points in the abstract of the journal articles. (What, you think s/he has time to read the whole thing?) You want a randomized, double-blind placebo-controlled multi-centre study with a large sample of the general population . If these keywords are there, the doc jumps to the "conclusion" heading and generally acknowledges it to be valid. The main text of the article is just for medical students who have to do their presentations. :)

• randomized: what/who determines which patient goes into which comparison group? If you ask for patients to volunteer to "try the experimental drug" or "try the placebo", you get selection bias: the patients willing to try the drug may be sicker (this could mean that the drug works better, or doesn't work as well). If you say, "Okay, everyone at Dr. Smith's clinic gets the real drug, and everyone at Dr. Jones' gets placebo," there may be other biases. If it ain't randomized, people get purty suspishus.
• double-blind: not only do the patients not know whether they're taking the real drug (that would be single-blind), but the doctors who treat the patients also don't know (double-blind). This prevents a doctor from saying, "I'm not telling you whether you're taking the real drug, but do you feel better? Are you sure? Don't you feel just a teensy weensy bit better?" All patients are assigned a number, and at the end of the trial, some oversight comittee reveals which patients got which drug.
• placebo-controlled: you have to compare it with something, even if it's just placebo. Got a group where 60% of people improved on the drug? If 50% of the people improved on placebo, it certainly puts a different light on things.
• multi-centre: this means that the study took place at various locations, hopefully drawing on different groups. If it were one inner city hospital, that's one thing; if it's five different urban centres in five different countries, that's another.
• large sample size: Having a thousand patients (500 per group) would be nice. I generally ignore anything less than N=200. (Multiple low-budget studies can be combined by meta-analysis to achieve the same effect, *if* the experimental conditions are similar.)
• general population: some studies target certain segments of the population, such as "diabetics over age 65", which would narrow the market. The wider the population to whom the study is applicable, the better.

There are other study methodologies with such names as "cohort studies", "case-controlled", "retrospective" which get some respect but are viewed generally as a steppingstone to getting funding for the ultimate gold standard, the randomized clinical trial. So if a company's drug trial works with case-controlled, that might not be enough to get FDA approval for whatever labeled use they're after.

Everybody knows

[Uplore]

that given enough time, you are able to prove anything using statistics. Not only that, but depending on how the 'facts' are presented, they may be interpreted differently as well. We will never be able to completely erradicate ambiguous or even downright fraudulently presented research, but at least this is a start.

Re:Finally... Null Hypothesis

[Anonymous Coward]

At least one such exists:
Journal of Atricles in Support of the Null Hypothesis