Slashdot Mirror
Majority of Landmark Cancer Studies Cannot Be Replicated
New submitter Beeftopia writes with perhaps distressing news about cancer research. From the article: "During a decade as head of global cancer research at Amgen, C. Glenn Begley identified 53 'landmark' publications — papers in top journals, from reputable labs — for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development. Result: 47 of the 53 could not be replicated. He described his findings in a commentary piece published on Wednesday in the journal Nature (paywalled) . ... But they and others fear the phenomenon is the product of a skewed system of incentives that has academics cutting corners to further their careers."
As is the fashion at Nature, you can only read the actual article if you are a subscriber or want to fork over $32. Anyone with access care to provide more insight?
Update: 04/06 14:00 GMT by U L : Naffer pointed us toward informative commentary in Pipeline. Thanks!
But they and others fear the phenomenon is the product of a skewed system of incentives that has academics cutting corners to further their careers.
As I've said before, back when I was in academia, there were always grant-whores and academics more interested in their own interests than science around. Too many people have come to treat science with a reverence more appropriate to a religion than a system of knowledge and discovery, however. And so when I point out that there are scientists out there willing to cook the numbers, exaggerate, play to politics and/or public opinion, etc. I inevitably run into those who say "Science wouldn't allow that" (like my friend who's still in the field). But science is only as good as the people practicing it. And, in any field, there are always those willing to put their own personal interests ahead of the greater good.
I just hope this doesn't cast a shadow over those out there who *are* doing good work and *are* trying to do honest work. Sadly, some of the best researchers out there are the ones who make the least noise, get the least attention, get the least grants, and are least likely to get tenure.
What political party do you join when you don't like Bible-thumpers *or* hippies?
See this discussion of the same paper on In the Pipeline, a blog devoted to organic chemistry and drug discovery. http://pipeline.corante.com/archives/2012/03/29/sloppy_science.php
There is no greater "absolute power" than knowing that if you say or write something that others will like, they will pay you lots of money and make you famous.
It's not that money corrupts. Money is not the root of all evil; the full quote is "the love of money is the root of all evil." When our society decided that money was more important than truth, we surrendered truth to the void.
A research scientist thinks about his day. He can slightly fudge his cancer study, make big headlines, get a ton of grant money and get appointed chair at the university. Or he can go down the long hall to his boss and say, "Nope, this one didn't work either, and while I'd like to start a religion based on false hope, this isn't the false hope you're looking for." If he does that, he can then watch one of his subordinates fudge the cancer study, make big headlines, and be his boss at the same time next year.
Which choice would you make?
Most people pursue careers in atmospheric science for the dollars
Many of these scientists are getting big grants to do their research. At least a few of them might be skewing their results, even just a little, to give the answers their backers want, in order to keep the money flow open. This goes for a lot of scientific research. (Not to mention the politics of getting published if your research contradicts a heavyweight in your field).
Two weeks ago said the social science studies are usually not replicated. Either because they are not true or too expensive. They were trying to explain the rise in psychology paper retractions and job firings as poor science.
My second thought is "Hmmm, academics/scientists skewing results for the sake of their own careers. Global warming, anyone?"
Your second thought is completely off because every single time someone actually tried to replicate global warming research, they DID get the same results. Unlike in the case of those medical papers TFA is about.
Given the expense, I flat don't believe that a private company just decided to replicate 53 studies.
And he claims that authors "made" his team sign confidentiality agreements. How do authors force that?
So, he claims, he can't even tell us which studies failed.
Now he works at a different cancer research company. Conflict of interests?
I don't doubt that we've got problems in the "medical industry", but the linked article reeks of bullshit.
Has anyone looked at Nature?
Sheesh, evil *and* a jerk. -- Jade
While certainly there are those who will publish findings they know to be false, that's not really the big issue I see here. Good science demands that studies be replicated so they can be upheld or refuted. Sure, there's confirmation bias in science all over the place - the bigger problem I see is that there's very little incentive to publish a paper that simply refutes another. Busting existing studies should be a glorious field, but it's not. If big-name scientist A publishes a result in nature, and no-name scientist B publishes a paper in the journal-of-no-one-reads-it, everyone just assumes scientist B is just a bad scientist (assuming he even managed to actually get published at all).
Another major issue is that the null hypothesis is a very un-enticing story. No one wants to publish the paper: "New Drug does nothing to cure cancer". If you spent a year and a ton of money researching New Drug, you're damn sure going to try and make it work. It's unfortunate, because often the null hypothesis is very informative, but it doesn't get you paid or published. Or how about the psychology paper: "Brain does not respond to stimulus A in any meaningful way", don't remember that paper? That's because it never got published.
I think this is less about malicious behavior, and more about a lack of interest (which can somewhat be blamed on the way universities/journals/grants handle funding, notoriety, etc) in replicating and refuting studies.
Do you want to be the guy who cured cancer, or the guy who disproved a bunch of studies?
Ze Atomic Device! It iz Ztolen!
are famously bad at sceince and statistics.
And there's no benefit (to the researcher) in replicating a study that's already been done which makes for an obvious problem.
Medical science isn't alone in this of course, it just seems to be worse than most.
A recent PLOS article (free to view!) analyses modern research, coming to the conclusion that most research findings are false.
TLDR: Because of the nature of the statistics used and the fact that only positive results are reported.
http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124
It's their methodology; to wit -
Skeptic: I do not believe that your results accurately reflect reality, and therefore would like to see further experimentation.
Neocon "Skeptic:" Uh-huYUK, I no dat I ain't come frum no durn monkey, cuz da preacher-man done told me so!
Living in the midwest, I tend to see the latter far more than the former.
An enigma, wrapped in a riddle, shrouded in bacon and cheese
Let's publish it quickly, then get a tenured job and change topic before others find out it's all wrong!
Yeah, I'm going to get me a job with Big Green. I hear Greenpeace is paying new grads six-figure starting salaries*.
* Energy companies in north-eastern Alberta are actually offering six-figure salaries to new grad electricians among other trades jobs.
Fanatically anti-fanatical
These findings are no surprise to those who have been following medical science and research for the past decades. See for example what Dr John Ioannidis has to say about the consistency, accuracy and honesty (or lack thereof) of medical science in general: "as much as 90 percent of the published medical information that doctors rely on is flawed","There was plenty of published research, but much of it was remarkably unscientific, based largely on observations of a small number of cases", "he was struck by how many findings of all types were refuted by later findings" - and not just in epidemiological (statistical) studies, but also in randomized, double-blind clinical trials: "Baffled, he started looking for the specific ways in which studies were going wrong. And before long he discovered that the range of errors being committed was astonishing: from what questions researchers posed, to how they set up the studies, to which patients they recruited for the studies, to which measurements they took, to how they analyzed the data, to how they presented their results, to how particular studies came to be published in medical journals ... 80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 percent of supposedly gold-standard randomized trials, and as much as 10 percent of the platinum-standard large randomized trials."
Gary Taubes too denounced this accumulation of bias back in 2007: compliance bias, information bias, confirmation bias, etc. routinely introduce non-uniform effects that can be bigger than what you try to measure. And you cannot compensate for them because you cannot quantify them.
As Sander Greenland, one of the editor/authors of Modern Epidemiology, wrote in chapter 19 "Bias Analysis":
John Ioannidis, a medical statistics researcher on a small island in the Aegean, leads a group that has done significant work in this area. Here is an article in The Atlantic about his work.
From the article: ". . . Ioannidis laid out a detailed mathematical proof that, assuming modest levels of researcher bias, typically imperfect research techniques, and the well-known tendency to focus on exciting rather than highly plausible theories, researchers will come up with wrong findings most of the time. Simply put, if you’re attracted to ideas that have a good chance of being wrong, and if you’re motivated to prove them right, and if you have a little wiggle room in how you assemble the evidence, you’ll probably succeed in proving wrong theories right."
The glass is half glass.
Well, it's time to start a new journal (among 100 000 of already existing journals). Create a huge lab, get lots of funding to pay them for their work. And tell the to reproduce every result from every submitted paper. Publish only if result was reproductible. Expensive as hell, but soon that would be the only journal that people will bother to spend time on reading.
#
#\ @ ? Colonize Mars
#
Physicists get cranky if you show them research with p values higher than one in ten thousand, but medical research routinely produces p values as high as 0.2. On top of that, you have a large number of wealthy corporations who have powerful financial motivations to influence the results of medical research fraudulently, which probably doesn't come up as often in other fields. There has been a lot of news about problems with medical research lately, mostly because the medical research community itself is discussing these issues, which is a promising sign.
These replicated studies were a cost saving measure. Doing a clinical trial on people is more expensive than doing a preclinical trail on cell cultures. Before committing to the more expensive trail they tried to repeat the research the trial was based on. Over 10 years they did this 53 times and only got positive results 6 times. They then focused their money on those results.
A drug that works no better than the placebo could be used for years or decades before anyone figures out that it doesn't do anything but create side effects.
Not years, more like decades, maybe even a century....
As a child, I was subjected to almost every known type of prescription cough syrup/suppressant stuff to almost no effect (even the heavy duty codine laced stuff). Now ~30years later, they are only just admitting that the non-prescription stuff cough stuff is no better than placebo. Next thing you know they'll say the same thing about the prescription stuff.
By the time they take this stuff off the market I'm sure it'll have been sold for 100 years and then the OTC stuff will probably morph into something like Dr Pepper, Coca-cola, and Hires RootBeer which started life on the medicinal side of the fence.
Most pre-meds go on to do something unrelated to medicine. The acceptance rate to medical school is quite low, especially when people looked to medicine as a "guaranteed job" during uncertain economic times. In my undergraduate class, there were roughly 400 pre-meds initially, ~10 went to the affiliated medical school, and ~10 went to other medical schools. Even a single bad grade will dramatically worsen one's odds, hence why so many pre-meds succumb to the temptation of cheating.
Research is similar, because grants & funding are also stupidly competitive. This drives people to behave unethically as a calculated risk. Even well-intentioned people lose their objectivity when they dedicate their careers to something, and unobjective science is an oxymoron.