Most Scientists 'Can't Replicate Studies By Their Peers' (bbc.com)

Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. From a report: This is frustrating clinicians and drug developers who want solid foundations of pre-clinical research to build upon. From his lab at the University of Virginia's Centre for Open Science, immunologist Dr Tim Errington runs The Reproducibility Project, which attempted to repeat the findings reported in five landmark cancer studies. "The idea here is to take a bunch of experiments and to try and do the exact same thing to see if we can get the same results." You could be forgiven for thinking that should be easy. Experiments are supposed to be replicable. The authors should have done it themselves before publication, and all you have to do is read the methods section in the paper and follow the instructions. Sadly nothing, it seems, could be further from the truth.

Comments

Fake science/sloppy science

[dave3548]

If you can't reproduce it, it's either fake or you were just being sloppy. Either way, it's no wonder ordinary civilians have doubts.

Re:Fake science/sloppy science

[Baron_Yam]

I have to disagree. To me, "a fair amount of success comes down to technique, not the written protocol" means you're not documenting your protocol adequately.

It would certainly be fair to say that some manual actions could take a lot of practice before the experimenter would likely be skilled enough to perform them, but there shouldn't be anything missing from the protocol documentation that someone attempting to reproduce the results would have to learn from scratch.

Excepting well-established standard practices of the field, of course. You don't have to teach from kindergarten up to post-grad.

I'm no bio researcher, but I am an IT guy and we could fill a library with books on substandard documentation making it difficult for others to follow in our footsteps.

Re:Fake science/sloppy science

[Thud457]

This pretty sums up my experiences reproducing experiments in the lab : ob. Electron Band Structure In Germanium, My Ass

Re:Fake science/sloppy science

[sjames]

If it's not a standard protocol, why isn't it documented?

Re:Fake science/sloppy science

[Salgak1]

. . . or that an unknown and unrecognized variable is at play.

Mind you, this goes back 40 years, but in my undergrad days, we were required to do a needle powder mount of a ground mineral, and use it in an x-ray diffractometry device. in order to identify the mineral by diffraction patterns.

The year my class did it, we were **all** off by about 5-6% from the reference shots, made years earlier (the sample sources remained the same). Turns out the manufacturer of the adhesive we used for the powder mounts (it was office-type rubber cement) had undergone a minor formula change, they had a new source of one ingredient, which had some metal dust contamination.

We had students doing the same experiment for 20 years previously, used identical techniques, and we STILL got different results. And we wouldn't have figured it out, if one of the TAs recalled that she had to get a new jar of rubber cement, the old one had solidified. . . . and it still took comparative chemical analyses of two different needle mounts of the same sample, but different years, to identify the difference.

Re:Fake science/sloppy science

[Dissenter]

I suppose that's a key element to the issue this article is discussing. If only the standard methods are in the publication and some novel augmentation of a process is necessary to produce those results, there is missing data and it could not be reproduced. Too many people are anxious to publish simply because it is part of their job to do so, but if some novel component is being persued through patent or other non-disclosed intellectual property, the publication should probably be either post-poned or not submitted. It's an odd catch 22 for folks in this area of research. I tend to agree that publishing something incomplete, however, simply extends ignorance rather than contributing to the education of your peers.

Reproducibility is hard.

[FellowConspirator]

In the experimental protocols listed in a paper, it is not unusual to have a method's section that is more or less an executive summary rather than a very detailed account of the underlying protocol. This is for two reasons: to great a level of detail leads to a methods section as big as the publication that the paper appears in, and second because many protocols more or less boil down to using a particular series kit or out-sourced lab service. Most journals require data supplements where an author must share their datasets in electronic form as an online addendum to the publication. I would support a similar requirement for a long-form protocol for reproduction of the study.

That said, some protocols necessarily take a lot of money, special equipment, a carefully selected population of volunteers, and time. Reproducing some studies can be outright impractical.

In computational biology and other computational extensions of the physical science, the reproducibility basically comes in the form of requirements to provide the software and raw data for a study. It's easy for the individual that compiles this information to verify that they get the same result as the one they report in the article. The concern there boils down to the provenance of the source data, which may be from registries, public data sets, or some combination of public and private data.

Re:Reproducibility is hard.

[RyanFenton]

Why are we still using printed journals?

Why is the amount of space a report takes up still an issue?

Details are important. If you want a short version, then make a summary, but don't cut out the detail available to do that.

In terms of ascii/unicode text, we're not going to run out of bytes to explain important scientific details.

Heck - make videos of the processes, mention part numbers, and even show mistakes that you encountered along the way in your notes! Video hosting is free, and shouldn't be going away anytime soon. Making a process replication video should be a normal thing.

If you're spending so much time anyway, so much of your life in these studies, what's the value in holding back important information?

Ryan Fenton

Re: s/drug trials/climate change/g

[reanjr]

Many studies have been done on anthropic climate change, but almost no experiments.

Science discourages reproducing

[Anonymous Coward]

Scientists are not rewarded for reproducing/debunking previous work. You can't easily get it published, because it is not regarded as new. Honestly, I think grad student projects should be almost entirely reproducing other results. It would insure that every important result is reproduced, and increase the emphasis of doing the science correctly rather than finding some novel result (which is usually a 2 sigma result which again can't be reproduced).

Re:Science discourages reproducing

[WrongMonkey]

You are vastly underestimating grad students. All grad students already have a bachelor's degree, had high enough grades to get accepted to grad school and had previous undergraduate research experience. Most incoming grad students have already published research papers. Many of them have had industry experience between undergrad and grad school. The average age of grad students is 33 years old. We're talking about mid-career professionals, not wet-behind the ears newbies.

Not Science, Medicine

[Roger W Moore]

This is medical research, not science. Medicine uses science because often the best way to cure something is to understand it but, very importantly, it has a very different motivation to science. Finding a "magic" pill which cures disease X without side effects but whose mechanism is completely unknown is great medicine but appalling science. Science is all about understanding how things work, medicine is all about treating human ailments.

This leads to a different approach using the tools of science. Medical researchers tend to focus far more on correlation over causation because that is what is most important to this. Unfortunately this approach leaves them open to random statistical effects which require a very good understanding of statistics to avoid and even then it can still be very easy to fool yourself e.g. the Monty Hall effect.

So lets call this problem what it is: a problem with medical research.

Re: s/drug trials/climate change/g

[Geoffrey.landis]

Many studies have been done on anthropic climate change, but almost no experiments.

The infrared absorption of carbon dioxide is experimentally measured in the laboratory. And there is a vast amount of measured data on the earth's atmosphere and climate, from surface, atmospheric, and orbital probes, not to mention probes of other planets; and we acquire terabytes of additional data every year.
The basics of Earth's energy balance are well understood, and they are understood, in part, because of this vast amount of experimental and observational data.

Questions require listening

[Geoffrey.landis]

It's not heresy if the science is sound. Simply questioning isn't valid, though.

Questioning, of course, is always valid. But "questioning" is useless when the questioner has no interest in listening to anybody answering the question.

Far too much of the "questioning" about climate science is from people who have no interest in any of the science, the measurements, or the data, and won't bother to learn anything about it.