Slashdot Mirror
Most Scientists 'Can't Replicate Studies By Their Peers' (bbc.com)
Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. From a report: This is frustrating clinicians and drug developers who want solid foundations of pre-clinical research to build upon. From his lab at the University of Virginia's Centre for Open Science, immunologist Dr Tim Errington runs The Reproducibility Project, which attempted to repeat the findings reported in five landmark cancer studies. "The idea here is to take a bunch of experiments and to try and do the exact same thing to see if we can get the same results." You could be forgiven for thinking that should be easy. Experiments are supposed to be replicable. The authors should have done it themselves before publication, and all you have to do is read the methods section in the paper and follow the instructions. Sadly nothing, it seems, could be further from the truth.
If you can't reproduce it, it's either fake or you were just being sloppy. Either way, it's no wonder ordinary civilians have doubts.
In the experimental protocols listed in a paper, it is not unusual to have a method's section that is more or less an executive summary rather than a very detailed account of the underlying protocol. This is for two reasons: to great a level of detail leads to a methods section as big as the publication that the paper appears in, and second because many protocols more or less boil down to using a particular series kit or out-sourced lab service. Most journals require data supplements where an author must share their datasets in electronic form as an online addendum to the publication. I would support a similar requirement for a long-form protocol for reproduction of the study.
That said, some protocols necessarily take a lot of money, special equipment, a carefully selected population of volunteers, and time. Reproducing some studies can be outright impractical.
In computational biology and other computational extensions of the physical science, the reproducibility basically comes in the form of requirements to provide the software and raw data for a study. It's easy for the individual that compiles this information to verify that they get the same result as the one they report in the article. The concern there boils down to the provenance of the source data, which may be from registries, public data sets, or some combination of public and private data.
Scientists are not rewarded for reproducing/debunking previous work. You can't easily get it published, because it is not regarded as new. Honestly, I think grad student projects should be almost entirely reproducing other results. It would insure that every important result is reproduced, and increase the emphasis of doing the science correctly rather than finding some novel result (which is usually a 2 sigma result which again can't be reproduced).
It's not heresy if the science is sound. Simply questioning isn't valid, though.
Questioning, of course, is always valid. But "questioning" is useless when the questioner has no interest in listening to anybody answering the question.
Far too much of the "questioning" about climate science is from people who have no interest in any of the science, the measurements, or the data, and won't bother to learn anything about it.
http://www.geoffreylandis.com