Slashdot Mirror
Most Scientists 'Can't Replicate Studies By Their Peers' (bbc.com)
Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. From a report: This is frustrating clinicians and drug developers who want solid foundations of pre-clinical research to build upon. From his lab at the University of Virginia's Centre for Open Science, immunologist Dr Tim Errington runs The Reproducibility Project, which attempted to repeat the findings reported in five landmark cancer studies. "The idea here is to take a bunch of experiments and to try and do the exact same thing to see if we can get the same results." You could be forgiven for thinking that should be easy. Experiments are supposed to be replicable. The authors should have done it themselves before publication, and all you have to do is read the methods section in the paper and follow the instructions. Sadly nothing, it seems, could be further from the truth.
This is medical research, not science. Medicine uses science because often the best way to cure something is to understand it but, very importantly, it has a very different motivation to science. Finding a "magic" pill which cures disease X without side effects but whose mechanism is completely unknown is great medicine but appalling science. Science is all about understanding how things work, medicine is all about treating human ailments.
This leads to a different approach using the tools of science. Medical researchers tend to focus far more on correlation over causation because that is what is most important to this. Unfortunately this approach leaves them open to random statistical effects which require a very good understanding of statistics to avoid and even then it can still be very easy to fool yourself e.g. the Monty Hall effect.
So lets call this problem what it is: a problem with medical research.
. . . or that an unknown and unrecognized variable is at play.
Mind you, this goes back 40 years, but in my undergrad days, we were required to do a needle powder mount of a ground mineral, and use it in an x-ray diffractometry device. in order to identify the mineral by diffraction patterns.
The year my class did it, we were **all** off by about 5-6% from the reference shots, made years earlier (the sample sources remained the same). Turns out the manufacturer of the adhesive we used for the powder mounts (it was office-type rubber cement) had undergone a minor formula change, they had a new source of one ingredient, which had some metal dust contamination.
We had students doing the same experiment for 20 years previously, used identical techniques, and we STILL got different results. And we wouldn't have figured it out, if one of the TAs recalled that she had to get a new jar of rubber cement, the old one had solidified. . . . and it still took comparative chemical analyses of two different needle mounts of the same sample, but different years, to identify the difference.
I suppose that's a key element to the issue this article is discussing. If only the standard methods are in the publication and some novel augmentation of a process is necessary to produce those results, there is missing data and it could not be reproduced. Too many people are anxious to publish simply because it is part of their job to do so, but if some novel component is being persued through patent or other non-disclosed intellectual property, the publication should probably be either post-poned or not submitted. It's an odd catch 22 for folks in this area of research. I tend to agree that publishing something incomplete, however, simply extends ignorance rather than contributing to the education of your peers.
Dissenter
"There is no knowledge that is not power."