Slashdot Mirror
Most Scientists 'Can't Replicate Studies By Their Peers' (bbc.com)
Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. From a report: This is frustrating clinicians and drug developers who want solid foundations of pre-clinical research to build upon. From his lab at the University of Virginia's Centre for Open Science, immunologist Dr Tim Errington runs The Reproducibility Project, which attempted to repeat the findings reported in five landmark cancer studies. "The idea here is to take a bunch of experiments and to try and do the exact same thing to see if we can get the same results." You could be forgiven for thinking that should be easy. Experiments are supposed to be replicable. The authors should have done it themselves before publication, and all you have to do is read the methods section in the paper and follow the instructions. Sadly nothing, it seems, could be further from the truth.
Drug trials are limited in scope because there are restrictions on the patents of the studied compounds, which greatly limits the capacity to replicate the trial. Multiple studies have been done on climate, which is more open access.
As per the subject, this comes from the collision of two things that are completely counter to the process of science.
1) Patent theory. Since many more nations have access to patents than actually respect patents, it is self-destructive to put enough detail in a patent to actually build what the patent is for. For research papers, this has the benefit of being informed of any attempts to replicate the study, because the other labs will call in and ask questions to find out what was left out. This lets the guy who signed the paper know who is working in the same field and can decide whether to be helpful or antagonistic.
2) Fiction. "Publish or perish! " "No one actually reads the papers!" "Everyone else is too busy writing their own papers to even look at replicating the experiment, who cares if it doesn't actually work?" And other little labor-saving excuses that show scientists are just as dishonest and self-serving as politicians.
Then why not describe the novel techniques you developed to complete the research in the paper? Any process that is claimed to require special abilities is actually one the needs training.
The purpose of peer review is to identify incorrect theories and throw them out. This article says an awful lot more about the state of research today than it does about peer review which is doing what it's supposed to be doing.
Build a Man a Fire, and He'll Be Warm for a Day. Set a Man on Fire, and He'll Be Warm for the Rest of His Life.
This is medical research, not science. Medicine uses science because often the best way to cure something is to understand it but, very importantly, it has a very different motivation to science. Finding a "magic" pill which cures disease X without side effects but whose mechanism is completely unknown is great medicine but appalling science. Science is all about understanding how things work, medicine is all about treating human ailments.
This leads to a different approach using the tools of science. Medical researchers tend to focus far more on correlation over causation because that is what is most important to this. Unfortunately this approach leaves them open to random statistical effects which require a very good understanding of statistics to avoid and even then it can still be very easy to fool yourself e.g. the Monty Hall effect.
So lets call this problem what it is: a problem with medical research.
sometimes thats done purposefully so that you can get to the next paper before the other person. /p
not a good system but I know plenty of researchers who put in enough to explain what they did, but not enough so you could reproduce it without a bit of effort/research yourself. /p
definitely not the ideal, but people care about getting the next paper first so they can advance their career. publish or perish
www.RacquetUp.org - Helping Detroit Youth
. . . or that an unknown and unrecognized variable is at play.
Mind you, this goes back 40 years, but in my undergrad days, we were required to do a needle powder mount of a ground mineral, and use it in an x-ray diffractometry device. in order to identify the mineral by diffraction patterns.
The year my class did it, we were **all** off by about 5-6% from the reference shots, made years earlier (the sample sources remained the same). Turns out the manufacturer of the adhesive we used for the powder mounts (it was office-type rubber cement) had undergone a minor formula change, they had a new source of one ingredient, which had some metal dust contamination.
We had students doing the same experiment for 20 years previously, used identical techniques, and we STILL got different results. And we wouldn't have figured it out, if one of the TAs recalled that she had to get a new jar of rubber cement, the old one had solidified. . . . and it still took comparative chemical analyses of two different needle mounts of the same sample, but different years, to identify the difference.
I suppose that's a key element to the issue this article is discussing. If only the standard methods are in the publication and some novel augmentation of a process is necessary to produce those results, there is missing data and it could not be reproduced. Too many people are anxious to publish simply because it is part of their job to do so, but if some novel component is being persued through patent or other non-disclosed intellectual property, the publication should probably be either post-poned or not submitted. It's an odd catch 22 for folks in this area of research. I tend to agree that publishing something incomplete, however, simply extends ignorance rather than contributing to the education of your peers.
Dissenter
"There is no knowledge that is not power."
The part of Eisenhower's speech that is rarely quoted, for some reason: "Today, the solitary inventor, tinkering in his shop, has been overshadowed by task forces of scientists in laboratories and testing fields. In the same fashion, the free university, historically the fountainhead of free ideas and scientific discovery, has experienced a revolution in the conduct of research. Partly because of the huge costs involved, a government contract becomes virtually a substitute for intellectual curiosity. For every old blackboard there are now hundreds of new electronic computers.
The prospect of domination of the nation's scholars by Federal employment, project allocations, and the power of money is ever present and is gravely to be regarded.
Yet, in holding scientific research and discovery in respect, as we should, we must also be alert to the equal and opposite danger that public policy could itself become the captive of a scientific technological elite."