Slashdot Mirror
Can Bad Scientific Practice Be Fixed?
HughPickens.com writes: Richard Horton writes that a recent symposium on the reproducibility and
reliability of biomedical research discussed one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with science (PDF), one of our greatest human creations. The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. According to Horton, editor-in-chief of The Lancet, a United Kingdom-based medical journal, the apparent endemicity of bad research behavior is alarming. In their
quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world or retrofit hypotheses to fit their data.
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative. Tony Weidberg says that the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication following several high-profile errors. By filtering results through independent working groups, physicists are encouraged to criticize. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. "The good news is that science is beginning to take some of its worst failings very seriously," says Horton. "The bad news is that nobody is ready to take the first step to clean up the system."
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative. Tony Weidberg says that the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication following several high-profile errors. By filtering results through independent working groups, physicists are encouraged to criticize. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. "The good news is that science is beginning to take some of its worst failings very seriously," says Horton. "The bad news is that nobody is ready to take the first step to clean up the system."
Feynman's take:
We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.
Two more examples from Ignition! by John Clark.
James Dewar (later Sir James, and the inventor of the Dewar flask and hence of the thermos botde), of the Royal Institute in London, in 1897 liquefied fluorine, which had been isolated by Moisson only eleven years before, and reported that the density of the liquid was 1.108. This wildly (and inexplicably) erroneous value (the actual density is 1.50) was duly embalmed in the literature, and remained there, unquestioned, for almost sixty years, to the confusion of practically everybody.
Bill Doyle, at North American, had also fired a small fluorine motor in 1947, but in spite of these successes, the work wasn't immediately followed up. The performance was good, but the density of liquid fluorine (believed to be 1.108 at the boiling point) was well below that of oxygen, and the military (JPL was working for the Army at that time) didn't want any part of it.
This situation was soon to change. Some of the people at Aerojet simply didn't believe Dewar's 54-year-old figure on the density of liquid fluorine, and Scott Kilner of that organization set out to measure it himself. (The Office of Naval Research put up the money.) The experimental difficulties were formidable, but he kept at it, and in July, 1951, established that the density of liquid fluorine at the boiling point was not 1.108, but rather a little more than 1.54. There was something of a sensation in the propellant community, and several agencies set out to confirm his results. Kilner was right, and the position of fluorine had to be re-examined. (ONR, a paragon among sponsors, and the most sophisticated —by a margin of several parsecs — funding agency in the business, let Kilner publish his results in the open literature in 1952, but a lot of texts and references still list the old figure. And many engineers, unfortunately, tend to believe anything that is in print.)
For years people had noted that a standing drum of acid slowly built up pressure, and had to be vented periodically. But they assumed that this pressure was a by-product of drum corrosion, and didn't think much about it. But then, around the beginning of 1950, they began to get suspicious. They put WFNA in glass containers and in the dark (to prevent any photochemical reaction from complicating the results) and found, to their dismay, that the pressure buildup was even faster than in an aluminum drum. Nitric acid, or WFNA at least, was inherently unstable, and would decompose spontaneously, all by itself. This was a revolting situation.
All of this goes to show that even well-respected scientists and engineers are not immune to bad science.
That's part of it... But I believe the biggest problem is science fairs. Once heralded as a great way to get kids involved in science and the scientific method has been ruined by a culture of excessive safety, pandering to kids, and incompetent science teachers. First, every kids science toy has been neutered by safety culture. I'm not saying we should have kits with mercury and radioactive materials like we did in the 50s, but "science" kits where you make kitchen goo instead of actual chemical reactions is lame and boring. Kids are not fooled.
Second, the increasing pressure to pass all kids or give them participation ribbons is very present at the science fair. Many kids are forced to participate, and in many fairs judges have to assign a minimum score of "good" or some such term. I have judged at the STATE LEVEL (as in, they had to do very well at the school and county levels) and have had to assign this minimum score which was still a gift. Kids come up with buzzword laden projects and make elaborate art projects that get ooohs and ahhs from non-technical people while doing no research and offering conclusions that are demonstrably wrong. Don't believe me? Go to a science fair some time and count the number of "experiments" showing ethanol has more energy content than gasoline. There are usually a dozen at the state science fair I judge. I also wonder how many projects are done primarily by the parents who don't want their kids to do poorly.
Finally, the incompetency of science teachers... This is not applicable to all teachers, but especially in poorer areas and in under performing schools, science teachers have no science background and don't understand the scientific method. They don't understand research, citations, hypotheses, or conclusions. They don't even take the time to verify experimental results with a quick Google search. The comforting thing I've noticed from judging student science projects is that most of the kids KNOW their teachers are incompetent and bullshitted their way to a good score at the science fair. At the state level, they are completely unprepared for actual questions on subject matter by professionals in the various fields. I'm a civil engineer, and I've had to shake my head in disbelief that projects are off by an order of magnitude from what they should be and it is a shock for the student to hear that as no one has reviewed or questioned their work before the state level.
What we need is a new science fair system where teachers can mentor students on projects, but teachers don't judge projects. Projects should only be judged by people familiar with the subject matter and the scientific method. If they can't scrape together the judges, maybe the science fair needs to go away or there needs to be an active campaign to recruit and support professionals to judge school science fairs. It should be no surprise that the science fair kids have grown up to do research that panders to public opinion, are lazy, have poor citations, and are filled with self-confirming results.
Yes it's an anecdote! Were you expecting original research in a Slashdot comment?
I think part of the problem is that nobody wants to publish a paper where the experiment failed--but they should.
Failures are useful; they're not wasted time. You've almost certainly learned something from a failed experiment. Maybe you learned that the setup wasn't rigorous enough, or maybe you just learned that a certain avenue of research wasn't viable for one reason or another. I get that journals are looking for breakthroughs, but it would be so useful to read a paper in your field and find out that someone already tried the thing you're attempting, and now you don't have to fail in exactly the same way.
But that requires a much more collaborative system, and one where the community is interested in finding answers, not glory.
I agree with your general tone and statement. However it is important to note the inherent limitations of biomedical research. Generally one CANNOT do large scale studies needed to get a statistically robust result. All of physics and astrophysics generally use the 5 sigma discover requirement which means you have to measure the effect to 3e-7. You cannot do this with people as subjects. It is hard to do this with ANY biological subject. Many of the issues brought up stem from this.
I think much of the problem is exacerbated by the public-or-perish mentality but is even more affected by the total lack of reporting null results (when you DO NOT see anything). This skews your overall distribution. It is like not accounting for trials (because you aren't). In biomedical research they need to spend more time quantifying their trials and placing their results in the proper statistical context. Just staying that you are less likely to get parkinson's disease if you drink coffee because we asked a bunch of people isn't the whole story. How many questions did you ask? Was it 100? Did you treat all those as essentially trials?