Slashdot Mirror
Can Bad Scientific Practice Be Fixed?
HughPickens.com writes: Richard Horton writes that a recent symposium on the reproducibility and
reliability of biomedical research discussed one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with science (PDF), one of our greatest human creations. The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. According to Horton, editor-in-chief of The Lancet, a United Kingdom-based medical journal, the apparent endemicity of bad research behavior is alarming. In their
quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world or retrofit hypotheses to fit their data.
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative. Tony Weidberg says that the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication following several high-profile errors. By filtering results through independent working groups, physicists are encouraged to criticize. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. "The good news is that science is beginning to take some of its worst failings very seriously," says Horton. "The bad news is that nobody is ready to take the first step to clean up the system."
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative. Tony Weidberg says that the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication following several high-profile errors. By filtering results through independent working groups, physicists are encouraged to criticize. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. "The good news is that science is beginning to take some of its worst failings very seriously," says Horton. "The bad news is that nobody is ready to take the first step to clean up the system."
The case against journalism is straightforward: much of the news articles, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, journalists has taken a turn towards darkness. The apparent endemicity of bad journalist behavior is alarming. In their quest for telling a compelling story, journalists too often sculpt facts to fit their preferred narrative of the world or retrofit hypotheses to fit their data.
Unlike journalists, however, science will always have to bow to reality. So, yeah, bad science practice will eventually run aground when reality hits, no matter how many epicycles one add to the model. But bad journalism will persists as long as it attracts eyeballs.
Oliver.
Much of the problem comes from studies being published whose data is not robust because the sample size is too small to be meaningfully significant. This needs to be headlined in the abstract if it is published at all; the best magazines should refuse anything without a decent sample size, whilst the ones further down the food chain should have statisticans on hand to ask hard questions.
Discovering an apparent effect should result in more research - not a rush to believe...
Your post hurts Michael Mann's feelings, and should be modded down for that reason alone. ;)
Feynman's take:
We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.
Two more examples from Ignition! by John Clark.
James Dewar (later Sir James, and the inventor of the Dewar flask and hence of the thermos botde), of the Royal Institute in London, in 1897 liquefied fluorine, which had been isolated by Moisson only eleven years before, and reported that the density of the liquid was 1.108. This wildly (and inexplicably) erroneous value (the actual density is 1.50) was duly embalmed in the literature, and remained there, unquestioned, for almost sixty years, to the confusion of practically everybody.
Bill Doyle, at North American, had also fired a small fluorine motor in 1947, but in spite of these successes, the work wasn't immediately followed up. The performance was good, but the density of liquid fluorine (believed to be 1.108 at the boiling point) was well below that of oxygen, and the military (JPL was working for the Army at that time) didn't want any part of it.
This situation was soon to change. Some of the people at Aerojet simply didn't believe Dewar's 54-year-old figure on the density of liquid fluorine, and Scott Kilner of that organization set out to measure it himself. (The Office of Naval Research put up the money.) The experimental difficulties were formidable, but he kept at it, and in July, 1951, established that the density of liquid fluorine at the boiling point was not 1.108, but rather a little more than 1.54. There was something of a sensation in the propellant community, and several agencies set out to confirm his results. Kilner was right, and the position of fluorine had to be re-examined. (ONR, a paragon among sponsors, and the most sophisticated —by a margin of several parsecs — funding agency in the business, let Kilner publish his results in the open literature in 1952, but a lot of texts and references still list the old figure. And many engineers, unfortunately, tend to believe anything that is in print.)
For years people had noted that a standing drum of acid slowly built up pressure, and had to be vented periodically. But they assumed that this pressure was a by-product of drum corrosion, and didn't think much about it. But then, around the beginning of 1950, they began to get suspicious. They put WFNA in glass containers and in the dark (to prevent any photochemical reaction from complicating the results) and found, to their dismay, that the pressure buildup was even faster than in an aluminum drum. Nitric acid, or WFNA at least, was inherently unstable, and would decompose spontaneously, all by itself. This was a revolting situation.
All of this goes to show that even well-respected scientists and engineers are not immune to bad science.
"Publish or Perish", Degrees that require new original ideas, Strict hierarchy structure...
Academic institutions are culturally stuck in victorian times. So if you want to work up, get the choice projects and research, you need to publish. The more your publish, the higher the chances you will move up. Because there is so much published material, people don't read it much, so they found that they can get credit for half ass work.
Your name becomes your brand, so when you try to get a grant your name+institution you will work for will get you the grant money.
There isn't any reason why Say State University of New York Buffalo can't get a grant to study seismology, but chances are it will go to University of California Berkeley not because they will do a better job, but because of the name.
Finally institutions haven't learned how to deal with today's political climate with the attempt for breaking news. Every Hypothesis is sold to the public as a new Theory... Then if that Hypothesis is shown false (as it is common in science) then the media who may have a political slant will go and say see Science is Wrong again, just like our political stance has predicted!
Science for the most part is quite work, collaborating with like minded people, with checks and balances to try to filter out strong egos. But it has gone commercial so these checks and balances are weaken as strong egos will win out.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
Half? In many fields (like medical research) it's essentially all, and there's no "at some point." Many places offer one or two year starting faculty appointments, at the end of which you're expected to have a major grant (success rate is somewhere around 10% on those). So you better get busy writing applications. Once you're established, you better keep writing them, because now you've got a lab full of people depending on you for their livelihood.
It's well more than half their in engineering disciplines as well. I worked for a research university for two decades and know that the more successful professor/researcher spends almost all their time on grant writing, with the best ones getting buy-out of their salaries so adjunct instructors can be brought in to teach their classes while they and their grad students focus on fulfilling the needs of one grant while working on the next three or five proposals. These faculty will often teach one undergrad and one grad class and that's about it. The rest of the time they are doing project management and business development tasks with the occasional sabbatical where they actually get to do research themselves. These profs also travel a lot in order to keep connections to research collaborators at other universities, private sector companies that either benefit from their research or are supplying equipment or other needs for their research and with program directors of NSF funding areas that are either current or former colleagues. They are, basically, mini-CEOs once they get to the point where they are pulling in $1 million or more per year in grant funds.