Slashdot Mirror
Can Bad Scientific Practice Be Fixed?
HughPickens.com writes: Richard Horton writes that a recent symposium on the reproducibility and
reliability of biomedical research discussed one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with science (PDF), one of our greatest human creations. The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. According to Horton, editor-in-chief of The Lancet, a United Kingdom-based medical journal, the apparent endemicity of bad research behavior is alarming. In their
quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world or retrofit hypotheses to fit their data.
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative. Tony Weidberg says that the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication following several high-profile errors. By filtering results through independent working groups, physicists are encouraged to criticize. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. "The good news is that science is beginning to take some of its worst failings very seriously," says Horton. "The bad news is that nobody is ready to take the first step to clean up the system."
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative. Tony Weidberg says that the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication following several high-profile errors. By filtering results through independent working groups, physicists are encouraged to criticize. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. "The good news is that science is beginning to take some of its worst failings very seriously," says Horton. "The bad news is that nobody is ready to take the first step to clean up the system."
Much of the problem comes from studies being published whose data is not robust because the sample size is too small to be meaningfully significant. This needs to be headlined in the abstract if it is published at all; the best magazines should refuse anything without a decent sample size, whilst the ones further down the food chain should have statisticans on hand to ask hard questions.
Discovering an apparent effect should result in more research - not a rush to believe...
Half? In many fields (like medical research) it's essentially all, and there's no "at some point." Many places offer one or two year starting faculty appointments, at the end of which you're expected to have a major grant (success rate is somewhere around 10% on those). So you better get busy writing applications. Once you're established, you better keep writing them, because now you've got a lab full of people depending on you for their livelihood.
It's well more than half their in engineering disciplines as well. I worked for a research university for two decades and know that the more successful professor/researcher spends almost all their time on grant writing, with the best ones getting buy-out of their salaries so adjunct instructors can be brought in to teach their classes while they and their grad students focus on fulfilling the needs of one grant while working on the next three or five proposals. These faculty will often teach one undergrad and one grad class and that's about it. The rest of the time they are doing project management and business development tasks with the occasional sabbatical where they actually get to do research themselves. These profs also travel a lot in order to keep connections to research collaborators at other universities, private sector companies that either benefit from their research or are supplying equipment or other needs for their research and with program directors of NSF funding areas that are either current or former colleagues. They are, basically, mini-CEOs once they get to the point where they are pulling in $1 million or more per year in grant funds.