Slashdot Mirror
Can Bad Scientific Practice Be Fixed?
HughPickens.com writes: Richard Horton writes that a recent symposium on the reproducibility and
reliability of biomedical research discussed one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with science (PDF), one of our greatest human creations. The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. According to Horton, editor-in-chief of The Lancet, a United Kingdom-based medical journal, the apparent endemicity of bad research behavior is alarming. In their
quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world or retrofit hypotheses to fit their data.
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative. Tony Weidberg says that the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication following several high-profile errors. By filtering results through independent working groups, physicists are encouraged to criticize. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. "The good news is that science is beginning to take some of its worst failings very seriously," says Horton. "The bad news is that nobody is ready to take the first step to clean up the system."
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative. Tony Weidberg says that the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication following several high-profile errors. By filtering results through independent working groups, physicists are encouraged to criticize. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. "The good news is that science is beginning to take some of its worst failings very seriously," says Horton. "The bad news is that nobody is ready to take the first step to clean up the system."
It's not.
Much of the problem comes from studies being published whose data is not robust because the sample size is too small to be meaningfully significant. This needs to be headlined in the abstract if it is published at all; the best magazines should refuse anything without a decent sample size, whilst the ones further down the food chain should have statisticans on hand to ask hard questions.
Discovering an apparent effect should result in more research - not a rush to believe...
Some of you may be visiting that Amazon link and wondering how the hell weilawei ended up spending $1,000 on a book ("Ignition!"). The book is too wonderful to be limited to the big spenders. Search for it online and you will find a PDF easily enough. It's an awesome read.
Half? In many fields (like medical research) it's essentially all, and there's no "at some point." Many places offer one or two year starting faculty appointments, at the end of which you're expected to have a major grant (success rate is somewhere around 10% on those). So you better get busy writing applications. Once you're established, you better keep writing them, because now you've got a lab full of people depending on you for their livelihood.
I agree it's not a problem. As can be seen at Retraction Watch, lots of bad science if found out and retracted. That's a good thing not a bad thing. One could ask how much of published science is made up and undetected but a better question would be how many results are simply crappy in the data or crappy in the analysis. It surely dwarfs the latter. But who cares. If the result is important it will be replicated. if it's not important then no one will cite it.
ultimately it's the well cited articles that also get vetted by reproduction. Those constitute the body of science moving forward. the rest goes into the gutter of history.
In skiing the saying is, if you fall and your fall isn't forward your not being aggressive enough. It's the same in science. People will make errors. If they weren't then then were not paying for aggressive enough research.
Some drink at the fountain of knowledge. Others just gargle.
Climate science is probably the most scrutinized field of science right now. And despite people saying the whole field is a crock, nothing of substance is found wrong.
I work in a science field. I have *never* seen "truth" or "fact" set by polling. Scientific controversy exists and disagreements exist. Researchers attempt to use carefully designed experiments with measurements to resolve those disagreements. Fringe researchers get a voice, and once in a while, an "out there" idea does pay off. I have attended many conferences where unconventional theories were presented.
However, when conveying scientific results to the public or policy makers, discussing what is consensus and what is not consensus does make sense. For that purpose, we as scientists and public trust holders shouldn't let the fringe distract us from our best understanding of the world. Because 99.99% of those fringe ideas are pretty much junk.
Science does permit challenges to existing models. But in most cases those challenges will have an extraordinarily difficult time of it. Difficult, not because of some kind of popularity contest. The current best models are there for a reason. They have survived knock-down fights with other models. They have been run through multiple experimental gauntlets. Theoretical models usually have multiple consequences and these consequences can be tested in multiple, cross-linking ways. A new model can be proposed, a new way of thinking put forth, but the burden is upon the proposer to show how this new way is consistent with all previous experiments. Fringe theories should not get a free pass because they are new or fringe.
Half? In many fields (like medical research) it's essentially all, and there's no "at some point." Many places offer one or two year starting faculty appointments, at the end of which you're expected to have a major grant (success rate is somewhere around 10% on those). So you better get busy writing applications. Once you're established, you better keep writing them, because now you've got a lab full of people depending on you for their livelihood.
It's well more than half their in engineering disciplines as well. I worked for a research university for two decades and know that the more successful professor/researcher spends almost all their time on grant writing, with the best ones getting buy-out of their salaries so adjunct instructors can be brought in to teach their classes while they and their grad students focus on fulfilling the needs of one grant while working on the next three or five proposals. These faculty will often teach one undergrad and one grad class and that's about it. The rest of the time they are doing project management and business development tasks with the occasional sabbatical where they actually get to do research themselves. These profs also travel a lot in order to keep connections to research collaborators at other universities, private sector companies that either benefit from their research or are supplying equipment or other needs for their research and with program directors of NSF funding areas that are either current or former colleagues. They are, basically, mini-CEOs once they get to the point where they are pulling in $1 million or more per year in grant funds.
Papers get 'impact scored'. Based on the number of times they are cited by other papers, especially other high impact papers. Basically Google page rank for papers. If Google ever tried to patent 'page rank', scientific papers 'impact scores' are prior art. It's even done 'on the internet'.
Not surprisingly, this is also gamed.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
Mann will sue your ass off, an innovation he has personally added to the scientific method.
Mann didn't sue anyone for hurting his feelings, or claiming he was wrong - he sued them for claiming, very explicitly, that he had committed fraud, and for calling him "the Jerry Sandusky of climate science".