Slashdot Mirror
How Science Goes Wrong
dryriver sends this article from the Economist:
"A simple idea underpins science: 'trust, but verify'. Results should always be subject to challenge from experiment. That simple but powerful idea has generated a vast body of knowledge. Since its birth in the 17th century, modern science has changed the world beyond recognition, and overwhelmingly for the better. But success can breed complacency. Modern scientists are doing too much trusting and not enough verifying — to the detriment of the whole of science, and of humanity. Too many of the findings that fill the academic ether are the result of shoddy experiments or poor analysis (see article). A rule of thumb among biotechnology venture-capitalists is that half of published research cannot be replicated. Even that may be optimistic. Last year researchers at one biotech firm, Amgen, found they could reproduce just six of 53 'landmark' studies in cancer research. Earlier, a group at Bayer, a drug company, managed to repeat just a quarter of 67 similarly important papers. A leading computer scientist frets that three-quarters of papers in his subfield are bunk. In 2000-10 roughly 80,000 patients took part in clinical trials based on research that was later retracted because of mistakes or improprieties. Even when flawed research does not put people's lives at risk — and much of it is too far from the market to do so — it squanders money and the efforts of some of the world's best minds. The opportunity costs of stymied progress are hard to quantify, but they are likely to be vast. And they could be rising."
Possibly more importantly, pseudoscience is the articles worst nightmare.
The defensiveness now built into some fields (and here I'm thinking climate science), because of unrelenting, personal attacks does put important discussions like this into a defensive context.
And this is another bitter fruit produced by the anti-science industry, because these discussions are important to have. There are a lot of mistakes in science, but (seeming to me increasingly) there is also data falsification and fraud. [Retraction watch](http://retractionwatch.wordpress.com/) is a great website, but it makes sickening reading, and I suspect that it only scratches the surface.
I mean, sometimes, no fucks whatsoever are given. How that got past peer review blows the mind. And any of these.
Remember this letter to Nature (FFS!) pointing out that 70% of the papers in one of their issues didn't say what the error bar represented. How that got past the reviewers is mind boggling. Imagining how it got past the authors requires mental gymnastics. (Since the letter, Nature articles are much better, but Peer Review is not what is catching the errors).
So, lets talk about errors in scientific research, and lets talk about scientific fraud. It's important because its rampant, and despite that there are nutjobs seeing it in their peculiar light lets not be put off. This conversation needs to be had more often, because the problem is dug in at the highest levels of academic prestige.
Props to the Economist for bringing this up. I'd like to see this discussed in Cell, Nature and Science. And I'd like to see credible career protection for whistle-blowers.
You really have no idea how 'publish or perish' is involved?
Here's a clue: when was the last time you delayed publication (of eminently publishable results) to run some extra tests, or perform alternative forms of verification? I've never had a supervisor allow such things in my entire career. It's always a case of publishing as soon as possible (i.e. as soon as a study has the remotest chance of getting past reviewers).
I tend to be very cautious in my approach to things, and I've often wanted to do additional verification work. Not to target a better journal or a second publication, but just for the sake of more solid conclusions. I'm never allowed to do this, and I even recognise that it's not part of my job to cause any problems over it, for the very economic reasons that you mention. This bothers me deeply, but it doesn't seem to bother the kind of people who care more about their careers than about the veracity of their results. I've even been told on a few occasions that my reticence to publish some of my own simulation work that "should already be out there" is bad for my career.
In my perception, those who are more career-driven have an advantage in gaming the system. They are rewarded for publishing multiple papers of shallow scope and relatively minor significance; spreading what should be presented once as thin as possible across multiple publications. We all know it's a game to be played; that those evaluating our early-career performance really have no clue whether a publication is important or not. By the time they find out, those who've gamed the system well will already have tenure.