How Science Goes Wrong

dryriver sends this article from the Economist: "A simple idea underpins science: 'trust, but verify'. Results should always be subject to challenge from experiment. That simple but powerful idea has generated a vast body of knowledge. Since its birth in the 17th century, modern science has changed the world beyond recognition, and overwhelmingly for the better. But success can breed complacency. Modern scientists are doing too much trusting and not enough verifying — to the detriment of the whole of science, and of humanity. Too many of the findings that fill the academic ether are the result of shoddy experiments or poor analysis (see article). A rule of thumb among biotechnology venture-capitalists is that half of published research cannot be replicated. Even that may be optimistic. Last year researchers at one biotech firm, Amgen, found they could reproduce just six of 53 'landmark' studies in cancer research. Earlier, a group at Bayer, a drug company, managed to repeat just a quarter of 67 similarly important papers. A leading computer scientist frets that three-quarters of papers in his subfield are bunk. In 2000-10 roughly 80,000 patients took part in clinical trials based on research that was later retracted because of mistakes or improprieties. Even when flawed research does not put people's lives at risk — and much of it is too far from the market to do so — it squanders money and the efforts of some of the world's best minds. The opportunity costs of stymied progress are hard to quantify, but they are likely to be vast. And they could be rising."

Comments

Re:Peer review stretched to its limit by money

[fermion]

Also cancer research/biotech is barely science. No one would say Thomas Edison was a scientist, he was just someone who tried a bunch of stuff until something worked, with not necessarily a great deal of understanding of the basic science. We can argue the specifics of the points, but most of these people are just highly trained technicians paid to write papers that say what they are supposed to. Remember all the paper than came out supporting tobacco?

Most of this 'research' has significant consequences if they do not play out. No one is going to rock the boat by verifying results because such knowledge will keep the product from coming to market. Pharmaceuticals has a built in buffer to pay for future deaths, but those cannot be made if the product is never on the market.

Now for real science, this is not an issue. Researches will regularly ignore bad research, but the results are not often the most critical thing. It is the methods. Even bad result can lead to innovative experiment. I recall one case where the results of a technique were bad for years, but the technique proved very useful, and the errors were eventually discovered.In fact the damaging papers are sometimes where bad techniques are promulgated.