How Science Goes Wrong

dryriver sends this article from the Economist: "A simple idea underpins science: 'trust, but verify'. Results should always be subject to challenge from experiment. That simple but powerful idea has generated a vast body of knowledge. Since its birth in the 17th century, modern science has changed the world beyond recognition, and overwhelmingly for the better. But success can breed complacency. Modern scientists are doing too much trusting and not enough verifying — to the detriment of the whole of science, and of humanity. Too many of the findings that fill the academic ether are the result of shoddy experiments or poor analysis (see article). A rule of thumb among biotechnology venture-capitalists is that half of published research cannot be replicated. Even that may be optimistic. Last year researchers at one biotech firm, Amgen, found they could reproduce just six of 53 'landmark' studies in cancer research. Earlier, a group at Bayer, a drug company, managed to repeat just a quarter of 67 similarly important papers. A leading computer scientist frets that three-quarters of papers in his subfield are bunk. In 2000-10 roughly 80,000 patients took part in clinical trials based on research that was later retracted because of mistakes or improprieties. Even when flawed research does not put people's lives at risk — and much of it is too far from the market to do so — it squanders money and the efforts of some of the world's best minds. The opportunity costs of stymied progress are hard to quantify, but they are likely to be vast. And they could be rising."

Comments

This is a real problem and conflict of interest

[shaitand]

All researchers in the sciences have a motivation to be published, in the form of recognition, academic progress, and financial motivation. Not many of them have an incentive stop working on looking great for producing results and check the work of someone else.

Re:Anti-science? See, now you have proof!

[phantomfive]

If you are trying to prove to young-earth creationists that the earth is old because they should trust scientists, you're doing the wrong thing. If you do that, you turn it into a fight about, "the guy I trust" vs "the guy you trust."

Instead, if you really want to talk to a young earth creationist (I don't know why you would), you need to show them the evidence. Really dig deep. If they want to discuss carbon dating, then dig in and show the evidence we have of why carbon dating works. Eventually, if they are willing to go along with you (and it will take a lot of work so they might not), they will turn into an old-earth creationist.

And you will absolutely learn something along the way. Never turn the discussion into an argument about "the guys I trust" vs "the guys you trust" because that argument is never won, by either side.

Re:Sounds Like Work...

[lgw]

It's not about being lazy. Feynman famously addressed this in his "Cargo Cult Science" rant in his Caltech commencement address given in 1974. (There's no recording AFAIK, that link is to someone reading the transcript).

He makes very good points: funding is for new results. Attempting to repeat another scientists published work is not a new result (unless you can't), and many places won't even allow you to try, unless it's something very sexy like observing the Higgs boson or something. It's an important structural problem, and it was worth calling attention to forty years ago.

There's no doubt that some unscrupulous researchers have noticed this and are gaming the system. The incentives to do so are particularly high in biochem.