Slashdot Mirror
Are Research Papers Less Accurate and Truthful Than in the Past? (economist.com)
An anonymous reader shares an Economist report: An essential of science is that experiments should yield similar results if repeated. In recent years, however, some people have raised concerns that too many irreproducible results are being published. This phenomenon, it is suggested, may be a result of more studies having poor methodology, of more actual misconduct, or of both. Or it may not exist at all, as Daniele Fanelli of the London School of Economics suggests in this week's Proceedings of the National Academy of Sciences. First, although the number of erroneous papers retracted by journals has increased, so has the number of journals carrying retractions. Allowing for this, the number of retractions per journal has not gone up. Second, scientific-misconduct investigations by the Office of Research Integrity (ORI) in America are no more frequent than 20 years ago, nor are they more likely to find wrongdoing.
In 2016, The Journal Nature published a story by Monya Baker, where more than 70% of 1,576 researchers tried and failed to reproduce other scientist's experiments [2].
Even worse, many did claim to have reproduced the Pons and Fleischmann Cold Fusion experiment shortly after their press release in 1989 [3]. So many in fact, Nathan Lewis of Cal Tech quipped "Cold fusion has been verified by no university without a good football team" [4].
The problem has been around for decades. I'm thinking there might be reasons, like patents, contracts, grants, money, and prestige. It could be that science, or at least a bunch of scientists, ain't what they're cracked up to be. Or maybe football appendages and their cozens just aren't that important.
[1]https://en.wikipedia.org/wiki/Replication_crisis
[2]https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970
[3]https://en.wikipedia.org/wiki/Fleischmann%E2%80%93Pons_experiment
https://bwi.forums.rivals.com/threads/scientists-fleischmann-and-pons-cold-fusion-or-cold-illusion-25-years-later.10260/
You aren't a scientist, but those of us that are know 80% of our job is writing grant proposals and networking for opportunities to fund our research. If you do not provide results beneficial to a grant provider, there will not be a second grant.
This applies to mostly medicine and social science see John Ioannidis's research paper "Why Most Published Research Findings Are False" : http://journals.plos.org/plosm... It seems to me the sciences that deal with statistical p-value significance are all subject to false published research findings , for instance, see Craig Bennet's "Neural correlatates of interspecies perspectitve taking in post-mortem Salmon : An argument for multiple comparisons corrections". http://prefrontal.org/files/po... The paper is a deadpan gag and a veiled attack on sloppy methodology among neuroimaging researchers. Also, researchers run the Baltimore Stockbroker scam : https://somemathematicalmusing... When they selectively choose not to publish certain results in favor of other ones etc... So on and so forth etc...
This isn't a matter of misconduct, that's the wrong way to look at the current failure of science to... do science. (I am a scientist.)
Other metrics are more useful. My favorite is "research efficiency." This is a decidedly commercial metric, it's the amount of revenue or economic activity (in dollars) generated by $1 of scientific research investment. It's been going down since about 1980. Surprisingly, research areas pitched as "basic research" (i.e. math, astronomy) tend to do well with this metric. It's the research that's sold to the public as industrially focused (i.e. my field, nanotechnology) that tends to do the worst.
Another useful metric is the % of science PhDs who stay in science for at least 10 years after getting their degree. This measures how effective we are at training our scientific workforce. That's down significantly over the last 30 years as well. What we teach people now is not what they need to succeed in science after training (which is getting longer and longer).
The metric most scientists are looking for is reproducibility, or the percentage of papers which can be repeated by simply following the instructions in the paper. Papers have grown in length and complexity in the last 40 years. It's pretty hard to argue that reproducibility has actually gone down because older papers simply don't include details we now expect. Of course, this is very hard to measure in any case. That's the thesis of TFA. It doesn't change the very real feeling (and data) that science is somehow not delivering on our investment.
Misconduct is... you're going to have some when there are people involved. You're also going to have mistakes and papers which are disproven very quickly. I have a personal pet peeve for papers that promise extraordinarily cheap hardware by assuming labor is free, manufacturing can be done at large scale without investment in tooling, and working capital is free. Things like this are not actually misconduct, no matter how misleading they are.