Slashdot Mirror

← Back to Stories (view on slashdot.org)

Are Research Papers Less Accurate and Truthful Than in the Past? (economist.com)

Posted by msmash on from the debunking-myths dept.

An anonymous reader shares an Economist report: An essential of science is that experiments should yield similar results if repeated. In recent years, however, some people have raised concerns that too many irreproducible results are being published. This phenomenon, it is suggested, may be a result of more studies having poor methodology, of more actual misconduct, or of both. Or it may not exist at all, as Daniele Fanelli of the London School of Economics suggests in this week's Proceedings of the National Academy of Sciences. First, although the number of erroneous papers retracted by journals has increased, so has the number of journals carrying retractions. Allowing for this, the number of retractions per journal has not gone up. Second, scientific-misconduct investigations by the Office of Research Integrity (ORI) in America are no more frequent than 20 years ago, nor are they more likely to find wrongdoing.

18 of 119 comments (clear)

  1. My research says.. by mnemotronic · · Score: 5, Funny

    My research says that overall accuracy has declined at 0.65 radians per fortnight, factoring out verisimilitude mitigation factors where tensile strength is less than 2.227BeV per leapyear. Use of odd numbers and fractional percentages leads to higher levels of acceptance among those who don't know how to spell per centage.

    --
    The Russians have won. They have made the world a cesspool of distrust, greed, fear and hate.
    1. Re:My research says.. by IDrinkFatCashews · · Score: 3, Insightful

      The quantity of information available today is so staggering that we cannot know everything about a subject. For example, it's estimated that anyone attempting to research what's known about depression would have to read over 100,000 studies on the subject. And there's the problem of trying to decide which studies have produced reliable results.

      Similarly, for information on other topics, not only is there a huge quantity available but with a very uneven level of quality. You don't want to rely on the news in the headlines of sensational tabloids near supermarket checkout counters, and it's just as hard to know how much to accept of what's in all the books, magazines, pamphlets, newspapers, journals, brochures, Web sites, and various media reports that are available. People want to convince you to buy their products, agree with their opinions, rely on their data, vote for their candidate, consider their perspective, or accept them as experts. In short, you have to sift and make decisions all the time, and you want to make responsible choices that you won't regret.

      Evaluating sources is an important skill. It's been called an art as well as work—much of which is detective work. You have to decide where to look, what clues to search for, and what to accept. You may be overwhelmed with too much information or too little. The temptation is to accept whatever you find. But don't be tempted. Learning how to evaluate effectively is a skill you need both for writing papers and for your life.

    2. Re:My research says.. by jellomizer · · Score: 2

      The flaw isn’t in science, but in the education fields motto, publish or perish.
      To be funded scientists are expected to show results. Most of the time these results are no conclusive evidence. Which doesn’t get those grants in the door. And will not get your name known.
      Scientists are people too, so like all of us when under pressure, will sacrifice their ideals for a paycheck. Emblish a paper, write on a outlier action that was interesting.
      Being the people who pay for these grants rarely read the paper past a quick summary. Means we need to do what it takes to keep funding.
      If they can come up with a consistent way to pay scientists to do science knowing quite well this is science not engineering, where outcomes are knowledge, not products. Where such knowledge may be used in the future by the engineers to make a viable tool or product.

      --
      If something is so important that you feel the need to post it on the internet... It probably isn't that important.
    3. Re:My research says.. by liquid_schwartz · · Score: 2

      I largely agree with you. Unfortunately support for government becomes mixed when it dabbles in multiple realms and conflates the two. For example I love the idea of government supported basic research and allowing US companies to license that under preferential terms. This is a win for the US in general. However knowing that the priorities of many in government is not getting things done but having their preferred person getting things done I don't trust the government to be a good steward anymore. They want a new antibiotic sure but they also want a certain number of discoveries to happen from aggrieved group X/Y/Z. If push comes to shove which is more important, good discoveries or who discovers them? If I had confidence that they were only interested in good discoveries regardless of who discovers them then I'd wholeheartedly support basic research. However by consistently demonizing me, a (gasp) white male who works in technology, I have no confidence in their willingness to fund winning research based on merit.

  2. No, the transparency is just better today. by Anonymous Coward · · Score: 4, Interesting

    Lots of old reports, research, etc are known to have been grossly biased, outright doctored, or using questionable results towards questionable confirmations or refutations of the hypothesis.

    As a simple example, go read up on the Coca plant, and UN level attempts to eradicate wild plants from its ENTIRE HABITAT RANGE and the ecological damage that has been done as a result.

    Hint: While refined cocaine in recreational quantities is dangerous and addictive, individual leaves contain 5 percent of coca extracts per leaf mass, and have non-recreational uses especially in their natural range, as well as modern commercial success as teas and other herbal supplements. Furthermore use as a topical anesthetic requires 1/100th of the dosage used by people to get high. The danger is that unlike marijuana, and more like opiates, cocaine can saturate receptors to the point of causing cardiac arrest or other serious medical issues if misused.

    My point being: Any science currently considered controversial will have politically motivated research aimed to either prove or refute the stance that is most politically favorable to the people in power, whether those people are religious, governmental, or commercial in nature. Always has been, likely always will be.

  3. Everything was always better by rsilvergun · · Score: 5, Insightful

    in the good 'ole days. Especially the good 'ole days before computers made it easy to track things.

    --
    Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
  4. Why are are the headlines now questions? by Notabadguy · · Score: 5, Insightful

    Almost every slashdot article in the last couple days has been a question.

    Fucking knock it off.

  5. Re:Simple answer by alvinrod · · Score: 3, Funny

    I recall reading one such paper about this problem, but I don't know if anyone has been able to reproduce the results it had.

  6. This ain't news, nerds. by Humbubba · · Score: 5, Informative
    The term 'replication crisis' has been around since 2010, when more and more scientists found they could not reproduce the results of experiments of others [1].

    In 2016, The Journal Nature published a story by Monya Baker, where more than 70% of 1,576 researchers tried and failed to reproduce other scientist's experiments [2].

    Even worse, many did claim to have reproduced the Pons and Fleischmann Cold Fusion experiment shortly after their press release in 1989 [3]. So many in fact, Nathan Lewis of Cal Tech quipped "Cold fusion has been verified by no university without a good football team" [4].

    The problem has been around for decades. I'm thinking there might be reasons, like patents, contracts, grants, money, and prestige. It could be that science, or at least a bunch of scientists, ain't what they're cracked up to be. Or maybe football appendages and their cozens just aren't that important.

    [1]https://en.wikipedia.org/wiki/Replication_crisis

    [2]https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970

    [3]https://en.wikipedia.org/wiki/Fleischmann%E2%80%93Pons_experiment

    https://bwi.forums.rivals.com/threads/scientists-fleischmann-and-pons-cold-fusion-or-cold-illusion-25-years-later.10260/

  7. Cart before horse? by TapeCutter · · Score: 3, Insightful

    The summary seems to suggest that results should be reproduced before a paper is published?

    --
    And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
  8. Grants May Have Agendas by eggman9713 · · Score: 4, Interesting

    I may be dead wrong on this, but it seems like many of the studies and papers put out today are funded by grants from organizations which often have a (even if subtle) political or ideological agenda. And if the studies they fund support their position, they hand out more grants. If the studies go against it, that university sees its grants from that organization reduced. Perhaps this has an effect on the results of the studies? I'd like to hope not but it seems like anything we think is right is upside down anymore. I haven't looked into this very closely to see if my anecdotal data point is valid, but I'd like to see if anyone can validate it.

    1. Re:Grants May Have Agendas by PopeRatzo · · Score: 2

      I may be dead wrong on this, but it seems like many of the studies and papers put out today are funded by grants from organizations which often have a (even if subtle) political or ideological agenda.

      I doubt the political or ideological agenda of current scientists is anywhere near as obvious, over-arching, and narrowly-defined as that of the Royal Society or Oxford of a century ago.

      --
      You are welcome on my lawn.
    2. Re:Grants May Have Agendas by Anonymous Coward · · Score: 3, Informative

      You aren't a scientist, but those of us that are know 80% of our job is writing grant proposals and networking for opportunities to fund our research. If you do not provide results beneficial to a grant provider, there will not be a second grant.

    3. Re:Grants May Have Agendas by tlhIngan · · Score: 4, Insightful

      I may be dead wrong on this, but it seems like many of the studies and papers put out today are funded by grants from organizations which often have a (even if subtle) political or ideological agenda. And if the studies they fund support their position, they hand out more grants. If the studies go against it, that university sees its grants from that organization reduced. Perhaps this has an effect on the results of the studies? I'd like to hope not but it seems like anything we think is right is upside down anymore. I haven't looked into this very closely to see if my anecdotal data point is valid, but I'd like to see if anyone can validate it.

      No, that's always been the case. Lead in gasoline, smoking, acid rain, pesticides, etc., they all had studies to prove that they caused no harm at all. In fact, there are people who basically "manage' this sort of publication, a playbook if you will. As in, the best way to counter something bad in your industry is to manufacture controversy, and the way you do that is by getting studies done in your favor. I cite those cases above because those were "managed" by the same group of people who basically do just that - manufacture controversy. (And yes, that same group is behind climate change opposition as well).

      Some history of that can be found in Merchants of Doubt. They came up with the playbook on how to manufacture controversy and thus push regulations out.

      And let's not forget other cases like vaccines causing autism and plenty of food related papers all paid for by various aspects of industry.

      What's happened is recently the Internet has made it much easier to find information, so hunting down who the sponsors of a paper out is much easier even when they hide through 10 layers of corporate shields, and people are able to seek out the original document much more easily and thus analyze the results. The fact that everyone is moving towards open data as well makes it much easier to spot frauds.

  9. Article Self-contradicts by Roger+W+Moore · · Score: 4, Interesting

    This article appears to contradict itself. It claims that the rate of inaccurate papers as a fraction of the total is not increasing. However, it also notes that countries with weaker misconduct policies, like China and India, have far higher rates of problem papers. It also notes that while the fraction of papers in these countries with issues is approximately constant the overall share of papers coming from these countries is increasing. Hence, the overall fraction of papers with problems must be increasing too because more and more papers are coming from countries with higher rates of inaccuracies while each country's individual accuracy rate (as a fraction) is constant.

    It also seems very narrowly focussed on deliberate attempts to mislead since it concentrates on discipline procedures and investigations. However, the reproducibility problem is generally acknowledged to be mainly due to poor scientific practice, e.g. claiming that correlation implies causation or not understanding statistics, and not due to deliberate malfeasance.

    The data also show that there does appear to be a slight increase in the number of corrections per journal - although this is only small and the plot fails to provide error bars so it is impossible to know whether or not this is statistically meaningful. It also cryptically mentions that this is for journals which issue corrections suggesting that there are journals which do not issue them.

    The number of invitations I get to be an editor on new journals by predatory publishers has markedly increased over the past few years so, at least based on my experience, that there appear to be many more predatory journals than there used to be and I would be amazed if any cared enough to publish errata given that there is no money in it for them so, if the fraction of junk publications has increased this might entirely hide a large source of irreproducible papers from this study.

  10. Office of Research Integrity has more resources? by drnb · · Score: 2

    scientific-misconduct investigations by the Office of Research Integrity (ORI) in America are no more frequent than 20 years ago

    Does the Office of Research Integrity have access to more resources than they did 20 years ago? If the number of misconduct investigations is limited by their capacity to investigate then "no more frequent" is meaningless. If ORI can expand their capacity as needed, as more questionable research is reported, then "no more frequent" may be meaningful.

  11. This applies mostly to medicine and social science by kfburke39 · · Score: 3, Informative

    This applies to mostly medicine and social science see John Ioannidis's research paper "Why Most Published Research Findings Are False" : http://journals.plos.org/plosm... It seems to me the sciences that deal with statistical p-value significance are all subject to false published research findings , for instance, see Craig Bennet's "Neural correlatates of interspecies perspectitve taking in post-mortem Salmon : An argument for multiple comparisons corrections". http://prefrontal.org/files/po... The paper is a deadpan gag and a veiled attack on sloppy methodology among neuroimaging researchers. Also, researchers run the Baltimore Stockbroker scam : https://somemathematicalmusing... When they selectively choose not to publish certain results in favor of other ones etc... So on and so forth etc...

  12. misconduct is the wrong way to look at this by Goldsmith · · Score: 3, Informative

    This isn't a matter of misconduct, that's the wrong way to look at the current failure of science to... do science. (I am a scientist.)

    Other metrics are more useful. My favorite is "research efficiency." This is a decidedly commercial metric, it's the amount of revenue or economic activity (in dollars) generated by $1 of scientific research investment. It's been going down since about 1980. Surprisingly, research areas pitched as "basic research" (i.e. math, astronomy) tend to do well with this metric. It's the research that's sold to the public as industrially focused (i.e. my field, nanotechnology) that tends to do the worst.

    Another useful metric is the % of science PhDs who stay in science for at least 10 years after getting their degree. This measures how effective we are at training our scientific workforce. That's down significantly over the last 30 years as well. What we teach people now is not what they need to succeed in science after training (which is getting longer and longer).

    The metric most scientists are looking for is reproducibility, or the percentage of papers which can be repeated by simply following the instructions in the paper. Papers have grown in length and complexity in the last 40 years. It's pretty hard to argue that reproducibility has actually gone down because older papers simply don't include details we now expect. Of course, this is very hard to measure in any case. That's the thesis of TFA. It doesn't change the very real feeling (and data) that science is somehow not delivering on our investment.

    Misconduct is... you're going to have some when there are people involved. You're also going to have mistakes and papers which are disproven very quickly. I have a personal pet peeve for papers that promise extraordinarily cheap hardware by assuming labor is free, manufacturing can be done at large scale without investment in tooling, and working capital is free. Things like this are not actually misconduct, no matter how misleading they are.