Slashdot Mirror

← Back to Stories (view on slashdot.org)

Researchers Unable To Replicate Findings of Published Economics Studies (businessinsider.com)

Posted by samzenpus on from the but-can-you-do-it-twice? dept.

An anonymous reader writes: Federal Reserve economists Andrew Chang and Phillip Li looked at 67 papers in 13 reputable academic journals. Their findings were shocking. Without the help of the authors, only a third of the results could be independently replicated. Even with the author's help, only about half, or 49%, could. Business Insider reports: "It's a pretty massive issue for economics, especially given the impact that the subject has on public policy. Li and Chang use a well-known paper by Carmen Reinhart and Ken Rogoff as an example. The study showed a significant growth drop-off once a country's national debts reached 90% of gross domestic product, but three years after being published the study was found to contain a significant Microsoft Excel error that changed the magnitude of the effect." With cancer studies and most recently psychology studies all having replication trouble, these economics papers have some company.

18 of 213 comments (clear)

  1. Why would anyone be shocked? by riverat1 · · Score: 5, Insightful

    Economics has always been one of the least predictive of "sciences". Economists with an ideological bent make things up with no relationship to the real world and people believe them.

    1. Re:Why would anyone be shocked? by Chirs · · Score: 4, Insightful

      How hard could it be to design real-world economic experiments that actually yield useful and reproducible results?

      I'm guessing that most countries wouldn't want to be the subject of a real-world economic experiment...

    2. Re:Why would anyone be shocked? by Anonymous Coward · · Score: 5, Insightful

      As someone who has a degree in this goop. I can safely say most of it is bunk. When you show me how to measure 'happiness' and 'utility' (real variables in economics) will be the day I think you are onto something.

      Most economic models work *very* well in a static system *if* you can perfectly measure *everything*. Oh and when I say everything I mean it. Take for example the recent 2008 fiasco. Very few saw it coming. Why? Because they could not measure what if say 20% defaulted.

      Economics only works in hand wavy generalizations. Take for example min wage. Many cities raising it. Yet they have no idea what it will really do good or bad. From a pure econ view it is probably bad. But from a humanitarian view it is probably a good thing. But neither side can really back it up with hard facts what will happen. How do you measure a job that never existed? You dont.

    3. Re:Why would anyone be shocked? by guacamole · · Score: 4, Interesting

      Most of academic economists are non-partisan. Or if they are, they normally have pretty solid theory and numbers to back their claims. The economists who do give the study of economics a bad reputation are those who actually make very opinionated and authoritative statements and reports, but hardly publish anything in a peer reviewed academic journal, where such antics wouldn't pass. Unfortunately, there are a few well known economists who had built a solid reputation in the research circles years ago, and then moved on past producing publishable academic research into the realm of partisanship. Any economist knows who they are. There are some well known liberals and conservatives among those.

    4. Re:Why would anyone be shocked? by AthanasiusKircher · · Score: 5, Insightful

      Economics is a "science" in the same way that a pickle is "candy".

      The best that can be done is to make some generalized guesses based on hazy metrics and barely-understood past events.

      While that may be true, that is not the problem with these studies.

      TFS is misleading here in referencing the problems in other disciplines, because there the replication problems often had to do with other scientists running an empirical experiment, collecting new data, and seeing whether the same trend occurs.

      In THIS study, the "replication" problems were solely due to insufficient documentation. The categories for reasons that studies could not be replicated were listed as: (1) missing public data or code, (2) incorrect public data or code, (3) missing software, or (4) proprietary data.

      There were no empirical "experiments" re-tested here. All they did was try to replicate data analysis, and "replication failed" when they couldn't access the relevant datasets or analytical software.

      This is still a significant problem in economics, but the failure in "scientific" methodology here was of a VERY different kind -- it just had to do with access to research tools, not new data that conflicted with previous findings.

    5. Re:Why would anyone be shocked? by chipschap · · Score: 5, Insightful

      I'm guessing that most countries wouldn't want to be the subject of a real-world economic experiment...

      Except that most countries are the subject of such experiments ... it's just that they are called "policy."

    6. Re:Why would anyone be shocked? by AthanasiusKircher · · Score: 5, Insightful

      But at the same time the Austrian School (not that I subscribe to them) gets dismissed for not relying on studies and poorly supported models.

      That's a bit of an understatement, don't ya think? It's not that they don't "rely on studies" and "models" -- they actually fundamentally deny the possibility of empirical study! To wit:

      Mises stated that praxeology could be used to deduce a priori theoretical economic truths and that deductive economic thought experiments could yield conclusions which follow irrefutably from the underlying assumptions. He claimed conclusions could not be inferred from empirical observation or statistical analysis and argued against the use of probabilities in economic models.

      I'll give you that the field of economics is a mess. But the "Austrian School" denies the fundamental existence of SCIENCE. They believe that one can just "make a priori assumptions" and do "thought experiments" and they will always get the right answer. But -- if they are honest about it -- they have no way of ever proving themselves right, because they deny the ability of using empirical data to support any argument.

      That may be a good methodology for a theoretical system of pure logic or for a religion, but if you deny empiricism, then there's simply no way your system could have any contact with the real world!

    7. Re:Why would anyone be shocked? by riverat1 · · Score: 4, Insightful

      As opposed to the likes of the Chicago and Austrian school who yell "Austerity!" and "Inflation!" with no evidence to back them up?

    8. Re:Why would anyone be shocked? by AthanasiusKircher · · Score: 4, Interesting

      I just realized my post was slightly unclear -- the vast majority of "unsuccessful replication" had to do with problems with public access to data or data analysis methods.

      But there were also about 1/4 of those studies which "failed replication" which failed due to errors in the data or the analysis of that data. In any case, no new data was collected here -- but "failure" here was mostly about lack of access.

      (Of course, whether full access would have allowed successful replication is another question. Regardless, they weren't actually collecting new empirical data and "running experiments" again in the usual scientific sense.)

    9. Re: Why would anyone be shocked? by silentcoder · · Score: 5, Interesting

      Hell not only do they lack evidence they even use their own custom definition of inflation as "increased money supply". The proper definition is "decreased buying power". Those things do not always correlate. A lot of inflation happens in static currencies as well for example. By their definition it is true that austerity prevents inflation but also utterly meaningless since thats just what they defined the word to mean. However they want you to assume the connotations of the proper definition still apply (it does not). Increased money supply does not have to equal decreased buying power. Especially if it is offset by taxes (which they ignore).
      So the claim is nothing but a ruse intended to drive a political ideology. It has absolutely no bearing on useful economic analysis.

      As an aside ignoring taxes is stupid because austerity reduces tax revenue and provably it always does so by several orders of magnitude more than it saves in expenses. Austerity can, as baseline mathematics, never ever achieve anythiny except to make the deficit much larger much faster.

      --
      Unicode killed the ASCII-art *
  2. Lies, damned lies, and statistics by Anonymous Coward · · Score: 4, Interesting

    I did a brief course in cognitive psychology during my masters. The course was given by a fairly well known name in the field - an editor of one of the standard texts.

    He specifically told us that we were to do 'anything we liked' to get our data to say what we wanted. He told us that it was vastly more important to publish and defend than not and get sacked. Very much a "it's easier to ask forgiveness than it is to get permission" sort of atmosphere.

    It wouldn't surprise me that this sort of attitude is rampant in other areas of 'science'.

    Only the hard sciences seem to have any real legitimacy and even then I wouldn't trust a biologist all that much.

    The level of trust I'd give to any statement by someone working in a given area is directly proportional to the area's 'purity': http://xkcd.com/435/

    1. Re:Lies, damned lies, and statistics by pz · · Score: 4, Insightful

      Only the hard sciences seem to have any real legitimacy and even then I wouldn't trust a biologist all that much.

      I got started in life as an Engineer (3rd or 4th generation, as far as I can tell), and became a Biologist. One of the first things that shocked me is the notion of noise. In Electrical Engineering, noise is well-managed and understood. When you say you have a good fit to your data, it means errors of less than 1%. In Biology, there's so much noise and inherent variability that when you say you have a good fit, it means errors of less than 50%.

      There are very few biological processes we understand well enough to say that we really, deeply understand them. Unlike, say, a transistor.

      --

      Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
  3. These aren't failures to replicate by Anonymous Coward · · Score: 5, Informative

    This paper just finds that in many cases when journals require that replication files be posted, they aren't. Of the half of studies that aren't "replicated", the majority are due to the fact that replication files simply aren't available. It's not like these people read the papers, got the data, and reran the analysis on their own. All this paper is saying is that posted replication files often either don't exist or don't work. Their work doesn't show that the results can't be replicated, just that they can't be replicated from public code.

  4. old warning that should be repeated over and over by supernova87a · · Score: 4, Interesting

    Issues like this were already being flagged in 2013:

    http://www.nytimes.com/2013/04/19/opinion/krugman-the-excel-depression.html
    http://www.washingtonpost.com/news/wonkblog/wp/2013/04/16/is-the-best-evidence-for-austerity-based-on-an-excel-spreadsheet-error/

    First of all, shame on authors for either not checking their models enough, not asking others to check them, and not opening their models for others to see before publishing "important" results.

    Secondly, and perhaps more importantly, shame on the rest of us (and especially policymakers) for relying on such kinds of work so quickly and without validation to support generally political agendas. It's almost the equivalent of funding vaccine-skeptic studies by choosing which doctors will speak in your favor without regard to a rigorous scientific review process.

  5. Re:Wait a minute. by CanadianMacFan · · Score: 4, Interesting

    Yes but the peer review system is flawed. Take someone that is pretty high up in their field and they have their research, people to manage, and they are probably at a university so some classes too. Add in all of their day to day home life stuff that needs to be done. And now they are asked to review a paper from someone else for free. Even if they want to do a great job on it they are on a tight deadline from the publisher. So how thorough of a job do you think that they can do? Can they try to replicate the experiment? Look for an error in an Excel spreadsheet?

    It's like what a lot of software shops I've seen do with their testing. The development team races to finish it and gets the application done the day before release and hands it off to QA. Then everyone wonders why the program is full of errors in production.

    If you want a better peer review system you are going to have to give the reviewers more time and pay them for it.

  6. Re:Are we blaming Microsoft for this? by ShanghaiBill · · Score: 5, Insightful

    in situations such as "Economics Research" that there are all sorts of incentives to cook the book

    This is not unique to economics. Most scientific fields have problems with replication. Journals are strongly biased toward publishing positive results, and nobody gets tenure for negative results or replication. I believe the last Nobel Prize for a failed experiment was Albert Michelson in 1907. There are strong incentives to cheat, or at least cut corners.

  7. You have failure backwards by Roger+W+Moore · · Score: 5, Informative

    This is not unique to economics. Most scientific fields have problems with replication. Journals are strongly biased toward publishing positive results, and nobody gets tenure for negative results or replication.

    Economics is not a scientific field and the fields which seems to have the most problems with this seem to be medical, not scientific ones and "nobody gets tenure for negative results" is simply not true because I did! Indeed it is common in particle physics where we search for evidence of new physics beyond the Standard Model and, with only one exception so far, keep coming up empty handed. As for the most recent Nobel for a "failed" experiment try the one of two days ago: this was awarded to two experiments which failed to show that the Standard Model description of neutrinos was correct.

    I think your definition of "failed experiment" needs almost completely reversing. Michelson-Morley was a stunning success: it completely destroyed the luminiferous aether model for light. It was not the result that was expected but that does not make it a failure. The same applies to neutrino oscillations. Not getting a result you expect from an experiment is the thing every scientist hopes for it because means that you have learnt something new about the universe which is why these experiments often win Nobel prizes. If anything is a failed experiment it is those that just end up confirming existing theories because you were hoping you might learn something new and instead just ended up confirming what you already knew.

    1. Re:You have failure backwards by The+Real+Dr+John · · Score: 5, Interesting

      Couldn't agree more Roger. Economics is not a field of scientific inquiry. It is far more prone to fudging than biochemistry, or my field, neuroscience. Any scientist can manipulate or cherry pick the data, but in the end it will only hurt their reputation and so most experienced researchers go to great lengths to make sure their experimental results are reproducible. I am notorious in the lab for insisting on more repetition of experiments that have worked well a number of times. Nobody likes doing repetitive and tedious experiments but it is part of the job.

      I do however agree that fudging in science may be increasing, but that seems more due to the pressures of scarce funding. If science were funded as well as the military is, there would be no temptation for fudging on anyone's part.

      --
      A brain is a terrible thing to waste... Mind? That's debatable.