Researchers Unable To Replicate Findings of Published Economics Studies (businessinsider.com)

An anonymous reader writes: Federal Reserve economists Andrew Chang and Phillip Li looked at 67 papers in 13 reputable academic journals. Their findings were shocking. Without the help of the authors, only a third of the results could be independently replicated. Even with the author's help, only about half, or 49%, could. Business Insider reports: "It's a pretty massive issue for economics, especially given the impact that the subject has on public policy. Li and Chang use a well-known paper by Carmen Reinhart and Ken Rogoff as an example. The study showed a significant growth drop-off once a country's national debts reached 90% of gross domestic product, but three years after being published the study was found to contain a significant Microsoft Excel error that changed the magnitude of the effect." With cancer studies and most recently psychology studies all having replication trouble, these economics papers have some company.

Comments

Why would anyone be shocked?

[riverat1]

Economics has always been one of the least predictive of "sciences". Economists with an ideological bent make things up with no relationship to the real world and people believe them.

Re:Why would anyone be shocked?

[Anonymous Coward]

As someone who has a degree in this goop. I can safely say most of it is bunk. When you show me how to measure 'happiness' and 'utility' (real variables in economics) will be the day I think you are onto something.

Most economic models work *very* well in a static system *if* you can perfectly measure *everything*. Oh and when I say everything I mean it. Take for example the recent 2008 fiasco. Very few saw it coming. Why? Because they could not measure what if say 20% defaulted.

Economics only works in hand wavy generalizations. Take for example min wage. Many cities raising it. Yet they have no idea what it will really do good or bad. From a pure econ view it is probably bad. But from a humanitarian view it is probably a good thing. But neither side can really back it up with hard facts what will happen. How do you measure a job that never existed? You dont.

Re:Why would anyone be shocked?

[AthanasiusKircher]

Economics is a "science" in the same way that a pickle is "candy".

The best that can be done is to make some generalized guesses based on hazy metrics and barely-understood past events.

While that may be true, that is not the problem with these studies.

TFS is misleading here in referencing the problems in other disciplines, because there the replication problems often had to do with other scientists running an empirical experiment, collecting new data, and seeing whether the same trend occurs.

In THIS study, the "replication" problems were solely due to insufficient documentation. The categories for reasons that studies could not be replicated were listed as: (1) missing public data or code, (2) incorrect public data or code, (3) missing software, or (4) proprietary data.

There were no empirical "experiments" re-tested here. All they did was try to replicate data analysis, and "replication failed" when they couldn't access the relevant datasets or analytical software.

This is still a significant problem in economics, but the failure in "scientific" methodology here was of a VERY different kind -- it just had to do with access to research tools, not new data that conflicted with previous findings.

Re:Why would anyone be shocked?

[chipschap]

I'm guessing that most countries wouldn't want to be the subject of a real-world economic experiment...

Except that most countries are the subject of such experiments ... it's just that they are called "policy."

Re:Why would anyone be shocked?

[AthanasiusKircher]

But at the same time the Austrian School (not that I subscribe to them) gets dismissed for not relying on studies and poorly supported models.

That's a bit of an understatement, don't ya think? It's not that they don't "rely on studies" and "models" -- they actually fundamentally deny the possibility of empirical study! To wit:

Mises stated that praxeology could be used to deduce a priori theoretical economic truths and that deductive economic thought experiments could yield conclusions which follow irrefutably from the underlying assumptions. He claimed conclusions could not be inferred from empirical observation or statistical analysis and argued against the use of probabilities in economic models.

I'll give you that the field of economics is a mess. But the "Austrian School" denies the fundamental existence of SCIENCE. They believe that one can just "make a priori assumptions" and do "thought experiments" and they will always get the right answer. But -- if they are honest about it -- they have no way of ever proving themselves right, because they deny the ability of using empirical data to support any argument.

That may be a good methodology for a theoretical system of pure logic or for a religion, but if you deny empiricism, then there's simply no way your system could have any contact with the real world!

Re: Why would anyone be shocked?

[silentcoder]

Hell not only do they lack evidence they even use their own custom definition of inflation as "increased money supply". The proper definition is "decreased buying power". Those things do not always correlate. A lot of inflation happens in static currencies as well for example. By their definition it is true that austerity prevents inflation but also utterly meaningless since thats just what they defined the word to mean. However they want you to assume the connotations of the proper definition still apply (it does not). Increased money supply does not have to equal decreased buying power. Especially if it is offset by taxes (which they ignore).
So the claim is nothing but a ruse intended to drive a political ideology. It has absolutely no bearing on useful economic analysis.

As an aside ignoring taxes is stupid because austerity reduces tax revenue and provably it always does so by several orders of magnitude more than it saves in expenses. Austerity can, as baseline mathematics, never ever achieve anythiny except to make the deficit much larger much faster.

These aren't failures to replicate

[Anonymous Coward]

This paper just finds that in many cases when journals require that replication files be posted, they aren't. Of the half of studies that aren't "replicated", the majority are due to the fact that replication files simply aren't available. It's not like these people read the papers, got the data, and reran the analysis on their own. All this paper is saying is that posted replication files often either don't exist or don't work. Their work doesn't show that the results can't be replicated, just that they can't be replicated from public code.

Re:Are we blaming Microsoft for this?

[ShanghaiBill]

in situations such as "Economics Research" that there are all sorts of incentives to cook the book

This is not unique to economics. Most scientific fields have problems with replication. Journals are strongly biased toward publishing positive results, and nobody gets tenure for negative results or replication. I believe the last Nobel Prize for a failed experiment was Albert Michelson in 1907. There are strong incentives to cheat, or at least cut corners.

You have failure backwards

[Roger W Moore]

This is not unique to economics. Most scientific fields have problems with replication. Journals are strongly biased toward publishing positive results, and nobody gets tenure for negative results or replication.

Economics is not a scientific field and the fields which seems to have the most problems with this seem to be medical, not scientific ones and "nobody gets tenure for negative results" is simply not true because I did! Indeed it is common in particle physics where we search for evidence of new physics beyond the Standard Model and, with only one exception so far, keep coming up empty handed. As for the most recent Nobel for a "failed" experiment try the one of two days ago: this was awarded to two experiments which failed to show that the Standard Model description of neutrinos was correct.

I think your definition of "failed experiment" needs almost completely reversing. Michelson-Morley was a stunning success: it completely destroyed the luminiferous aether model for light. It was not the result that was expected but that does not make it a failure. The same applies to neutrino oscillations. Not getting a result you expect from an experiment is the thing every scientist hopes for it because means that you have learnt something new about the universe which is why these experiments often win Nobel prizes. If anything is a failed experiment it is those that just end up confirming existing theories because you were hoping you might learn something new and instead just ended up confirming what you already knew.

Re:You have failure backwards

[The Real Dr John]

Couldn't agree more Roger. Economics is not a field of scientific inquiry. It is far more prone to fudging than biochemistry, or my field, neuroscience. Any scientist can manipulate or cherry pick the data, but in the end it will only hurt their reputation and so most experienced researchers go to great lengths to make sure their experimental results are reproducible. I am notorious in the lab for insisting on more repetition of experiments that have worked well a number of times. Nobody likes doing repetitive and tedious experiments but it is part of the job.

I do however agree that fudging in science may be increasing, but that seems more due to the pressures of scarce funding. If science were funded as well as the military is, there would be no temptation for fudging on anyone's part.