Researchers Unable To Replicate Findings of Published Economics Studies (businessinsider.com)

An anonymous reader writes: Federal Reserve economists Andrew Chang and Phillip Li looked at 67 papers in 13 reputable academic journals. Their findings were shocking. Without the help of the authors, only a third of the results could be independently replicated. Even with the author's help, only about half, or 49%, could. Business Insider reports: "It's a pretty massive issue for economics, especially given the impact that the subject has on public policy. Li and Chang use a well-known paper by Carmen Reinhart and Ken Rogoff as an example. The study showed a significant growth drop-off once a country's national debts reached 90% of gross domestic product, but three years after being published the study was found to contain a significant Microsoft Excel error that changed the magnitude of the effect." With cancer studies and most recently psychology studies all having replication trouble, these economics papers have some company.

Comments

These aren't failures to replicate

[Anonymous Coward]

This paper just finds that in many cases when journals require that replication files be posted, they aren't. Of the half of studies that aren't "replicated", the majority are due to the fact that replication files simply aren't available. It's not like these people read the papers, got the data, and reran the analysis on their own. All this paper is saying is that posted replication files often either don't exist or don't work. Their work doesn't show that the results can't be replicated, just that they can't be replicated from public code.

You have failure backwards

[Roger W Moore]

This is not unique to economics. Most scientific fields have problems with replication. Journals are strongly biased toward publishing positive results, and nobody gets tenure for negative results or replication.

Economics is not a scientific field and the fields which seems to have the most problems with this seem to be medical, not scientific ones and "nobody gets tenure for negative results" is simply not true because I did! Indeed it is common in particle physics where we search for evidence of new physics beyond the Standard Model and, with only one exception so far, keep coming up empty handed. As for the most recent Nobel for a "failed" experiment try the one of two days ago: this was awarded to two experiments which failed to show that the Standard Model description of neutrinos was correct.

I think your definition of "failed experiment" needs almost completely reversing. Michelson-Morley was a stunning success: it completely destroyed the luminiferous aether model for light. It was not the result that was expected but that does not make it a failure. The same applies to neutrino oscillations. Not getting a result you expect from an experiment is the thing every scientist hopes for it because means that you have learnt something new about the universe which is why these experiments often win Nobel prizes. If anything is a failed experiment it is those that just end up confirming existing theories because you were hoping you might learn something new and instead just ended up confirming what you already knew.