Researchers Unable To Replicate Findings of Published Economics Studies (businessinsider.com)

An anonymous reader writes: Federal Reserve economists Andrew Chang and Phillip Li looked at 67 papers in 13 reputable academic journals. Their findings were shocking. Without the help of the authors, only a third of the results could be independently replicated. Even with the author's help, only about half, or 49%, could. Business Insider reports: "It's a pretty massive issue for economics, especially given the impact that the subject has on public policy. Li and Chang use a well-known paper by Carmen Reinhart and Ken Rogoff as an example. The study showed a significant growth drop-off once a country's national debts reached 90% of gross domestic product, but three years after being published the study was found to contain a significant Microsoft Excel error that changed the magnitude of the effect." With cancer studies and most recently psychology studies all having replication trouble, these economics papers have some company.

Comments

Why would anyone be shocked?

[riverat1]

Economics has always been one of the least predictive of "sciences". Economists with an ideological bent make things up with no relationship to the real world and people believe them.

Re:Why would anyone be shocked?

[JustAnotherOldGuy]

Economics is a "science" in the same way that a pickle is "candy".

The best that can be done is to make some generalized guesses based on hazy metrics and barely-understood past events.

Re:Why would anyone be shocked?

[Chirs]

How hard could it be to design real-world economic experiments that actually yield useful and reproducible results?

I'm guessing that most countries wouldn't want to be the subject of a real-world economic experiment...

Re:Why would anyone be shocked?

[Anonymous Coward]

As someone who has a degree in this goop. I can safely say most of it is bunk. When you show me how to measure 'happiness' and 'utility' (real variables in economics) will be the day I think you are onto something.

Most economic models work *very* well in a static system *if* you can perfectly measure *everything*. Oh and when I say everything I mean it. Take for example the recent 2008 fiasco. Very few saw it coming. Why? Because they could not measure what if say 20% defaulted.

Economics only works in hand wavy generalizations. Take for example min wage. Many cities raising it. Yet they have no idea what it will really do good or bad. From a pure econ view it is probably bad. But from a humanitarian view it is probably a good thing. But neither side can really back it up with hard facts what will happen. How do you measure a job that never existed? You dont.

Re:Why would anyone be shocked?

[AthanasiusKircher]

Economics is a "science" in the same way that a pickle is "candy".

The best that can be done is to make some generalized guesses based on hazy metrics and barely-understood past events.

While that may be true, that is not the problem with these studies.

TFS is misleading here in referencing the problems in other disciplines, because there the replication problems often had to do with other scientists running an empirical experiment, collecting new data, and seeing whether the same trend occurs.

In THIS study, the "replication" problems were solely due to insufficient documentation. The categories for reasons that studies could not be replicated were listed as: (1) missing public data or code, (2) incorrect public data or code, (3) missing software, or (4) proprietary data.

There were no empirical "experiments" re-tested here. All they did was try to replicate data analysis, and "replication failed" when they couldn't access the relevant datasets or analytical software.

This is still a significant problem in economics, but the failure in "scientific" methodology here was of a VERY different kind -- it just had to do with access to research tools, not new data that conflicted with previous findings.

Re:Why would anyone be shocked?

[chipschap]

I'm guessing that most countries wouldn't want to be the subject of a real-world economic experiment...

Except that most countries are the subject of such experiments ... it's just that they are called "policy."

Re:Why would anyone be shocked?

[AthanasiusKircher]

But at the same time the Austrian School (not that I subscribe to them) gets dismissed for not relying on studies and poorly supported models.

That's a bit of an understatement, don't ya think? It's not that they don't "rely on studies" and "models" -- they actually fundamentally deny the possibility of empirical study! To wit:

Mises stated that praxeology could be used to deduce a priori theoretical economic truths and that deductive economic thought experiments could yield conclusions which follow irrefutably from the underlying assumptions. He claimed conclusions could not be inferred from empirical observation or statistical analysis and argued against the use of probabilities in economic models.

I'll give you that the field of economics is a mess. But the "Austrian School" denies the fundamental existence of SCIENCE. They believe that one can just "make a priori assumptions" and do "thought experiments" and they will always get the right answer. But -- if they are honest about it -- they have no way of ever proving themselves right, because they deny the ability of using empirical data to support any argument.

That may be a good methodology for a theoretical system of pure logic or for a religion, but if you deny empiricism, then there's simply no way your system could have any contact with the real world!

Re:Why would anyone be shocked?

[riverat1]

As opposed to the likes of the Chicago and Austrian school who yell "Austerity!" and "Inflation!" with no evidence to back them up?

Well, duh

[Anonymous Coward]

Economics is just a religion, it's a human invention. It's a way of saying "We have the highest technology, energy resources and best materials in the history of humanity, but we will continue to enforce millenia-old ideas of "work"."

Hey, as long as there's numbers and formulas and lots of PhDs, it's gotta be true, right?

Re:Are we blaming Microsoft for this?

[ShanghaiBill]

in situations such as "Economics Research" that there are all sorts of incentives to cook the book

This is not unique to economics. Most scientific fields have problems with replication. Journals are strongly biased toward publishing positive results, and nobody gets tenure for negative results or replication. I believe the last Nobel Prize for a failed experiment was Albert Michelson in 1907. There are strong incentives to cheat, or at least cut corners.

Re:old warning that should be repeated over and ov

[Anonymous Coward]

"The only danger of deficit spending is the risk of inflation; that managed, there is absolutely no issue."

Right. "Managed"... This is up there with fairies and unicorns in that it will never happen in the real world.

Re:Lies, damned lies, and statistics

[pz]

Only the hard sciences seem to have any real legitimacy and even then I wouldn't trust a biologist all that much.

I got started in life as an Engineer (3rd or 4th generation, as far as I can tell), and became a Biologist. One of the first things that shocked me is the notion of noise. In Electrical Engineering, noise is well-managed and understood. When you say you have a good fit to your data, it means errors of less than 1%. In Biology, there's so much noise and inherent variability that when you say you have a good fit, it means errors of less than 50%.

There are very few biological processes we understand well enough to say that we really, deeply understand them. Unlike, say, a transistor.