Slashdot Mirror
Researchers Unable To Replicate Findings of Published Economics Studies (businessinsider.com)
An anonymous reader writes: Federal Reserve economists Andrew Chang and Phillip Li looked at 67 papers in 13 reputable academic journals. Their findings were shocking. Without the help of the authors, only a third of the results could be independently replicated. Even with the author's help, only about half, or 49%, could. Business Insider reports: "It's a pretty massive issue for economics, especially given the impact that the subject has on public policy. Li and Chang use a well-known paper by Carmen Reinhart and Ken Rogoff as an example. The study showed a significant growth drop-off once a country's national debts reached 90% of gross domestic product, but three years after being published the study was found to contain a significant Microsoft Excel error that changed the magnitude of the effect." With cancer studies and most recently psychology studies all having replication trouble, these economics papers have some company.
That sounds about right.
Economics has always been one of the least predictive of "sciences". Economists with an ideological bent make things up with no relationship to the real world and people believe them.
Isn't errors in the data analysis exactly the sort of issue the peer-review system is supposed to catch?
I did a brief course in cognitive psychology during my masters. The course was given by a fairly well known name in the field - an editor of one of the standard texts.
He specifically told us that we were to do 'anything we liked' to get our data to say what we wanted. He told us that it was vastly more important to publish and defend than not and get sacked. Very much a "it's easier to ask forgiveness than it is to get permission" sort of atmosphere.
It wouldn't surprise me that this sort of attitude is rampant in other areas of 'science'.
Only the hard sciences seem to have any real legitimacy and even then I wouldn't trust a biologist all that much.
The level of trust I'd give to any statement by someone working in a given area is directly proportional to the area's 'purity': http://xkcd.com/435/
This paper just finds that in many cases when journals require that replication files be posted, they aren't. Of the half of studies that aren't "replicated", the majority are due to the fact that replication files simply aren't available. It's not like these people read the papers, got the data, and reran the analysis on their own. All this paper is saying is that posted replication files often either don't exist or don't work. Their work doesn't show that the results can't be replicated, just that they can't be replicated from public code.
Issues like this were already being flagged in 2013:
http://www.nytimes.com/2013/04/19/opinion/krugman-the-excel-depression.html
http://www.washingtonpost.com/news/wonkblog/wp/2013/04/16/is-the-best-evidence-for-austerity-based-on-an-excel-spreadsheet-error/
First of all, shame on authors for either not checking their models enough, not asking others to check them, and not opening their models for others to see before publishing "important" results.
Secondly, and perhaps more importantly, shame on the rest of us (and especially policymakers) for relying on such kinds of work so quickly and without validation to support generally political agendas. It's almost the equivalent of funding vaccine-skeptic studies by choosing which doctors will speak in your favor without regard to a rigorous scientific review process.
I majored in Economics, an area of study with a huge problem with relying on mathematical formula without worrying about how accurately the formula reproduce reality. I'm honestly surprised even half are reproducible. The honest opinion of many professors therein is "pfft, the numbers agree with themselves perfectly! That's all that matters."
The solution is pretty simple. Every author must reveal the codes that were used to produce the results.
However, one interesting issue is that once every author reveals the codes, you could find out that half of them code in MATLAB with a proficiency comparable to a 3rd grader who just learned BASIC programming up to about the "goto" statement. Not only there is lots of spaghetti code, but it may also contain serious errors that may filter through into the papers. Hence, I suspect a lot of people will not be happy to reveal their codes.
If you think all economics is Republican, it's evidence that you haven't been paying attention. Of course, the rest of economics is no more reliable than the part you've noticed, but there *are* other schools of economics. There are even Marxian economists.
I think we've pushed this "anyone can grow up to be president" thing too far.
"The only danger of deficit spending is the risk of inflation; that managed, there is absolutely no issue."
Right. "Managed"... This is up there with fairies and unicorns in that it will never happen in the real world.
I'm ready for a good laugh. Demonstrate how "supply and demand" don't affect prices.
Contribute to civilization: ari.aynrand.org/donate
"So, the fact that Western economies make life longer...
Get your head out of your ass."
Exactly the kind of thing being talked about here: on these kind of topics people let them go by their gut feelings, even when hard data can be easily found.
Life expectancy at birth, years 1985-1995*1:
USA | Cuba
1985 74.56 | 76.34
1986 74.61 | 76.43
1987 74.77 | 76.34
1988 74.77 | 76.49
1989 75.02 | 76.53
1990 75.21 | 76.53
1991 75.37 | 76.59
1992 75.64 | 76.65
1993 75.42 | 76.65
1994 75.57 | 76.65
1995 75.62 | 76.65
So no, "western economies" doesn't make life longer and, in fact, USA has always done merely so-so in this regard: look at the OECD tables and you'll see it belongs to the awkward squad, and the OECD doesn't even publish data about USA's infant mortality rates, probably because they are outright embarrasing for a first world country*3.
*1http://www.indexmundi.com/
*2 https://data.oecd.org/healthst...
*3 https://www.cia.gov/library/pu...
"...With cancer studies and most recently psychology studies all having replication trouble, these economics papers have some company."
Unless we're actually going to institute some sort of reform when it comes to peer review and documented result validation, the only "company" these papers will be in is an endless sea of ignorance that assumes it won't keep happening over and over again.
There is far too much money to be made in simply publishing papers (a.k.a. bullshit) to actually validate results. Therefore, a whole new breed of fucking liars (sorry, no other words suffice) has been manipulating policy for quite a long time now.
Now society, the onus is on you. What are you going to do about this issue? Sit around and wait for it to happen again? That's what you did the last dozen times.
I am an econometrician (well sort of), which is probably worse, but at least we know that. But economics, independent of any data set availability or actual method problems, is broadly handicapped by the generally unobservable nature of the actual data that would enable the verification (or refutation) of a hypothesis. That is, much of the data is quite noisy with many variables mixed in with each other, and as such a big part of the work is trying to determine the extent to which the data itself is a useful measure of the thing being tested. Sometimes getting to a useful dataset is dependent on some awkward assumptions. As such, one of the biggest faults of Economic Theory is assuming a can opener (https://en.wikipedia.org/wiki/Assume_a_can_opener).
"The first thing to do when you find yourself in a hole is stop digging."
For most studies of any kind, the margin for error is around +/-3%. For example, a study covering the United States population using a sample size of 1000 will yield a margin of error of 3.1%.
So a study says that texting while driving increases your risk of a fatal crash by 23 times. That sounds like a lot! But hold the phone...the overall rate of traffic fatalities is about 10 per 100,000 people, or about 0.01%. Multiply that by 23, and you get 0.23%. A big change, right? But that 0.23% is still well within the margin for error of most any study.
I'm not saying that texting while driving isn't dangerous. I'm just saying it's a lot harder to prove a link than it would seem.
Still, people are wowed by big multipliers, and news writers love to tout dramatic statistics, whether the subject of the study is economics, medicine, or traffic safety. But if you understand statistics, you know that most of these studies don't really tell us much. It's no wonder we keep getting contradictory study results!
This is not unique to economics. Most scientific fields have problems with replication. Journals are strongly biased toward publishing positive results, and nobody gets tenure for negative results or replication.
Economics is not a scientific field and the fields which seems to have the most problems with this seem to be medical, not scientific ones and "nobody gets tenure for negative results" is simply not true because I did! Indeed it is common in particle physics where we search for evidence of new physics beyond the Standard Model and, with only one exception so far, keep coming up empty handed. As for the most recent Nobel for a "failed" experiment try the one of two days ago: this was awarded to two experiments which failed to show that the Standard Model description of neutrinos was correct.
I think your definition of "failed experiment" needs almost completely reversing. Michelson-Morley was a stunning success: it completely destroyed the luminiferous aether model for light. It was not the result that was expected but that does not make it a failure. The same applies to neutrino oscillations. Not getting a result you expect from an experiment is the thing every scientist hopes for it because means that you have learnt something new about the universe which is why these experiments often win Nobel prizes. If anything is a failed experiment it is those that just end up confirming existing theories because you were hoping you might learn something new and instead just ended up confirming what you already knew.