Slashdot Mirror
Researchers Unable To Replicate Findings of Published Economics Studies (businessinsider.com)
An anonymous reader writes: Federal Reserve economists Andrew Chang and Phillip Li looked at 67 papers in 13 reputable academic journals. Their findings were shocking. Without the help of the authors, only a third of the results could be independently replicated. Even with the author's help, only about half, or 49%, could. Business Insider reports: "It's a pretty massive issue for economics, especially given the impact that the subject has on public policy. Li and Chang use a well-known paper by Carmen Reinhart and Ken Rogoff as an example. The study showed a significant growth drop-off once a country's national debts reached 90% of gross domestic product, but three years after being published the study was found to contain a significant Microsoft Excel error that changed the magnitude of the effect." With cancer studies and most recently psychology studies all having replication trouble, these economics papers have some company.
I did a brief course in cognitive psychology during my masters. The course was given by a fairly well known name in the field - an editor of one of the standard texts.
He specifically told us that we were to do 'anything we liked' to get our data to say what we wanted. He told us that it was vastly more important to publish and defend than not and get sacked. Very much a "it's easier to ask forgiveness than it is to get permission" sort of atmosphere.
It wouldn't surprise me that this sort of attitude is rampant in other areas of 'science'.
Only the hard sciences seem to have any real legitimacy and even then I wouldn't trust a biologist all that much.
The level of trust I'd give to any statement by someone working in a given area is directly proportional to the area's 'purity': http://xkcd.com/435/
Issues like this were already being flagged in 2013:
http://www.nytimes.com/2013/04/19/opinion/krugman-the-excel-depression.html
http://www.washingtonpost.com/news/wonkblog/wp/2013/04/16/is-the-best-evidence-for-austerity-based-on-an-excel-spreadsheet-error/
First of all, shame on authors for either not checking their models enough, not asking others to check them, and not opening their models for others to see before publishing "important" results.
Secondly, and perhaps more importantly, shame on the rest of us (and especially policymakers) for relying on such kinds of work so quickly and without validation to support generally political agendas. It's almost the equivalent of funding vaccine-skeptic studies by choosing which doctors will speak in your favor without regard to a rigorous scientific review process.
Econ is a heckalot more predictive than most sciences!
If it has "science" as part of the name of the field, it isn't science. If it was, it wouldn't need it.
Recognition that 2500 year old methods work is not a bad thing. The difference is that there is documented empirical evidence of the positive effect gained meditation using the scientific method. More, the underlying mechanisms of meditation are beginning to be understood in terms of neural and physiological systems. I'm not actually sure what you have against this research, but I doubt you would find one serious researcher that will describe meditation as a panacea.
Good one. But I always assume in situations such as "Economics Research" that there are all sorts of incentives to cook the books so to speak. Writing what policy makers want to hear will get you hired on as a White House advisor, where you can join the economic team that uses the flawed data to implement horrible economic plans of the type that the Republicans and DLC Democrats have been shoving down our throats since Reagan. So I am not entirely sure that those were just "honest mistakes".
A brain is a terrible thing to waste... Mind? That's debatable.
I majored in Economics, an area of study with a huge problem with relying on mathematical formula without worrying about how accurately the formula reproduce reality. I'm honestly surprised even half are reproducible. The honest opinion of many professors therein is "pfft, the numbers agree with themselves perfectly! That's all that matters."
Yes but the peer review system is flawed. Take someone that is pretty high up in their field and they have their research, people to manage, and they are probably at a university so some classes too. Add in all of their day to day home life stuff that needs to be done. And now they are asked to review a paper from someone else for free. Even if they want to do a great job on it they are on a tight deadline from the publisher. So how thorough of a job do you think that they can do? Can they try to replicate the experiment? Look for an error in an Excel spreadsheet?
It's like what a lot of software shops I've seen do with their testing. The development team races to finish it and gets the application done the day before release and hands it off to QA. Then everyone wonders why the program is full of errors in production.
If you want a better peer review system you are going to have to give the reviewers more time and pay them for it.
The solution is pretty simple. Every author must reveal the codes that were used to produce the results.
However, one interesting issue is that once every author reveals the codes, you could find out that half of them code in MATLAB with a proficiency comparable to a 3rd grader who just learned BASIC programming up to about the "goto" statement. Not only there is lots of spaghetti code, but it may also contain serious errors that may filter through into the papers. Hence, I suspect a lot of people will not be happy to reveal their codes.
Most of academic economists are non-partisan. Or if they are, they normally have pretty solid theory and numbers to back their claims. The economists who do give the study of economics a bad reputation are those who actually make very opinionated and authoritative statements and reports, but hardly publish anything in a peer reviewed academic journal, where such antics wouldn't pass. Unfortunately, there are a few well known economists who had built a solid reputation in the research circles years ago, and then moved on past producing publishable academic research into the realm of partisanship. Any economist knows who they are. There are some well known liberals and conservatives among those.
I just realized my post was slightly unclear -- the vast majority of "unsuccessful replication" had to do with problems with public access to data or data analysis methods.
But there were also about 1/4 of those studies which "failed replication" which failed due to errors in the data or the analysis of that data. In any case, no new data was collected here -- but "failure" here was mostly about lack of access.
(Of course, whether full access would have allowed successful replication is another question. Regardless, they weren't actually collecting new empirical data and "running experiments" again in the usual scientific sense.)
Hell not only do they lack evidence they even use their own custom definition of inflation as "increased money supply". The proper definition is "decreased buying power". Those things do not always correlate. A lot of inflation happens in static currencies as well for example. By their definition it is true that austerity prevents inflation but also utterly meaningless since thats just what they defined the word to mean. However they want you to assume the connotations of the proper definition still apply (it does not). Increased money supply does not have to equal decreased buying power. Especially if it is offset by taxes (which they ignore).
So the claim is nothing but a ruse intended to drive a political ideology. It has absolutely no bearing on useful economic analysis.
As an aside ignoring taxes is stupid because austerity reduces tax revenue and provably it always does so by several orders of magnitude more than it saves in expenses. Austerity can, as baseline mathematics, never ever achieve anythiny except to make the deficit much larger much faster.
Unicode killed the ASCII-art *
Couldn't agree more Roger. Economics is not a field of scientific inquiry. It is far more prone to fudging than biochemistry, or my field, neuroscience. Any scientist can manipulate or cherry pick the data, but in the end it will only hurt their reputation and so most experienced researchers go to great lengths to make sure their experimental results are reproducible. I am notorious in the lab for insisting on more repetition of experiments that have worked well a number of times. Nobody likes doing repetitive and tedious experiments but it is part of the job.
I do however agree that fudging in science may be increasing, but that seems more due to the pressures of scarce funding. If science were funded as well as the military is, there would be no temptation for fudging on anyone's part.
A brain is a terrible thing to waste... Mind? That's debatable.