Slashdot Mirror
Researchers Unable To Replicate Findings of Published Economics Studies (businessinsider.com)
An anonymous reader writes: Federal Reserve economists Andrew Chang and Phillip Li looked at 67 papers in 13 reputable academic journals. Their findings were shocking. Without the help of the authors, only a third of the results could be independently replicated. Even with the author's help, only about half, or 49%, could. Business Insider reports: "It's a pretty massive issue for economics, especially given the impact that the subject has on public policy. Li and Chang use a well-known paper by Carmen Reinhart and Ken Rogoff as an example. The study showed a significant growth drop-off once a country's national debts reached 90% of gross domestic product, but three years after being published the study was found to contain a significant Microsoft Excel error that changed the magnitude of the effect." With cancer studies and most recently psychology studies all having replication trouble, these economics papers have some company.
That sounds about right.
Economics has always been one of the least predictive of "sciences". Economists with an ideological bent make things up with no relationship to the real world and people believe them.
"If all the economists were laid end to end, they'd never reach a conclusion." - George Bernard Shaw
They don't call them soft sciences for nothing. Although I believe economics is also known as the dismal science -- unsurprising with results like this.
Economists usually publish their data and code, unlike other sciences. This is a good thing and helps find problems, like were found in this case.
here's the link: http://www.nature.com/news/rep...
You took all the economists in the world, and laid them end to end, they'd point in different directions.
So rise up, all ye lost ones, as one, we'll claw the clouds.
Isn't errors in the data analysis exactly the sort of issue the peer-review system is supposed to catch?
I did a brief course in cognitive psychology during my masters. The course was given by a fairly well known name in the field - an editor of one of the standard texts.
He specifically told us that we were to do 'anything we liked' to get our data to say what we wanted. He told us that it was vastly more important to publish and defend than not and get sacked. Very much a "it's easier to ask forgiveness than it is to get permission" sort of atmosphere.
It wouldn't surprise me that this sort of attitude is rampant in other areas of 'science'.
Only the hard sciences seem to have any real legitimacy and even then I wouldn't trust a biologist all that much.
The level of trust I'd give to any statement by someone working in a given area is directly proportional to the area's 'purity': http://xkcd.com/435/
Their statistics on reproducibility--without their (the authors) help, can anyone reproduce their results?
...or the entire sweater of our existence will come unraveled.
Cloudiot: A person who does not see offsite storage as a way to lose control over access to his or her own data.
This paper just finds that in many cases when journals require that replication files be posted, they aren't. Of the half of studies that aren't "replicated", the majority are due to the fact that replication files simply aren't available. It's not like these people read the papers, got the data, and reran the analysis on their own. All this paper is saying is that posted replication files often either don't exist or don't work. Their work doesn't show that the results can't be replicated, just that they can't be replicated from public code.
Economics is just a religion, it's a human invention. It's a way of saying "We have the highest technology, energy resources and best materials in the history of humanity, but we will continue to enforce millenia-old ideas of "work"."
Hey, as long as there's numbers and formulas and lots of PhDs, it's gotta be true, right?
Issues like this were already being flagged in 2013:
http://www.nytimes.com/2013/04/19/opinion/krugman-the-excel-depression.html
http://www.washingtonpost.com/news/wonkblog/wp/2013/04/16/is-the-best-evidence-for-austerity-based-on-an-excel-spreadsheet-error/
First of all, shame on authors for either not checking their models enough, not asking others to check them, and not opening their models for others to see before publishing "important" results.
Secondly, and perhaps more importantly, shame on the rest of us (and especially policymakers) for relying on such kinds of work so quickly and without validation to support generally political agendas. It's almost the equivalent of funding vaccine-skeptic studies by choosing which doctors will speak in your favor without regard to a rigorous scientific review process.
I majored in Economics, an area of study with a huge problem with relying on mathematical formula without worrying about how accurately the formula reproduce reality. I'm honestly surprised even half are reproducible. The honest opinion of many professors therein is "pfft, the numbers agree with themselves perfectly! That's all that matters."
It is a religion where you are asked to believe so much hateful Republican garbage. Because of economists we have millions of children starving. They created scams like Adam Smith's racist hate book that demands poor children die. They must die according to them. Adam Smith's book is second only to the xian Bibble in the number of people it has killed. So much hate. So much.
Evidence for the benefit of austerity is so weak as to be non-existant. However the effects of austerity are very real and hurtful. The only danger of deficit spending is the risk of inflation; that managed, there is absolutely no issue.
Finally. I get all sorts of shit around here because of my opinion that Economics is considerably less rigorous than Parapsychology. It's the softest of all sciences, and yet the pronouncements of some academic pronouncements of a group of those jackoffs have given us "supply-side economics". They come up with "laws" like "supply and demand" and hang on to them despite the fact that they fly in the face of reality.
To make matters worse, economists' math is just awful.
I'm old enough to have actually taken a course from Milton Friedman. I was just a 17 year-old university freshman at the time, but even in my hormone-addled state I could tell he was full of shit, and he was better than most.
You are welcome on my lawn.
The solution is pretty simple. Every author must reveal the codes that were used to produce the results.
However, one interesting issue is that once every author reveals the codes, you could find out that half of them code in MATLAB with a proficiency comparable to a 3rd grader who just learned BASIC programming up to about the "goto" statement. Not only there is lots of spaghetti code, but it may also contain serious errors that may filter through into the papers. Hence, I suspect a lot of people will not be happy to reveal their codes.
Trying to predict future GDP is a non-starter for me. Even invoking the term GDP puts you outside the realm of economics as far as I am concerned. I think economics is poorly applied by using aggregated numbers. The more you aggregate, the less you understand.
Current economic theory could well substitute for some primitive magical cult. Hence such monstrosities such as the Black–Scholes model (a rehashed nuclear physics formula). The magic formula that caused the global economic meltdown. You might as well wave a dead chicken at your Quotron/Bloomberg terminal.
Liar's Poker by Michael Lewis
Austerity hurts people on government dole. Without the dole, they can't get mind numbing drugs; they have to work instead of going to "Occupy" rape camps and riots.
I suppose that can't be specified as a "risk", because it's more of a certainty.
Contribute to civilization: ari.aynrand.org/donate
"The only danger of deficit spending is the risk of inflation; that managed, there is absolutely no issue."
Right. "Managed"... This is up there with fairies and unicorns in that it will never happen in the real world.
Good economy was already done quite some time ago.
But to enable any kind of social change, crisis are engineered.
The documentary "Princes of the yen" [https://www.youtube.com/watch?v=p5Ac7ap_MAY] is a good example of it.
The current situation worldwide is also probably linked to TPP, TTIP and the likes.
"...With cancer studies and most recently psychology studies all having replication trouble, these economics papers have some company."
Unless we're actually going to institute some sort of reform when it comes to peer review and documented result validation, the only "company" these papers will be in is an endless sea of ignorance that assumes it won't keep happening over and over again.
There is far too much money to be made in simply publishing papers (a.k.a. bullshit) to actually validate results. Therefore, a whole new breed of fucking liars (sorry, no other words suffice) has been manipulating policy for quite a long time now.
Now society, the onus is on you. What are you going to do about this issue? Sit around and wait for it to happen again? That's what you did the last dozen times.
The problem with economics is that is is the study of human interaction. There is no way to predict what will actually occur. The best you can do is derive general statements from first principles. The problem with this is the conclusions you reach tell politicians to do very little and the more they try to help the more they will hurt. This obviously isn't popular with politicians so they get people from schools that like to play with data fitting and pretend they can predict what the outcome will be when they inject hundreds of billions of dollars into the market. This is just what the politicians want to hear.
I love Jesus, except for his foreign policy.
$deity help us.
or die. All important studies should be replicated before being cited.
I am an econometrician (well sort of), which is probably worse, but at least we know that. But economics, independent of any data set availability or actual method problems, is broadly handicapped by the generally unobservable nature of the actual data that would enable the verification (or refutation) of a hypothesis. That is, much of the data is quite noisy with many variables mixed in with each other, and as such a big part of the work is trying to determine the extent to which the data itself is a useful measure of the thing being tested. Sometimes getting to a useful dataset is dependent on some awkward assumptions. As such, one of the biggest faults of Economic Theory is assuming a can opener (https://en.wikipedia.org/wiki/Assume_a_can_opener).
"The first thing to do when you find yourself in a hole is stop digging."
For most studies of any kind, the margin for error is around +/-3%. For example, a study covering the United States population using a sample size of 1000 will yield a margin of error of 3.1%.
So a study says that texting while driving increases your risk of a fatal crash by 23 times. That sounds like a lot! But hold the phone...the overall rate of traffic fatalities is about 10 per 100,000 people, or about 0.01%. Multiply that by 23, and you get 0.23%. A big change, right? But that 0.23% is still well within the margin for error of most any study.
I'm not saying that texting while driving isn't dangerous. I'm just saying it's a lot harder to prove a link than it would seem.
Still, people are wowed by big multipliers, and news writers love to tout dramatic statistics, whether the subject of the study is economics, medicine, or traffic safety. But if you understand statistics, you know that most of these studies don't really tell us much. It's no wonder we keep getting contradictory study results!
The data can also be a problem. Economic analyses of the income and lifestyle of people with chronic medical conditions is often privileged, and the raw data cannot necessarily be made public.
The gist seems to be that he read about an omission in climate models, claims it's grounds to question the evidence for climate change, and takes umbrage that others disagree. In particular, he quotes:
From this he concludes:
Earlier in the article he stated:
I don't find the article convincing, because I think you do need an understanding of a model to determine how much of an impact a discrepancy in particular data will be likely to have on the model's reliability.
In some particular model, some data may be critical, and other data less so, and it may be possible to approximate some data without a large effect on the outcome. To take an extreme example, astronomical models often approximate stars and planets as point masses, thereby approximating all of us out of existence. This is clearly a gross approximation for many purposes, but often not so much of an issue for astronomy.
I know something about astronomical models, at least with regards to Newtonian physics, and am fairly comfortable making the above claim. I don't know much about climate models, and don't claim to know the significance of the discrepancy. There's nothing in the article, however, to convince me that the author knows any more about this than I do.
A new initiative for peer-reviewed replication: http://rescience.github.io
ReScience is a peer-reviewed journal that targets computational research and encourages the explicit replication of already published research, promoting new and open-source implementations in order to ensure that the original research is reproducible. To achieve this goal, the whole publishing chain is radically different from other traditional scientific journals. ReScience lives on GitHub where each new implementation of a computational study is made available together with comments, explanations and tests. Each submission takes the form of a pull request that is publicly reviewed and tested in order to guarantee that any researcher can re-use it. If you ever replicated computational results from the literature in your research, ReScience is the perfect place to publish your new implementation.
This is not unique to economics. Most scientific fields have problems with replication. Journals are strongly biased toward publishing positive results, and nobody gets tenure for negative results or replication.
Economics is not a scientific field and the fields which seems to have the most problems with this seem to be medical, not scientific ones and "nobody gets tenure for negative results" is simply not true because I did! Indeed it is common in particle physics where we search for evidence of new physics beyond the Standard Model and, with only one exception so far, keep coming up empty handed. As for the most recent Nobel for a "failed" experiment try the one of two days ago: this was awarded to two experiments which failed to show that the Standard Model description of neutrinos was correct.
I think your definition of "failed experiment" needs almost completely reversing. Michelson-Morley was a stunning success: it completely destroyed the luminiferous aether model for light. It was not the result that was expected but that does not make it a failure. The same applies to neutrino oscillations. Not getting a result you expect from an experiment is the thing every scientist hopes for it because means that you have learnt something new about the universe which is why these experiments often win Nobel prizes. If anything is a failed experiment it is those that just end up confirming existing theories because you were hoping you might learn something new and instead just ended up confirming what you already knew.
Economy is not science. There's your reason.
"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." - https://en.wikipedia.org/wiki/...
Of course news about a fake are Fake News.
Math, and its application, never was a strength of the economists. Nor was or is logical reasoning or application of scientific methods.
Ages ago, I made a fake economics whitepaper. It looked like the stuff I've seen in their libraries, the text was equally braindead, and it contained a lot of made-up formulas. The formulas were completely irrelevant (as was the text), but if anyone had followed the pattern to read the "input data" in the text before and actually pass it through the shown calculations, he would have noticed a) that the actual result did not match whatever was written in the text, and b) that all formulas resulted in an 8-digit number (with the decimal point in varying places) like 1991.0401 or 199104.01 (I'm no longer sure aboute the "1991" part, it could have been "1992" or "1993", don't care).
Of course this ended up in their library, along with a matching card in the index (they still had a paper-based index back then).
I actually got asked by someone if he could base his doctoral thesis on my "phenomenal" findings. I told him to do the math, and contact me again if he still considered this a good idea.
That's just a small area of economic research. A lot of studies work with publicly accessible data after it had been 'massaged' one way or another. That's because typical census/survey data sets come in multi-gigabyte mess of numbers, while a typical research question needs to operate on a small subset or a transformation of that subset. How the researcher gets from the raw survey data file down to the data set he is analyzing is not always clear. Also, the statistical codes, which are not always trivial.
I'm intimately aware of one on the US state level: Brownbackistan. It's being transformed into a Koch Brothers Randian paradise (i.e., hell on earth for the 99%).
To their credit, they have proven that by cutting taxes and gutting the state's regulatory, financial, and educational systems, you can indeed end up with far, far fewer real jobs and a demonstrably lower standard of living for those that can't even leave the state Grapes of Wrath style. Also having a White Power advocate as Secretary of State is just gravy.
I don't care what anybody says, economics isn't a science, and can't be a science ... because far too much about how you interpret and use economics is determined by how you ideologically believe it should work.
Economics in large part is bad math, with unfounded assumptions, making hand-waving conclusions about something which happened (or you believe should happen) to explain it according how you need it to be explained to match your world view.
Economics is not and never can be an objective science.
Lost at C:>. Found at C.
Astrology is more accurate.
Science in which experimental controls are illegal, isn't.
Experiments conducted on unwilling human subjects isn't ethical.
All of the social "sciences" are unethical non-sciences.
The solution?
Sort proponents of social theories into governments that test them.
Seastead this.
No true professor would you use such a fallacy as you have.
Further, if you are, as you claim, a particle physicist, you know that "not finding a particle" is not a failed experiment, but a refudiation of a theory, and consequently is consequently a positive result, not a negative one.
So, wait. Are you saying the "market" for scientific research is driving people to publish fake science? How can that be when the invisible hand is supposed to save us all from human foibles.
Maybe you are blind, but most of us, even a schooler, can see the difference between a 45 slope (predicted) and a 0 slope (measured). And that's why climate change has become a "science" that you only find discussed in sites that lean heavily to the left side of the political spectrum, as those people are totally impervious and blind to facts, and truth is whatever their media repeat enough times.
And to be on-topic, this is how the economics field works. Because the vast majority of academic economists are socialists. You know, there is a reason it is called a "social" science.
You are correct, the scientific validity of a paper does not rest in where it is published or who has reviewed it... but simply in its repeatability.
It should probably be made part of degree progression (somewhere between honors and doctorate) to show or not show the repeatability of existing papers... and only those papers that are shown to be repeatable become part of the accepted knowledge, the unrepeated papers to be considered speculative at best, and the unrepeatable to be abandoned.
that Republican economic theory is terrible in theory AND practice
Climate models are a dime a thousand and most have turned out to be rubbish. None of them predicted a 19+ year pause in the warming "trend" (which was coaxed out of adjusting historical temperatures to get the "right" result, by the way). James Delingpole demonstrates clearly why many non-scientists have stopped listening to the prophets of climate doom: their nay-saying has been around long enough for us to learn that after two decades of telling us the sky will fall - when it hasn't - they're just rent seekers dancing to the tune of government grant money. Pure and simple. We need to get beyond this sad state of affairs and redirect this wasted money to actual progress.
Mark Steyn has written an excellent book on that predicted 45-degree slope, it's called "A Disgrace to the Profession". The disgrace he refers to, of course, is Michael Mann's so-called "hockey stick" graph, the one Al Gore relied upon so much in his "Inconvenient Truth" beat up of a non-problem.
I don't think there's anything about this in the linked article, which is all I was responding to.
Sorry, just checked this thread.
I'm not very well read on this topic, and thought I'd better read something before responding, so had a browse through the Wikipedia page on the hockey stick graph, but don't feel I know a great deal more, except that it all sounds rather complicated. (Thanks, BTW, I didn't know what the AC's "45-degree slope" referred to--I assumed it was some temperature prediction, and had heard of the hockey stick graph, but didn't make a connection.)
My "understanding" of climate models still basically goes no further than: People are putting more CO2 in the atmosphere, and atmospheric CO2 traps heat. I guess it's the sceptics' view that this effect is relatively minor in the scheme of things?