Slashdot Mirror
Researchers Unable To Replicate Findings of Published Economics Studies (businessinsider.com)
An anonymous reader writes: Federal Reserve economists Andrew Chang and Phillip Li looked at 67 papers in 13 reputable academic journals. Their findings were shocking. Without the help of the authors, only a third of the results could be independently replicated. Even with the author's help, only about half, or 49%, could. Business Insider reports: "It's a pretty massive issue for economics, especially given the impact that the subject has on public policy. Li and Chang use a well-known paper by Carmen Reinhart and Ken Rogoff as an example. The study showed a significant growth drop-off once a country's national debts reached 90% of gross domestic product, but three years after being published the study was found to contain a significant Microsoft Excel error that changed the magnitude of the effect." With cancer studies and most recently psychology studies all having replication trouble, these economics papers have some company.
That sounds about right.
Economics has always been one of the least predictive of "sciences". Economists with an ideological bent make things up with no relationship to the real world and people believe them.
"If all the economists were laid end to end, they'd never reach a conclusion." - George Bernard Shaw
Economists usually publish their data and code, unlike other sciences. This is a good thing and helps find problems, like were found in this case.
here's the link: http://www.nature.com/news/rep...
You took all the economists in the world, and laid them end to end, they'd point in different directions.
So rise up, all ye lost ones, as one, we'll claw the clouds.
Isn't errors in the data analysis exactly the sort of issue the peer-review system is supposed to catch?
I did a brief course in cognitive psychology during my masters. The course was given by a fairly well known name in the field - an editor of one of the standard texts.
He specifically told us that we were to do 'anything we liked' to get our data to say what we wanted. He told us that it was vastly more important to publish and defend than not and get sacked. Very much a "it's easier to ask forgiveness than it is to get permission" sort of atmosphere.
It wouldn't surprise me that this sort of attitude is rampant in other areas of 'science'.
Only the hard sciences seem to have any real legitimacy and even then I wouldn't trust a biologist all that much.
The level of trust I'd give to any statement by someone working in a given area is directly proportional to the area's 'purity': http://xkcd.com/435/
...or the entire sweater of our existence will come unraveled.
Cloudiot: A person who does not see offsite storage as a way to lose control over access to his or her own data.
This paper just finds that in many cases when journals require that replication files be posted, they aren't. Of the half of studies that aren't "replicated", the majority are due to the fact that replication files simply aren't available. It's not like these people read the papers, got the data, and reran the analysis on their own. All this paper is saying is that posted replication files often either don't exist or don't work. Their work doesn't show that the results can't be replicated, just that they can't be replicated from public code.
Economics is just a religion, it's a human invention. It's a way of saying "We have the highest technology, energy resources and best materials in the history of humanity, but we will continue to enforce millenia-old ideas of "work"."
Hey, as long as there's numbers and formulas and lots of PhDs, it's gotta be true, right?
Issues like this were already being flagged in 2013:
http://www.nytimes.com/2013/04/19/opinion/krugman-the-excel-depression.html
http://www.washingtonpost.com/news/wonkblog/wp/2013/04/16/is-the-best-evidence-for-austerity-based-on-an-excel-spreadsheet-error/
First of all, shame on authors for either not checking their models enough, not asking others to check them, and not opening their models for others to see before publishing "important" results.
Secondly, and perhaps more importantly, shame on the rest of us (and especially policymakers) for relying on such kinds of work so quickly and without validation to support generally political agendas. It's almost the equivalent of funding vaccine-skeptic studies by choosing which doctors will speak in your favor without regard to a rigorous scientific review process.
I majored in Economics, an area of study with a huge problem with relying on mathematical formula without worrying about how accurately the formula reproduce reality. I'm honestly surprised even half are reproducible. The honest opinion of many professors therein is "pfft, the numbers agree with themselves perfectly! That's all that matters."
The solution is pretty simple. Every author must reveal the codes that were used to produce the results.
However, one interesting issue is that once every author reveals the codes, you could find out that half of them code in MATLAB with a proficiency comparable to a 3rd grader who just learned BASIC programming up to about the "goto" statement. Not only there is lots of spaghetti code, but it may also contain serious errors that may filter through into the papers. Hence, I suspect a lot of people will not be happy to reveal their codes.
If you think all economics is Republican, it's evidence that you haven't been paying attention. Of course, the rest of economics is no more reliable than the part you've noticed, but there *are* other schools of economics. There are even Marxian economists.
I think we've pushed this "anyone can grow up to be president" thing too far.
Current economic theory could well substitute for some primitive magical cult. Hence such monstrosities such as the Black–Scholes model (a rehashed nuclear physics formula). The magic formula that caused the global economic meltdown. You might as well wave a dead chicken at your Quotron/Bloomberg terminal.
Liar's Poker by Michael Lewis
"The only danger of deficit spending is the risk of inflation; that managed, there is absolutely no issue."
Right. "Managed"... This is up there with fairies and unicorns in that it will never happen in the real world.
So, the fact that Western economies make life longer and more pleasant than, e.g. the Soviet Union or Mao's China, is evidence of the Republican Adam Smith starving children?
Get your head out of your ass.
Contribute to civilization: ari.aynrand.org/donate
I'm ready for a good laugh. Demonstrate how "supply and demand" don't affect prices.
Contribute to civilization: ari.aynrand.org/donate
Nonsense. They are mathematically trivial problems.
It's just that there are too many variables, and many of them cannot be measured. Hence, there's a lot of space for bullshit.
"So, the fact that Western economies make life longer...
Get your head out of your ass."
Exactly the kind of thing being talked about here: on these kind of topics people let them go by their gut feelings, even when hard data can be easily found.
Life expectancy at birth, years 1985-1995*1:
USA | Cuba
1985 74.56 | 76.34
1986 74.61 | 76.43
1987 74.77 | 76.34
1988 74.77 | 76.49
1989 75.02 | 76.53
1990 75.21 | 76.53
1991 75.37 | 76.59
1992 75.64 | 76.65
1993 75.42 | 76.65
1994 75.57 | 76.65
1995 75.62 | 76.65
So no, "western economies" doesn't make life longer and, in fact, USA has always done merely so-so in this regard: look at the OECD tables and you'll see it belongs to the awkward squad, and the OECD doesn't even publish data about USA's infant mortality rates, probably because they are outright embarrasing for a first world country*3.
*1http://www.indexmundi.com/
*2 https://data.oecd.org/healthst...
*3 https://www.cia.gov/library/pu...
"...With cancer studies and most recently psychology studies all having replication trouble, these economics papers have some company."
Unless we're actually going to institute some sort of reform when it comes to peer review and documented result validation, the only "company" these papers will be in is an endless sea of ignorance that assumes it won't keep happening over and over again.
There is far too much money to be made in simply publishing papers (a.k.a. bullshit) to actually validate results. Therefore, a whole new breed of fucking liars (sorry, no other words suffice) has been manipulating policy for quite a long time now.
Now society, the onus is on you. What are you going to do about this issue? Sit around and wait for it to happen again? That's what you did the last dozen times.
The problem with economics is that is is the study of human interaction. There is no way to predict what will actually occur. The best you can do is derive general statements from first principles. The problem with this is the conclusions you reach tell politicians to do very little and the more they try to help the more they will hurt. This obviously isn't popular with politicians so they get people from schools that like to play with data fitting and pretend they can predict what the outcome will be when they inject hundreds of billions of dollars into the market. This is just what the politicians want to hear.
I love Jesus, except for his foreign policy.
I am an econometrician (well sort of), which is probably worse, but at least we know that. But economics, independent of any data set availability or actual method problems, is broadly handicapped by the generally unobservable nature of the actual data that would enable the verification (or refutation) of a hypothesis. That is, much of the data is quite noisy with many variables mixed in with each other, and as such a big part of the work is trying to determine the extent to which the data itself is a useful measure of the thing being tested. Sometimes getting to a useful dataset is dependent on some awkward assumptions. As such, one of the biggest faults of Economic Theory is assuming a can opener (https://en.wikipedia.org/wiki/Assume_a_can_opener).
"The first thing to do when you find yourself in a hole is stop digging."
For most studies of any kind, the margin for error is around +/-3%. For example, a study covering the United States population using a sample size of 1000 will yield a margin of error of 3.1%.
So a study says that texting while driving increases your risk of a fatal crash by 23 times. That sounds like a lot! But hold the phone...the overall rate of traffic fatalities is about 10 per 100,000 people, or about 0.01%. Multiply that by 23, and you get 0.23%. A big change, right? But that 0.23% is still well within the margin for error of most any study.
I'm not saying that texting while driving isn't dangerous. I'm just saying it's a lot harder to prove a link than it would seem.
Still, people are wowed by big multipliers, and news writers love to tout dramatic statistics, whether the subject of the study is economics, medicine, or traffic safety. But if you understand statistics, you know that most of these studies don't really tell us much. It's no wonder we keep getting contradictory study results!
The gist seems to be that he read about an omission in climate models, claims it's grounds to question the evidence for climate change, and takes umbrage that others disagree. In particular, he quotes:
From this he concludes:
Earlier in the article he stated:
I don't find the article convincing, because I think you do need an understanding of a model to determine how much of an impact a discrepancy in particular data will be likely to have on the model's reliability.
In some particular model, some data may be critical, and other data less so, and it may be possible to approximate some data without a large effect on the outcome. To take an extreme example, astronomical models often approximate stars and planets as point masses, thereby approximating all of us out of existence. This is clearly a gross approximation for many purposes, but often not so much of an issue for astronomy.
I know something about astronomical models, at least with regards to Newtonian physics, and am fairly comfortable making the above claim. I don't know much about climate models, and don't claim to know the significance of the discrepancy. There's nothing in the article, however, to convince me that the author knows any more about this than I do.
This is not unique to economics. Most scientific fields have problems with replication. Journals are strongly biased toward publishing positive results, and nobody gets tenure for negative results or replication.
Economics is not a scientific field and the fields which seems to have the most problems with this seem to be medical, not scientific ones and "nobody gets tenure for negative results" is simply not true because I did! Indeed it is common in particle physics where we search for evidence of new physics beyond the Standard Model and, with only one exception so far, keep coming up empty handed. As for the most recent Nobel for a "failed" experiment try the one of two days ago: this was awarded to two experiments which failed to show that the Standard Model description of neutrinos was correct.
I think your definition of "failed experiment" needs almost completely reversing. Michelson-Morley was a stunning success: it completely destroyed the luminiferous aether model for light. It was not the result that was expected but that does not make it a failure. The same applies to neutrino oscillations. Not getting a result you expect from an experiment is the thing every scientist hopes for it because means that you have learnt something new about the universe which is why these experiments often win Nobel prizes. If anything is a failed experiment it is those that just end up confirming existing theories because you were hoping you might learn something new and instead just ended up confirming what you already knew.
"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." - https://en.wikipedia.org/wiki/...
Of course news about a fake are Fake News.
Math, and its application, never was a strength of the economists. Nor was or is logical reasoning or application of scientific methods.
Ages ago, I made a fake economics whitepaper. It looked like the stuff I've seen in their libraries, the text was equally braindead, and it contained a lot of made-up formulas. The formulas were completely irrelevant (as was the text), but if anyone had followed the pattern to read the "input data" in the text before and actually pass it through the shown calculations, he would have noticed a) that the actual result did not match whatever was written in the text, and b) that all formulas resulted in an 8-digit number (with the decimal point in varying places) like 1991.0401 or 199104.01 (I'm no longer sure aboute the "1991" part, it could have been "1992" or "1993", don't care).
Of course this ended up in their library, along with a matching card in the index (they still had a paper-based index back then).
I actually got asked by someone if he could base his doctoral thesis on my "phenomenal" findings. I told him to do the math, and contact me again if he still considered this a good idea.
It was called "the dismal science" by Thomas Carlyle because economists claimed that slavery was economically inefficient.
Watch this Heartland Institute video
That's just a small area of economic research. A lot of studies work with publicly accessible data after it had been 'massaged' one way or another. That's because typical census/survey data sets come in multi-gigabyte mess of numbers, while a typical research question needs to operate on a small subset or a transformation of that subset. How the researcher gets from the raw survey data file down to the data set he is analyzing is not always clear. Also, the statistical codes, which are not always trivial.
I'm intimately aware of one on the US state level: Brownbackistan. It's being transformed into a Koch Brothers Randian paradise (i.e., hell on earth for the 99%).
To their credit, they have proven that by cutting taxes and gutting the state's regulatory, financial, and educational systems, you can indeed end up with far, far fewer real jobs and a demonstrably lower standard of living for those that can't even leave the state Grapes of Wrath style. Also having a White Power advocate as Secretary of State is just gravy.
I don't care what anybody says, economics isn't a science, and can't be a science ... because far too much about how you interpret and use economics is determined by how you ideologically believe it should work.
Economics in large part is bad math, with unfounded assumptions, making hand-waving conclusions about something which happened (or you believe should happen) to explain it according how you need it to be explained to match your world view.
Economics is not and never can be an objective science.
Lost at C:>. Found at C.
Science in which experimental controls are illegal, isn't.
Experiments conducted on unwilling human subjects isn't ethical.
All of the social "sciences" are unethical non-sciences.
The solution?
Sort proponents of social theories into governments that test them.
Seastead this.
You are correct, the scientific validity of a paper does not rest in where it is published or who has reviewed it... but simply in its repeatability.
It should probably be made part of degree progression (somewhere between honors and doctorate) to show or not show the repeatability of existing papers... and only those papers that are shown to be repeatable become part of the accepted knowledge, the unrepeated papers to be considered speculative at best, and the unrepeatable to be abandoned.
The US has only one of the "western economies", albeit the largest by far, and is not representative of the others in many ways. I believe Cuba has universal health care, for example, and although it almost certainly isn't nearly as good as US health care everybody gets it.
"When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
Climate models are a dime a thousand and most have turned out to be rubbish. None of them predicted a 19+ year pause in the warming "trend" (which was coaxed out of adjusting historical temperatures to get the "right" result, by the way). James Delingpole demonstrates clearly why many non-scientists have stopped listening to the prophets of climate doom: their nay-saying has been around long enough for us to learn that after two decades of telling us the sky will fall - when it hasn't - they're just rent seekers dancing to the tune of government grant money. Pure and simple. We need to get beyond this sad state of affairs and redirect this wasted money to actual progress.
Mark Steyn has written an excellent book on that predicted 45-degree slope, it's called "A Disgrace to the Profession". The disgrace he refers to, of course, is Michael Mann's so-called "hockey stick" graph, the one Al Gore relied upon so much in his "Inconvenient Truth" beat up of a non-problem.
I don't think there's anything about this in the linked article, which is all I was responding to.
Sorry, just checked this thread.
I'm not very well read on this topic, and thought I'd better read something before responding, so had a browse through the Wikipedia page on the hockey stick graph, but don't feel I know a great deal more, except that it all sounds rather complicated. (Thanks, BTW, I didn't know what the AC's "45-degree slope" referred to--I assumed it was some temperature prediction, and had heard of the hockey stick graph, but didn't make a connection.)
My "understanding" of climate models still basically goes no further than: People are putting more CO2 in the atmosphere, and atmospheric CO2 traps heat. I guess it's the sceptics' view that this effect is relatively minor in the scheme of things?