Slashdot Mirror
Results Are In From Psychology's Largest Reproducibility Test: 39/100 Reproduced
An anonymous reader writes: A crowd-sourced effort to replicate 100 psychology studies has successfully reproduced findings from 39 of them. Some psychologists say this shows the field has a replicability problem. Others say the results are "not bad at all". The results are nuanced: 24 non-replications had findings at least "moderately similar" to the original paper but which didn't quite reach statistical significance. From the article: "The results should convince everyone that psychology has a replicability problem, says Hal Pashler, a cognitive psychologist at the University of California, San Diego, and an author of one of the papers whose findings were successfully repeated. 'A lot of working scientists assume that if it’s published, it’s right,' he says. 'This makes it hard to dismiss that there are still a lot of false positives in the literature.'”
I think it could have something to do with this XKCD:
https://xkcd.com/882/
Please remember that this applies to Psychology, a field that is rife with lots of historical issues (and it is getting better).
In the past, they've practiced thinly veiled religion (Freud, if you don't believe me it's because my unverifiable model explains it, and you are in denial)
They've overstated their findings, to the tune of "and because of this experiment, we can extrapolate that EVERYONE is just the same!" (Stanford experiment, one of the ones with reproduciblity problems too, coincidentally).
They've put up roadblocks to proper scientific evidence gathering (and so this experiment was done before we adopted an ethical code that made verification of its outstanding results, that is reproduciblity, possible, but we are going to believe the conclusions anyway).
It was always called a "soft" Science because that way they could dodge the bullet coming from the real Sciences. However, when you read some of their works (especially their older works) you begin to see a pattern. They come from a history that probably poised them to have a long and hard road to understanding much of what we really do. In their defense they were attempting to build a model of the "mind" which is something that they assumed existed in a particular way, but didn't really test (it took a long time for Turing to come around). Finally, they are burdened with a lot of thinking that doesn't meet Philosophical rigor, because they shore it up with testing that doesn't meet Scientific rigor.
I'm glad to see the new wave of Psychology coming through. The now base a lot of their findings on biochemical analysis and stronger testing (including better attention to controls and double blind testing, which to their credit, they invented). It's just disheartening that the field lacks respect in other ways because in every intro to Psychology class they keep pushing the sensational "Dogma" experiments as facts when in reality they often fail to reproduce the results.
Just taking this quick opportunity to post a link to my favorite journal, the Journal of Articles in Support of the Null Hypothesis: http://www.jasnh.com/ .
JASNH is one of the few places where you can submit a paper that says "we tested for X effect on Y and found no evidence that X affects Y". Generally this research is unpublishable and people will tweak parameters to get something career-advancing out of their research; I like JASNH because of the reminder that "falsifiability" can really happen.
We recently had heard in the office over one of the Yellow Machine that's made by Anthology Solutions.