Study: More Than Half of Psychological Results Can't Be Reproduced

Bruce66423 writes: A new study trying to replicate results reported in allegedly high quality journals failed to do so in over 50% of cases. Those of us from a hard science background always had our doubts about this sort of stuff — it's interesting to see it demonstrated — or rather, as the man says: 'Psychology has nothing to be proud of when it comes to replication,' Charles Gallistel, president of the Association for Psychological Science. Back in June a crowd-sourced effort to replicate 100 psychology studies had a 39% success rate.

Comments

I think 538 covered this better.

[nucrash]

Yes there is a problem, and yes there needs to be a solution.

http://fivethirtyeight.com/dat...

Re:Comparison?

[Halo1]

In spite of the gut feeling of the submitter, it's not much better in at least computer science: http://reproducibility.cs.ariz...

39% without secondary false-positives.

[JoshuaZ]

If you have 100 studies you are replicating, by sheer chance you are likely still going to have a few who you successfully replicate but aren't real. So the problem may be worse than that (slightly). Psychology isn't the only field with these issues. There have been a lot of problems in medicine also. See https://www.newscientist.com/article/mg21528826-000-is-medical-science-built-on-shaky-foundations/. Part of the problem is one of incentives: the incentives to do a study which simply replicates a pre-existing study is low, and many journals won't even publish them. This also combines in bad ways with post-hoc analysis where you look at your data and find a pattern in it that is worth publishing; the worst offender here is medicine where people use different statistical tests and different subgroup analysis until they get a positive result.

Re:Comparison?

[Halo1]

In spite of the gut feeling of the submitter, it's not much better in at least computer science: http://reproducibility.cs.ariz...

And to clarify: they only checked for what they call "weak repeatability": was it possible to get the code from the original researchers and if yes, was it possible to build it (or did the author at least explain how he himself managed to build it). They did not even investigate whether they could replicate the reported results with the the code that built.

Feynman and Crichton

[Orgasmatron]

Feynman talked about this fairly often, most notably in Cargo Cult Science. The problem seems to be most common in soft sciences, such as the rat running example he gives from psychology. See also his commentary on science education in Brazil.

The Brazil report appears to be unrelated, but hear me out.

Brazil's problem was cultural. Their textbooks included all of the right information, but it wasn't presented as things that the student could learn about the real world, just as facts to be memorized. Scientists without the culture of science will make lousy experiments because they don't understand what they want to do or why, or how or why they need to keep themselves honest.

The culture of physics in the US was very good, but they were unable to export that culture to Brazil when they tried.

In the same way, other branches of science were unable to duplicate the physics culture. The rat runners in the example given didn't understand what they were trying to do, so they didn't pay attention to Young's work, which would have helped keep them honest.

Crichton's Aliens Cause Global Warming lecture was given nearly 30 years after Feynman's Cargo Cult Science, and it shows a creeping degeneration of the culture of science.

I go a step further, and say that the decline of the science culture has been part of a general cultural decline. There has been no great art or literature or music in decades.

The good news is that people are waking up. The internet is connecting people to each other, to science, and to culture. We are pissed about the decline of the past century, the decline that we've allowed, or at least failed to prevent, and are steeling our resolve to do the hard work to restore our greatness.

Articles like this show the stirring of the cultural revival. Keep them coming, please.