Slashdot Mirror
Study: More Than Half of Psychological Results Can't Be Reproduced
Bruce66423 writes: A new study trying to replicate results reported in allegedly high quality journals failed to do so in over 50% of cases. Those of us from a hard science background always had our doubts about this sort of stuff — it's interesting to see it demonstrated — or rather, as the man says: 'Psychology has nothing to be proud of when it comes to replication,' Charles Gallistel, president of the Association for Psychological Science. Back in June a crowd-sourced effort to replicate 100 psychology studies had a 39% success rate.
Does anybody know how this compares to the hard sciences? How many published math papers turn out to be incorrect? How many physics experiments cannot be reproduced?
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
You can fool all of the people, some of the time, or some of the people, all of the time.
"National Security is the chief cause of national insecurity." - Celine's First Law
It's hard to believe psychology studies are more reproducible than cancer studies (11% reproducible): http://www.nature.com/nature/j...
1^2=1; (-1)^2=1; 1^2=(-1)^2; 1=-1; 1=0.
Yes there is a problem, and yes there needs to be a solution.
http://fivethirtyeight.com/dat...
Place something witty here
The idea that this means "Psychology is wrong" is the opposite of what should be gleaned here. 1. This looked at a journal from 2008. That's pretty recent in scientific terms. 2. What is the replicability of new scientific findings in "hard" science fields like physics, chemistry, and medicine? Well, we don't know very well, because there hasn't been a concerted effort to explore that like there has been in psychology (At least not that I was able to find; I thought there was an effort. Please link if you know of it). 3. This is GREAT. This is what science is supposed to do, test itself. Now we have a much better understanding of what is "true" in psychology. 4. The author is something of a fool and probably not an actual scientist. A phrase like, "those of us in the hard sciences" speaks strongly to a high level of ignorant arrogance. People who do real science understand the noisiness of real data and difficulty of discovering truly new effects.
If you have 100 studies you are replicating, by sheer chance you are likely still going to have a few who you successfully replicate but aren't real. So the problem may be worse than that (slightly). Psychology isn't the only field with these issues. There have been a lot of problems in medicine also. See https://www.newscientist.com/article/mg21528826-000-is-medical-science-built-on-shaky-foundations/. Part of the problem is one of incentives: the incentives to do a study which simply replicates a pre-existing study is low, and many journals won't even publish them. This also combines in bad ways with post-hoc analysis where you look at your data and find a pattern in it that is worth publishing; the worst offender here is medicine where people use different statistical tests and different subgroup analysis until they get a positive result.
Has this study been replicated?
Or is it perhaps a replication of an earlier study?
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
And what do you actually know about psychological research? Have you conducted any or read any full articles from the original authors? What do you know about scientific research at all? What scientific studies have you performed? I am a psychologist. I am also a fan of the "hard" sciences and have read several full articles in physics, chemistry, medicine, and several other fields.. I can tell you that experiments in the social sciences (when done correctly) are far more controlled (relatively speaking) and reviewed before they are conducted than experiments in "hard" fields. There is no IRB review process in experiments not involving human subjects. Fields that study non-living things like matter and energy have it far easier. Those things behave deterministically and predictably. With humans, animals, and other living things, the noise factor is intense. We have had to develop highly sophisticated techniques to be able to perform science and uncover truth. The truth of that science needs to be taken with a large grain of salt because we are talking about summarizing a very wide set of behaviors, outcomes, and causes with a relatively small amount of words.
I mean it just seems too true to not be real https://en.wikipedia.org/wiki/...
Did you know 80 to 90% of the moderators on slashdot wouldn't recognize a troll even if one dragged them under a bridge.
... you don't make any important decisions based on a single paper. That's true for hard sciences as well as social sciences.
Science by its very nature deals in contradictory evidence. I'd argue that openness to contradictory evidence is the distinguishing characteristic of science. A and not A cannot be true at the same time, but their can be, and normally is, evidence for both positions. So that means science often generates contradictory papers.
What you need to do is read the literature in a field widely so you can see the pattern of evidence, not just a single point. Or, if you aren't willing to invest the time for that you can find what's called a review paper in a high-impact factor journal. A review paper is supposed to be a fair summary of recent evidence on a question by someone working in the field. For bonus points read follow-up letters to that paper. Review papers are not infallible, but they're a heck of a lot more comprehensive than any other source of information.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
Feynman talked about this fairly often, most notably in Cargo Cult Science. The problem seems to be most common in soft sciences, such as the rat running example he gives from psychology. See also his commentary on science education in Brazil.
The Brazil report appears to be unrelated, but hear me out.
Brazil's problem was cultural. Their textbooks included all of the right information, but it wasn't presented as things that the student could learn about the real world, just as facts to be memorized. Scientists without the culture of science will make lousy experiments because they don't understand what they want to do or why, or how or why they need to keep themselves honest.
The culture of physics in the US was very good, but they were unable to export that culture to Brazil when they tried.
In the same way, other branches of science were unable to duplicate the physics culture. The rat runners in the example given didn't understand what they were trying to do, so they didn't pay attention to Young's work, which would have helped keep them honest.
Crichton's Aliens Cause Global Warming lecture was given nearly 30 years after Feynman's Cargo Cult Science, and it shows a creeping degeneration of the culture of science.
I go a step further, and say that the decline of the science culture has been part of a general cultural decline. There has been no great art or literature or music in decades.
The good news is that people are waking up. The internet is connecting people to each other, to science, and to culture. We are pissed about the decline of the past century, the decline that we've allowed, or at least failed to prevent, and are steeling our resolve to do the hard work to restore our greatness.
Articles like this show the stirring of the cultural revival. Keep them coming, please.
See that "Preview" button?
Study: More Than Half of Psychological Results Can't Be Reproduced
That's not what my study said.
systemd is Roko's Basilisk.
Psychology is complicated. Even identical twins don't have minds that function exactly the same under the closest of possible circumstances. Failure to reproduce the results of a study don't necessarily mean it was a bad study; it just means that our understanding of the study is incomplete.
I would rather we have studies disproven or adjusted by additional work than have those studies not published at all. The human brain is a very complicated thing that we actually know very little about; if we discard psychology entirely out of hand we will then do very little to further our understanding of how it actually works.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Psychology is today where Medicine was 200 years ago.
There's nothing inherently wrong with treating behavioral maladies, and such treatment could eventually be classified as medicine.
It's just not there yet.
(He also didn't have a very high opinion of chiropractors...)
"Reality is that which, when you stop believing in it, doesn't go away." - Philip K. Dick
And you clearly don't understand what science is, nor, despite what you may think, do you hang out or listen to people who do.
Let me help you. Science isn't about WHAT you study. It's about HOW you study it.
To avoid researcher biases creeping in and affecting the results. Even with double-blind studies, there can be issues of researcher bias. http://centerforopenscience.gi... This is exactly why what we're seeing here is science at work, not evidence of its failure. We must constantly review the findings of new (and even old) science to fully distinguish what is real from what is false. The world is complex place and it is often the case that things are true in one context and not true in others.
The physician must not have been very familiar with Psychology. Psychology is no medicine, nor is it like medicine 200 years ago. Further, psychology is far more than the study and treatment of mental illnesses. It is the study of behavior. Proper functioning is a far broader field than mal-functioning. Likewise, the body of psychological scientific literature extends far beyond mental disorders. He's totally right about chiropractors.
We are right to hold discoveries of science and the scientific method in high regard. But that approval is distinct from respecting scientists as a class. The problem of non-reproducibility is no fault in the scientific method but instead indication of the rotten state of modern academia.
Earlier in my career I worked at universities writing software used for psychology and neuroscience experiments. On the basis of that experience I can offer an explanation for why about 1/2 of experiments are not reproducible: A lot of psychology faculty are terrible liars. While some demonstrate perfect integrity, others, probably the ones generating all those irreproducible results, lied whenever it suited their purposes. Still others were habitual liars who lied not to achieve some specific outcome but out of habit or compulsion. The center director of one research group confided to me, after a dispute with the faculty, that he had not been able to control his compulsion to lie. And when I claim that faculty "lie", I do not mean what could, by any stretch, be characterized as errors, oversights, or honest differences of opinion. I mean abusive, sociopathic, evident and deliberate lying. Like being told that the inconvenient evidence which you have in hand, "does not exist."
The lying is enforced by implicit threat. One time I responded to an email message, correcting an error, and then immediately after that a prominent member of the faculty, somewhat creepily, follows me into the restroom, stands too close to me while I am using the urinal, and explains to me in a threatening tone the error of my reasoning, which according to him, was that, "it would not do that because it would not do that." The dean imposed a disciplinary penalty on me for objecting to that. Though that was unusual, typically challenging lies elicited, a yelling, screaming fit from a faculty member. So it's not just lying, but lying backed up by threatening, thuggish, behavior of the faculty and university administration. This was a highly-regarded department with generous NIH funding, which makes me think that lying in that field is kind of a mainstream thing.
The root cause here has little to do with science, per se, and has more to do with the rotten management of colleges and universities. Regardless of what the employee handbook states, there are few de facto restrictions on faculty conduct and university administrations act to cover up problems by disciplining and threatening the whistle-blowers. Jerry Sandusky was not a scientist, he was a football coach, but if you look at the way Penn State concealed child molestation and protected him, that is typical of the way universities respond to faculty misconduct as welll, and explains why academic dishonesty is tolerated. One full-time faculty member in the department in which I worked had not set foot in the department in over five years nor ever appeared in any of the classes which she "taught." According to the department chairman, every time she was contacted to encourage her retirement was, whe was, "drunk off her ass in the middle of the day." It was tolerated and covered up.
I am not claiming that all scientists, fields, or academic departments are full of liars. I have never worked closely with physicists, computer scientists or mathematicians on a daily basis, but none whom I know personally have behaved like that.
To sum it all up, psychology has a problem with poor reproducibility of published results, many of the psychology faculty I knew were terrible liars; there might be a causal connection between the two.
Ceci n'est pas une signature.
We're talking about the demarcation problem here. Still, I'll answer your sillyness.
Well when more then 1/2 of your studies and research can't be repeated, you lose.
Then I guess we should toss out the whole of modern medicine as well, eh? Don't be foolish. This is how science is supposed to work. What concerns me most is that when faced with a failed replication, your first reaction is to reject the original research. It could have easily been a failure on the part of the second experimenters. To sort that out, you need are more replications. Science is riddled with contradictory results. That's normal, which is why replication is so damned important. See, a single experiment doesn't often tell you very much. It sure as hell doesn't result in gospel truth. All you generally get is "this is what we did, this was the result".
For instance if I call myself the world best engineer and yet over 1/2 of everything i do is wrong and fails, that would be totally in disagreement.
Here's an interesting statistic: Ty Cobb has the highest batting average in Major League Baseball history. He is, put simply, the best batter in the history of the world. What was his batting average? 0.366 Let that sink in.
Required reading for internet skeptics
(...as of about 8-9 years ago) The psych department had its own stats class, taught by a psych professor. You couldn't get an exemption if you had a high-level statistics course under your belt already, they insisted that psych stats were 'special' somehow, and needed to be taught differently.
If by 'special', you mean 'less rigorous' and 'taught by people that literally don't understand the definition of a function', then yes, the classes were special, and failed to prepare the students in any significant way for good statistical analysis.
I'm sure the story is the same at many universities.