Slashdot Mirror
Study: More Than Half of Psychological Results Can't Be Reproduced
Bruce66423 writes: A new study trying to replicate results reported in allegedly high quality journals failed to do so in over 50% of cases. Those of us from a hard science background always had our doubts about this sort of stuff — it's interesting to see it demonstrated — or rather, as the man says: 'Psychology has nothing to be proud of when it comes to replication,' Charles Gallistel, president of the Association for Psychological Science. Back in June a crowd-sourced effort to replicate 100 psychology studies had a 39% success rate.
It's hard to believe psychology studies are more reproducible than cancer studies (11% reproducible): http://www.nature.com/nature/j...
1^2=1; (-1)^2=1; 1^2=(-1)^2; 1=-1; 1=0.
Big Pharma, in its chase for the ever mighty dollar, has made medical science into a farce of what it should be by now. Don't get me wrong, there's been a lot of progress made...but a lot of the information coming out from the companies backing the publishing of irreproducible results is leaving a large shadow over that progress; it's beginning to give me the perception that we're coming upon a plateau in our rate of advancement. It's also not easing my cynicism any.
Excellent point. From the article http://www.washingtonpost.com/...
"Despite the rather gloomy results, the new paper pointed out that this kind of verification is precisely what scientists are supposed to do: “Any temptation to interpret these results as a defeat for psychology, or science more generally, must contend with the fact that this project demonstrates science behaving as it should.”"
This is the kind of stuff that needs to be done.
Let's be mindful that academically, CS is not considered one of the hard sciences. It might look and smell like one, but nobody in physics, math, biology, engineering, or chemistry takes CS seriously. There is a pretty sharp intellectual and attitude divide at the engineering oriented schools between the EE majors and professors and the CS majors and professors. Much sharper divide than two fields with so much in common and a decent number of overlapping classes. There's a reason why IT/IS is taught at the business school rather than engineering or science, and it is because CS was so far into liberal arts-like territory that they weren't going to take a second chance when IT/IS emerged as a field needing their own degrees.
A biology researcher I know was pretty disgusted by the science when she worked with some psychologists. The gist came down to that the field considers results only as results for the particular cohort used in the study, not as a repeatable experiment.
It's interesting nonetheless seeing what studies come up as bunk and which get confirmed. For example, I opened up their data file and started pulling up random entries about gender differences for fun:
"Sex differences in mate preferences revisited: Do people know what they initially desire in a romantic partner?" - The original study claimed that while men often self-report having their selection criteria for a partner being a lot more hinged around appearance than women do, that in practice this isn't the case, and more to the point, people's self-reporting for what they want most in a partner has little bearing on what they actually find most important in partner selection in practice.
The re-analysis confirmed this study.
"Perceptual mechanisms that characterize gender differences in decoding women's sexual intent" - This was a followup study to an earlier study that claimed that women often perceive men's sexual interest as friendliness while men often perceive women's friendliness as sexual interest. This study found, by contrast, that while men often misperceive friendliness as sexual interest, they also often misperceive sexual interest as friendliness - that they're just worse in general than reading sexual interest than women.
The re-analaysis was thus in a way responding to both the original and the followup. And found neither to be true. They found no difference between men and women in ability to read sexual interest vs. friendliness.
"Loving those who justify inequality: the effects of system threat on attraction to women who embody benevolent sexist ideals." - this study was to test - and reported confirmation - of the hypothesis that men who don't trust the government will also tend to find attractive women who embody "benevolent sexist" stereotypes - that is, that women are vulnerable, need to be saved, belong in the house, are there to complete men, etc, vs. women who have interest in careers or activities outside of the family, expect to be seen as equals, etc.
The reanalysis showed no correlation at all.
"The Best Men Are (Not Always) Already Taken: Female Preference for Single Versus Attached Males Depends on Conception Risk" - this study claimed that women in relationships find single men more attractive when they're ovulating and partnered men when they're not, but that single women show no preference. They argued that this result is expected given selective factors.
The reanalysis showed no correlation at all in any of the above cases.
Stale pastry is hollow succor to one who is bereft of ostrich.
We are right to hold discoveries of science and the scientific method in high regard. But that approval is distinct from respecting scientists as a class. The problem of non-reproducibility is no fault in the scientific method but instead indication of the rotten state of modern academia.
Earlier in my career I worked at universities writing software used for psychology and neuroscience experiments. On the basis of that experience I can offer an explanation for why about 1/2 of experiments are not reproducible: A lot of psychology faculty are terrible liars. While some demonstrate perfect integrity, others, probably the ones generating all those irreproducible results, lied whenever it suited their purposes. Still others were habitual liars who lied not to achieve some specific outcome but out of habit or compulsion. The center director of one research group confided to me, after a dispute with the faculty, that he had not been able to control his compulsion to lie. And when I claim that faculty "lie", I do not mean what could, by any stretch, be characterized as errors, oversights, or honest differences of opinion. I mean abusive, sociopathic, evident and deliberate lying. Like being told that the inconvenient evidence which you have in hand, "does not exist."
The lying is enforced by implicit threat. One time I responded to an email message, correcting an error, and then immediately after that a prominent member of the faculty, somewhat creepily, follows me into the restroom, stands too close to me while I am using the urinal, and explains to me in a threatening tone the error of my reasoning, which according to him, was that, "it would not do that because it would not do that." The dean imposed a disciplinary penalty on me for objecting to that. Though that was unusual, typically challenging lies elicited, a yelling, screaming fit from a faculty member. So it's not just lying, but lying backed up by threatening, thuggish, behavior of the faculty and university administration. This was a highly-regarded department with generous NIH funding, which makes me think that lying in that field is kind of a mainstream thing.
The root cause here has little to do with science, per se, and has more to do with the rotten management of colleges and universities. Regardless of what the employee handbook states, there are few de facto restrictions on faculty conduct and university administrations act to cover up problems by disciplining and threatening the whistle-blowers. Jerry Sandusky was not a scientist, he was a football coach, but if you look at the way Penn State concealed child molestation and protected him, that is typical of the way universities respond to faculty misconduct as welll, and explains why academic dishonesty is tolerated. One full-time faculty member in the department in which I worked had not set foot in the department in over five years nor ever appeared in any of the classes which she "taught." According to the department chairman, every time she was contacted to encourage her retirement was, whe was, "drunk off her ass in the middle of the day." It was tolerated and covered up.
I am not claiming that all scientists, fields, or academic departments are full of liars. I have never worked closely with physicists, computer scientists or mathematicians on a daily basis, but none whom I know personally have behaved like that.
To sum it all up, psychology has a problem with poor reproducibility of published results, many of the psychology faculty I knew were terrible liars; there might be a causal connection between the two.
Ceci n'est pas une signature.