Slashdot Mirror
Study: More Than Half of Psychological Results Can't Be Reproduced
Bruce66423 writes: A new study trying to replicate results reported in allegedly high quality journals failed to do so in over 50% of cases. Those of us from a hard science background always had our doubts about this sort of stuff — it's interesting to see it demonstrated — or rather, as the man says: 'Psychology has nothing to be proud of when it comes to replication,' Charles Gallistel, president of the Association for Psychological Science. Back in June a crowd-sourced effort to replicate 100 psychology studies had a 39% success rate.
In spite of the gut feeling of the submitter, it's not much better in at least computer science: http://reproducibility.cs.ariz...
Donate free food here
Or this one from just under 2 weeks ago:
The requirement that medical researchers register in detail the methods they intend to use in their clinical trials, both to record their data as well as document their outcomes, caused a significant drop in trials producing positive results. From Nature: "The study found that in a sample of 55 large trials testing heart-disease treatments, 57% of those published before 2000 reported positive effects from the treatments. But that figure plunged to just 8% in studies that were conducted after 2000.
Big Pharma, in its chase for the ever mighty dollar, has made medical science into a farce of what it should be by now. Don't get me wrong, there's been a lot of progress made...but a lot of the information coming out from the companies backing the publishing of irreproducible results is leaving a large shadow over that progress; it's beginning to give me the perception that we're coming upon a plateau in our rate of advancement. It's also not easing my cynicism any.
So is medical science not "real science" because we've had quite a few stories over the last few years that a ton of results from medical research and drug trials can't be reproduced.
A large percentage of medical studies are funded by manufacturers, and it's fairly well understood that most of those don't get published unless they produce the "right" results. And those that are published are often really "preliminary", based on too little data to be considered reliable. But if a test on 10 or 20 patients gives the "right" results, there is a lot of marketing pressure to get the paper published right away.
This easily explains the growing problem of medical products that are found to be worthless (or even harmful) to the patients, after years of heavy marketing has produced large profits.
There's also the age-old problem that studies with "negative" results usually don't get published at all. As usual, there's a good xkdc comic that explains the methodology in a way that even the minimally numerate reader can understand.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
That sounds like a pretty weak test, but not a bad one. To my mind, this crosses one of my areas of expertise since, I have had his job as a professional sysadmin. I worked in a shop where, for better or worst, we decided that all free software we used on Solaris would be compiled from source.
This quickly became a huge mess as updates would sometimes bring changes and there was always the question "who built it last time and what options did they choose", so quickly we found a need to fix that, and I started scripting. (its where my competence with shell really began)
Once you have even solved the easy part, then you have to think about versions and dependencies.
In fact, later on we were getting involved in research computing, that wasn't my project but one of the topics that came up was... researchers will build this software, just like we are talking about, and use the data.,...now someone wants to audit it down the road....
What happens if the libraries have changed and the old code doesn't compile? What if there is an error in a calculation that was introduced by a particular library version being used?
The reality is, you write the code, but it gets run in an environment. That entire environment has the potential to have an effect, a full specification needs to capture at least some of that as well.
"I opened my eyes, and everything went dark again"
Exactly. This applies to "hard" sciences too. The jackass submitter couldn't get his head out of his ass long enough to read the article and understand the implication. One of the problems the article mentioned was journals not willing to publish null results and research that just replicates other research. That's rather important because you need more than one researching publishing similar findings to feel confident about the results. The other point is that some the results, though valid, were valid for very narrowly defined scenarios and therefore not generalization.
Yo Bruce66423, RTFA sometime.
Feynman talked about this fairly often, most notably in Cargo Cult Science. The problem seems to be most common in soft sciences, such as the rat running example he gives from psychology. See also his commentary on science education in Brazil.
The Brazil report appears to be unrelated, but hear me out.
Brazil's problem was cultural. Their textbooks included all of the right information, but it wasn't presented as things that the student could learn about the real world, just as facts to be memorized. Scientists without the culture of science will make lousy experiments because they don't understand what they want to do or why, or how or why they need to keep themselves honest.
The culture of physics in the US was very good, but they were unable to export that culture to Brazil when they tried.
In the same way, other branches of science were unable to duplicate the physics culture. The rat runners in the example given didn't understand what they were trying to do, so they didn't pay attention to Young's work, which would have helped keep them honest.
Crichton's Aliens Cause Global Warming lecture was given nearly 30 years after Feynman's Cargo Cult Science, and it shows a creeping degeneration of the culture of science.
I go a step further, and say that the decline of the science culture has been part of a general cultural decline. There has been no great art or literature or music in decades.
The good news is that people are waking up. The internet is connecting people to each other, to science, and to culture. We are pissed about the decline of the past century, the decline that we've allowed, or at least failed to prevent, and are steeling our resolve to do the hard work to restore our greatness.
Articles like this show the stirring of the cultural revival. Keep them coming, please.
See that "Preview" button?
It's interesting nonetheless seeing what studies come up as bunk and which get confirmed. For example, I opened up their data file and started pulling up random entries about gender differences for fun:
"Sex differences in mate preferences revisited: Do people know what they initially desire in a romantic partner?" - The original study claimed that while men often self-report having their selection criteria for a partner being a lot more hinged around appearance than women do, that in practice this isn't the case, and more to the point, people's self-reporting for what they want most in a partner has little bearing on what they actually find most important in partner selection in practice.
The re-analysis confirmed this study.
"Perceptual mechanisms that characterize gender differences in decoding women's sexual intent" - This was a followup study to an earlier study that claimed that women often perceive men's sexual interest as friendliness while men often perceive women's friendliness as sexual interest. This study found, by contrast, that while men often misperceive friendliness as sexual interest, they also often misperceive sexual interest as friendliness - that they're just worse in general than reading sexual interest than women.
The re-analaysis was thus in a way responding to both the original and the followup. And found neither to be true. They found no difference between men and women in ability to read sexual interest vs. friendliness.
"Loving those who justify inequality: the effects of system threat on attraction to women who embody benevolent sexist ideals." - this study was to test - and reported confirmation - of the hypothesis that men who don't trust the government will also tend to find attractive women who embody "benevolent sexist" stereotypes - that is, that women are vulnerable, need to be saved, belong in the house, are there to complete men, etc, vs. women who have interest in careers or activities outside of the family, expect to be seen as equals, etc.
The reanalysis showed no correlation at all.
"The Best Men Are (Not Always) Already Taken: Female Preference for Single Versus Attached Males Depends on Conception Risk" - this study claimed that women in relationships find single men more attractive when they're ovulating and partnered men when they're not, but that single women show no preference. They argued that this result is expected given selective factors.
The reanalysis showed no correlation at all in any of the above cases.
Stale pastry is hollow succor to one who is bereft of ostrich.