Slashdot Mirror
Study: More Than Half of Psychological Results Can't Be Reproduced
Bruce66423 writes: A new study trying to replicate results reported in allegedly high quality journals failed to do so in over 50% of cases. Those of us from a hard science background always had our doubts about this sort of stuff — it's interesting to see it demonstrated — or rather, as the man says: 'Psychology has nothing to be proud of when it comes to replication,' Charles Gallistel, president of the Association for Psychological Science. Back in June a crowd-sourced effort to replicate 100 psychology studies had a 39% success rate.
Does anybody know how this compares to the hard sciences? How many published math papers turn out to be incorrect? How many physics experiments cannot be reproduced?
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
You can fool all of the people, some of the time, or some of the people, all of the time.
"National Security is the chief cause of national insecurity." - Celine's First Law
It's hard to believe psychology studies are more reproducible than cancer studies (11% reproducible): http://www.nature.com/nature/j...
1^2=1; (-1)^2=1; 1^2=(-1)^2; 1=-1; 1=0.
Yes there is a problem, and yes there needs to be a solution.
http://fivethirtyeight.com/dat...
Place something witty here
we have a pill for that
A study by more than 270 researchers from around the world has found that just 39 per cent of the claims made in psychology papers published in three prominent journals could be reproduced unambiguously – and even then they were found to be less significant statistically than the original findings.
The non-reproducible research includes studies into what factors influence men's and women's choice of romantic partners, whether peoples’ ability to identify an object is slowed down if it is wrongly labelled, and whether people show any racial bias when asked to identify different kinds of weapons.
The idea that this means "Psychology is wrong" is the opposite of what should be gleaned here. 1. This looked at a journal from 2008. That's pretty recent in scientific terms. 2. What is the replicability of new scientific findings in "hard" science fields like physics, chemistry, and medicine? Well, we don't know very well, because there hasn't been a concerted effort to explore that like there has been in psychology (At least not that I was able to find; I thought there was an effort. Please link if you know of it). 3. This is GREAT. This is what science is supposed to do, test itself. Now we have a much better understanding of what is "true" in psychology. 4. The author is something of a fool and probably not an actual scientist. A phrase like, "those of us in the hard sciences" speaks strongly to a high level of ignorant arrogance. People who do real science understand the noisiness of real data and difficulty of discovering truly new effects.
Psychology Research.... the field of people who don't know how to set up an actual experiment, and who are incapable of correctly interpreting any data they actually stumble onto, in any meaningful way other than to show they really don't have a clue what they're doing.... who for some reason seem to think that you're more "correct" if that horrendously bad idea you've had has been brought up by some other numbskull prior to you.
Fascism: An authoritarian and nationalistic right-wing system of government and social organization. See also: NAZI's
Isn't this like statistics, where half of all the statistics are made up? Or half of all sandwiches are not finished? Or half of the slashdot topics get skipped :)
If you have 100 studies you are replicating, by sheer chance you are likely still going to have a few who you successfully replicate but aren't real. So the problem may be worse than that (slightly). Psychology isn't the only field with these issues. There have been a lot of problems in medicine also. See https://www.newscientist.com/article/mg21528826-000-is-medical-science-built-on-shaky-foundations/. Part of the problem is one of incentives: the incentives to do a study which simply replicates a pre-existing study is low, and many journals won't even publish them. This also combines in bad ways with post-hoc analysis where you look at your data and find a pattern in it that is worth publishing; the worst offender here is medicine where people use different statistical tests and different subgroup analysis until they get a positive result.
Those of us from a hard science background always had our doubts about this sort of stuff
Maybe you should have been using those doubts for introspection? I can easily find numerous retractions from "hard science" journals plus it's easy to find a number of similar studies showing reproducibility in "hard" science medical research and drug trials.
Has this study been replicated?
Or is it perhaps a replication of an earlier study?
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
I mean it just seems too true to not be real https://en.wikipedia.org/wiki/...
Did you know 80 to 90% of the moderators on slashdot wouldn't recognize a troll even if one dragged them under a bridge.
The study linked to in the story and the "Back in June" study are ONE AND THE SAME.
...why it's not actually called a 'science' by anyone who understands what science is.
-Styopa
Can't be reproduced, data has to be doctored and cooked to within an inch of its life to stay consistent, ideological agenda overrides any pretense of scientific method.
Yep, that's modern science alright!
SJW's don't eliminate discrimination. They just expropriate it for themselves.
... you don't make any important decisions based on a single paper. That's true for hard sciences as well as social sciences.
Science by its very nature deals in contradictory evidence. I'd argue that openness to contradictory evidence is the distinguishing characteristic of science. A and not A cannot be true at the same time, but their can be, and normally is, evidence for both positions. So that means science often generates contradictory papers.
What you need to do is read the literature in a field widely so you can see the pattern of evidence, not just a single point. Or, if you aren't willing to invest the time for that you can find what's called a review paper in a high-impact factor journal. A review paper is supposed to be a fair summary of recent evidence on a question by someone working in the field. For bonus points read follow-up letters to that paper. Review papers are not infallible, but they're a heck of a lot more comprehensive than any other source of information.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
Psychology doesn't exist. Holy crap, Scientology got something right!
Feynman talked about this fairly often, most notably in Cargo Cult Science. The problem seems to be most common in soft sciences, such as the rat running example he gives from psychology. See also his commentary on science education in Brazil.
The Brazil report appears to be unrelated, but hear me out.
Brazil's problem was cultural. Their textbooks included all of the right information, but it wasn't presented as things that the student could learn about the real world, just as facts to be memorized. Scientists without the culture of science will make lousy experiments because they don't understand what they want to do or why, or how or why they need to keep themselves honest.
The culture of physics in the US was very good, but they were unable to export that culture to Brazil when they tried.
In the same way, other branches of science were unable to duplicate the physics culture. The rat runners in the example given didn't understand what they were trying to do, so they didn't pay attention to Young's work, which would have helped keep them honest.
Crichton's Aliens Cause Global Warming lecture was given nearly 30 years after Feynman's Cargo Cult Science, and it shows a creeping degeneration of the culture of science.
I go a step further, and say that the decline of the science culture has been part of a general cultural decline. There has been no great art or literature or music in decades.
The good news is that people are waking up. The internet is connecting people to each other, to science, and to culture. We are pissed about the decline of the past century, the decline that we've allowed, or at least failed to prevent, and are steeling our resolve to do the hard work to restore our greatness.
Articles like this show the stirring of the cultural revival. Keep them coming, please.
See that "Preview" button?
Study: More Than Half of Psychological Results Can't Be Reproduced
That's not what my study said.
systemd is Roko's Basilisk.
That's a field that's full of crap if there ever was one.
Years ago I was into NLP. One of the things that sold me on the idea of learning about it was that the original guys who gave it the name NLP wanted to make reproducible psychological changes to people. One of the first things I learned was to remove phobias and had something like a 90% success rate even as a beginner.
Surprise, surprise, the mainstream psychological community, instead of trying to encourage and build it, has done its best to ensure that NLP is labelled as quackery and a personality cult. There's even some truth in the cult label but that only came about because the psychology community left few other doors open.
There worst thing is (and the reason I gave it up) because so few of the mainstream psychology authorities gave it any credence, it started attracting a lot of nut-jobs. And the nut-jobs couldn't fix phobias or anything else because they had too many problems of their own.
I denounce this propaganda attack piece paid for by Koch brothers seeking to destroy the planet and drown the poor for profit!!!!
Oh, this is not about Climate science? Never mind...
In Soviet Washington the swamp drains you.
I didn't have the patience to read through the whole article in detail, but I didn't see anything about how long back the study had checked - this may be important for the reulst. Eksperiments in psychology must be particularly difficult to set up and evaluate rigorously, and I suspect we weren't too good at it in the early years. Even in modern, physical medicine, where there now good practices, it can be very difficult to get strong data.
Psychology is complicated. Even identical twins don't have minds that function exactly the same under the closest of possible circumstances. Failure to reproduce the results of a study don't necessarily mean it was a bad study; it just means that our understanding of the study is incomplete.
I would rather we have studies disproven or adjusted by additional work than have those studies not published at all. The human brain is a very complicated thing that we actually know very little about; if we discard psychology entirely out of hand we will then do very little to further our understanding of how it actually works.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Psychology is today where Medicine was 200 years ago.
There's nothing inherently wrong with treating behavioral maladies, and such treatment could eventually be classified as medicine.
It's just not there yet.
(He also didn't have a very high opinion of chiropractors...)
"Reality is that which, when you stop believing in it, doesn't go away." - Philip K. Dick
Psychology is not science.
now that we really need him?
it's not science, it opinion.
The physician must not have been very familiar with Psychology. Psychology is no medicine, nor is it like medicine 200 years ago. Further, psychology is far more than the study and treatment of mental illnesses. It is the study of behavior. Proper functioning is a far broader field than mal-functioning. Likewise, the body of psychological scientific literature extends far beyond mental disorders. He's totally right about chiropractors.
It is a descriptive, non-applicable "science" at best. The sad thing is that psychologists have too much power in their hands. A job a applicant has to go through one of them where I live.
We are right to hold discoveries of science and the scientific method in high regard. But that approval is distinct from respecting scientists as a class. The problem of non-reproducibility is no fault in the scientific method but instead indication of the rotten state of modern academia.
Earlier in my career I worked at universities writing software used for psychology and neuroscience experiments. On the basis of that experience I can offer an explanation for why about 1/2 of experiments are not reproducible: A lot of psychology faculty are terrible liars. While some demonstrate perfect integrity, others, probably the ones generating all those irreproducible results, lied whenever it suited their purposes. Still others were habitual liars who lied not to achieve some specific outcome but out of habit or compulsion. The center director of one research group confided to me, after a dispute with the faculty, that he had not been able to control his compulsion to lie. And when I claim that faculty "lie", I do not mean what could, by any stretch, be characterized as errors, oversights, or honest differences of opinion. I mean abusive, sociopathic, evident and deliberate lying. Like being told that the inconvenient evidence which you have in hand, "does not exist."
The lying is enforced by implicit threat. One time I responded to an email message, correcting an error, and then immediately after that a prominent member of the faculty, somewhat creepily, follows me into the restroom, stands too close to me while I am using the urinal, and explains to me in a threatening tone the error of my reasoning, which according to him, was that, "it would not do that because it would not do that." The dean imposed a disciplinary penalty on me for objecting to that. Though that was unusual, typically challenging lies elicited, a yelling, screaming fit from a faculty member. So it's not just lying, but lying backed up by threatening, thuggish, behavior of the faculty and university administration. This was a highly-regarded department with generous NIH funding, which makes me think that lying in that field is kind of a mainstream thing.
The root cause here has little to do with science, per se, and has more to do with the rotten management of colleges and universities. Regardless of what the employee handbook states, there are few de facto restrictions on faculty conduct and university administrations act to cover up problems by disciplining and threatening the whistle-blowers. Jerry Sandusky was not a scientist, he was a football coach, but if you look at the way Penn State concealed child molestation and protected him, that is typical of the way universities respond to faculty misconduct as welll, and explains why academic dishonesty is tolerated. One full-time faculty member in the department in which I worked had not set foot in the department in over five years nor ever appeared in any of the classes which she "taught." According to the department chairman, every time she was contacted to encourage her retirement was, whe was, "drunk off her ass in the middle of the day." It was tolerated and covered up.
I am not claiming that all scientists, fields, or academic departments are full of liars. I have never worked closely with physicists, computer scientists or mathematicians on a daily basis, but none whom I know personally have behaved like that.
To sum it all up, psychology has a problem with poor reproducibility of published results, many of the psychology faculty I knew were terrible liars; there might be a causal connection between the two.
Ceci n'est pas une signature.
That is pretty much the charitable explanation that Feynman offered in his notorious public talk on this subject.
That and confirmation bias. The classic example is Millikan's initial but not-quite-accurate measurement of the charge of an electron followed by subsequent results that "drifted" slowly but surely to the more modern measurement value.
Gee, Millikan is way off, but I can't publish this, I need to go over my apparatus and procedures to find what is wrong. In that way, only small changes from the "accepted" value get published until converging on a more accurate value.
Feynman was as much as saying that research in behavioral psychology was a Cargo Cult -- going through the motions that brought the planes and ships to our island without understanding that the arrival of the planes and ships had something to do with a war fought far from the horizon of the island and that actions taken by people on the island have no influence on when that war started, how long it continued, and when it ended.
He pretty much gives the benefit of the doubt on fraud, but he calls them out on experimental control, giving the example of one investigator going to the trouble to isolate the cues rats were relying upon in a maze experiment, finding it to be the sounds their feet made on the wooden boards of the maze floor, suppressing that sound by placing the maze box in sound-dampening sand, and finding the behavior of the rats to be entirely different when deprived of that cue. The amazing thing, to Feynman, was that setting the maze in sand was never adopted by subsequent studies -- the scientists in that "community" just plain didn't care.
We're inherently too biased. Only an AI with scalable human-like intelligence will give us real answers about ourselves.
This may be why one will never be built.
Please do not read this sig. Thank you.
Didn't someone mention 11% reproducibility rate for cancer research above? You also seem to misunderstand what they mean by psychology.
Very interesting. Xenu would be proud.
Remember kids, if you're not paying for the service, YOU ARE THE PRODUCT THAT IS BEING SOLD.
Sartre said that the in-itself and the for-itself both existed, and went on about freedom and responsibility from the for-itself. But only the in-itself actually exists, and the for-itself is only a delusion we pick up as a consequence of the in-itself trying to understand itself. (Like you trying to understand your body which is doing trillions of trillions of things every second.) Judges pull these delusions out of their arse every day and write them into law, no voter approval necessary.
(...as of about 8-9 years ago) The psych department had its own stats class, taught by a psych professor. You couldn't get an exemption if you had a high-level statistics course under your belt already, they insisted that psych stats were 'special' somehow, and needed to be taught differently.
If by 'special', you mean 'less rigorous' and 'taught by people that literally don't understand the definition of a function', then yes, the classes were special, and failed to prepare the students in any significant way for good statistical analysis.
I'm sure the story is the same at many universities.
LOL
Science has disproved itself. It is not even internally consistent. Actual hard sciences get the same results. If you have no reason to trust the a result returned from science is more likely to be true that 50-50, then the entire system is just worthless. I think we need to rethink the whole process.
Troll is not a replacement for I disagree.
The sad truth is psychology is one big f'ing fraud. The people and organizations behind it vote on what is to be an *illnesses* and the industry has an incentive to add as many mental illnesses as possible. It gives them more things that they can diagnose people with, sell more drugs, sell more sessions, etc. They don't have ANY interest in actually helping people.
There is no science behind it. There is no evidence behind it. The American Psychological Association listed homosexuality in it's diagnostics manual which is THE manual American psychologists use to diagnose patients (victims). Psychology was *outlawed* hundreds of years ago because of the fraud that it was, but the industry persisted. Early on they gained some of the first lobbyists and eventually came out with new ways to "cure people" using electricity, lobotomy, and then drugs. Every time it was proven fraudulent they'd just come out with something new. Much to no effect as these things were little more than scams that harmed there victims.
Just because someone isn't responding doesn't mean there not in pain. You've just put them into a state that they *can't* express that pain.
All the major universities must have the prestige of staff who are published researchers. So there is this constant push to publish or lose your job. Not just Psychology, right across the board. The system creates a stream of mediocrity.
It's hardly surprising it's not replicatable. A lot of effects are culturally based. If you didn't share the same nurturing environment, what is normal is different. What you expect and how you react are different. Anything that codifies one particular set of "standards" is only trying to entrench their own biases and culture.
At least in this regard Scientology is right, Psychology is usually just voodoo. Just like Scientology, Psychology is another religion.
For those who have forgotten , or never knew in the first place, a New Mexico state senator
once proposed a law to deal with this, which would probably have been the greatest single reform of the legal system in US History, and still makes me grin. It passed the state senate with a unanimous vote! The timid morons in the state House, however, (probably aided by "campaign contributions" from professionals in the affected fields) killed it. The proposed law said:
"When a psychologist or psychiatrist testifies during a defendant’s competency hearing, the psychologist or psychiatrist shall wear a cone-shaped hat that is not less than two feet tall. The surface of the hat shall be imprinted with stars and lightning bolts. Additionally, a psychologist or psychiatrist shall be required to don a white beard that is not less than 18 inches in length, and shall punctuate crucial elements of his testimony by stabbing the air with a wand. Whenever a psychologist or psychiatrist provides expert testimony regarding a defendant’s competency, the bailiff shall contemporaneously dim the courtroom lights and administer two strikes to a Chinese gong."
Just imagine all the pseudo-scientific clap-trap that this would have destroyed in numerous legal cases where "experts" pretend to know what somebody is thinking, has thought, or will or won't do in the future.
Puh leeze. I would not be surprised to find that you couldn't reproduce 50% of the published papers in "hard sciences" couldn't be replicated either. Do the words "cold fusion" ring a bell? Depends what you call hard sciences though. Note that several of the irreproductible examples given are brain scans etc. which is definitely on the harder side of the softer sciences.
Star Trek transporters are just 3d printers.
nothing op said disputes the broader study of behavior. if we can agree to that, the only real bone of contention is whether psychology is still in its infancy, because medicine certainly is. and given the scope of what psychologists want to achieve, I thought 200 was generous.