Slashdot Mirror
Study: More Than Half of Psychological Results Can't Be Reproduced
Bruce66423 writes: A new study trying to replicate results reported in allegedly high quality journals failed to do so in over 50% of cases. Those of us from a hard science background always had our doubts about this sort of stuff — it's interesting to see it demonstrated — or rather, as the man says: 'Psychology has nothing to be proud of when it comes to replication,' Charles Gallistel, president of the Association for Psychological Science. Back in June a crowd-sourced effort to replicate 100 psychology studies had a 39% success rate.
Does anybody know how this compares to the hard sciences? How many published math papers turn out to be incorrect? How many physics experiments cannot be reproduced?
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
You can fool all of the people, some of the time, or some of the people, all of the time.
"National Security is the chief cause of national insecurity." - Celine's First Law
It's hard to believe psychology studies are more reproducible than cancer studies (11% reproducible): http://www.nature.com/nature/j...
1^2=1; (-1)^2=1; 1^2=(-1)^2; 1=-1; 1=0.
Yes there is a problem, and yes there needs to be a solution.
http://fivethirtyeight.com/dat...
Place something witty here
A study by more than 270 researchers from around the world has found that just 39 per cent of the claims made in psychology papers published in three prominent journals could be reproduced unambiguously – and even then they were found to be less significant statistically than the original findings.
The non-reproducible research includes studies into what factors influence men's and women's choice of romantic partners, whether peoples’ ability to identify an object is slowed down if it is wrongly labelled, and whether people show any racial bias when asked to identify different kinds of weapons.
The idea that this means "Psychology is wrong" is the opposite of what should be gleaned here. 1. This looked at a journal from 2008. That's pretty recent in scientific terms. 2. What is the replicability of new scientific findings in "hard" science fields like physics, chemistry, and medicine? Well, we don't know very well, because there hasn't been a concerted effort to explore that like there has been in psychology (At least not that I was able to find; I thought there was an effort. Please link if you know of it). 3. This is GREAT. This is what science is supposed to do, test itself. Now we have a much better understanding of what is "true" in psychology. 4. The author is something of a fool and probably not an actual scientist. A phrase like, "those of us in the hard sciences" speaks strongly to a high level of ignorant arrogance. People who do real science understand the noisiness of real data and difficulty of discovering truly new effects.
If you have 100 studies you are replicating, by sheer chance you are likely still going to have a few who you successfully replicate but aren't real. So the problem may be worse than that (slightly). Psychology isn't the only field with these issues. There have been a lot of problems in medicine also. See https://www.newscientist.com/article/mg21528826-000-is-medical-science-built-on-shaky-foundations/. Part of the problem is one of incentives: the incentives to do a study which simply replicates a pre-existing study is low, and many journals won't even publish them. This also combines in bad ways with post-hoc analysis where you look at your data and find a pattern in it that is worth publishing; the worst offender here is medicine where people use different statistical tests and different subgroup analysis until they get a positive result.
Those of us from a hard science background always had our doubts about this sort of stuff
Maybe you should have been using those doubts for introspection? I can easily find numerous retractions from "hard science" journals plus it's easy to find a number of similar studies showing reproducibility in "hard" science medical research and drug trials.
Has this study been replicated?
Or is it perhaps a replication of an earlier study?
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
And what do you actually know about psychological research? Have you conducted any or read any full articles from the original authors? What do you know about scientific research at all? What scientific studies have you performed? I am a psychologist. I am also a fan of the "hard" sciences and have read several full articles in physics, chemistry, medicine, and several other fields.. I can tell you that experiments in the social sciences (when done correctly) are far more controlled (relatively speaking) and reviewed before they are conducted than experiments in "hard" fields. There is no IRB review process in experiments not involving human subjects. Fields that study non-living things like matter and energy have it far easier. Those things behave deterministically and predictably. With humans, animals, and other living things, the noise factor is intense. We have had to develop highly sophisticated techniques to be able to perform science and uncover truth. The truth of that science needs to be taken with a large grain of salt because we are talking about summarizing a very wide set of behaviors, outcomes, and causes with a relatively small amount of words.
I mean it just seems too true to not be real https://en.wikipedia.org/wiki/...
Did you know 80 to 90% of the moderators on slashdot wouldn't recognize a troll even if one dragged them under a bridge.
...why it's not actually called a 'science' by anyone who understands what science is.
-Styopa
Can't be reproduced, data has to be doctored and cooked to within an inch of its life to stay consistent, ideological agenda overrides any pretense of scientific method.
Yep, that's modern science alright!
SJW's don't eliminate discrimination. They just expropriate it for themselves.
... you don't make any important decisions based on a single paper. That's true for hard sciences as well as social sciences.
Science by its very nature deals in contradictory evidence. I'd argue that openness to contradictory evidence is the distinguishing characteristic of science. A and not A cannot be true at the same time, but their can be, and normally is, evidence for both positions. So that means science often generates contradictory papers.
What you need to do is read the literature in a field widely so you can see the pattern of evidence, not just a single point. Or, if you aren't willing to invest the time for that you can find what's called a review paper in a high-impact factor journal. A review paper is supposed to be a fair summary of recent evidence on a question by someone working in the field. For bonus points read follow-up letters to that paper. Review papers are not infallible, but they're a heck of a lot more comprehensive than any other source of information.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
Psychology doesn't exist. Holy crap, Scientology got something right!
Like medical research.....
Feynman talked about this fairly often, most notably in Cargo Cult Science. The problem seems to be most common in soft sciences, such as the rat running example he gives from psychology. See also his commentary on science education in Brazil.
The Brazil report appears to be unrelated, but hear me out.
Brazil's problem was cultural. Their textbooks included all of the right information, but it wasn't presented as things that the student could learn about the real world, just as facts to be memorized. Scientists without the culture of science will make lousy experiments because they don't understand what they want to do or why, or how or why they need to keep themselves honest.
The culture of physics in the US was very good, but they were unable to export that culture to Brazil when they tried.
In the same way, other branches of science were unable to duplicate the physics culture. The rat runners in the example given didn't understand what they were trying to do, so they didn't pay attention to Young's work, which would have helped keep them honest.
Crichton's Aliens Cause Global Warming lecture was given nearly 30 years after Feynman's Cargo Cult Science, and it shows a creeping degeneration of the culture of science.
I go a step further, and say that the decline of the science culture has been part of a general cultural decline. There has been no great art or literature or music in decades.
The good news is that people are waking up. The internet is connecting people to each other, to science, and to culture. We are pissed about the decline of the past century, the decline that we've allowed, or at least failed to prevent, and are steeling our resolve to do the hard work to restore our greatness.
Articles like this show the stirring of the cultural revival. Keep them coming, please.
See that "Preview" button?
Study: More Than Half of Psychological Results Can't Be Reproduced
That's not what my study said.
systemd is Roko's Basilisk.
I denounce this propaganda attack piece paid for by Koch brothers seeking to destroy the planet and drown the poor for profit!!!!
Oh, this is not about Climate science? Never mind...
In Soviet Washington the swamp drains you.
I didn't have the patience to read through the whole article in detail, but I didn't see anything about how long back the study had checked - this may be important for the reulst. Eksperiments in psychology must be particularly difficult to set up and evaluate rigorously, and I suspect we weren't too good at it in the early years. Even in modern, physical medicine, where there now good practices, it can be very difficult to get strong data.
Psychology is complicated. Even identical twins don't have minds that function exactly the same under the closest of possible circumstances. Failure to reproduce the results of a study don't necessarily mean it was a bad study; it just means that our understanding of the study is incomplete.
I would rather we have studies disproven or adjusted by additional work than have those studies not published at all. The human brain is a very complicated thing that we actually know very little about; if we discard psychology entirely out of hand we will then do very little to further our understanding of how it actually works.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Psychology is today where Medicine was 200 years ago.
There's nothing inherently wrong with treating behavioral maladies, and such treatment could eventually be classified as medicine.
It's just not there yet.
(He also didn't have a very high opinion of chiropractors...)
"Reality is that which, when you stop believing in it, doesn't go away." - Philip K. Dick
Psychology is not science.
Ignorant is a strong word to some. I don't mean it to be an insult here. You are ignorant of too much to understand my points it seems. You're missing what I am saying or purposefully misinterpreting or focusing on minor supplementary points. I had a point-by-point response to your questions, but they were lost right as I was about to post them. I don't have the time to re-do. I recommend boning up on the philosophy of science. Here is a good start: https://en.wikipedia.org/wiki/...
This is precisely what goes on. The more complex the object of study the weaker the criteria for evidence. For psych/medicine/etc it is 5e-2 for a "significant" result, for particle physics it is 3e-7, five orders of magnitude difference. That is just for illustration of the bar lowering, I do not recommend using the concept of statistical significance with regards to a null hypothesis, but if it wasn't that it would be some other way.
Well, be careful. There is real, scientific NLP. It is used in a wide variety of clinical and professional (and other) settings. It is a hot area of psychological research.
However, there is also the popular "NLP" concept advanced by "pick-up artists" like Magic and his band of bafoons. That is the nonsense stuff.
The buzzword/terminology issues definitely make it hard to know what to trust these days.
The physician must not have been very familiar with Psychology. Psychology is no medicine, nor is it like medicine 200 years ago. Further, psychology is far more than the study and treatment of mental illnesses. It is the study of behavior. Proper functioning is a far broader field than mal-functioning. Likewise, the body of psychological scientific literature extends far beyond mental disorders. He's totally right about chiropractors.
We are right to hold discoveries of science and the scientific method in high regard. But that approval is distinct from respecting scientists as a class. The problem of non-reproducibility is no fault in the scientific method but instead indication of the rotten state of modern academia.
Earlier in my career I worked at universities writing software used for psychology and neuroscience experiments. On the basis of that experience I can offer an explanation for why about 1/2 of experiments are not reproducible: A lot of psychology faculty are terrible liars. While some demonstrate perfect integrity, others, probably the ones generating all those irreproducible results, lied whenever it suited their purposes. Still others were habitual liars who lied not to achieve some specific outcome but out of habit or compulsion. The center director of one research group confided to me, after a dispute with the faculty, that he had not been able to control his compulsion to lie. And when I claim that faculty "lie", I do not mean what could, by any stretch, be characterized as errors, oversights, or honest differences of opinion. I mean abusive, sociopathic, evident and deliberate lying. Like being told that the inconvenient evidence which you have in hand, "does not exist."
The lying is enforced by implicit threat. One time I responded to an email message, correcting an error, and then immediately after that a prominent member of the faculty, somewhat creepily, follows me into the restroom, stands too close to me while I am using the urinal, and explains to me in a threatening tone the error of my reasoning, which according to him, was that, "it would not do that because it would not do that." The dean imposed a disciplinary penalty on me for objecting to that. Though that was unusual, typically challenging lies elicited, a yelling, screaming fit from a faculty member. So it's not just lying, but lying backed up by threatening, thuggish, behavior of the faculty and university administration. This was a highly-regarded department with generous NIH funding, which makes me think that lying in that field is kind of a mainstream thing.
The root cause here has little to do with science, per se, and has more to do with the rotten management of colleges and universities. Regardless of what the employee handbook states, there are few de facto restrictions on faculty conduct and university administrations act to cover up problems by disciplining and threatening the whistle-blowers. Jerry Sandusky was not a scientist, he was a football coach, but if you look at the way Penn State concealed child molestation and protected him, that is typical of the way universities respond to faculty misconduct as welll, and explains why academic dishonesty is tolerated. One full-time faculty member in the department in which I worked had not set foot in the department in over five years nor ever appeared in any of the classes which she "taught." According to the department chairman, every time she was contacted to encourage her retirement was, whe was, "drunk off her ass in the middle of the day." It was tolerated and covered up.
I am not claiming that all scientists, fields, or academic departments are full of liars. I have never worked closely with physicists, computer scientists or mathematicians on a daily basis, but none whom I know personally have behaved like that.
To sum it all up, psychology has a problem with poor reproducibility of published results, many of the psychology faculty I knew were terrible liars; there might be a causal connection between the two.
Ceci n'est pas une signature.
That is pretty much the charitable explanation that Feynman offered in his notorious public talk on this subject.
That and confirmation bias. The classic example is Millikan's initial but not-quite-accurate measurement of the charge of an electron followed by subsequent results that "drifted" slowly but surely to the more modern measurement value.
Gee, Millikan is way off, but I can't publish this, I need to go over my apparatus and procedures to find what is wrong. In that way, only small changes from the "accepted" value get published until converging on a more accurate value.
Feynman was as much as saying that research in behavioral psychology was a Cargo Cult -- going through the motions that brought the planes and ships to our island without understanding that the arrival of the planes and ships had something to do with a war fought far from the horizon of the island and that actions taken by people on the island have no influence on when that war started, how long it continued, and when it ended.
He pretty much gives the benefit of the doubt on fraud, but he calls them out on experimental control, giving the example of one investigator going to the trouble to isolate the cues rats were relying upon in a maze experiment, finding it to be the sounds their feet made on the wooden boards of the maze floor, suppressing that sound by placing the maze box in sound-dampening sand, and finding the behavior of the rats to be entirely different when deprived of that cue. The amazing thing, to Feynman, was that setting the maze in sand was never adopted by subsequent studies -- the scientists in that "community" just plain didn't care.
We're inherently too biased. Only an AI with scalable human-like intelligence will give us real answers about ourselves.
This may be why one will never be built.
Please do not read this sig. Thank you.
Very interesting. Xenu would be proud.
Remember kids, if you're not paying for the service, YOU ARE THE PRODUCT THAT IS BEING SOLD.
(...as of about 8-9 years ago) The psych department had its own stats class, taught by a psych professor. You couldn't get an exemption if you had a high-level statistics course under your belt already, they insisted that psych stats were 'special' somehow, and needed to be taught differently.
If by 'special', you mean 'less rigorous' and 'taught by people that literally don't understand the definition of a function', then yes, the classes were special, and failed to prepare the students in any significant way for good statistical analysis.
I'm sure the story is the same at many universities.
Science has disproved itself. It is not even internally consistent. Actual hard sciences get the same results. If you have no reason to trust the a result returned from science is more likely to be true that 50-50, then the entire system is just worthless. I think we need to rethink the whole process.
Troll is not a replacement for I disagree.
I'm going to suggest that it's possible to understand how science works without actually practicing it. I'm not suggesting that it's possible to understand it without reading a lot of papers.
The experimental psych papers I read were pretty detailed on the protocol and fairly rigorous in the analysis. They tended to take a hypothesis, test it, and publish in enough detail to replicate and confirm. I don't know about the papers I didn't read.
(The most masterful use of the "least publishable unit" theory I ever saw was in psychology, FWIW.)
"When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
The truth of that science (psychology) needs to be taken with a large grain of salt because we are talking about summarizing a very wide set of behaviors, outcomes, and causes with a relatively small amount of words.
What you're saying is, even in the best circumstances, you arrive at nothing more than generalizations and conjecture. That's fine for a social science i suppose, but not science.
What i have seen from many papers i've read (since the 1980s) is that confirmation bias is pretty common, typically in very obvious if not glaring ways. Some of the conclusions in papers i've seen are so far removed from what data was actually taken you have to wonder whether the research even looked at it, so it's either a massively flawed experiment design, or people too incompetent to read or use basic logic.... and this is the stuff that actually passes through peer review.
Also to point out something else:
I can tell you that experiments in the social sciences (when done correctly) are far more controlled (relatively speaking) and reviewed before they are conducted than experiments in "hard" fields.
Here you are simply wrong.
Fascism: An authoritarian and nationalistic right-wing system of government and social organization. See also: NAZI's
All the major universities must have the prestige of staff who are published researchers. So there is this constant push to publish or lose your job. Not just Psychology, right across the board. The system creates a stream of mediocrity.
Puh leeze. I would not be surprised to find that you couldn't reproduce 50% of the published papers in "hard sciences" couldn't be replicated either. Do the words "cold fusion" ring a bell? Depends what you call hard sciences though. Note that several of the irreproductible examples given are brain scans etc. which is definitely on the harder side of the softer sciences.
Star Trek transporters are just 3d printers.