Psychologists Strike a Blow For Reproducibility

ananyo writes "Science has a much publicized reproducibility problem. Many experiments seem to be failing a key test of science — that they can be independently verified by another lab. But now 36 research groups have struck a blow for reproducibility, by successfully reproducing the results of 10 out of 13 past experiments in psychology. Even so, the Many Labs Replication Project found that the outcome of one experiment was only weakly supported and they could not replicate two of the experiments at all."

Comments

Good news, everyone!

[Anonymous Coward]

We ought to be encouraging this sort of thing.

Re:Good news, everyone!

[Jeremiah Cornelius]

Have you ever BEEN to an APA conference?

My dear god, man! The last thing you want is these people REPRODUCING!

Psychology

[Anonymous Coward]

If only Psychology was a science.

Re:Psychology

[__aaltlg1547]

If the experiments are reproducible, it's science.

Apparently it's biochemistry that is not a science.
http://online.wsj.com/news/articles/SB10001424052970203764804577059841672541590

Re:Psychology

[FatdogHaiku]

If only Psychology was a science.

Lol -- psychiatrists and psychologist doing experiments. It's is a weak science at its best; a modern day priesthood at its worst.

For your heresy and disbelief you have been diagnosed as a paranoid schizophrenic!
We will monitor you to see if medication will be adequate to silence you, I mean, control your symptoms...

Re:Psychology

[Lamps]

The ironic thing about statements like these is that they usually come from people with no scientific training in any field, nor any meaningful training in statistics, but only a "sciency" inclination and questionable, popular distillation-derived knowledge of some principles from what they consider "the hard sciences".

Sadly, this irony will be lost on the people making such statements, who will, for some unfathomable reason, continue to disparage people doing meaningful work in the sciences, while never coming close to accomplishing anything of the sort themselves.

Actual academics have an idea of the hard work involved in contributing to the human knowledge base in all scientific disciplines, and thus, tend to respect each other's work (as long as others don't step on their own toes in their particular area of specialization, in which case, prepare for turbulence).

Re:Psychology

[Michael Woodhams]

Psychology is a huge field. Perception, experimental analysis of animal behaviour, clinical psychology, cognitive biases etc. etc. (Note that only one of those involves psychiatrists.) Some bits allow for harder science than other bits.

I personally don't know enough about psychiatry to form a judgement on how scientific they are, but unlike you, at least I know what a psychologist is (or something of the range that they could be.) Your trite dismissal says much about your ignorance and nothing about psychology.

Re:Psychology

[antifoidulus]

Unfortunately there are 2 major hurdles that limit all but the most groundbreaking experiments:

1. Money
2. Glory

Money is probably the biggest factor, there just isn't enough money allotted to trying to reproduce experiments. Most budgets only exist for new/continuing research, not verifying experiments done by others. And as the cost of doing experiments rises(more sophisticated equipment necessary, lots of paid "volunteers" etc) this is only going to get worse.

Second, although not as important, is the "glory" factor. Very few talented scientists want to spend their time and research money on reproducing experiments done by others. There aren't a whole lot of publishing opportunities in doing so, esp. if you cannot refute what they have done. You can see this to a certain extent in the open source world as well, for all but the most famous of projects you tend to have a very large # of projects that essentially do the same thing. Why? Because a lot of people want to "invent" a new program rather than improve upon what is there. Fortunately in the OSS world money isn't nearly as much an issue, so you do have large #s of people improving OSS rather than trying to re-invent the wheel.

Re:Psychology

[Anonymous Coward]

Angry.

Mostly angry.

Re:Psychology

[rtb61]

Psychology ran in a major hiccup, as many of it's experiments are no longer reproducible not because of bad 'science' but because they are considered naughty and not something that should really be done to people to test out psychological theories, as in http://www.bps.org.uk/what-we-do/ethics-standards/ethics-standards (I used British standards rather than US ones, as the US ones have so badly been mauled by the US government and their fully medically and psychological researched mass torture facility at GITMO that the US ones are rules that 'should be' broken as defined by the US government) and http://mentalfloss.com/article/52787/10-famous-psychological-experiments-could-never-happen-today.

Re:Psychology

[steelfood]

Psychology is a soft science because of the numerous variables that in studies are often simplified into a constant often for simplicity's sake and nothing else. Economics and politics are the same, mostly because they're based on psychology.

It's an inexact science because the human condition is imperfect. As opposed to the hard sciences, which are exact, because the universe around us is "perfect". And then, there's computer science, which is a mathematical, computational science that's absolute. It's not even "perfect" anymore; it's exactly what the maths say it is, and any failure sits between keyboard and chair.

Anyway, psychology is important, because the only way to truly understand the imperfect conditions of humans is via an inexact science. And it's something only fully understood by humans (computers can simulate the hard sciences to a calculable degree of accuracy, but they'll never be able to simulate the soft sciences in the same way), and innately at that.

The way to think about psychology is using fractals. X% | X is > statistical significance, of the population behaves in manner a. X * (100-X)% of the population behaves in manner b. X * (100 - X * (100-X))% of the population behaves in manner c. Etc. a, b, c, etc. are up to you to figure out. And when you change the test, the individual that falls into one category is not guaranteed to fall into the same category again.

Note that the human mind can comprehend infinity (poorly for most, but very possible for a few), both countable and uncountable variants, but a computer will never be able to calculate it. So the fractal analogy works really, really well.

Re:Psychology

[VortexCortex]

Psychology is a huge field. Perception, experimental analysis of animal behaviour, clinical psychology, cognitive biases etc. etc.

No, the field you're thinking of is Neuroscience and Cybernetics -- These have evidence based on observation and models which have predictive power. Psychology is just confirmation bias. You must prove the null hypothesis more implausible than the original hypothesis, yet Psychology does not do this. For every ridiculous Sexual Epistemology, there's an equally valid Scatological Epistemology.

The truth is that neurons fire in brains, and that complexity gives rise to emergent behaviours. Leaping the gulf in understanding to arrive at the explanations that Psychology and Philosophy give is akin to claiming a God in a Chariot pulls the Sun across the sky.

Re:Psychology

[rtaylor]

Science is a process, not a field of study or a result.

That process can be applied to anything where you want to find a fact.

Re:Psychology

[Dcnjoe60]

Psychology is a huge field. Perception, experimental analysis of animal behaviour, clinical psychology, cognitive biases etc. etc.

No, the field you're thinking of is Neuroscience and Cybernetics -- These have evidence based on observation and models which have predictive power. Psychology is just confirmation bias. You must prove the null hypothesis more implausible than the original hypothesis, yet Psychology does not do this. For every ridiculous Sexual Epistemology, there's an equally valid Scatological Epistemology.

The truth is that neurons fire in brains, and that complexity gives rise to emergent behaviours. Leaping the gulf in understanding to arrive at the explanations that Psychology and Philosophy give is akin to claiming a God in a Chariot pulls the Sun across the sky.

Your argument is only valid for recent times. For most of the history of modern psychology, there was not a separate field between neuroscience and psychology. Often in med school today, the psych departments and the neurology departments have been combined because we have found that the two are interrelated.

Since most psychological research deals with evidence based observation and models which have predictive power, what distinguishes it from the subclassification of neuroscience? It's a little bit like saying that nuclear physics is science but theoretical physics is not (particularly the whole observational part). I think the flaw in your position is that you feel that science has at it's root the desire to prove the null hypothesis as more implausible than the original hypothesis. In the hard sciences, such as chemistry and physics, that type of test would be meaningless. Just because Newton's theory of gravity has been uprooted by quantum theory, does that mean Newtonian Physics isn't science? No, of course not. That's because most science is about proving what is, not what is not.

If the measure of science is the ability to prove the null hypothesis being more implausible than the original or non-null hypothesis, then theology is a science because it is a lot more difficult to prove that a deity does not exist than it is to prove one actually does.

Not bad at all

[TwineLogic]

So 2 or 3 out of 13 were not reproduced in these attempts. I imagine the standard was "P 0.05." If you then consider ANOVA, the collection of 13 studies did not perform poorly at all.

Re:Not bad at all

[Anonymous Coward]

The fact that people are trying to reproduce the experiments is good news in and of itself.

Re:Not bad at all

[bondsbw]

If the scientific community valued reproducibility as much as original work, we would solve 2 problems:

1) Science without confirmation can lead us astray for years.
2) There are plenty of scientists who a great at experimentation but lousy at coming up with new ideas, and these scientists (or potential scientists) may not be finding their full potential.

And while we're at it, let's value failed experiments as much as successful experiments.

Re:Not bad at all

[phantomfive]

No, unfortunately, because they didn't choose the studies to reproduce randomly. FTA:

[The studies chosen for reproduction] included classic results from economics Nobel laureate and psychologist Daniel Kahneman at Princeton University in New Jersey

At least some of these were fairly important research, which ideally would have been verified more than once. That there was any doubt that they would be reproducible is worrisome in itself.

Re:Not bad at all

[toQDuj]

This is very important, as they did not randomly pick studies but rather chose the ones they "Deemed Worthy". As they did not want to be proven bad scientists (I assume), their conscious or unconscious bias will have been towards sound or easy studies.

Re:Not bad at all

[noobermin]

Did you read TFA? Or did you choose sentences to read randomly? Those we're quoted as the results that worked. In fact, here is the original paragraph:

Ten of the effects were consistently replicated across different samples. These included classic results from economics Nobel laureate and psychologist Daniel Kahneman at Princeton University in New Jersey, such as gain-versus-loss framing, in which people are more prepared to take risks to avoid losses, rather than make gains1; and anchoring, an effect in which the first piece of information a person receives can introduce bias to later decisions2. The team even showed that anchoring is substantially more powerful than Kahneman’s original study suggested.

Two that didn't were about social priming, one was currency priming, in which participants supported what I assume is the current state of capitalism after seeing money, and the other, priming feelings of patriotism with a flag. Moreover, both original authors we're positive about it:

Social psychologist Travis Carter of Colby College in Waterville, Maine, who led the original flag-priming study, says that he is disappointed but trusts Nosek’s team wholeheartedly, although he wants to review their data before commenting further. Behavioural scientist Eugene Caruso at the University of Chicago in Illinois, who led the original currency-priming study, says, “We should use this lack of replication to update our beliefs about the reliability and generalizability of this effect”, given the “vastly larger and more diverse sample” of the Many Labs project. Both researchers praised the initiative.

There you go, quoting the article directly since you can't be bothered to read it. It is true that they apparently chose what some consider to be important effects and the evidence against social priming is upsetting to some. Still, the fact that verification actually happened and people are happy about it shows science is alive and kicking.

Anyway, another cool thing about this study should be that it uses this thing, the open science framework which I haven't heard about until today, but seems pretty cool.

Science has a reproducibility problem?

[TheloniousToady]

I'll believe it when I see it.

The problem isn't necessarily reproducibility

[Dahamma]

The "problem" with experiments that aren't reproducible may not be with the experiments as much as with the popular media that decides to make sweeping generalizations based on one result. Though I guess some blame definitely needs to be applied to the researcher who allows unverified results to be misrepresented to get that 15 minutes of fame in a quote in The Guardian or USA Today...

Re:The problem isn't necessarily reproducibility

[Impy the Impiuos Imp]

The whole point of an experiment is to run enough trials to gain statistical confidence. It's supposed to be it's own validation in that sense.

So it's either a systematic error in their experiment, or fraud.

Re:The problem isn't necessarily reproducibility

[_Shad0w_]

If you're going to pick a paper then The Guardian was not a good choice, given Dr Ben Goldacre writes a regular column for the called "Bad Science" where he critiques terrible science reporting in the media (amongst other things).

I read the headline...

[trudyscousin]

...and thought, "Now there's a ray of sunshine for Slashdotters."

As a group, that is.

Re:A quick question

[dabadab]

And recently, the The New England Journal of Medicine reported depression meds have no effect.

That's patently untrue. The Huffingtonpost article you link to is wildly inaccurate, self-contradictory and much more about sensationalism than the actual NEJoM article.

Re:A quick question

[steelfood]

Medication belongs to the field of psychiatry. And for the most part, they do have an effect. But it's only temporary, and the human body gets used to it after a while. So in the long term, medicine is largely useless, and in fact is counterproductive, as they tend to cause other, worse effects ("side" effects). But in the short term, it helps.

All systems have a state of equilibrium, a state of stability. The same holds for the body and the mind, two different but related and dependent systems. They'll always tend towards the state of equilibrium because that's the path of least resistance.

Psychological ills are not the equilibrium being tipped, but the point of equilibrium itself changing. To truly "cure" someone of depression or OCD or bipolar, you have to change the point of equilibrium itself. That's much, much harder than you can imagine, and a far greater challenge than any pill will ever resolve. Those whose equilibrium were changed by an event in their life are easier to change back than those who are born with a certain equilibrium. Some people call the former nature vs. nurture. I call it, again, the past of least resistance.

Psychology is not attempting to medicate everyone. It's attempting to explain in terms familiar to the scientific-minded humanity.