Psychologists Strike a Blow For Reproducibility

ananyo writes "Science has a much publicized reproducibility problem. Many experiments seem to be failing a key test of science — that they can be independently verified by another lab. But now 36 research groups have struck a blow for reproducibility, by successfully reproducing the results of 10 out of 13 past experiments in psychology. Even so, the Many Labs Replication Project found that the outcome of one experiment was only weakly supported and they could not replicate two of the experiments at all."

Comments

Psychology

[Anonymous Coward]

If only Psychology was a science.

Re:Psychology

[__aaltlg1547]

If the experiments are reproducible, it's science.

Apparently it's biochemistry that is not a science.
http://online.wsj.com/news/articles/SB10001424052970203764804577059841672541590

Re:Psychology

[FatdogHaiku]

If only Psychology was a science.

Lol -- psychiatrists and psychologist doing experiments. It's is a weak science at its best; a modern day priesthood at its worst.

For your heresy and disbelief you have been diagnosed as a paranoid schizophrenic!
We will monitor you to see if medication will be adequate to silence you, I mean, control your symptoms...

Re:Psychology

[Lamps]

The ironic thing about statements like these is that they usually come from people with no scientific training in any field, nor any meaningful training in statistics, but only a "sciency" inclination and questionable, popular distillation-derived knowledge of some principles from what they consider "the hard sciences".

Sadly, this irony will be lost on the people making such statements, who will, for some unfathomable reason, continue to disparage people doing meaningful work in the sciences, while never coming close to accomplishing anything of the sort themselves.

Actual academics have an idea of the hard work involved in contributing to the human knowledge base in all scientific disciplines, and thus, tend to respect each other's work (as long as others don't step on their own toes in their particular area of specialization, in which case, prepare for turbulence).

Re:Psychology

[Michael Woodhams]

Psychology is a huge field. Perception, experimental analysis of animal behaviour, clinical psychology, cognitive biases etc. etc. (Note that only one of those involves psychiatrists.) Some bits allow for harder science than other bits.

I personally don't know enough about psychiatry to form a judgement on how scientific they are, but unlike you, at least I know what a psychologist is (or something of the range that they could be.) Your trite dismissal says much about your ignorance and nothing about psychology.

Re:Psychology

[rtb61]

Psychology ran in a major hiccup, as many of it's experiments are no longer reproducible not because of bad 'science' but because they are considered naughty and not something that should really be done to people to test out psychological theories, as in http://www.bps.org.uk/what-we-do/ethics-standards/ethics-standards (I used British standards rather than US ones, as the US ones have so badly been mauled by the US government and their fully medically and psychological researched mass torture facility at GITMO that the US ones are rules that 'should be' broken as defined by the US government) and http://mentalfloss.com/article/52787/10-famous-psychological-experiments-could-never-happen-today.

Re:Psychology

[steelfood]

Psychology is a soft science because of the numerous variables that in studies are often simplified into a constant often for simplicity's sake and nothing else. Economics and politics are the same, mostly because they're based on psychology.

It's an inexact science because the human condition is imperfect. As opposed to the hard sciences, which are exact, because the universe around us is "perfect". And then, there's computer science, which is a mathematical, computational science that's absolute. It's not even "perfect" anymore; it's exactly what the maths say it is, and any failure sits between keyboard and chair.

Anyway, psychology is important, because the only way to truly understand the imperfect conditions of humans is via an inexact science. And it's something only fully understood by humans (computers can simulate the hard sciences to a calculable degree of accuracy, but they'll never be able to simulate the soft sciences in the same way), and innately at that.

The way to think about psychology is using fractals. X% | X is > statistical significance, of the population behaves in manner a. X * (100-X)% of the population behaves in manner b. X * (100 - X * (100-X))% of the population behaves in manner c. Etc. a, b, c, etc. are up to you to figure out. And when you change the test, the individual that falls into one category is not guaranteed to fall into the same category again.

Note that the human mind can comprehend infinity (poorly for most, but very possible for a few), both countable and uncountable variants, but a computer will never be able to calculate it. So the fractal analogy works really, really well.

Not bad at all

[TwineLogic]

So 2 or 3 out of 13 were not reproduced in these attempts. I imagine the standard was "P 0.05." If you then consider ANOVA, the collection of 13 studies did not perform poorly at all.

Re:Not bad at all

[Anonymous Coward]

The fact that people are trying to reproduce the experiments is good news in and of itself.

Re:Not bad at all

[bondsbw]

If the scientific community valued reproducibility as much as original work, we would solve 2 problems:

1) Science without confirmation can lead us astray for years.
2) There are plenty of scientists who a great at experimentation but lousy at coming up with new ideas, and these scientists (or potential scientists) may not be finding their full potential.

And while we're at it, let's value failed experiments as much as successful experiments.

Re:Not bad at all

[phantomfive]

No, unfortunately, because they didn't choose the studies to reproduce randomly. FTA:

[The studies chosen for reproduction] included classic results from economics Nobel laureate and psychologist Daniel Kahneman at Princeton University in New Jersey

At least some of these were fairly important research, which ideally would have been verified more than once. That there was any doubt that they would be reproducible is worrisome in itself.

Re:Not bad at all

[noobermin]

Did you read TFA? Or did you choose sentences to read randomly? Those we're quoted as the results that worked. In fact, here is the original paragraph:

Ten of the effects were consistently replicated across different samples. These included classic results from economics Nobel laureate and psychologist Daniel Kahneman at Princeton University in New Jersey, such as gain-versus-loss framing, in which people are more prepared to take risks to avoid losses, rather than make gains1; and anchoring, an effect in which the first piece of information a person receives can introduce bias to later decisions2. The team even showed that anchoring is substantially more powerful than Kahneman’s original study suggested.

Two that didn't were about social priming, one was currency priming, in which participants supported what I assume is the current state of capitalism after seeing money, and the other, priming feelings of patriotism with a flag. Moreover, both original authors we're positive about it:

Social psychologist Travis Carter of Colby College in Waterville, Maine, who led the original flag-priming study, says that he is disappointed but trusts Nosek’s team wholeheartedly, although he wants to review their data before commenting further. Behavioural scientist Eugene Caruso at the University of Chicago in Illinois, who led the original currency-priming study, says, “We should use this lack of replication to update our beliefs about the reliability and generalizability of this effect”, given the “vastly larger and more diverse sample” of the Many Labs project. Both researchers praised the initiative.

There you go, quoting the article directly since you can't be bothered to read it. It is true that they apparently chose what some consider to be important effects and the evidence against social priming is upsetting to some. Still, the fact that verification actually happened and people are happy about it shows science is alive and kicking.

Anyway, another cool thing about this study should be that it uses this thing, the open science framework which I haven't heard about until today, but seems pretty cool.

The problem isn't necessarily reproducibility

[Dahamma]

The "problem" with experiments that aren't reproducible may not be with the experiments as much as with the popular media that decides to make sweeping generalizations based on one result. Though I guess some blame definitely needs to be applied to the researcher who allows unverified results to be misrepresented to get that 15 minutes of fame in a quote in The Guardian or USA Today...