Slashdot Mirror
Psychologists Strike a Blow For Reproducibility
ananyo writes "Science has a much publicized reproducibility problem. Many experiments seem to be failing a key test of science — that they can be independently verified by another lab. But now 36 research groups have struck a blow for reproducibility, by successfully reproducing the results of 10 out of 13 past experiments in psychology. Even so, the Many Labs Replication Project found that the outcome of one experiment was only weakly supported and they could not replicate two of the experiments at all."
We ought to be encouraging this sort of thing.
If only Psychology was a science.
So 2 or 3 out of 13 were not reproduced in these attempts. I imagine the standard was "P 0.05." If you then consider ANOVA, the collection of 13 studies did not perform poorly at all.
Scientists must have *more sex* if they are to reproduce... :P
I do not fail; I succeed at finding out what does not work.
I'll believe it when I see it.
I've been barking up that tree for years.
The "problem" with experiments that aren't reproducible may not be with the experiments as much as with the popular media that decides to make sweeping generalizations based on one result. Though I guess some blame definitely needs to be applied to the researcher who allows unverified results to be misrepresented to get that 15 minutes of fame in a quote in The Guardian or USA Today...
...and thought, "Now there's a ray of sunshine for Slashdotters."
As a group, that is.
Those who can, do. Those who can't, write technology blogs.
I thought at first it was saying 36 groups each tried to reproduce the results of 13 experiments, and all 36 were successful with 10 of 13 (though not necessarily the same ten), successfully reproducing the results of a reproducibility meta-experiment.
I also agree with your opinion
Okay, so... I'm going to preface this with: I have a degree in Economics (blood type hater) and support many heterodox ideologies as well as much of modern microeconomic theory. I'm a strange duck.
So while I firmly believe psychology is not a science and that's because, I believe, often it's not viewed in a mathematical context. This weakness is generally rectified by behavioral economists which synthesize the results into a more "probable" (I.e. Being able to compute a probability); which, alludes and often precludes a truth (however you want interpret "truth").
In the end it all ends up in the realm of biologists, and neurologists, who discover the true nature (causality, not correlation) that will solve the symptom at hand.
I say symptom because I believe if we better understood how to understand this (the root) these symptoms would fade.
The original model held that psychotherapy could cure depression. Talk to your analyst once a week and after years of treatment you got better.
Then it was discovered that low norepinephrine caused depression, and tricyclics fixed that and cured depression.
Then it was discovered that low serotonin caused depression, and SSRIs fixed that and cured depression.
Then it was discovered that low dopamine caused depression, and MAOIs fixed that and cured depression.
And recently, the The New England Journal of Medicine reported depression meds have no effect.
One last question... just one*.
Is psychology evidence-driven, or belief-driven?
(*) This isn't just me asking. Here's a quote from the The New England Journal of Medicine article:
Evidence-based medicine is valuable to the extent that the evidence base is complete and unbiased. Selective publication of clinical trials — and the outcomes within those trials — can lead to unrealistic estimates of drug effectiveness and alter the apparent risk–benefit ratio.
(**) Also, I have no meaningful training in science or statistics. If you want, you can win the argument by pointing this out in your response.
So...they couldn't reproduce the reproducibility problem...?
blindly antisocialist = antisocial
Were they not able to reproduce the outcome of an experiment, or were they not able to reproduce the whole experiment (as in "We assume that during a total eclipse in the month of may" or... "to reproduce this, take any old Large Hadron Collider lying around...")
bickerdyke
Is this news? The Decline Effect has been known about (and generally ignored) for years.
Some scientists should not be allowed to reproduce. The problem will take care of itself.
But isn't really able to hold on to much of the scientific method because there's so much we don't know.
Alchemy isn't a science, but it led to chemistry which was.
Astrology isn't a science, but it led to astrophysics which was.
Because the methods Alchemy and Astrology used to get BETTER predictions was the nascent scientific method and keeping what worked and ejecting what didn't was part of that method (which a lot of social science doesn't do, worse luck). And the remains were the germs of the science that later came forward to replace it.
reproducibility is the ability to be reproduceable. It may not be *reproduced*, but it is reproducible. Contrasted that with a lot of result in psychology being ad-hoc, and having no reproducibility, in other word, you cannot ever reproduce the results no matter how you would try.
The article you linked does not say anything about reproducibility, it only state experiment were not reproduced. The problem of psychology is that it has a lot of stuff which has zero reproducibility !
Good post Woodhams, I'll use an analogy I formed when discussing Psychology with my girlfriend whose been in the field a while: Psychology today is like studying Chemistry in the bronze age. Back then, they didn't have the means to understand the why of this chemical working with this chemical, they just knew it worked and did Chemistry via trial and error and guessing. Today, psychology is classifying things based on relations and forming best practices, but we don't understand why things are the way they are because of our limited understanding of the brain.
Maybe things will change in 100 years, maybe not. I think the field is worth its weight in gold though, there's a lot of good that can be/is being done and a lot of progress still to be made.
I worked in a bio tech lab for a while. Optimizing reactions is a lengthy and expensive process. Because of that, scientists like to keep at least part of the process secret and not let their competitors know all the minute dteaila required. Papers very rarely conatain all of the nitty grtitty tricks required to reproduce the result.
I have a particular experiment which, in my mind, highlights an aspect of human nature we would all prefer to deny. We all have within us the capacity to ruthlessly abuse others, even and especially friends and family when given the opportunity. This famous experiment a group of peers where some were assigned the role of prison guard and others that of prisoner. It really didn't take long before things went really bad.
I have often heard that corruption is a problem of opportunity more than of character. I believe it is generally true. I believe this has been common knowledge for centuries if not thousands of years. I believe the reason the US constitution was written as it is was to delay if not entirely prevent certain inevitable human behaviors. Our very nature as humans is our undoing. What enables us to survive beyond our primitive selves is our recognition and understanding of our natures and to inhibit and limit those aspects which are the most harmful.
"The goal of a good society is to structure social relations and institutions so that cooperative and generous impulses are rewarded, while antisocial ones are discouraged. The problem with capitalism is that it best rewards the worst part of us: ruthless, competitive, conniving, opportunistic, acquisitive drives, giving little reward and often much punishment to honesty, compassion, fair play, many forms of hard work, love of justice, and a concern for those in need...the enormous gap between what US leaders do in the world and what Americans think their leaders are doing is one of the great propaganda accomplishments of the dominant political mythology ...the real danger we face is not from terrorism, but what is being done under the pretext of fighting it." ~ Michael Parenti
I am not a socialist or a communist. I believe we should all be allowed to seek our fortunes and to build lives of our choosing. However, we can only get to a certain point in development before we start tearing each other down. We have exceeded that point. We need to return to some middle ground.
"Science has a much publicized reproducibility problem" links to an article that says "as many as 17â"25% of such findings are probably false", but a group reproducing 10 out of 13 experiments (23% not replicable) is striking a blow for reproducibility?
Science does not a "reproducibility problem" ... it's shoddy voodoo perpetrated in a lab environment that can't withstand scrutiny.
Reproducing 10 out of 13 experiments is really quite good.
I'm a materials physicist, and if I could reproduce that ratio of experiments in my field, I would be very happy and a little bit surprised.
This problem of non-reproducible results in science is due to poor training, poor writing, poor experiment design and a direct link between citations and funding.
Psychology is a soft science because of the numerous variables that in studies are often simplified into a constant often for simplicity's sake and nothing else. Economics and politics are the same, mostly because they're based on psychology.
It's an inexact science because the human condition is imperfect. As opposed to the hard sciences, which are exact, because the universe around us is "perfect". And then, there's computer science, which is a mathematical, computational science that's absolute. It's not even "perfect" anymore; it's exactly what the maths say it is, and any failure sits between keyboard and chair.
Anyway, psychology is important, because the only way to truly understand the imperfect conditions of humans is via an inexact science. And it's something only fully understood by humans (computers can simulate the hard sciences to a calculable degree of accuracy, but they'll never be able to simulate the soft sciences in the same way), and innately at that.
The way to think about psychology is using fractals. X% | X is > statistical significance, of the population behaves in manner a. X * (100-X)% of the population behaves in manner b. X * (100 - X * (100-X))% of the population behaves in manner c. Etc. a, b, c, etc. are up to you to figure out. And when you change the test, the individual that falls into one category is not guaranteed to fall into the same category again.
Note that the human mind can comprehend infinity (poorly for most, but very possible for a few), both countable and uncountable variants, but a computer will never be able to calculate it. So the fractal analogy works really, really well.
Ironically, the same things that make psychology a soft science are the same things used by theoretical physicists. Both rely heavily on probability versus observable inputs (although for different reasons). So, I would posit that what makes psychology and the others you mentioned a soft science versus a hard science is not aobut exactitude, but semantics. That and the fact that it is was the hard sciences that created the definition in the first place.
Because that's how "science" works in Psychology.
You come up with some ridiculous simple experiment, like giving people the same amount of money for a simple boring task, or a complex creative task.
Turns out most people go for the simple boring one.
Study Conclusion: People prefer simple boring tasks!
My Conclusions: why take more risk in screwing up when you can make the same money with something easy?
Who says these researchers didn't select experiments they a-priori thought seemed reasonable?