Psychologists Strike a Blow For Reproducibility

ananyo writes "Science has a much publicized reproducibility problem. Many experiments seem to be failing a key test of science — that they can be independently verified by another lab. But now 36 research groups have struck a blow for reproducibility, by successfully reproducing the results of 10 out of 13 past experiments in psychology. Even so, the Many Labs Replication Project found that the outcome of one experiment was only weakly supported and they could not replicate two of the experiments at all."

Comments

Good news, everyone!

[Anonymous Coward]

We ought to be encouraging this sort of thing.

Re:Good news, everyone!

[i kan reed]

Yes, that's a good point. What measures do you think "we," which presumably refers to mild-to-moderately informed laypeople, can put into place to help?

Not trying to be sarcastic, I genuinely wonder what practical options exist.

Re:Good news, everyone!

[Jeremiah Cornelius]

Have you ever BEEN to an APA conference?

My dear god, man! The last thing you want is these people REPRODUCING!

Re:Good news, everyone!

[Jeremiah Cornelius]

The only thing sadder than that joke is the fact that you thought it was worth making.

McLaughlin Group answer: Mods say "WRONG!"

Psychology

[Anonymous Coward]

If only Psychology was a science.

Re:Psychology

[__aaltlg1547]

If the experiments are reproducible, it's science.

Apparently it's biochemistry that is not a science.
http://online.wsj.com/news/articles/SB10001424052970203764804577059841672541590

Re:Psychology

[FatdogHaiku]

If only Psychology was a science.

Lol -- psychiatrists and psychologist doing experiments. It's is a weak science at its best; a modern day priesthood at its worst.

For your heresy and disbelief you have been diagnosed as a paranoid schizophrenic!
We will monitor you to see if medication will be adequate to silence you, I mean, control your symptoms...

Re:Psychology

[Lamps]

The ironic thing about statements like these is that they usually come from people with no scientific training in any field, nor any meaningful training in statistics, but only a "sciency" inclination and questionable, popular distillation-derived knowledge of some principles from what they consider "the hard sciences".

Sadly, this irony will be lost on the people making such statements, who will, for some unfathomable reason, continue to disparage people doing meaningful work in the sciences, while never coming close to accomplishing anything of the sort themselves.

Actual academics have an idea of the hard work involved in contributing to the human knowledge base in all scientific disciplines, and thus, tend to respect each other's work (as long as others don't step on their own toes in their particular area of specialization, in which case, prepare for turbulence).

Re:Psychology

[Anonymous Coward]

How do you feel about political science or economics?

Re:Psychology

[Michael Woodhams]

Psychology is a huge field. Perception, experimental analysis of animal behaviour, clinical psychology, cognitive biases etc. etc. (Note that only one of those involves psychiatrists.) Some bits allow for harder science than other bits.

I personally don't know enough about psychiatry to form a judgement on how scientific they are, but unlike you, at least I know what a psychologist is (or something of the range that they could be.) Your trite dismissal says much about your ignorance and nothing about psychology.

Re:Psychology

[Anonymous Coward]

"popular distillation-derived knowledge "

That dash implies the knowledge comes from a crowd at a bar, is this correct?

Re:Psychology

[antifoidulus]

Unfortunately there are 2 major hurdles that limit all but the most groundbreaking experiments:

1. Money
2. Glory

Money is probably the biggest factor, there just isn't enough money allotted to trying to reproduce experiments. Most budgets only exist for new/continuing research, not verifying experiments done by others. And as the cost of doing experiments rises(more sophisticated equipment necessary, lots of paid "volunteers" etc) this is only going to get worse.

Second, although not as important, is the "glory" factor. Very few talented scientists want to spend their time and research money on reproducing experiments done by others. There aren't a whole lot of publishing opportunities in doing so, esp. if you cannot refute what they have done. You can see this to a certain extent in the open source world as well, for all but the most famous of projects you tend to have a very large # of projects that essentially do the same thing. Why? Because a lot of people want to "invent" a new program rather than improve upon what is there. Fortunately in the OSS world money isn't nearly as much an issue, so you do have large #s of people improving OSS rather than trying to re-invent the wheel.

Re:Psychology

[bakes]

You forgot 'Politics'. Not sure where that fits in the ordering though.

Re:Psychology

[Anonymous Coward]

Angry.

Mostly angry.

Re:Psychology

[Anonymous Coward]

How do you feel

I see what you did there.

Re:Psychology

[rtb61]

Psychology ran in a major hiccup, as many of it's experiments are no longer reproducible not because of bad 'science' but because they are considered naughty and not something that should really be done to people to test out psychological theories, as in http://www.bps.org.uk/what-we-do/ethics-standards/ethics-standards (I used British standards rather than US ones, as the US ones have so badly been mauled by the US government and their fully medically and psychological researched mass torture facility at GITMO that the US ones are rules that 'should be' broken as defined by the US government) and http://mentalfloss.com/article/52787/10-famous-psychological-experiments-could-never-happen-today.

Re:Psychology

[steelfood]

Psychology is a soft science because of the numerous variables that in studies are often simplified into a constant often for simplicity's sake and nothing else. Economics and politics are the same, mostly because they're based on psychology.

It's an inexact science because the human condition is imperfect. As opposed to the hard sciences, which are exact, because the universe around us is "perfect". And then, there's computer science, which is a mathematical, computational science that's absolute. It's not even "perfect" anymore; it's exactly what the maths say it is, and any failure sits between keyboard and chair.

Anyway, psychology is important, because the only way to truly understand the imperfect conditions of humans is via an inexact science. And it's something only fully understood by humans (computers can simulate the hard sciences to a calculable degree of accuracy, but they'll never be able to simulate the soft sciences in the same way), and innately at that.

The way to think about psychology is using fractals. X% | X is > statistical significance, of the population behaves in manner a. X * (100-X)% of the population behaves in manner b. X * (100 - X * (100-X))% of the population behaves in manner c. Etc. a, b, c, etc. are up to you to figure out. And when you change the test, the individual that falls into one category is not guaranteed to fall into the same category again.

Note that the human mind can comprehend infinity (poorly for most, but very possible for a few), both countable and uncountable variants, but a computer will never be able to calculate it. So the fractal analogy works really, really well.

Re:Psychology

[blue trane]

No, create more. Let the Fed fund the govt by buying T-bills. Since the Fed returns the interest to the Treasury, the borrowing cost is zero.

Give Republicans low (or no) taxes, but in return have them eliminate the debt ceiling laws.

Re:Psychology

[Anonymous Coward]

it was is easy to reproduce , they just made it up the same way they made up the the first result. psychology you'd have to be out of your mind to believe it.

Re:Psychology

[VortexCortex]

Psychology is a huge field. Perception, experimental analysis of animal behaviour, clinical psychology, cognitive biases etc. etc.

No, the field you're thinking of is Neuroscience and Cybernetics -- These have evidence based on observation and models which have predictive power. Psychology is just confirmation bias. You must prove the null hypothesis more implausible than the original hypothesis, yet Psychology does not do this. For every ridiculous Sexual Epistemology, there's an equally valid Scatological Epistemology.

The truth is that neurons fire in brains, and that complexity gives rise to emergent behaviours. Leaping the gulf in understanding to arrive at the explanations that Psychology and Philosophy give is akin to claiming a God in a Chariot pulls the Sun across the sky.

Re:Psychology

[VortexCortex]

If only Psychology was a science.

The sad thing is that it's on par with the level of TV Tropes. They first use their pattern matching brains to notice some pattern, then go seek out quantifications for it. That's ridiculous. That's why literally everything is a trope -- even the tropeless story is a trope. The same goes for psychological classifications and categorizations of behaviours. Some psychologists claim to study cognitive bias -- Yet their own confirmation bias has blinded them to the fact that their distinctions themselves were biased to begin with.

The answer is to start at the bottom -- Understand through observation the mechanisms of neurobiology. Model them through cybernetics. Verify them mathematically, experimentally, and link the behaviours to the physical world, instead of guessing from the top down through the fog of their own cognition.

The primary problems humans face is that their minds are too small, and they don't live long enough. They must specialize if they are to become experts in an area of study before their lives run out. Contrast this with the real world, existing for billions of years, timeless on human scales, and the fact that there are no divisions in sciences. The quantum physics of spin giving predictive power to macroscopic magnetism is linked to human behaviour via eddy currents in their electrochemical neural network brains.

The sciences remain separated quite foolishly. Eg: The Philosopher should start with Information Theory and Entropy and Cybernetics and build their epistemology on mathematically provable principals and cybernetic models with entropic parameters set by observations in quantum physics. THEN you they can say what knowing means. Instead they get continually tripped up by one aspect of a self reflecting cybernetic being: Their data could be tampered with. Yes, indeed, troves are dedicated to the simple subject, and still they have not done as I propose and utilize information theory and observations of thermodynamics to quantify the apparent degree of which knowing can exist. Fools.

One must make due with the planet one finds oneself on. Taking cognitive limits into concern they need not specialize deeply in each field, but no Psychiatrist or Philosopher or Psychologist should be ignorant of the governing field of Cybernetics -- Especially not the cybernetically provable social dynamics that result from information disparity: The Mathematics of Secrets. I would venture to say they should be able to explain the uncertainty principal as well -- since it can be directly applied to perception.

Re:Psychology

[rvw]

The ironic thing about statements like these is that they usually come from people with no scientific training in any field, nor any meaningful training in statistics, but only a "sciency" inclination and questionable, popular distillation-derived knowledge of some principles from what they consider "the hard sciences".

Could you please show me a reasonable experiment with proper statistics supporting this claim?

Well to be honest - we have to confess something to you. Slashdot was once setup as a psychological experiment.... Just take a look at the users and comments, no more proof needed!

Re:Psychology

[rtaylor]

Science is a process, not a field of study or a result.

That process can be applied to anything where you want to find a fact.

Re:Psychology

[LearningHard]

That is a very disingenuous statement. While I would agree that there are many aspects of Psychology/Psychiatry that are not very scientific, there are some areas that are rigorous. Neuropsychology can be a good example. Testing and measurement is another good example. There is a lot more to Psychology than the hokey therapy that we think of.

Re:Psychology

[wisnoskij]

How is this rated Funny? Psychology is not a Science.

Re:Psychology

[khallow]

Even if you reduce it by an order of magnitude, that's still $60 trillion.

And if you reduce it by two orders of magnitude, that's still $6 trillion.

There's no shortage of money. The only scarcity is the lack of political will to fund basic social services and science. There is no economic or physical necessity preventing us from funding food stamps, science research, health care, a basic income.

I agree that's there's no "shortage of money". But these things which you mention do not have that much value to them. For example, most people earn enough to pay for their own food, health care, and still have a basic income.

Also, efforts to insure that these services are available in turn harms the providing of these services. A lot of effort has been made to turn corn inefficiently into a gasoline additive. The motives are mostly selfish IMHO, but the public justification is protection of the food supply.

And a "basic income" is notorious for making a lot of people unemployable since they aren't worth hiring at the resulting mandated minimum wage, especially with all the other service taxes imposed. And a bunch of the unemployable people don't have incentive to improve since they can get a guaranteed income from the state without much effort.

Finally, it's worth noting that in the US, science remains mostly funded by the private world. My view is that government is only notable in funding science that has poor value to it (for example, big science items like fusion research or the International Space Station). And the FDA probably is the single biggest obstacle to science (via the very expensive regulation and obstruction of medical technologies) in the world today.

The reason there is a shortage of political will is because there is a shortage of value from the things you mention. And given that proposed solutions to these issues have a tendency to either make the problem worse or make it a problem in the first place, I oppose most such efforts.

Re:Psychology

[Russ1642]

And there you have it. Getting chemistry to work in the lab is hard enough. Experimental psychology results would never stand up to the same scrutiny.

Re:Psychology

[RabidReindeer]

Without a doubt, there is a LOT of unscientificness going on in the field of psychology. Look at the satanic ritual abuse situation of the 80s for an obvious example, especially when you realize that even today there are still psychologists treating patients for this 'malady.'

Another example is the DSM, which is a joke from a scientific perspective (though this is perhaps an indication of the difficulty of the field of psychology as much as anything).

Psychology is like quantum physics, except that not only does observing something change the behavior, but the degree and kind of change is likewise unpredictable. Isaac Asimov understood that decades ago. "Psycohistory" depended on A) large populations, to smooth out personal variances and B) keeping most of the mechanism out of sight so that people wouldn't factor the predicted results into their behavior.

A lot of what's wrong with mental health in general is that we're still chipping flints. As new studies come out and new medications are developed, we progress, but there's a LOT of work before it becomes a cookbook procedure. In the mean time, psychology experiences fads where each new methodology is applied indiscriminately to as many people as possible like the small child with a hammer.

Rather like the trail of tarnished Silver Bullets that we see in the more "scientific" art of software development.

Re:Psychology

[K. S. Kyosuke]

For your heresy and disbelief you have been diagnosed as a paranoid schizophrenic!

Go straight to asylum; don't pass go, don't receive telepathic messages.

Re:Psychology

[phantomfive]

Oh yeah, I'm sure bringing science fiction into psychology will fix everything (sarcasm).

Re:Psychology

[Dcnjoe60]

Psychology is a huge field. Perception, experimental analysis of animal behaviour, clinical psychology, cognitive biases etc. etc.

No, the field you're thinking of is Neuroscience and Cybernetics -- These have evidence based on observation and models which have predictive power. Psychology is just confirmation bias. You must prove the null hypothesis more implausible than the original hypothesis, yet Psychology does not do this. For every ridiculous Sexual Epistemology, there's an equally valid Scatological Epistemology.

The truth is that neurons fire in brains, and that complexity gives rise to emergent behaviours. Leaping the gulf in understanding to arrive at the explanations that Psychology and Philosophy give is akin to claiming a God in a Chariot pulls the Sun across the sky.

Your argument is only valid for recent times. For most of the history of modern psychology, there was not a separate field between neuroscience and psychology. Often in med school today, the psych departments and the neurology departments have been combined because we have found that the two are interrelated.

Since most psychological research deals with evidence based observation and models which have predictive power, what distinguishes it from the subclassification of neuroscience? It's a little bit like saying that nuclear physics is science but theoretical physics is not (particularly the whole observational part). I think the flaw in your position is that you feel that science has at it's root the desire to prove the null hypothesis as more implausible than the original hypothesis. In the hard sciences, such as chemistry and physics, that type of test would be meaningless. Just because Newton's theory of gravity has been uprooted by quantum theory, does that mean Newtonian Physics isn't science? No, of course not. That's because most science is about proving what is, not what is not.

If the measure of science is the ability to prove the null hypothesis being more implausible than the original or non-null hypothesis, then theology is a science because it is a lot more difficult to prove that a deity does not exist than it is to prove one actually does.

Re:Psychology

[mcgrew]

is to

science as a virgin is to a hooker.

(I assume you were looking for a punch line and didn't misspell "too". How could anyone misspell a three letter word?)

Not bad at all

[TwineLogic]

So 2 or 3 out of 13 were not reproduced in these attempts. I imagine the standard was "P 0.05." If you then consider ANOVA, the collection of 13 studies did not perform poorly at all.

Re:Not bad at all

[Anonymous Coward]

The fact that people are trying to reproduce the experiments is good news in and of itself.

Re:Not bad at all

[sandytaru]

I agree. Science goes through the upgrade of hypothesis, to tested results, to verified results, to working theories, and eventually laws (although the line between the last two is arbitrary in modern science, really.) The more results that are tested again and again, the better science is as a whole.

Re:Not bad at all

[bondsbw]

If the scientific community valued reproducibility as much as original work, we would solve 2 problems:

1) Science without confirmation can lead us astray for years.
2) There are plenty of scientists who a great at experimentation but lousy at coming up with new ideas, and these scientists (or potential scientists) may not be finding their full potential.

And while we're at it, let's value failed experiments as much as successful experiments.

Re:Not bad at all

[phantomfive]

No, unfortunately, because they didn't choose the studies to reproduce randomly. FTA:

[The studies chosen for reproduction] included classic results from economics Nobel laureate and psychologist Daniel Kahneman at Princeton University in New Jersey

At least some of these were fairly important research, which ideally would have been verified more than once. That there was any doubt that they would be reproducible is worrisome in itself.

Re:Not bad at all

[fatphil]

That 20% failed to be reproduceable is just as worrying.,

Re:Not bad at all

[toQDuj]

This is very important, as they did not randomly pick studies but rather chose the ones they "Deemed Worthy". As they did not want to be proven bad scientists (I assume), their conscious or unconscious bias will have been towards sound or easy studies.

Re:Not bad at all

[drinkypoo]

1) Science without confirmation can lead us astray for years.

It's not science if it's not tested. It's just making assumptions.

It's also not science if you just take their word for it when they said they tested it. It might have been originally, but not any more.

Re:Not bad at all

[noobermin]

Did you read TFA? Or did you choose sentences to read randomly? Those we're quoted as the results that worked. In fact, here is the original paragraph:

Ten of the effects were consistently replicated across different samples. These included classic results from economics Nobel laureate and psychologist Daniel Kahneman at Princeton University in New Jersey, such as gain-versus-loss framing, in which people are more prepared to take risks to avoid losses, rather than make gains1; and anchoring, an effect in which the first piece of information a person receives can introduce bias to later decisions2. The team even showed that anchoring is substantially more powerful than Kahneman’s original study suggested.

Two that didn't were about social priming, one was currency priming, in which participants supported what I assume is the current state of capitalism after seeing money, and the other, priming feelings of patriotism with a flag. Moreover, both original authors we're positive about it:

Social psychologist Travis Carter of Colby College in Waterville, Maine, who led the original flag-priming study, says that he is disappointed but trusts Nosek’s team wholeheartedly, although he wants to review their data before commenting further. Behavioural scientist Eugene Caruso at the University of Chicago in Illinois, who led the original currency-priming study, says, “We should use this lack of replication to update our beliefs about the reliability and generalizability of this effect”, given the “vastly larger and more diverse sample” of the Many Labs project. Both researchers praised the initiative.

There you go, quoting the article directly since you can't be bothered to read it. It is true that they apparently chose what some consider to be important effects and the evidence against social priming is upsetting to some. Still, the fact that verification actually happened and people are happy about it shows science is alive and kicking.

Anyway, another cool thing about this study should be that it uses this thing, the open science framework which I haven't heard about until today, but seems pretty cool.

Re:Not bad at all

[phantomfive]

Work on your reading comprehension, man. You failed to show any understanding of the post you responded to.

Re:Not bad at all

[Trogre]

Okay, but don't confuse a failed experiment with a successful experiment that returned negative results.

Clearly

[msobkow]

Scientists must have *more sex* if they are to reproduce... :P

Science has a reproducibility problem?

[TheloniousToady]

I'll believe it when I see it.

Re:Science has a reproducibility problem?

[Anonymous Coward]

And then see it again.

It's hopeless.

[Confusedent]

I've been barking up that tree for years.

The problem isn't necessarily reproducibility

[Dahamma]

The "problem" with experiments that aren't reproducible may not be with the experiments as much as with the popular media that decides to make sweeping generalizations based on one result. Though I guess some blame definitely needs to be applied to the researcher who allows unverified results to be misrepresented to get that 15 minutes of fame in a quote in The Guardian or USA Today...

Re:The problem isn't necessarily reproducibility

[Impy the Impiuos Imp]

The whole point of an experiment is to run enough trials to gain statistical confidence. It's supposed to be it's own validation in that sense.

So it's either a systematic error in their experiment, or fraud.

Re:The problem isn't necessarily reproducibility

[Lamps]

That's not really a problem from the perspective of scientists - in the fields of psychology, cog sci, and neuroscience, I've never encountered an instance of a researcher using any popular media distillation of some study as a meaningful source of info on that study (aside from making them aware of the study's existence).

Also, you seem to be assigning some a priori status of reproducibility or lack thereof to some studies, which really confuses the issue. For example, what does it mean for an experiment to be "not reproducible"? You can fail to reproduce the results of an experiment, but proving that a result is not reproducible, or somehow knowing that it isn't, is a different issue altogether.

Re:The problem isn't necessarily reproducibility

[_Shad0w_]

If you're going to pick a paper then The Guardian was not a good choice, given Dr Ben Goldacre writes a regular column for the called "Bad Science" where he critiques terrible science reporting in the media (amongst other things).

Re:The problem isn't necessarily reproducibility

[Oligonicella]

The whole point of karking on a single misplaced apostrophe is to elevate yourself with pomposity.

Re:The problem isn't necessarily reproducibility

[Oligonicella]

For example, what does it mean for an experiment to be "not reproducible"?

It means that, as a scientist, you have failed to produce a worthwhile experiment. Hypothesis is first. A road map for reproducibility is next. If said map does not function, you have failed.

Re:The problem isn't necessarily reproducibility

[Lamps]

I guess I should clarify the larger epistemic point at which I was hinting. That others may not, in some reasonable number of attempts, reproduce an experiment does not mean that the experiment is categorically not reproducible. Any number of things, such as lab conditions (which are not, in practical, absolute terms, reproducible within a lab, much less between different labs), can influence the results of experiments, and while adhering to certain sound methodological principles abstracts away a lot of these real-world complications, and while statistics can tell us a certain probability of one thing or another, these are idealizations. There's a distinction between an experiment as an abstraction (i.e. that thing that gets written up in a methods section) and an instance of an experiment (i.e. that thing that gets written up in the results section) - only the latter exists in the tangible world, while the former exists as an idea. Thus, while we can comfortably talk of experiments that haven't been reproduced, it's another thing to casually throw around terms such as "not reproducible". We should be much more cautious about the latter term.

Re:The problem isn't necessarily reproducibility

[femtobyte]

Isn't the ability to reproduce results based on the "idea" in the methods section central to the concept of scientific "reproducibility"? If I claim I applied one set of methods and got a certain result --- but those methods are different from the physical reality of my setup that reasonable adherence to the stated methods will produce a vastly different result --- then I have failed at publishing a "reproducible" experiment.

Example:
"By the method of releasing lead spheres at rest into the air, I have observed that lead spheres have a tendency to rise upward (padhering to methods as written, which need to be sufficiently precise to capture all relevant "lab-specific" conditions which would significantly effect the result?

I think if differences between abstract ideal and physical implementation are sufficient to preclude replication of results, then this renders an experiment "not reproducible," rather than "explaining away" the inability of following experimentalists to produce consistent answers.

Re:The problem isn't necessarily reproducibility

[femtobyte]

Sorry, I have no idea how Slashdot managed to completely mangle my post, cutting out a big chunk in the middle, after previewing and submitting. I don't have the patience to re-write it, so just ignore the garbled mess left after Slashdot's unexpected redaction of a whole middle paragraph.

Re:The problem isn't necessarily reproducibility

[fatphil]

Something passed through our hands here at the office recently. A "scientific" study, in the very soft field of human behaviour, where their sample size was 27. And that set was split into 4 groups. Absolutely any result from that experiment was possible and could be explained as pure random chance, not deviating from the null hypothesis.

Reading works like that, and others from the same authors and institutes, we got the feeling that it was mostly a delusion that they were doing science. Like how when a 4-year-old is given a tub of water, some washing up liquid, and various containers and vessels - there's a delusion that it's actually doing the washing up, and you really don't want to spoil its fun by breaking that illusion - it's not doing any harm, is it?

Re:The problem isn't necessarily reproducibility

[Lamps]

Don't worry about the post getting cut off - the point you were trying to make is clear. To address the idea that "the methods are different from the physical reality of my setup" - it's simply not possible to be comprehensive in the instructions you provide when you write up your methods, and furthermore, in reality, variables are often introduced in a lab which impact experimental results, but which are not accounted for in the methods writeup because the authors are not aware of these variables (this is known to impact the likelihood that results will be reproduced). Adhering to certain methodological principles may mitigate the latter issue, but you can't be sure that the issue has been resolved.

Consider that it's believed that many older studies in psych, if conducted today, would produce non-significant p-values simply for the reason that over time, the population from which participants will be drawn has become different from the population from which participants were drawn for the original study (the changes may be cultural, but may be also be something that we cannot readily pinpoint). Methods sections cannot be comprehensive in addressing all concerns, and don't account for this sort of thing. Are the original studies failed studies? Should the conclusions of the original studies simply be revised to qualify the generalizability of these studies, and if so, at what point can we be reasonably confident that the conclusions of a study are generally applicable? So that's a commonplace scenario illustrating that the notion of being able to reproduce results is more slippery than we may have thought.

Re:The problem isn't necessarily reproducibility

[drinkypoo]

there's a delusion that it's actually doing the washing up, and you really don't want to spoil its fun by breaking that illusion - it's not doing any harm, is it?

The harm comes when they "dry" the dishes with the dish towel, getting it all nasty and dirty, and then put the dishes away in the cupboard still dirty, contaminating the other dishes they come into contact with. There's a direct analogy to be made here.

Re:The problem isn't necessarily reproducibility

[Chris Mattern]

But don't be quick to assume it's fraud, because it's very, very easy to make a systematic error in designing an experiment.

Re:The problem isn't necessarily reproducibility

[Chris Mattern]

For example, what does it mean for an experiment to be "not reproducible"? You can fail to reproduce the results of an experiment, but proving that a result is not reproducible, or somehow knowing that it isn't, is a different issue altogether.

You make absolutely no sense here. If I follow the original researcher's notes and experiment design correctly and cannot obtain his result, the experiment was not reproducible. That's what "not reproducible" *mean*--I couldn't reproduce his results.

Re:The problem isn't necessarily reproducibility

[femtobyte]

No, I still think that, if the methods are insufficiently described (do not sufficiently capture the nuances of the physical experiment) to permit reliable replication of results, then this makes an experiment un-reproducible by definition.

Indeed, allowances must be made for the study of ephemeral phenomena --- people's perceptions in some particular time and place; a comet passing by once --- that preclude actual recreation of the experiment. So, I think there is a "methodological description" requirement for reproducibility which is itself an abstract concept rather than (necessarily) physically enactable. Which is: the methods should be sufficiently well described that another experimenter, with access to the same resources as described in the method, could repeat the results (with allowance for stated statistical fluctuations). This covers the case where the "resources" include "students at Random U. in the year 2009," which may be impossible to acquire again without a time machine. But, the "methods" description ought to be thorough enough to cover all factors that would significantly change the outcome: if you would get a different answer by selecting participants from a different pool, then precise description of how the selection was accomplished is absolutely necessary. Otherwise, the result is useless unreproducible garbage.

So, I agree with your point that actual ability to repeat an experiment performed is not a requirement for reproducibility. I disagree that differences between described method and lab reality "excuse" any experiment from being reproducible: an experiment is "reproducible" only to the extent that any such differences are too small to alter the result (at the level of confidence stated). This may be only an "abstract, in theory" requirement for ephemeral phenomena --- but the methods and result claims need to fully reflect that, not claiming results about "The Psychology of Humans" when all that has been demonstrated is "the psychology of caucasian females in Western Pennsylvania in 1992, self-selected by response to flyers on coffee shop bulletin boards."

Re:The problem isn't necessarily reproducibility

[Dahamma]

But unfortunately not all of his colleagues share his standards for scientific repairing - there have been plenty of horribly reported scientific studies in the Guardian, as well as almost all "popular media" sources that just can't help it...

Re:The problem isn't necessarily reproducibility

[Dahamma]

That's not really a problem from the perspective of scientists

Most people here are not looking it from the perspective of scientists, though, but from the (as the article states) "much publicized" perspective.

For example, what does it mean for an experiment to be "not reproducible"?

Exactly. You should be asking that question about the article, not to me :) In fact I do have a neuroscience background, even if that's not what I am doing these days... but that is pretty much irrelevant to this thread, which was to start a conversation about how the mainstream media tends to latch onto any unverified/reproduced study and report it as the canonical truth. And that sometimes this is in fact the fault of the researcher, when some media pundit picks up on it and they just can't help basking in their work a bit too prematurely...

I read the headline...

[trudyscousin]

...and thought, "Now there's a ray of sunshine for Slashdotters."

As a group, that is.

Re:I read the headline...

[steelfood]

Funny, because there's another study recently published that said technology is killing everyone's sex lives. It's probably old news here, as it's just a downward extrapolation of the extreme case found here.

I thought at first...

[ysth]

I thought at first it was saying 36 groups each tried to reproduce the results of 13 experiments, and all 36 were successful with 10 of 13 (though not necessarily the same ten), successfully reproducing the results of a reproducibility meta-experiment.

A quick question

[Okian Warrior]

The original model held that psychotherapy could cure depression. Talk to your analyst once a week and after years of treatment you got better.

Then it was discovered that low norepinephrine caused depression, and tricyclics fixed that and cured depression.

Then it was discovered that low serotonin caused depression, and SSRIs fixed that and cured depression.

Then it was discovered that low dopamine caused depression, and MAOIs fixed that and cured depression.

And recently, the The New England Journal of Medicine reported depression meds have no effect.

One last question... just one*.

Is psychology evidence-driven, or belief-driven?

(*) This isn't just me asking. Here's a quote from the The New England Journal of Medicine article:

Evidence-based medicine is valuable to the extent that the evidence base is complete and unbiased. Selective publication of clinical trials — and the outcomes within those trials — can lead to unrealistic estimates of drug effectiveness and alter the apparent risk–benefit ratio.

(**) Also, I have no meaningful training in science or statistics. If you want, you can win the argument by pointing this out in your response.

Re:A quick question

[dabadab]

And recently, the The New England Journal of Medicine reported depression meds have no effect.

That's patently untrue. The Huffingtonpost article you link to is wildly inaccurate, self-contradictory and much more about sensationalism than the actual NEJoM article.

Re:A quick question

[steelfood]

Medication belongs to the field of psychiatry. And for the most part, they do have an effect. But it's only temporary, and the human body gets used to it after a while. So in the long term, medicine is largely useless, and in fact is counterproductive, as they tend to cause other, worse effects ("side" effects). But in the short term, it helps.

All systems have a state of equilibrium, a state of stability. The same holds for the body and the mind, two different but related and dependent systems. They'll always tend towards the state of equilibrium because that's the path of least resistance.

Psychological ills are not the equilibrium being tipped, but the point of equilibrium itself changing. To truly "cure" someone of depression or OCD or bipolar, you have to change the point of equilibrium itself. That's much, much harder than you can imagine, and a far greater challenge than any pill will ever resolve. Those whose equilibrium were changed by an event in their life are easier to change back than those who are born with a certain equilibrium. Some people call the former nature vs. nurture. I call it, again, the past of least resistance.

Psychology is not attempting to medicate everyone. It's attempting to explain in terms familiar to the scientific-minded humanity.

Re:A quick question

[maxwell demon]

If about half of the studies showed a positive effect, then it is hardly a proof that there is no effect. It may not be sufficient to show that it has an effect, but it's a clear hint that it might have. To make better statements one would have to have a closer look at the studies in question, because not every study has the same quality. In the extreme case, all of the studies showing a positive effect might be flawed, while those showing no effect might be sound (in which case, the claim that there's no positive effect would be justified). In the opposite extreme case, all the studies showing no effect are flawed, and all the studies showing an effect are fine (in which case the effect would be proven). But even if no single of the studies is flawed, the different studies most likely have different significance levels which have to be used to weight those studies. A highly significant result is more important than less significant results.

Whom to ask? Well, at first, the authors of the study, because they should know best what they wrote. Then, some well-published researchers who already demonstrated that they know their shit.

A doctor is specialized in practising medicine. I'd also not go to an experimental physicist to let me explain anything non-basic about a physics theory, or to a theoretical physicist for an explanation of the finer points of an experiment (or to an engineer for both; he would, however, be the one to ask about the details of some engine, or about engineering practice). Which doesn't mean that there are none of them who could give a good answer. But unless you've got independent evidence that you would, you better ask those who are doing exactly what your question is about.

Re:A quick question

[rvw]

One last question... just one*.

Is psychology evidence-driven, or belief-driven?

(*) This isn't just me asking. Here's a quote from the The New England Journal of Medicine article:

Evidence-based medicine is valuable to the extent that the evidence base is complete and unbiased. Selective publication of clinical trials — and the outcomes within those trials — can lead to unrealistic estimates of drug effectiveness and alter the apparent risk–benefit ratio.

(**) Also, I have no meaningful training in science or statistics. If you want, you can win the argument by pointing this out in your response.

Read this book: Why zebras don't get ulcers. It explains how stress influences our life, and how complex the system works that tries to regulate this. It shows that it all works beautifully, for people living in the wild, but the system is not so good for us.

This book won't give you the solution to depression, but it will show that the body uses many methods to accomplish several things, like redirecting sources when in danger or in rest. There is not one solution - any solution will have side effects, and this goes for the solutions of the body as well.

Re:A quick question

[Lamps]

(**) Also, I have no meaningful training in science or statistics. If you want, you can win the argument by pointing this out in your response.

It's not my intention to get into any arguments or win anything. When I got snarky above, it was to get some people to consider whether they're qualified to disparage the work of others. Anyway, you raise a good question.

Like all sciences, psychology entails a set of beliefs/theories/ideas/models. These constructs should be informed by evidence gathered through a certain methodology. As new evidence is gathered, old models are continually revised or superceded by new ones in an iterative process that's intended to provide a better understanding of things than the understanding we had.

All sciences come up with models that are wrong or grossly oversimplified (thus, the need for scientists to reproduce and build upon others' research). However, the presence of wrongful models in a scientific field shouldn't discredit the field, and doesn't even necessarily mean that the field is headed in the wrong direction, because in principle, we will eventually replace ideas with better ones.

You've presented some findings from the field of psychiatry (which is a medical field, and not typically classed as a subfield of psychology; the medical field has its own set of complications, including pharma companies pushing researchers to pump out products) which (hopefully) show that researchers are continually revising their understanding of depression. This is comparable to, let's say, the various models of an atom that physicists came up with until they came to what we have today.

As far as psychology goes, consider the complexity of the human body, the nervous system, possible environmental inputs to the nervous system, and the potential range of human behaviors. Consider the complexity of the interactions between those systems. That complexity should give some idea of the difficulties intrinsic to the field of psychology.

Re:A quick question

[strikethree]

Some people call the former nature vs. nurture. I call it, again, the past of least resistance.

Freudian slip? ;)

Eh

[sociocapitalist]

So...they couldn't reproduce the reproducibility problem...?

"not been able to reproduce"

[bickerdyke]

Were they not able to reproduce the outcome of an experiment, or were they not able to reproduce the whole experiment (as in "We assume that during a total eclipse in the month of may" or... "to reproduce this, take any old Large Hadron Collider lying around...")

Re:"not been able to reproduce"

[fatphil]

Apparently, there were no "experiments", in the lab/contrivance sense - it was just a questionaire.

Regarding the failures:
"Of the 13 effects under scrutiny in the latest investigation, one was only weakly supported, and two were not replicated at all. Both irreproducible effects involved social priming. In one of these, people had increased their endorsement of a current social system after being exposed to money[3]. In the other, Americans had espoused more-conservative values after seeing a US flag[4]."

[3] Caruso, E. M., Vohs, K. D., Baxter, B. & Waytz, A. J. Exp. Psych: Gen. 142, 301â€"306 (2013).
[url was a redirect loop for me]

[4] Carter, T. J., Ferguson, M. J. & Hassin, R. R. Psych. Sci. 22, 1011â€"1018 (2011).
http://pss.sagepub.com/content/22/8/1011
"""
A Single Exposure to the American Flag Shifts Support Toward Republicanism up to 8 Months Later

      1. Travis J. Carter[1],
      2. Melissa J. Ferguson[2] and
      3. Ran R. Hassin[3]

      1. Center for Decision Research, Booth School of Business, University of Chicago
      2. Department of Psychology, Cornell University
      3. Department of Psychology and The Center for the Study of Rationality, Hebrew University

Abstract

There is scant evidence that incidental cues in the environment significantly alter peopleâ€(TM)s political judgments and behavior in a durable way. We report that a brief exposure to the American flag led to a shift toward Republican beliefs, attitudes, and voting behavior among both Republican and Democratic participants, despite their overwhelming belief that exposure to the flag would not influence their behavior. In Experiment 1, which was conducted online during the 2008 U.S. presidential election, a single exposure to an American flag resulted in a significant increase in participantsâ€(TM) Republican voting intentions, voting behavior, political beliefs, and implicit and explicit attitudes, with some effects lasting 8 months after the exposure to the prime. In Experiment 2, we replicated the findings more than a year into the current Democratic presidential term. These results constitute the first evidence that nonconscious priming effects from exposure to a national flag can bias the citizenry toward one political party and can have considerable durability.
"""

It operates the scientific method

[Anonymous Coward]

But isn't really able to hold on to much of the scientific method because there's so much we don't know.

Alchemy isn't a science, but it led to chemistry which was.

Astrology isn't a science, but it led to astrophysics which was.

Because the methods Alchemy and Astrology used to get BETTER predictions was the nascent scientific method and keeping what worked and ejecting what didn't was part of that method (which a lot of social science doesn't do, worse luck). And the remains were the germs of the science that later came forward to replace it.

An apt analogy

[PPalmgren]

Good post Woodhams, I'll use an analogy I formed when discussing Psychology with my girlfriend whose been in the field a while: Psychology today is like studying Chemistry in the bronze age. Back then, they didn't have the means to understand the why of this chemical working with this chemical, they just knew it worked and did Chemistry via trial and error and guessing. Today, psychology is classifying things based on relations and forming best practices, but we don't understand why things are the way they are because of our limited understanding of the brain.

Maybe things will change in 100 years, maybe not. I think the field is worth its weight in gold though, there's a lot of good that can be/is being done and a lot of progress still to be made.

Re:An apt analogy

[Dcnjoe60]

Good post Woodhams, I'll use an analogy I formed when discussing Psychology with my girlfriend whose been in the field a while: Psychology today is like studying Chemistry in the bronze age. Back then, they didn't have the means to understand the why of this chemical working with this chemical, they just knew it worked and did Chemistry via trial and error and guessing. Today, psychology is classifying things based on relations and forming best practices, but we don't understand why things are the way they are because of our limited understanding of the brain.

Maybe things will change in 100 years, maybe not. I think the field is worth its weight in gold though, there's a lot of good that can be/is being done and a lot of progress still to be made.

That is an extremely narrow view of psychology today and pretty much views it in terms of therapy. Let me ask you this, when Warren Buffet invests in the market using a contrarian strategy, are you stating that there is no underlying science backing him up? I ask, because he and many others seem to be quite successful at it.

Real psychology has a lot more depth than the therapist's couch. Should the determination of what is science be based on if it can fulfill the requirements of the scientific method versus preconceived notions?

Astronomy, what many would call a science, uses probability to tell us there is life on other planets in the galaxy. Psychology, what many are saying is not a science, uses the same probability to predict various behaviors. What's the difference? One has been deemed science and one has not and yet they both use similar tools to come to their results. It seems pretty subjective.

But here is the crux of the matter. The scientific method states that you first ask a question, then pose a hypothesis, then test your hypothesis, analyze your data and draw a conclusion. If a psychology experiment does all of that, how is it not science?

One of my favorite experiments

[erroneus]

I have a particular experiment which, in my mind, highlights an aspect of human nature we would all prefer to deny. We all have within us the capacity to ruthlessly abuse others, even and especially friends and family when given the opportunity. This famous experiment a group of peers where some were assigned the role of prison guard and others that of prisoner. It really didn't take long before things went really bad.

I have often heard that corruption is a problem of opportunity more than of character. I believe it is generally true. I believe this has been common knowledge for centuries if not thousands of years. I believe the reason the US constitution was written as it is was to delay if not entirely prevent certain inevitable human behaviors. Our very nature as humans is our undoing. What enables us to survive beyond our primitive selves is our recognition and understanding of our natures and to inhibit and limit those aspects which are the most harmful.

"The goal of a good society is to structure social relations and institutions so that cooperative and generous impulses are rewarded, while antisocial ones are discouraged. The problem with capitalism is that it best rewards the worst part of us: ruthless, competitive, conniving, opportunistic, acquisitive drives, giving little reward and often much punishment to honesty, compassion, fair play, many forms of hard work, love of justice, and a concern for those in need...the enormous gap between what US leaders do in the world and what Americans think their leaders are doing is one of the great propaganda accomplishments of the dominant political mythology ...the real danger we face is not from terrorism, but what is being done under the pretext of fighting it." ~ Michael Parenti

I am not a socialist or a communist. I believe we should all be allowed to seek our fortunes and to build lives of our choosing. However, we can only get to a certain point in development before we start tearing each other down. We have exceeded that point. We need to return to some middle ground.

Hold on a second

[GrumpySteen]

"Science has a much publicized reproducibility problem" links to an article that says "as many as 17â"25% of such findings are probably false", but a group reproducing 10 out of 13 experiments (23% not replicable) is striking a blow for reproducibility?

this is really great

[Goldsmith]

Reproducing 10 out of 13 experiments is really quite good.

I'm a materials physicist, and if I could reproduce that ratio of experiments in my field, I would be very happy and a little bit surprised.

This problem of non-reproducible results in science is due to poor training, poor writing, poor experiment design and a direct link between citations and funding.

Ironically...

[Dcnjoe60]

Psychology is a soft science because of the numerous variables that in studies are often simplified into a constant often for simplicity's sake and nothing else. Economics and politics are the same, mostly because they're based on psychology.

It's an inexact science because the human condition is imperfect. As opposed to the hard sciences, which are exact, because the universe around us is "perfect". And then, there's computer science, which is a mathematical, computational science that's absolute. It's not even "perfect" anymore; it's exactly what the maths say it is, and any failure sits between keyboard and chair.

Anyway, psychology is important, because the only way to truly understand the imperfect conditions of humans is via an inexact science. And it's something only fully understood by humans (computers can simulate the hard sciences to a calculable degree of accuracy, but they'll never be able to simulate the soft sciences in the same way), and innately at that.

The way to think about psychology is using fractals. X% | X is > statistical significance, of the population behaves in manner a. X * (100-X)% of the population behaves in manner b. X * (100 - X * (100-X))% of the population behaves in manner c. Etc. a, b, c, etc. are up to you to figure out. And when you change the test, the individual that falls into one category is not guaranteed to fall into the same category again.

Note that the human mind can comprehend infinity (poorly for most, but very possible for a few), both countable and uncountable variants, but a computer will never be able to calculate it. So the fractal analogy works really, really well.

Ironically, the same things that make psychology a soft science are the same things used by theoretical physicists. Both rely heavily on probability versus observable inputs (although for different reasons). So, I would posit that what makes psychology and the others you mentioned a soft science versus a hard science is not aobut exactitude, but semantics. That and the fact that it is was the hard sciences that created the definition in the first place.

That's 10/10 if you leave out those 3 bad ones!

[tommeke100]

Because that's how "science" works in Psychology.
You come up with some ridiculous simple experiment, like giving people the same amount of money for a simple boring task, or a complex creative task.
Turns out most people go for the simple boring one.
Study Conclusion: People prefer simple boring tasks!
My Conclusions: why take more risk in screwing up when you can make the same money with something easy?

Who says these researchers didn't select experiments they a-priori thought seemed reasonable?