Slashdot Mirror
Does Google Actually Make Us Dumber? (buzzfeednews.com)
Another spate of high-profile and provocative psychology studies have failed to replicate, dealing blows to the theories that fiction makes readers empathetic, for example, or that the internet makes us dumber. From a report: At a time when psychology researchers are increasingly concerned about the rigor of their field, five laboratories set out to repeat 21 influential studies. Experiments in just 13 of those papers -- or 62% -- held up, according to an analysis published Monday. The eight papers that did not fully replicate -- seven in Science, one in Nature -- have been cited hundreds of times in scientific literature and many were widely covered by the media.
Failing to replicate isn't definitive proof that a finding is false, particularly in cases where other studies support the same general idea. And some scientists told BuzzFeed News they do not agree with how the replications were done. Still, the new findings are part of an overwhelming, and troubling, trend. The so-called reproducibility crisis has hit research in many fields of science, from artificial intelligence to cancer. Shoddy psychology research has received the most attention, with a 2015 report replicating just 36% of 97 studies. It makes sense that scientists want to publish data that is surprising or counterintuitive. "That's not a bad thing in science, because that's how science breaks boundaries," said Brian Nosek, a University of Virginia psychologist and executive director of the Center for Open Science, which led the replication project. But too few scientists, he said, recognize the inherent uncertainty of their splashy results. "It's okay if some of those turn out to be wrong," he said.
Failing to replicate isn't definitive proof that a finding is false, particularly in cases where other studies support the same general idea. And some scientists told BuzzFeed News they do not agree with how the replications were done. Still, the new findings are part of an overwhelming, and troubling, trend. The so-called reproducibility crisis has hit research in many fields of science, from artificial intelligence to cancer. Shoddy psychology research has received the most attention, with a 2015 report replicating just 36% of 97 studies. It makes sense that scientists want to publish data that is surprising or counterintuitive. "That's not a bad thing in science, because that's how science breaks boundaries," said Brian Nosek, a University of Virginia psychologist and executive director of the Center for Open Science, which led the replication project. But too few scientists, he said, recognize the inherent uncertainty of their splashy results. "It's okay if some of those turn out to be wrong," he said.
The crisis isn't really in research: it's neither surprising nor problematic that most experimental and theoretical results reported in the literature have problems, are hard/impossible to replicate, or contain outright errors. That's a normal and healthy part of science.
The actual crisis is in the rush of modern society has to use scientific results as the basis of decision making. You should usually only consider scientific results to be "established" or "true" after several generations of scientists have passed and after they have been reproduced many times by many different approaches. If you use scientific results earlier than that, it's a big gamble.
This is going to be hard to suggest, but these results are actually good news. I used to be in the field of psychology research, so I know all the dirt on how these studies work and the techniques researchers have to do to get published. The fact that over 60% of the findings replicated is very surprising, for a lot of reasons.
1. Top journals publish flashy research. Rather than doing technical research on the mechanisms of empathy, a study showing "fiction reading increases empathy" is more likely to be published in Science, and for a grad student, get you a grant/job/life. Building up more incremental research with stronger theoretical foundations is a lot harder to get published.
2. There's a lot of competition. People want grants/jobs/life and so they have to publish. But anyone can steal your ideas. One of my greatest ideas was stolen as a grad student, which is why I pretty much left the field. It's easy for someone with a bigger lab to do the same study, find similar results, and then publish it. So you have to get your ideas out quickly and you really can't share many of the flashy ones, because this could happen. So this creates a perverse incentive not to replicate the most flashy findings, because they are the ones that give you all the glory.
3. People make it very hard to replicate their research, because once someone gets a program of research, that is their gravy train. If I have a great finding that gets me a great paper and a top-level job, I need to keep up that research to get tenure. All those replication studies are necessary for my job, so again there's an incentive not to replicate other labs' research.
4. The sample sizes are way too low for the type of research being done, which is why false positives are very likely. fMRI research uses sample sizes that are 20-40. My dissertation used 30 participants, because it was all I could afford. To make great generalizable conclusions, I'd probably need 100 or more. So should I stop doing research? No, I should publish it and others should replicate it, or not. By replicating it in slightly different conditions, it increases the external validity of the findings.
So long story short, 60% replication is a good number, and should be how science goes. It's not the researchers' fault or the fault of psychology as a science, since studying complex systems is very hard. It's just the fault of a broken system that equates a published paper with truth, rather than one piece of evidence.