Slashdot Mirror
Why Published Research Findings Are Often False
Hugh Pickens writes "Jonah Lehrer has an interesting article in the New Yorker reporting that all sorts of well-established, multiply confirmed findings in science have started to look increasingly uncertain as they cannot be replicated. This phenomenon doesn't yet have an official name, but it's occurring across a wide range of fields, from psychology to ecology and in the field of medicine, the phenomenon seems extremely widespread, affecting not only anti-psychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants. 'One of my mentors told me that my real mistake was trying to replicate my work,' says researcher Jonathon Schooler. 'He told me doing that was just setting myself up for disappointment.' For many scientists, the effect is especially troubling because of what it exposes about the scientific process. 'If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved?' writes Lehrer. 'Which results should we believe?' Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to 'put nature to the question' but it now appears that nature often gives us different answers. According to John Ioannidis, author of Why Most Published Research Findings Are False, the main problem is that too many researchers engage in what he calls 'significance chasing,' or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher. 'The scientists are so eager to pass this magical test that they start playing around with the numbers, trying to find anything that seems worthy,'"
That's not a given. Particularly in the soft sciences - psychology, for instance - it is extremely difficult to control for all factors (I'm more inclined to say nearly impossible) and so replication of results can be subsumed by other effects, or even simply not work at all. You know that whole generation gap thing? That's a good example of groups of people who are different enough that the reactions they will have to certain subject matter can be polar opposites. So something that was "definitively determined" in 1960 may be statistically irrelevant among the current generation.
That's just one example of how squishy this all is. Without having to bring lying into it at all. And then, there will be liars; and there will be people who draw conclusions without scientific rigor at all, simply because it's just too difficult, expensive or time-consuming to attempt to confirm the ideas at hand. And there is the outlier personality; the one who accounts for those other few percent -- all the declarations of "this is how it is" are false for them right out of the gate.
Hard sciences simply lend themselves a lot better to repeatability. Where I think we go wrong is assigning the same certainties to the claims of the soft scientists. I have personally seen psychiatrists, best intent not in doubt, completely err in characterizing a situation to the great detriment of the people involved, because the court took the psychiatrist's word as gospel truth.
All science is an exercise in metaphor, but soft science is an exercise of metaphor that is almost always far too flexible. One place you can see this happening is the trendy / cyclic adherence to Froyd, Jung, Maslow, Rogers and so forth... the "correct" way to raise babies... Ferberizing, etc. This stuff isn't generally lies at all, but it also generally isn't "right." Good intentions do not automatically make good science.
Serious medicine is another good example. Something that might work very well for you might not work at all for me; get the wrong group of test subjects, and your results will skew or worse. This is an area that I think is fair to call a hard science, but where we just don' t know enough about the systems involved. Generally speaking, I don't think our oncologist lies to us; further, I think he's pretty well aware of the limitations of his practice and the state of knowledge that informs it; but they just don't know enough. To which I hopefully add, "yet."
On a personal level - since that's all I can really affect - I treat soft science about the same way I do astrology. If you believe it, you'll probably attempt to modify your behavior because of the predictions, which in turn may, or may not, affect your actual outcome. If you don't, it's either irrelevant or too uncertain to trust anyway. So it's low confidence all the way.
I do, however, still place very high confidence in Boyle's law for gasses. Hard science works very well. :)
I've fallen off your lawn, and I can't get up.
No, scientists in many fields (and some of which you would expect the opposite) do not understand statistics well.
If you dig through your well gathered data you will find correlations that are purely chance. Which is why you are supposed to be looking for the predetermined correlation not just any correlation. But when you've spend a lot of time and effort gathering a set of data, digging into it to find other things seems like a reasonable plan - and as long as you do another completely separate data gathering study to check what you find it is (but there's a great pressure to publish something now since you just spent a huge wad of cash and your performance is measured by what you publish not by actual scientific progress).
Scientists do this. Traders at investment banks (and elsewhere) do this. People just do this.
"Fooled by Randomness" by Taleb is a good look into this from the trading perspective. Assuming you don't mind his writing style, "ego-centric and pompous" is a common description (though I don't find it so).
I'm pretty sure investment banking is dominated by "rightoids" which nullifies your ridiculous injection of politics into the universal human bias to see patterns in randomness.
>>so you cannot hunt for a statistical significance just somewhere in the data and then re-formulate your hypothesis
Cannot? Or should not?
I work as an external evaluator on federal projects, and have been told by one group I worked with, after I delivered a negative result on their data, that "we know that the stats can say anything - why don't you take another look at the stats and find something that makes us look better?" I refused, saying it would be dishonest to change the analysis. They fired me, saying "most evaluators make us look better than the data, but you're making us look worse."
The entire point of an external evaluator is to have a third party looking at your data, so as to prevent this kind of analysis fudging, but when I reported it to the federal case officer overseeing the grant, they just shrugged and didn't care. They don't want any drama to crop up in the grants they oversee. Makes them look bad to *their* bosses.