Slashdot Mirror
Most Science Studies Tainted by Sloppy Analysis
mlimber writes "The Wall Street Journal has a sobering piece describing the research of medical scholar John Ioannidis, who showed that in many peer-reviewed research papers 'most published research findings are wrong.' The article continues: 'These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. [...] To root out mistakes, scientists rely on each other to be vigilant. Even so, findings too rarely are checked by others or independently replicated. Retractions, while more common, are still relatively infrequent. Findings that have been refuted can linger in the scientific literature for years to be cited unwittingly by other researchers, compounding the errors.'"
Furthermore, this epidemiologist primarily studied medically-related publications, and in fact focused mostly on high-profile research that make broad claims, or relied heavily on statistics to support a conclusion. Many research publications at the cell/molecular level do not rely on subtle statistical comparisons to prove a point. This guy is singling out research that is based heavily on correlations (like people with x, y, z are more likely to get a, b, or c diseases). He is only an expert in his own field, and I don't think he is qualified to judge every level of scientific publication, but he certainly doesn't mind the media attention.
"We all know..." What are you basing this on??? As a postdoc, I've committed myself to a massive amount of work and I'm certainly not doing it for pay (which is meager), but a LITTLE amount of respect would be nice. I've published a few studies and it was incredibly hard work to do the kind of careful science that gets published. A small amount of scandals and people like you who swallow any sensationalist piece of news out there really cast things in an unfair light. I encourage you to read more scientific literature and actually try and understand how the scientific process works. Do you really think we live in the kind of technological age as we do in spite of "a good portion of all studies" being "bogus" or "based on nothing"? I find this incredibly insulting.
People need to realise that a lot of those calling themselves scientists are not really scientists at all. They don't apply the scientific method. They massage data regularly. They misapply statistics constantly. They don't subject their theories to falisfiability. They waffle, hand wave, engage in rhetoric, and generally do just about everything except an honest to goodness, old fashioned solid, scientific experiment.
Feynman spotted them over 30 years ago. He called them Cargo Cult Scientists. They put on the appearance of science, but have none of its substance. They give a good performance, like an actor playing a scientists on TV. They wear the clothes, speak the language, seemingly apply the methods. But it's all empty. There's no rigor. There's no insight. There's no real testing going on. It's all just people waving around graphs, and lines, and their qualifications, and formulae they don't understand, to support the theories they want to be true, regardless of whether they are true or not.
It's because in this day and age, you can't be a witchdoctor. You can't appeal to spirits, or gods, or karma, or any of the other philosophical reason thrown up in past ages. We live in "The Age of Reason", and people expect things to be proven to them "scientifically". So all the people who in the past would have risen high by browbeating, appealing to authority and writing great prose, are forced to dress themselves up in white coats and go through the motions of an experiment before they proclaim their great revelations to the world. The experiments however, are just as empty as all the old techniques, and bear only superficial relation to actual science.
Personally, I think it's gotten worse over the last 30 years. The unwillingness of actual scientific communities to challenge the misapplication of their methods by unscientific ones has lead to a dilution of the authority of science as a whole. Under the current regime any half baked psychiatrists can show pictures to 20 undergraduates, record a few squiggles on an MRI, run the numbers through R over and over until he gets what he wants, and proclaim to the world just about whatever he likes, and still be called a scientist! No wonder it's all too easy for the Intelligent Design movement to pose as "real science". Just look at how low the threshold for real science is.
There's only one way to deal with Cargo Cult Scientists. You have to call them out. You have to show how flimsy and false their supposed science really is. You also need to learn all the old rhetorical techniques, because faced with someone who actually knows what they're doing, the Cargo Culter will fall back to very old and time honored methods which enable him to win from a weak or false position. I think the real scientific community owes it to itself to show up these charlatans for what they really are, Con men. If they don't, science will just become more diluted in the long run until the public regards it in the same way it regards homeopathy.
May the Maths Be with you!
After all, studies show that most studies are wrong.
Clever.
The fact is, good science is hard work. In fact, it is damn hard work, requiring not only a supremely keen intellect but a very high tolerance for tedium, great attention to detail, and usually a big fat wad of cash. Also, it requires a profound lack of ego (and the ability to cope with failure and keep trying), given that a trememdous amount of effort could (and frequently does) wind up being completely discounted by a peer-review or another study.
The endeavor of scientific research obviously provides us tremendous benefits, and is furthering the evolution of our species at a blindingly fast rate (depending on how you look at it, of course). It is very important, very hard, and very expensive.
There are many, many people who would like to be scientists but really don't have the brain for it (as I stated above, it isn't just intelligence that matters). Unfortunately, a lot of them wind up doing research anyway, and they cause problems. Hopefully there are enough good scientists with enough funding to clean up their mess.