Most Science Studies Tainted by Sloppy Analysis

mlimber writes "The Wall Street Journal has a sobering piece describing the research of medical scholar John Ioannidis, who showed that in many peer-reviewed research papers 'most published research findings are wrong.' The article continues: 'These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. [...] To root out mistakes, scientists rely on each other to be vigilant. Even so, findings too rarely are checked by others or independently replicated. Retractions, while more common, are still relatively infrequent. Findings that have been refuted can linger in the scientific literature for years to be cited unwittingly by other researchers, compounding the errors.'"

Comments

Medical research vs. basic research

[BWJones]

It should be noted that "medical research" (epidemiology, clinical studies etc...) is very different from basic research (mechanisms, pathways, etc...) and the threshold for acceptance in journals that cover basic research is much higher than that for medical journals. i.e. There is significantly higher oversight and peer review criticism over basic research than there is medical research and the two fields should not be confused.

I came here for the global warming ref

[benhocking]

And I wasn't disappointed. You also managed to please me with your inclusion of the bogus "The Great Global Warming Swindle", "Global Cooling", and conspiracy theories. Excellent!

As an interesting aside, I thought that this argument had been dropped because it was a little too easy to shoot down:

Because the ocean can hold less CO2 when it is warmed.

The interesting thing is that, despite warming temperatures, the oceans are holding more CO2 than before (which lowers their pH level as CO2 + H20 = C2H03, carbonic acid). This is possible because increasing the partial pressure of CO2 in the atmosphere (as we've done significantly) more than counteracts the decreased solubility due to temperature rises. It's possible that in the past this was a factor (although you should read up on those time courses and realize that your 800 year figure is also bogus), but it's clearly not true today. Global warming theories aren't based on correlations, they're based on fundamental principles of science.

Fairly common knowlege

[everphilski]

It is fairly common knowledge that 3 things factor into tenure (in this order): (1) being published (2) bringing funding into the university and (3) teaching.

1. A good number to shoot for is 15 journal articles in your first 6 years. If you don't have tenure in 6 years chances are you are never going to get it. The point of being published is to get the name of the university out.

2. Should be self-explanatory. You need to bring in $$$ to the university. The more you bring, the more profitable you are and the more they need to keep you around. But publishing is still more important.

3. Teaching, while as students we all feel is important, is actually the least important thing towards tenure. A mediocre or even bad teacher who writes papers (that get accepted by excellent journals) at a rapid pace will get tenure where an excellent teacher who can't write for the life of him will not. This is why you often see people from industry teaching. They teach for the love, tenured professors are there for the research and for the higher level teaching (where it is more a relation of facts, not an educational process).

The 'sloppy analysis' referred to is not 'fraud' as you cite. There is a difference between fraud and sloppy analysis. The rush to put out papers (between 2 and three a year, by this guide, for tenure) causes some slop to occur. As a reference, I've been working on a paper with my advisor and a (yet-to-be-tenured) professor for almost a year already, and we are just submitting it to a major journal. And the paper is based mostly off of my thesis work completed a year ago! A good paper and good research takes time. But please, do not mistake sloppy analysis for fraud. Mistakes are one thing, deception entirely another.

SOURCE: Advice to rocket scientists: A Career Survival Guide for Scientists and Engineers. Dr. Jim Longuski, published by the AIAA in 2004. But again, this is fairly common knowlege and can be found anywhere you look. As a postdoc (I am too) I'm suprised you didn't know ...

Devastating the world's economy

[benhocking]

On the other hand, devastating the world's economy in order to reduce (not eliminate) growth in CO2 production will cause millions to die, and everyone else to live at a significantly reduced quality of life.

Most studies, however, suggest that we do not need to devastate the world's economy. In fact, I'm not aware of any studies that suggest we would. Are you?