Slashdot Mirror
Most Science Studies Tainted by Sloppy Analysis
mlimber writes "The Wall Street Journal has a sobering piece describing the research of medical scholar John Ioannidis, who showed that in many peer-reviewed research papers 'most published research findings are wrong.' The article continues: 'These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. [...] To root out mistakes, scientists rely on each other to be vigilant. Even so, findings too rarely are checked by others or independently replicated. Retractions, while more common, are still relatively infrequent. Findings that have been refuted can linger in the scientific literature for years to be cited unwittingly by other researchers, compounding the errors.'"
It should be noted that "medical research" (epidemiology, clinical studies etc...) is very different from basic research (mechanisms, pathways, etc...) and the threshold for acceptance in journals that cover basic research is much higher than that for medical journals. i.e. There is significantly higher oversight and peer review criticism over basic research than there is medical research and the two fields should not be confused.
Visit Jonesblog and say hello.
Oh, say does that Star-Spangled Banner entwine / The myrtle of Venus with Bacchus's vine?
And I wasn't disappointed. You also managed to please me with your inclusion of the bogus "The Great Global Warming Swindle", "Global Cooling", and conspiracy theories. Excellent!
As an interesting aside, I thought that this argument had been dropped because it was a little too easy to shoot down: The interesting thing is that, despite warming temperatures, the oceans are holding more CO2 than before (which lowers their pH level as CO2 + H20 = C2H03, carbonic acid). This is possible because increasing the partial pressure of CO2 in the atmosphere (as we've done significantly) more than counteracts the decreased solubility due to temperature rises. It's possible that in the past this was a factor (although you should read up on those time courses and realize that your 800 year figure is also bogus), but it's clearly not true today. Global warming theories aren't based on correlations, they're based on fundamental principles of science.Ben Hocking
Need a professional organizer?
It is fairly common knowledge that 3 things factor into tenure (in this order): (1) being published (2) bringing funding into the university and (3) teaching.
...
1. A good number to shoot for is 15 journal articles in your first 6 years. If you don't have tenure in 6 years chances are you are never going to get it. The point of being published is to get the name of the university out.
2. Should be self-explanatory. You need to bring in $$$ to the university. The more you bring, the more profitable you are and the more they need to keep you around. But publishing is still more important.
3. Teaching, while as students we all feel is important, is actually the least important thing towards tenure. A mediocre or even bad teacher who writes papers (that get accepted by excellent journals) at a rapid pace will get tenure where an excellent teacher who can't write for the life of him will not. This is why you often see people from industry teaching. They teach for the love, tenured professors are there for the research and for the higher level teaching (where it is more a relation of facts, not an educational process).
The 'sloppy analysis' referred to is not 'fraud' as you cite. There is a difference between fraud and sloppy analysis. The rush to put out papers (between 2 and three a year, by this guide, for tenure) causes some slop to occur. As a reference, I've been working on a paper with my advisor and a (yet-to-be-tenured) professor for almost a year already, and we are just submitting it to a major journal. And the paper is based mostly off of my thesis work completed a year ago! A good paper and good research takes time. But please, do not mistake sloppy analysis for fraud. Mistakes are one thing, deception entirely another.
SOURCE: Advice to rocket scientists: A Career Survival Guide for Scientists and Engineers. Dr. Jim Longuski, published by the AIAA in 2004. But again, this is fairly common knowlege and can be found anywhere you look. As a postdoc (I am too) I'm suprised you didn't know
As another post-doc, I have to say that while the article writer is blunt, tactless, and overdramatic, he has something of a point. Although the reviewers of papers submitted to conferences are usually diligent and careful (and give excellent feedback), I would be very surprised if any of them attempted to replicate the experiments described in the papers -- simply because it's an infeasible amount of work for them to have to do. So if the authors have made any mistakes that aren't obvious just from the text, they're unlikely to have be picked up by the reviewers. Most experiments never get re-performed elsewhere, so conclusions do not get confirmed as often as perhaps they should (academics are rewarded for new work, not reproducing and checking the work of others).
.05 level -- ie a 5% chance that this result was due to random chance. So, up to 1 in 20 conclusions would be utterly wrong - it was just random chance after all. But since even conference papers cite a good 10 or 12 others, around half the papers would use an incorrect result in their discussion of an issue...
In fact on a good day, even if all experiments were carried out perfectly, the literature would still contain a lot of incorrect conclusions. Statistical tests in many fields are done at the
I'm painting the picture a little vaguely and perhaps just as overdramatically as the original author, but my point is that just because something is published in the literature does not mean you can check your critical judgement at the door. Scientists like you and me do not abandon their critical judgement, but sadly many science journalists (and hence the public at large) do. And that, I think, is the point the original author was trying to make.
This guy's main beef appears to be with medical studies and other sciences which rely heavily on statistics (sociology, psychology and the other wannabe-sciences). This is not surprising, to be honest. Statistical analysis isn't difficult, but I've known many social science students. They consider statistics to be extremely advanced and have no other mathematical background. As a result, they don't have a very deep understanding of how to mathematically model a system. Naturally, this will lead to bogus conclusions and incompetent analysis work. Medicine has a similar problem, albeit on a smaller scale. Most of the time, statistical analysis will yield correlations, but they won't tell you anything about the mechanism behind what you're seeing, which is what's important in science.
I'd expect the rate of error for physics experiments to be much lower than that of, say, sociology.
>According to the scientists
According to the scientists, the program misrepresented what they said.
>the concept of CO2 warming was a fairly small area of research that wasn't taken very seriously
On the contrary, it goes back to Arrhenius and is generally agreed to be the reason the oceans aren't frozen over. The existence of a "greenhouse" effect was in science textbooks decades ago.
>CO2 levels rose about 800 years AFTER the temperature rose.
After the temperature BEGAN to rise. Temperature and CO2 feed on each other in a positive feedback cycle. The Milankovitch cycles, by themselves, aren't enough to account for the temperature swings in the geological record. There needs to be some mechanism that amplifies the temperature swings, and CO2 accounts for it.
That positive feedback implies some important things for making policy. In particular, it means warming will go further than you'd expect -- CO2 production leads to more CO2 production, rising temperatures cause temperature to go up further.
MOD PARENT UP!!! Excellent contribution to the discussion!
:: cough ::" You know something
is wrong when even first posters complain about accuracy.
The media often contributes by being dishonest and over-interpreting results.
Most "scientific" papers aren't really scientific. The first clue is that they are poorly written, suggesting that the writers want to hide their poor contribution behind bad expression.
Slashdot editors often are fooled by "junk science", I notice. For example, this article was fraudulent in my opinion: Imaging Breakthrough "Sees" Lung Disease.
The Slashdot article The Car That Makes Its Own Fuel [slashdot.org] has a +5 moderated First Post that expresses the consensus of the comments on that story: "first post to call bullshit!
The Slashdot article Hydrogen Won't Save Our Economy said that water was rare. That's a stretch considering most of the surface of the planet is covered with deep water. Maybe Slashdot editors had never heard of the Pacific ocean. Then there's that small pond called the North and South Atlantic ocean.
You probably don't care, so let's go to the refutation of your claim:
From Wikipedia: "Carl Wunsch, professor of Physical Oceanography at MIT, was originally featured in the programme. Afterwards he said that he was "completely misrepresented" in the film and had been "totally misled" when he agreed to be interviewed.[23][5] He called the film "grossly distorted" and "as close to pure propaganda as anything since World War Two."
You can also check out the "responses to scientists" part of the Wikipedia article to see how he deals with being questioned.
Being bitter is drinking poison and hoping someone else will die
Some Epidemiological Claims of Sex Differences for Genetic Effects Not Replicated.
This is a *very* small number of claims from a subsection of a single field of one small bit of science. Tarring all of science based on some potentially dubious epidemiology is badly out of line. It would be like claiming that since some spinach has made people sick, all food is unsafe to eat. Absurd.
Epidemiology itself has a bit of a reputation of having a hard time finding really solid effects, partly because the effects that are measured are frequently multi-variate with lots on confounding effects, partly because you need huge numbers to have very much analysis power, partly because such studies are generally more observational then experimental. This guy has published a bunch of papers in the past arguing (and presenting models for) exactly this kind of problem. He comes up with the logical (if rather obvious) suggestions that amongst others: 1. Smaller studies are less likely to be true. 2. Smaller observed effects are less likely to be true. 3. The greater the financial interests there are in the study, the less likely it is to be true. 4. The "hotter" a topic is, the less likely a study is to be true. Largely these are no shit, sherlock kinds of things.
So, to sum up, there are lots of epidemiological claims in published articles out there that might not be right. This represents neither a new idea, nor a meaningful comment on anything but epidemiology.
-Ted
-=-=- Quantum physics - the dreams stuff are made of.
Ben Hocking
Need a professional organizer?