Slashdot Mirror
Study Shows One Third of All Studies Are Nonsense
SydShamino writes "CNN has a report on new research to confirm claims made in initial, well-publicized studies. According to the new study, about a third of all major studies from the last 15 years were subsequently shown to be inaccurate or overblown. The study abstract is available."
four out of three people have problems with fractions.
Yes, but this wasn't talking about surveys with leading questions. TFA was talking about clinical studies published in medical journals like JAMA and the New england Journal of Medicine.
Even worse are the lazy journalists who report it. After a New York Times piece last week claimed bisexual males were "lying" based on results from a highly questionable study, I reminded their editors of this excellent piece Blinded by Science in Columbia Journalism Review.
This kind of sloppy reporting is perfect for lazy journalists-- it's a three-for-one deal. They get to break the news, and then later they have a second story when real experts point out the flaws, and a third when the people finally get discredited. More evidence of the shameful state of journalism in this country.
Evil sig is livE.
Drink 8 Glasses of Water a Day (does beer count?)
Depends. One would have to calculate the water content of beer versus the rate of dehydration produced by the alcohol content. Following through, one would conclude that 8 glasses of beer would fall short of the goal of 8 glasses of pure water, with the only recourse being to drink more beer.
This, kids, is a practical demonstration of how to make science work for you.
Most obvious comment ever has been taken. On November 12, 2001 (61 days after the WTC attack in NYC) American Airlines flight 587 took off from JFK and promptly crashed into a Queens neighborhood. Obviously, most Americans suspected the worst. That day, I was watching CNN when one of these so-called "experts" (sic) came on and actually said in plain english:
"This is not a very good time for something like this to happen."
So my question is this: when is a good time for an airplane full of people to crash into a residential neighborhood? This guy should designate a day for us so we can make sure all the airlines and pilots know when the good day for crashing is. Morons.
Turkish shepherds look at dead sheep in the town of Gevas, near the city of Van, eastern Turkey, Thursday, July 7, 2005. First one sheep jumped to its death. Then stunned Turkish shepherds, who had left the herd to graze while they had breakfast, watched as nearly 1,500 others followed, each leaping off the same cliff, according to the Turkish media reported on Friday July 8, 2005. In the end, 450 dead animals lay on top of one another in a billowy white pile. Those who jumped later were saved as the pile got higher.
Yahoo link
I feel like alot of posters are not understanding what the study is... this is probably because the abstract (or, if you have access, the actual article) is much more meaningful than the CNN report.
First, notwithstanding the many good jokes about a self-referrential study that will proven to be exaggerated, this study specifically checked whether highly cited clinical studies had claims that were later contradicted or softened due to other research. This study was not claiming that 1/3 of all scientific studies published were wrong in some way. It's worth noting that doing clinical research is very difficult, and that the error bars will always be quite large. It's also important to keep in mind that sometimes clinical research may be unduly influenced by financial pressures... and that clinical research undergoes very heavy scrutiny.
So having 1/3 of all clinical studies be later contradicted should not make us worry that clinical research is being done wrong. We should be happy that so much verification occurs, that any erroneous conclusions will (probably) be checked again. One line from the CNN article rings true:
Experts say the report is a reminder to doctors and patients that they should not put too much stock in a single study and understand that treatments often become obsolete with medical advances.
I think that should be the take-home message for the casual reader. Science is doing its job of verification, but people need to stop jumping to conclusions (or worse, changing their life habits) based on the results of a single study. The results need to be double-checked. The study may have been a fluke, or have flaws, or the data may have been manipulated. Whatever the reason, we should not trust single experiments, especially where human lives are at stake!
Having (partially) read the JAMA article, I think their result is sobering and useful. It really shows how intense the competition is in that field (which leads both to people making exagerrated claims, but also alot of pressure to dis-prove other's claims and get at the "right answer").
It's not that boggling when you consider 62% of all Insightful's are Overrated and 83% of all Funny's are Redundant.
-Rabbit
Obviously errors are not completely avoidable because people are fallible; that's why we try to reproduce results and practice peer review. But I should think we ought to do better than having 33% of our supposedly "proven" hypotheses eventually disproved by subsequent experiments.
Note that I'm not talking here about trivial things like Netwon's laws of motion being "disproved" by relativity. Relativity is more like a generalization of Newton's laws than a refutation, and that *is* a part of the normal scientific process. I'm talking here about medical studies which come up with conflicting results or the innumerable global warming studies that the scientific community can't make up its mind on (for example).
main(c,r){for(r=32;r;) printf(++c>31?c=!r--,"\n":c<r?" ":~c&r?" `":" #");}
Thats a pretty good one, my favorite was this one from the CNN news ticker:
"Public split on whether Bush is a divider."
Okay, I know that everyone likes throwing out wisecracks about the headline, which was ever-so-cleverly chosen by the article submitter, but consider the article for a moment.
This is about the accuracy of clinical trial research. This is not about market research studies in the latest clothes fashions. Medicine is an extremely lucrative and risky field -- being associated with the group that pushes through the next Viagra can ensure that your family becomes the next Rockefellers. Your only opposition is the FDA (and the politicians that influence it, which are always hungry for money, which you have lots of).
There is a tremendous amount of pressure on pharmaceutical researchers to produce favorable results. Let's say that you're a new, idealistic researcher who runs some tests on a new drug that your employer wants to market. Your tests show that our drug produces an increased rate of cancer? Well, been nice having you work here...bye. Bob down the hall has consistently gotten us much better results to feed to the FDA for approval. We really don't know how or why he gets better results, but he's definitely the man we want on the job. Sure, maybe twenty years down the road there will be some complaining, but *we didn't know*...*we did all our due dilligence and somehow our results just wound up showing that our drug was okay*.
And even the more innocent "conclusive results" become suspect. A pharmaceutical doesn't want "inconclusive results", where "further tests are recommended". They have a bloody lifetime on the product ticking away, and a competition breathing down their neck. They want some scientist to sign off on this thing with a nice firm "Okay" or "Not Okay", or else what are they paying the guy for? He's not here to do ivory tower work -- he's here to serve the company, which is in the business of extracting savings from aging and achy baby boomers and subsidies paid for by their tax-paying children.
What is being said is that a full third of examined clinical trials were essentially horseshit. This is really not a laughing matter.
Any program relying on (nontrivial) preemptive multithreading will be buggy.