Study Shows One Third of All Studies Are Nonsense

SydShamino writes "CNN has a report on new research to confirm claims made in initial, well-publicized studies. According to the new study, about a third of all major studies from the last 15 years were subsequently shown to be inaccurate or overblown. The study abstract is available."

Comments

Re:more like

[cyclopropene]

all studies are biased and show exactly what the person who is doing them want. much like stats, you can make anything prove anything through working the #'s and asking the right questions

While I agree that some form of bias infects essentially all research, I don't think that's the whole picture here. Publishing new scientific results has two components. One is to put the data out into the literature where other researchers can read it, comment on it, build on it and improve on it. The other is to provide your interpretation of it, to draw what are supposed to be the best conclusions you can based on the data set that you have, even if it's small. Many conclusions based on initial data or small data sets may well be proved wrong (or may well be biased toward what the investigators want), but that doesn't mean that all of the data is wrong, or that it has no value to other researchers. Scientific literature is a dialog involving the presentation of data along with an explanation. It is expected that the interpretation will be amended as larger data sets become available. The mistake is to take an old, highly referenced paper and treat it as gospel.

Re:nice

[nickj6282]

Most obvious comment ever has been taken. On November 12, 2001 (61 days after the WTC attack in NYC) American Airlines flight 587 took off from JFK and promptly crashed into a Queens neighborhood. Obviously, most Americans suspected the worst. That day, I was watching CNN when one of these so-called "experts" (sic) came on and actually said in plain english:

"This is not a very good time for something like this to happen."

So my question is this: when is a good time for an airplane full of people to crash into a residential neighborhood? This guy should designate a day for us so we can make sure all the airlines and pilots know when the good day for crashing is. Morons.

Re:Irony meter!

[Soul-Burn666]

Well there's that Guardian story about the transportation company suing 10 cleaning women for carpooling instead of using their overpriced and horrible service.

Re:Obviously flawed

[A1kmm]

I think that this study is measuring what it says it does correctly(so isn't flawed by its own criterion), but it is measuring the wrong thing.

Think about it: they are measuring highly cited studies that get a "stronger" result than other subsequent studies. However, they simply say "stronger"(at least in the abstract). Whenever you measure something other than by a census, you take a sample. Therefore, as anyone familiar with elementary statistics should know, you have sampling variation. Researchers then usually appeal to the 'central limit theorem' to assume that the mean comes from a normal distribution, and then, knowing the distribution of the mean, make a statement like "We are 95% confident that x is between a and b". The later experiment will make a similar such claim.

Assume the experiment to measure x was performed correctly and identically both times, and there is no change in the effect with respect to time. Each time the experiment is performed, a different mean value of x will be obtained due to sampling error. Since both measurements of x come from an identically distributed population, by symmetry 50% of the time the earlier one will be "stronger" than the later one shows. However, not all highly cited studies are repeated after publication, and not all "me too" studies are published, hence the figure less than 50%.

Clearly, what they should be doing is comparing the confidence intervals, and looking for a statistically percentage of studies which do not overlap. This then would be relevant, as it would show us about non-sampling errors, rather than sampling errors, such as experiments designed or performed or interpreted incorrectly.

This study is actually alarming

[typical]

Okay, I know that everyone likes throwing out wisecracks about the headline, which was ever-so-cleverly chosen by the article submitter, but consider the article for a moment.

This is about the accuracy of clinical trial research. This is not about market research studies in the latest clothes fashions. Medicine is an extremely lucrative and risky field -- being associated with the group that pushes through the next Viagra can ensure that your family becomes the next Rockefellers. Your only opposition is the FDA (and the politicians that influence it, which are always hungry for money, which you have lots of).

There is a tremendous amount of pressure on pharmaceutical researchers to produce favorable results. Let's say that you're a new, idealistic researcher who runs some tests on a new drug that your employer wants to market. Your tests show that our drug produces an increased rate of cancer? Well, been nice having you work here...bye. Bob down the hall has consistently gotten us much better results to feed to the FDA for approval. We really don't know how or why he gets better results, but he's definitely the man we want on the job. Sure, maybe twenty years down the road there will be some complaining, but *we didn't know*...*we did all our due dilligence and somehow our results just wound up showing that our drug was okay*.

And even the more innocent "conclusive results" become suspect. A pharmaceutical doesn't want "inconclusive results", where "further tests are recommended". They have a bloody lifetime on the product ticking away, and a competition breathing down their neck. They want some scientist to sign off on this thing with a nice firm "Okay" or "Not Okay", or else what are they paying the guy for? He's not here to do ivory tower work -- he's here to serve the company, which is in the business of extracting savings from aging and achy baby boomers and subsidies paid for by their tax-paying children.

What is being said is that a full third of examined clinical trials were essentially horseshit. This is really not a laughing matter.

Politics and money drives the FUD.

[TapeCutter]

"...innumerable global warming studies that the scientific community can't make up its mind on (for example)." - Bad example, climate scientists "know" the planet is warming and also why it is warming, but fossil fuel politics creates an enormous amount of FUD in an attempt to make you and me think the scientists are contradicting each other and basically haven't got a clue. A similar thing occured when medical scientists said tabacoo was bad for your health. Incredibly some of the same "researchers" who "proved" tabacco was harmless have also been involved in "proving" climate scientists are wrong.