Slashdot Mirror
Study Shows One Third of All Studies Are Nonsense
SydShamino writes "CNN has a report on new research to confirm claims made in initial, well-publicized studies. According to the new study, about a third of all major studies from the last 15 years were subsequently shown to be inaccurate or overblown. The study abstract is available."
My apology in advance for being a MC-person, but if 1/3 of the studies are inaccurate, which means this study can be 1/3 inaccurate, does it mean that the actual inaccuracy is 1/3 * 2/3 = 2/9 of all major studies are inaccurate?
Rock that crushes, Paper & Scissors that don't matter.
Since when is "inaccurate" or "overblown" nonsense? That's what science is: study something, make a theory, and just about dare someone else to prove it wrong, because that's what makes for a better theory.
[
I think it states that most studies have inaccurate or overblown results not that 1/3 of studies are completely off.
Just because a study has inaccuracies doesn't mean the whole thing is nonsense.
According to a recent study involving 100 clones based on DNA fragments of Karl Popper, a statistically significant number of the clones agree that this is pretty goddamn good result, considering that that's how science is supposed to work.
You know - that silly process whereby you make a falsifiable claim, run an experiment, report your results, and encourage others to add to the store of scientific knowledge by attempting to falsify your original hypothesis?
The fact that they get away with it is a shame. It's even worse when they have an influence on government policy. Ugh.
Lots of people can't think of a good reason to do science, maths and statistics at school. Well, a bloody good reason is so you can prevent the wool being pulled over your eyes.
Deleted
Aaargh. Comments like this turn up in droves in every story that mentions statistics (in any light, whether good or bad) and ... wait for it ... they're always wrong. 100% of the time. I can state that with absolute certainty. No margin of error.
The fact is, yes, statistics can be misused. So can every other field of study. But used right, statistics are a tremendously powerful way to understand our world, and often reveal information that can't be obtained any other way. And believe me, nobody gets more peeved at statistics abuse than statisticians do.
But that's okay, pal. Just keep on making fun of things you don't understand. The smart people of the world will keep on working, keep doing things that make your and everyone else's life better, whether you know it or not.
The correlation between ignorance of statistics and using "correlation is not causation" as an argument is close to 1.
When companies can buy reports and studies to say whatever the fuck they want them to say (*cough*microsoft*cough*), of course they are going to be bullshit.
Who's surprised by this? Seriously.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
Even worse are the lazy journalists who report it. After a New York Times piece last week claimed bisexual males were "lying" based on results from a highly questionable study, I reminded their editors of this excellent piece Blinded by Science in Columbia Journalism Review.
This kind of sloppy reporting is perfect for lazy journalists-- it's a three-for-one deal. They get to break the news, and then later they have a second story when real experts point out the flaws, and a third when the people finally get discredited. More evidence of the shameful state of journalism in this country.
Evil sig is livE.
No - the standard way to report statistics is with a 95% confidence interval (i.e. there is a 95% chance that the true value lies within the interval quoted). It's just a norm that has nothing to do with overblown results (at least not directly) and nothing to do with the study.
No, really: internet traffic really is doubling every six months!
It is!
No, please don't call my bluff!
What are those cuffs about?
Screw you all! SCREW YOU ALL!
Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
Even worse are the lazy journalists who report it. After a New York Times piece last week claimed bisexual males were "lying" [nytimes.com] based on results from a highly questionable study, I reminded their editors of this excellent piece Blinded by Science [cjr.org] in Columbia Journalism Review.
This kind of sloppy reporting is perfect for lazy journalists-- it's a three-for-one deal. They get to break the news, and then later they have a second story when real experts point out the flaws, and a third when the people finally get discredited. More evidence of the shameful state of journalism in this country.
Slashdotters: You are all a bunch of faggots.
Do you hear me, you repulsive faggots? NO DIGG.
Connectivity - global media, the internet -- have created a winner-take-all world that drives both the creators of studies and the reporters of studies toward hyperbole. If someone wants their 15 minutes of fame, they need to do (or appear to do) something spectacular. When attention is a scarce resource (because of an explosion of applications/demands for attention), then it drives people toward excessive behavior in crafting and reporting the results of studies.
At the same time, I wonder if the long tail efect means that an increasing number of once-obscure, high-quality studies are being discovered, read, and used by an increasing number of people. Those that do create unbiased studies may not get much popular press, but they do become more widely read due to Google.
Ultimately, we seem to be floating in a rising tide of both good and bad studies. Perhaps the ratio of studies is being biased toward the bad (winner take all) but the ratio of impressions -- the numbers of times that good studies have been accessed -- has actually improved due to long-tail effects.
Two wrongs don't make a right, but three lefts do.
The title to this "Study Shows One Third of All Studies Are Nonsense" is just wrong. If the author read the article it would be obvious the problem. Hell the first line is this:
"Controversy and uncertainty ensue when the results of clinical research on the effectiveness of interventions are subsequently contradicted."
Effectiveness of interventions! This only looked at studies in which interventions were involved. NOT ALL STUDIES.
To hyperbolize in the same manner..... I think the author is the dumbest person in the entire universe.
The research was done for MEDICAL studies, not tech studies, or animal habitat studies, or psychological studies, or sociological studies... only medical studies. Nowhere in the title or the post's main body is this mentioned. This is very poor reporting of the news. It is misleading. The study also only measured studies from 1990 to 2003. That's 13 years not 15 years!
Word to the wise, don't trust the press at face value. Expect sources, preferably cited and available for you to review, and check your facts before you buy into whatever the press happens to be reporting today.
Being able to make up stats and get away with it is one of the nice things about having a PhD... most people do not have the qualifications nor the data necessary to expose the made-up nature.
This appears to be particularly frequent in more abstract (non-maths) sciences like environment. (I once had lectures on the topic where the speaker cited stats that did not match the notes and were inconsistent across presentations.)
So if a third of all studies are overblown, how do we know that this study isn't overblown or inaccurate? Hmmm?
This signature has Super Cow Powers
The main reason that journalism is "flawed" is because in the MSM or Main Stream News, either print or broadcast, the main focus is on making money. It's a "business" first. You will have to somehow make it be "news" first to get more accuracy and objectivity in reporting.
And then it could segue into something roughly analagous to the debates over for-sale closed source software and collaborative information-sharing free software. Could a news reading public be persuaded to actually become critical reporters and "share" news freely? Could it replace the expensive and established profit motive design of "news" as we know it today?
Some might say blogging is at least an attempt in that direction.
Just how do we know this study isnt one of the 1/3?
Phredd - "I have found people tend to take you far less seriously once you start waving your genitals at them..."
However, statistics are not determinative. This is a mistake I've heard from both laymen and experts. The fact that, according to what's known (and factored in to the calculation) an event is 99.999999999% likely to happen... well, that doesn't mean it will happen. Really, it doesn't matter how unlikely your statistics demonstrate something to be, it won't prevent the unlikely from happening.
In fact, it's demonstrable from statistical analysis that we should expect tremendously improbable events to happen quite often, and that the chances of the most probable outcomes to occur at every instance is an incredibly unlikely outcome as time stretches on.
So statistics are an interpretive tool, not an answer. Statistics alone cannot tell you what will happen, they can't tell you what has happened, and they certainly cannot tell you what should happen. And all these comments you're talking about, I think they come from a valid frustration borne from sloppy reporting telling us "scientists have discovered that 75% of" this and "they now know that 25% of" that outside of any meaningful context.
And what's the likelihood that all these percentages are correct? What's the margin of error, and what's the margin of error's margin of error? Certainly the people telling us these "facts" (reporters) have no idea.
Presumably the expert was alluding to the fact that an incident such as this, so soon after WTC, could take on a greater significance than had it been looked at in isolation.
It could lead to an escalation in hysteria, racial hatred and so forth from otherwise-rational people who believe the end of the World is nigh, or something.
Case in point: Following the London attacks there was a bomb scare in Birmingham (about 120 miles North) which resulted in 20,000+ people being evacuated from the city centre. It turned out to be a firework. Taken in isolation I'm sure the response to something like this would be a lot more measured, but coming so soon after the events in London everyone (including the Police) reacted in a typical knee-jerk fashion.
I guess my point is that there is obviously never "a good time" for a plane to crash into a neighbourhood, but there is certainly a less dramatic time for it to happen.
I'm curious to know how Mr. Gribbin defines a religious nut.
While there are a large number of people who reject facts and reason due to their a priori commitment to a religious beliefs, there are a great number who do the same whose religion is science itself.
That is to say, preconceived notions and personal bias prevent many so-called scientists from acknowledging facts and realizing that their pet theories are baseless.
As an example, I offer Carl Sagan. Here was a man who made a nice living talking about extraterrestrial life. Is there ANY evidence of extraterrestrial life? Is there ANY science that supports it? After all, the best that the SETI institute has is Drake's equation which at best merely multiplies speculation upon speculation.
Is SETI science? Perhaps, but Sagan's beliefs and public discussions were based on fantasy and hope rather than fact.
My point is this: Bias appears in religion and in the name of science. Science has dirty hands, too.
Remember, power tends to corrupt, regardless of world view. I'd be willing to bet that a similar book could be written demonstrating horrible abuses of human rights where science was allowed to 'progress' unchecked by morality.
Finally, it is important to note that much of science has been advanced by people with strong religious convictions. Pascal, Pasteur, Lister, Knuth, Kelvin, Joule, Carver, Bacon, Boyle, and many many others. Strong religious conviction is NOT the antithesis of scientific advancement, as demonstrated by the legacy of those I listed above and I could list many more.
Respectfully,
Anomaly
But Herr Heisenberg, how does the electron know when I'm looking?