Slashdot Mirror
Study Shows One Third of All Studies Are Nonsense
SydShamino writes "CNN has a report on new research to confirm claims made in initial, well-publicized studies. According to the new study, about a third of all major studies from the last 15 years were subsequently shown to be inaccurate or overblown. The study abstract is available."
There's a new study to be published next week showing that only one quarter of all studies are overblown, including this one.
I wonder if this is one of those studies that is nonsense :)
I soo knew this was true.. But no one believed me.
What does your Credit Report look like?
My apology in advance for being a MC-person, but if 1/3 of the studies are inaccurate, which means this study can be 1/3 inaccurate, does it mean that the actual inaccuracy is 1/3 * 2/3 = 2/9 of all major studies are inaccurate?
Rock that crushes, Paper & Scissors that don't matter.
A third?! I'm sure this one is at least a 'litte' overblown too...
(\_/)
(O.o) This is Bunny. (> <)
What... are you guys trying to blow up our heads?
;)
I think it is possible this is the most amusingly ridiculous piece of "legitemate" news I've read in awhile...
Anyone got anything to beat it? Post it, I need to shock my brain a little more
MoM++ - A Classic Expanded - [Master of Magic 1.5]
http://mompp.sourceforge.net/
Since when is "inaccurate" or "overblown" nonsense? That's what science is: study something, make a theory, and just about dare someone else to prove it wrong, because that's what makes for a better theory.
[
Not to be too blunt, But shouldn't a CNN report equal some kind of study, which also is overblown and innaccurate.
I can imagine the next headline:
Study finds that previous study is also nonsense
int cents = 0;
cents += 2;
This study will cause an infinite loop..PLEASE SHUTDOWN IMMEDIATELY. 0x381F
all studies are biased and show exactly what the person who is doing them want. much like stats, you can make anything prove anything through working the #'s and asking the right questions
because of probability calculus
Zapp: Kiff we have a conundrum!
"There is no time, sir, at which ties do not matter," Jeeves, (Jeeves and the Impending Doom)
Because this is from the other 2/3rd studies talking about the other 1/3rd studies.
What does your Credit Report look like?
Dr. Spin and his pragmatic stats says, that's the fact Jack!
Crunch!
Well, it says 1/3 of the studies are inaccurate, so let us rank that on a percentage scale, say the study is inaccurate, we give them a 0 value.
... *GASP FOR AIR* ... okay... so you have 66% chance of that particular study being at least somewhat accurate...
... (rolls percentile dice)... OOOOHH! Man, rolled a 72, looks like I can't believe it.
Accurate studies, lets say 100 (I know nothing is 100% accurate, and I know most studies even if they are somewhat accurate probably don't exceed 70% probability even in the specific environments they are enacted in, but lets just be over-generous since this whole thing is rather ridiculous anyway)
Right?
Rats.
MoM++ - A Classic Expanded - [Master of Magic 1.5]
http://mompp.sourceforge.net/
I think it states that most studies have inaccurate or overblown results not that 1/3 of studies are completely off.
Just because a study has inaccuracies doesn't mean the whole thing is nonsense.
Obligatory Simpsons quote:
"Oh, people can come up with statistics to prove anything, Kent. 14% of people know that."
First recursive study!
All studies is not the same thing as all studies in the field of clinical research.
According to a recent study involving 100 clones based on DNA fragments of Karl Popper, a statistically significant number of the clones agree that this is pretty goddamn good result, considering that that's how science is supposed to work.
You know - that silly process whereby you make a falsifiable claim, run an experiment, report your results, and encourage others to add to the store of scientific knowledge by attempting to falsify your original hypothesis?
The fact that they get away with it is a shame. It's even worse when they have an influence on government policy. Ugh.
Lots of people can't think of a good reason to do science, maths and statistics at school. Well, a bloody good reason is so you can prevent the wool being pulled over your eyes.
Deleted
No doubt the study of studies is itself of dubious quality.
Personally I would have expected that the Pareto principle (20% of anything is the important part) or Sturgeon's law (90% of everything is crap) would have been the operational forces here.
Talk about recursive self reference - I think this study is best suited for you number theorists out there!
What a nice paradox this story presents. "I am lying."
The way the topic makes this sound, this is some sort of blatantly obvious study...
:)
... so it's kind of... ambiguous...
it isn't, really
It is about the effectiveness of interventions... if you skipped over it, its worth a perusal to a skim... at the very least... but it would seem to me that the whole thing has lead to almost no positive conclusion itself... with 44% of the experiments being replications and 24% unchallenged... the 66% really don't seem to have much value...
Ahhh, academic research... only there can you get paid well to tell us absolutely nothing...
MoM++ - A Classic Expanded - [Master of Magic 1.5]
http://mompp.sourceforge.net/
it was published three times?
four out of three people have problems with fractions.
Questionable studies like Second-Hand Smoke Is Bad or Drink 8 Glasses of Water a Day (does beer count?)
"I'd rather be a lightning rod than a seismometer." -Ken Kesey
Since there is a 33% chance that this study itself is wrong, I hereby scoff and dismiss it!
"The power of accurate observation is frequently called cynicism by those who don't have it." - G.B. Shaw
According to the new study, about a third of all major studies from the last 15 years were subsequently shown to be inaccurate or overblown.
The actual figure turns out to have been 26.4% - much closer to 1/4 than 1/3.
I did my own study and it shows that one half of all studies are nonsense.
Homer: "Oh, Kent, you can make up statistics to prove anything. 14% of all people know that."
It's true. Those letters your aunt, grandmother, and that complete stranger who thinks you are their college friend because you have the same initials are the most reliable source of scientific, health, and political information:
Aspartame causes MS. Andy Rooney has definitive proof that the founding fathers wanted prayer in the classroom. Microwaving food in plastic causes cancer. George Carlin lays out why librals are dumb. Bill Gates wants to buy you a car for forwarding this email.
All true!
Stefan
Most published studies include statistics claiming something like a 95% (or higher) chance of being correct. I've always thought that was bogus and it looks like someone finally got around to proving it.
/me rolls 4d6
7
didn't make the saving throw, I don't belive the story.
I heard this last night on the news, and couldn't help but laugh. Anyone catch the irony in this? What if this study is inherantly false? That would give greater creadance to studies being true (..goes running for his cat and buttered bread...)
This CNN story is antiscience nonsense. The report might have something useful to say about the 45 pharma studies that were analyzed - if its results can be repeated by another analyst, and its methodology is sound (assumedly vetted by the JAMA publishers). But it doesn't say anything about "science". Science doesn't give final answers. Of course many studies are subsequently shown to be inaccurate: the reproducability of the experiment tests the theory, and more data eventually converge on the actual statistical models.
Even this study needs to be repeated, to test its accuracy. For example, as far as the CNN article states, only 45 studies were examined. Out of thousands every year. And they are from the sector of science that has the most money put into promoting studies, pharma research. How about a random sample of every study in the JAMA, for instance? Even that would have to be tested against a random sample of, say, Lancet, to discover comparative bias in the two journals. And of course they'd have to do multiple studies on the same journal, each with different random samples.
--
make install -not war
When companies can buy reports and studies to say whatever the fuck they want them to say (*cough*microsoft*cough*), of course they are going to be bullshit.
Who's surprised by this? Seriously.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
Even worse are the lazy journalists who report it. After a New York Times piece last week claimed bisexual males were "lying" based on results from a highly questionable study, I reminded their editors of this excellent piece Blinded by Science in Columbia Journalism Review.
This kind of sloppy reporting is perfect for lazy journalists-- it's a three-for-one deal. They get to break the news, and then later they have a second story when real experts point out the flaws, and a third when the people finally get discredited. More evidence of the shameful state of journalism in this country.
Evil sig is livE.
..and one third of the replies to this article will be lame jokes about this study being nonsense.
...was that 64% of all statistics are made up on the spot. Yeah, I've got nothing.
I sure hope this study falls into that category :)
For he today that sheds his blood with me shall be my brother.
A new study has shown that the previous study on the subject was flawed and reached an inaccurate conclusion. The next study scheduled is anticipated to show this new study as flawed and inaccurate in its conclusion, thus reinforcing the initial assumtion and ironically contradicting itself. Upon being reached for comment, all respondents promptly suffered violent seizures as their brains clawed their way out of their skulls.
It must be Windows. It needs half a gig of RAM and a hardware-accelerated graphics card just to run Solitaire.
1] Liars
2] Damn Liars
3] Staticians
79% of statistics are made up on the spot.
how many studies are produced by creationists, or the oil lobbies' climatologists.
At work we did a study using state DOE data, we showed a correlation between high scoring (we review Juvenile detention programs by visiting them and scoring them on different benchmarks) programs and students rate of returning to school and graduating. Seems pretty obvious, and the results were as we expected. Odd thing is another team did a followup study with their own methods, both of which seemed to make sense, except they were studying kids released a year later than the ones we studied. They produced absolutly no correlation in their results. I methods seemed perfectly fine, I still havn't run their cohort against our methods to test it out. But otherwise I can't figure out what went wrong. But I guess thats just the nature of studies sometimes.
Wow! I didn't know Microsoft commission so many studies!
Oh, people can come up with statistics to prove anything, Kent. 14% of people know that. --Homer Simpson Simpsons episode 1F09 (Homer the Vigilante)
Laziness is a virtue, anyone who bothers to tell you otherwise, is clearly lacking it.
One might argue experimenters should observe better, but they are often trying to study very subtle stuff (in terms of getting their instrumentation to show them a reliable result), and often, as in medicine, going not for either/or results but for 45, 55% likely, whatever -
--and then it's reviewed by their peers, or new tests are done. Hallelujah, science at work.
(Of course all the other posters' comments about statistics and bought studies apply, too.)
Physicist 1 to physists 2: One in three scientists believe their peers fabrictate research.
Physicist 3, to herself: He's making that up...
Physicist, consultant, science communicator
Yeah, this reminds me of the fact that 87.6% of all statistics are made up on the spot!
made up on the spot. This study proves that!
No - the standard way to report statistics is with a 95% confidence interval (i.e. there is a 95% chance that the true value lies within the interval quoted). It's just a norm that has nothing to do with overblown results (at least not directly) and nothing to do with the study.
Well now they were refering to medical research; which is kind of like the humanities of the hard sciences - so really, who knows - as long as i can still get my viox .... oh wait... damn!
This study, found in the scholarly journal Duh...
In walking, just walk. In sitting, just sit. Above all, don't wobble.
-- Yun-Men
This guy alone probably tipped the scales considerably...
Which 1/3 does this study fall under?
---- Booth was a patriot ----
One Third of All Slashdot Comments are Nonsense. This isn't one of them.
Unfortunately, no research was done whether the invaluable publications had something in common, e.g. corperate sponsered.
If we get two groups to repeat this study and falsify the claims of this one, what exaclty have we proved?
I feel a bit like Marvin right now.
I wonder if _this_ study is in the 1/3 of them that are found to be nonsense.
My head hurts.
Don't think that a small group of dedicated individuals can't change the world. It's the only thing that ever has.
Well if you look at all the studies for and against global warming you quickly realize that there are hundreds of studies out there that have to be overblown garbage. How else can you explain so much contradicting data?
“Common sense is not so common.” — Voltaire
How much did the taxpayers pump out to pay for or subsidize this?
We need the studies equivalent to python's Ministry of Silly Walks.
This sounds similar to the study that showed that 83% of people lie regularly.
See subject
97% of articles about scientific studies over simplify the findings in the search for a headline
When its reposted tomorrow, you'll have another 66% chance its correct.
http://www.toddsnider.net/newconnection.html
STATISTICIAN'S BLUES
Todd Snider ©2002
==============
They say 3 percent of the people use 5 to 6 percent of their brain
97 percent use 3 percent and the rest goes down the drain
I'll never know which one I am but I'll bet you my last dime
99 percent think with 3 percent 100 percent of the time
64 percent of all the world's statistics are made up right there on the spot
82.4 percent of people believe 'em whether they're accurate statistics or not
I don't know what you believe but I do know there's no doubt
I need another double shot of something 90 proof
I got too much to think about
Too much to think about
Too much to figure out
Stuck between hope and doubt
It's too much to think about
They say 92 percent of everything you learned in school was just bullshit you'll never need
84 percent of everything you got you bought to satisfy your greed
Because 90 percent of the world's population links possessions to success
Even though 80 percent of the wealthiest 1 percent of the population
Drinks to an alarming excess
More money, more stress
It's too much to think about
Too much to figure out
Stuck between hope and doubt
It's too much to think about
Pick it now
84 percent of all statisticians truly hate their jobs
They say the average bank robber lives within say about 20 miles of the bank that he robs
There's this little bank not far from here I've been watching now for a while
Lately all I can think about's how bad I wanna go out in style
And it's too much to think about
Too much to figure out
Stuck between hope and doubt
It's too much to think about
That's right... It's too much to think about... Amen... It's too much to think about... Mmm hmmm
"An unarmed man can only flee from evil, and evil is not overcome by fleeing from it." Col. Jeff Cooper
Maybe because those studies were done by researchers and professors in University?
I've seen many (now) tenures who had spent more time writting articles/jounals and conducting research to keep their jobs than actually teach to keep their jobs.
We should start giving condoms to kids in University, because they are getting royally screwed by unprotected University education.
"Don't let fools fool you. They are the clever ones."
I feel like alot of posters are not understanding what the study is... this is probably because the abstract (or, if you have access, the actual article) is much more meaningful than the CNN report.
First, notwithstanding the many good jokes about a self-referrential study that will proven to be exaggerated, this study specifically checked whether highly cited clinical studies had claims that were later contradicted or softened due to other research. This study was not claiming that 1/3 of all scientific studies published were wrong in some way. It's worth noting that doing clinical research is very difficult, and that the error bars will always be quite large. It's also important to keep in mind that sometimes clinical research may be unduly influenced by financial pressures... and that clinical research undergoes very heavy scrutiny.
So having 1/3 of all clinical studies be later contradicted should not make us worry that clinical research is being done wrong. We should be happy that so much verification occurs, that any erroneous conclusions will (probably) be checked again. One line from the CNN article rings true:
Experts say the report is a reminder to doctors and patients that they should not put too much stock in a single study and understand that treatments often become obsolete with medical advances.
I think that should be the take-home message for the casual reader. Science is doing its job of verification, but people need to stop jumping to conclusions (or worse, changing their life habits) based on the results of a single study. The results need to be double-checked. The study may have been a fluke, or have flaws, or the data may have been manipulated. Whatever the reason, we should not trust single experiments, especially where human lives are at stake!
Having (partially) read the JAMA article, I think their result is sobering and useful. It really shows how intense the competition is in that field (which leads both to people making exagerrated claims, but also alot of pressure to dis-prove other's claims and get at the "right answer").
Connectivity - global media, the internet -- have created a winner-take-all world that drives both the creators of studies and the reporters of studies toward hyperbole. If someone wants their 15 minutes of fame, they need to do (or appear to do) something spectacular. When attention is a scarce resource (because of an explosion of applications/demands for attention), then it drives people toward excessive behavior in crafting and reporting the results of studies.
At the same time, I wonder if the long tail efect means that an increasing number of once-obscure, high-quality studies are being discovered, read, and used by an increasing number of people. Those that do create unbiased studies may not get much popular press, but they do become more widely read due to Google.
Ultimately, we seem to be floating in a rising tide of both good and bad studies. Perhaps the ratio of studies is being biased toward the bad (winner take all) but the ratio of impressions -- the numbers of times that good studies have been accessed -- has actually improved due to long-tail effects.
Two wrongs don't make a right, but three lefts do.
for two more scientific studies on the validity of scientific studies before I...
Ah, forget it.
The 1/3rd that are useful
The 1/3rd that are nonsense
or the 1/3rd that are somewhere in between?
Correction: Study shows 66.66% chance that 1/3 of studies are nonsense.
I'm an American. I love this country and the freedoms that we used to have.
I don't think the problem is a surfeit of nonsensical studies, so much as a misrepresentation of them by the media. All the time, studies are reported as news. Yet even a well-carried out, accurate study does not prove anything. Studies by their nature must be corroborated to be meaningful at all. When a new study comes out, it doesn't mean that we know anything new. It means that somebody suspects something which may in the long run turn out to be true. The problem is that the latter is not sensational enough for news, so studies are represented as the former.
English is easier said than done.
I only read the last two-thirds of any study. This just confirms my intuition -- why bother with the nonsense at the start?
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
.008). Among randomized trials, studies with contradicted or stronger effects were smaller (P = .009) than replicated or unchallenged studies although there was no statistically significant difference in their early or overall citation impact. Matched control studies did not have a significantly different share of refuted results than highly cited studies, but they included more studies with "negative" results.
John P. A. Ioannidis, MD
JAMA. 2005;294:218-228.
ABSTRACT
Context Controversy and uncertainty ensue when the results of clinical research on the effectiveness of interventions are subsequently contradicted. Controversies are most prominent when high-impact research is involved.
Objectives To understand how frequently highly cited studies are contradicted or find effects that are stronger than in other similar studies and to discern whether specific characteristics are associated with such refutation over time.
Design All original clinical research studies published in 3 major general clinical journals or high-impact-factor specialty journals in 1990-2003 and cited more than 1000 times in the literature were examined.
Main Outcome Measure The results of highly cited articles were compared against subsequent studies of comparable or larger sample size and similar or better controlled designs. The same analysis was also performed comparatively for matched studies that were not so highly cited.
Results Of 49 highly cited original clinical research studies, 45 claimed that the intervention was effective. Of these, 7 (16%) were contradicted by subsequent studies, 7 others (16%) had found effects that were stronger than those of subsequent studies, 20 (44%) were replicated, and 11 (24%) remained largely unchallenged. Five of 6 highly-cited nonrandomized studies had been contradicted or had found stronger effects vs 9 of 39 randomized controlled trials (P =
Conclusions Contradiction and initially stronger effects are not unusual in highly cited research of clinical interventions and their outcomes. The extent to which high citations may provoke contradictions and vice versa needs more study. Controversies are most common with highly cited nonrandomized studies, but even the most highly cited randomized trials may be challenged and refuted over time, especially small ones.
INTRODUCTION
Jump to Section
Top
Introduction
Methods
Results
Comment
Author information
References
Clinical research on important questions about the efficacy of medical interventions is sometimes followed by subsequent studies that either reach opposite conclusions or suggest that the original claims were too strong. Such disagreements may upset clinical practice and acquire publicity in both scientific circles and in the lay press. Several empirical investigations have tried to address whether specific types of studies are more likely to be contradicted and to explain observed controversies. For example, evidence exists that small studies may sometimes be refuted by larger ones.1-2
Similarly, there is some evidence on disagreements between epidemiological studies and randomized trials.3-5 Prior investigations have focused on a variety of studies without any particular attention to their relative importance and scientific impact. Yet, most research publications have little impact while a small minority receives most attention and dominates scientific thinking and clinical practice. Impact is difficult to measure in all its dimensions. However, the number of citations received by a publication is a surrogate of the attention it has received in the scientific literature and its influence on scientific debate and progress. Citations are readily and objectively counted in established databases.6 High citation count does not necessarily mean that these studies are accepted;
The study basically just says that there are alot of nonrandomized studies and some small-sample-size randomized studies, that they are often wrong, and that they are cited alot anyway.
I think most people would already agree that those kinds of studies are 'bad' for exactly those reasons, anyway.
Presumbly the "true value" is outside the "confidence interval" for 1 in 3 of the studies investigated which is a whole lot worse than 1 in 20 implied by a 95% confidence level.
Studies show only 0.002% of Slashdot viewers will ever get laid. Additionally the study showed that the slashdot population has the largest population of virgins over the age of 37.
That would mean 1/3 of THAT study is nonsense so only 2/3 of 1/3 of all studies are nonsense. 2/3 of 1/3 is 2/9. So really, only 2/9 of all studies are nonsense. So then 2/9 of 1/3 of all studies are nonsense. 2/9 of 1/3 is 2/27. Therefore only 2/27 of all studies are nonsense.... I suck at math so at some point, that might converge to 0 and thereby totoally invalidating the study! Since the reduction depends on the validity of the original study, we have a logical paradox!
PS - Yes, my math is probably wrong. No, I didn't RTFA. My logic is probably off too. I just wanted to see myself post.
EvilCON - Made Famous by
What they don't say is the forementioned study has a 33% chance of being a bunch of crap too...
Additionally, wouldn't it be nice to name the organization that sponsored the study that the AP (not CNN!) is reporting about? (I know there's a link, but why can't we have it in the description too?!)
It's an easy thing to fix, and it would make Slashdot seem more trustworthy...
I can personally relate to this. While in AP Statistics, we practiced our mad stat skillz on the real world. We were encouraged to bring in newspaper clippings of studies and experiments and see if they were statistically sound. Most weren't. The most common form of bias in those studies were known as negative response. In this form of bias, only people with negative, strong feelings reply to the study question.
I am willing to bet that the CNN study is correct in it's assumption that most studies are incorrect.
Ordinary decent people are sick and tired of being told that ordinary decent people are fed up with being sick and tired. Well I'm certainly not, and I'm sick and tired of being told I am!
That which does not kill her only prolongs my agony.
Studies show that 1\3rd of all slashdot posters use confusing links.
Wouldn't it make more sense to make the word "report" the hyperlink to the story as opposed to "inaccurate?"
___
It's the end of my comment as I know it and I feel fine.
well that would be more truthful wouldn't it?
Engineering is the art of compromise.
Posting this URL is Irony.
You are being MICROattacked, from various angles, in a SOFT manner.
this means that 1/3 of studies are paid for by Microsoft???
// TODO: Insert Cool Sig
It's a whole lot messier than that. There is not enough information to make any meaningful conclusions like that.
Notwithstanding that, and as you note, in a perfect world 1 in 20 studies using normal statistical standards would be 'wrong'. However, if the confidence interval is, say, 5-6 and all you care about is that the thing is positive it may not make much difference that it is 'wrong'.
Oh this is just absurd enough to make the "news". Haven't we all known this all along? Don't our local and on "up the ladder" governments know this?
Juggling data is fairly easy, that's how these "consulting firms" get paid.
In my town there's a developing need for a study...
Bowen Branch CreekWell, I'm an "expert" in < insert field name here >. Our firm will do a detailed assesment of the abatement issues involved with this site for $326,525. Man bulldoze the *hit! That'll be 2 cents please.
-me
> i.e. there is a 95% chance that the true value lies within the interval quoted
Unfortunately, statistics will never tell you such a direct answer(i.e. the probability that you are right) unless you are using a Bayesian approach(which they are not, and if they were, they would need to specify a priori assumptions and then update those by the evidence. If you don't agree with the a priori assumptions, the number is meaningless to you).
A more accurate statement of a confidence interval:
Assuming that the measurements(perhaps of the true mean value) really come from the normal distribution, and is equal to some specific value outside the interval [a,b], the probability that we would see evidence as strong as we did is less than or equal to 0.05
That is not equivalent to what you said, although what you said is a common misinterpretation of confidence intervals.
X-Has-Sig: yes
This study falls in the nonsense category because it makes the assumption that correlation equals proof of cause (on way or the other) between citations and falsified studies.
And fails to mention studies get published for the prupose of trying to find flaws in them hoping it cuts the nonsense in at least half!
Microsoft is pure dog-ma. FreeBSD is pure cat-ma.
I'm writing a study about all the studies that do not study themselves. I'm having trouble finishing it.
Music speeds up when you yawn, but does not change pitch.
RTFA before posting people, the way the submitter has stated this study's findings is too inflamatory and trollish in my opinion.
The AACS key is NOT 0xF606EEFD628B1CA427BEA93A9CA9773F
According to the new study, about a third of all major studies from the last 15 years were subsequently shown to be inaccurate or overblown.
if they # they provided was overblown, that means this article itself has a higher chance of being legit. on the otherhand, if the figure they provided was understated, then this study is most likely inaccurate making this discussion nonsense.
my head hurts now...
HD Trailers
Is that the third that start "Get the Facts"?
Hmmmmmm..... Deep fried and look like Squirrel.
Everybody knows that 56% of all statistics are made up on the spot...
After all, Sturgeon's Law already states that 90% of EVERYTHING is crap.
With inflation, this has been raised to 98%. I hold out for 99% myself.
So this study is at least 65% wrong if it claims 33% of studies are wrong.
Richard Steven Hack - This sig is TOO GODDAMN SHORT TO DO ANYTHING USEFUL WITH! MORONS!
The research was done for MEDICAL studies, not tech studies, or animal habitat studies, or psychological studies, or sociological studies... only medical studies. Nowhere in the title or the post's main body is this mentioned. This is very poor reporting of the news. It is misleading. The study also only measured studies from 1990 to 2003. That's 13 years not 15 years!
Word to the wise, don't trust the press at face value. Expect sources, preferably cited and available for you to review, and check your facts before you buy into whatever the press happens to be reporting today.
Totally and utterly bogus. Actually, real science is about failure, and if only 1/3 of the research doesn't produce useful results, that would be an incredibly good batting average for real-world science. Actually, I think the real success rate is more like 5%, but of course most of the failures never get as far as being published, and especially not in relatively prestigious journals.
Coincidentally, I just now happen to be reading the excellent Science: A History 1543-2001 by John Gribbin. A recurrent theme is how religious lunacy interferes with scientific progress. Can't stop it, though the main historical effect is that the progress is slowed up in one place, and takes off in some other place where the religious nuts have less control.
Freedom = (Meaningful - Coerced) Choice != (Speech | Beer^2), and sad sock puppets' bad mods avail them naught.
This study is a little misleading, because different studies get different amounts of exposure. The most amazing and unbelievable studies probably make it into the news the most. You'd have to weight the studies by exposure to accurately tell readers the probability that the next study they read will be nonsense. The 80/20 rule probably applies here.
and 87% of statistics are made up on the spot....
well i dont know if i would trust the results of this study. I think the results might be biased slightly in the direction of studies being inaccurate. I think that I will have to perform a study to test the results of this studies results of studies. HA now try to say that 3 times fast.
So if a third of all studies are overblown, how do we know that this study isn't overblown or inaccurate? Hmmm?
This signature has Super Cow Powers
The main reason that journalism is "flawed" is because in the MSM or Main Stream News, either print or broadcast, the main focus is on making money. It's a "business" first. You will have to somehow make it be "news" first to get more accuracy and objectivity in reporting.
And then it could segue into something roughly analagous to the debates over for-sale closed source software and collaborative information-sharing free software. Could a news reading public be persuaded to actually become critical reporters and "share" news freely? Could it replace the expensive and established profit motive design of "news" as we know it today?
Some might say blogging is at least an attempt in that direction.
/Obvious
This is not linearly related so . . . It is a fairly commonly known meme that when a person is measuring the way in which light travels, s/he will achieve results based on what they are looking for. Perhaps people make manifest their desired outcome.
we are never passive observers
This is a joke right? Just a rephrasing of the old "10% of statistics are incorrect" paradox?
A smile sure spread across my face when I read the title. It made me think back to the scrolling funny headlins in the old Sim City games. I must not be the only one who though that right?
134% of Statistics are made up on the spot :)
................a study was used to find that studies don't work. Genius move on their part.
Monkeys... own..... ASCII Slashdot: |/.|
I have been saying that for years!
I'll believe this when I see two more studies showing the same thing. I like to be at least 95% sure of something like this.
Just how do we know this study isnt one of the 1/3?
Phredd - "I have found people tend to take you far less seriously once you start waving your genitals at them..."
All that studies report is their results. They don't gaurantee other studies will find the same thing. If they did, there'd be no reason to have replications. This is a basic part of doing science.
And who's to say the replications aren't the ones that missed the mark?
1/3 right
1/3 wrong
1/3 we have no idea what the answer means.
That's what I was told to expect from research in my first semester of grad school. Not from reading it -- from DOING it.
They really should teach science in school. Not the just areas of science, but the subject of science itself.
"I may be synthetic, but I'm not stupid." -- Bishop 341-B
Statistics can be made up to prove anything, forfty percent of all people know that(sic) - Homer Simpson
Internet watchdogs report that some so-called "News for nerds" and "Stuff that matters" content found on popular websites, may be neither nerdy nor matter much at all...to anyone.
Most of Usenet content is probably fine, apparently. "Minus the digitally enhanced pr0n. That might be fake too", says one user who simply goes by the pseudonym of Anonymous Coward.
No sig.
Read the EFF's Fair Use FAQ
So wait....one third of all studies? Which 1/3? Do they underestimate or overestimate? So If a study said one quarter of all college students smoke, does that mean that:
1/4 - (1/4 x 1/3) = 1/6
So really only one sixth do?
Or would that be that be that really:
1/4 + (1/4 x 1/3) = 1/3
So one third does.
How do you know which 1/3 is garbage?? This is some really useful information the freaking study should let us know.
Wait, so are 4/9ths of studies nonsense or 2/9ths?
is that study itself in that 1/3 of all studies?
It could be that every study performed is accurate with the exception of this one.
Fortunately, most physicians do not change there practice based on a single study. Medicine has a long history of interventions and medications that seemed promising, but when they were looked at in a prospective, randomized control trial (RCT) they did not pan out. Vitamin E for heart disease is a recent example. The article notes out that large RCT's were not as likely to be refuted. These are the studies that physicians pay the most credence to. Learning to evaluate the methodology and statistics of a clinical trial is an important skill that is developed over time. Medical school and residency places an emphasis on this skill. As such, a physician should evaluate each trial in a rigorous fashion before deciding to incorporate the findings of that trial in his practice. Patients reading the New York Times Journal of Medicine typically do not evaluate trials in this manner. For that matter, the media usually reports the conclusions of trial authors, rather than the actual results. At times, the two can be different.
I feel like exploring the nuances so I'll make some response.
Central Limit Theorem's mean that it doesn't really matter what the underlying distribution is, the mean will asymptotically have a normal distribution. (Couldn't be bothered worrying about t-distributions.)
As for the statements, I think it becomes one more of semantics and reducing ambiguity than anything else. The true value is not a random variable, but one could interpret a statment such as "there is a 95% chance that the true value lies within the interval" as implying that is was one. A more correct statement is that "there is a 95% chance the random interval includes the true value". Mathematically they are equivalent (becaue mathematics does not allow confusion about what is a parameter and what is a random variable) and, provided the context is clear, very few people would interpret the parameter as a random variable.
Thus, I would argue that the statements are equivalent provided one does not interpret the true value as a random variable. The first statement permits an ambiguous interpretation that is not equivalent, but otherwise they are equivalent.
This is a problem, and they're starting to have to acknowledge it.
The problem is one of integrity. Now, admitably, consumers of talk show/'news' outlets like Rush Limbaugh or Al Franken are around, and can sustain a market. On the other hand, news outlets like ABC, NBC, Fox News, the New York Times, Time Magazine, and others are advertised as news programs. Thus people expect factual news out of them. When they hear mis-statement after mis-statement, they lose viewership to other things, such as blogs.
It also helps that many blogs act as special interest sites, posting articles relating to their themes. Thus a reader can hear about news that didn't make it to the national sites.
As for the articles, I wonder how many of them are related to enviromental issues? I've read that some of the most frequently used climate models have some serious problems(IE they're unstable in the long run, and tend to run hot over time no matter what the initial inputs are).
I don't read AC A human right
Okay, I know that everyone likes throwing out wisecracks about the headline, which was ever-so-cleverly chosen by the article submitter, but consider the article for a moment.
This is about the accuracy of clinical trial research. This is not about market research studies in the latest clothes fashions. Medicine is an extremely lucrative and risky field -- being associated with the group that pushes through the next Viagra can ensure that your family becomes the next Rockefellers. Your only opposition is the FDA (and the politicians that influence it, which are always hungry for money, which you have lots of).
There is a tremendous amount of pressure on pharmaceutical researchers to produce favorable results. Let's say that you're a new, idealistic researcher who runs some tests on a new drug that your employer wants to market. Your tests show that our drug produces an increased rate of cancer? Well, been nice having you work here...bye. Bob down the hall has consistently gotten us much better results to feed to the FDA for approval. We really don't know how or why he gets better results, but he's definitely the man we want on the job. Sure, maybe twenty years down the road there will be some complaining, but *we didn't know*...*we did all our due dilligence and somehow our results just wound up showing that our drug was okay*.
And even the more innocent "conclusive results" become suspect. A pharmaceutical doesn't want "inconclusive results", where "further tests are recommended". They have a bloody lifetime on the product ticking away, and a competition breathing down their neck. They want some scientist to sign off on this thing with a nice firm "Okay" or "Not Okay", or else what are they paying the guy for? He's not here to do ivory tower work -- he's here to serve the company, which is in the business of extracting savings from aging and achy baby boomers and subsidies paid for by their tax-paying children.
What is being said is that a full third of examined clinical trials were essentially horseshit. This is really not a laughing matter.
Any program relying on (nontrivial) preemptive multithreading will be buggy.
You this is why you should publish in Journal of Irreproducible Results. No can check up on your work. I think balderdash has a nicer ring to it than nonsense. Sounds more academic. 31.4159% (+/- 2.718% margin of error) of all studies are balderdash and poppycock.
OT but amusing: From Without a Clue:
Holmes: How can I be expected to maintain the character when you belittle me in front of those hooligans?
Watson: Character? Are we talking about the same man who once declared with total conviction that the late Colonel Howard had been bludgeoned to death with a blunt excrement?
Holmes: Is it my fault you have such poor handwriting?
"You'll get nothing, and you'll like it!"
Studies on research show that research may cause cancer, be a cult ringleader, is the neighbour from hell, cons unsuspecting single mothers and is causing obesity in our kids. More on A Current Affair, tonight.
The easy part was getting the brain out, but the hard part was getting the brain out.
Comment removed based on user account deletion
66% of the time, it works everytime
A study carried out in Australia back in 1943 found that 50% of all people currently married were female.
Is it a joke or not?
Study Finds Women Who Drink Way More Fun To Study
This is not a dream, not a dream...we are transmitting from the year 1-9-9-9.
Except That. And That. And that and that and that and that and that.
And that.
87.34% of all statistics are made up on the spot!
Just sounds like something that Congress spent millions to find out.
No member of Comgress has ever seen a nickle they couldn't waste. If you send it they WILL spend it!
Professional Politicians are not the solution, they ARE the problem.
1/3 of every study is nonsense
There is something weird about that corner of Turkey...
I want to know what the other two studies concluded.
What makes the study highly questionable? And even if you find it so, what makes the article bad? Rather than simply reporting the claims, they had good coverage of the method used.
i nd.html It makes a pretty good case for active suppression of good quality research suggesting that there is, in fact, a link between breast cancer risk and abortion, and active promotion of a methodologically flawed study as definitive simply because it reports no link.
The "Blinded by Science" article, on the other hand, starts off with an interesting anecdote about what is presented as unreasonable balance, then goes into a rant on global warming that constitutes the bulk of the piece. Furthermore, it focuses exclusively on conservative positions as ridiculous.
It's a standard piece of stealth propaganda. It takes some of the worst excesses of conservative pseudoscience and fringe science and quietly slips opposition to the extremely political and not-especially-scientific Kyoto Protocol into the same category, and for good measure implicitly equates the scientific consensus that human activity is causing some measure of global warming with a consensus that it is causing a global warming emergency, and with mainstream support for extreme claims such as a quarter of all species being wiped out in the next 50 years.
But the initial bit, on breast cancer... are we to simply assume that those wacky conservative legislators have done it again, as he suggests? You don't have to search hard on the internet to find an opposing piece. Here's the first thing I found on google: http://www.leaderu.com/ftissues/ft9705/opinion/br
Looking around more, there seems to be a genuine, legitimate controversy on this point. Studies continue to be published in reputable journals suggesting a link. This is a terrible example of false balance.
Just because the law was written trying to scare girls away from abortions (as I'm willing to assume) rather than from a sincere concern for their health is no excuse for inaccurate reporting about research into the health concerns.
I think it is possible this is the most amusingly ridiculous piece of "legitemate" news I've read in awhile...
Let's try this one... Today's headline: Guess what? Men don't mind seeing naked women...
http://news.yahoo.com/s/nm/20050711/od_nm/italy_n"The majority of interview-based studies have indicated a link - some are statistically significant - but there is debate as to their reliability."
Seriously though, I'm not surprised that 1/3 of all studies are garbage. It's what happens when your society places a higher value on sensationalism than accuracy.
If later research discredits a study, then the system works. Published peer reviewed studies put out an idea, others get to read that idea and respond with other studies or facts.
Science gets things wrong from time to time for a whole host of reasons from accidental to intentional manipulation of data. The review process ensures that we fix the errors and get it all right in the end.
In the 70s and 80s some studies showed we were entering another ice age. In the 90s and 00s some studies show we are in a run-away green-house warming age. I don't think either of them are right.
Article X: The powers not delegated... by the Constitution...are reserved...to the people
If this study itself is nonsense, does it make the other studies good? Looks like a paradox. If the study is right, there is a 33% probability that this study is wrong. And if this study is wrong, then the other studies are right!
So what was my point? Aaaah!!
O this learning! What a thing it is - William Shakespeare
"...innumerable global warming studies that the scientific community can't make up its mind on (for example)." - Bad example, climate scientists "know" the planet is warming and also why it is warming, but fossil fuel politics creates an enormous amount of FUD in an attempt to make you and me think the scientists are contradicting each other and basically haven't got a clue. A similar thing occured when medical scientists said tabacoo was bad for your health. Incredibly some of the same "researchers" who "proved" tabacco was harmless have also been involved in "proving" climate scientists are wrong.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
50% of people are of below average inteligence
They don't mention that 34.8% of statistics are pulled out of the speaker's ass on the spot.
It is pitch black. You are likely to be eaten by a grue.
Now, I was slightly interested in reading this article, but I always read a few posts before clicking anything. The posts here have actually chased me away. I want nothing to do with this article. I'm well aware of the irony of the previous sentence.
From what I gather, this statistic applies to scientific studies. Am I surprised that this happens? Not very. Most people believe that scientists are impartial observers of natural phenomena that follow the evidence where it leads. This is how Science is supposed to work, but it doesn't work like this all the time.
Why do scientists jeopardize there careers by falsifying evidence and studies? Because they are human. Some do this unintentionally by subconsciously ignoring evidence that is contrary to their preconceived hypothesis. Others do this intentionally. They do this because if their hypothesis is found wanting; they will lose prestige and/or funding. Or sometimes the evidence points to concepts that they are unable or unwilling to understand or acknowledge as true. Every scientist has preconceived beliefs on what they will discover in a given study; the good ones do not let these beliefs get in the way of the evidence.
History is littered with studies and discoveries that were later shown to be hoaxes. The Piltdown Man is a very famous one; the supposed early human fossil was created to promote a certain view of human evolution. Another recent hoax was the Chinese "feathered dinosaur" fossils that were heavily promoted by the National Geographic to be ancestral to birds (circa 1996). In fact, National Geographic created an evolutionary timeline based on these fossils and presented it as fact. Not only were these supposed fossils later exposed as elaborate hoaxes, it was revealed that National Geographic had a major lapses in scientific and journalistic ethics. First off, the fossils were stolen and smuggled out of China; secondly they were not verified as being authentic. It was shown that National Geographic knew this and still published the story. Why? Because National Geographic is a big proponent of the theropods to birds view of bird evolution, so any evidence that fit this preconceived belief was accepted and not questioned. If on the other hand the evidence had supported the tree-reptiles to birds view, the fossils would have been very critically examined and rejected.
Of course this is not limited just to archeology; many other branches of science suffer from the same problem. But I can somewhat sympathize with these researchers. Imagine arguing your whole life that your hypothesis is correct, all the evidence for the past 50 years points to your conclusion, and then suddenly new evidence pops up that invalidates your life's work. Would you not be tempted to suppress the evidence? Like I said, scientists are human just like us, and while I do not condone suppressing contrary evidence, I can understand why someone would do just that. On the upside, history shows that evidence cannot be suppressed indefinitely, it will always surface. It may take several decades, as was the case with the plate tectonics theory, but evidence cannot be silenced forever.
a quick study shows that one third of all slashdot headlines are nonsense.
And another third is inaccurate or overblown.
A: 3.215 ± 3%
. . . 95% of the time.
Free Software: Like love, it grows best when given away.
The entire story subject and paragraph on Slashdot is misleading and wrong. On CNN, DeAngelis actually talks about not thinking critically and analyzing the results of single medical studies. The article was NOT about all research in all branches of science.
I think a quote from the actual CNN article sums up this Slashdot story perfectly "(DeAngelis) said the media can complicate matters with misleading or exaggerated headlines about studies."
I mean, literally, that headline would be equally plausible on the Onion as it is here on /. !!
THE MAGIC WORDS ARE SQUEAMISH OSSIFRAGE
So does this obey sand pile physics? ;)
"sweet dreams are made of this..."
Human mind can see (thanks his memory) the past but due to his poor mathematic instict can't see the future. Now things seems in the way to change, but I desagree with KurzeweilAI.net'illusion to find inmortality:I fell we all that live now shall Die,ours children and nephews etc. will die but I also feel Science,thank also computers and networks,is growing so much that just passed this 100-200 years there is a 66% probability to win aging and death. So when we,ours children,ours nephews are going to die but we can feel that then death is going to be win,this is the typical immage of a loosing war,does anybody knows witch war are we loosing?
Because its three studies in one? Therefore ... o wait ...
Caesar si viveret, ad remum dareris.
Didn't you know that 46% of all statistics are made up?
My other comment is funny
This slashdot headline in itself is highly inaccurate.
The cnn article talks about medical studies between 1990 and 2003, not all studies. It says that 16 % of those have been contradicted by other studies and another 16 % had "weaker results". This does NOT mean that 1/3 of all studies are nonsense.
When I think about this and how much CNN might have already altered reality in their article I have to ask myself what news stories I can still take for granted these days...
Yep.
Their own statistics show that there is a 33.33% chance that this study is wrong. How can they be more than 67% sure?
"...who search the reason of things
Are those who bring the most sorrow on themselves." --Euripides, The Medea
Shouldn't that be "...found that a single study cannot be studied as multiple studies " . I don't know ... I'm just asking. But if such were the case, I'd understand what you're saying.
How many beans make five, anyhow ?
Staistics state that 90% of statistics are made up on the spot -.^
This was MEDICAL studies in peer reviewed journals. As such, this is a pretty high percentage.
Now think of all the other studies that are funded by someone with an axe to grind, like those MS funded studies that say linux sux.
If 1/3 of peer reviewed medical studies are innaccurate, I'm betting a MINIMUM of 50% of all other studies are the same.
I'm curious to know how Mr. Gribbin defines a religious nut.
While there are a large number of people who reject facts and reason due to their a priori commitment to a religious beliefs, there are a great number who do the same whose religion is science itself.
That is to say, preconceived notions and personal bias prevent many so-called scientists from acknowledging facts and realizing that their pet theories are baseless.
As an example, I offer Carl Sagan. Here was a man who made a nice living talking about extraterrestrial life. Is there ANY evidence of extraterrestrial life? Is there ANY science that supports it? After all, the best that the SETI institute has is Drake's equation which at best merely multiplies speculation upon speculation.
Is SETI science? Perhaps, but Sagan's beliefs and public discussions were based on fantasy and hope rather than fact.
My point is this: Bias appears in religion and in the name of science. Science has dirty hands, too.
Remember, power tends to corrupt, regardless of world view. I'd be willing to bet that a similar book could be written demonstrating horrible abuses of human rights where science was allowed to 'progress' unchecked by morality.
Finally, it is important to note that much of science has been advanced by people with strong religious convictions. Pascal, Pasteur, Lister, Knuth, Kelvin, Joule, Carver, Bacon, Boyle, and many many others. Strong religious conviction is NOT the antithesis of scientific advancement, as demonstrated by the legacy of those I listed above and I could list many more.
Respectfully,
Anomaly
But Herr Heisenberg, how does the electron know when I'm looking?
In which category is this study in?
This is kinda like that 80-million dollar study that was done back in 2002 to figure out "why convicts want to escape prison."
LOL
Anyone?
Or is 'obvious' more suitable here?
Sorry. It had to be said. :)
Duh! They actually needed a "study" to tell them this?!
The geek shall inherit the earth.
I just found it funny, I'm Elite!
So I guess there's no point in doing a study on the likelyhood of missiles turning into sperm whales or potted plants.
I find it fascinating that rather than address the substance of my posting, you choose instead to speculate on my character and mental condition - about which you know little. Should I consider this to be representative of how scientists think and act?
Do you have any interest in addressing the content of my thoughts rather than presume about my mental state?
Respectfully,
Anomaly
But Herr Heisenberg, how does the electron know when I'm looking?
But it's according to 35% of those polled!
This post encoded with ROT26. If you can read it, you've violated the DMCA. Handcuffs please, sergeant.
I'm sure this will be fodder for ever crackpot trying to sell magic beads as an alternative to regular medical avenues.
I see it differently. This is the strength of the scientific process. If you make a claim, you had not only be able to back it up, but also be able to have people in other labs replicate your results. Yes, surveys are at the far end of the scientific spectrum in terms of validity of provability. It is way to easy to get some sort of statistical anomaly, and accuracy of data is hard to maintain simply based on the fact that people lie ("oh yeah, I only smoke on cigarette a day"). But in the end, the scientific process demands that these results be replicated. This study is simply showing the scientific mechanism in progress.
2) Career. Some studies are done to further a career or make a career. Screw up - make another study! No really, this does happen. I have seen it when I was in college. Congress has funded a number of people's careers that way. Some career studies have to do with "civil rights." Don't like the result, have another study! Listen for "this study is right" sometime. Uh huh.
3) Incompetentcy. Some studies are done by people who have no clue about what they are doing. A famous "consumer" advocate fits into this category that I'm sure we have all heard of. Since he is very litigous I won't name him by name, however he has been a US Presidential candidate. Often times this category should probably go into category 1) Political.
One has to wonder if more studies are wrong than just 1/3.
After reading through the various comments, I notice that there are a number of legitimately serious comments and discussions on the original posting. However, there seems to be a measurable temptation to respond to the humorous self-referential headline. So, not to be left out, I feel compelled (somewhat apologetically) to chime in with yet another silly self-referential comment.
Roughly, a third of all comments are superfluous or redundant.
The notion that corporations are spending money to mislead both the public and the government is indeed frightening. However, you have to expect a certain amount of noise with every discussion.
{ return clarity; }
However, my roommate doesn't understand any of this, and believes that the generator is, in fact, a toaster.
It'll be duped twice.
Just couldn't resist a little progressive/vegan/environmentalist/neopagan bashing, could you? Did you just completely make that up or can you point to something written by these progressives/vegans/environmentalists/neopagans that actually proves your point? Care to provide a link? Could you explain how this reply is AT ALL germain to the grandparent poster? Nah, didn't think so.
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
Today is opposite day. Also: This statement is false.
What the subject says.
Regards,
Ryan Pritchard
Fun Extends All Basic Life Expectancies
Just reading the title makes me think hmm.....is it april 1st and someone didn't tell me? Or did the submitter wake up drunk and think it was april 1st
So, if this study shows that 1/3 of ALL studies are nonsense, then wouldn't that make this study 1/th nonsense?
It turns out they were concentrating too much on the "Craft" part and too little on Networking.
You should really read this book by Horace Freeland Judson.
He outlines the motivations, the excuses, the history, and even plenty of known and not so well known case studies. It is not an easy read; but it is a recent, scholarly, and very comprehensive treatment of this subject.
Judson may have his critics. However, his arguments are quite well thought out and well founded. This is a festering problem, not just with current medical research, but also with many other hard sciences. It discusses the very manner in which we conduct modern scientific research. It even opens up a substantial can of worms by questioning the supposed value of peer review.
Like it or hate it, it's worth your time to read it.
Nearly fifty percent of all graduates come from the bottom half of the class!
They may be highly cited for the simple reason they were obviously wrong to the authors of the contradicting study.
This study says nothing about that larger proportion of papers that are subtly wrong...
Conduct more studies on the subject. No, this isn't a smart ass answer, I'm serious. Any type of study is best confirmed or denied by conducting additional studies, using different samples, varying methodologies, wider scope, etc.
Maybe the correct answer isn't exactly one-third (this is only an estimate after all), but additional studies can help home in on a more precise number.
The ultimate plays for Madden 2006
A study done about other studies, saying that 1/3 of all studies are BS, meaning this CNN study has a 33.33% chance of being BS!! Enlightening!!