Science and the Shortcomings of Statistics

Kilrah_il writes "The linked article provides a short summary of the problems scientists have with statistics. As an intern, I see it many times: Doctors do lots of research but don't have a clue when it comes to statistics — and in the social science area, it's even worse. From the article: 'Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted. As a result, countless conclusions in the scientific literature are erroneous, and tests of medical dangers or treatments are often contradictory and confusing.'"

Comments

Lies, Damned Lies, and Statistics.

[Shadow of Eternity]

In other news math may not lie but people still can, all the honesty and good statistics in the world doesnt help end-user stupidity, and there are statistically two popes per square kilometer in the vatican.

What it actually said

[williamhb]

Contrary to the parent poster's claim, the article does not focus on correlation vs causation. It focuses on people getting the correlation wrong in the first place. It lists several common mistakes scientists make when writing up research studies. (Not all scientists are very good at stats). These include:

• If you run enough studies you are almost certain to find a difference that appears statistically significant at the p<0.05 level through chance alone. (It is incredibly unlikely that you will win the lottery; but across the whole pool of tickets someone wins it most weeks.) That makes studies that bulk analyze large amounts of data against many different factors, actively hunting for something that is significantly different, erroneous.
• "p < 0.05" does not mean there is a 95% chance of your result being "true"; it just means that someone else rolling dice has a 5% chance of achieving the same result through chance alone.
• Tests are often combined in ways that are mathematically inconsistent
• Finding a statistical effect does not mean it is a strong effect
• You cannot simply compare effect sizes between two studies because the results of their control groups may differ ("effect size analysis" is usually wrong)
• Failing to find a significant effect does not mean there is no effect ("we found there was no significant effect on..." is misleading because "no satistical significance" is "no information" [your study didn't tell anybody anything] not "no effect" -- to prove "no effect" you need a different statistical test)

And lots of others. It then suggests Bayesian reasoning as an alternative to traditional statistical tests.

Most post-PhD scientists are aware of the common mistakes, but being aware that we make mistakes doesn't necessarily stop us from making them. If you chose a random set of conference proceedings, it is almost certain you will find at least one paper (and I suspect usually a dozen or more) that have statistical mistakes in them.

Re:Summery?

[Saroful]

And what's the law about spelling/grammar corrections that incorrectly correct the supposed spelling error? (Redundancy is purposefully deliberate.) "Its" is possessive. "It's" is a contraction of "it" and "is". -- This has been a message from your friendly neighborhood Spelling Nazi.