Slashdot Mirror
Science and the Shortcomings of Statistics
Kilrah_il writes "The linked article provides a short summary of the problems scientists have with statistics. As an intern, I see it many times: Doctors do lots of research but don't have a clue when it comes to statistics — and in the social science area, it's even worse. From the article: 'Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted. As a result, countless conclusions in the scientific literature are erroneous, and tests of medical dangers or treatments are often contradictory and confusing.'"
In other news math may not lie but people still can...
Usually (in science at least) it's not even a matter of lying. Part of the problem is that the multi-headed monster that statistics has become has a tendency to lead people to over-use numerical "answers" vomited up by stats packages, without really understanding what they are for, or how to interpret them.
Statistics are very useful for predicting certain things, but all too often they are submitted as "proof" of a given condition, which is dangerous. Sometimes we need to throw away statistics and start applying common sense.
You are a jerk.
You are insulting your sister because she is bad at mental math? It is a skill; one not required for extensive knowledge of the social sciences. Additionally, maybe if sales tax is simple in your state like 10%, but where I live it is 4.5% which is not always easy to get exactly right in your head.
I had a roommate who was brilliant,funny, a singer and an artist, and yet, he couldn't calculate tip to save his life, but I don't certainly hold that against him.
You are missing the point - he did not know what a standard deviation means! That is unforgivable for anyone with a medical degree...hell, it's unforgivable for anyone who has passed a course in statistics in school.
...And then it struck me - most of the research I had read had applied parametric statistical tests to their data - that it, the researchers made an assumption that the underlying distribution of results would fall on a normal curve. Yet this simple assumption may be all it takes to skew the data when they should have chosen a non-parametric test instead.
So yes, stats are vitally important, badly taught, and focus too much on the maths rather than the concepts. Remember that we're doctors, not mathematicians - the last set of sums I did were in high school. If I need to analyse data, I'll probably plug it into SPSS - although now with my eyes open.
That's a good insight. I'm a statistics professor, and some of the problems I see are a) people generally get exposed to a single course in statistics; b) they're usually mathematically unprepared for it; c) so much gets squeezed into that one opportunity that heads are exploding; d) because of (a) - (c), everybody wants you to "just give 'em the formula"; e) since statistics is so widely used, there's a plethora of courses that are being taught by people who themselves are victims/products of (a) - (d), and are very happy to "just give 'em the formula"; and so e) most people plug and chug data through a stats package with no idea of the applicability, limitations, and interpretation of the results. The sheer volume of bad analyses is enough to make you weep, and contributes to the widely held perception about "lies, damned lies, and statistics". And that completely ignores the intentional falsehoods propagated by people who are trying to support various advocacy viewpoints, and will happily mislead the public with biased samples, Simpson's paradox, invalid assumptions, etc.
And 77.335% of all statistics claim more accuracy than their expected deviation warrants.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
There are some things you should never be able to forget - the definitions and meanings of probability, mean, median, standard deviation and variance come to mind. You find yourself in situations everyday where you need to apply some of these things. Am I wrong about this? Do people forget basic definitions so easily?
Standard deviation is what you learn very early in school.
So early in fact that by you forget the details by the time you have had some serious study under your belt. Do you have any idea of the stuff you have to keep in your head to be an endocrinologist? So long as he remembers that it's a measure of variance (which he obviously does), it hardly matters whether he can explain to a mathematician how to derive it? And if OP gets off tripping up specialists with such minutae it ain't the specialist who has issues.
And you are telling me that it's not his "job" to know?
YMMV, but I would prefer to visit an endocrinologist who was an expert on the subject of hormones etc rather than stats.
Better to be despised for too anxious apprehensions, than ruined by too confident a security. --Edmund Burke
Actually the subtler issue here has nothing to do with statistics, they are implying peer-review does not work.
"Peer review" is another of the things that has been over-sold to the public. A science research group spends six months and a hundred thousand dollars conducting a research study using highly specialised equipement. They submit a paper to an academic conference or a small journal. It gets put out to review by three people who each spend about four hours reading it and reviewing it, and who usually do not have access to the equipment or the original data that was used in the study. Do you really think we're likely to catch every mistake at review? We certainly can't check the stats (except for the most egregious errors) because we don't have the full data tables they analyzed.
Scientists actually accept that inevitably some incorrect results will be published. More often in the smaller conferences than in the most prestigious journals, but even the journals have to publish a retraction every now and then. We also accept that most studies are never repeated, and so the "objective repeatable experiment" is rarely really tested for being either objective or repeatable. However, science has long had the "many eyes" effect at work. There are hundreds of thousands of scientists reading papers and using them in our own experiments. If some theorised effect out there is wrong, usually we'll find out eventually.
People who deal with raw physical measurements (radar engineers, astronomers, the guy who makes airspeed sensor of the B2--er,um...) have had this problem figured out for a while.
The result, repeatedly proven mathematically and by experience, is that the magic number is always Signal-to-Noise-Ratio. You can't get good information from crappy, scant, data.
Humanities and social-"science" types, and unfortunately the med school set, are by and large composed of people with varying degrees of pathological fear of mathematics, computation, and computer programming. I'd be willing to bet that a largish portion of even the post-PhD scientists who 'know' how to make a proper calculation for a statistical test don't really understand the physical meaning of the numbers they're copying and pasting in and out of excel.
When your attention and skill set are focused on looking through a microscope, or cutting up lab rats, or synthesizing chemicals, you probably never have the experience of being up to your eyeballs in noise estimates and P_FA's that bludgeon in the fact that your data really sucks because it's too noisy, and never need to answer fundamental questions like 'what's the probability that the ruskies will fire off a missile and this radar won't see it'/[insert biologically relevant example here], which *requires* learning the right way to do statistics.
I am in the life sciences (ie not a computer programmer).
If MD's are reading medical journals and interpreting their results, which they all are expected to do (especially those with a Board Certified Specialty like Endocrinology) then there is no excuse for them to have forgotten what what the standard deviation is a measure of. They should be using the variance estimates provided in a data table to interpret the results it contains every time they read an article. If not, then they aren't worth the exorbinant fee's they are charging, because critical thinking is part of a physicians job description, and accepting whatever gets publish in the New England Journal of Medicine at face value is not.
I can accept forgetting the equation, but there is NO EXCUSE for forgetting that SD is a measure of varition (along with SEM, SED, and CV) as opposed to a measure of central tendancy (mean, mode, median). That is something they teach you in the first week of a statistics course, and is used every subsequent class because it is so fundamental to the interpretation of statistics. If I were cytoman, I'd be looking for a new Endocrinologist.
Bureaucracy expands to meet the needs of the expanding bureaucracy.-Oscar Wilde
The problem is that a lot of people believe statistics produced by an expert such as a doctor. Sri Roy Meadow had people sent to prison, and lots of children taken away from their parents, by misinterpreting statistics.
There's a reason why you keep getting modded up and those disagreeing with you keep getting modded down.
You're exactly right. Modern diagnostic medicine is predicated on interpreting statistical studies to make diagnoses. It is practically incompetence for a practicing medical doctor to not know what standard deviation means.
Has it ever been demonstrated that social scientists have a worse understanding of statistics than physical scientists? I ask because my observations are the opposite. The physical scientists run a t-test and declare the matter resolved (significant or not-significant). Given the complexities of social sciences, these scientists check the assumptions required to use a test (e.g., normalcy) and have a good understanding of the statistics involved. (The obligatory exception is statistical genetics: physical science with a solid statistical basis.)
I believe that forgetting something usually means you never really understood it. I don't think that if you really understand something, you'll ever forget it. There are things I do rather rarely yet I don't forget them because I understand them and could re-derive them from first principles.
Usually if I forget something, it means I never quite understood it in the first place.
I think that real understanding implies almost indefinite retention, and lack of retention can be usually be explained by lack of understanding.
It's very easy to forget something if all you know about it is a memorized definition and an equation and two.
If you use statistical terms and concepts daily, you should be able to explain them after being woken up in the middle of your sleep, after several hours of partying with lots of booze. Anything less probably means you're acting things out rather than understanding them.
Feynman often talked about the issue of real understanding. One could summarize his view thusly: if you cannot explain it to a non-specialist of decent intelligence, you probably don't understand it.
A successful API design takes a mixture of software design and pedagogy.