Slashdot Mirror
Is Statistical Significance Significant? (npr.org)
More than 850 scientists and statisticians told the authors of a Nature commentary that they are endorsing an idea to ban "statistical significance." Critics say that declaring a result to be statistically significant or not essentially forces complicated questions to be answered as true or false. "The world is much more uncertain than that," says Nicoole Lazar, a professor of statistics at the University of Georgia. An entire issue of the journal The American Statistician is devoted to this question, with 43 articles and a 17,500-word editorial that Lazar co-authored.
"In the early 20th century, the father of statistics, R.A. Fisher, developed a test of significance," reports NPR. "It involves a variable called the p-value, that he intended to be a guide for judging results. Over the years, scientists have warped that idea beyond all recognition, creating an arbitrary threshold for the p-value, typically 0.05, and they use that to declare whether a scientific result is significant or not. Slashdot reader apoc.famine writes: In a nutshell, what the statisticians are recommending is that we embrace uncertainty, quantify it, and discuss it, rather than set arbitrary measures for when studies are worth publishing. This way research which appears interesting but which doesn't hit that magical p == 0.05 can be published and discussed, and scientists won't feel pressured to p-hack.
"In the early 20th century, the father of statistics, R.A. Fisher, developed a test of significance," reports NPR. "It involves a variable called the p-value, that he intended to be a guide for judging results. Over the years, scientists have warped that idea beyond all recognition, creating an arbitrary threshold for the p-value, typically 0.05, and they use that to declare whether a scientific result is significant or not. Slashdot reader apoc.famine writes: In a nutshell, what the statisticians are recommending is that we embrace uncertainty, quantify it, and discuss it, rather than set arbitrary measures for when studies are worth publishing. This way research which appears interesting but which doesn't hit that magical p == 0.05 can be published and discussed, and scientists won't feel pressured to p-hack.
Then I took a course on statistics, and the stats professor told me that 47.37% of all statisticians make up their own statistics.
Some drink at the fountain of knowledge. Others just gargle.
100% of all published incorrect results have a P value above 0.05
Some drink at the fountain of knowledge. Others just gargle.
A prime number is divisible only by itself and 1
1 is prime (by this definition)
3 is prime
5 is prime
7 is prime
11 is prime
13 is prime
9 is experimental error.
The proposition that "all odd numbers are prime" has a P value above 0.05.
Some drink at the fountain of knowledge. Others just gargle.
Nope. I'll delete it from Wikipedia later today.
882: Significant
If intelligent life is too complex to evolve on its own, who designed God?
This way research which appears interesting but which doesn't hit that magical p == 0.05 can be published and discussed
The significance value is essentially a measurement of how good a researcher is at their job. Unfortunately, a lot of academics feel that they shouldn't be bothered by silly things like "accountability", because they've chosen the noble ivory tower of research.
If your experiment can't hit that level of certainty, redesign your experiment. Go get more samples, run more simulations, and grow more cultures. Alternatively, go ahead and publish, but include the note that the job isn't actually finished. Use the partial result to justify asking for more funding so you can complete the work.
(These are all things I saw first- or secondhand during my time in academia)
I'd be fine getting rid of the p-value, but it would have to be replaced by something else that does an equal job of filtering out the half-assed crank "research" that makes more headlines than discoveries. The only replacement I can think of that wouldn't be vulnerable to similar "hack" methods would be to require that every experiment go through an exhaustive process inspection before, during, and after the run. That's an even more painful thing to deal with than making sure your experiment can produce significant results.
You do not have a moral or legal right to do absolutely anything you want.
Plus they are almost all from biology or medicine. Just because their fields don't seem to understand what statistically significant means does not mean that the rest of us do not. Their example when two results measure the same value but one is within one sigma of a null result and the other is not they claim that people interpret this as two incompatible results!? I do not know of any physicist who would look at those data and make that assertion.
Their paper reads more like a "I wish our colleagues understood simple statistics". Banning certain terms is not going to address the underlying problem they clearly have. The solution to ignorance is education, not censorship as they really ought to know, working in universities!
Even without a magical "significant/insignificant" threshold, researchers will still evaluate, judge, and compare levels of significance. The pressure will just shift to come up with results that are "MORE significant" rather than "LESS significant," and thus p-hacking will continue by those that were willing to cross that line in the first place.
The root cause is going to remain until peer reviewers force researchers to commit to how they're going to evaluate their measurements before they take those measurements. But the likely outcome would be either a lot less research would get published at all or published research would start to lose some of the imprimatur it now enjoys, including that of the peer reviewers. So that's unlikely to happen.
1 is prime by that definition, but it's mostly called a unit and defined as *not* prime to make factorising integers into primes unique (up to the order of the factors): Prime number - Primality of 1
Sure, in a perfect world we would all discuss the exact probabilities. The reality is we all (even professionals in an industry) have a limited attention span. Benchmarks are useful, even imperfect benchmarks. This is just another example of some purists thinking we should move to some idealized but impractical situation
I'm really curious about what people think about this comment and my attempt to defend p-values and statistical significance testing as a concept. I used to hate p-values like any respectable scientist, but in teaching intro college stats class (targeted to behavioral science), I've come to appreciate them, for one major reason.
1. We have to take uncertain science and make certain decisions about the conclusions. Science gets simplified to dichotomous decisions. You either approve the drug or not. You either eat eggs or don't eat eggs. The defendant is guilty or not guilty. In each of these cases, we take scientific and other evidence and have to make a decision: do we trust these data. Confidence intervals, odds ratios, etc, help give a picture but they don't give a clear guideline about what to accept.
2. It's really hard to understand (and teach) Bayesian and other approaches. I think that statistical significance is a decent proxy, as long as the limitations are well-understood. I am a big believer in teaching science research to people who have no desire to ever be "researchers", and in order to evaluate their studies, statistical significance is a good proxy. If you are doing an intro biology lab testing whether there are more bacteria on your hands after washing your hands versus hand sanitizer, a t-test with a p .05 criterion is a good approach. It won't get published in JAMA, but it's good for teaching research concepts.
3. Reviewers still want p-values. Each time I have submitted a manuscript without p-values, I get a nasty reviewer who requires p-values. Maybe I've had bad luck, but I'm guessing this is pretty common in the literature. Any time I try a statistical technique that goes beyond null hypothesis testing, there is at least one reviewer who doesn't understand the technique and gripes because there are no p-values or decision criteria. As long as this is required to publish, we need to do it.
So these aren't very good defenses, but it's why I'm still teaching p-values and null hypothesis testing. Maybe we will get rid of it, but like some other comments here, it leaves the question of what the alternative would be.
When I took statistics, the text made it clear that a P-value of 0.05 is *somewhat* arbitrary, in that for any individual analysis, it is a useful threshold, but by itself not an absolute indicator of significance. I think the people in this group are guilty of overstating their argument. Determining P-value, or any other statistical measure of significance, is the *start* of a study, and then comes all the hard work of determining if that value is pointing to something truly significant. But a p value of 0.05 is certainly going to suggest that the finding is significant, but it is not THE definitive test.
The world's burning. Moped Jesus spotted on I50. Details at 11.
And this is why there is so little truth to be found in the humanities.
Here's a scenario: A white nationalist kills dozens of Muslims. Someone looks at this and sees evidence that the normalization of fringe views, characteristic of the way president Trump talks, is emboldening these maniacs to act violently. Someone else looks at this and sees evidence that white middle-class uneducated men have been marginalized by our economic system and are at their wits' end, which is the same phenomenon that lead to Trump being elected.
The kind of narrative-based elaborate analyses that you advocate doesn't help us decide which of the points of view above is right, and we carry on with our preconceptions, unable to learn anything.
Narratives allow you to explain the past perfectly using models that have no predictive value. The only way to make progress when trying to understand a complex system is to come up with very simple hypotheses and try to validate them empirically. Of course this is very hard to do, but I think people in the humanities do a poor job and fool themselves into thinking they understand things they don't understand.
The real problem is when scientists aren't interested in finding something significant, they are interested in getting published. In that situation, even setting the threshold at .0005 will end up with p value hacking.
"First they came for the slanderers and i said nothing."
Actually 1 is neither prime nor composite by some deep mathematical definitions which go beyond the integers -- they go into the structure of algebraic rings which are generalizations of the integers. If you allow 1 (a unit) to be prime then you break some properties and theorems which everyone generally accepts in the algebra of the integers. The most well known such property is that of unique factorization -- any natural number is factored uniquely into prime factors. If you let 1 be prime then the prime factorization of a composite number can have any number of factors of 1 in it.
The deeper definition of a prime (from my old abstract algebra book) is, "In the Euclidean ring R a nonunit p is said to be a prime element of R if whenever p = ab, where a, b are in R, then one of a or b is a unit in R."
And there is a king which gives the definitive definition -- it is the accepted body of mathematical definitions by the world's mathematical community. There are sometimes differing definitions of a term, but those differences are usually well spelled out in any discussions. You can choose not to accept the definitions as the professionals in the field use them but then don't claim your definition is as good or useful as that of the pros.
Narratives allow you to explain the past perfectly using models that have no predictive value. The only way to make progress when trying to understand a complex system is to come up with very simple hypotheses and try to validate them empirically. Of course this is very hard to do, but I think people in the humanities do a poor job and fool themselves into thinking they understand things they don't understand.
A person is not a dice, no matter how much you want it to be. You can ask a fairly simple question like "Would you pose for nude art?" and get a survey answer. But if you break it down there'll be a ton of factors and the more answers you get and the more fine masked you make your model you'll only end up finding more and more differences plus the answer will not remain constant in place or time with a strong group dynamic and feedback loops. And you still will not have found a meaningful answer to why, only a bunch of correlated variables. Qualitative studies do the exact opposite, they don't generalize they ask one and one subject to explain their reasoning and try to summarize them into common sentiments. It's a much more accurate description for each person and the group as a whole. It's just really hard to compare scores because it's not on a measurable willingness scale.
Yes, we've vaguely identified some risk factors that are usually present in a terrorist. We've got a long manifestos on why exactly that person turned into a terrorist. But everyone at risk are somewhere in between, they're not just risk factors and they're not clones of the terrorist. It's something like the Heisenberg's uncertainty principle for the social sciences, the more specific knowledge you have of an individual the less applicable it's to the group and the more general knowledge you have on the group the less accurate it's for the individual. They're both circling what nobody knows for sure, what exactly goes on in somebody else's head. Until we discover mind-reading technology that's going to be an approximation at best. Just because you can sell power tools to most Americans if you throw a dart at a map you could hit an Amish community.
Live today, because you never know what tomorrow brings