Slashdot Mirror
Is Statistical Significance Significant? (npr.org)
More than 850 scientists and statisticians told the authors of a Nature commentary that they are endorsing an idea to ban "statistical significance." Critics say that declaring a result to be statistically significant or not essentially forces complicated questions to be answered as true or false. "The world is much more uncertain than that," says Nicoole Lazar, a professor of statistics at the University of Georgia. An entire issue of the journal The American Statistician is devoted to this question, with 43 articles and a 17,500-word editorial that Lazar co-authored.
"In the early 20th century, the father of statistics, R.A. Fisher, developed a test of significance," reports NPR. "It involves a variable called the p-value, that he intended to be a guide for judging results. Over the years, scientists have warped that idea beyond all recognition, creating an arbitrary threshold for the p-value, typically 0.05, and they use that to declare whether a scientific result is significant or not. Slashdot reader apoc.famine writes: In a nutshell, what the statisticians are recommending is that we embrace uncertainty, quantify it, and discuss it, rather than set arbitrary measures for when studies are worth publishing. This way research which appears interesting but which doesn't hit that magical p == 0.05 can be published and discussed, and scientists won't feel pressured to p-hack.
"In the early 20th century, the father of statistics, R.A. Fisher, developed a test of significance," reports NPR. "It involves a variable called the p-value, that he intended to be a guide for judging results. Over the years, scientists have warped that idea beyond all recognition, creating an arbitrary threshold for the p-value, typically 0.05, and they use that to declare whether a scientific result is significant or not. Slashdot reader apoc.famine writes: In a nutshell, what the statisticians are recommending is that we embrace uncertainty, quantify it, and discuss it, rather than set arbitrary measures for when studies are worth publishing. This way research which appears interesting but which doesn't hit that magical p == 0.05 can be published and discussed, and scientists won't feel pressured to p-hack.
100% of all published incorrect results have a P value above 0.05
0.05 has always intended to be the bare minimum, not a guarantee of absolute truth. If you hit 0.05, and you haven't engaged in P hacking, it indicates that there may be an effect there and that more study is warranted.
Plus they are almost all from biology or medicine. Just because their fields don't seem to understand what statistically significant means does not mean that the rest of us do not. Their example when two results measure the same value but one is within one sigma of a null result and the other is not they claim that people interpret this as two incompatible results!? I do not know of any physicist who would look at those data and make that assertion.
Their paper reads more like a "I wish our colleagues understood simple statistics". Banning certain terms is not going to address the underlying problem they clearly have. The solution to ignorance is education, not censorship as they really ought to know, working in universities!
Worse than that, if you only publish one out of 20 studies, you are reporting noise.
while (sig==sig) sig=!sig;
And this is why there is so little truth to be found in the humanities.
Here's a scenario: A white nationalist kills dozens of Muslims. Someone looks at this and sees evidence that the normalization of fringe views, characteristic of the way president Trump talks, is emboldening these maniacs to act violently. Someone else looks at this and sees evidence that white middle-class uneducated men have been marginalized by our economic system and are at their wits' end, which is the same phenomenon that lead to Trump being elected.
The kind of narrative-based elaborate analyses that you advocate doesn't help us decide which of the points of view above is right, and we carry on with our preconceptions, unable to learn anything.
Narratives allow you to explain the past perfectly using models that have no predictive value. The only way to make progress when trying to understand a complex system is to come up with very simple hypotheses and try to validate them empirically. Of course this is very hard to do, but I think people in the humanities do a poor job and fool themselves into thinking they understand things they don't understand.
Or perhaps the thing you thought was sure, isn't at all and you just proved that your idea was wrong.
A researcher should prove and disprove, not only prove.
Don't fight for your country, if your country does not fight for you.
Of course 0.05 is arbitrary. But researchers have to run studies using budgets that limit the amount of subjects in the study and they also are up against the level of accuracy of the test / instrument / survey. Obtaining extremely low p-values requires one or more of these:
1. Very large sample sizes.
2. Extremely effective intervention that produces huge differences between your groups.
3. Extremely accurate instruments / measures.
4. Lying.
These things all come at a cost, which has to be balanced between doing fewer studies at higher cost or more studies at less cost.
Worse than that, if you only publish one out of 20 studies, you are reporting noise.
All publicly funded research should be published.
Often the failed experiments are more important than the successes.
Where would we be today if Michelson and Morley hadn't published their failure to measure the ether?