Is Statistical Significance Significant? (npr.org)

More than 850 scientists and statisticians told the authors of a Nature commentary that they are endorsing an idea to ban "statistical significance." Critics say that declaring a result to be statistically significant or not essentially forces complicated questions to be answered as true or false. "The world is much more uncertain than that," says Nicoole Lazar, a professor of statistics at the University of Georgia. An entire issue of the journal The American Statistician is devoted to this question, with 43 articles and a 17,500-word editorial that Lazar co-authored.

"In the early 20th century, the father of statistics, R.A. Fisher, developed a test of significance," reports NPR. "It involves a variable called the p-value, that he intended to be a guide for judging results. Over the years, scientists have warped that idea beyond all recognition, creating an arbitrary threshold for the p-value, typically 0.05, and they use that to declare whether a scientific result is significant or not. Slashdot reader apoc.famine writes: In a nutshell, what the statisticians are recommending is that we embrace uncertainty, quantify it, and discuss it, rather than set arbitrary measures for when studies are worth publishing. This way research which appears interesting but which doesn't hit that magical p == 0.05 can be published and discussed, and scientists won't feel pressured to p-hack.

Comments

Statistically, yes.

[Narcocide]

But not always.

I used to think so

[goombah99]

Then I took a course on statistics, and the stats professor told me that 47.37% of all statisticians make up their own statistics.

Re:I used to think so

[dargaud]

On average humans have one tit and one testicle...

Re: I used to think so

[narcc]

I head 47.379

Re:I used to think so

[apoc.famine]

Only if you round to the nearest integer of each.

Re:I used to think so

[Aighearach]

On average humans have one tit

You understanding of mammal bodies is substantially lacking.

P-hacking

[goombah99]

100% of all published incorrect results have a P value above 0.05

Re:P-hacking

[Impy the Impiuos Imp]

It would be nice to see how accurate .005 is with longitudinal studies of papers and their ultimate truthiness down the road.

Like weather predictions of 30% chance of rain at 2 pm, did it actually rain 30% of the time?

Re: P-hacking

[c6gunner]

100% of all published incorrect results have a P value above 0.05

0.05 has always intended to be the bare minimum, not a guarantee of absolute truth. If you hit 0.05, and you haven't engaged in P hacking, it indicates that there may be an effect there and that more study is warranted.

Re:P-hacking

[Anonymous Coward]

30% chance of rain means that 30% of the given area will experience rain at 2pm.

Not that is rains 30% of a given amount of time.

Also, this is regardless of the rain volume.

Re: P-hacking

[Sique]

But 0.05 is as arbitrary as any value. A p-value of 0.05 means, that out of 20 studies, that consider themselves significant because of their p-value, one is a pure statistical fluke. So why not 0.1? Or 0.01? Or even 0.000,001?

Re:P-hacking

[apoc.famine]

Like weather predictions of 30% chance of rain at 2 pm, did it actually rain 30% of the time?

That sort of research is done all the time. Usually it's on far more specific parts of weather models than the overall model. Weather models are ridiculously complicated, and scientists spend a lot of time on minor components of them like modeling aerosols better since they form the nuclei of clouds and thus rain, or the vertical humidity profile, or boundary layer dynamics. There are so many minor processes that make up weather that most of the research effort goes into things that 99.9% of the population never will even know even exist. In conjunction, all of these things will be what predict rain or temperature at a certain time.

However, once in awhile someone revisits the models as a whole, and you get something like this: http://www.inscc.utah.edu/~pu/...

For hurricanes in particular: http://science.sciencemag.org/...
(If you want the pop journalism coverage of that article: https://www.theatlantic.com/sc...)

Re: P-hacking

[WhiplashII]

Worse than that, if you only publish one out of 20 studies, you are reporting noise.

Re: P-hacking

[Anonymous Coward]

No. The p-value is the probability of observing the value, or a more extreme value, given that the null hypothesis is true.

That does not mean it is more random. It could be centered around a different mean, in which case you would say:

Given the data, I am going to reject the null hypothesis in favor of the alternati e hypothesis because the p-value is less than .05

Re: P-hacking

[fropenn]

Of course 0.05 is arbitrary. But researchers have to run studies using budgets that limit the amount of subjects in the study and they also are up against the level of accuracy of the test / instrument / survey. Obtaining extremely low p-values requires one or more of these:

1. Very large sample sizes.

2. Extremely effective intervention that produces huge differences between your groups.

3. Extremely accurate instruments / measures.

4. Lying.

These things all come at a cost, which has to be balanced between doing fewer studies at higher cost or more studies at less cost.

Re: P-hacking

[ShanghaiBill]

Worse than that, if you only publish one out of 20 studies, you are reporting noise.

All publicly funded research should be published.

Often the failed experiments are more important than the successes.

Where would we be today if Michelson and Morley hadn't published their failure to measure the ether?

Re:P-hacking

[mysticgoat]

I'm across the river from Washington, in Oregon.

Here, a 30% chance of rain means that if you be on it raining, you will win 3 out of 10 bets.

Re: P-hacking

[denzacar]

Because 95% accuracy is GOOD ENOUGH for everyday life. And should the results of at least 19 studies agree... that would be more than enough.

Also, p-value of 0.05 doesn't mean one study out of 20 is a pure statistical fluke.
It means that in 1 case out of 20 WITHIN the study - we don't know if it is a statistical fluke.
Which is why studies needs large samples - so that way 2 guys out of 20 who just happen to be allergic to something in the room don't mess up the entire study.
56 guys out of a 1000 on the other hand... Now that's significant.

Objection

[Anonymous Coward]

> 850 scientists and statisticians

Not a statistically significant representation of the scientific community.

Re:Objection

[0ld_d0g]

It probably means that they were not sure of the exact count. Happens when you have to get your article out on a deadline..

All odd numbers are prime

[goombah99]

A prime number is divisible only by itself and 1
1 is prime (by this definition)
3 is prime
5 is prime
7 is prime
11 is prime
13 is prime
9 is experimental error.

The proposition that "all odd numbers are prime" has a P value above 0.05.

Nope.

[dohzer]

Nope. I'll delete it from Wikipedia later today.

Obligatory XKCD cartoon

[nickovs]

882: Significant

Re:Obligatory XKCD cartoon

[Solandri]

That is actually one of the problems with statistical significance. It's only relevant if you're reporting one single result. If you're reporting multiple results, then that creates a second layer of statistical significance, where on average you expect several of those results to surpass your single-sample threshold of significance just by random chance. And so your findings are only noteworthy if you get more than a certain number of results which surpass your single-sample threshold.

If you've got just one result, the confidence level in that result is 95%. If you've got two and one is "statistically significant", the confidence level drops to 90%. If you've got three it drops to 86%. etc. It converges on 1/e when the number of results you're reporting equals the single-confidence level. i.e. For statistical significance, since your threshold is a 1 in 20 chance of obtaining the result by random chance, the odds of getting one "statistically significant" result from 20 results is 1/e. (This might be easier to see in lottery tickets, where if you've go a 1 in 1 million chance of winning, and you buy a million random tickets, your odds of winning are 1/e.) When reporting larger numbers of results, you actually expect more than a single result to surpass your 95% threshold, so the odds of finding just one or even no "statistically significant" results begin to approach zero.

The 1/e value is the only constant point in all these probabilities. The chance for everything else varies depending on your threshold level and the number of results. So the threshold for "statistical significance" actually varies with the number of results you're reporting. But scientists report their results using the single-sample threshold for everything, as if that's the constant point, not 1/e. Which I'm guessing is what TFA (didn't read) is complaining about.

Re:Obligatory XKCD cartoon

[ceoyoyo]

"That is actually one of the problems with statistical significance. It's only relevant if you're reporting one single result."

No, it's not. That's one of the problems with not knowing what you're doing. You're *supposed to* formulate a detailed hypothesis and analysis plan. That plan should include criteria for deciding what tests you'll do, and what combination of tests you will judge to be supporting each part of the hypothesis. Then you perform multiple comparisons correction based on the number of individual tests that contribute to your conclusions.

Just because nobody bothers to do it doesn't mean the procedure doesn't exist.

The Standards of Particle Physics

[Anonymous Coward]

In particle physics, (the field in which I have my Ph.D. but--full disclosure--no longer work), the standard is 3 sigma to claim evidence for an effect, and 5 sigma to claim discovery. Publication of results below 3 sigma is not only encouraged, but required...it's unethical to conceal such results. A null result can be a theory killer.

Re:The Standards of Particle Physics

[Anonymous Coward]

Physics is an outlier, in human bio-mechanics (where I also no longer work) publishing anything but a positive and significant results is outright impossible. This poisoned the field with cooky models that have no predictive power. Consequence of this, I think Boston Dynamics will get to a full body human gait model over variable terrain before actual scientists conducting research in this field.

Re:The Standards of Particle Physics

[habig]

I still do work in particle physics. Yes, I do understand that particles are way easier to be careful with the error propagation on than anything medical. But still, we do spend 90% of the time on any given analysis "embracing uncertainty, quantifying it, and discussing it,", as TFA says. Figuring out the error bars is the hard part, and also usually the part that referees pick at to make sure you did it right.

(BTW: Isn't p=0.05 only a 2-sigma result? Ick.)

Re:The Standards of Particle Physics

[sfcat]

(BTW: Isn't p=0.05 only a 2-sigma result? Ick.)

Its a bit less than 2-sigma. It should be more like 3-sigma (about p=0.01) which would make p-hacking much more difficult as it would take 100 variations to see a probable null hypothesis. Although the exact methods of conversion are complex.

Re:The Standards of Particle Physics

[ceoyoyo]

The publication of inconclusive results is a problem outside physics. Particle physics does have an advantage though: the data and analyses tend to be from only a few places. In parts of physics where Joe Anybody can ask a few undergrads a handful of questions and then write a paper, there's likely less publication of all those inconclusive results.

Quant vs Qual

[Nidi62]

In my International Relations graduate program there was a big push towards quantitative research and analysis; there were two mandatory classes on it. However, I always felt that it broke things down into too simplistic a view, and while it could tell things might be correlated, it never told you why. And with human systems like societies, states, conflict, politics, etc, there are so many inputs, so many factors that contribute to why people act the way they do, what decisions they make, that to boil it down to one or two that are "statistically significant" isn't missing the forest for a tree, it's missing the forest for a bush. Complex systems very often have complex inputs.

That's why I preferred a more qualitative approach: there was no arbitrary line of significance. It allowed you to explore more complicated or elaborate analyses. There was no worry about getting bogged down in what regression method you used or why, whether your math was wrong, or you excluded/included a variable that you shouldn't have. It gives you the chance to simply lay out your theory, your interpretation, and the evidence to back up that interpretation. And best of all, it allows you do it in such a way that it makes your research much more accessible to other people. I also prefer a more narrative style of writing anyway. Now, of course this for a humanities discipline. A more scientific discipline would require significantly more math.

Re:Quant vs Qual

[PacoSuarez]

And this is why there is so little truth to be found in the humanities.

Here's a scenario: A white nationalist kills dozens of Muslims. Someone looks at this and sees evidence that the normalization of fringe views, characteristic of the way president Trump talks, is emboldening these maniacs to act violently. Someone else looks at this and sees evidence that white middle-class uneducated men have been marginalized by our economic system and are at their wits' end, which is the same phenomenon that lead to Trump being elected.

The kind of narrative-based elaborate analyses that you advocate doesn't help us decide which of the points of view above is right, and we carry on with our preconceptions, unable to learn anything.

Narratives allow you to explain the past perfectly using models that have no predictive value. The only way to make progress when trying to understand a complex system is to come up with very simple hypotheses and try to validate them empirically. Of course this is very hard to do, but I think people in the humanities do a poor job and fool themselves into thinking they understand things they don't understand.

Re:Quant vs Qual

[Nidi62]

Here's a scenario: A white nationalist kills dozens of Muslims. Someone looks at this and sees evidence that the normalization of fringe views, characteristic of the way president Trump talks, is emboldening these maniacs to act violently. Someone else looks at this and sees evidence that white middle-class uneducated men have been marginalized by our economic system and are at their wits' end, which is the same phenomenon that lead to Trump being elected.

The kind of narrative-based elaborate analyses that you advocate doesn't help us decide which of the points of view above is right, and we carry on with our preconceptions, unable to learn anything.

You've proven my point: they're both right. When people in power either espouse certain views or give support (whether implicit or explicit) for those views it emboldens others who hold those same views. At the same time, it's a commonly held belief that marginalization, perceived or actual, can lead one to more extremist views. Both of those very likely factored into why the person in your scenario acted the way that he did. Using numbers tries to break everything down into black and white. With people, everything is grey. There is no one right answer, but there may be many correct answers.

And, as I like to say around here at work with our increasing focus on measurable and metrics (which we've fortunately gotten away from somewhat), numbers lie. They may tell you what's happening (assuming you are actually measuring the correct thing) but they cannot tell you why it's happening, and any solution made based on the numbers may not be the truly correct solution.

Re: Quant vs Qual

[phantomfive]

Best description of narratives ever. It also explains why marketers like them so much.

Re:Quant vs Qual

[Kjella]

Narratives allow you to explain the past perfectly using models that have no predictive value. The only way to make progress when trying to understand a complex system is to come up with very simple hypotheses and try to validate them empirically. Of course this is very hard to do, but I think people in the humanities do a poor job and fool themselves into thinking they understand things they don't understand.

A person is not a dice, no matter how much you want it to be. You can ask a fairly simple question like "Would you pose for nude art?" and get a survey answer. But if you break it down there'll be a ton of factors and the more answers you get and the more fine masked you make your model you'll only end up finding more and more differences plus the answer will not remain constant in place or time with a strong group dynamic and feedback loops. And you still will not have found a meaningful answer to why, only a bunch of correlated variables. Qualitative studies do the exact opposite, they don't generalize they ask one and one subject to explain their reasoning and try to summarize them into common sentiments. It's a much more accurate description for each person and the group as a whole. It's just really hard to compare scores because it's not on a measurable willingness scale.

Yes, we've vaguely identified some risk factors that are usually present in a terrorist. We've got a long manifestos on why exactly that person turned into a terrorist. But everyone at risk are somewhere in between, they're not just risk factors and they're not clones of the terrorist. It's something like the Heisenberg's uncertainty principle for the social sciences, the more specific knowledge you have of an individual the less applicable it's to the group and the more general knowledge you have on the group the less accurate it's for the individual. They're both circling what nobody knows for sure, what exactly goes on in somebody else's head. Until we discover mind-reading technology that's going to be an approximation at best. Just because you can sell power tools to most Americans if you throw a dart at a map you could hit an Amish community.

Science is hard

[Sarten-X]

This way research which appears interesting but which doesn't hit that magical p == 0.05 can be published and discussed

The significance value is essentially a measurement of how good a researcher is at their job. Unfortunately, a lot of academics feel that they shouldn't be bothered by silly things like "accountability", because they've chosen the noble ivory tower of research.

If your experiment can't hit that level of certainty, redesign your experiment. Go get more samples, run more simulations, and grow more cultures. Alternatively, go ahead and publish, but include the note that the job isn't actually finished. Use the partial result to justify asking for more funding so you can complete the work.

• Half of your samples died unexpectedly? If you were a better researcher with better lab practices, you'd have had someone check that the equipment stayed plugged in over spring break.
• Nobody responded to your survey? Maybe you should try something more effective than standing in a corner of the local pub for an hour asking the drunks if you can "get something good from them real quick".
• You can't get enough reagents for your chemical process? Perhaps you should have actually budgeted for supplies, rather than host an open-bar party celebrating that you received that grant.
• You ran out of time on the cluster computer? Next time try asking the computer science students to review your program for efficiency, rather than trying to run a direct implementation of your whiteboard notes.

(These are all things I saw first- or secondhand during my time in academia)

I'd be fine getting rid of the p-value, but it would have to be replaced by something else that does an equal job of filtering out the half-assed crank "research" that makes more headlines than discoveries. The only replacement I can think of that wouldn't be vulnerable to similar "hack" methods would be to require that every experiment go through an exhaustive process inspection before, during, and after the run. That's an even more painful thing to deal with than making sure your experiment can produce significant results.

Re:Science is hard

[Anonymous Coward]

This is absolute horseshit. There is often background noise in a measurement that you CAN NOT GET RID OF. Therefore you will never get a perfect 0 p-value. In fact, you will often be unable to reduce it beyond a certain point NO MATTER HOW GOOD YOUR EXPERIMENT IS.

What the article is arguing is that we should not be using a blunt instrument like a p-value which is often a lazy person's (like the parent poster) substitute for quality, but instead should be assessing research on its relative merit and making judgments about quality from a deeper understanding of the problems that some experiments face. Attittudes like the one the parent poster gives are why p-hacking and its associated problems exist - dilletantes like Sarten-X substiute p-values for quality, whereas actual statisticians know it cannot be used in that way.

Re:Science is hard

[Anonymous Coward]

They should publish p-value and effect size. I'm also a big advocate of robust statistics; the assumption of an underlying normal distribution is not always justified for real world data, central limit theorem notwithstanding.

Re:Science is hard

[houghi]

If your experiment can't hit that level of certainty, redesign your experiment.

Or perhaps the thing you thought was sure, isn't at all and you just proved that your idea was wrong.

A researcher should prove and disprove, not only prove.

Re:Science is hard

[AmiMoJo]

Alternatively, go ahead and publish, but include the note that the job isn't actually finished.

That's what they are arguing for. Too many scientists won't submit and too many journals will reject based on the p value alone, which means some interesting ideas and data goes unpublished.

They are not arguing for "crank" research, just that the p value isn't the be-all and end-all, and in fact no single data point should be. Work should be considered on all its merits.

Re:Science is hard

[jeff4747]

The significance value is essentially a measurement of how good a researcher is at their job

Oh dear god no.

Proving something is not true (aka, results are not significant) is an incredibly valuable thing.

Demanding that all researchers produce experiments that prove their hypothesis true and only true is awful, and how you get p-hacking. And it is also what you are demanding here.

Re:Science is hard

[werepants]

The significance value is essentially a measurement of how good a researcher is at their job.

This is totally wrong, and reflects the exact misconception that the article is talking about. For quite a while my job was doing experiments on hardware that cost as much as $100k per sample, where test time would cost $1000/hr or more, and you needed hundreds of hours of testing to get any kind of reasonable certainty. Budgets are finite, and at some point you have to decide how good is good enough, or even if isn't good enough, there just isn't any money left to do better. We could only estimate effects to within a couple orders of magnitude at times. However, we put error bars on fucking everything, so we were very explicit about how much slop there was in the answers. How good a researcher is at their job is determined by how much they can get done with finite resources, and how deeply they understand the limitations of their knowledge. All researchers should be trying to get maximal knowledge per dollar (or per time, in some cases), and sometimes an experiment with large uncertainty is the appropriate approach, or the only thing that is feasible within time/funding/physics constraints.

Sure, if you are doing something basic like surveys, it's not hard to increase statistics. But if you are doing medical research on a new drug, costs can run into billions and you've got major ethical quandaries every step along the way. If you are developing a drug for a rare condition, there might only be a handful of test candidates in the world, and so you literally can't increase your sample size unless you wait a decade for more incidences to crop up. In that interval, depending on the specifics of the disease, people could be suffering or dying needlessly because you haven't gotten your drug approved.

Yes, bad research is bad, and journals are replete with examples of terrible studies being published. But the p-value doesn't help that situation - it makes it worse, because it's treated as a binary marker of success. You can easily produce a great p-value by approaching science in the exact wrong way... look for significant correlations in a large, highly multivariate dataset and you are guaranteed to find some total nonsense correlations that look flawless (like the insanely tight correlation between swimming pool drowning deaths and Nicolas Cage movies... true story).

What we actually need is more rigorous peer review and greater transparency and information sharing in science. If it becomes standard practice to make all of your raw data and calculations public, then it will become obvious very quickly when people are fudging numbers and inflating their stats.

Re:Science is hard

[Sarten-X]

I have a fair coin that always lands on heads, just with about 50% background noise.

The whole point of an experiment is to remove the "background noise", which is another way of saying "uncontrolled variables". If your experiment can't isolate the target variable, then you need to fix your experiment. In the extremely rare case that the experiment can't be fixed, like in cases where a small number of particles matters (including the very small number of photons hitting a telescope sensor), you still should be acknowledging your experimental problems. Own up to having a low p-value, and explain how you did absolutely everything possible with today's technology to pull signal from that background noise.

[We] instead should be assessing research on its relative merit and making judgments about quality from a deeper understanding of the problems that some experiments face.

I agree, but to do that, we'd need a good way to quickly educate every other scientist on that "deeper understanding", and why it's not possible to do any other experiment that does a better job of isolating the variables. Without that, it's easy to simply claim that an overly-complicated random-number generator with cherry-picked results is really an extremely-sensitive test apparatus supporting some pet theory.

Re:Science is hard

[Sarten-X]

What they're arguing for, in their words:

The editors [of The American Statistician] introduce the collection with the caution “don’t say ‘statistically significant’”. Another article with dozens of signatories also calls on authors and journal editors to disavow those terms.

We agree, and call for the entire concept of statistical significance to be abandoned.

TFA summarizes this as a "ban on p-values".

I'm all in favor of evaluating work on its merits, but the p-value is still a useful tool for measuring one of the most important merits: the chance that the result was completely coincidental.

Re:Science is hard

[Sarten-X]

As I said, go ahead and publish, but include the note that the job isn't actually finished. Use the partial result to justify asking for more funding so you can complete the work.

For a while, my job was finding the handful of drug-trial candidates you mentioned. I understand that there are times when you realistically cannot hit a high degree of probability, but that doesn't mean a p-value isn't critically important, and it certainly doesn't mean a p-value should be "banned" from peer-reviewed journals.

If you only ran three trials before funding ran out, that's fine... just make absolutely certain that it's clear that's what happened. In that case, your published result isn't so much a claim that "this is a discovery", but "we did this, and saw this, and more work might show a discovery".

That's how we get the "greater information sharing" that we all love. We don't get it by sweeping poor rigor under the rug.

Re: Science is hard

[phantomfive]

The scientists in the article are complaining that people conclude two things are the same when there is no statistical difference between the two. You can't conclude that: all you can say is "we aren't sure."

Re:Science is hard

[Falconnan]

Well, that's the rub, isn't it? Rare is the experiment that proves an idea true. Most experiments are designed to falsify an hypothesis. Statistical noise comes in and complicates this, and can't always be accounted for. Hell, there's a philosophical case to be made that reality is the result of noise and its cancellation.

It all really comes down to acknowledging that there's always some uncertainty to any measurement, whether due to limits of the measuring device, random noise, or previously unknown variables. This is normal. This is life. Acting like science happens in a vacuum (the quantum physics and astronomy crowds will love that metaphor) is missing the point.

The big problem is we don't talk about degrees of uncertainty in the core research. The next big problem is our focus on positive implications as opposed to negative results, especially as publications go. And it would probably help if we eliminated the stigma attached to retractions. "We found a problem with the study," isn't supposed to be shameful, it's supposed to improve the process. Even in science, shit happens.

Re:Science is hard

[Anonymous Coward]

As I said, go ahead and publish, but include the note that the job isn't actually finished. Use the partial result to justify asking for more funding so you can complete the work.

No! My god, no, that's a terrible idea.

If there is no effect, then your p-value will not show significance. If you run a larger trial, your p-value will still not show significance. If you run your trial over a longer period, your p-value will still not show significance. How much funding should you spend devising more extensive experiments to search for an elusive result?

A statistical test like this can have two results. One is that there is a significant result, beyond some threshold (e.g. p0.05), meaning that there is probably an effect. The other is that there is no significant result, in which case you can place an upper limit (with some confidence) on the size of the effect. Both are valid results that tell us something about the world.

Re:Science is hard

[ceoyoyo]

Meh, you just have to call it something else. Confidence interval. Likelihood ratio. Bayes factor. The dirty little secret is that these things are all mathematically equivalent, or very nearly so, for the vast majority of analyses that are actually conducted.

People like simple solutions. A demon to exorcise. P-values fit that. The real problem is lazy interpretation. Any single result is questionable, no matter how well the data is collected and analyzed. A journal article is not truth, it's an observation. But that uncertainty bothers people.

Re: Science is hard

[ceoyoyo]

They are absolutely right on that score. People do that ALL the time. I tell my students it is the first sin of statistics because I'm sure it's responsible for the vast majority of committed statistical fallacies.

It's the root of the "difference of difference" error, which is apparently present in 50% of neuroscience papers that have the opportunity to make it.

Re: Science is hard

[phantomfive]

What's the difference of difference error?

Re: Science is hard

[ceoyoyo]

Here's a blog (with a link to a published paper) discussing the error and it's incidence in neuroscience: https://www.theguardian.com/co...

Basically, imagine you've got a control group and two different treatments. You determine that treatment group A is not significantly different than control, but treatment group B is. So you conclude that treatment B works better than treatment A. Implicitly, you've assumed that the non-significance of group B means "no difference" or at least "less difference." Both of these options are fallacies.

That *seems* like a silly example, except that it's present in so many papers. But the error can be much more subtle. Have you ever seen a pair of fMRI images side by side, with blobs on one that aren't on the other? fMRI "scans" are actually statistical maps (a t-test per pixel, hopefully mostly corrected for multiple comparisons) with the coloured blobs indicating p some threshold like 0.05. Comparing two of them is committing the difference of differences sin on a massive scale.

Re: Science is hard

[phantomfive]

Huh, so basically they are taking .05 and pretending it's zero. That is fascinating, I will look out for that now. I'm glad we had this conversation.

Re:Science is hard

[epine]

If your experiment can't hit that level of certainty, redesign your experiment. Go get more samples, run more simulations, and grow more cultures.

Ridiculous. You have to budget before data collection. But this approach could be valid, I suppose, anywhere money grows on trees.

Bio/Medical Fields

[Roger W Moore]

Plus they are almost all from biology or medicine. Just because their fields don't seem to understand what statistically significant means does not mean that the rest of us do not. Their example when two results measure the same value but one is within one sigma of a null result and the other is not they claim that people interpret this as two incompatible results!? I do not know of any physicist who would look at those data and make that assertion.

Their paper reads more like a "I wish our colleagues understood simple statistics". Banning certain terms is not going to address the underlying problem they clearly have. The solution to ignorance is education, not censorship as they really ought to know, working in universities!

Re:Bio/Medical Fields

[omnichad]

Statistics in medicine are inherently messier. We don't clone people to do experiments and they don't intentionally kill people. You don't get clean control subjects.

Re:Bio/Medical Fields

[oh_my_080980980]

It's about publishing Potsy. Something you would know if you were an actual researcher. The publishing game is about publishing statistical significant results. If you don't have that, your research does not get published.

If you had RTFA you would know that no one is banning anything:

"We are not calling for a ban on P values."

"Nor are we saying they cannot be used as a decision criterion in certain specialized applications (such as determining whether a manufacturing process meets some quality-control standard)."

" And we are also not advocating for an anything-goes situation, in which weak evidence suddenly becomes credible."

"Rather, and in line with many others over the decades, we are calling for a stop to the use of P values in the conventional, dichotomous way — to decide whether a result refutes or supports a scientific hypothesis."

Reading is fundamental and you failed. Moron.

Re:Bio/Medical Fields

[Roger W Moore]

It's about publishing Potsy. Something you would know if you were an actual researcher.

I am an actual researcher. Given your lack of understanding of statistics and reliance on ad hominem attacks, if you are a researcher too then you are clearly the target audience that this paper is trying to help by reducing your exposure to simple statistical concepts that you are likely to misinterpret.
I never said that they were calling for a ban on p-values, I said that they were calling for an end to "statistical significance". To quote:

We agree, and call for the entire concept of statistical significance to be abandoned.

This is just stupid. You do not stop using a valuable and sensible concept simply because some people who should know better do not properly understand it. Drawing a conclusion from your analysis is a fundamental part of doing science and it is completely proper that an author of a paper should make a statement of their conclusions based on the data. When this is whether a particular hypothesis is correct or not then you have to address the binary nature of the result otherwise you have not done your job. How strong a statement you can make will, of course, depend on how good your data are. This can vary from "the data are consistent with the Standard Model but do not rule out the presence of new physics" to "The $EXOTIC_NEW_PROCESS is ruled out at the 95% confidence level".

Reading is fundamental and you failed.

Please do not project your own failings onto others.

Re:Bio/Medical Fields

[ceoyoyo]

Sad but true. And I do medical research.

One time a particularly annoying research assistant came running down the hall all excited about two recently published papers that showed exactly the situation you mentioned. Look! Contradictory results! Who's wrong? Uh, those results are compatible with each other. One of them had a confidence interval that completely included the other.

Re:Bio/Medical Fields

[ceoyoyo]

No it's not. *Data* in medicine is inherently messier. That makes good statistics more important. In most cases the actual stats are easier: measurements in medicine tend to be so much crap averaged together that the central limit theorem works quite well, Gaussian assumptions are valid, and the t-test reigns supreme.

Re:Bio/Medical Fields

[ceoyoyo]

Yes. They basically want people to stop saying p 0.05 and instead say p = 0.xxxxx. It's a great idea. As far as I can tell, it mostly happened twenty years ago when people learned how to use computers. Every once in a while I review a paper by someone who didn't get the memo and make them include their actual p-values.

Re:Bio/Medical Fields

[omnichad]

That's just a narrow definition of statistics. Gathering that data falls under "statistics" as well.

Re:Bio/Medical Fields

[ceoyoyo]

I absolutely include designing the study and gathering data under the heading of statistics.

Re:Bio/Medical Fields

[Roger W Moore]

Statistics in medicine are inherently messier.

I agree that it is harder to quantify your uncertainties because you have so many variables but this is what leads to incorrect uncertainty values. What we are talking about here is the correct interpretation of a stated uncertainty which is an entirely different problem to whether the stated uncertainty is correct.

Re:Bio/Medical Fields

[omnichad]

I don't think you could ever accurately quantify the uncertainty.

Re:Bio/Medical Fields

[Roger W Moore]

That's actually a given in any experiment in any field: all uncertainties are themselves uncertain to some degree. However, this in no way stops you from being able to correctly interpret what a stated uncertainty means.

This won't address the underlying problem

[SlaveToTheGrind]

Even without a magical "significant/insignificant" threshold, researchers will still evaluate, judge, and compare levels of significance. The pressure will just shift to come up with results that are "MORE significant" rather than "LESS significant," and thus p-hacking will continue by those that were willing to cross that line in the first place.

The root cause is going to remain until peer reviewers force researchers to commit to how they're going to evaluate their measurements before they take those measurements. But the likely outcome would be either a lot less research would get published at all or published research would start to lose some of the imprimatur it now enjoys, including that of the peer reviewers. So that's unlikely to happen.

Re:This won't address the underlying problem

[Falconnan]

I agree 90%. Another thing that would help would be to reduce pressure to always get a solid result at the end. "We did the experiment as designed and approved and we got very little to show for it," needs to be acceptable to avoid fraud, as well as to improve processes. "This experiment fails because 'x'," is a beautiful thing, since it has value: This experiment for this purpose doesn't work, so skip it or improve on it, please.

Re:This won't address the underlying problem

[ceoyoyo]

Nope. What needs to happen is we need to give up on this idea that papers must be True. Scientific papers evolved from personal letters and presentations at scientific society meetings. A published paper is basically "hey guys, I think I found this thing that might be cool. Take a look?"

The key being the last part. Have a look and see if you see the same thing. If a bunch of us do, we might be onto something. If, instead, you get a reputation for finding random crap nobody else can replicate, well....

I prefer to use an average for my mis-information

[Anonymous Coward]

On average, humans have one breast and one testicle.

It's even worse when economic trends are reported in the popular press.

Re:All odd numbers are prime

[colinwb]

1 is prime by that definition, but it's mostly called a unit and defined as *not* prime to make factorising integers into primes unique (up to the order of the factors): Prime number - Primality of 1

These statisticians are idealists

[plague911]

Sure, in a perfect world we would all discuss the exact probabilities. The reality is we all (even professionals in an industry) have a limited attention span. Benchmarks are useful, even imperfect benchmarks. This is just another example of some purists thinking we should move to some idealized but impractical situation

Re:These statisticians are idealists

[plague911]

Nonsense. We trust in science, not because we as humans are perfect, but because its a process we use to try and get as close the truth as we can. Admitting to inherent imperfection in humans in no way takes away from the scientific process. Peoples belief is fading because we have a couple political dogmas floating around that pump out the idea that the goberment/scientists/the illuminati is lying to them. Humans are flappable and yet the scientific process is ineffable when compared to "my neighbor told me"

In defense of the p-value

[psychic_bacon]

I'm really curious about what people think about this comment and my attempt to defend p-values and statistical significance testing as a concept. I used to hate p-values like any respectable scientist, but in teaching intro college stats class (targeted to behavioral science), I've come to appreciate them, for one major reason.

1. We have to take uncertain science and make certain decisions about the conclusions. Science gets simplified to dichotomous decisions. You either approve the drug or not. You either eat eggs or don't eat eggs. The defendant is guilty or not guilty. In each of these cases, we take scientific and other evidence and have to make a decision: do we trust these data. Confidence intervals, odds ratios, etc, help give a picture but they don't give a clear guideline about what to accept.

2. It's really hard to understand (and teach) Bayesian and other approaches. I think that statistical significance is a decent proxy, as long as the limitations are well-understood. I am a big believer in teaching science research to people who have no desire to ever be "researchers", and in order to evaluate their studies, statistical significance is a good proxy. If you are doing an intro biology lab testing whether there are more bacteria on your hands after washing your hands versus hand sanitizer, a t-test with a p .05 criterion is a good approach. It won't get published in JAMA, but it's good for teaching research concepts.

3. Reviewers still want p-values. Each time I have submitted a manuscript without p-values, I get a nasty reviewer who requires p-values. Maybe I've had bad luck, but I'm guessing this is pretty common in the literature. Any time I try a statistical technique that goes beyond null hypothesis testing, there is at least one reviewer who doesn't understand the technique and gripes because there are no p-values or decision criteria. As long as this is required to publish, we need to do it.

So these aren't very good defenses, but it's why I'm still teaching p-values and null hypothesis testing. Maybe we will get rid of it, but like some other comments here, it leaves the question of what the alternative would be.

Re:In defense of the p-value

[jeff4747]

The idea of this proposal is not to abandon p values. It's to stop using p less than 0.05 as a magical threshold.

Loosening that limit is also far more useful in studies involving the "softer" sciences, where it's not possible to control all confounding variables. I wouldn't expect much of a benefit in astrophysics as you'd get in nutrition.

So you'd still report p values, and something with a p-value that indicates the result is basically random probably wouldn't get published. But there'd also be more papers and discussion around things where a particular experiment only reached, say, p=0.1. And that discussion could lead to further research and better designs that get a better p-value.

Or several different approaches to study the subject at hand, each of which with a p over 0.05 because it's studying something where controls are limited or not possible (eg, can't kill your test humans). Several studies with p a bit over 0.05 that show the same result is probably as significant as one study with a p less 0.05 when you're talking about complex and uncontrollable interactions.

Re:In defense of the p-value

[fropenn]

You have to teach it because students need to be able to read articles and interpret what they are reading. The p-value will be reported in scientific studies for a long time to come, so this is an essential skill.

Re: In defense of the p-value

[phantomfive]

Everyone needs to understand statistics in the modern world. People who don't get lost and very, very confused.

The problem is more that people don't understand

[Opportunist]

Mostly, they don't understand that the world isn't black and white.

People want answers. That's a given. And they used to turn to science for this. I say used to, because more and more people think that woo has better answers for their questions. The reason is less that science does not have answers, but that the answers science has require thinking and understanding. They are rarely YES or NO. There's a lot of ifs and buts attached, but people don't want that. They want easy answers.

And reality has rarely easy answers.

"Statistically significant" doesn't mean "resoundingly YES". But that was what was read into it, and of course that expected YES cannot be delivered.

Yes, reading statistics requires some effort by those trying to understand them. Unfortunately that's not what people want to do when they're looking for answers.

Yes

[Chris Mattern]

If you understand what it means and how to apply it. If you blindly slap on the formula and use the resulting number to say, "Look, it's significant!", then, no, it isn't.

Re:Hail incoherentism!

[MightyMartian]

When I took statistics, the text made it clear that a P-value of 0.05 is *somewhat* arbitrary, in that for any individual analysis, it is a useful threshold, but by itself not an absolute indicator of significance. I think the people in this group are guilty of overstating their argument. Determining P-value, or any other statistical measure of significance, is the *start* of a study, and then comes all the hard work of determining if that value is pointing to something truly significant. But a p value of 0.05 is certainly going to suggest that the finding is significant, but it is not THE definitive test.

0.051

[cheaphomemadeacid]

meh just set it to 0.051 and watch 90% of "science" publication burn

Re:0.051

[Actually, I do RTFA]

I'm guessing you don't understand p-values and statistical significance. Setting the limit of publishability to 0.051 would increase the number of papers that passed the test.

Re:0.051

[goose-incarnated]

meh just set it to 0.051 and watch 90% of "science" publication burn

You mean 0.049.

Re:p-hack?

[omnichad]

The wait is over - YOU did!

Re:All odd numbers are prime

[rossdee]

"A prime number is divisible only by itself and 1
    1 is prime (by this definition)"

When I was learning Maths (Mathematics is plural where I come from) I was taught that 1 is not prime, it is a special case.

Anyway for 1 the statement becomes:
1 is divisible by 1 and 1

But 1and1 is now IONOS

 

Re:The problem is more that people don't understan

[andrewbaldwin]

And reality has rarely easy answers.

Which is why engineers answer most questions with "it depends".

Re:Hail incoherentism!

[willaien]

Having P = 0.05 has led to "P-Hacking". One example was the "chocolate" trial that had a small group eating different diets and tracked a whole slew of things, looking for a correlation in any of them. It happened to be that in the small group eating 1.5oz of chocolate and otherwise dieting lost slightly more weight than the group just dieting. Due to not having a specific goal in mind, and small sample size, they were bound to determine some sort of "positive correlation" somewhere, and there you go. If it hadn't been weight loss, it would have been heart rate or something else.

https://io9.gizmodo.com/i-fool...

Re:All odd numbers are prime

[fibonacci8]

https://en.wikipedia.org/wiki/...

Nature is not always Gaussian

[pz]

The issue I find with nearly every single biological application of p-value testing is that either the wrong test is used, or, far more frequently, the necessary validations of the assumptions of the test have not been made. I assume that among those many articles from The American Statistician (a journal that I do not read) that point will have been made because although it is a subtle one, it isn't that subtle, and it is important.

The most commonly used statistical tests assume that unaccounted experimental variability will be Gaussian in nature. That assumption is patently false for the general case. Noise sources for some things are Gaussian -- thermal noise in an electrical signal for example -- but many, many biological sources are not.

When Nature is non-Gaussian, you have to be extra super careful with your tests of significance. And nearly every paper that I've read skips doing noise analysis to validate their tests. Even the lowly mean and standard deviation functions assume Gaussian variability for correct interpretation. The alternative is to have p-values that are so small that results are obvious by inspection --- and then you don't need statistics.

That's the sort of science I strive to perform.

Re:Nature is not always Gaussian

[ceoyoyo]

No, there's a reason most people learn how to do Gaussian stats and then stop. MOST measurements have Gaussian error because that's what you get when you average or add up a bunch of random variables. Most measurements are really composites like that, and the noise is quite Gaussian.

It's very important to recognize situations where that's not true though. Counts, surveys, ordinal scales, data that's been transformed, etc. People are legitimately terrible at doing that.

It's Elemetrary Statistics 101, from 60 years ago

[guacamole]

If you browse around a typical statistics textbook, you will probably find a brief discussion about the difference between statistical significance and real world significance. It seems like a lot of people in sciences, specially in the soft sciences are chasing after the statistical significance because it's now some kind of a prerequisite to get published. However, their findings can amount to very little in the real world. Imagine for example that you find out there the commute distance is statistically significant between people who drink diet coke and tomato juice. Sounds like a great title for a click-bait report. But in reality, your estimates can be 7.34 miles vs 7.36 miles, a difference of 40 meters.

Just an excuse to excuse publishing crap papers.

[Chas]

Yay.

So we can look forward to even MORE broken, badly researched, pointless garbage being published as academically or scientifically relevant.

Look at the finances of any journal pushing this crap. They're probably on borrowed time, in the financial sense.

Re:The problem is more that people don't understan

[Opportunist]

And this is why we don't make good politicians. Politics need easy answers. They needn't be correct or even solve anything, but they have to be easy to understand.

The P-Value Song

[Anonymous Coward]

By Michael Greenacre & Gurdeep Stephens
Barcelona, Catalonia & Victoria, Canada, March 2015
[Video link]

Statistics, logistics, cladistics seem to me
To have a common theme scientifically,
Economists, biologists, with PhD degrees,
They all need some proof of their theories.
A letter is the key, you'll see clearly,
Not B nor G nor V -- but it's the P !

There's no values like P-values
Like no values I know
Think of something that is not worth proving,
An hypothesis that everyone calls null,
If your P is too large to reject it
Then your experiment is rather dull.

There's no values like P-values,
Especially when they are low,
Don't be sad if your P's over point-O-five,
Just try again with samples twice the size,
Everything is possible, just trust in me:
Put your faith in the P.

The F test, the Z test, the chi-square and the T
And other cryptic terminology
Anova, regression, tests distribution-free,
They all need some sort of guarantee.
So if you find a tiny effect size
The P-value will be a good disguise.

There's no values like P-values,
The frequentist's hero,
When you get that data modeling feeling
But results you have are not a lot,
You will need some stats that are appealing
To show the journals your work is hot!

There's no values like P-values
Especially when they're low
Don't be sad if your P's over point-O-five,
Just try again with samples twice the size
Everything is possible, just trust in me:
Put your faith in the P!

Too many studies

[rechtco]

We have too many researchers doing too many studies about the same topic and we incorrectly view each study as a separate event. Without quantification of the number of unpublished, published, and significant studies on a given topic, an individual study's relevance is unknown. If 200 separate researchers did 50 studies each (or 2000 did 5 studies) for a total of 10,000 studies, at .05 p, we could expect 500 false positives. When a study is published without knowing the universe of all studies on that topic, we do not know if any report of a significance level is really significant. Add that there is a bias to report and document positive over negative results. There is also data mining, where an existing database is used to search for any relationship among the historical variables at a p value and then report that relationship. With a large universe of studies and with data mining of historical data, an individual studies significance level is unknown and reproducible results is very low. Combining the data of published studies does not help, since their is a bias is what is reported and published.

Re: Hail incoherentism!

[phantomfive]

The real problem is when scientists aren't interested in finding something significant, they are interested in getting published. In that situation, even setting the threshold at .0005 will end up with p value hacking.

Re:All odd numbers are prime

[MooseTick]

"Prime numbers have to be greater than 1 so 1 is not a prime."

According to your definition. Like most terms, there is no king to give the definitive definition. To me, prime is a cut of meat.

Also, remember that math is just a mental contruct that allows our human minds to interpret the universe around us.

Re: All odd numbers are prime

[gnick]

The even number 2

"All odd numbers are prime" does not imply "no even numbers are prime".

Re:All odd numbers are prime

[thrich81]

Actually 1 is neither prime nor composite by some deep mathematical definitions which go beyond the integers -- they go into the structure of algebraic rings which are generalizations of the integers. If you allow 1 (a unit) to be prime then you break some properties and theorems which everyone generally accepts in the algebra of the integers. The most well known such property is that of unique factorization -- any natural number is factored uniquely into prime factors. If you let 1 be prime then the prime factorization of a composite number can have any number of factors of 1 in it.

The deeper definition of a prime (from my old abstract algebra book) is, "In the Euclidean ring R a nonunit p is said to be a prime element of R if whenever p = ab, where a, b are in R, then one of a or b is a unit in R."

And there is a king which gives the definitive definition -- it is the accepted body of mathematical definitions by the world's mathematical community. There are sometimes differing definitions of a term, but those differences are usually well spelled out in any discussions. You can choose not to accept the definitions as the professionals in the field use them but then don't claim your definition is as good or useful as that of the pros.

It's not p, it's n

[WillAffleckUW]

I don't care how significant your p value is, if your n is less than 40 case/control match your values are meaningless, other than proof of concept for further study.

Wake me up when you get 256/256 fully matched case/control with true randomization. Then we'll talk p values.

Re:Not Statistically Significant

[WillAffleckUW]

To get a truly significant number, you would need 850 scientists, and 850 controls (or non-scientists). And you would need a truly randomized sample of both. If all the scientists are the same age BMI and gender, it's not even close to randomized. Throw in some post-docs.

Re:Hail incoherentism!

[ceoyoyo]

p 0.05 is supposed to be kind of a minimum threshold. Higher than that and you really can't draw any conclusions. Less than that, and you might have something. Maybe. It's basically a first level filter.

Does that mean you get some false negatives? Absolutely. And you also get lots of false positives.

There seem to be a bunch of people who want to look at confidence intervals and say "well, a good part of my confidence interval is over here, which is interesting, so this is important!" There are also a bunch of people who think that too many false positives get published and want to clean things up. These two things are at odds with each other.

Re:Hail incoherentism!

[ceoyoyo]

I much prefer Bayes factor hacking. Sounds way fancier.

Lies, damned lies, and statistics

[NewYork]

Lies, damned lies, and statistics is a phrase describing the persuasive power of numbers, particularly the use of statistics to bolster weak arguments
https://en.m.wikipedia.org/wik...

Re:Hail incoherentism!

[parkinglot777]

Then don't use the word statistical significance to express the word "real". This "p value exaggerate" has been discussed over 10 years ago, but the p-value is still being used because many people, like you, are, as you are saying, have to accept some thing or method in order to be "real". As MightyMartian said, p-value could be used in initial test, but it should never be used as "statistical significance" at all.

Doesn't "significant" mean "important"?

[RespekMyAthorati]

I've always thought "statistical reliability" was a better name.