Slashdot Mirror
Social Science Journal 'Bans' Use of p-values
sandbagger writes: Editors of Basic and Applied Social Psychology announced in a February editorial that researchers who submit studies for publication would not be allowed to use common statistical methods, including p-values. While p-values are routinely misused in scientific literature, many researchers who understand its proper role are upset about the ban. Biostatistician Steven Goodman said, "This might be a case in which the cure is worse than the disease. The goal should be the intelligent use of statistics. If the journal is going to take away a tool, however misused, they need to substitute it with something more meaningful."
This is social science. Mathematics and statistics aren't even relevant.
http://xkcd.com/1478/
I thought libtards love statistics and studies? Someone must have been submitting work they could not spin their own way, so I don't blame them for protecting their interests.
It is the job of the reviewer to check that the statistic was used ion the proper context. not to check the result, but the methodology. It sounds like social journal simply either have bad reviewer or sucks at methodology.
C. Sagan : A demon haunted world:
http://www.amazon.com/gp/product/0345409469/
visit randi.org
My immediate thought would be that hard math in this field doesn't tow the groupthink by revealing too much that they want to be able to argue around, so their solution is to try to eliminate the math.
I don't know that this is the case or anything: it's just the only real motivation that would lead to this. Like, the studies show stuff that no one wants to talk about or the math prevents people from coming to a conclusion opposite reality.
It's a war, I tell you, a war on frequentists! I'm 95% certain!
https://xkcd.com/882/
What's the point of a journal if not for peer review? Isn't that the point? Ohhhhh, sociology. I get it now, ok cool.
Don't worry, we'll find another panacea statistic.
From a blog by a colleague of mine on the subject: "Questions that p-values can answer" != "Interesting questions about the world'.
"Look at this experimental evidence and tell me what you see?"
Revolution is the opium of the intellectuals.
Given enough banning of math and science, all libtard fantasies may proceed to become their destined colossal failures.
miss used them, it is right to ban them. To not ban them is to support racism.
At least one president of the American Psychological Association published a statistics book intelligent enough that it used to be required in university statistics intro classes: http://books.google.com/books/about/How_to_use_and_misuse_statistics.html
Not that he would have disagreed with the comment about social psychologists...
Ok, let me enlighten the readers a bit. The reviewers tend to be the typical researcher within the field. The typical social researcher does not have a very strong math background. There is a lot of them into qualitative research and quantitative tends to stop at ANOVA. I have multiple masters in business and social science and worked on a Ph.D. in social science (Being vague here for a reason). However, I have a dual bachelors in comp sci and math. I know statistical analysis very well. My master's thesis for my MBA was an in-depth analysis of survey responses. 30 pages of body and really good graphs. My research professor, an econometrics professor, and I submitted it to a second tier journal associated with the field I specialized in...
... 6 pages got published. 6?!? They took out the vast majority of the math. Why? "Our readers are really bad at math," said the editor. If you knew the field... you would be scared shitless. The reviewers suggested we took out the math because it confused them. This is why they want P value out... it is misunderstood and abused. The reviewers have NO idea if it is being used correctly.
just because you use the word science doesn't make something science, social science is not science.
Blindfold. Check.
Math textbook. Check.
Bedroom. Check.
Girlfriend to enjoy my fetish with??? Oh wait, this is Slashdot.
This is why we can't have nice things.
used to be required in university statistics intro classes: http://books.google.com/books/about/How_to_use_and_misuse_statistics.html
I suspect that book is still foundational in most University advertising/marketing progams.
the growth in cynicism and rebellion has not been without cause
Just because you put the word science behind your name, doesn't mean you're doing scientific work. Psychology is great example of where the term "science" has been grossly misused and misdirected. Psychology is really about the pursuit to understand why we're all fucked up and explain away behaviour that no one wants to take ownership of. If this publication is going to block anything, block anyone using the term science, because psychology is not science, it's people trying to make excuses about the way they feel and act, which is all boils down to scape goating responsibility for your actions.
With the exception of chemical imbalance, every single person is directly responsible for there actions, case closed, now lets stop using the term science to describe excuse generation.
While p-values are routinely misused in scientific literature, many researchers who understand its proper role are upset about the ban.
Do they also know whether "p-values" is plural or singular?
systemd is Roko's Basilisk.
I don't think you even need to be pushing people to do Bayesian stats. You just need to force them to graph their data properly. In *a lot* of biological and social science sub-fields it's standard practice to show your raw data only in the form of a table and the results of stats tests only in the form of a table. They aren't used to looking at graphs and raw data. You can hide a lot of terrible stuff that way, like weird outliers. Things would likely improve immediately in these fields if they banned tables and forced researchers to produce box plots (ideally with overlaid jittered raw data), histograms, overlaid 95% confidence intervals corresponding to their stats tests, etc, etc.
Having seen some of these people work, it's clear that many of them never make these plots in the first place. All they do is look at lists of numbers in summary tables. They have no clue in the first place what their data really look like, and know good knowledge of how to properly analyse data and make graphs. Before they even teach stats to undergrads they should be making them learn to plot data and read graphs. It's obvious most of them can't even do that.
soylentnews.org
Use of the p-value gave us conclusions that weren't politically correct. We have corrected the issue by banning the use of the p-value so that only True Science may be published.
...and this isn't even the first journal to do this. It's probably happening now because an entire book has just come out walking people how universally abused p-values are as statistical measures.
http://www.statisticsdonewrong...
The book is nice in that it does give one replacements that are more robust and less likely to be meaningless, although nothing can substitute for having a clue about data dredging etc.
rgb
Even when the experts all agree, they may well be mistaken. --- Bertrand Russell.
IMO the main problem with p-values is it equates p(a|b) with p(b|a) ... disregarding that different values of p(b) and p(a) could make that equation woefully inaccurate.
it really needs to be a bayesian estimate. they need to look at p(a) and p(b) in addition to p(a|b) (or p(b|a)).
regarding "... they need to substitute it with something more meaningful", that's the more meaningful thing they need to substitute it with.
If this is important enough of an issue to consider such a radical change to policy, then they should also have considered other possible solutions, like requiring a statistician be included in the pool of reviewers. The journal I submit to most frequently uses 2 to 3 ad hoc reviewers plus the associate section editor. It could be possible to require the section editor who choses the ad hoc reviewers to include a statistician as the 3rd reviewer. They would then review for the soundness of the statistical procedures, and the appropriateness of the conclusions based on the model used, and analysis conducted.
I have better stats chops than most in my field (dunning kruger delusion on my part, possibly), but I know that I'm no statistician. I think that getting an actual statistician involved in reviewing most papers as a content expert is far more valuable to science as a whole than simply banning a statistical convention that can be, but is not universally, abused. The comments from the statistician would improve the statistical prowess of the corresponding author, thus reducing the tendency for conclusions based on poor stats to be accepted at face value. This move just hides the ignorance behind confidence intervals, which can also be abused if they are not calculated correctly.
Bureaucracy expands to meet the needs of the expanding bureaucracy.-Oscar Wilde
The Q are intrigued. Pray they don't intervene.
It is the job of the reviewer to check that the statistic was used ion the proper context. not to check the result, but the methodology. It sounds like social journal simply either have bad reviewer or sucks at methodology.
That's a good sentiment, but it won't work in practice. Here's an example:
Suppose a researcher is running rats in a maze. He measures many things, including the direction that first-run rats turn in their first choice.
He rummages around in the data and finds that more rats (by a lot) turn left on their first attempt. It's highly unlikely that this number of rats would turn left on their first choice based on chance (an easy calculation), so this seems like an interesting effect.
He writes his paper and submits for publication: "Rats prefer to turn left", P<0.05, the effect is real, and all is good.
There's no realistic way that a reviewer can spot the flaw in this paper.
Actually, let's pose this as a puzzle to the readers. Can *you* spot the flaw in the methodology? And if so, can you describe it in a way that makes it obvious to other readers?
(Note that this is a flaw in statistical reasoning, not methodology. It's not because of latent scent trails in the maze or anything else about the setup.)
====
Add to this the number of misunderstandings that people have about the statistical process, and it becomes clear that... what?
Where does the 0.05 number come from? It comes from Pearson himself, of course - any textbook will tell you that. If P<0.05, then the results are significant and worthy of publication.
Except that Pearson didn't *say* that - he said something vaguely similar and it was misinterpreted by many people. Can you describe the difference between what he said and what the textbooks claim he said?
====
You have a null hypothesis and some data with a very low probability. Let's say it's P<0.01. This is such a good P-value that we can reject the null hypothesis and accept the alternative explanation.
P<0.01 is the probability of the data, given the (null) hypothesis. Thus we assume that the probability of the hypothesis is low, given the data.
Can you point out the flaw in this reasoning? Can you do it in a way that other readers will immediately see the problem?
There is a further calculation/formula that will fix the flawed reasoning and allow you to make a correct inference. It's very well-known, the formula has a name, and probably everyone reading this has at least heard of the name. Can you describe how to fix the inference in a way that will make it obvious to the reader?
I studied and tutored experimental design and this use of inferential statistics. I even came up with a formula for 1/5 the calculator keystrokes when learning to calculate the p-value manually. Take the standard deviation and mean for each group, then calculate the standard deviation of these means (how different the groups are) divided by the mean of these standard deviations (how wide the groups of data are) and multiply by the square root of n (sample size for each group). But that's off the point. We had 5 papers in our class for psychology majors (I almost graduated in that instead of engineering) that discussed why controlled experiments (using the p-value) should not be published. In each case my knee-jerk reaction was that they didn't like math or didn't understand math and just wanted to 'suppose' answers. But each article attacked the math abuse, by proficient academics at universities who did this sort of research. I came around too. The math is established for random environments but the scientists control every bit of the environment, not to get better results but to detect thing so tiny that they really don't matter. The math lets them misuse the word 'significant' as though there is a strong connection between cause and effect. Yet every environmental restriction (same living arrangements, same diets, same genetic strain of rats, etc) invalidates the result. It's called intrinsic validity (finding it in the experiment) vs. extrinsic validity (applying in real life). You can also find things that are weaker (by the square root of n) by using larger groups. A study can be set up in a way so as to likely find 'something' tiny and get the research prestige, but another study can be set up with different controls that turn out an opposite result. And none apply to real life like reading the results of an entire population living normal lives. You have to study and think quite a while, as I did (even walking the streets around Berkeley to find books on the subject up to 40 years prior) to see that the words "99 percentage significance level" means not a strong effect but more likely one that is so tiny, maybe a part in a million, that you'd never see it in real life.
OK a new size TV
indicates that authors incorrectly measure p-values to their study results 86.5% of the time (P 0.001).
Modern social psychology is notorious for rejecting objective evidence, since it can often uncover facts about human nature which society (and many social psychologists) don't like. Stan Milgram's experiments come to mind.
On the other hand, get rid of p-values and other forms of objective verification and you can make up anything you want to. You can come up with any amount of airy-fairy so-called 'evidence' to support your pet theory. Get rid of that annoying inconvenience of 'logic'. These days, it's all inference and innuendo, especially since the "critical" / "discursive" crowd have gotten a hold.
and the other crazies were right all along, that psychiatry is not a real science?
Or does it just prove that the general understanding of math and statistics (except among matematicians) are fields that are in free fall, and that a few years from now, college graduates won't even be able to recite the multiplication table up to 10?
-- Another senseless waste of fine bytes.
I suspect that book is still foundational in most University advertising/marketing progams.
I think historically, a more influential book has been Darrell Huff's "How To Lie With Statistics", the second book in this list.
It was originally written in 1954. And while less rigorous, it is an entertaining read and probably gets its point across to a much wider audience.
I know for a fact that Huff's book is still used as a text in college statistics courses... but probably only the lower-level classes.
I used to have one of those old fashioned uniplication tables but ordered a new multiplication table because it collects spilled drinks in its crevices rather than letting it drip on the floor. Much more sanitary. Why would I need to recite? I already cited amazon for one, are you saying they are usually defective?
The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives
Stephen T. Ziliak and Deirdre N. McCloskey
http://www.press.umich.edu/titleDetailDesc.do?id=186351
The University of Michigan Press
https://www.press.umich.edu/pdf/9780472070077-fm.pdf
Citing one table will get them to grade you as a D at most. You have to recite at least ten times to get the benefits of being in the A group. Even then it is up to chance since it is graded on a curve.
Classical statistics mentions the significance level, alpha=0.05. It mentions beta -- (1-beta) is the power of the test to conclude the null hypothesis. Classical statistics never mentions R, the background ratio of true to false relationships in a field. While R lies in the interval [0,infinity], you could think instead about the background probability of true relationships. PLOS had an article several years ago that showed the probability a published article falsely touts a relationship as true, a probability they called the Positive Predictive Probability,
PPV = 1 / [1 + alpha / ((1 - beta) * R))]
The person designing an experiment seeks a large power, 1 - beta, so is bounded away from 0 and at most 1, so this factor becomes irrelevant (remember, the article gets published). When R is much less than alpha; eg, R=0.001 is less than 0.05, then PPV is about
R / alpha
or often
R / 0.05
The background proportion of true relationships R dominates over alpha and over beta in the probability the relationship is true PPV.
You do a statistical test in a "field" of relationships where most of the relationships are wrong, otherwise any relationship stated has a good chance to be correct and the "field" is easy if not boring. Consider the search for some 30 genes that might cause a genetic disease out of 30,000 genes in a genome. Then R is 1 / 1000 and (about)
PPV =. 1/(1 + 0.05/(1/1000)) = 1/51 =. 0.02
That is, such published genetics articles tout relationships that are very unlikely (0.02) to be correct.
The German pharmaceutical Bayer called a large sample of published article authors, duplicated their procedures, yet found 70 percent of the publications' touted results could not be confirmed (probably wrong). Many statistical tools will give fame -- hypothesis tests or even more so data mining tools -- these are often charlatan's tools.
So on a scale of 1 to freedom, how much sympathy do we have for the authors who got banned for using p-values?
I suspect that book is still foundational in most University advertising/marketing progams.
I think historically, a more influential book has been Darrell Huff's "How To Lie With Statistics", the second book in this list. It was originally written in 1954. And while less rigorous, it is an entertaining read and probably gets its point across to a much wider audience. I know for a fact that Huff's book is still used as a text in college statistics courses... but probably only the lower-level classes.
Not to be confused with the much more exciting book, "How to lie with Statisticians"