Slashdot Mirror

← Back to Stories (view on slashdot.org)

Why Standard Deviation Should Be Retired From Scientific Use

Posted by Soulskill on from the hope-it-gets-a-good-pension dept.

An anonymous reader writes "Statistician and author Nassim Taleb has a suggestion for scientific researchers: stop trying to use standard deviations in your work. He says it's misunderstood more often than not, and also not the best tool for its purpose. Taleb thinks researchers should use mean deviation instead. 'It is all due to a historical accident: in 1893, the great Karl Pearson introduced the term "standard deviation" for what had been known as "root mean square error." The confusion started then: people thought it meant mean deviation. The idea stuck: every time a newspaper has attempted to clarify the concept of market "volatility", it defined it verbally as mean deviation yet produced the numerical measure of the (higher) standard deviation. But it is not just journalists who fall for the mistake: I recall seeing official documents from the department of commerce and the Federal Reserve partaking of the conflation, even regulators in statements on market volatility. What is worse, Goldstein and I found that a high number of data scientists (many with PhDs) also get confused in real life.'"

3 of 312 comments (clear)

  1. So you want to retire a statistical term... by Anonymous Coward · · Score: 5, Insightful

    ...because people use it incorrectly in economics? Get bent. The standard deviation is a useful tool for statistical analysis of large populations.

    1. Re:So you want to retire a statistical term... by Fouquet · · Score: 5, Insightful

      +1 this. The problem here is the author's impression that "social scientists" and economists are scientists. The groups that he excludes in the first paragraph (physicists) are scientists. Anyone attempting to implement a statistical model designed for a large (and Gaussian) data set on a small number of data points (as the article's example does) should expect to get an answer that is at best marginal. Any scientists who ever received even the most basic of statistics and/or data analysis training knows this. Understand the problem first, then take enough data points, then carry out your statistical analysis & formulate conclusions.

  2. Re:That's not the problem. by JoeMerchant · · Score: 5, Insightful

    The problem is that peoples' attention spans are rapidly approaching that of a water-flea.

    Up until the past 50 or so years, people who learned about Standard Deviation would do so in environments with far less stimulation and distraction. Their lives weren't so filled with extra-curricular activities and entertainments that they never sat for a moment from waking until sleep without some form of stimulus based pastime. When they "understood" the concept, there was time for it to ruminate and gel into a meaningful set of connections with how it is calculated and commonly applied. Today, if you can guess the right answer from a set of 4 choices often enough, you are certified expert and given a high level degree in the subject.

    Not bashing modern life, it's great, but it isn't making many "great thinkers" in the mold of the 19th century mathematicians. We do more, with less understanding of how, or why.