Slashdot Mirror
Metrics Mania and the Countless Counting Problem
mobkarma writes "Einstein once said, 'Not everything that can be counted counts, and not everything that counts can be counted.' A New York Times article suggests that unless we know how things are counted, we don't know if it's wise to count on the numbers. The problem isn't with statistical tests themselves, but with what we do before and after we run them. If a person starts drinking day in and day out after a cancer diagnosis and dies from acute cirrhosis, did he kill himself? The answers to such questions significantly affect the count."
Cancer itself could be considered a form of killing yourself.
We all live our lives as we wish to live them, and realize that statistics are incredibly important to making the world a better, easier place to live in. Sure, they can be wrong sometimes, but I would imagine the general public trusts them a lot less than they should actually be trusted. I mean, global warming is like 99.99999% true, same with evolution, but we still have people who don't have a clue and doubt blatant facts because they don't understand things like the specific heat capacity of water, or that evolution isn't globs of crap off the ground suddenly turning into animals and people.
Sure, the numbers can sometimes be wrong, but they are not wrong 75% of the time. Not even 50% or 25%, but less. And yes, sometimes we are further off, but it is rare. Should we really ignore important numbers because their is a small chance they are wrong? I am not saying anyone should change everything about their lives due to a single number, but common, this is a bit crazy. I am not trying to be debatative here, just saying hey, it is what it is.
Where is the mod rating for "scary"? Also,
Q: What did the clinically depressed alcoholic man with acute cirrhosis get for Christmas?
A: Cancer
We're all just afraid of uncertainty. It is the shadow from which anything potentially could arise. Our brains are just hardwired to be much more fearful than hopeful (for obvious evolutionary reasons).
It's like the whole "critical vulnerability" count in software
And having read The Opinion Makers I have zero trust in them.
Increasing the number of times I spank the monkey a day isn't going to increase my chances of getting a girl into my basement. But I've lost count either way.
You don't die of cirrhosis by drinking heavily for a short time. You may die of alcohol poisoning.
Intron: the portion of DNA which expresses nothing useful.
Sounds like a restatement of the simultaneously-discovered Goodhart's Law, Lucas critique, and Campbell's Law.
Basically, once you start measuring something as a proxy for what you really want to know, people start to take the proxy into account when making decisions, to the point where it becomes useless as a measure for whatever it was intended.
Here, people take these cancer tests as a measure of their probability of cancer. But once they start to treat them as reliable, they start doing more self-destructive things, destroying the correlation between the proxy (the cancer test) and the actual probability of cancer.
Information theory is life. The rest is just the KL divergence.
So the problem isn't one of having too much data but rather unreliable correlation of that data to draw conclusions. What exactly is new here?
I finally learn how to count and now they tell me it's useless. What's next, I learn how to type and I find out nobody is reading what I write?
There are more things in heaven and earth than are dreamt of in your philosophy.
Counting things counts for 23% less than it did last millennium.
Is 1563649 a prime number?
You can count on metrics being a problem.
Many years ago, I had an in-depth discussion about gathering statistics on heart disease with a woman on the board of the American Heart Association. This was a big deal. Serious ethical issues were in play and there was a great deal of infighting going on.
I asked her how you make a definitive decision that someone has heart disease. I was trying to figure out what to measure. Her answer surprised me. She said "You wait till they die. Then you cut out their heart and have a look." She then went on to patiently explain to me that the only thing that could be measured and evaluated were "markers" of heart disease. Those markers, as revealed by various disgnostic tests, could be mighty reliable. But you never know if someone is going to die of heart disease until they...you know...actually *die*.
Thus informed, I came to realize that what we measure is almost never what we really want to know. Measuring the right stuff is simply too hard to do. No matter where you look, this is almost universally true. In my job, for example, we fix computer problems. Thus, we measure how many incidents get closed and how much time it took. If you quickly close an incident, then surely you've provided good service, right? Most slashdotters should realize that's not true. In fact, my job is actually to get other, more important workers back to work asap. The only way to measure that would be to interview my customers and their bosses. We'd have to pry for an hour into their effectiveness to find out if I properly completed a job that took me five minutes. That's too much trouble, so we look for markers. Closed incidents. Timeliness of closures.
Measures are inadequate so often that I pretty much don't trust anything that contains them. After years of training in Quality Improvement Processes, I came to realize that the amount of time needed to understand a process and perfectly spec out what needs to be measured is 452% of the expected life cycle of the project, plus or minus a 17.5% margin of error. (Aside - How much do you trust those statistics?)
Almost no one can devote the time required to do the job (no matter what "the job" is) right. We just hope people do their best and trust to good intentions.
As a computer guy who wants things to be either "yes" or "no", unambiguously, I found this state of affairs very difficult to accept. But it's just part of being human.
I have heard this issue raised regarding reports of the health of the economy. Retail sales are shown to be up, but only because stores that go out of business are dropped from the counting. If there were still there counting as big fat goose eggs the average would show that the economy is in fact contracting.
Scientists, etc. use statistical tests to get information about something they can measure, and how well that measurable quantity can be predicted from their data set. They form a hypothesis that variable X can be predicted from the data. They test their hypothesis, and calculate the probability that knowledge of the data set will lead to a correct prediction of X. If they get something like 68%, 99.9%, etc., they're happy and they write it up. Perfect, but Suppose in some parallel universe (that some string theorist might try to sell you) that similar scientists had been more diligent, and conducted statistical analyses on not just X, but on 10^10 other variables. Maybe X is global temperature or the price of oil, and the smart folks in this universe can measure a million things at a million time points that might potentially affect X. Same as in the other universe, they find that 99.9% of the variance in X can be predicted by the data set, but since they tested so many variables, they can't claim significance. By random chance, a lot of other variables did even better than X. Then what matters is whether the scientist tells you about all those other tests. That's not exactly conducive to getting published, or getting grant money, but who knows- it might be right. It's not just how things are counted, but how many different things are counted, and in what parallel universe, that really counts.
He once inserted random mutations into his code, just so he could have the experience of debugging.
Often people gather statistics in a deliberately-biased way, to intentionally make the numbers be wrong, so they can 'prove' a point (which may be false) that benefits them in some way.
People will shine a positive light on their own products or services, and will shine a negative light on anything that competes with or otherwise threatens them. Such people will deliberately mis-count, or misrepresent the properly counted numbers, in order to get their way.
So I don't think that the numbers are right as often as you think they are.
This comment is insightful, not funny.
I always interpreted that quote as a comment on the existence of the real numbers.
...'unless we know how things are counted, we don't know if it's wise to count on the numbers'.... Which is why we still spend money on, public health research, and other so called 'soft' social sciences! GIGO...
excellent title for an album ... "Metrics Mania and the Countless Counting Problem", like "Mellon Collie and the Infinite Sadness" or "Cobra and Phases Group Play Voltage in the Milky Night"
will run to register it right now ... thanks
This got censored on climateprogress.org so stop reading if you are sensitive.
The question of what counts has put coal mining fatality statistics into a strange state. http://blogs.wvgazette.com/coaltattoo/2010/05/05/will-latest-death-be-counted-as-mining-related/ Has President Obama allowed 2010 coal mining fatalities to double 2009 fatalities or is he still one shy of that dubious distinction? If the Mine Safety and Health Administration (MSHA) bites the bullet and counts the death as a coal mining fatality, then there will have been 36 so far this year, twice as many as for all of last year. If not, they keep the administration away from that statistic at least for a while. The stakes are high. At 37 fatalities, President Obama will have the highest percentage annual increase in coal mining fatalities of any President ever. His only way out is to close and fence dangerous mines. Current efforts focus on explosion dangers so roof collapses continue apace (fatalities #34 and #35 last month).
And, these statistics are also coming in the context of investigations into bribery at both the MSHA (Department of Labor) and likely at the Minerals Management Service again (Department of Interior) suggesting that coal mine and oil rig fatalities are caused by official and widespread corruption. Does that bribe count as taxable income?