Metrics Mania and the Countless Counting Problem

mobkarma writes "Einstein once said, 'Not everything that can be counted counts, and not everything that counts can be counted.' A New York Times article suggests that unless we know how things are counted, we don't know if it's wise to count on the numbers. The problem isn't with statistical tests themselves, but with what we do before and after we run them. If a person starts drinking day in and day out after a cancer diagnosis and dies from acute cirrhosis, did he kill himself? The answers to such questions significantly affect the count."

Comments

Technically

[Monkeedude1212]

Cancer itself could be considered a form of killing yourself.

Re:Technically

[ShadyG]

Whoa, dude. Ease up. The OP obviously meant that cancer by its nature is your own body attacking itself, not that all cancer is the result of some intentional decision by the mind of the afflicted. My mother is at this moment probably just about one week from dying of brain cancer, and I was still able to read it for what it was and not require an apology.

Re:How about this

[TheCarp]

I don't disagree, but, it cuts both ways. I think the article has a point... the numbers only have meaning in context.

If I tell you "X people die every year from being shot, in their home, with their own gun", that tells you something. It evokes images of burglars or irresponsible people playing. It SOUNDS like a statement about how safe or dangerous it is to have a gun in your own house.

However, if my number "X" includes suicides, well, then how much of a statement about the relative danger of owning a gun am I making? How about if I can find no link between owning a gun and committing suicide?

Clearly the statement is correct, "shot, in their home, with their own gun" but, even so, its misleading if you then use the numbers wrong.

Take texting while driving. The claim is 900 deaths a year. How do they come at that number? Even better, how big is that number? 900 sounds like a lot.. However... its less than the estimated number of serial murder victims in the US. Overall driving deaths are more like 40,000 a year. Context is everything. If I said "about 1 driving death in 40 is related to txting while driving" thats suddenly a lot smaller, yet, represents the same data.

frankly, I tend to think a LOT of statistics are meaningless. NY state enacted a law against handhed phone use while driving. It resulted in a 70% decrease in OBSERVED use. There was no decrease at all in accident rate.

What this tells me is, someone really believed that this was going to make a difference, came up with numbers and statistics and, in reality, the one little item that he picked out had about as much bearing on accident rates as the price of butter in bangladesh does.

Much of the time statistics are used to just bullshit and make it look like we aren't playing blindfolded darts when we make public policy.

-Steve

Re:cirrhosis

[Rakshasa Taisab]

I believe you just counted something that didn't need to be counted.

Re:How about this

[natehoy]

had about as much bearing on accident rates as the price of butter in Bangladesh does.

Thanks, now I have to check the price of butter in Bangladesh to see whether it's safe to drive home, you insensitive clod!

Re:How about this

[Anonymous Coward]

I'd wager that the 1 in 40 people who've died from texting while driving came out of a sample of 1 in 40 drivers who don't put enough importance into paying attention to the road. So, take away texting drivers, and you'll still have 1 in 40 people dying because they were adjusting their radio one station at a time without looking up, or rolling up the rear-passenger window by hand because they don't have power windows.

I don't think texting while driving has increased accidents, I just think it's made it easier to point out who the stupid drivers are.

Re:No counting problem that I can see

[maxwell demon]

Actually the "counting" problem they mentioned is a categorization problem. Depending how you define your categories, you get different counts. But that's because those are really different categories (they are defined differently). So the question is not really one of counting, but one of the "correct" definition of the category.

Lies, damned lies, and statistics

[BenEnglishAtHome]

Many years ago, I had an in-depth discussion about gathering statistics on heart disease with a woman on the board of the American Heart Association. This was a big deal. Serious ethical issues were in play and there was a great deal of infighting going on.

I asked her how you make a definitive decision that someone has heart disease. I was trying to figure out what to measure. Her answer surprised me. She said "You wait till they die. Then you cut out their heart and have a look." She then went on to patiently explain to me that the only thing that could be measured and evaluated were "markers" of heart disease. Those markers, as revealed by various disgnostic tests, could be mighty reliable. But you never know if someone is going to die of heart disease until they...you know...actually *die*.

Thus informed, I came to realize that what we measure is almost never what we really want to know. Measuring the right stuff is simply too hard to do. No matter where you look, this is almost universally true. In my job, for example, we fix computer problems. Thus, we measure how many incidents get closed and how much time it took. If you quickly close an incident, then surely you've provided good service, right? Most slashdotters should realize that's not true. In fact, my job is actually to get other, more important workers back to work asap. The only way to measure that would be to interview my customers and their bosses. We'd have to pry for an hour into their effectiveness to find out if I properly completed a job that took me five minutes. That's too much trouble, so we look for markers. Closed incidents. Timeliness of closures.

Measures are inadequate so often that I pretty much don't trust anything that contains them. After years of training in Quality Improvement Processes, I came to realize that the amount of time needed to understand a process and perfectly spec out what needs to be measured is 452% of the expected life cycle of the project, plus or minus a 17.5% margin of error. (Aside - How much do you trust those statistics?)

Almost no one can devote the time required to do the job (no matter what "the job" is) right. We just hope people do their best and trust to good intentions.

As a computer guy who wants things to be either "yes" or "no", unambiguously, I found this state of affairs very difficult to accept. But it's just part of being human.

Re:Lies, damned lies, and statistics

[Anonymous Coward]

In fact, my job is actually to get other, more important workers back to work asap.

A refreshing point of view. A surprising amount of IT weenies seem think that what they do is the most important thing in the entire company, and that the rest of the organization needs to bow down to their whims.

I remember having to explain to an IT worker that if they weren't going to change the schedule of the forced anti-virus full scan from 10:30am, I was going to delete the software since it was keeping me from doing my job. He didn't seem to understand that, no matter what he thought, having almost every machine be unusable for three hours in the middle of the work day cost a lot of productive time. His rationale was that doing it in the middle of the morning was the only way to be sure the machines were likely to be on since the machines might hibernate over night.

When 15 people started sitting in the lunch room for an hour or so, HR eventually managed to explain it to him that it wasn't really OK to cause all of the machines to lock up and become unusable in the middle of the day.

Re:How about this

[Suffering Bastard]

frankly, I tend to think a LOT of statistics are meaningless. NY state enacted a law against handhed phone use while driving. It resulted in a 70% decrease in OBSERVED use. There was no decrease at all in accident rate.

Ah, the irony of using a statistic to prove that statistics are meaningless.