Slashdot Mirror
Visualizing False Positives In Broad Screening
AlejoHausner writes "To find one terrorist in 3000 people, using a screen that works 90% of the time, you'll end up detaining 300 people, one of whom might be your target. A BBC article asks for an effective way to communicate this clearly. 'Screening for HIV with 99.9% accuracy? Switch it around. Think also about screening the millions of non-HIV people and being wrong about one person in every 1,000.' The problem is important in any area where a less-than-perfect screen is used to detect a rare event in a population. As a recent NYTimes story notes, widespread screening for cancers (except for maybe colon cancer) does more harm than good. How can this counter-intuitive fact be communicated effectively to people unschooled in statistics?"
"To find one terrorist in 3000 people, using a screen that works 90% of the time, you'll end up detaining 300 people, one of whom might be your target."
If you have a screen that works 90% of the time, and you detain 300 people, 270 will be terrorists.
http://www.amazon.com/Manga-Guide-Statistics-Shin-Takahashi/dp/1593271891
I hate math, always did. I was good at it but just could not stand it. As such I skipped out on about anything math related beyond algebra (college level). Didn't impede my programming ability at all.
Still there are times where I like to learn how stuff works and honestly this series of books, Manga Guide to ......, has given me a quick leg up on a few subjects I would never have gained from traditional text books.
* Winners compare their achievements to their goals, losers compare theirs to that of others.
Though it may be the first time that people are trying to draw general population attention to it. I believe the first place I saw this sort of concept revealed was by Cory Doctorow. Though the below article isn't necessarily where I saw it, it recants the same message.
http://www.guardian.co.uk/technology/2008/may/20/rare.events
addendum - I shouldn't have hit submit yet sorry
Where it says "Assuming again a 90% accuracy, you are still wrongly intimidating well over 3500 people." I should add "per group of 35,000". 10% of 420 million passengers per year is 42 million people per year being harangued for no reason at all.
Seven puppies were harmed during the making of this post.
Switch around what the percentage means: instead of 90% meaning there is 90% chance to successfully ascertain whatever you're screening for, make 90% stand for the analogy of LD50 (Lethal dose for 50%+ of the population). So the screening method would be SE50 (screening effective 50%) if the number of positive cases correctly detected are 50%+ of all positive cases detected.
It takes a man to suffer ignorance and smile
Be yourself no matter what they say
I read the court documents for the trial a while ago. There were two issues:
1. They quoted the statistics assuming the deaths were independent (i.e. the squared the probability of one SIDS death). The error of this was pointed out.
2. No one mentioned the prosecutors fallacy.
In the end the jury were won over by an argument along the lines of: "Ignore the statistics. You *know* it's really unlikely that these were two SIDS deaths.".
http://en.wikipedia.org/wiki/Sally_Clark
Well, that example sorta fails because there are no actual terrorists in it.
A more sane example might be to get a group of a few hundred people, in, say, a auditorium, give them an envelope with a sheet of instructions, and a 20-sided die.
The instructions tell them if they're a terrorist or not, and tells them to roll the die and, depending on what it lands on, go to a specific labeled area. I.e., everyone goes to an area, differently, depending on what they rolled. You need about 20 areas.
Terrorists, of which you have a few, have a 9/10 chance to be sent to a 'terrorist' area, and everyone else has a 1/10 chance.
Everyone rolls, and goes to wherever, and at the end you realize which area was 'prison' and who got caught, and you ask everyone in that area to reveal their status, and you also ask all the other terrorists in other groups to step forward.
Or you could do this with badges and colored stickers, or something. The important thing is that people who are identified as 'terrorists' don't actually know they are until they're pointed out, which is a powerful psychological effect. (Which also means they shouldn't know who is a terrorist, or they'll realize it when grouped together.)
Could be a fairly powerful demonstration when people realize that 90% of the 'terrorist' group are innocent, and, while that group did catch most actual terrorists, there are a few still roaming around.
If corporations are people, aren't stockholders guilty of slavery?
The same idea works any time you want to illustrate the effects of low-incidence events on a large population.
XKCD can be used to illustrate this too
Mouse-over: "You can do this one in every 30 times and still have 97% positive feedback."
Blasphemy is a human right. Blasphemophobia kills.
Instead of talking about false positives and negatives and dependent distributions (which fly right over the head of the average joe), boil it down to the "amplification power" of the test. A random person "presumed innocent until proven guilty" has a chance of 1/3000 to be a terrorist. If you apply your 90% test, people failing it will be terrorists ~1/333 of the time. So the test as an "amplification power" of ~9x. Now everything becomes intuitive. You are looking for a 1-in-3000 needle in a haystack with an amplification power of ~9x, you now need to look for a ~1-in-333 needle in a haystack. The term "90% accuracy" doesn't appear anywhere to confuse things, and it is something everyone can easily grasp. And yes, I know, this ignores the terrorists false negatives; for that you say the test has a "miss rate" of 1/9 so about 1 in nine terrorists will slip through. These three numbers - (1) how rare what you are looking for is, (2) what's the "amplification power" of the test, and (3) what is the "miss rate", give you enough info to intuitively convey all you need to get a good feel for how effective the test really is.