Slashdot Mirror
Visualizing False Positives In Broad Screening
AlejoHausner writes "To find one terrorist in 3000 people, using a screen that works 90% of the time, you'll end up detaining 300 people, one of whom might be your target. A BBC article asks for an effective way to communicate this clearly. 'Screening for HIV with 99.9% accuracy? Switch it around. Think also about screening the millions of non-HIV people and being wrong about one person in every 1,000.' The problem is important in any area where a less-than-perfect screen is used to detect a rare event in a population. As a recent NYTimes story notes, widespread screening for cancers (except for maybe colon cancer) does more harm than good. How can this counter-intuitive fact be communicated effectively to people unschooled in statistics?"
How can this counter-intuitive fact be communicated effectively to people unschooled in statistics?
Hmm, teach them statistics?
That's easy, just tell them that the screenings work about as well as speech recognition. It's 95% accurate and everyone knows how much it sucks.
Great! Thank you for identifying yourself as one of those "unschooled in statistics" people the summary mentioned. Now we just need to experiment with different ways to get you to understand this simple concept.
Wow. Way to illustrate the point. Remember, terrorists are roughly zero percent of the population (at least, of the population going on plane trips in the U.S./U.K.). Odds are, at most one of those 3000 actually is a terrorist. So if it is 90% accurate in identifying terrorist vs. non-terrorist (and vice versa), then 10% of the non-terrorists will be identified as terrorists (or ~300), while the 0-1 terrorists will be missed 10% of the time. And of course, since you don't know for sure if there was a terrorist in the group, an in-depth search of the 300 will usually be a waste of time.
$_ = "wftedskaebjgdpjgidbsmnjgcdwatb"; tr/a-z/oh, turtleneck Phrase Jar!/; print
http://www.amazon.com/Manga-Guide-Statistics-Shin-Takahashi/dp/1593271891
I hate math, always did. I was good at it but just could not stand it. As such I skipped out on about anything math related beyond algebra (college level). Didn't impede my programming ability at all.
Still there are times where I like to learn how stuff works and honestly this series of books, Manga Guide to ......, has given me a quick leg up on a few subjects I would never have gained from traditional text books.
* Winners compare their achievements to their goals, losers compare theirs to that of others.
Exactly. So, someone who doesn't have a grasp on the terminology wants to educate folks who don't have a grasp on it either.
And this kids -- we call journalism.
I am the lawn!
If you have a screen that works 90% of the time, and you detain 300 people, 270 will be terrorists.
Congratulations, you got it wrong exactly the way that is being complained about.
The test accuracy is measured compared to the population tested. In fact, a test that consistently says "no cancer" in all cases is 99% accurate when run on the general population.
Finally! A year of moderation! Ready for 2019?
Back during the TQM fad they'd make this point by giving everyone a clear plastic box with 10,000 little balls in it. There was a cribbage board like affair in it, with 1,000 holes, such that by inverting and shaking the box, then turning it upright, 1,000 of the balls would settle into the holes more or less at random, but still be visible through the clear box. The balls were color coded -- 10 red balls, 40 black ones, 50 blue ones, and the rest white. The odds of getting no red and no black are lower than 1%, contrary to most people's expectations.
This was used to drive home a point about the difficulty of "testing in quality" (quality tests suffer false negatives and if there are, say, 1000 such individual measurements on a piece of machinery it's nearly impossible to ship a machine without at least one thing wrong unless the tolerances are well controlled at the point of manufacture). The same idea works any time you want to illustrate the effects of low-incidence events on a large population.
I've always wondered how much injustice is perpetrated by drug screening on large populations, since false positives do occur and statistically must occur twice in a row at least some of the time, which is the threshold considered conclusive proof of abuse by most employers and the courts.
The article itself started out by oversimplifying the test. It would be an astounding coincidence if the test had both a 10% false-positive and a 10% false-negative rate. In fact, any normal test has a very different false-positive and false-negative rate. People who describe the test should mention both, not this meaningless "90% accurate" number.
The BBC article, while claiming to want to reduce confusion, actually perpetuates the problem by using the meaningless "90%" number instead of the specific positive and negative failure rates. If every article describing tests would quote both failure rates, that would go a long way to getting people to understanding the situation.
I hate it when I make a joke and I get modded "+5 insightful". Mod the stupid comments "funny", not "insightful", pleas
There's a second opinion in the US when you're put on the no fly list? Or in the UK, when you're detained without charge for weeks (the Government wanted three months)?
The point is that idiots as described in the article think that a "90% scanner" means 90% probability they are guilty, and use to urge action on such people without further checks. And even in a court of law, the point being made is still important: imagine the prosecution telling the jury that the fingerprint/DNA test is 99.99% accurate, therefore he must be guilty? In other words, these further checks are useless if they also fall on the same flawed statistics.
You're okay with your medical analogy, because most doctors have an understanding of basic statisics - unlike the police, politicians, and random members of a jury.
Anyone can write software to look for a turban
This sort of racist bollocks is what has been getting people attacked in the US for wearing them, despite them being an optional part of the Muslim faith so most turban wearers are from entirely different religions which actually require them.
http://en.wikipedia.org/wiki/Turban
Please educate yourself before posting such drivel.
I dont read
The given version of "terrorist" is arbitrary and thus subject to change over time - from people who hijack planes with guns and explosives, to apparently nowadays, Iceland, however I think that if you're starting with a number of 1 in 3000 you are so far from reality anyway that what you really want to do is harass innocent people.
Let's look at ALL the hijackings from 1970 to 2000, a total of 924 hijackings. I couldn't find more recent figures quickly, but let's assume that hijackings have continued at a rate of around 30 per year (the average from 1970-2000), that would add another 30 * 9 = 270 hijackings, for a total of 1194 ok I will be generous 1200 hijackings.
Now let's assume (and this is a BIG assumption - I am again going to be very generous) that TEN people, (the terrorists), board the plane for EACH hijacking event. So now we have 12,000 terrorists.
Now let's just look at the passenger data for the LAST YEAR ALONE for the top 5 airlines. They carried last year 420 million people. LAST YEAR. Now assuming that since 1970 till today there have been a total of 12000 "terrorists" (a VERY generous number), when you divide 420 million by that, you would be looking at 1:35,000 people being a "potential terrorist". However do remember that I am only including passenger data for ONE SINGLE YEAR. Assuming again a 90% accuracy, you are still wrongly intimidating well over 3500 people.
If I was to go through year by year and gouge up the billions of people that have been transported by air, the actual chances of the person being screened actually being a terrorist drops to almost zero.
I will not argue against the value of security as a deterrent. However I think that airport security employees should be well aware that they are, more likely than not, harassing innocent people. Therefore all the excessive bullying, posturing, abuse, privacy and rights violations are completely unnecessary in this context. Airline terrorism is NOT a real threat, be it ever so dramatic on the few times when it does happen. Use technology to screen for the obvious, and lock the god damned cockpit door with a solid lock, for the not so obvious.
Seven puppies were harmed during the making of this post.
I think they totally forget that there is ALSO a 10% possibility that you _don't_ detect the terrorist...
Watch this TED : http://www.ted.com/talks/peter_donnelly_shows_how_stats_fool_juries.html
Privacy is terrorism.
You'd have a really good point if there weren't actually bigoted assholes and/or ignorant people in the world who agree with the great-grand-parent. Earlier in my life I may have been one of them.
I remember painting Muslims with a very broad and unfair brush. People would tell me that all Muslims aren't bad and most want the same thing I do, peace and prosperity. Why don't they speak out against the bigoted extremist representatives then? I would ask.
I didn't have the slightest understanding of the culture and environment those types of ideas breed in and probably still don't. However, I can come out of my own bubble enough to ask myself the question - What motivation would I have to speak out against wrongs being done against a culture who shows repeated disrespect and ignorance for my own?
I'm not suggesting we adopt sharia law and that all North American women start wearing burqa as a sign of respect. There is a very thick line between embracing and adopting a culture and respecting it.
Turbans are worn by Sikhs. This is a completely different religion to Islam which is alleged to harbour these terrorists.