Slashdot Mirror


False Positives, Few Matches Plague 'No-Fly' List

lindner writes "According to a recent article in the San Francisco Chronicle, the United States No-Fly List uses a soundex algorithm to match names. Designed 'to quickly summon passenger names or to catch deal-hunting passengers making duplicate bookings.' The system has only managed to rack up a slew of false-positives, including everyone matching soundex ("J. Adams") at one point in time. The problem has gotten so bad that there is now a "Fly List" for chronically misidentified passengers."

37 of 325 comments (clear)

  1. Soundex??? by Surak · · Score: 4, Interesting

    That algorithm is so fundamentally broken as to be practically useless for anything but as an aid in simple searches. Why anyone would use soundex in a mission critical application designed to positively identify individuals is beyond me. What, was the 'No Fly' database written by 1st year comp sci major or something? Sheesh.

    1. Re:Soundex??? by BWJones · · Score: 4, Informative

      That algorithm is so fundamentally broken as to be practically useless for anything but as an aid in simple searches. Why anyone would use soundex in a mission critical application designed to positively identify individuals is beyond me. What, was the 'No Fly' database written by 1st year comp sci major or something? Sheesh.

      Hey, wanna make a good living in these uncertain economic times? Come up with a better alternative and propose it to DARPA. There is actually some very cool research going on that is funded by DARPA in terms of biometrics and database centric comp sci.

      --
      Visit Jonesblog and say hello.
    2. Re:Soundex??? by Randseed · · Score: 5, Insightful

      One of the major clinical automation systems used in American hospitals uses soundex as a primary matching algorithm for patient lookups in the admitting department. Everyone is smart enough not to use it for names like "Juan Garza," but for names like "Steve Franklin" the chance of getting false-positives on your search algorithm is REALLY high. This is largely because of how the system itself implements things.

      Two notable occasions have occured where patients were admitted as the incorrect "Steve Franklin" (name make up for use here, of course). Needless to say, this might be a bit of a problem when the medical and nursing staff then takes that admission record and looks back at labs, radiographs, and such ON THE WRONG PATIENT.

      Of course, this same "highly advanced" system is really just a set of SQL tables that don't even use variable lengths for fields like comments (instead restricting the user to something obscene like 38 characters). The user interface is really just a Curses program that reads the columns on the table and displays them, allowing the user to edit them. Nearest I can tell, SQL functions handle all the data verification and such, and don't even do a good job at it.

      I've worked with this computer system for four years, suffering through it's stupidity.

      The point is that one should never assume that sucky, disgusting software is written by first year comp sci majors. There are enough professional programmers out there to cause a far bigger disaster.

      Never underestimate the power of stupid people in large groups, or in corporate culture.

    3. Re:Soundex??? by kaisa_sosey · · Score: 5, Insightful

      Who cares what algorithm they use? Why someone would support a 'No Fly' database is beyond me.


      I think people are either criminal (means they should be kept in prison) or not.


      Guys like you make me really afraid. For you it's only a technical problem, is it?



    4. Re:Soundex??? by Old+Wolf · · Score: 5, Funny

      That's a bit insulting to first-years, don't you think?
      On my course, in the section on name recognition, we first learned Soundex, and then learned Obershelp, along with the fact that the latter is far more accurate, and Soundex is pretty crap.

      Perhaps it was written by people with no education..

    5. Re:Soundex??? by mindriot · · Score: 4, Insightful

      Interestingly enough, the original Soundex was based on English language only. So when feeding it foreign names, it will obviously match names from different languages that in reality are far from sounding alike. Admittedly, their algorithms are merely based on Soundex and maybe a bit better.

      But to me, finding terrorists by checking their names against no-fly lists sounds just about as useful as checking IP packets for an Evil bit, doesn't it?

    6. Re:Soundex??? by Surak · · Score: 4, Funny

      But to me, finding terrorists by checking their names against no-fly lists sounds just about as useful as checking IP packets for an Evil bit, doesn't it?

      Are you trying to tell me that the RFC 3514 patch I wrote for the Linux kernel back in April is useless?!

  2. Heard about this by Anonymous Coward · · Score: 5, Funny

    This also happened when Cowboy Neal was mistakenly identified as Kh'alid bin Naoul.

  3. Deal-Hunting is illegal? by mosch · · Score: 4, Insightful
    I'm a little confused... are the airlines now prohibiting people to fly on the basis that they went "deal-hunting"?

    I understand that the airline industry is a little tight right now, but that's just insane.

    1. Re:Deal-Hunting is illegal? by PeteJones · · Score: 5, Informative

      I think the deal hunting they are refering to is the practice of buying 2 return tickets and then using the first set to get to your destination and the second to get back. This way you can get the cheap tickets that span weekend stays but instead travel on a, say, Tuesday and come home on Wednesday of the same week.

      Even though you bought two complete return fares, it's still cheaper most of the time that buying one return ticket during a business week. They are trying to fleese the business traveller to subsidize the vacation traveller. I think you probably get both sets of airmiles from buying 2 sets as well

  4. The problem... by maelstrom · · Score: 4, Insightful

    Unfortunately, the officials implementing a system such as this are going to get crucified either way. If they let a known terrorist onto a plane and a terrorist act happens, their heads are going to roll. Every journalist will be screaming that, "this terrorist has been on the FBI watch list for 2 years, a simple misspelling of his name allowed him to foil the multi-million dolar no fly system".

    On the other hand, false positives are going to make the system useless as the boy who cried wolf one too many times found out. There doesn't seem to be an easy solution to this problem.

    --
    The more you know, the less you understand.
    1. Re:The problem... by Anonymous Coward · · Score: 5, Insightful

      Solution is simple. The odds of you being killed in the U.S. by a terrorist is so small that only an ignortant and fearfull population would even worry about it. Just /dev/null wipe hands go to bed.

    2. Re:The problem... by MisterMook · · Score: 4, Interesting

      Screening for terrorists by name is a nice idea, and maybe eventually it will pop up one or two dumb ass terrorists travelling around the country by plane in a nation that's so paranoid that they're willing to sign away their most basic freedoms they've enjoyed for 200+ years. I think it's pretty stupid to throw the baby out with the bathwater though. Before 9/11 we had Timothy McVeigh and no one was hollering that rednecks shouldn't be allowed to drive trucks. So a terrorist gets on a plane, hopefully he'll be going home. If just make sure that the planes have security in place like police, cockpit doors, and some common sense I'm sure we'll all breath easier than if we've got Big Brother and his faulty software deciding who is a threat to society.

      Heck, there's a guy on death row with the exact same name as me out there somewhere...I'd hate for this great idea of the government to lead to my law abiding ass getting thrown in jail even "just in case"

    3. Re:The problem... by MillionthMonkey · · Score: 5, Insightful

      The system relies on a false premise. Terrorists don't have "careers" anymore. If you were planning a terrorist attack, you could easily find 20 guys with no records whose names appear on no lists in any form. Recruiting people for suicide terrorist activities has become very easy as of late. (I can't say why or I'll be attracting a bunch of AC replies from dittoheads.)

      This is a system designed to give you a false sense of security. It bothers and harasses people so much that they feel safe when they get on the plane (if the plane doesn't leave before they get through the bullshit). It will not stop the next hijacking at all- although it strongly discourages discretionary air travel, and is rapidly destroying the airline industry.

      We should go back to the system we had before 9/11, that served us well for many years. Terrorists may still be able to crash airplanes, but they will no longer succeed in crashing airplanes into buildings. Now that everyone knows how that type of attack works, it is unlikely to succeed again. Note how it didn't even succeed once they got wind of it via cellphone during that flight over Pennsylvania.

      But since the public has it in their head that terrorism can be magically prevented at the airport somehow, we should put up some sort of pretense for them at the security checkpoint and the gate. I'm thinking about some sort of prop that you would see Scotty using on Star Trek- a sort of stick with colored lights inside that you wave over a person. If they're a terrorist, the lights turn red and the stick makes a funny sort of buzzing noise. That way we could wave people through, and have them convinced that they've been inconvenienced enough to be safe.

    4. Re:The problem... by Cipster · · Score: 4, Insightful

      This list is designed to be like a screening test: False positives are acceptable (since you clear them up with a follow up) but false negatives are not since they can lead to disaster.
      It's a lot like the AIDS/HIV test. You want every potential positive test to show up so you can follow it up but you do not want to miss a potentially infected person.

    5. Re:The problem... by eyegone · · Score: 5, Funny

      Before 9/11 we had Timothy McVeigh and no one was hollering that rednecks shouldn't be allowed to drive trucks.

      I've been hollering that rednecks shouldn't be allowed to drive trucks (or anything else) since I moved to Texas in '96.

      --
      "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety."
    6. Re:The problem... by Sique · · Score: 4, Insightful

      It's a lot like the AIDS/HIV test. You want every potential positive test to show up so you can follow it up but you do not want to miss a potentially infected person.

      Normally every medical test in fact consists of two tests. The first one is called the efficient test, the second one the effective test.

      The first one is to throw out as much uninteresting people (the ones without any sign of sickness) as possible without missing any of the interesting people (the ones that are sick). In Computer Science you call a test like this "Trivial Reject". It should be fast and cheap to save on costs (of every type, like monetary, computational time...).

      The second test is supposed to catch as much of the interesting people (sick ones) as possible while throwing out pretty surely all of those who are in fact healthy and thus uninteresting for the matters of the test. This leaves you with a test population which consists of almost all sick people and some very few false positives. Because of the shrunken number of persons this test can be more elaborate and expensive.

      The problem with all those tests is, that with the low frequency of the usual dangerous plague you are testing for, the false positives (even though they may be very small compared to the whole of the population) still contains more people than the actual sick ones. I remember some exercises where we had to calculate the risk of actual being sick after a positive test for given frequencies of occurance of the plague in the population and the characteristics of the test. Often the result was that you have a 99% chance of being a false positive even though the test itself was pretty good.

      Same is valid for the frequency of the actual terrorists in the population of all U.S. domestic flyers. While there were 19 terrorists in the 9/11 attack, there have been billions of individual flights in the recent years. If your tests are being 99.999% sure in clearing a innocent flyer, it would still mean that this test applied to one billion flight passengers gives you 10 000 false positives.

      (This number is one of the reasons why the Romans once decided for "in dubio pro reo". Because the number of lawful citizens is quite high compared to the number of criminals, the average screening of larger populations results in more false positives than actually caught thieves. So you have always to prove individually that someone is guilty beyond any doubt before you can actually call them guilty.)

      In this special case the screening for names itself is a very inaccurate test. There are names in some countries being too prevalent to distinguish between people. For Corea it may be the name 'Kim', for India 'Singh', for Russia 'Gorbachev', for the U.S. 'Smith' or 'Adams'.

      The inapprobriate way to match spelling and pronouncing with the Soundex system (which only works with english names) just adds to the problem. In french names you often can't make a difference between 'en', 'an', 'on', 'ent', 'ant', and 'ont'. Famous are the words 'son' (his/her, but also tone), 'sont' ([they] are), 'sans' (without), 'sang' (blood), which are all pronounced the same.

      In german names there is no difference between 'tz' and 'z' or between 't', 'tt' and 'dt'. For my family name I know of the spellings 'Sigmund', 'Siegmund', 'Siegmundt', 'Sigemund', 'Siegemund', 'Siegemundt', 'Sigesmund', 'Siegesmund', 'Siegesmundt', 'Zygmunt', and 'Zygmont' (it appears that no one I ever met was able to spell my name right without me helping him, I have several documents with a falsely spelled name...).

      Add to this list the number of falsely entered records (people accused of things, later cleared, but not erased from lists compiled while they were under suspicion, typing errors, missunderstandings...). So I am quite amazed that the SFO airport had only about three hundred false positives. On the other hand there were no rightful positives (no one got convicted of terrorism yet who was caught with the CAPPS system), and we don't know about the false negatives (actual terrorists flying without being caught) because they slipped through the controls.

      --
      .sig: Sique *sigh*
  5. Obviously this would happen by Anonymous Coward · · Score: 5, Insightful

    It should be obvious to anyone that any mechanism designed to target a small group out of a large group will would have to have an extremely small false positive rate to be of any use.

    And the false negative rate had better be small, too.

    Something 99% accurate is far from good enough; if only 0.01% of possible individuals are actual targets, you'll be getting 100 times as many false positives as correct positives.

  6. Will I use my alias name ? by Alain+Williams · · Score: 4, Insightful

    Will I use my alias name which is Alain Williams, or will I use my real name which is Osama Bin Laden the next time that I book a flight to the USA ?

    The trouble with this sort of thing is that it inconveniences Joe Public while doing little to deter a real terrorist.

  7. Man... by Faust7 · · Score: 5, Funny
    Many airlines rely on name-searching software derived from "Soundex," a 120- year-old indexing system first used in the 1880 U.S. census.

    ...and you thought mainframes were legacy technology.

  8. Ten Problems with Soundex by notestein · · Score: 4, Informative

    Here's a good run down on soundex and ten problems with it.

  9. Pity the unfortunately named by sssmashy · · Score: 4, Funny

    Soundex gives each name a key using its first letter and dropping the vowels and giving number codes to similar-sounding vowels (like "S" and "C"). The system gives the same code, L350, for "Laden" and all similar-sounding names: Lydon, Lawton, and Leedham.

    Boy, I'd hate to be a guy with a name like "Sam Lawton" or something. I wonder how many similarly-named middle-aged salesmen are getting red-flagged on flights... because you just never know, what if Osama Bin Laden disguised as a portly white guy from Milwaukee, and he never bothered to pick a false name that sounded sufficiently different from the original.

  10. Re:Deal hunting? by heli0 · · Score: 4, Informative

    what's wrong with passengers hunting for the best deal?

    Story: to catch deal-hunting passengers making duplicate bookings.

    This means that people were using different travel agencies to reserve lots of seats on the same flight, and then simply going with the agency who quoted the lowest price. This means that the seats end up being empty, and the rest of us end up paying for them. Sorry, but even the "rich elite" can no longer make duplicate reservations. Almost all airlines now use software to automatically cancel duplicate bookings.

    --
    Whenever the offence inspires less horror than the punishment, the rigour of penal law is obliged to give way...
  11. Reinventing the wheel by CodeMaster · · Score: 5, Informative

    This is unbelievable. Why reinvent the wheel, while there are at least 3 countries that have implemented similar restrictions and tracking systems succesfully for more than 10 years now (England, Israel, and Germany - that I know of).

    This sounds like the work of some consultants with no idea of what they are faced with and fresh out of collage where they have analyzed a couple of algorithms... sorry - I have had way too much of these running around the office lately

    If for once someone would just poke his head out and instead of trying to find a solution to an age old problem, look and see how others are handling it, we (taxpayers) would all be much more content (and safe...).

    Just my 2c.

  12. David Nelson [TSA most wanetd??] by FreeLinux · · Score: 4, Interesting

    The situation is really bad if you are named David Nelson. Here is a sad but true story about no fly lists and the very common name, David Nelson. There was also a followup story to this one but I am unable to find a link.

    But, you feel safe. Don't you?

  13. Why not use ID#? by jaclu · · Score: 5, Informative

    I guess all US people would screem for such a "threat to your privacy" ;)

    But at least in here in sweden basically anytime you book a flight you give your ID number (similar to a social security #)
    Two benefits:
    1. Name is just a courtesy, so doesnt really matter for security if somebody get my name wrong.
    2. On checkin, it must be _you_ not somebody with a similar name

    Of course IDs can be stolen or forged, but that is a problem regardles of how you ID your self.

  14. But Bushies don't care Re:Obviously this would by leoaugust · · Score: 4, Insightful

    Your point is very valid if there is a reasonable and rational discussion of the tradeoff's - You know kind of Type I and Type II errors. But the Bushies don't believe in that. Goebellian Ashcroft said that they are willing to use every legal tool available to them to achieve their goals - even if it means ignoring the spirit of the law, and reinterpreting the letters of the law to do whatever they want.

    The willingness, in fact eagerness, to overlook collatoral damage is the Hallmark of the Bush Administration. They have rammed policies that wouldn't pass muster support anywhere. It is almost as if they are willing to kill 9 innocent people to prevent the 10th guilty one from escaping.

    This mentality shows up in the No Fly list. It shows up in how the Arab immigrants were rounded up, and are now being deported by the thousands. It shows up in how to get to the Saddam "WMD's" they were willing to slaughter Iraqi's. Two or 3 Sept 11 bombers entered with Student visa's so everyone on that visa now gets grandly screwed.

    So, logic applies only when the hysteria subsides. If you want you can never let the hysteria subside. And Donald Rumsfeld is a genius - almost lunatic - in that. Like he said in almost poetic form, on Feb. 12, 2002, Department of Defense news briefing, (which means that he could use the concept described in his "poem" below prove anything that he wants - it is almost like dividing by Zero.)

    The Unknown

    As we know,

    There are known knowns.

    There are things we know we know.

    We also know

    There are known unknowns.

    That is to say

    We know there are some things

    We do not know.

    But there are also unknown unknowns,

    The ones we don't know

    We don't know.

    --
    To see a world in a grain of sand, and then to step back and see the beach where the sand lies ...
  15. Re:credit checks - wtf? by juuri · · Score: 4, Informative

    Not every check shows up on your reports, it is possible to make an overall query that doesn't get mentioned. FICOs scores are done this way.

    --
    --- I do not moderate.
  16. You need a constitution by Psiren · · Score: 5, Insightful

    You US folks could really do with a constitution to stop this sort of crap happening. Oh wait, you do have one. Oh well, back to the drawing board. Land of the free indeed.

  17. false sense of security by Destree · · Score: 5, Interesting

    I think the idea behind this scheme is not to catch terrorists, or even deter them. But to keep the public under a false sense of security, thinking "hell, if they are searching a lot of people, they must be getting the real ones too!" Although it never works out that way.

    I think I will be flying private planes if they start looking into your credit. A credit check could be like "Well, you evaded child support and paying the bank $5,000, we can't let you board, if you have the money for a flight, you can pay them!"

    They know they they won't be able to get this to work right, they are just pocketing money and putting out a crap system, but I think that it may have better use for private organizations, such as "Well, he evaded taxes and bills, but we see him having a one way ticket to (place), search for him there."

  18. It doesn't matter who gets on... by onallama · · Score: 4, Insightful

    ...as long as they're barred from entering the cockpit. The success of the 9/11 attacks can mainly be credited to 1970s-era hijacking guidelines directing pilots to comply with the terrorists' demands, on the assumption that they were going to fly the plane to Cuba or something similar, rather than use it as a weapon. Those guidelines made sense in their time, but clearly, they're no longer applicable.

    Here's an idea -- instead of inconveniencing millions of innocent passengers, how about securing the cockpits instead? So long as the pilots remain in control of the plane, it's a flying prison for anyone who commits any criminal act back in the passenger compartment. Let the cockpit crew notify the ground of a failed terrorist attack and land the plane at the nearest airport, with the police and FBI waiting. End of story.

  19. Good news for potential terrorists, then ... by legLess · · Score: 4, Interesting

    If you want to make sure your hijacking works fly around innocently as "Sam bin Laden" for a few months, get your name on all the "Fly lists," and then hijack a plane.

    --
    This isn't as much "normalization" as it is "don't take so many drugs when you're designing tables."
  20. This had me laughing so hard... by mark-t · · Score: 4, Funny
    ... I had to take a break to catch my breath.

    From the article:

    Soundex gives each name a key using its first letter and dropping the vowels and giving number codes to similar-sounding vowels (like "S" and "C").
    Hate to break it to these guys but neither "S" nor "C" are vowels. Heck, they don't even make vowel sounds.

    Geeze, no wonder the system is broken.

  21. Re:Deal hunting? by The+Fanta+Menace · · Score: 4, Insightful

    Perhaps if airlines weren't so elusive about their pricing, potential passengers would be able to easily compare various flight options without having to do this. But obviously it's in the interests of the airlines to keep passengers in the dark.

    I had to change a flight that I was booked for a couple of months back, and I couldn't even get them to give me a firm figure on how much it would cost to alter it until I'd committed myself to doing that. Now that is ridiculous.

    --
    -- Even if a god did exist, why the fsck should I worship it?
  22. John Gilmore is suing for freedom of travel by morcheeba · · Score: 4, Informative

    John gilmore is suing for the right to travel anonymously(sp).

    From the website:
    He does so "because he believes persons have a right to travel by air without the government requiring that they relinquish their anonymity. No security threat is as important as the threat to American society caused by erosion of the right to travel, the right to be free from unreasonable searches, and the right to exercise First Amendment rights anonymously."

    Check out the FAQ's, which are well written and explain the other reasons - including being subject to secret laws - he is opposing this.

  23. Or safer cars. And bathtubs. by Anonymous Coward · · Score: 5, Insightful

    In the US, more people are killed in car accidents _every month_ than were killed in the attacks on the WTC. Even a tiny 2% decrease in the number of car-accident deaths would save more lives every decade than were lost in all terrorist attacks the US has ever suffered.

    Over the last 10 years, an American's odds of dying in a terrorist attack are about 1 in 100,000. That's less than your odds of drowning in your own bathtub, less than your odds of drinking yourself to death, and less than your odds of accidentally suffocating in your own bed! (http://www.nsc.org/lrs/statinfo/odds.htm)

    Frankly, the current atmosphere of fear of terrorism is little more than hysteria. Why on earth aren't we showing the world we have some balls and are strong enough to not let a few terrorists make us live in fear? If you live in fear or give up freedoms, you've let the terrorists win!

  24. Working Just Fine by Markus+Registrada · · Score: 4, Insightful
    If you think the "no-fly" list is not working well, you fundamentally misunderstand its purpose. In fact, for its intended purpose, it has been working fabulously. Like harrassing people about nail files in their bathroom kits, it leads people to think that something is being done about security, without the need actually to do something. How many people have you heard say that while having their shoes X-rayed was inconvenient, it made them feel safer about flying?

    X-raying shoes doesn't make for effective security, but it's intrusive enough to give the impression that at least something is being done.

    Articles and editorials that call attention to the violations that come with the bogus no-fly list are essential components of the system -- they make everybody else experience it, vicariously. Everybody who is a little bit stupid (i.e. most people) feels a little safer for it. Sure it inconveniences some people, but not enough to make much political difference.

    Even better than the impression of intrusive security, it leads to demands for what amounts to a system of internal passports, where you can't travel by air without registering, and getting -- and maintaining --- official permission. "What, no internal passport? Sorry, sir, I can't let you board." At first felons will have their passports pulled, then "suspected terrorists", then political undesirables of all sorts.