Slashdot Mirror
Terror Watchlist "Crippled By Technical Flaws"
I Don't Believe in Imaginary Property writes "The database used by the government to generate lists like the No-Fly List is 'crippled by technical flaws,' according to the chairman of a House technology oversight subcommittee. And the upgrade may be worse than the original. Rep. Brad Miller (D-NC) says that 'if actually deployed, [the upgrade] will leave our country more vulnerable than the existing yet flawed system in operation today.' It seems that the current database doesn't have any easy way to do plain-text matching, forcing users to enter SQL queries. That might not sound so bad until you learn that the database contains 463 poorly indexed tables. How long until there's a terrorist named Robert'); DROP DATABASE; —?"
That's what happens when your interview questions are a political loyalty test.
The grass is only greener, if you don't take care of your own lawn.
Oh yes. Little Bobby Datas, we call him.
xkcd. Always relevant.
The amount of people they want to include on their "t3rr0rz l1zt" it'll only be a matter of time before we have
Osama Bin CREATE INDEX;
and
Saddam OPTIMIZE TABLE;
Then everything will be hunk dory again.
That might not sound so bad until you learn that the database contains 463 poorly indexed tables.
This is not a good measure of how good or bad a database is. Its good to have a table for every type of data and every data type. Read about normalization. You can go overboard, but as long as your database is designed well, having 463 tables might be just fine.
I say this because once I heard consultant say something like "This web application shouldn't need more than 40 tables, when in fact they didn't know much about the details of the web app, which were quite sophisticated and the real application had more than 100 tables."
The same US government that screws everything else up should be expected to screw up the terror DB. It was probably written by a junior developer who had never heard of a SQL injection. Isn't making a search form about the easiest project there is to build? I hate to say it, but I'm glad our government is so full of screw ups: pity the list exists at all...
Comment removed based on user account deletion
to code an exploit that automatically populates tables in the watchlist with entries from the TSA employee database wins.
Good people go to bed earlier.
It's crippled by being a moronic concept in the first place ("You've got the wrong name and _maybe_ the wrong date of birth, and you're not flying.") and an absolutely arbitrary process of putting names on the list, and no way of ever getting a name off the list.
Fix those points first, and _then_ worry about technical details.
In the comic, it's "DROP TABLE." In the summary, it's "DROP DATABASE."
The problem is not the number of tables, but the fact that they are apparently 'poorly indexed'. Table indexes are important, both for the speed of queries, and data integrity.
The grass is only greener, if you don't take care of your own lawn.
Because theres' nothing a non-USian can learn in such a "story", except that US-ians are teh morons.
Hold on, that's not true! In this story, we learn that the terrorist watch list is not only a bad idea, but it is poorly implemented!
The worst part is that the government hasn't figured out that some contractors, with few exceptions, are just routinely bad and should be avoided at all costs.
What makes you think they haven't figured it out? There is good money, bad money, corrupt money, etc ... but the best type of money is *my* money, ie money in my hand. Frankly, if I was purchasing something that's of no benefit to me I'd hardly bother with quality, I'd just like to keep as much money in my hands as possible.
For example *ahem* if I was forced to purchase something (say, furniture) for ex-wife after she moved out, why would I bother spending money on anything more than patio chairs and a plastic table?
I'm a minority race. Save your vitriol for white people.
For those interested: the size of the terror watchlist compared to US cities and States.
Silver Clipboard: Time Management Tips
In the comic, it's "DROP TABLE." In the summary, it's "DROP DATABASE."
I wonder if I'm the only SQL noob who had to look up the "drop database" command to see that indeed it is valid?
Granted, not everyone gets to play with their first database with the rights to even use the 'drop database' command...
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
how are we going to recognise all those terrorists now?
it may have been a good idea, but the implementation was horrible, come on....
"I was gratified to be able to answer promptly, and I did. I said I didn't know." -- Mark Twain
Since he flew a lot for work, the unfortunate consequence was being FULLY searched EVERY time he went through the airport. He finally called up the TSA once and told them, "How about I just come into your office. If I am your man, ARREST ME! If I'm not, then get me off of this list!" to which they responded, "I'm sorry sir, but it doesn't work that way."
All in all, it took him over 3 years to finally get his name off. I think the criteria for being on the terror watch list are pretty well summed up here:
-If you have the same name, initials or hair color as a felon, you're on the list.
-If you've ever lived withing a 5 mile radius of a felon, you're on the list.
-If you've ever flown on an airline that a terrorist has ever attacked before, you're on the list. and finally.
-If airport security is bored, you're on the list.
Any thoughts?
"Thank you for using Stop-n-Drop, America's favorite suicide booth since 2008"
"Miller also alleged that some of the $500 million spent on Railhead already had been improperly used to renovate a facility owned by contractor Boeing."
Its easy to waste a lot of money when a department that has a virtually unlimited budget outsources with little to no oversight.
We had similar problems in Canada with the Long Gun Registry, which was a dumb idea to begin with. Then they outsourced it. All told it cost more then $1 billion to set up, and didn't work properly at first. (It does work now, though its still a dumb idea.)
-- "So they told me that using the download page to download something was not something they anticipated." - Bill Gates
One could wonder whether the project was set up to adress terrorism OR it was setup to generate media-attention ?
It was both and then some.
I'm trying to find the link of the guy who started this BS. It was a private citizen who, IIRC, was the one who was involved with Choicepoint. He wrote some code and his algorithm pulled up most of the 9/11 hijackers and then some. He had some false positives even then, but it was the Government's wet dream and it solved some of their problems (such as that pesky little Constitutional problem of spying on Americans. It's OK if a private company does it -Choicepoint.) and it makes great security theater and it creates some big fat Governemtn contracts for some big fat cats with Government connections.
Need more caffeine and I'm getting tons of false hits from Google trying to find the cite - it is over 7 years old, ya know.
I'm not sure you can call having names on the list matching 1/3 of the population of the earth a "technical flaw".
What they really need to do to make it useful is get it down to perhaps a couple thousand real concerns.
I just put you on the list.
So, the question that comes to mind for me is this: what if I were a database architecture guru who had been asked to build this system (or its replacement)? At first, my thought is that I'd refuse on grounds of my opposition to the whole thing...but now I'm suddenly wondering if some of the better options did just that, and then it got designed and built by the knob who would take the job. Unlikely, sure, but it's something that I've never thought about before. Is the ethical cost of not doing something like this (that's going to get done anyways one way or another) when you're the right guy for the job potentially higher than the ethical cost of doing it?
For your security, this post has been encrypted with ROT-13, twice.
http://video.aol.com/video-detail/snl-funny-terrorist-names/4040669571
"M'balz es-Hari"
"Haid D'Salaami"
"Mustaf Herod Apyur Poupr"
"Usuqa M'diq"
"Hous bin Phartin"
"I'zheet m'drawrz"
----------------------------------- My Other Sig Is Hilarious -----------------------------------
A friend of mine is the security manager for a fairly large company. They have offices all over the world and business in many countries. He tells me that there are at least three "terrorist" lists. The EU list, the UN list and the US list. They are listed from poor to really shitty.
If a person or a company is on either of these lists then they aren't allowed to do business with them as they are suspected terrorists r terrorist backers.
The US list can contain things like "Muhammad, Saudi Arabia", or "Iqbal, Pakistan".
The lists are of no use to them and impossible to follow, but they are required to do so or risk sanctions from EU or the US.
Happy times!
This happened to one of the guys at the company I work at, who has a pretty common name and flew at least once a month. Every single time, he'ld be datained a couple of hours.
It took several years and several thousand dollars of lawyer fees to fix (company paid, I assume, since they needed him to travel).
My uncle had a similar experience to your relative when he was returning from Jamaica (he was there for his anniversary). He had the exact name (middle too) of a wanted felon and was detained in customs for hours before they finally figured out he was from the other side of the country as his evil name-twin. As he pointed out at the time, "If I was the person they were looking for, would I be quite so stupid as to travel under my real name with genuine IDs in my name?" It's not like the guy was just "suspected"...he was pretty much a known criminal/fugitive.
Please do explain how data INTEGRETY is affected by the way you define indexes, as opposed to the ways in which you have denormalized tables for performance.
From the article, it would be good to see an explanation of just what they mean by "poorly indexed". That seems much more likely to refer to the need for more indexes for faster search results, rather than indexes done badly...
"There is more worth loving than we have strength to love." - Brian Jay Stanley
I'm sure somebody at the Justice Department decided that this database should be easy to build ("It's just a list!"), and rather than bring in some professionals to design it, they slapped it together on their own.
If you'd bother reading the report, available at http://democrats.science.house.gov/Media/File/Commdocs/Staff_Memo_toBM_terror_watch_8.21.08.pdf, you'd see that Boeing is responsible for the current system. So, yes, a private professional company, employing experienced DBAs is responsible for the current system. If you'd spent much time consulting for private industry you'd know that this sort of thing isn't unique to the government. It's just that it's much more likely to come to light if it's a government project. I've seen many examples in private industry where companies, large and small, end up in the same same bind. This is what happens when rapidly evolving requirements are shoehorned into databases whose original designs could never have anticipated those requirements. Projects like this don't have scope creep so much as scope leap. Software messes that are difficult to migrate almost invariably occur.
I wonder if I'm the only SQL noob who had to look up the "drop database" command to see that indeed it is valid?
Why look it up when you can test it out for yourself?
-If you have the same name, initials or hair color as a felon, you're on the list.
-If you've ever lived withing a 5 mile radius of a felon, you're on the list.
Any thoughts?
It takes more than just being a felon.
I have a felony conviction (non-violent). I've flown 3 times since being discharged from parole and haven't run into any difficulties at the airports.
There are many different types of felonies. Many felons are, indeed, very very bad people. However, I personally know several convicted felons who I would trust to babysit my children, or loan money to. Most of the people I know in that category got their felony convictions as a result of substance abuse issues and have since cleaned up their act.
Just wanted to point out that having a felony conviction doesn't necessarily mean somebody is an evil person.
One could wonder whether the project was set up to adress terrorism OR it was setup to generate media-attention ?
I work at an airport, in administration, and trust me when I say this has very little to do with dark political conspiracies, and a lot to do with the government's haste to show they were "doing something" after 9/11. This project was quickly rushed into service, and has been widely reviled by airports and airport police departments across the country. And other similar measures... the current background check process for giving access to secured areas, and the very creation of TSA itself, were all measures to reassure the public that something was getting done. The problem is that government enterprises like these tend to become bipartisan boondoggles, with every state and major city wanting a piece of the political and funding action these things entail. Federal agencies tend to become monsters that need to justify their own existence by constant growth. TSA in particular is quickly becoming a large federal law enforcement agency, not just a baggage security team. When they were first set up, several of their nascent teams moved and basically tried to take control of several airports... I know of one major southern airport where they simply showed up one day, declared that a series of offices now belonged to them, and when the airport director came down to see what was going on, they tried to have him arrested by his own police force for "violating federal facilities". Anyone that works with AAAE members (airport execs group) knows what incident I'm talking about.
Did you know that TSA will now be issued police-like blue uniforms, with metal badges, just like cops? Airport police and the metropolitan police departments that supplement them just looooove that, and there's the inevitable talk of actually giving said TSA agents firearms. Unlike some other police departments, TSA agents are being encouraged to wear their uniforms and badges in their spare time, in order to enhance the agency's "visibility" to the public. There are already jokes that TSA SWAT teams are inevitable at airports. The problem is, the laughter doesn't last very long when we realize that the way things are going, that might not be a joke so much as a prediction of the future.
Life is hard, and the world is cruel
So I'm guessing their Access database just looks like this:
ID | First | Last
I guess they just took KISS way too far.
Splitters!
[UID-HeinzIntel]
I'm sure he'll not be missed.
My co-workers 2 year old Daughter was on the list. It took 4 years to get her name removed.
It must have been her evil plot to drop a bomb in her diaper.
If you think it's expensive to hire a professional to do the job, wait until you hire an amateur. --Red Adair
Just wanted to point out that having a felony conviction doesn't necessarily mean somebody is an evil person.
... nah, it just means that they didn't have a very good lawyer.
Hope these thoughts are useful:
(1) Contractors bid to get the contract. The lowest bidder able to perform gets the contract. They'll provide to exact spec, and not a bit more. They make their REAL money with change orders. Change orders are not bid out. Contemplate this.
(2) Government specs can suck. This is quite understandable, because the people who want to use the software may not know how the software works. This invites the change orders described above. (When a consultant is hired to write the specs, you magnify the problems outlined in (3) and (4).)
(3) At the outset, contractors usually know nothing about the nuts and bolts of the government business that they are writing software for. The BUSY government people have to teach them. Power and control issues also play a role here. Details can easily get lost.
(4) Interactions between the government and the contractor become really intertwined. How would the government ever prove to a jury that any defect is the contractor's fault when the final product is an inseverably intermixed product of both the contractor and the government?
They also thought there was a comma and that it was hiring advice: "Keep it simple, stupid".
Sorry, you're not allowed to create an unnecessary and disruptive large system and then pull the excuse that "large systems are hard!" when it fails badly.
If DHS created a program with a goal of kicking every single American citizen square in the nuts, and that program ended up being fraught with budget overruns, cases of mistaken identity, citizens getting kicked square in the nuts twice, some citizens not getting kicked square in the nuts at all, and people complained about the system, would you stand up and say "don't criticize them too much, large systems are hard"?
A sane person should say that TSA does a pointless job in a worthless manner, and this, not the fact that it's a "large system", is the root of the problem.
If you mod me Overrated, you are admitting that you have no penis.
I think your comment is pretty irrelevant since the TSA has only gotten worse since it's inception. Not only has it made mistakes, but it has expanded and pursued its mistakes to a point where, like the GP implied, we have to take a step back and say "this has gotten a little out of control."
Well let me give you my personal experience about it. I have a relative named "David Hall." Pretty common name huh? Well he was put on the terror watch list years ago because there is a suspicious person named David Hall. He was able to determine that the person they were after was many years older, had a different birthdate, SSN, and even lived in a state he had never been in.
Since he flew a lot for work, the unfortunate consequence was being FULLY searched EVERY time he went through the airport. He finally called up the TSA once and told them, "How about I just come into your office. If I am your man, ARREST ME! If I'm not, then get me off of this list!" to which they responded, "I'm sorry sir, but it doesn't work that way."
Yeah, I was on the watch list myself, in some relatively minor category I guess. "Chris Burke" isn't exactly an uncommon name. Despite not being hassled by security since a few months after 9/11 (obviously I fell into some random Scary Hippie profile that they grew enough of a clue to stop using), suddenly I started getting the super-search every time I went through security, and couldn't use self check-in, and other minor inconveniences.
I found out when I asked an airline ticketing clerk what was up. She said there must be an evil Chris Burke out there (hey, I thought that was me!), made a phone call, said it was all cleared up, and after getting the super-MEGA-search going through security, I haven't had any problem since.
So not nearly as annoying as the cases where it takes years to get off the list and requires some act of God -- I guess there must be different levels of watch list that you can be arbitrarily placed in -- but still stupid.
The enemies of Democracy are
I just put you on the list.
Yea, but did you add his name to all 463 tables?
tomorrow who's gonna fuss
Having designed a couple of poorly-designed databases myself, I can understand how this can happen.
What I don't understand is why the hell there are 463 tables in this thing?
I mean, what all information do they need in there? Names, maybe a list of known addresses, social security numbers, phone numbers, other identifying information?
Perhaps a reason why they're on the watchlist at all? List of evidence putting them there? Political activities they've been involved in, letters to congress they've written? Types of books they've checked out of the library?
Maybe a list of all flights they've taken, and notes on how much trouble they've given to the TSA people when going through the checkpoints?
OK, that's three tables. What on earth are the other 460 for??
If the masses can keep you down, you're not the Ubermensch.
Was this built in-house or by a contractor?
I ask, because I've been involved with government contracting work, specifically for the FAA. One aspect of the relationship I've repeatedly seen is private business' efforts to cripple the in-house engineering and software expertise of government agencies they do business with. We'd hire their key people away and call the legislators we owned to get funding for in-house projects killed just to drive the work out to us. Once the agency fell on its face a few times, political pressure would grow to quit wasting money and contract it out. To us. For big money.
Back when I was still in that biz, the Australian government's equivalent of the FAA, CASA, had undertaken a project to build some advanced air traffic control systems in-house. The attitude of our management was rage. "If this had been the United States, we'd have had them shut down."
If you need work done fast, you need people who can do it on the inside. Even if it goes out for contract, you've got to get the requirements written down correctly.
Have gnu, will travel.
...and yet despite it's failure to protect us, we have not been attacked.
Perhaps, just perhaps, this is evidence that it is not necessary?
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
I was for a while. I apparently got taken off of it a few months before they publicly admitted its existence.
It was fun. During my time on it, I flew 37 times. I got "randomly" selected for the extra search all 37 times. I ran the numbers for a TSA agent once who insisted it was purely random, and came up with something like one in a few hundred quintillion chance of that actually happening if it was truly random. Still failed to convince the agent it was not, though.
It was great when I had to fly out of LAX. Unlike most airports, that one had a special line for the special searches. So, instead of standing in line for an hour and a half to walk through the metal detector in ten seconds like most people, I waited in line for five minutes, then spent another 2-3 getting searched.
Most airports made me wait in line with the non-terrorists, though.
I'm still not sure what it was that got me on the list, whether it was carrying a knife onto the plane, twice, or the rather obvious joke I made while taking off my shoes. ("It's a good thing that that guy didn't put the bomb in his underwear").
Did you know that it's illegal to even say the word "bomb" in an airport? TSA explained this to me at great length that day.
(The knife, by the way, was a cub scout pocket knife, and it had already been through three searches without being noticed. Four if you count my checking the bag before I left to make sure I didn't leave anything in it.)
Anyway, at some point I got dropped off the list. I don't know why. Maybe it got too full, or maybe they decided that after 37 flights I wasn't a threat, or perhaps they were cleaning up the database before they publicly admitted its existence.
Before I dropped off of it, though, I purchased one-way tickets for a couple of friends who'd helped me move to another state. (We drove out, they flew back). They've both been pulled over for the extra "random" searches now, too.
If the masses can keep you down, you're not the Ubermensch.
...when the watch list hit 1,000,000 names...
Holy crap...that's like one in three hundred Americans on the watch list. Think about that for a second. This means on any given airliner, chances are the government considers at least one of your fellow passengers a person of interest.
Me thinks the signal to noise ratio of this list is mighty, mighty low (not that I expected much, but still, 1:300).
MCSE? No, sir...I don't do Windows. Yes, I am an idealist. What's your point?