Slashdot Mirror
Terror Watchlist "Crippled By Technical Flaws"
I Don't Believe in Imaginary Property writes "The database used by the government to generate lists like the No-Fly List is 'crippled by technical flaws,' according to the chairman of a House technology oversight subcommittee. And the upgrade may be worse than the original. Rep. Brad Miller (D-NC) says that 'if actually deployed, [the upgrade] will leave our country more vulnerable than the existing yet flawed system in operation today.' It seems that the current database doesn't have any easy way to do plain-text matching, forcing users to enter SQL queries. That might not sound so bad until you learn that the database contains 463 poorly indexed tables. How long until there's a terrorist named Robert'); DROP DATABASE; —?"
Because theres' nothing a non-USian can learn in such a "story", except that US-ians are teh morons.
Smile, don't click...
That's what happens when your interview questions are a political loyalty test.
The grass is only greener, if you don't take care of your own lawn.
Oh yes. Little Bobby Datas, we call him.
xkcd. Always relevant.
The amount of people they want to include on their "t3rr0rz l1zt" it'll only be a matter of time before we have
Osama Bin CREATE INDEX;
and
Saddam OPTIMIZE TABLE;
Then everything will be hunk dory again.
That might not sound so bad until you learn that the database contains 463 poorly indexed tables.
This is not a good measure of how good or bad a database is. Its good to have a table for every type of data and every data type. Read about normalization. You can go overboard, but as long as your database is designed well, having 463 tables might be just fine.
I say this because once I heard consultant say something like "This web application shouldn't need more than 40 tables, when in fact they didn't know much about the details of the web app, which were quite sophisticated and the real application had more than 100 tables."
The same US government that screws everything else up should be expected to screw up the terror DB. It was probably written by a junior developer who had never heard of a SQL injection. Isn't making a search form about the easiest project there is to build? I hate to say it, but I'm glad our government is so full of screw ups: pity the list exists at all...
Comment removed based on user account deletion
to code an exploit that automatically populates tables in the watchlist with entries from the TSA employee database wins.
Good people go to bed earlier.
It's crippled by being a moronic concept in the first place ("You've got the wrong name and _maybe_ the wrong date of birth, and you're not flying.") and an absolutely arbitrary process of putting names on the list, and no way of ever getting a name off the list.
Fix those points first, and _then_ worry about technical details.
Well, isn't this normal? We create as many tables as we need during normalisation, and then create views/stored procs/frontend scripts/whatever for the user to use.
Sounds like they only did the half the job. The other half still needs to be done - I see no "crippling" here.
I'm a minority race. Save your vitriol for white people.
Some contractors win by hiring really cheap labor, and then bidding so low that the contractors with decent software engineers and database developers cannot compete. Yeah, I'm looking at you, SAIC!
The worst part is that the government hasn't figured out that some contractors, with few exceptions, are just routinely bad and should be avoided at all costs.
In the comic, it's "DROP TABLE." In the summary, it's "DROP DATABASE."
*cleans spectacles*
Dammit
I'm a minority race. Save your vitriol for white people.
The problem is not the number of tables, but the fact that they are apparently 'poorly indexed'. Table indexes are important, both for the speed of queries, and data integrity.
The grass is only greener, if you don't take care of your own lawn.
What is sad about this is some security-cleared development company probably charged $20 million to make it.
Little Bobby Tables http://xkcd.com/327/
You just know Little Bobby Tables is going to grow up to be a terrorist one day.
'Every story, if continued long enough, ends in death.' --Ernest Hemingway
For those interested: the size of the terror watchlist compared to US cities and States.
Silver Clipboard: Time Management Tips
In the comic, it's "DROP TABLE." In the summary, it's "DROP DATABASE."
I wonder if I'm the only SQL noob who had to look up the "drop database" command to see that indeed it is valid?
Granted, not everyone gets to play with their first database with the rights to even use the 'drop database' command...
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Foolish, poorly designed, wasteful, corrupt Federal Database application fails to deliver on promises,
Film at eleven !
how are we going to recognise all those terrorists now?
it may have been a good idea, but the implementation was horrible, come on....
"I was gratified to be able to answer promptly, and I did. I said I didn't know." -- Mark Twain
I, for one, welcome our meme neutralizing slashdot editor-overlords, film at eleven
There, fixed that for you (which is, in fact, my favourite meme.)
"Miller also alleged that some of the $500 million spent on Railhead already had been improperly used to renovate a facility owned by contractor Boeing."
Its easy to waste a lot of money when a department that has a virtually unlimited budget outsources with little to no oversight.
We had similar problems in Canada with the Long Gun Registry, which was a dumb idea to begin with. Then they outsourced it. All told it cost more then $1 billion to set up, and didn't work properly at first. (It does work now, though its still a dumb idea.)
-- "So they told me that using the download page to download something was not something they anticipated." - Bill Gates
Someone seeing the terrorist's name: "Hey, you're that American Comic!"
Terroists looking very serious and dangerous: "NO! I AM NOT!"
Person: "Yes you are! You're so funny!"
Terroist looking even more pissed: "No! Absolutely not!"
Gotta watch it.
It runs on PBS every once in a while Ameican Crossroads.
One could wonder whether the project was set up to adress terrorism OR it was setup to generate media-attention ?
It was both and then some.
I'm trying to find the link of the guy who started this BS. It was a private citizen who, IIRC, was the one who was involved with Choicepoint. He wrote some code and his algorithm pulled up most of the 9/11 hijackers and then some. He had some false positives even then, but it was the Government's wet dream and it solved some of their problems (such as that pesky little Constitutional problem of spying on Americans. It's OK if a private company does it -Choicepoint.) and it makes great security theater and it creates some big fat Governemtn contracts for some big fat cats with Government connections.
Need more caffeine and I'm getting tons of false hits from Google trying to find the cite - it is over 7 years old, ya know.
Is it irony that this was the tagline for the article?
"Kludge, n.: An ill-assorted collection of poorly-matching parts, forming a distressing whole. -- Jackson Granholm, "Datamation""
The "watchlist" should be more like a credit-rating report: It gives the front-line screener a "score" or more likely a redlight-greenlight.
If there is a redlight, and the person has previously been mis-identified, he can whip out his "get on the plane free" card and the screener swipes the card and fingerprint or scans his face or something. If they match, he's green-lighted.
Otherwise, he goes to further screening conducted by trained, relatively trustworthy professionals who look at the actual information: This person has visited Pakistan 5 times, his uncle is a member of the Indian Mafia, he's paying cash, etc. They talk to him and make a decision whether to okay him for this flight, this itinerary, or in the case of mistaken identity, photograph or fingerprint him, and give him a "get on the plane free" card.
In a perfect world, this wouldn't be necessary, but as long as there are people who match my physical description and share my name and birth-date running around trying to blow up planes, I expect to endure the inconvenience of a single delayed flight and having to show and authenticate my "get on the plane free" card every flight. If I ever do start doing things that "fit the profile" like pay cash to flights to Pakistan or marry someone whose family is in organized crime, I expect greater scrutiny.
There are a few things that should be off-limits though:
Race, gender, sexual orientation, religion, place of birth, etc. However, country of citizenship and non-trivial, clandestine association with known criminals or terrorists is fair game.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Why not just put up the list on "terroristwatchlist.com" and do site: google searches.. Would probably be much easier and more reliable.
I'm not sure you can call having names on the list matching 1/3 of the population of the earth a "technical flaw".
What they really need to do to make it useful is get it down to perhaps a couple thousand real concerns.
That could work.Risk it airlines, where there are no security checks to get on board and the only security measures are to detect when a plane has been hijacked and once confirmed a killswitch is activated to simply blow it out of the sky. Might have to pay the pilots more but I'd travel on one of those.
College-Pages.com - Online Colleges, Degrees, and Programs
So, the question that comes to mind for me is this: what if I were a database architecture guru who had been asked to build this system (or its replacement)? At first, my thought is that I'd refuse on grounds of my opposition to the whole thing...but now I'm suddenly wondering if some of the better options did just that, and then it got designed and built by the knob who would take the job. Unlikely, sure, but it's something that I've never thought about before. Is the ethical cost of not doing something like this (that's going to get done anyways one way or another) when you're the right guy for the job potentially higher than the ethical cost of doing it?
For your security, this post has been encrypted with ROT-13, twice.
First of all, many terrorists have names in a non-Latin alphabet. There are multiple ways to transcribe Arabic and Cyrillic letters to the Latin alphabet.
Funnily enough there is a lot of software around to do this and not all of it by the CIA. Since the unPatriotic act (actually before it) banks have been supposed to ID check new account holders.
As far as the other data is concerned, often watch lists contain poor data, and passports may contain inexact data (Chinese gymnasts' DOBs, anyone). You need to have a scoring match, i.e. if the name sort-of matches and the age is within 2 yrs and place of birth is a similar sounding town in the right country, and then flag for a human to review. If there is a positive match using approximation, you really do need a trained person to make any final decision.
See my journal, I write things there
And our so super smart congress could not figure out how to hire Google to complete the project?
Fred Grott(aka shareme) http://mobilebytes.wordpress.com
As usual, the government has taken taxpayers' dollars and wasted it. I'm sure somebody at the Justice Department decided that this database should be easy to build ("It's just a list!"), and rather than bring in some professionals to design it, they slapped it together on their own. They probably decided that the data is too sensitive to have an outside data architect design it. Not only is the government messing with private citizen's rights, but they are doing it badly.
http://video.aol.com/video-detail/snl-funny-terrorist-names/4040669571
"M'balz es-Hari"
"Haid D'Salaami"
"Mustaf Herod Apyur Poupr"
"Usuqa M'diq"
"Hous bin Phartin"
"I'zheet m'drawrz"
----------------------------------- My Other Sig Is Hilarious -----------------------------------
What laws are invoked by the DHS in putting people onto this list and not allowing them to fly. I mean there is a law, isn't there. Now how about getting onto a no-water-boarding list .. :)
davecb5620@gmail.com
A good start is usually hiring IBM, or one of several other large 'service' firms.
IBM specifically does some good things lately, but if you're truly looking to get serviced, the big consulting companies can help you out.
Hiring relatives also generally helps.
I mean how hard would it be to take all of the news stories about these poor Joes who for no fault of their own end up on the no-fly list.
Then publish the daylight out of the list and force some accountability for the responsibility.
A friend of mine is the security manager for a fairly large company. They have offices all over the world and business in many countries. He tells me that there are at least three "terrorist" lists. The EU list, the UN list and the US list. They are listed from poor to really shitty.
If a person or a company is on either of these lists then they aren't allowed to do business with them as they are suspected terrorists r terrorist backers.
The US list can contain things like "Muhammad, Saudi Arabia", or "Iqbal, Pakistan".
The lists are of no use to them and impossible to follow, but they are required to do so or risk sanctions from EU or the US.
Happy times!
Ok ouch ok I'll shut up and sit ouch in a ouch stop it! corner now.
If your a terrorist please stop reading here! It would not do at all for this information to get into terrorist hands!
This story says it's possible to bypass the list by using a legal variant of your name ie Capt. James Robinson said he has learned that "Jim Robinson" and "J.K. Robinson" are not on the list.
I want to be on the terrorist watch list.
"Technical flaws"? I think "technical flaws" are the least of their concerns. The whole system has been marred with logical flaws from its inception.
Arguing the system is hampered by database problems is the equivalent of building a car without an engine and then complaining it doesn't run because the tires are the wrong size.
George Carlin rant on the general topic of airport security...
http://www.youtube.com/watch?v=KBxzvSbGJ2w
" All this airport security the searches, the screenings, the cameras, the questions. Its just one more way to reduce your liberty and remind you that they can fuck with you anytime they want."
Please do explain how data INTEGRETY is affected by the way you define indexes, as opposed to the ways in which you have denormalized tables for performance.
From the article, it would be good to see an explanation of just what they mean by "poorly indexed". That seems much more likely to refer to the need for more indexes for faster search results, rather than indexes done badly...
"There is more worth loving than we have strength to love." - Brian Jay Stanley
You know, since it is the government, I think we should probably let them know that sending around DVD's with Terrorists.mdb is probably not the thing to do. I wonder what will happen when they have more than 2gb of suspects?
This is my sig.
I wonder if I'm the only SQL noob who had to look up the "drop database" command to see that indeed it is valid?
Why look it up when you can test it out for yourself?
Its called an arrest warrant. This other thing is an extra-constitutional fallacy.
The incompetance of US Government contractors is nothing short of astonishing.
How has this been allowed to happen? From the little info I've managed to glean from TFA it seems the schema has been designed by somebody who doesn't know even the rudiments of database design.
How did somebody like this get such a lucrative contract? Is there no accountability in your Gov?
At best it seems the Government gives these contracts out at random, without even the most basic vetting procedures ("Does your company have anything to do with software design?"). A more distrustful person might suggest there is something more sinister going on...
You feel sleepy. Close your eyes. The opinions stated above are yours. You cannot imagine why you ever felt otherwise.
...I thought I might share my train of thought:
1.) Why are all Federal Government databases a complete mess ?
2.) Isn't it actually a good thing that all Federal Government databases a complete mess ?
3.) It's possible that somebody is ensuring that all Federal Government databases are a complete mess ?
4.) I for one welcome our new Federal Government Database Overlords !
One could wonder whether the project was set up to adress terrorism OR it was setup to generate media-attention ?
I work at an airport, in administration, and trust me when I say this has very little to do with dark political conspiracies, and a lot to do with the government's haste to show they were "doing something" after 9/11. This project was quickly rushed into service, and has been widely reviled by airports and airport police departments across the country. And other similar measures... the current background check process for giving access to secured areas, and the very creation of TSA itself, were all measures to reassure the public that something was getting done. The problem is that government enterprises like these tend to become bipartisan boondoggles, with every state and major city wanting a piece of the political and funding action these things entail. Federal agencies tend to become monsters that need to justify their own existence by constant growth. TSA in particular is quickly becoming a large federal law enforcement agency, not just a baggage security team. When they were first set up, several of their nascent teams moved and basically tried to take control of several airports... I know of one major southern airport where they simply showed up one day, declared that a series of offices now belonged to them, and when the airport director came down to see what was going on, they tried to have him arrested by his own police force for "violating federal facilities". Anyone that works with AAAE members (airport execs group) knows what incident I'm talking about.
Did you know that TSA will now be issued police-like blue uniforms, with metal badges, just like cops? Airport police and the metropolitan police departments that supplement them just looooove that, and there's the inevitable talk of actually giving said TSA agents firearms. Unlike some other police departments, TSA agents are being encouraged to wear their uniforms and badges in their spare time, in order to enhance the agency's "visibility" to the public. There are already jokes that TSA SWAT teams are inevitable at airports. The problem is, the laughter doesn't last very long when we realize that the way things are going, that might not be a joke so much as a prediction of the future.
Life is hard, and the world is cruel
The only problem with the watch list is my buddy Bob Smith. Real guy, accountant, nice Audi. He's on the list. He flies about 70 times a year for his job. Every time it takes him about 4hours to get through security. It's things like this that cause people to snap and become terrorists.
They should have just hired TicketMaster, as absurd as it sounds, they're already experts in the field. Although you'd have to pay an extra $8 to get your ticket printed right then at the train station.
Splitters!
[UID-HeinzIntel]
The FBI ran into a similar problem with their case automation system. Investigative databases contain items like "informant 345 reports white male, 20s, tan windbreaker, called "Harry" was seen in a bar on 4th street talking about a robbery at 10th and Main last week". How do you utilize a few million items like that? The usual approach is to start with fully-identified people and work outward, but this leads to the traditional cop vice of finding info that reinforces preconceptions. True correlation of loosely identified items is tough, although there are similarity metrics which can help.
Worse, the terrorism people have to deal with names from cultures that have low name uniqueness.
*wooosh*
You're right! They do get cuter when they're small.
[UID-HeinzIntel]
http://xkcd.com/327/
yeah guise?
lol!
lol?
After all, it's also crippled by moral and logical flaws.
I should point out that meta tables are not "wide" tables by themselves. It's also possible to use wide tables without meta (attribute) tables, so I suppose that's a third option. But generally for a larger system, those who want to avoid a large quantity of tables will use a combination of wide tables for attributes that are stable and common and meta(s) table for attributes and/or components that change often or are rare.
Table-ized A.I.
Rather than bash government, to be fair, large systems are difficult to get right regardless of whether its public or private. Private companies also have big messes. However, they are closed such that outsiders generally don't hear about it. Or else they fail because of their mess and disappear. Economic darwinism thus plays a role for private companies and companies with bad systems eventually fade into oblivion or get bought out by companies with better systems.
But, generally such trial-and-error is not an option for public agencies. They are expected to get it right the first time. I think its realistic to let them get something up and running that half-works, and then apply those lessons to a second generation design. Trial-and-error is sometimes a necessary evil.
Table-ized A.I.
How long until there's a terrorist named Robert'); DROP DATABASE; â"?"
Reminds me of this comic: http://xkcd.com/327/ (xkcd.com)
I thought the first thing you learned with SQL was creating and deleting databases. I've never met anyone who learned the former, but not the latter. You must have a TON of databases on your system.
It's not SUICIDE bombers, it's HOMICIDE bombers! Just watch Fox news! They know that unlike suicide bombers who die when they blow themselves and everyone around them into little pieces, these eveil terrorsists didn't commit suicide. So they can kill again and again and again!!!!
Stick the server in a burlap sack and pour water over it...
they should stop using MSAccess.
Having designed a couple of poorly-designed databases myself, I can understand how this can happen.
What I don't understand is why the hell there are 463 tables in this thing?
I mean, what all information do they need in there? Names, maybe a list of known addresses, social security numbers, phone numbers, other identifying information?
Perhaps a reason why they're on the watchlist at all? List of evidence putting them there? Political activities they've been involved in, letters to congress they've written? Types of books they've checked out of the library?
Maybe a list of all flights they've taken, and notes on how much trouble they've given to the TSA people when going through the checkpoints?
OK, that's three tables. What on earth are the other 460 for??
If the masses can keep you down, you're not the Ubermensch.
period
Julio Henrique Morimoto juliohm@gmail.com
"If I was the person they were looking for, would I be quite so stupid as to travel under my real name with genuine IDs in my name?"
Criminals have been known to return to the scene of a crime to fetch their dropped wallet, write "This is a bank robbery, give me all of your money" on the backs of their own bills, checks, etc... They have even been known to withdraw money from their own bank account before robbing said bank. Some criminals are stupid enough to call the police for other reasons and don't understand that the cops will identify and arrest them once they show up.
Criminals are largely stupid. Most people realize that the amount of effort and planning it takes to be a successful criminal can be used to legitimately make money with far less risk.
Was this built in-house or by a contractor?
I ask, because I've been involved with government contracting work, specifically for the FAA. One aspect of the relationship I've repeatedly seen is private business' efforts to cripple the in-house engineering and software expertise of government agencies they do business with. We'd hire their key people away and call the legislators we owned to get funding for in-house projects killed just to drive the work out to us. Once the agency fell on its face a few times, political pressure would grow to quit wasting money and contract it out. To us. For big money.
Back when I was still in that biz, the Australian government's equivalent of the FAA, CASA, had undertaken a project to build some advanced air traffic control systems in-house. The attitude of our management was rage. "If this had been the United States, we'd have had them shut down."
If you need work done fast, you need people who can do it on the inside. Even if it goes out for contract, you've got to get the requirements written down correctly.
Have gnu, will travel.
> Actually, I think the SQL 2012 standard only supports the short form, "SADDAMIZE TABLE".
Isn't that an MS-SQL-only extension? I'm sure that I've seen it used on both Microsoft Access & Microsoft SQL Server...
here about the dangers of posting a link to xkcd on /.
shame on you. I missed 1 meeting, half of lunch, and didn't do anything all morning.
not only is time travel possible, it's irrelevant.
Some people never, ever, ever, delete their email, either.
Its simple ... talented DBAs don't want to work for peanuts, which is why they don't work for the government.
Just wanted to point out that having a felony conviction doesn't necessarily mean somebody is an evil person.
That's exactly what an evil person would say.
paintball
...and yet despite it's failure to protect us, we have not been attacked.
Perhaps, just perhaps, this is evidence that it is not necessary?
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
Sure, but there's a huge difference between talented and incompetent. Surely this database can't be so complicated that it requires almost 500 tables - I bet I could design a database for the Encyclopaedia Britannica with fewer tables than that
The way to repeal this monster is to insert the names of all of our Congressmen, their families, and their office staff. Add the names of every confirmed administration official, with their families and staffs, and the repeal bill will be signed instantly.
Alternatively, insert the name of every federal judge, and it will get declared unconstitutional in a few days.
Throw in some Governors and other state-level politicians, some state judges, and all of the candidates for office (this is an election season), and it will really get noticed.
I thought the first thing you learned with SQL was creating and deleting databases. I've never met anyone who learned the former, but not the latter. You must have a TON of databases on your system.
That may be a philosophical choice for the DBA to make regarding the SQL noobs in question. Couldn't an argument be made that each SQL noob should have the ability to create and delete tables, not databases, within an instance?
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
How long until there's a terrorist named Robert'); DROP DATABASE; --?"
I have a friend from Southeast Asia whose name, when you zero out the high-order bits of the usual UTF-8 encoding, comes out to just that. It'll be fun seeing what happens the next time he flies home to visit his family.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
If you look at the firehose, you will see that I, the submitter, was the one to add that XKCD link. The editors didn't do it.
I also changed the SQL injection because I didn't want to guess or type in all 463 tables. DROP DATABASE is so much more efficient.
- I Don't Believe in Imaginary Property
The TSA also distributed the lists as separate searchable Excel spreadsheets. The last time I saw them, (roughly 6 months ago) the selectee and no fly lists were pushing 40MB and 15MB respectively. The airline security department would download a new list from the TSA on a daily basis. The Excel version was used as a backup in case the airline's database driven version rolled over and died (which it did frequently).
If you're wondering, I'm a former airline employee. I'm the one that the ticket counter agents would call to bypass the check-in restricted response when someone was flagged as a selectee or No-Fly. After verifying that they are or aren't a match based on the information provided, I'd make an entry in the computer that allowed the counter agent to check the person in.
The lists included all sorts of interesting data, like Birthdate/Birthplace, SSN, Driver's License Numbers, Passport Numbers, Description of the Person, State of Residence, etc.
I'd love to have downloaded them and posted them for the world to see, but I don't cherish the federal prison time that would have come with it.
Leaving one party with the power of the executive branch and the other with the power of the legislative branch ensures both active branches of government will battle each other and accomplish very little. This is perhaps the best a reasonable person can hope for and I believe it's that way by design. Given reign of both branches one party can do a great deal of damage to civil liberty in a very short time. Properly done, though, we should trade off which is which so that what energies they have left from fighting each other can be employed in reversing the actions of their predecessors. That way each generation will begin the same place the last generation did, and the power seekers will have been successfully turned from the tyrannical ways they seek and harnessed to the useful task of depleting the surplus productivity.
Please, don't hope for efficiency in government. History is full of efficient governments and living was free and easy under none of them.
Help stamp out iliturcy.
As a man with the same name as an ex-Gitmo inmate, I get hassle.
Thankfully I'm not on the no-fly, but because my namesake was an inmate I'm denied luxuries like online checkin on US flights, use of automated checkin machines at airports and a bunch of other stuff. Not only that but at checkin desks I get, at the very least, the agent having to call upstairs and more usually a stern looking man taking my passport off for checks for a few minutes and then coming back and giving the all clear.
This speaks to me of broken systems because surely they have my passport number on record...
Anyway, it's an annoyance.