Anti-Terrorist Data Mining Doesn't Work Very Well

Presto Vivace and others sent us this CNet report on a just-released NRC report coming to the conclusion, which will surprise no one here, that data mining doesn't work very well. It's all those darn false positives. The submitter adds, "Any chance we could go back to probable cause?" "A report scheduled to be released on Tuesday by the National Research Council, which has been years in the making, concludes that automated identification of terrorists through data mining or any other mechanism 'is neither feasible as an objective nor desirable as a goal of technology development efforts.' Inevitable false positives will result in 'ordinary, law-abiding citizens and businesses' being incorrectly flagged as suspects. The whopping 352-page report, called 'Protecting Individual Privacy in the Struggle Against Terrorists,' amounts to [be] at least a partial repudiation of the Defense Department's controversial data-mining program called Total Information Awareness, which was limited by Congress in 2003."

Comments

Bets....?

[Gat0r30y]

I bet this will not change what they are doing or how they are doing it one bit.

Well, that's a shocker!

[frank_adrian314159]

In other news, water is wet, the Pope is Catholic, and Ursines excrete solid wastes in silviculture.

Seems

[speroni]

What we really need are spies. Not so much in the US, here good old fashioned detective work (with Warrants) should work.

But over seas a standing army isn't going to do anything to quell terrorism. Tanks and plans will only inspire more terrorism. What we need are good old fashioned black ops. Undercover agents penetrating the terrorist groups and talking to the bad guys. Much less collateral damage as well.

We'd get a lot further with a couple guys with silenced pistols rather than a whole army.

Re:Seems

[Martin Blank]

I seem to recall that much of this was gutted by Congress in the 1990s when they didn't want intelligence operatives paying off criminals for information, on the risk that the money might be tied back to the United States. This severely nerfed the ability of the CIA (among others) to gather HUMINT, as paid informants were a significant source of the information required to infiltrate the groups in the first place. I don't recall if this was ever overturned, though.

Re:Seems

[Fulcrum of Evil]

No, what you need is to stop making people hate you - go after al queda, sure, but the guys killing soldiers in Iraq aren't terrorists for the most part, they're resisting a foreign invader. Tell me, does Canada have a big problem with foreign terrorists?

Re:Seems

[Ethanol-fueled]

Don't forget about the military, who stupidly booted some of their translator recruits (yes, middle-eastern languages) for being....OMG TEH GAY!!!1!

Re:In other news,

[wizardforce]

and it is broken constantly... arg what did we expect of our government when the vast majority of people leave it up to the government to police its self... the constitution only bites those who violate it if it is upheld by the people for its intended purpose, to defend the rights of the people against actions by the government.

Profiles also work in reverse

[davidwr]

As any Cold War spy can tell you, if you "fit the profile" of a normal law-abiding person with just enough "off-perfect" things in your life so you don't seem "too perfect," it's much easier to blend in.

Re:Profiles also work in reverse

[jbeaupre]

For example, never ever have 2.1 kids. It's suspiciously normal. Go with 2, maybe 3, to blend in.

Didn't we already know?

[MozeeToby]

I thought we already knew this. If the algorithm comes back with even .1% false positives the system is totally worthless. There's 365 million people in the US, .1% means that the FBI/CIA/NSA would have 365,000 people to investigate. Now go and talk to someone in the AI field and see if even .1% false positive is possible.

I'm betting that if a system is going to catch any decent percentage of terrorists (greater than 50%) the false positive rate will be above 1%. Even if you only apply the system to a relatively small number of people (say people entering a leaving the country) you are going to have hundreds of thousands, if not millions, of people to investigate. Combine any kind of realistic false positive rate with the fact that about .00001% of the population deserves to be investigated and the system is worse than worthless; all it will do is distract from the people who should be investigated.

I'd run on that platform.

[khasim]

I would seriously consider voting for either one that came forward and promised to cut TSA's authority and streamline the process, getting back to only those people who are basically confirmed problems being on the list, no matter what their views might be on Iraq, Afghanistan, the economy, or offshore drilling.

Vote for me.

I'd take their "no fly" list and identify every single person on it who was a legitimate threat and either have them under 24 hour surveillance or arrested.

The mere concept of a list of names of people who are too "dangerous" to let fly ... but not dangerous enough to track ... that just fucking stupid.

Think about how many people could be killed in the airport terminal itself WITHOUT getting on a plane ... say during the Thanksgiving or Christmas rushes there.

What idiot would let the people on that list (if they were really a threat) into a terminal? Wouldn't you expect them to STOP them BEFORE they get into a position to do that kind of damage?

Re:I'd run on that platform.

[Fulcrum of Evil]

The no fly list doesn't identify people, just names, and it's very exact, so changing charles to chuck will defeat it. The upshot is that it's utterly useless for stopping bad guys, so you can't even identify who's on there - John Smith is on the list, but there are 10,000 of them.

Re:I'd run on that platform.

[Geoffrey.landis]

The no fly list doesn't identify people, just names, and it's very exact, so changing charles to chuck will defeat it.

No, actually it won't. The newspapers are full of stories of people who were detained or forbidden from flying because their name was similar to a name on the list, or a nickname of a name on the list, or a possible alternative spelling of a name on the list, or names that had once been used as an alias of names on the list.

for example, the name "T. Kennedy" was on the list. Senator Edward Kennedy (whose name does not begin with "T", but who is nicknamed "Teddy") was stopped:
from Wikipedia

"In August 2004, Senator Ted Kennedy (D-MA) told a Senate Judiciary Committee discussing the No Fly List that he had appeared on the list and had been repeatedly delayed at airports. He said it had taken him three weeks of appeals directly to Homeland Security Secretary Tom Ridge to have him removed from the list. Kennedy said he was eventually told that the name "T Kennedy" was added to the list because it was once used as an alias of a suspected terrorist. There are an estimated 7,000 American men whose legal names correspond to "T Kennedy". (Senator Kennedy, whose first name is Edward and for whom "Ted" is only a nickname, would not be one of them.)"

Re:I'd run on that platform.

[Free the Cowards]

It doesn't matter, because the only place where you have to get your ID checked is at the TSA checkpoint, and they don't check it against any databases.

So, the easy recipe for bypassing the no-fly list is:

1. Purchase tickets in a fake name.
2. Check in at home before your flight, and print your boarding pass on your home printer.
3. Using any number of techniques which are trivial to the computer literate, capture that boarding pass, alter it to match your real name, and print a second copy.
4. When you arrive at the airport, go straight to the security checkpoint.
5. Use the altered pass with your real name in combination with your real ID to get through security.
6. Use the original, non-altered pass to board the plane.

I flew as recently as last month and was not subjected to anything which would defeat this scheme. It fails if you need to check luggage, but I doubt a terrorist is going to be doing that. The no-fly list is such an obvious joke.

The actual report

[Americano]

I know this is slashdot and all, but if anybody's actually interested in looking at the full report, it's available for reading in pdf format online.

Re:There's a bigger problem with that.

[Duradin]

Reminds me of a bit from Discworld.

To summarize, Ankh-Morpork was over run by rats. The obvious solution was to put a bounty on rats, payable per tail. Soon, the rat infestation was under control but the number of tails being brought in kept increasing.

The Patrician's solution: tax the rat farms.

Paradox of the False Positive

[gknoy]

I realize this is likely starting to sound old, but Cory Doctorow's Little Brother should be required reading for people doing something like this. His writings about the "Paradox of the False Positive" are enumerated there, but also in other sources:

http://www.guardian.co.uk/technology/2008/may/20/rare.events

Statisticians speak of something called the Paradox of the False Positive. Here's how that works: imagine that you've got a disease that strikes one in a million people, and a test for the disease that's 99% accurate. You administer the test to a million people, and it will be positive for around 10,000 of them because for every hundred people, it will be wrong once (that's what 99% accurate means). Yet, statistically, we know that there's only one infected person in the entire sample. That means that your "99% accurate" test is wrong 9,999 times out of 10,000!

Terrorism is a lot less common than one in a million and automated "tests" for terrorism data-mined conclusions drawn from transactions, Oyster cards, bank transfers, travel schedules, etc are a lot less accurate than 99%. That means practically every person who is branded a terrorist by our data-mining efforts is innocent.

(emphasis mine)

And, as others have pointed out, this system is likely to have a false positive rate higher than 1%.

Re:There's a bigger problem with that.

[bendodge]

The GP isn't calling for vigilante groups turning in terrorists. He's calling for old-fashioned cloak-and-dagger HUMINT. It works far, far better than the technological circus we are operating now. Humans will always outsmart machines made by humans. The only real accomplishment of mass government data mining is the oppression of the general public who aren't interesting in outwitting the government. They're just trying to live their lives.

In the old days (Revolution, World Wars, Cold War), when we were aware of our enemies, spies, analysts and cryptographers defeated the enemies with courage, brainpower and skill. Now we've replaced them almost entirely with people in offices. This isn't going to change until we have another wakeup call, and the next one will probably come from Russia. The red bear is back, and we aren't prepared to deal with it (or China). Much of Russia's new technology is ahead of the US, particularly in aerospace submarine areas. We do not have a real missile shield, we do not have space-based weapons, we do not have supercavitating torpedoes (or anything to stop them). About the only encouraging developments we do have are in robotics and lasers.

China isn't very technological (except for those nasty anti-sat weapons), but they have an enormous mountain of people they don't mind sacrificing for whatever they dream up. Their standing army is over 2 million. They're also currently building and testing over one ballistic missile a week.
2005 article 2007 Article Oct 6, 2008

Terrorist data mining won't help much of anything when an EMP hits and the computers are fried.

The purpose isn't to flag terr'ists

[sydbarrett74]

Why do people still stubbornly insist that flagging 'terrorists' was ever the reason for all of this data-mining? Don't people understand the hidden agenda is to develop detailed dossiers on every single ordinary US national?