Data-Sifting For Timely Intelligence Still an Elusive Goal

gyrogeerloose writes "Although there was evidence to suggest that the Japanese navy was up to something in December 1941, that information was scant and came too late. Today's intelligence agencies have another problem altogether — more information than they can deal with, and computers aren't helping as much as one might expect for reasons that will be familiar to Slashdot readers: computers can crunch numbers faster and more accurately than humans, but they're still easily baffled by language as it is commonly used in the real world. Metaphor, slang and simple figures of speech can confuse the best algorithm and, as quoted in the linked article in the San Diego Union-Tribune, 'A system that takes a week to discover a bombing that will occur in a day isn't very useful.'"

Comments

Feed it Monty Python

[Quiet_Desperation]

Squadron Leader: "Top hole. Bally Jerry pranged his kite right in the how's your father. Hairy blighter dicky-birded, feathered back on his Sammy, took a waspy, flipped over on his Betty Harper's and caught his can in the Bertie."

Computer: WTF?

Pilot: "Bunch of monkeys on your ceiling, sir! Grab your egg and fours and let's get the bacon delivered."

Computer: (explodes)

Re:Feed it Monty Python

[Quiet_Desperation]

No.

Obviously...

[fuzzyfuzzyfungus]

This is why we need to legally mandate that all human communication occur in newspeak. Ambiguity is the enemy of security.

Re:Obviously...

[SEWilco]

Ambiguity IS security

That depends upon what the meaning of "is" is.

This stuff is hard. I do it every day...

[Anonymous Coward]

We use an expert system to try and figure out what air traffic controllers are doing in a simulation. It is a big system and making it fast means trading alot of memory for speed. Identifying rules to categorize what a subject is doing is hard, especially because you see things that arent' expected when you think about what rules the system can use to identify a category of interaction. Looking at a stream of recording of system events is similar to looking at a stream of intelligence hits like 'subject crossed border x', 'subject a called subject b', 'subject purchased x, y, and z with credit card #k at mid #l with location coordinants (m,n)' The hardest part is that the system wants context but computers don't do context very well. To do it fast, you have to come up with vectors representing context state and rules and accept a certain amount of errors. Data can easily run into the hundreds of gigabytes for only 1 hour of monitoring a self contained experiment. It is fun though...

Any statistician could have told them that

[Opportunist]

There's one thing that's worse than too little data: Too much data that may or may not be relevant to your task.

It's bad to have no data. But that can be remedied. Having more data than you can process, worse, data where you cannot discriminate between wheat and chaff is pretty much useless. And that's basically what we have now. They were busy collecting data left and right, not asking whether that data could be relevant. Now they're stuck with a buttload of data that may or may not be relevant.

The best solution? Toss it and start over. And this time, collect only what's relevant.

Re:Any statistician could have told them that

[SpeedyDX]

Isn't that kind of begging the question? The problem here is, as you said, not being able to discriminate between useful and useless data. So how do we know what's relevant (a.k.a. useful)? Do we only collect data by using humans interpret the data? If so, then the role of the computer is much diminished. Do we automate the process by having computers discriminate between useful and useless data? Well, that's exactly the problem - we can't figure out how to do that yet. Even if we only have relevant data, how do we assign semantic value to the data in order for the computer to properly parse the data and give us semantically useful results?

It's not as simple as just collecting relevant data. Even if it were, that in and of itself is a major hurdle.

The Real Issue

[carp3_noct3m]

Is not that technology can't sift through some language as easy as it wants to. The problem is that (I am speaking for US and some allies) we have moved away from HUMAN intel. All the technology in the world can be rendered useless when for example, terrorists cells start to use face to face only communication in a tree like scheme. Only a few people ever talk to the people that pull the strings, and they talk to only a few people, only by passing letters or by talking face to face. I took a counter-surveillance course where I was amazed at the relative ease it took to shake even trained professionals. (It was also very fun to learn how to make drops and such) The point being that if someone really doesn't want to get caught, especially in a foreign country, its not too difficult. Humans are vulnerable, weak, and irrational beings capable of cognitive dissonance at every corner. If you want real intel, start focusing on HUMINT again. I read a very good book written by the guy whom the movie "Syriana" was based. Basically it boiled down to the CIA moving away from tried and true practices of gathering intel through human means, and becoming heavily reliant on both technology and politics to get stuff done, a major factor why he retired. Anyway, just my two cents.

Re:The Real Issue

[darkmeridian]

Intelligence is worthless without an intelligent decision-making process. During the run-up to the second Iraq War, the CIA sent Iraqi-Americans related to Iraqi nuclear scientists to inquire about the status of that country's nuclear program. Thirty Iraqi-Americans were debriefed by the CIA and sent independently of each other. All thirty returned with news that the Iraqi nuclear program had been run into the ground by a relative of Saddam, that the scientists lied about their progress to Saddam to stay in his good graces, and that Saddam was bluffing by denying UN inspectors.

In fact, a few scientists reported that Iraq had no real capability to make nuclear bombs since the early nineties. A crucial centrifuge facility had been destroyed in the first Gulf War. The facility had been unknown to Western intelligence until Saddam's hand-picked boss ordered it to be moved to a safer location. American intelligence detected the activity. They didn't realize that was a nuclear processing facility but knew it was a military target. Thus, the facility was put as a secondary target on the Air Tasking Order designating targets for air bombardment.

One day, a fighter-bomber returning to its carrier had unexpended laser-guided munitions because its primary target had been masked by weather. Back then, American planes could not land with unexpended munitions because the explosives were not inert and posed a risk of fire or explosion. The air traffic controller directed him to the nuclear facility. The bombs hit their target and that was the end of the Iraqi nuclear program.

Re:The Real Issue

[timeOday]

OK, let's look at the recent case of Nidal Hasan: sigint detects his communications with somebody considered an extremist, law enforcement detects some troubling internet postings maybe attributable to him, other officers report him having potential problems, his case is looked into very carefully by the right people, but they decide not to act and disaster ensues. What went wrong here?

I would argue, probably nothing. In hindsight, it is always easy to second-guess why the system "didn't work," but in fact, all these same clues occur in thousands of other cases where nothing ever comes of it. So, much of this disappointment comes from people unconsciously (and always in retrospect) hold systems up to an impossible standard - not only where an infinite number of KGB agents have an infinite amount of time to track everybody, not only where a super-intelligent AI acts in the most rational possible manner, but beyond that, demanding prediction of the future given some information which is relevant, but nevertheless insufficient.

Re:HUMINT SIGINT

[houstonbofh]

Which is why human intelligence is much more useful than signal intelligence (data mining).

Exactly. If people can't sift through the mass of information (and misinformation) we have today, what hope does a computer have? Just look at how hard it is to find "The Truth" in todays news, or on the Internet...

America forced Japan's hand

[BadAnalogyGuy]

Japan had only one real front in the lead up to Pearl Harbor. For all intents and purposes Japan was only focused on expanding westward into Asia. They envisioned "breathing space" like Germany did and meant to build an "Eastern co-prosperity sphere" led by an enlightened Japanese government. Naturally there was some resistance from the neighboring countries, but America and Japan didn't really have any reason to fight except that Japan was allied with Germany and there was a greater anti-imperialist zeitgeist among the Allies.

So when America decided to blockade South Asian shipping routes to effectively starve Japan of steel and other necessary resources, the Japanese had only one recourse. They bombed Pearl Harbor in an attempt to destroy as much of the American fleet as possible in the shortest amount of time. It was strategically the right move.

Now, if you want to say that the American military had its head up its ass that fateful morning, you'll find support from most historians. But to make the claim that no one expected an attack is simply absurd.

Re:America forced Japan's hand

[gyrogeerloose]

It was strategically the right move.

I think the outcome of World War II demonstrates that it was not the right move strategically. Tactically, perhaps, if the Japanese military planners were expecting the U.S. to enter into war, but it was a strategic disaster for them over the long run.

Re:America forced Japan's hand

[hardburn]

Granted, this was from the movie, but leaving planes out on the airfield and lots of shore leave for the sailors on the ships docked at Pearl Harbor is not what I'd call being on alert.

Planes were out on the field because they thought the biggest threat was from Japanese spys laying bombs in the hangers. They put them all together so they'd be easier to guard. To their credit, not one plane was lost due to a Japanese spy.

It's an interesting case of a defense against one form of attack making you more vulnerable against another.

Re:America forced Japan's hand

[MozeeToby]

You're missing the point, denying the Japanese access to steal and other resources during wartime was, for all intents and purposes, and act of war. Without those resources, Japan wouldn't have been able to hold the ground they had already taken, let alone continue advancing. When the US cut off access to critical war resources, Japan had only two choices: End the war almost immediately and retreat back to Japan proper, or take control of the resources by force. For political and ideological reason, the former option wasn't much of an option at all.

Imagine if the US were fighting a major war (against a powerful, conventional enemy) and OPEC said "No more oil exports for a while". You don't think the US govt would see that as an act of war?

Re:America forced Japan's hand

[gyrogeerloose]

I've got to disagree with you almost completely on this one. Japan made a slick tactical maneuver with their attack on Pearl Harbor but a huge strategical blunder. If Japanese war planners had paid any attention to their own intelligence, they would have realized that even if they had succeeded in knocking out the American Pacific fleet, all that would have accomplished was delaying the inevitable.

As it was, all the Pearl Harbor attack accomplished was to give the Imperial Japanese Navy six months of free reign in the western Pacific. It wasn't enough. The industrial capacity of the U.S. dwarfed that of Japan, which the Japanese military had known but chosen to ignore. The reasons given for that vary--perhaps arrogance, possibly belief in a divine destiny but no matter how you spin it, it was a strategic mistake.

And, by the way, the U.S. was not actively involved in the war in Europe at the time of the Pearl Harbor attack, although it was supplying weapons and materials to Great Britain to help them in their fight against Germany.

Re:America forced Japan's hand

[downhole]

If the Japanese had succeeded in wiping out the Pacific fleet in one fell swoop, it would have unlocked the entire Asian continent for them. American was already fighting a war on the European front. A crippled Pacific fleet would have made it impossible to keep up a second front.

No, it wouldn't. The bottom line is that our industrial capacity was just too much for them. It wouldn't matter what they did in the opening stages; we would have out-produced them and defeated them eventually no matter what. And there were more than enough production resources to smother both the Germans and the Japanese. (I know, technically the Soviets did most of the smothering of the Germans, but that's beside the point here).

The fundamental problem with the Axis powers is that they just weren't big enough to take on the whole rest of the world, and that their nationalistic/racist ideologies made it impossible to build true alliances. The only thing that might have actually worked for both Germany and Japan would have been to conquer just enough territory that nobody would complain too much, then consolidate and industrialize that territory, and repeat until they have a bigger war machine than their rivals. But that probably isn't possible if you believe that the citizens in all of the neighboring countries are all sub-human.

Re:HUMINT SIGINT

[Maximum Prophet]

Which is why human intelligence is much more useful than signal intelligence

People lie.

The US government is especially good at sending bogus signals. There's no reason to believe other governments aren't as good.

All intelligence has it's problems. The trick is to put together enough different sources to weed out the bogus, and home in on the truth, all while keeping everything secret. Basically, it's an impossible task, but sometimes it's good enough, and sometimes you go to war looking for WMD that aren't there.

Big Brother is watching you

[br00tus]

In the mid 1990s I watched a video tape of "The Falcon and the Snowman". It is based on a real story of a young man who worked at a sensitive location at TRW (his father was in the FBI and got him the job through the old boy network) which was responsible for sending and receiving CIA cables from overseas. Sometimes, they mixed up the TWXs and they saw cables they weren't supposed to. It was by this that he learned of how the CIA helped in the overthrow the government of Australia in the 1970s, the famous Whitlam constitutional crisis. You can read about it on Wikipedia. It can be debated how effective or ineffective the CIA's efforts were, but they've never denied their involvement, and in fact it was alleged that John Kerr was a CIA asset. Anyhow, so one day in 1997 or 1998 I was sitting at my SunOS x86 workstation at work, back before NAT had become popular, and I decided to surf the web and visit some lefty Australian web sites that discussed the extent of CIA involvement in overthrowing Australia's government in the 1970s. Several days later, I noticed SNMP requests coming into my workstation, scanning for any information about it. If I hadn't set my workstation to log absolutely everything, if it wasn't a UNIX workstation, if I didn't control the Cisco router and access list and so on and so forth I never would have seen it, it would have been a standard SNMP request. In fact, I didn't log for everything and who knows what other queries came to the machine. I saved the request for years but then lost it in a hard drive crash. It came out of a US army intelligence division (.MIL) that was based in Quantico, Virginia and which had some long acronym which I now forget. I thought the military wasn't supposed to monitor the communications of US citizens, but apparently not in this case. Also, as soon as I saw this, I thought of how I had read about Whitlam and the CIA on the Australian web site days before, and that was the only thing I had done on the machine that they might have been interested in. With the Patriot Act etc. who knows what will be happening.

No More Data Needed

[hardburn]

Today's intelligence agencies have another problem altogether — more information than they can deal with . . .

This is the ultimate argument against those defending increased surveillance activities to fight terrorism (or any other crime). Intelligence agencies already have way more information than they can deal with just from public sources. 99.999% of it is the noise of people going about their normal lives. Getting out the interesting bits is a hard problem, and adding more is only going to slow you down. It can help if you've already nailed down a good list of suspects and therefore have a small, targeted list of people to watch. But if that's the case, what's the big deal about getting a warrant?

The problem is misunderstoood...

[petaflop]

...at least as presented in the article, which frequently assumes that the text of a communication carries its meaning. The article keeps hinting at the problems, but comes back to the position that you just to make more links with more text. That's simply not true, and shows a rather understanding of the nature of language.

The meaning of a piece if a communication involves not just the text, but the specific context (who is the source, who is the recipient), the social context, and the cultural context.

For an example of the first - a 8 year old who says "I'm going to shoot her" (especially if the context is a game of cops and robbers) should be understood differently to an adult to says the same thing. And the meaning also varies depending on whether the adult is a photographer or not, and whether 'her' refers to a model or an ex-wife. None of these things may be made explicit anywhere in a any intercepted communication.

As another example, a description of a gory murder by a wild animal carries a very different meaning if the text starts with the words "Once upon a time".

You can't separate text, meaning and culture and consciousness. Which is why the problem of interpreting natural language is so hard; harder than even the article author seems to acknowledge.

Re:HUMINT SIGINT

[megamerican]

Human SIGINT is flawed because they can easily be manipulated, compartmentalized and shut down when neccessary. You can also be relying on people who are flawed morally, intellectually, etc...

An Example

15 of the 19 hijackers fail to fill in visa documents properly in Saudi Arabia. Only six are interviewed. All 15 should have been denied entry to the US. [Washington Post, 10/22/02, ABC, 10/23/02] Two top Republican senators say if State Department personnel had merely followed the law, 9/11 would not have happened.[ AP, 12/18/02More]

At least 13 of the Sept. 11, 2001, hijackers were never interviewed by U.S. consular officials before being granted visas to enter the United States, according to a congressional report issued yesterday. The finding contradicts previous assurances from the State Department that most of them had been thoroughly screened.

The General Accounting Office also found that, for 15 hijackers whose applications could be found, none had filled in the documents properly.

...

The GAO report found that all 15 of the hijackers from Saudi Arabia applied for visas in Jeddah or Riyadh; two others applied in their native United Arab Emirates. The remaining two, including ringleader Mohamed Atta, an Egyptian citizen, applied as "third-country" applicants in Berlin.

None of 18 separate visa applications by 15 of the hijackers was completed properly, the report said. Thirteen of the 15, who were from Saudi Arabia or UAE, were never interviewed before being approved for a visa, the report found. Investigators were unable to review the applications for four other hijackers, including Atta, because they were destroyed.

If you want to see the actual Visas of some 9/11 hijackers you can go here.

If you want to know why people with such obviously fraudulent Visa applications can get in to the country consider the testimony of J. Michael Springmann. He worked at the Jeddah consulate approving Visas and says he was occasionally overruled by the CIA. Remember that when Springmann was working there they weren't known as terrorists, they were still called freedom fighters.

Re:For example

[Lumpy]

You think that is fun, Try doing a commercial shoot with a team at an airport.

"Ok, we need to set up there and there. we can start shooting when the 12:30 planes arrive to get good coverage as the most people will be here."

Homeland security and TSA people are very dumb and can not understand Film or TV jargon. Nothing like having your entire team detained past the shooting time because the morons refuse to call to administration to clear things up.

Time for a Reminder ...

[foobsr]

Computers and Common Sense, the Myth of Thinking Machines. 1961 by Mortimer Taube.

Still valid, but mostly unheard of.

Interesting that in the English Wikipedia there is even no article on him.

CC.

Easy Fix

[AK Marc]

The problem is both hard and simple at the same time: Hire Google-like problem solvers. There are a lot of people really good at logic problems. However, companies (including the government) don't hire smart people because they are smart. They hire skillsets and such (when a smart person could be taught the skill set and get more done in 6 months than hiring a mediocre person that already has the skillset for the position). Don't look for programmers. Don't look for linguists. Find people that can solve complex problems. Motivate them. Give them resources like programmers and linguists, and the ability to study those subjects directly as well. And then the problem will be worked on. But having bureaucrats trying to fix the problem will have them not even consider the route that will get them the solution fastest.

It's a cryptography problem. There's information stored in codes. Sometimes the code is regular language, sometimes slang, sometimes coded language, but it's all decoding meaning from words. Problem solvers are better at solving the problem than having someone program a solution when no one actually figured out the solution, or having some linguists come up with direct matches that miss a large portion of what they want and get huge numbers of false positives.

But hiring someone that doesn't know what they are doing and training them is anti-American. We'll import our labor at a higher cost than actually train someone for the position. So I don't think anyone will ever do it. Google proved me wrong, but it doesn't seem anyone else is following their lead.

A week? A day? Whatever.

[ljwest]

A system taking a week to discover a bombing that will occur in a day's time will (by Moore's law) break even in 33.9 month's time. I.e., in 33.9 month's time, it won't take week - it'll take a day. So keep developing - it'll be viable in three years.

Re:For example

[Hognoxious]

Are you a TSA agent?

Re:HUMINT SIGINT

[flyingsquid]

Exactly. If people can't sift through the mass of information (and misinformation) we have today, what hope does a computer have?

An individual person can't, the volume is too large, and the information is often dispersed. However, large networks of people could potentially gather and bring together that intelligence in a timely fashion so that it can be acted on. Obviously, this didn't happen before Pearl Harbor, or 9/11. But can this type of intelligence gathering be done in the real world- can we collect dispersed information, bring that information together, and do in hours, rather than days? The answer is "yes", and to see why, just look at the DARPA balloon challenge.

I really didn't get what DARPA's red balloons were all about until I read this article. It seemed sort of abstract, something about social networks. I suspect that this question (how do we collect and assemble those needles of intelligence from a vast haystack of noise?) is the question that DARPA is trying to answer with those balloons. In the case of Pearl Harbor, or 9/11, or the Fort Hood shootings, there was actionable intelligence. The problem was that there wasn't a mechanism to collect that intelligence; the people who knew the facts, and the people who needed the facts, didn't know each other. What DARPA asked was: how can we collect intelligence when the intelligence is held by different groups of people (think different government agencies, like CIA, NSA, FBI, Customs and Immigration, different governments, or just people in the streets) and those people don't know the people who need the information (higher-ups in Homeland Security or the White House)? We know what failure looks like (Pearl Harbor, 9/11) but what does success look like? How and why do certain intelligence-gathering systems actually work, when they do work?

Their unconventional approach here was to set up an intelligence problem (balloons dispersed all over the country, need to collect info within 24 hours) and then let other people figure out how to solve it. Obviously terrorists will not be painted bright red, clearly marked, and stationary, but the principles of effective network intelligence can be applied to more difficult problems. I suspect that DARPA is going to spend a lot of time studying the data about how the intel came in for these various efforts.