Open Source Intelligence

Artifice_Eternity writes: "Time magazine is running a story highlighting the US government's neglect of open source intelligence, or OSINT. OSINT includes stuff on the Internet and in various newspapers and periodicals, as well as "gray literature" (limited-availability publications like dissertations, local phone directories, etc.). It also includes foreign-language experts, and commercial data (satellite maps, news archives, scientific research). The mass of data to be crunched indicates how intelligence is an information processing problem in today's world."

Comments

Interesting premise, but realisable?

[ari{Dal}]

Unfortunately, our spies and our satellites have lost touch with reality, for they collect less than 10% of the relevant information that we must digest to understand the complex multi-cultural world that is now capable of producing very wealthy and suicidal terrorists.

There's a good reason for the above mentioned figure... While I agree that there's a lot of useful information on the net, there's also a lot of crap.

Any intelligence agency looking to filter out the 99.9% of nonsense that's out there to glean the remaining .01% of useful information faces an incredible challenge. That's not to say that it's impossible, just very, very difficult, time-consuming, and expensive. For every real threat being posted on the net, there are tens of thousands of harmless, steam-blowing rants posted. And how do you decipher between the two? Do we now get into investigating every idle threat someone posts on a bb, or in a chat channel? The sheer scope of this project would daunt even the most dedicated Government sanctioned snoop.

Just as an example, sift through the comments of a /. article at -1, and you're guaranteed to find at least one or two flame-ridden rants about god knows what. Or look at your own past history. How many 'harmless' comments have you made about the stupidity of this or that idea, or how you'd like to kill that person for doing this? You know its harmless. Anyone who knows you probably thinks the same. But how does someone completely outside of your community know?

The article does make a few valid points, however:

Shocking as it may seem, our intelligence community does not routinely strive to identify the top people in the world (not just Americans) on the various topics of concern -- from terrorism to the environment to human trafficking to corruption to disease and public health -- with the result that our analysis tends to be shallow and incestuous, relying on the same consultants again and again.

I think just about anyone who lives outside the US looking in (as I do) would agree with this statement; one has only to observe the lack of knowledge American citizens display with regards to the rest of the world to see that this attitude is quite widespread, and probably does affect intelligence gathering. Raise your hand anyone who's seen the (Canadian) 22 minutes special "Talking to Americans". It's rather depressing actually, to think that so many people, including prominant politicians, could believe that Canada works on a 20 hour clock, or that we're going to change the country's name to Chicago (I'm not exaggerating either... quite a few people were taken in by this).

The Recommended Open Source Initiatives proposed in the article are interesting, though idealistic. One example: Digital History Project ($5M) to digitize and translate key Islamic, Chinese, and other foreign language historical, political, economic, cultural, social, and technical materials.. Having been involved in translation projects (French to English and vice versa) myself, I think he has seriously underestimated how much this would cost..Translation is an incredibly difficult and time-consuming activity; it's not a simple matter of babelfishing an article. Localized phrases and slang do not translate well from one language to another.

Before you can even begin to sift through the plethora of information, you'll need people that are very net and tech savvy. Combining tech skills with those of an intelligence agent is just the beginning. I won't even go into the thorny privacy issues that could be touched on here... that's just a political bomb waiting to go off.

Open Source Intelligence -- ?

[ciurana]

I first read the article on Sunday afternoon. It caught my attention because, like many on /., I made the right associations and tried to match "intelligence" with "open source" in the espionage sense. I was disappointed after reading 2/3 of the article, and didn't finish it. This was written by someone mostly grandstanding. The author focused too much on gathering data from all sources without giving enough thought to interpretation of those data.

I had the privilege (misfortune?) to work with a few intelligence types. When you talk to the people in the field, not to the public figures, public affairs wags, or the pundits, you will almost universally get the same two answers, rated in order of importance:

1. Intelligence analysis
2. Human level intelligence (HUMINT)
3. Fewer whiz-bang gadgets

The open intelligence article advocates only points 2 and 3, and barely touches on 1. Just like in coding, where the problem is not writing code but writing code that does something useful, intelligence is all about interpreting the data so that policy and actions may be appropriately channeled.

Dr. Ray S. Cline (former deputy director, CIA; look him up) once said that the world needed fewer spies and more critical thinking (I'm paraphrasing here a bit). Everyday disasters and attacks that could have been prevented still happen because there are too many toys and budgets and bureacratic fiefdoms to protect and there aren't enough ears who understand the bad guys' language, not enough cooperation between three-letter agencies, and not enough brains focused on making sense of the data gathered through various channels.

Thus, while part of the problem is gathering data, making sense of it is what will prevent another catastrophe like the terrorist attacks last September.

E