Slashdot Mirror
Northern Light Technology Makes Deal WIth C.I.A.
Llywelyn writes: "The C.I.A. has evidently written up a contract with the group Northern Light Technology to develop a search engine that can sort through the C.I.A's increasing mound of unprocessed data. Unfortunately, one of the consequences of this is that Northern Light's public search engine is fated for destruction later this month. " It's inevitable, IMHO, that some of this happen - the search engine world is overpopulated right now, and with the economic downturn, more and more companies will move to where they can survive.
As much as the focus tends to land on it, information gathering is not by any means the weakest link in the intelligence system. Probably we hear most about it because
a) it is glamourous (think James Bond), and
b) it often affects our civil liberties.
But the real problem with intelligence is the processing of retrieved raw information. They gather so much of the stuff it's extremely difficult to sort through it to figure out what's relevant and what's not.
That is why whenever something bad happens (like Sept 11) the intelligence community looks sloppy. In retrospect they can dig out wads of unprocessed information that would have given advance warning of the disaster. Then they take a lot of heat for missing it, even though they may not really be at fault. Sometimes it's a matter of finding a needle in a haystack.
It's a little more interesting to geeks because it's an issue of pure computer science. Processing raw data into meaningful information is computing at its best.
But developing better algorithms as a response to a national disaster is never going to be a solution that catches the public's imagination.