Northern Light Technology Makes Deal WIth C.I.A.

Llywelyn writes: "The C.I.A. has evidently written up a contract with the group Northern Light Technology to develop a search engine that can sort through the C.I.A's increasing mound of unprocessed data. Unfortunately, one of the consequences of this is that Northern Light's public search engine is fated for destruction later this month. " It's inevitable, IMHO, that some of this happen - the search engine world is overpopulated right now, and with the economic downturn, more and more companies will move to where they can survive.

Comments

Huh?

[Anonymous Coward]

Why does Northern Light have to shutter its public operation in order to handle the CIA's content? Surely there has to be a way for them to separate the CIA's stuff from their public index. Or is the announcement of the end of their public service merely coincidental to the announcement of the CIA contract, and not a condition of it?

I too feel there are too many search engines. It is extremely rare that I don't find what I'm looking for on Google, but it's not good to put all your eggs in one basket. While Google may be the best today, there needs to be competition in every market (well, almost every market).

not necessarily

[MoceanWorker]

the search engine world is overpopulated right now, and with the economic downturn, more and more companies will move to where they can survive.

according to this article posted back in August.. there should be even more search engines popping up in the future.. most of the ones on that list are still in beta mode.. personally, i like Teoma a lot :-)

Well someone has to do something with that data

[CDWert]

The CIA and FBI (and others) have masses of data so large that their entire staff reading for 100 years couldnt possibly sort it all.

It makes you wonder about the QUALITY of the data they are collecting,

Are they going o index their HOWTO's aka
HOW TO WIRE A CAT WITH A MIC (Seen previoulsy on slashdot)

As they teach you DAY ONE in Naval Intelligence
Everything is simply a piece of a puzzle, expect nothing monumental, no matter how small it is a detail of a bigger picture.

Now without addressing information in this manner and looking for ONE big hit , the process of intelligence gathering is broken.

YOU NEED SOMEONE to al least try to review this data before its all clumped into a selective search engine and forgotten. The CIA has horrible record retention policies in place. My bet 10:1 this is the worst possible fate of this data.

Probably just be better to auction it off to some willing buyer and hire more spooks to gather more data to auction off again, then maybe then the CIA could be considered usefull at least as a govt profit center :)

On Intelligence

[Proaxiom]

This brings to light a large misconception about intelligence.

As much as the focus tends to land on it, information gathering is not by any means the weakest link in the intelligence system. Probably we hear most about it because
a) it is glamourous (think James Bond), and
b) it often affects our civil liberties.

But the real problem with intelligence is the processing of retrieved raw information. They gather so much of the stuff it's extremely difficult to sort through it to figure out what's relevant and what's not.

That is why whenever something bad happens (like Sept 11) the intelligence community looks sloppy. In retrospect they can dig out wads of unprocessed information that would have given advance warning of the disaster. Then they take a lot of heat for missing it, even though they may not really be at fault. Sometimes it's a matter of finding a needle in a haystack.

It's a little more interesting to geeks because it's an issue of pure computer science. Processing raw data into meaningful information is computing at its best.

But developing better algorithms as a response to a national disaster is never going to be a solution that catches the public's imagination.

Re:On Intelligence

[DarenN]

Er.... I thought this article meant that it would be searching the CIA's own databases for the mounds of information that they can't quite link together to find the master plan.
Like the way the KGB could never find Bond, despite the fact that he always introduced himself as "Bond, James Bond".

If they can link all their data together (e.g searching transcripts for Osama (not Bin!) might have an interesting result).

Maybe that'd just make too much sense.

Illumination

[Jucius Maximus]

It seems that Northern Light may be the magic lantern they have been looking for. Perhaps they will find some illuminating facts nestled in their databanks.

Signs of convergence

[PHAEDRU5]

There's a lot of stuff out there on the web. So much so that the question now is to find exactly the stuff you need. It's almost as if you need an observor monitoring your actions, constructing semantic nets and offering suggestions.

Microsoft is making a stab at this using SmartTags. Of course, the intent there is not to make the web more useful, but more Microsoft.

Then there's the w3 symantic net (http://www.w3.org/2001/sw/), and finally there's the grand daddy of them all: Xanadu (http://www.xanadu.net/).

No real thoughts here, just an observation.

Re:Still Available ... Why Not Google?

[brassman]

I was surprised last night to find that NL had literally 100 times as many hits as Google for the children's story, "The Hobyahs". Two for Google vs over 200 for NL.

Now, that's unlikely to be what's driving the NSA decision but it sure makes me wonder how "deep" Google really is.

Too bad

[phatlipmojo]

NL is my second-favorite search engine (second to the mighty google), due in large part to its massive index and its ability to use complex Boolean (try that on google). When you really need to drill down and/or cut through a major noise:signal ratio, there's no better engine on the web, in my experience.
Anyone know of another decent engine with a good boolean implementation?