Slashdot Mirror
Northern Light Technology Makes Deal WIth C.I.A.
Llywelyn writes: "The C.I.A. has evidently written up a contract with the group Northern Light Technology to develop a search engine that can sort through the C.I.A's increasing mound of unprocessed data. Unfortunately, one of the consequences of this is that Northern Light's public search engine is fated for destruction later this month. " It's inevitable, IMHO, that some of this happen - the search engine world is overpopulated right now, and with the economic downturn, more and more companies will move to where they can survive.
I -so- want to see the list of the top search terms for that...
"pr0n"
"Where is Osama?"
"missing equipment"
"JFK"
"Castro"
...et multiple cetera...
You think they'll let me pay a little extra to make sure my sites go to the top of the list?
I could use the hits..
.... will still have access to the site for a fee. The agreement isn't with the CIA for exclusive use.
One of the neater features of Northern Lights was the folders. I liked how they organized the info and let it flow out of your continuous clicking.
What is funny is that Northern Light is actually a better search engine than Google for specific info. Northern Light drills down on the subject only and doesn't take into consideration links to the info as Google does.
Why does Northern Light have to shutter its public operation in order to handle the CIA's content? Surely there has to be a way for them to separate the CIA's stuff from their public index. Or is the announcement of the end of their public service merely coincidental to the announcement of the CIA contract, and not a condition of it?
I too feel there are too many search engines. It is extremely rare that I don't find what I'm looking for on Google, but it's not good to put all your eggs in one basket. While Google may be the best today, there needs to be competition in every market (well, almost every market).
They aren't a contractor, they are fully funded and run by the CIA. I think they are also the venture capital arm of the CIA as well.
Has anyone ever read the book CIA and the cult of intelligence ?
Anyone notice the blank line at the top of this article?
What's going on here? I smell a censor!
The cia has [deleted] civil liberties
[deleted] Former Director [delted]
without any thought
[deleted]
won't stand for this!
(please don't mod this down if you haven't read the book)
Touch everywhere, even when inappropriate.
the search engine world is overpopulated right now, and with the economic downturn, more and more companies will move to where they can survive.
:-)
according to this article posted back in August.. there should be even more search engines popping up in the future.. most of the ones on that list are still in beta mode.. personally, i like Teoma a lot
"The ones who dont do anything are always the ones who try to pull you down" -- Henry Rollins
The CIA and FBI (and others) have masses of data so large that their entire staff reading for 100 years couldnt possibly sort it all.
:)
It makes you wonder about the QUALITY of the data they are collecting,
Are they going o index their HOWTO's aka
HOW TO WIRE A CAT WITH A MIC (Seen previoulsy on slashdot)
As they teach you DAY ONE in Naval Intelligence
Everything is simply a piece of a puzzle, expect nothing monumental, no matter how small it is a detail of a bigger picture.
Now without addressing information in this manner and looking for ONE big hit , the process of intelligence gathering is broken.
YOU NEED SOMEONE to al least try to review this data before its all clumped into a selective search engine and forgotten. The CIA has horrible record retention policies in place. My bet 10:1 this is the worst possible fate of this data.
Probably just be better to auction it off to some willing buyer and hire more spooks to gather more data to auction off again, then maybe then the CIA could be considered usefull at least as a govt profit center
Sig went tro...aahemmm.....fishing........
A few months back I pointed out that the economic model of search engines is a bit flaky. Here's yet another carcass in the search engine war to prove the point.
Intriguingly, according to some the growth of the web is slowing. The last search engine to index the web before it reaches quiescence is the likeliest candidate for survival.
This is a plug:
Lots of companies have vast amounts of data, both text and otherwise to which they need access. The company I work for, Maxim-IT, Inc focuses on search technologies for non-structured data (like CAD files and the like).
to me though, a contractor is a private entity [hired | created] for a job and funded by the sponsoring organization. Here, the private entity is In-Q-Tel, sponsor is the CIA.
The One Rule Of Chess You'll Ever Need: Don't play someone who carries a kit in their bookbag.
As much as the focus tends to land on it, information gathering is not by any means the weakest link in the intelligence system. Probably we hear most about it because
a) it is glamourous (think James Bond), and
b) it often affects our civil liberties.
But the real problem with intelligence is the processing of retrieved raw information. They gather so much of the stuff it's extremely difficult to sort through it to figure out what's relevant and what's not.
That is why whenever something bad happens (like Sept 11) the intelligence community looks sloppy. In retrospect they can dig out wads of unprocessed information that would have given advance warning of the disaster. Then they take a lot of heat for missing it, even though they may not really be at fault. Sometimes it's a matter of finding a needle in a haystack.
It's a little more interesting to geeks because it's an issue of pure computer science. Processing raw data into meaningful information is computing at its best.
But developing better algorithms as a response to a national disaster is never going to be a solution that catches the public's imagination.
For those non-English speakers, "IMHO" means "In My Honest Opinion".
Non-English speakers are probably going to have more trouble with the rest of the words in that sentence...not to mention your explanation.
It seems that Northern Light may be the magic lantern they have been looking for. Perhaps they will find some illuminating facts nestled in their databanks.
"Unfortunately, one of the consequences of this is that Northern Light's public search engine is fated for destruction later this month."
According to fuckedcompany.com, they will be converting their search-engine service to a paid-only model.
You make it sound like working for the CIA is some odious move of last resort. Perhaps the management and staff of Northern Light is excited about working with the intelligence agency. Perhaps they see it as a way to help their country. Perhaps the processing of terabytes of data is a thrilling prospect from a purely intellectual point of view.
The standard /. dislike of all things governmental is not necessarily mirrored through all geeks.
In-Q-Tel is a venture capital fund set up by the CIA to invest in technologies that may serve the intelligence community. This is not a CIA contract for search technology. The CIA was Verity Inc.'s first big customer and as far as I know, they're still mostly using Verity internally (I managed Verity Internet products for years).
And the article misspelled Gilman Louie's name...
Nick
This is just not true. Over the last year, more and more search engine companies are effectively consolidating - by licencing "search engine technology" from another company. The real down side of this is that the more popular (popular by the licencing...not by users) are a "pay" engine, whereby companies can move their listings higher in the rankings by paying a fee.
This has two main side effects.
One, there are a lot of search engines out there that are really the same search engine. Same query, same results.
Two, when you search with them, you're not really getting what you asked for, but what someone payed for.
I understand why companies are doing this - there isn't a really strong revenue model for search engines right now; banners don't cut it.
I suspect that soon, good search engines will just be a (hopefully) inexpensive pay site, where you pay $30 a year and can use that search engine.
/dev/psychic: No medium found
There's a lot of stuff out there on the web. So much so that the question now is to find exactly the stuff you need. It's almost as if you need an observor monitoring your actions, constructing semantic nets and offering suggestions.
Microsoft is making a stab at this using SmartTags. Of course, the intent there is not to make the web more useful, but more Microsoft.
Then there's the w3 symantic net (http://www.w3.org/2001/sw/), and finally there's the grand daddy of them all: Xanadu (http://www.xanadu.net/).
No real thoughts here, just an observation.
668: Neighbour of the Beast
Pretty stupid if you ask me. I doubt they are then intelligent enough to change the spider agent string and route the spider through various innocent proxies to disquise it is the FBI spider...
I mean, what does the FBI do? Enter in "warez" into a search engine and go out and bust heads depending on the results returned?
You know what I really thin@~.~.~.~.~.~~~..~~#~~
NO CARRIER
At the end of last year Kuro5hin went down. Part of the reason for this was a post in which one person detailed how easy it would be to whack a certain high ranking official in office right now. Soon after that post was made the Secret Service was banging on that guys door. The internet is not a secret place, it is not an anonymous place, and lots of people are interested in what happens on it. Don't delude yourself into thinking otherwise.
The middle mind speaks!
Anyone else remember "Snow Crash"? Where there's all that information-gathering so that "it got so there was no real difference between the CIA and the Library of Congress, so they merged and kicked out a big stock offering". Book was before the internet became big, but this sounds like the way it's going to go! :-)
Grab.
I suspect that soon, good search engines will just be a (hopefully) inexpensive pay site, where you pay $30 a year and can use that search engine.
But can you imagine all the bad possiblities if they were able to actually tie all your searches together and see WHAT YOU searched for? Sure they can do it by i.p. or cookie, but an actual account, probably verified by credit card?
On the otherhand, a search engine is a basic need to use the internet. And I'd be quite surprised if some of them didn't start heading this way, REALLY CHEAP though. Incidently I don't know exactly when google.com became my ONLY search engine, replacing altavista.com, but it happened. Probably because of the excellent results (not perfect though) and the light interface.
On another note, I get the BEST referrals from google.com to my site. I get the MOST referrals from msn.com to my site. I say BEST from google.com, because the people that find my site through them, most likely want to see my site, and end up staying. MSN's referrals are usually pretty broad topics.
Northern Light downgraded its relevance about two years ago when they weren't generating enough capital and decided to charge its visitors for "premium" articles. At that point, NL effectively ceased to be a search engine and became an information broker. Nothing's really changed.
"Folks just call him Buckethead." -- Les Claypool
NL is my second-favorite search engine (second to the mighty google), due in large part to its massive index and its ability to use complex Boolean (try that on google). When you really need to drill down and/or cut through a major noise:signal ratio, there's no better engine on the web, in my experience.
Anyone know of another decent engine with a good boolean implementation?
Nice things are nicer than nasty ones.
When you're looking for information, check out The Jargon File
// abbrev.
IMHO
[from SF fandom via Usenet; abbreviation for `In
My Humble Opinion'] "IMHO, mixed-case C names should be avoided, as mistyping something in the wrong case can cause hard-to-detect errors -- and they look too Pascalish anyhow." Also seen in variant forms such as IMNSHO (In My Not-So-Humble Opinion) and IMAO (In My Arrogant Opinion).
Anomaly
PS - God loves you and longs for relationship with you. If you'd like to know more, please email me directly.
But Herr Heisenberg, how does the electron know when I'm looking?
Reporter: You mean, the procurement officer paid $6,000,000 for grep?
Guy with face hidden: Yes.
Reporter: What happened to him when they found out?
Guy with face hidden: Well, they were going to hit him with a hammer, but they decided that would be hypocritical...
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
I couldn't care less about the search engine going away. I just always liked the AP News Feed they had there.
I hope it will still be available. I've tried others but not found a better one. Anyone else?
Better yet, a CIA version of the Google Zeitgeist.
Any sufficiently well-organized community is indistinguishable from Government.
what with Google, Yahoo, and ... MSN?
How about this definition: A contractor is a private entity with a contract.
Duh.
The CIA gives In-Q-Tel orders, they don't sign deals.
my old sig used to be funny, but then slashcode ate it and now it's not funny anymore
The CIA tells In-Q-Tel what they want, and IQT delivers it. How they do that (for the most part), doesnt matter to the CIA, as long as it's done within the given requirements. In this case, Northern Lights is being hired by IQT to handle some of the jobs related to IQT's CIA contract to handle data processing. Therefore, it is IQT hiring on Northern Lights, not the CIA.
The One Rule Of Chess You'll Ever Need: Don't play someone who carries a kit in their bookbag.
That isn't true anymore. Try searching for "to be or not to be". Google used to require you to put +'s on the stop words, but they stopped requiring that in phrases a while ago.
BTW, You might notice that some pages don't actually have the phrase you were looking for. That generally means that there's are links pointing to that page with that phrase in them. For example, if you search for "html 4 spec" you get this page. That page doesn't contain the phrase "html 4 spec", but there are lots of links pointing at it that contain that phrase, so it's probably the page you want if that's your query.
Interesting/amusing query: "click here"
Contractors get paid, and they get to decide how they are to fulfill their requirements. In-Q-Tel has no autonomy: anytime the CIA wants to change its marching orders, it just says aye.
We have a similar concept here in Canada: We call them Crown Corporations. The CBC (and its bigger cousin the BBC) is not a contractor, providing television services for the government of Canada. It's a part of the government. It's just not organized like a typical government agency, but rather organized like a corporation.
I wonder: Have you ever worked for a government contractor, or in government? The difference is pretty huge. (I've done both.)
my old sig used to be funny, but then slashcode ate it and now it's not funny anymore
Northern Lights is a contractor. (We're not arguing about that. We're technically arguing about whether they're a sub-contractor.)
You seem to believe that everything that receives government money is a contractor. This would make, for example, the BBC a contractor. This is my final post.
my old sig used to be funny, but then slashcode ate it and now it's not funny anymore