NSA To Use Cloud Model For Intelligence Analysis

Hugh Pickens writes "Information Week reports that the National Security Agency is taking a cloud computing approach in developing a new collaborative intelligence gathering system that will link disparate intelligence databases geographically distributed in data centers around the country. The system will house streaming data, unstructured text, large files, and other forms of intelligence data, and analysts will be able to add metadata and tags that, among other things, designate how securely information is to be handled and how widely it gets disseminated. For end users, the system will come with search, discovery, collaboration, correlation, and analysis tools. The intelligence agency is using the Hadoop file system, an implementation of Google's MapReduce parallel processing system, to make it easier to 'rapidly reconfigure data' and for Hadoop's ability to scale. The NSA's decision to use cloud computing technologies isn't about cutting costs or seeking innovation for innovation's sake; rather, cloud computing is seen as a way to enable new scenarios and unprecedented scalability. 'The object is to do things that were essentially impossible before,' says Randy Garrett, director of technology for NSA's integrated intelligence program."

Comments

Money well spent?

[siloko]

When will people realise than having more data often makes it more difficult to find the needle in the haystack. I am all up for utilising raw power in distributed networks to gain insights into patterns previously hidden but given the sheer breadth of proposed data formats to be mined this effort seems doomed to producing no tangible results at all. Except the ongoing expenditure of tax dollars.

Re:Money well spent?

[vertinox]

When will people realise than having more data often makes it more difficult to find the needle in the haystack.

The goal of government data collection isn't to find a needle in the haystack.

The goal is to turn more hay into needles.

Now how many crimes have you uknowlingly broke today, citizen?

Re:Money well spent?

[salimma]

You could say the same thing about search engines back in the mid-90s, before Google's PageRank. What's the point of indexing more and more information if it's impossible to find the relevant page? Necessity is the mother of all inventions. The only regret here is that whatever they come up with, probably won't see daylight until someone outside reimplements it.

Re:Money well spent?

[Bakkster]

"The system will house streaming data, unstructured text, large files, and other forms of intelligence data, and analysts will be able to add metadata and tags that, among other things, designate how securely information is to be handled and how widely it gets disseminated. For end users, the system will come with search, discovery, collaboration, correlation, and analysis tools."

More data is only bad if the signal to noise ratio doesn't also improve an equivalent amount. It doesn't matter if you've got 4x more hay to search through if you also can search the hay 4x (or more) faster. You might even find 4x more needles for the same effort.

Re:Money well spent?

[Anonymous Coward]

Siloko - It has produced tangible results in its original incarnation with a single data type - the question/bet Mr. Garrett is making is whether it will scale out and be able to handle much larger data files than Hadoop was originally built for. I hope he is hedging his bet.

Re:Money well spent?

[MindKata]

"Now how many crimes have you unknowingly broke today, citizen"

Well you've just committed one crime, you've shown decent, citizen, by joking about Big Brother ;). So we need to add you to more of our lists. (After all, we need to know who to round up first, when we next want to distract political opponents).

"more data often makes it more difficult to find the needle in the haystack"

The more haystacks they look through the more needles they can find. It'll just cost a lot more to build such a big system. Luckily money isn't such a worry for them. (For them that is, as they don't end up paying for it! ;) ... Ironic really, they use our money to spy on all of us.

As for tangible results, that's a very interesting question. Who exactly is this huge and ever growing distributed system going to data mine? ... I'm sure the idea of it is sold to everyone on the basis its all part of the war on fear, I mean war on terrorists (hiding on every street corner), but that's the point. Is this directed internally as well as externally?

We all know knowledge is power but with a system of such growing knowledge, what safe guards are in place to prevent greedy people in power exploiting this much growing knowledge (and power) for their own gain?

Because each time some more liberty is taken from us all, someone else (i.e. the people in power and their friends with the money) ends up profiting and gaining personally from our loss of yet more liberty.

In which case, Benjamin Franklin, must be spinning in his grave by now, i.e. "Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety." ... (Its amazing he said this over 200 years ago, yet still we make the same mistakes).

Re:Money well spent?

[siloko]

You could say the same thing about search engines back in the mid-90s, before Google's PageRank.

I couldn't agree more however the ramifications of inaccurate or misleading results are minimal - to the user if not Google's bottom line. What is at issue here is collating a resource from disparate sources, including vastly different formats, which would enable relevant agencies to better sort the wheat from the chaff. Having been part of a team aiming to standardise similar data sets to provide a search resource I speak from experience when I say this is no simple undertaking. Slashdot has recently posted an article on the difficulties of trawling datasets with tools which inevitably produce false positives, and the ramifications on false positives for the people in question are onerous to say the least. And this is without factoring in the liklihood that no more personnel will be available to follow up flagged content (people) so the net result is either an unused collection of positives or an accross the board policy change to haul in outliers under the pretense that they have been highlighted by 'powerful new technology'.

Re:Money well spent?

[WindowlessView]

Needles. Haystacks. How is it that it every government endeavor except intelligence agencies someone asks "Hey, exactly what is the cost per needle found in those haystacks?"

Why is there no commission that meets twice a year and announces to the public: we found 8 terrorists, killed 3. It cost 160 billion dollars. The commission should be composed of people the public knows and trusts. They can have their backgrounds examined by the agencies. They should give out as much information as possible w/o putting people or procedures at risk. But give the people who are paying for all some fucking idea of the efficacy of the operation. Or perhaps that is what they are really afraid of?

Re:Money well spent?

[salimma]

I guess it really depends on the details. The two bottlenecks are certainly worrisome: 1) the need for analysts familiar with both the system, and domain experts, to classify the data; 2) that the data is made available to a a wide range of users.

Let's hope the trial is realistic enough to bring up potential problems before real people get pulled in because of an overreliance on technology...

Centralized computing/storage

[Dareth]

Do the "Cloud Vendors" sell big cloud stickers to put over the data center portion of the design diagrams to hide the details from management?

This sounds like centralized computing and storage on dedicated servers. They are not going to buy a slice of some public cloud computing infrastructure.

   

Distributed == cloud?

[Red Flayer]

If in the cloud
The data you keep
The NSA
May get a peek.

Burma shave.

But seriously... Distributed data is now "the cloud"? Is my dirty laundry in "the cloud" because it is scattered in my bedroom?

Re:Distributed == cloud?

[Cro Magnon]

Only if your bedroom is online. Or at a very high altitude.

Tags For Cheney Intercepts: +1, PatRIOTic

[Anonymous Coward]

from his Naval Observatory spider-hole.

Yours In Espionage,
K. Trout

Cloud this cloud that

[croddy]

Listen, shut up about the "clouds" already. Just because you don't understand the architecture doesn't make it a "cloud". That word doesn't mean anything. I'm sick to death of hearing it already.

Re:Cloud this cloud that

[Em Emalb]

who are you talking to? Hate to tell you, Mr Preacherman, but the congregation is thataway. We're the choir.

Re:Cloud this cloud that

[thebian]

+1

And while we're at it, let's bury map-reduce, too.

Re:Cloud this cloud that

[El Torico]

Don't worry, if the Government wants to do it, then it's no longer hip. I expect the phrase "cloud computing" to go the way of "paradigm shift" and "mindshare" soon.
Too bad the NSA will piss away a few billion before they realize they aren't cool.

Re:Cloud this cloud that

[Anonymous Coward]

Everything are clouds these days. Slashdot is also about cloud computing. I had just defined it and hence it's true.

Re:Cloud this cloud that

[Anonymous Coward]

+1

And while we're at it, let's bury map-reduce, too.

Map-reduce actually has a specific meaning.

Re:Cloud this cloud that

[Anonymous Coward]

I like how you think the rest of the world gives a shit what you're tired of. I'm tired of morons but you're still breathing.

Re:Cloud this cloud that

[Anonymous Coward]

Isn't that called something along "divide et impera" ?

The Stasi, revisited . . .

[PolygamousRanchKid]

. . . the Stasi http://en.wikipedia.org/wiki/Stasi , the former East Germany's secret police, ended up collecting so much information on folks, that they couldn't process it all.

From the Wikipedia article: "When informants were included, the Stasi had one spy per 66 citizens of East Germany.[8] When part-time informer adults were included, the figures reach approximately one spy per 6.5 citizens."

Yo.

I guess the NSA thinks that they can do better than the Stasi with brute force computing power.

Re:The Stasi, revisited . . .

[gknoy]

Brute force computing power has certainly advanced a lot since the Stasi. Look at what Google does with a large volume of cheap hardware.

Re:The Stasi, revisited . . .

[DNS-and-BIND]

I swear to God, there has to be some extension of Godwin's Law for every time someone brings up the Stasi. What is the ratio of NSA informers to the American population?

Re:The Stasi, revisited . . .

[Bakkster]

I swear to God, there has to be some extension of Godwin's Law for every time someone brings up the Stasi. What is the ratio of NSA informers to the American population?

Exactly. Especially since the NSA is not (legally) allowed to analyze "signals" from US citizens, and the vast majority of their employees care about following that rule.

Re:The Stasi, revisited . . .

[Anonymous Coward]

Cold comfort when an NSA imp who doesn't care about that rule has targeted you.

Re:The Stasi, revisited . . .

[Anonymous Coward]

What is the ratio of NSA informers to the American population?

I don't know (obviously), as a German, I'll note that the reason the Stasi was evil was not that it had a lot of informers.

Rather, the reason it was evil that it did, well, evil things. The large number of informers merely allowed them to do that, and the GP's point is actually that the NSA does NOT need as many informers anymore now to do the same things the Stasi did.

Whether that means the NSA and the Stasi are comparable is another question entirely, though.

Re:The Stasi, revisited . . .

[Anonymous Coward]

I swear to God, there has to be some extension of Godwin's Law for every time someone brings up the Stasi. What is the ratio of NSA informers to the American population?

Measured in humans, infinitesimal.

Measured in megabytes of data and megahertz of processing power available, maybe not so small. There are only 300 megapeople to monitor, and we're talking about an organization that measures the sizes of its server rooms in acres.

Analysed Use of Intelligent Models on a Cloud NSFW

[srussia]

Geez, don't they teach editors to come up with snappy headlines anymore?

"have-fun-securing-that"

[John Hasler]

Google "air gap".

When too much is in the cloud it rains!

[Anonymous Coward]

terrorist assassinate csystems JUWTF Waco, Texas kilo class Telex fissionable benelux Ruby Ridge Operation Iraqi Freedom anarchy World Trade Center data haven covert video ASO anthrax United Nations Zachawi Plame cypherpunk jihad Capricorn Mole FTS2000 unclassified NSA ammunition NASA SAPO strategic Janet Reno COSCO jihad Zachawi Abu Ghraib JFK JPL Bletchley Park Craig Livingstone AMW spy illuminati terrorism radar

Information overload makes for a leaky cloud. Have fun weeding through the false positives assholes spook motherfuckers!

ammunition rs9512c Al Jazeera military encryption AIEWS AIMSX pipeline New World Order ANC Perl-RSA mindwar global Ruby Ridge Rand Corporation

Neat software

[Demonantis]

This would be really helpful for law enforcement with their need to be able to access data from other police counties. There is tonnes of cases where cases cross several jurisdictions and the police have trouble integrating the investigations.

Obligatory

[Anonymous Coward]

"Old Man Yells at Cloud"

http://i274.photobucket.com/albums/jj254/dear_die_hard/grandpa_simpson_yelling_at_cloud.jpg

Intelligent Intelligence?

[Anonymous Coward]

More geographic distribution = more points for hacker interception.
Intelligence has nothing to do with it.
Intelligent Intelligence = off-the-wall, off-the-air .
(Oh, but encryption will save us - yeah, right!)
Just a QA perspective

Re:Intelligent Intelligence?

[jamstar7]

No, this just means the data will 'disappear' into the cloud when some citizen comes walking up with a FOIA request to see the data. "Hey, all that shit got lost someplace on the net, we ain't got it!!"

Future cloudy, try again?

[gestalt_n_pepper]

Glad they're behind the eight ball on this one.

why not google docs?

[fluffernutter]

Since the average shashdotter seems to think third party server clusters a.k.a. "The Cloud" are good and secure and useful for all us human beings and the intentions of said third parties are all well and good; I'm surprised that no one has suggested that the NSA should just use Google Docs or Facebook and get over the supreme silliness and unneccesary cost of private servers. Oo. Or what about Google Wave. I'm sure there's something in that for the NSA, it is the new cool thing after all.

Good point

[sgt_doom]

Excellent point, besides, I thought the banks already had something like this called G.R.I.D. (Global Regulatory Information Database)?

Hey You!

[metaforest]

I store my data in a VM on the 99th U of the Rack
And I sit ~/ look at the windows
Imaging the world cash crops
Then in flies a guy who's all dressed up like a NSA Spook
And says, I've won five pounds if I have his kind of data packet

I said, Hey! You! Log off of my cloud
Hey! You! Log out of my cloud
Hey! You! Log off of my cloud
Don't login here two UIDs is a crowd
On my cloud, Baby

The VOIP is signaling
I say, "Hi it's me. Who is it there in the stream?"
A voice says, "Hi, hello, how are you"
Well I guess I'm doin' fine
He says, "It's three a.m. watch the SNR boy!
Don't you ever halt that simulation?
Just cause it runs so good do you have
to fill the drives to the end?

I say, Hey! You! Log off of my cloud
Hey! You! Log out of my cloud
Hey! You! Log off of my cloud
Don't hack around 'cause two UIDs is a crowd
On my cloud baby

I was hacking python, and fed up with spaces
And decided to connect to slashdot
It was so very quiet and peaceful
With all of the ACs banned no new postings around
I hibernated the laptop, I was so lagged I started to scream
In the morning the parked heads were just like
A write-flag stuck on my windows machine

I say, Hey! You! Log off of my cloud
Hey! You! Log out of my cloud
Hey! You! Log off of my cloud
Don't hack around 'cause two UIDs is a crowd
On my cloud baby

Hey! You! Log out of my cloud
Hey! You! Log off of my cloud
Hey! You! Log out of my cloud
Don't hang my process, baby two UID's a crowd!

-----

With apologies to Mick and the boys.