Snowden Used Software Scraper, Say NSA Officials
An anonymous reader writes with this excerpt from the New York Times: "Intelligence officials investigating how Edward J. Snowden gained access to a huge trove of the country's most highly classified documents say they have determined that he used inexpensive and widely available software to 'scrape' the National Security Agency's networks, and kept at it even after he was briefly challenged by agency officials. Using 'web crawler' software designed to search, index and back up a website, Mr. Snowden 'scraped data out of our systems' while he went about his day job, according to a senior intelligence official. 'We do not believe this was an individual sitting at a machine and downloading this much material in sequence,' the official said. The process, he added, was 'quite automated.'"
Who'd have thought? Experienced IT guy didn't manually download each file!?
"Inexpensive and widely available" - I hope they don't mean some evil subversive communist open-source tool.
You mean to tell me that an NSA tech contractor used wget or something, rather than loading up IE6 and clicking until his fingers fell off?
Knock me over with a feather, spooks. You fucking hired people to build what is probably the largest collection of signals intelligence scraping systems on the planet, targeted at a wide variety of differently structured systems. Why would you even consider, except as a last resort, the notion that you are dealing with a bunch of noobs?
(Oh, incidentally, maybe you should spend a bit less time reading everybody's email and work on that 'hilarious leaked diplomatic calls' problem, I'm told that sort of thing used to be your job at some point in the past...)
Maybe he typed it all...
Beta scrapes you!
If the network can't identify that something accessing the network sporadically and in repeated succession is a bot and should be stopped maybe the NSA shouldn't have access to this much data to start with....
What if a legitimate foreign hacker was able to get in and do the exact same thing? Obviously, they have very shitty standards when it comes to network security - you'd expect thousands of honey pots, ability to intercept attempted attacks, flat out network filtering of these kinds of requests. But alas, that would make sense!
This is why open source software is a good idea.
NO backdoors (that we know of)
FREE to download, modify, and redistribute.
FREE to use behind iron curtains even against dictatorships...
And for every open source tool used for evil there is one that can be used for good.
GNU wget and/or GPL httrack.
Go forth little lines of code, be fruitful and multiply.
*wipes tear from eye*
or httrack ftw. And you don't really need to write this stuff. :-)
"because he worked at an agency outpost that had not yet been upgraded with modern security measures."
"when he was questioned, Mr. Snowden provided what were later described to investigators as legitimate-sounding explanations for his activities"
Speechless.
I read it. Now, may I mod it down?
Anyone else notice that Snowden is increasingly being referred to as "Edward J. Snowden" instead of just "Edward Snowden"?
You use proper tools.
---- Booth was a patriot ----
Hey, NSA: ~$ rpm -q wget wget-1.15-1.fc21.x86_64
Geology - it's not rocket science; it's rock science
"This automated and indiscriminate bulk collection of data is unacceptable!"
Just modify your bookmark to
http://slashdot.org/?nobeta=1
Its the incantation used in the link at the bottom of the beta page, so its not as if you're using a commie hack. It just means that you're not randomly redirected to that festering midden of rancid shit that is Slashdot Beta.
Hmmmmmmmm.... Do you think my characterisation of the beta was a little extreme????
(Captcha "bellman". Ting Ting!!!)
Now the question is, how many other NSA contractors / staff / moles / spies have been doing the same thing, without Snowden's intention to disclose their behavior?
Gently reply
"Agency officials insist that if Mr. Snowden had been working from N.S.A. headquarters at Fort Meade .. he almost certainly would have been caught. But because he worked at an agency outpost that had not yet been upgraded with modern security measures, his copying .. raised few alarms."
This is retrospective ass-covering cyberbullshit. It is precisely at the edge that the security attacks would come from. What they were doing putting such material on Web servers and Wikis beggers credulity. Didn't senior management not realize that as keepers of the nations secrets they would be subject to attacks both internally and externally. Given the state of non-security at the NSA I would suspect that Snowden wasn't the only hostile with access to the “the keys to the kingdom”.
In his defense mr. Snowden explained that his scraper was only gathering metadata and therefore by their own standards the NSA has nothing to worry about.
When asked for comment NSA director James Clapper replied : "Whoa that's deep bro!"
And we were caught!
There's absolutely zero reason to believe anything the NSA says about how Snowden got the documents, or indeed, about anything. They believe they are entitled to lie to congress, so the public isn't even a question.
xkcd is not in the sudoers file. This incident will be reported.
Do you think he used wget or curl?
wget must be regulated in order to protect the freedom of the American people!
Please post this to new articles if it hasn't been posted yet. (Copy-paste the html from here so links don't get mangled!)
On February 5, 2014, Slashdot announced through a javascript popup that they are starting to "move in to" the new Slashdot Beta design. Slashdot Beta is a trend-following attempt to give Slashdot a fresh look, an approach that has led to less space for text and an abandonment of the traditional Slashdot look. Much worse than that, Slashdot Beta fundamentally breaks the classic Slashdot discussion and moderation system.
If you haven't seen Slashdot Beta already, open this in a new tab. After seeing that, click here to return to classic Slashdot.
We should boycott stories and only discuss the abomination that is Slashdot Beta until Dice abandons the project.
We should boycott slashdot entirely during the week of Feb 10 to Feb 17 as part of the wider slashcott
Moderators - only spend mod points on comments that discuss Beta
Commentors - only discuss Beta
http://slashdot.org/recent - Vote up the Fuck Beta stories
Keep this up for a few days and we may finally get the PHBs attention.
-----=====##### LINKS #####=====-----
Discussion of Beta: http://slashdot.org/firehose.pl?op=view&id=56395415
Discussion of where to go if Beta goes live: http://slashdot.org/firehose.pl?op=view&type=submission&id=3321441
Alternative Slashdot: http://altslashdot.org (thanks Okian Warrior (537106))
The idea of military specialists of whatever type being employed against the society they belong to, is treasonous and fucking retarded no matter what legal acrobatics are employed in their defense.
You may have some sort of mystic devotion to the law, but I believe laws are made by (generally corrupt) men for their own interests, and I am familiar enough with the world outside the borders and political influence of the United States to know there is an enormous difference between legality and rightousness. The U.S.A. may not be the kind of country where you are expected to bribe every public official however minor -- we generally reserve that for higher office. It takes a special kind of idiocy to use military forces against their homeland, though.
Government at its core is the body to which we have delegated our inherent right to violence -- a right being defined in this case as something which cannot be taken from you. We delegate this right to others, specialized in its use, with the express understanding that [a] as applied to civilian life, the exercise of violence by police will be applied fairly and equally as men can manage, and [b] that the unrestricted expression of this (as embodied by military force) be only employed against our enemies. War is hell, and we do not bring hell home.
Snowden is a patriot, and the NSA is treasonous -- whether or not the law can be made to serve whichever purpose. Beyond all other argument, potentially felonious violation of the law is so common with the continual proliferation of laws that lawfulness cannot be the only measure of either justice or rightousness. May all those who support the NSA have a fair trial.
Those who advocate genocide deserve every protection afforded by law, and none afforded by common human decency.
I'm trying to see how your sig. fits with your comment, and can only conclude you have no clue....
Down With Slashdot BETA!!! I've been around the corner and seen the oliphant; you can only abuse me from your perspecti
seriously? this is news?
Snowden used computer technology to gather the evidence??
GOOD ON HIM.
I look forward to the day when sNOwden hangs for high treason against the people of the United States of America.
Shouldn't the shock and horror be that Snowden was able to scrape the juiciest pages from the NSA information dump on basically everyone, without so much as a 403 error?
Actually most spy agencies are full of incompetent paranoid lunatics. The are the same people as the conspiracy nuts, just on the other side of the fence. They live in a bizarre world where they are so busy looking for enemy action and seeing enemies all over the place. If an operation fails its because the enemy was smarter, we need more resources to look closer. If they can't find a terrorist/spy its because they are really good at hiding and concealing their tracks.
They aren't able to entertain the idea that maybe what they are looking for doesn't actually exist.
Read this amusing essay about MI5
http://www.bbc.co.uk/blogs/adamcurtis/posts/BUGGER
SCP Script.
That spoonful of sugar can't mask the bitter taste of your own medicine, can it, fancy espionage agency?
The "Civilized World" jumped the shark ca. 1973.
one man didnt read millions of pages, he used software to get those millions. "this just in, someone followed the laws of sanity". idiots.
I'd have thought he went in each day with wheelbarrow full of 1.44" floppies and just copied until he got it all... That's some mighty fine detective work, Lou.
What is even more scary - nobody seems to have picked up that the floppy disks in question were 1.44 MB and were 3.5" in width - NOT 1.44" !!!
I love how they keep pretending they were on to him, but he was just so nefarious he escaped. You think a spy agency would be prepared to deal with spies. Or did they believe the polygraph results were actually accurate? There's a reason they are not allowed in courts of law.
For a spider (scraper?) to work, it has to get the filenames from somewhere, usually another file like ./index.html . I cannot see anyone building webpages of the memos, but they might very well be stored as files in some directory structure. Turning on dirlisting (or autoindex) is an invitation for total access -- http is a protocol for info you _want_ to spread. Not even the USG is that incompetent.
What might have happened is that netadmins like Snowden had uid/pwd that allowed ftp access (necessary to fix files). Then run the directories just as `archie` did 20+ years ago.
People are finding back door in open software on a daily basis. Wait until stories start coming out about hardware backdoors that have been coerced. In any case, get a clue.
slashdot troll = you make a compelling argument I do not like the implications of.
Is this supposed to convince me that snowden was a criminal instead of a patriot reporting on the behavior of corrupt government?
google is the biggest terrorist of them all then
SLASHCOTT FEB 10-17
putting it in my /etc/hosts file so I don't relapse :-)
/etc/hosts
/etc/hosts
BTW it is slightly evil of you to say
echo '127.0.0.1 slashdot.org' >
instead of
echo '127.0.0.1 slashdot.org' >>
Bye everybody! Bruce, you have my full mental support FWIW!
To be, or not to be: isn't that quite logical, Slashdot Beta?
Stop posting this message, attention whore !
What might have happened is that netadmins like Snowden had uid/pwd that allowed ftp access (necessary to fix files). Then run the directories just as `archie` did 20+ years ago.
The tool he had access to that was 'more advanced' than WGET was probably called Rsync
Who cares what tool he used? wget, some wrapper for it, etc... the point here is that the nsa is saying the information that snowden got was probably just what was available via the nsa's intranet, and maybe all shared drives with bad permissions, drop boxes, temp volumes that admins forget to clean out, etc...
and it can also be a subtle way of telling journalists to be careful about blindly trusting the data from snowden.
It's obvious that for someone trying to extract documents from the NSA network stealth is essential. Not spending too much time at work fine-tuning queries or wading through lots of material you don't have any business looking at is a start. Therefore using some sort of script or other software tool for the job is practically a necessity.
The suggestion that Snowden's "take" contained a disproportional amount of files related to military goings-on (as opposed to spying on US citizens) would need substantiation. Probably in the form of a copy of the script/tool he used plus the search terms used *plus* the ratio of military to non-military material in NSA's systems.
For example, it could very well be that search terms that cover how the NSA collects its data (Snowden's focus in publications) will return large amounts of militarily-oriented hits without specifically targeting military stuff. After all, NSA runs a large number of monitoring stations, most of whose data feed goes into military assessments rather than anti-terrorism related ones (I hear). So this mil/non-mil ratio would have significant impact on the ratio of mil/non-mil stuff that Snowden's query got him.
Somehow I doubt that NSA would like to divulge that ratio in a public court.
Who would host top secret documents on a web server, so any clown with HTTrack can download them all?
The Coen brothers' movie Fargo, as in " I'm not sure I agree with you a hundred percent on your police work, there, Lou. "
"Who would host top secret documents on a web server, so any clown with HTTrack can download them all?"
A government agency that wants a straw man to distract the public from other activities they are doing. I'm not worried about what Snowden is disclosing. I'm worried about what Snowden didn't get his hands on.
wget declared illegal. Film at 11.
Don't get me started on perl hackers. Heck, they even admit to be being evil, calling themselves hackers! They probably work in black magic too!
And you thought that Encrypted Media Extensions coming to HTML is evil!
If NSA followed future open (read "closed") standards to protect state secrets you would not have a problem like this.
Snowden would need to deal with serious messages like "Click here to purchase one copy of 'Guide to PRISM, part 2.' from NSA for only $4.99 incl. VAT"... And of course he would be limited to share such documents only with maximum of one additional reading device. That would definitely put the uncontrolled spread of information (for the people, by the people) to the end.
Plus if "wget" was not freaking-OSS he could read something like "wget is not licensed to be run on computer NSA-1AD1-489, please purchase full version"...
See how evil is OSS and open web?
Well, I've got to get back to work. When I stop rowing, the slave ship just goes in circles.
I have to wonder. Snowden's actions are right out of the playbook of the KGB. Maybe I've watched too many spy movies from the 80s but it's starting to ring like Snowden is a classic double agent working for Moscow.
Yeah, there's that Bond film too, where a KGB agent reveals to newspapers the illegal and intrusive plans of Bond and his agency. Wait...
Far from it. In fact if you're looking for illegal data-collection methods you know the mechanism but not the location, the name of the program, or the actual method.
Most database queries and WWW search engine queries work like that. Especially when the figure of merit is "don't overlook something we want to know about" and "minimise the effort on the query" instead of "minimise the number of hits returned".
After all, you can carry gigabytes of data with you on a USB stick and you can sift through it later when you're about to publish...
So err ... given the constraints I'd say it would be very hard to argue that Snowden went on a fishing trip.