Slashdot Mirror


Snowden Used Software Scraper, Say NSA Officials

An anonymous reader writes with this excerpt from the New York Times: "Intelligence officials investigating how Edward J. Snowden gained access to a huge trove of the country's most highly classified documents say they have determined that he used inexpensive and widely available software to 'scrape' the National Security Agency's networks, and kept at it even after he was briefly challenged by agency officials. Using 'web crawler' software designed to search, index and back up a website, Mr. Snowden 'scraped data out of our systems' while he went about his day job, according to a senior intelligence official. 'We do not believe this was an individual sitting at a machine and downloading this much material in sequence,' the official said. The process, he added, was 'quite automated.'"

36 of 227 comments (clear)

  1. Stunning. by quenda · · Score: 5, Insightful

    Who'd have thought? Experienced IT guy didn't manually download each file!?

    "Inexpensive and widely available" - I hope they don't mean some evil subversive communist open-source tool.

    1. Re:Stunning. by Anonymous Coward · · Score: 5, Funny

      Oh my god .... could it be .... wget ?

    2. Re:Stunning. by jsh1972 · · Score: 5, Funny

      I'd have thought he went in each day with wheelbarrow full of 1.44" floppies and just copied until he got it all... That's some mighty fine detective work, Lou.

    3. Re:Stunning. by fatphil · · Score: 3, Funny

      But you don't understand. Terrorists use wget for drug dealing with paedophiles who use slashdot beta. Wget must be banned!

      --
      Also FatPhil on SoylentNews, id 863
    4. Re:Stunning. by gl4ss · · Score: 4, Funny

      the malicious and now banned weapons grade tool is called "wget" by underground unix hackers.

      --
      world was created 5 seconds before this post as it is.
    5. Re:Stunning. by Arrogant-Bastard · · Score: 5, Insightful

      There's zero reason to believe the NSA's version of this and every reason to believe Snowden's

      Why?

      Because, so far, every single thing that Snowden has said has turned out to be true when cross-checked. And, so far, every NSA official spokesperson has been caught repeatedly lying.

    6. Re:Stunning. by VortexCortex · · Score: 4, Interesting

      The very first program (after "Hello World") I wrote in Java was a website scraper. I used it to download all of Sun's API and tutorial pages for Java and rewrite links to be relative. Younger and dumber. This created two copies of each set of docs: The scraped version and the compressed version, which I only discovered existed after having scraped the downloadable offline version of said docs.

      Point being: My scraper was written in a few hours and far less powerful than wget.

    7. Re:Stunning. by DarkOx · · Score: 4, Insightful

      Well if you knew a SIEM system had rules which might trigger alters if a database backup is started off hours or if the backup files are accessed for one. As apposed to normalish get query logs with 2XX results, its likely been trained to ignore.

      Questioning if who you might eventually leak the data to will have the technical chops and resources put the information together from the database file, as opposed to just reading through a bunch of handy precomputed html pages and office documents for another.

      --
      Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
    8. Re:Stunning. by dcollins117 · · Score: 4, Interesting

      Now that we have documented proof of a rogue government agency that ignores the law and then lies about it, I'm waiting for some enterprising criminal defense attorney to realize they've got the perfect patsy. Regardless of what crime their client is alleged to commit, just deny involvement in the crime and claim that it was committed by the NSA.

      I'm not a lawyer, I just play one on Slashdot. But it seems to me that should be sufficient to raise reasonable doubt.

    9. Re:Stunning. by s.petry · · Score: 3, Funny

      Slightly more powerful than wget to me is a wrapper around wget. Perl and Bash scripts are way beyond the average users. To politicians scripts can be used to claim "voodoo" or "saintly" depending on who writes the scripts. The NSAs scripts are obviously saintly, while anybody else is probably voodoo.

      --

      -The wise argue that there are few absolutes, the fool argues that there are no probabilities.

    10. Re:Stunning. by Jeremiah+Cornelius · · Score: 5, Insightful

      You're ALL missing the point.

      This is not a "news item" because he used some "arcane technology" from the view of the uninitiated.

      This is a calculated iteration in the advancement of declaring web automation tools "instruments of terror" and "cyber weapons", for prosecutorial purposes.

      You vilate a ToS with these, and WHAM! Terror suspect!

      --
      "Flyin' in just a sweet place,
      Never been known to fail..."
    11. Re:Stunning. by Chris+Mattern · · Score: 3, Funny

      Apparently the "wheelbarrow full of 1.44" floppies" was actually what their security was set up to prevent.

      And the joke was on them--he was actually stealing wheelbarrows!

    12. Re:Stunning. by Jane+Q.+Public · · Score: 4, Informative

      "Slightly more powerful than wget to me is a wrapper around wget. Perl and Bash scripts are way beyond the average users. To politicians scripts can be used to claim "voodoo" or "saintly" depending on who writes the scripts. The NSAs scripts are obviously saintly, while anybody else is probably voodoo."

      Even funnier is the assertion that such "web crawling" would be easy to detect. As someone who has done remote automation and data scraping for a living, I can tell you that it doesn't look any different than any other web traffic.

      About the only way to detect it is to do traffic analysis, to see if the same IP address is hitting nodes a lot, or hitting many nodes in a short period of time, and especially if they are rapid-fire.

      But the latter is easy to get around. I won't say just how here, because even if it's not hard to figure out it's still something of a trade secret.

    13. Re:Stunning. by Cyberax · · Score: 3, Funny

      Well, technically all Perl scripts are 'voodoo' by default.

    14. Re:Stunning. by cheesybagel · · Score: 4, Insightful

      Actually there are options in wget for that.

  2. Wow... by fuzzyfuzzyfungus · · Score: 5, Insightful

    You mean to tell me that an NSA tech contractor used wget or something, rather than loading up IE6 and clicking until his fingers fell off?

    Knock me over with a feather, spooks. You fucking hired people to build what is probably the largest collection of signals intelligence scraping systems on the planet, targeted at a wide variety of differently structured systems. Why would you even consider, except as a last resort, the notion that you are dealing with a bunch of noobs?

    (Oh, incidentally, maybe you should spend a bit less time reading everybody's email and work on that 'hilarious leaked diplomatic calls' problem, I'm told that sort of thing used to be your job at some point in the past...)

    1. Re:Wow... by fuzzyfuzzyfungus · · Score: 3, Insightful

      I suspect the posturing about 'zOMG, Snowden is clearly working for the commie russians and/or chinese taleban!!!!' to be the purest of bullshit; but if I were a member of the US clandestine services, I'd be shitting myself wondering about the existence of people who are working for somebody and running up against the same... impressive... security measures. If there are any actual moles, it is not looking good for what they were likely able to get their hands on.

    2. Re:Wow... by dcollins117 · · Score: 4, Insightful

      From TFA:

      Agency officials insist that if Mr. Snowden had been working from N.S.A. headquarters at Fort Meade, Md., which was equipped with monitors designed to detect when a huge volume of data was being accessed and downloaded, he almost certainly would have been caught. But because he worked at an agency outpost that had not yet been upgraded with modern security measures, his copying of what the agency's newly appointed No. 2 officer, Rick Ledgett, recently called "the keys to the kingdom" raised few alarms. "Some place had to be last" in getting the security upgrade, said one official familiar with Mr. Snowden's activities. But he added that Mr. Snowden's actions had been "challenged a few times".

      So they knew he was doing it, even questioned him, and he still got away with the data. To the people who maintain the NSA has the best and brightest security people perhaps they (NSA security) should use that expertise to improve their own security instead of weakening everyone else's.

      And yes, this is precisely why they must not be trusted with the data they are gathering due to mass surveillance.

  3. In Soviet Slashdot by Anonymous Coward · · Score: 5, Funny

    Beta scrapes you!

  4. ...and that makes it better? by Anonymous Coward · · Score: 4, Insightful

    If the network can't identify that something accessing the network sporadically and in repeated succession is a bot and should be stopped maybe the NSA shouldn't have access to this much data to start with....

    What if a legitimate foreign hacker was able to get in and do the exact same thing? Obviously, they have very shitty standards when it comes to network security - you'd expect thousands of honey pots, ability to intercept attempted attacks, flat out network filtering of these kinds of requests. But alas, that would make sense!

    1. Re: ...and that makes it better? by dk20 · · Score: 4, Insightful

      Seriously, they should have "accidented" the whole plane? So how many innocent people should have died to protect their poorly guarded secrets?

      "I'm all for whistleblower status, but Snowden should have been DEAD, HORRIBLY"
      You sure downt sound like you are for whistleblower. I dont many people forget that the NSA violated many US laws.

    2. Re: ...and that makes it better? by Somebody+Is+Using+My · · Score: 4, Insightful

      You know, whether you agree or disagree with what Snowden did, that in no way justifies killing him without a... oh, what was that quaint thing we used to require? That's right, a trial. Rule of law, and all that. I think that's what the country was based on originally.

      Of course, it's embarrassing for the NSA that Snowden waltzed out with so much confidential information, and arguably he should have been ARRESTED within 24 hours of "that flight to Hong Kong", but killed? To even think that sort of thing is disturbing.

      Having said that, I am glad he managed to get away, since his revelations are shining an absolutely necessary light on the murky behavior of our government and its actions. An educated populace is necessary to ensuring our freedoms and for too long the government has been hiding its wrong-doings from the ones it purports to serve. Whether Snowden acted as a foreign agent, or for his own advantage, or out of idealism, his actions were necessary and should not be so readily scorned.

      (oh right, and fuckbeta and all that jazz. It's gonna be hard doing that boycott tomorrow)

    3. Re: ...and that makes it better? by greenbird · · Score: 3, Insightful

      You know, whether you agree or disagree with what Snowden did, that in no way justifies killing him without a... oh, what was that quaint thing we used to require? That's right, a trial. Rule of law, and all that.

      If there were to be a trial it is almost certain they would exclude pretty much all avenues of defense that support what he actually did and why. Rule of Law is no more in this country. Just ask Aron Schwartz, Bradley Manning and the host of other whistle blowers prosecuted by the self proclaimed most open administration in history. If they want you gone they simple twist the millions of laws that exist and make up new interpretations if that's not enough. But you will be gone.

      --
      Who is John Galt?
  5. The trick by Anonymous Coward · · Score: 4, Interesting

    "because he worked at an agency outpost that had not yet been upgraded with modern security measures."

    "when he was questioned, Mr. Snowden provided what were later described to investigators as legitimate-sounding explanations for his activities"

    Speechless.

  6. Amused by Spad · · Score: 5, Funny

    "This automated and indiscriminate bulk collection of data is unacceptable!"

    1. Re:Amused by drinkypoo · · Score: 3, Informative

      Oddly, government is complaining that people will be able to take the various facts that he assembled and figure out what we're really up to. You know, the kind of thing they say they can't do with our metadata.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  7. How Many More NSA Employees? by retroworks · · Score: 5, Interesting

    Now the question is, how many other NSA contractors / staff / moles / spies have been doing the same thing, without Snowden's intention to disclose their behavior?

    --
    Gently reply
    1. Re:How Many More NSA Employees? by quenda · · Score: 5, Insightful

      Now the question is, how many other NSA contractors / staff / moles / spies have been doing the same thing, without Snowden's intention to disclose their behavior?

      I'm sure the NSA assumes they have moles, and none of the data Snowden released is a surprise to the Russians or Chinese.
      The NSA was just not prepared for the truth to leak to their real enemy - the general public.

    2. Re:How Many More NSA Employees? by Shaiken · · Score: 3, Insightful

      As others have said: "How so?"

      It's a logical conclusion based on the available evidence: No safeguards were in place to defend against an analyst stealing data and giving it to someone else, despite this being an obvious threat the NSA could not possibly have been unaware of.

      No such measures were taking until someone (i.e. Snowden) leaked this information to the public. Add this to the extremely negative way in which the NSA and the entire administration talks about journalists reporting on this, and the response to other whistleblowers and this really is the most likely explaination.

  8. Useless NYT article .. by DTentilhao · · Score: 5, Insightful

    "Agency officials insist that if Mr. Snowden had been working from N.S.A. headquarters at Fort Meade .. he almost certainly would have been caught. But because he worked at an agency outpost that had not yet been upgraded with modern security measures, his copying .. raised few alarms."

    This is retrospective ass-covering cyberbullshit. It is precisely at the edge that the security attacks would come from. What they were doing putting such material on Web servers and Wikis beggers credulity. Didn't senior management not realize that as keepers of the nations secrets they would be subject to attacks both internally and externally. Given the state of non-security at the NSA I would suspect that Snowden wasn't the only hostile with access to the “the keys to the kingdom”.

  9. Re:When you are a criminal by wonkey_monkey · · Score: 5, Insightful

    When you are anyone trying to do anything efficiently (such as the legally questionable automated gathering and storage of records of millions of phone calls and text messages?), you use proper tools.

    --
    systemd is Roko's Basilisk.
  10. In his defense by lxs · · Score: 4, Funny

    In his defense mr. Snowden explained that his scraper was only gathering metadata and therefore by their own standards the NSA has nothing to worry about.

    When asked for comment NSA director James Clapper replied : "Whoa that's deep bro!"

  11. A friendly reminder: by Vintermann · · Score: 4, Insightful

    There's absolutely zero reason to believe anything the NSA says about how Snowden got the documents, or indeed, about anything. They believe they are entitled to lie to congress, so the public isn't even a question.

    --
    xkcd is not in the sudoers file. This incident will be reported.
  12. Re:Middle Initial by wirefarm · · Score: 3, Funny

    Jerry: David Berkowitz, Ted Bundy, Richard Speck...
    Alice: What about them?
    Jerry: Serial killers. Serial killers only have two names. You ever notice that? But lone gunmen assassins, they always have three names. John Wilkes Booth, Lee Harvey Oswald, Mark David Chapman...
    Alice: John Hinckley. He shot Reagan. He only has two names.
    Jerry: Yeah, but he only just shot Reagan. Reagan didn't die. If Reagan had died, I'm pretty sure we probably would all know what John Hinckley's middle name was.

    --
    -- My Weblog.
  13. Re:When you are a criminal by Jah-Wren+Ryel · · Score: 4, Insightful

    Also, I don't care what think your motive is, you don't turn over classified documents to the enemy.

    And there it is ... the free press is nurb432's enemy.

    --
    When information is power, privacy is freedom.
  14. Snowden is a patriot; the NSA is treasonous by Tenebrousedge · · Score: 4, Insightful

    The idea of military specialists of whatever type being employed against the society they belong to, is treasonous and fucking retarded no matter what legal acrobatics are employed in their defense.

    You may have some sort of mystic devotion to the law, but I believe laws are made by (generally corrupt) men for their own interests, and I am familiar enough with the world outside the borders and political influence of the United States to know there is an enormous difference between legality and rightousness. The U.S.A. may not be the kind of country where you are expected to bribe every public official however minor -- we generally reserve that for higher office. It takes a special kind of idiocy to use military forces against their homeland, though.

    Government at its core is the body to which we have delegated our inherent right to violence -- a right being defined in this case as something which cannot be taken from you. We delegate this right to others, specialized in its use, with the express understanding that [a] as applied to civilian life, the exercise of violence by police will be applied fairly and equally as men can manage, and [b] that the unrestricted expression of this (as embodied by military force) be only employed against our enemies. War is hell, and we do not bring hell home.

    Snowden is a patriot, and the NSA is treasonous -- whether or not the law can be made to serve whichever purpose. Beyond all other argument, potentially felonious violation of the law is so common with the continual proliferation of laws that lawfulness cannot be the only measure of either justice or rightousness. May all those who support the NSA have a fair trial.

    --
    Those who advocate genocide deserve every protection afforded by law, and none afforded by common human decency.