Slashdot Mirror


'Google Search On Steroids' Brings Dark Web To Light

snydeq writes The government agency that brought us the Internet has now developed a powerful new search engine that is shedding light on the contents of the so-called deep Web. DARPA began work on the Memex Deep Web Search Engine a year ago, and this week unveiled its tools to Scientific American and 60 Minutes. "Memex, which is being developed by 17 different contractor teams, aims to build a better map of Internet content and uncover patterns in online data that could help law enforcement officers and others. While early trials have focused on mapping the movements of human traffickers, the technology could one day be applied to investigative efforts such as counterterrorism, missing persons, disease response, and disaster relief."

69 comments

  1. WebQL... by michael_rendier · · Score: 1

    couldn't this be done since like 2001 or so?

    --
    There are three kinds of people in the world. Those that can count, and those that can't.
    1. Re:WebQL... by Anonymous Coward · · Score: 0

      But now it's being done ...with a computer.

      Or maybe ...on the internet.

  2. 17 different contractor teams by Anonymous Coward · · Score: 4, Insightful

    "... being developed by 17 different contractor teams..."
    There's a recipe for failure if even I saw one!

    1. Re:17 different contractor teams by phantomfive · · Score: 1

      "... being developed by 17 different contractor teams..."
      There's a recipe for failure if even I saw one!

      That's exactly what I came here to say. My job is now done (by you).

      --
      "First they came for the slanderers and i said nothing."
    2. Re:17 different contractor teams by Sir_Substance · · Score: 1

      Maybe they made 17 different versions and picked the best one? vOv

    3. Re:17 different contractor teams by Anonymous Coward · · Score: 1

      That's exactly what I came here to say. My job is now done (by you).

      Outsourcing strikes again!

    4. Re:17 different contractor teams by phantomfive · · Score: 5, Funny

      Hey, looks like I'm ready to be promoted to manager.

      --
      "First they came for the slanderers and i said nothing."
    5. Re:17 different contractor teams by leehb · · Score: 1

      Wow! Software with at least seventeen different back-doors! That's gotta be some kind of record!

    6. Re:17 different contractor teams by dj245 · · Score: 1

      "... being developed by 17 different contractor teams..." There's a recipe for failure if even I saw one!

      Is that so bad for a search engine? If I had a bunch of people and needed to search a downtown neighborhood, I would break into teams and search different buildings all at the same time. Get the results of the search and organize according to relevence. Searching different networks is not much different. You could have a team of Tor specialists working on Tor, a team working on freenet, etc. Plus a team working on a common framework including a plugin or API system.

      --
      Even those who arrange and design shrubberies are under considerable economic stress at this period in history.
    7. Re:17 different contractor teams by Anonymous Coward · · Score: 0

      Can you say pork barrell?

    8. Re:17 different contractor teams by Anonymous Coward · · Score: 0

      Its a government job so efficiency probably isnt a primary goal.

    9. Re:17 different contractor teams by kcitren · · Score: 3, Informative

      It's 17 teams working on 17 different things. The way these projects work is they've got a bunch of contractors that have ideas they want to pursue that are relevant to DARPAs interests and requirements. DARPA funds them and they all work on their own things and try to create some synergy. Over the life of the contract they continue funding those projects that are most successful.

    10. Re:17 different contractor teams by kcitren · · Score: 1

      They're all working of different things related to the topic. It's not that they're building a consolidated system, they working on technologies related to the problem at hand.

    11. Re:17 different contractor teams by Anonymous Coward · · Score: 0

      17 different back doors.

  3. I'm sure this will bring us a better world by Anonymous Coward · · Score: 0

    What could possibly go wrong?

    1. Re:I'm sure this will bring us a better world by Anonymous Coward · · Score: 0

      ... and that captcha I got was "WATCHING" haha

  4. Magical Program that can cure cancer & warts! by Nyder · · Score: 1

    Sounds like this search engine runs on magic dust and could not only find the cure for cancer, but finally get rid of those facial warts that always come back! If you give us XXXX Billions of Dollars, it will keep you safe in your sleep from the bogey man, terrorists and spam!

    Ya, sounds like bullshit to me.

    --
    Be seeing you...
  5. search on steroids by rossdee · · Score: 3, Funny

    Whike I am sure that steroid abuse is assisted by :the dark web' , there are more dangerous drugs for sale there, not to mention actual violent crime they should crack down on

    1. Re:search on steroids by complete+loony · · Score: 2

      I don't see how searching for steroids is going to reveal the deep web...

      --
      09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
    2. Re:search on steroids by Kichigai+Mentat · · Score: 1

      not to mention actual violent crime they should crack down on

      Like murder for hire? And child exploitation rings? You do know entities involved in those things use the Dark Web to contact each other and organize, right? I mean, Dread Pirate Roberts used it to organize his hits.

      --
      Rawr
  6. go Xerox yourself by Anonymous Coward · · Score: 1

    Absolutely nothing about Google. But it's a search! Search is Google! That's why it's Google! Duh huh huh huhuh huhuhuh!

  7. The upshot is by Cafe+Alpha · · Score: 5, Interesting

    before, criminals could keep from being caught by having a robots.txt file.

    The sad thing is this isn't a joke

    1. Re:The upshot is by danknight48 · · Score: 1

      http://www.robotstxt.org/robot...

      There are two important considerations when using /robots.txt:

      robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
      the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.
      So don't try to use /robots.txt to hide information.

    2. Re:The upshot is by Anonymous Coward · · Score: 0

      There are a bunch of web crawlers that ignore robots.txt, or so the logs and statistics on my site tell me.

    3. Re: The upshot is by Anonymous Coward · · Score: 0

      Ignore it? If I'm looking for holes in a web site, that's one of the first things I check.

    4. Re:The upshot is by Uberbah · · Score: 1

      The sad thing is this isn't a joke

      It is for your friendly neighborhood fusion center. Just ask Ross Ulbricht.

    5. Re:The upshot is by Anonymous Coward · · Score: 0

      I'm sure they're already building a blacklist of IP/DNS entries and software that starts throwing dummy pages if their crawlers exceed a reasonable human-esque speed; if someone is going through 80 pages a minute, they're probably mapping the site.

  8. "...could one day be applied to...." by turkeydance · · Score: 0

    ANYthing we want to! Woo!Woo!

    1. Re: "...could one day be applied to...." by Anonymous Coward · · Score: 0

      "Now Officer James, sit right down and completely relax." "We'll be injecting the MeMeX 3000 deep in your ass." "Most importantly, tell us when it hurts, or if you start to feel good." "Remember, this is so we can save the world from the terrorists lurking around every corner." "You are one of the special few and very lucky at that."
      Dismissed

  9. Nice Memex :^) by Anonymous Coward · · Score: 0

    (^:

  10. Exactly. That was my takeaway as well. by tlambert · · Score: 4, Funny

    Exactly. That was my take-away as well.

    (1) Get a huge government contract

    (2) Ignore robots.txt

    (3) Profit!

  11. Digital Scarecrow by Guy+From+V · · Score: 5, Insightful

    I thought that this sounded ominous for a minute. Then I remembered that government projects like this are designed to have a chilling effect on activity that they cannot monitor, understand or enforce by their very existence and not by being actual potent tools to combat it (i.e. paper tiger). More likely this thing will become a money pit that contractors can use as a sandbox project to allow their employees to play in for implementation of IP that may be works-in-progress for future projects that may be useful, but are just lofty concepts that have no basis in reality. 17 contracting teams is about 15-16 too many hands in the cookie jar for this to be anything more than a Men In Black-wannabe training camp or a glorified propaganda project, most likely both.

    1. Re:Digital Scarecrow by CaptainDork · · Score: 1

      This.

      And ...

      How many tools does the government have that kids circumvent every day?

      This sounds a lot like the war on spam.

      --
      It little behooves the best of us to comment on the rest of us.
    2. Re:Digital Scarecrow by Anonymous Coward · · Score: 1

      How many tools does the government have that kids circumvent every day?

      Having lived through being a curious kid: all of them.

  12. Dark web, the necessary existence of ... by Anonymous Coward · · Score: 0

    If Darpa's latest venture is successful, dark net will be under the spotlight of Uncle Sam ... it is not only those who are using the dark net right now who are losing, we too lose

    Look, I am not condoning all the illegal / violent activity that might be related to those who use the dark net but as one that does not trust the authority very much I do recognize a crucial role that dark net plays --- to enable us a way to conceal something sensitive away from the prying eyes of the big brother

    Some of you may say "So what?" that Snowden gets to expose Uncle Sam's dirty linen without the use of the dark net

    Well, Snowden gets to do what he did because TPTB as it is, is split into several camps, and that right now Russia is giving protection to Snowden simply because the Russians do not like Uncle Sam too much

    But what if one day all the world's TPTB becomes one? Under that scenario no one can expose big brother's secret no more - no matter how important those secrets are to the citizens

    That is why I am worried --- with the lose of the dark net we are losing yet another venue to seek out (and expose) the truth

    1. Re:Dark web, the necessary existence of ... by Anonymous Coward · · Score: 0

      Snowden gets to expose Uncle Sam's dirty linen

      Uncle Sam refers to a meat packer who supplied the US military during World War I. The term does not apply to the President of the United States of Amerika no matter how many times the POTUS is called Uncle Sam. In the current context the POTUS is more akin to Uncle Tom than Uncle Sam.

    2. Re: Dark web, the necessary existence of ... by Anonymous Coward · · Score: 0

      Uncle Sam is some kind of personification / imaginary analogous representation of the US government. Nobody ever said he was ever president, you stupid turd burger.

    3. Re:Dark web, the necessary existence of ... by Anonymous Coward · · Score: 0

      Dude, that was a lot of work for a lame Uncle Tom joke.

    4. Re:Dark web, the necessary existence of ... by Anonymous Coward · · Score: 0

      No worries capcha understands.

    5. Re:Dark web, the necessary existence of ... by ganjadude · · Score: 1

      if this dark net gets a light shone on it, another darknet will pop up in its place.

      --
      have you seen my sig? there are many others like it but none that are the same
    6. Re:Dark web, the necessary existence of ... by Anonymous Coward · · Score: 0

      Those sort of tasks become harder as time goes by. al1 heil emp3r0r tom!

  13. Google search on steroids by wonkey_monkey · · Score: 0

    Argghhh!! Show me KITTENS!!!!111!!

    --
    systemd is Roko's Basilisk.
  14. Re: Magical Program that can cure cancer & war by Anonymous Coward · · Score: 0

    Umm, duct tape kills warts.

  15. Re:Magical Program that can cure cancer & wart by Anonymous Coward · · Score: 0

    Political attacks - If they don't find anything they'll make it up. Just look at their attack on an officer in Ferguson - 200 agents assigned to prosecute someone who was obviously defending themselves. It's all for PR and control of the masses.

  16. "Dark Web" by Arancaytar · · Score: 1

    Which is it, Deep Web or Darknet?

    Excellent reporting there.

    1. Re:"Dark Web" by Dominare · · Score: 1

      Whichever one will get us more funding, obviously.

    2. Re:"Dark Web" by jeffmeden · · Score: 1

      Which is it, Deep Web or Darknet?

      Excellent reporting there.

      TFA explains that it's both:

      Memex searches content typically ignored by commercial search engines, such as unstructured data, unlinked content, temporary pages that are removed before commercial search engines can crawl them, and chat forums[...]
      Memex also automates the mechanism of crawling the dark, or anonymous, Web where criminals conduct business. These hidden services pages, accessible only through the TOR anonymizing browser, typically operate under the radar of law enforcement selling illicit drugs and other contraband.

  17. Re: Magical Program that can cure cancer & war by Dominare · · Score: 1

    So does a shotgun. That's why you need to be specific about your use cases.

  18. "Deep" Web or "Dark" Web? by Rogue+Haggis+Landing · · Score: 3, Informative

    My understanding is that these are two different (though related) things. The Deep Web is simply the part of the Web that's not indexed by the major search engines. It might be purposefully hidden, or it might simply be a web page so out of the way that Google hasn't noticed it. The Dark Web is a subset of the Deep Web that is more purposefully hidden because people using it don't want The Man to know what's going on. Sometimes the Dark Web is defined as only places in which nefarious (or at least illegal) things go on, sometimes it's any place that's intentionally hidden, for whatever reason.

    Point is that the headline says "Dark Web" while the excerpt says "Deep Web", but then immediately starts talking about law enforcement, which means Dark Web.

    "Deep Web" and "Dark Web" are both useful concepts. We should avoid conflating them.

    1. Re:"Deep" Web or "Dark" Web? by jeffmeden · · Score: 2

      You are right, the "deep web" is not the same thing as the/a "darknet" or "dark web". They don't do a good job of keeping that clear in the headline. From TFA's own citation on wikipedia:

      "The deep web should not be confused with the dark Internet, computers that can no longer be reached via the Internet.

      However the article does assert that this Memex project is indexing both unpublicized content on the general internet (the deep part) plus anonymized content on Tor and other privacy services (the dark part).

    2. Re:"Deep" Web or "Dark" Web? by fustakrakich · · Score: 1

      "Deep Web" and "Dark Web" are both useful concepts. We should avoid conflating them.

      They also don't exist. People should stop believing they do. It all travels over the same wire. Each new 'encryption' protocol works exactly once, at best. The most functional you might find is Craigslist

      --
      “He’s not deformed, he’s just drunk!”
  19. I tried by Anonymous Coward · · Score: 0

    to search for steroids and all I got was sites about information on steroids.

  20. Disappointed by GuB-42 · · Score: 1

    No link to the search page... In fact it seems that there isn't a search page at all.
    The only thing Memex has in common with Google is the tracking.

  21. Other uses by Anonymous Coward · · Score: 0

    It could also be useful in outing malicious script kiddies and serial trolls, and thus deterring future nuisances of that sort.

  22. Please don't abuse disaster relief/response! by Opportunist · · Score: 0

    Look, we've been trained to treat anything you do for counter/anti/whatever-terrorism as an intrusion to our privacy and as a general way to screw us over and make money for your buddies. We've accepted that. And we've learned that it's bad for us and that we can't do jack about it, but at least we can ignore it.

    Now you start lumping disaster relief and disaster response into it. And that's where I draw the line. We need that, ok? That's something important, not like your war on pedophiles, war on terrorism, war on liberties or war on common sense. You can have that all and we'll play along because, well, as long as it keeps you entertained, we might have a little bit of peace from you. That's fine. Do not now lump disaster relief into the game because that's something you shouldn't play with. If we learned anything from the Hurricanes striking the Deep South in disturbing regularity then that this is were real people are in real danger and need real help.

    In short, that shit's way too important and too fragile to be left in a government's inapt hands.

    So please, go back to your war on terror and drugs and all the other little games you enjoy playing with your friends, but stay out of the grownup stuff, ok?

    --
    We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
  23. Or in other words... by Anonymous Coward · · Score: 0

    Built to further enforce IP interests of our corporate masters.

    Has nothing to do with finding missing children, preventing child exploitation, or human trafficking, as those things are actually helpful to the general population.

    Oh yeah, good ol' tax payers paid for this as well, but will never be able to use it for searching.

  24. "Memex" already has a famous meaning... by tgeller · · Score: 1

    ...among computer technologists, going back to 1945. Why reuse it for this? https://en.wikipedia.org/wiki/...

    --
    Tom Geller
  25. What does this have to do with Google? by Tony+Isaac · · Score: 1

    Other than that this is also a search engine, that is.

  26. The conspiracist in me says... by Anonymous Coward · · Score: 0

    "Memex, which is being developed by 17 different contractor teams, "(so only a select few get the complete source code) "aims to build a better map of Internet content and uncover patterns in online data that could help law enforcement officers and others"(,like political operatives, govt. employees who have a grudge or obsession.)" While early trials have focused on mapping the movements of human traffickers, the technology could one day be applied to investigative efforts such as counterterrorism, missing persons, disease response, and disaster relief"( and, well, anybody who accesses the internet.)

    FTFY

  27. Deep Web -ne Dark Web by Anonymous Coward · · Score: 0

    They aren't the same. Conflating the two dilutes the meaning of both, and shows that you don't understand the difference.

    Thanks, Slashdot-lite.

  28. So why do we need a root zone? by rs79 · · Score: 1

    If they're this clever at finding things then let them do TLD discovery and we can dispense with that trillion dollar ICANN nonsense that doesn't do anything.

    --
    Need Mercedes parts ?
  29. The key is babel by Anonymous Coward · · Score: 0

    11 And the whole earth is of one pronunciation, and of the same words,

    2 and it cometh to pass, in their journeying from the east, that they find a valley in the land of Shinar, and dwell there;

    3 and they say each one to his neighbour, `Give help, let us make bricks, and burn [them] thoroughly:' and the brick is to them for stone, and the bitumen hath been to them for mortar.

    4 And they say, `Give help, let us build for ourselves a city and tower, and its head in the heavens, and make for ourselves a name, lest we be scattered over the face of all the earth.'

    5 And Jehovah cometh down to see the city and the tower which the sons of men have builded;

    6 and Jehovah saith, `Lo, the people [is] one, and one pronunciation [is] to them all, and this it hath dreamed of doing; and now, nothing is restrained from them of that which they have purposed to do.

    7 Give help, let us go down, and mingle there their pronunciation, so that a man doth not understand the pronunciation of his companion.'

    8 And Jehovah doth scatter them from thence over the face of all the earth, and they cease to build the city;

    9 therefore hath [one] called its name Babel, for there hath Jehovah mingled the pronunciation of all the earth, and from thence hath Jehovah scattered them over the face of all the earth.

  30. mod_doorknock? by thogard · · Score: 1

    Some people have been using port knocking to allow remote admin yet cut down on the ssh bots trying to login.

    It would be trivial to do the same in a cgi where if your ip address is 1.2.232.121 you have to hit /target/232 then /target/121 to get the real data.

  31. Counterterrorism or disaster response by Anonymous Coward · · Score: 0

    It's basically the same thing.

    Fucking fearmongers.

  32. Temporary ads? by Anonymous Coward · · Score: 0

    SciAm:

    Evidence of criminals peddling such services online is hard to pinpoint because of the use of temporary ads and peer-to-peer connections within the deep Web.

    Because it's easier to track people if you serve them a permanent ad?

  33. If this is a memex by Anonymous Coward · · Score: 0

    Vannevar is Jeb's brother.