Slashdot Mirror

← Back to Stories (view on slashdot.org)

WebCrawler Turns 10 Today

Posted by ryuzaki0 on from the young-at-art dept.

Brian Pinkerton writes "WebCrawler, one of the first search engines on the 'Net, turns 10 today. You can read a short history of WebCrawler. When I wrote WebCrawler, one could do a credible job of crawling, indexing, and searching the Web from a single desktop PC. Today, the reality is a little bit different."

4 of 136 comments (clear)

  1. Re:e-mailing results by Brianwa · · Score: 5, Informative

    You can be emailed results from Google as well.
    Simply email google@capeclear.com with the search terms in the subject line, you will soon recieve a response with the results. I think there is a limit to how many times a day you can use this, but I cannot find the link to the project webpage.

  2. Re:I remember using Webcrawler before google... by qodfathr · · Score: 5, Informative

    You are remembering raging.com, still up-and-running today.

    --
    Yes, it's true. This man has no dick.
  3. Hardly one of the first by btempleton · · Score: 4, Informative

    Internet searching way predates 1994. Archie by Peter Deutsch (the one from Montreal, not the American one) was one of the most popular applications on the internet in the 80s. The http search engines like Webcrawler and Lycos came much, much later on internet time scales.

    --
    Has it been over a year since you last donated to the Electronic Frontier Foundation
  4. nope by millette · · Score: 4, Informative
    You just need 8 desktop machines and you can index a 10th of what google does. From a recent article:
    Gigablast runs on eight desktop machines, each with four 160-GB IDE hard drives, two gigs of RAM, and one 2.6-GHz Intel processor. It can hold up to 320 million Web pages (on 5 TB), handle about 40 queries per second and spider about eight million pages per day. Currently it serves half a million queries per day to various clients, including some meta search engines and some pay-per-click engines.
    I also read it was going to expand it's index this year, but I wasn't able to find where I read that.