Slashdot Mirror

← Back to Stories (view on slashdot.org)

WebCrawler Turns 10 Today

Posted by ryuzaki0 on from the young-at-art dept.

Brian Pinkerton writes "WebCrawler, one of the first search engines on the 'Net, turns 10 today. You can read a short history of WebCrawler. When I wrote WebCrawler, one could do a credible job of crawling, indexing, and searching the Web from a single desktop PC. Today, the reality is a little bit different."

18 of 136 comments (clear)

  1. Guess this celebration... by oberondarksoul · · Score: 5, Funny

    ...won't have an accompanying Google Doodle?

    --
    And tomorrow the stock exchange will be the human race
  2. They used to be my google.... by wo1verin3 · · Score: 5, Interesting

    I remember when webcrawler was the only search engine I touched...

    In 1996 it was nice and simple. Then as the time went on it got a bit too cluttered for my liking. Now looks like they're trying to googlize themselves with the current interface.

  3. Whoa! by outZider · · Score: 5, Interesting

    Holy crap!

    I remember WebCrawler, but lost touch with it in around 1996, when I started religiously using AltaVista. They sure have changed a bit. ... but do they have any relevance anymore? They're owned by InfoSpace. :P

    --
    - oZ
    // i am here.
  4. Wow by z0ink · · Score: 5, Interesting

    Does anybody else remember getting a WebCrawler promotional CD 10 years ago? I didn't even have a CD-ROM then!

    --
    Steal This Sig
  5. I remember using Webcrawler before google... by John+Seminal · · Score: 5, Insightful
    It was a good search engine. I dunno why I stopped using it, I think it was a bit on the slow side and Google had more pages.

    Heck, while reminiscing, I remember when excite was my start page, and when I used them for email. I remember they were the first "start" page to have groups. I stopped using them 4 years ago when their email stopped working.

    I guess if anything, we can learn the web is not going to be the same in 5 years as it is today. My question is, "is it better"? Personally, I think it was better back in the day. I would like to see a search engine that does not display any spam or sales or sex sites as hits. I now do most of my searches on google doing "search parameters site:edu".

    --

    Rosco: "If brains were gunpowder, Enos couldn't blow his nose."

    1. Re:I remember using Webcrawler before google... by Anonymous Coward · · Score: 5, Interesting

      I think I remember why I left WebCrawler.

      WebCrawler was simple and effective. But then AltaVista emerged. I started using AltaVista.digital.com, and from there WebCrawler went down hill - lots of advertising and junk that kind of made me hate it. What was once seemless and simple became noisy.

      I used AltaVista for a number of years, but once again advertising got the best of it. It turned super-sophisticated, with a lot of advertising fluff and "features". Altavista was becoming overly commercialized. They had a "simple" version that was better (I forget the name [begins with an "R"?]), but soon the result sets were scewed towards advertisers and abusers.

      In 2001, I made the switch to Google. It was everything that WebCrawler once was in terms of ease of use and quality of results. I've been more or less happy with Google ever since.

    2. Re:I remember using Webcrawler before google... by qodfathr · · Score: 5, Informative

      You are remembering raging.com, still up-and-running today.

      --
      Yes, it's true. This man has no dick.
  6. Then and now... by jdreed1024 · · Score: 5, Funny
    When I wrote WebCrawler, one could do a credible job of crawling, indexing, and searching the Web from a single desktop PC. Today, the reality is a little bit different.

    No kidding. Back then, one could serve a website from most any machine, and it would be there for all to see. Today only the largest websites can avoid a slashdotting with only 9 posts in the thread.

    --
    There is no sig, there is only Zuul.
    1. Re:Then and now... by mph · · Score: 5, Funny
      Today only the largest websites can avoid a slashdotting with only 9 posts in the thread.
      Imagine how bad it would be if everyone actually read the articles.
  7. Birthday party by jacobhoupt · · Score: 5, Funny

    I'll be hosting my tenth annual WebCrawler birthday party tonight in the back of my Yugo.

    Feel free to drop in, there should be plenty of seating available for those interested.

    --
    -- the only good thing the French ever did was two chicks at one time
  8. When did they give up.... by David+Hume · · Score: 5, Interesting

    ...on their own web search technology and become a metasearch engine? From the WebCrawler About Page:

    WebCrawler uses innovative metasearch technology to search the Internet's top search engines, including Google, Yahoo, Ask Jeeves, About, Teoma, FindWhat, LookSmart, and many more.

    With one single click, WebCrawler searches the best results from the combined pool of the world's leading search engines -- instead of results from only one single search engine.

    And WebCrawler makes it easy to refine your search so you can find the most meaningful results right away. No wonder it's a leader in the search industry.


    Was it 2001? The History states:

    2001 InfoSpace acquires WebCrawler. Excite, now Excite@Home, went belly up. In the bankruptcy, Infospace acquired WebCrawler. Today Infospace runs WebCrawler as a meta-search engine. And they've given Spidey a new name and turned him purple!


    Oh, and if it is not being otherwise used, has the code for the WebCrawler spider been open-sourced? :)

    --
    Only Women Bleed (Sex, Sharia remix)
  9. Wow. Just. Wow. by TWX · · Score: 5, Interesting

    I remember using Webcrawler back when I got my first 14.4 Slirp connection back in 1994. It was the only way to search!

    and then came the marvels of altavista.digital.com.

    I'm so glad that google came along...

    --
    Do not look into laser with remaining eye.
  10. Wow - the 1996 wayback WebCrawler page STILL WORKS by Anonymous Coward · · Score: 5, Interesting

    http://web.archive.org/web/19961023234707/http://w ww.webcrawler.com/

    Presumably connects to the current crawler which still accepts the old format :)

    --
    Callas

  11. Well isn't that ironic by Zygote-IC- · · Score: 5, Funny

    So, to read a story celebrating an anniversary about a search engine, we have to go through the cache of another search engine?

    Go figure.

  12. WebCrawler on NeXTStep - before Open Source by ben_kelley · · Score: 5, Interesting

    It is scary to think that at one point I e-mailed the WebCrawler people to ask them how it worked. In response they sent me a copy of the source (Objective C for NeXT) so I could compile it up on my NeXT PC (I had a "black" NeXT - 68000 based) to index my intranet web server.

    I doubt that someone like Google would send you a copy of their source these days - even if you asked nicely.

    I could never get it to compile, and I deleted it long ago, but I kind of wish I had kept it now. An interesting piece of internet history.

    1. Re:WebCrawler on NeXTStep - before Open Source by sacremon · · Score: 5, Interesting

      I sometimes sit back and think about some of the various projects that first saw life on the NeXT platform:

      the first web server
      Webcrawler
      Doom and DoomII

      Pretty good for a machine that only sold ~70,000 units total, not including the versions of NEXTSTEP for ix86/SPARC/PA-RISC.

      I still have a Color NeXTStation stashed away in a closet. I was using it as a print server till about two years ago.

      --
      If you can't beat them, embrace and extend them.
  13. The WebCrawler Search Voyeur by Faust7 · · Score: 5, Funny

    Anyone remember the WebCrawler Search Voyeur?

    It was a little Java applet that sat on your screen and displayed the pseudo-real-time search queries of other people.

    When I was a computer lab monitor at my college, we used to note in the log book any particularly amusing queries that we'd seen.

    "hairy woman"... "squirrel torture"... "tom AND cruise AND foot AND odor"... "asian girl underage spanking"...

    --
    The coolest voice ever.
  14. Re:e-mailing results by Brianwa · · Score: 5, Informative

    You can be emailed results from Google as well.
    Simply email google@capeclear.com with the search terms in the subject line, you will soon recieve a response with the results. I think there is a limit to how many times a day you can use this, but I cannot find the link to the project webpage.