WebCrawler Turns 10 Today

Brian Pinkerton writes "WebCrawler, one of the first search engines on the 'Net, turns 10 today. You can read a short history of WebCrawler. When I wrote WebCrawler, one could do a credible job of crawling, indexing, and searching the Web from a single desktop PC. Today, the reality is a little bit different."

Comments

I remember using Webcrawler before google...

[John Seminal]

It was a good search engine. I dunno why I stopped using it, I think it was a bit on the slow side and Google had more pages.

Heck, while reminiscing, I remember when excite was my start page, and when I used them for email. I remember they were the first "start" page to have groups. I stopped using them 4 years ago when their email stopped working.

I guess if anything, we can learn the web is not going to be the same in 5 years as it is today. My question is, "is it better"? Personally, I think it was better back in the day. I would like to see a search engine that does not display any spam or sales or sex sites as hits. I now do most of my searches on google doing "search parameters site:edu".

DNS failure (power outage)

[Anonymous Coward]

Try http://209.24.201.206/bp/WebCrawler/History.html for the history.

Re:Well isn't that ironic

[Adam9]

No worries, just go here

Re:WebCrawler on NeXTStep - before Open Source

[ArbitraryConstant]

I doubt that someone like Google would send you a copy of their source these days - even if you asked nicely.

The next best thing.

search appliance

Re:Then and now...

[/dev/trash]

In all fairness if the majority of the websites today that are Slashdotted would not use a mySQL dynamic solution to serve pages, they'd be okay.

The one before WebCrawler?

[SnappingTurtle]

I seem to remember that before WebCrawler there was actually a "big" search engine run by a non-profit. For the life of me I can't remember what it was, but I seem to remember one day going "Wow, this webcrawler thing is great, I'm never touching [whatever] again."

Of course a few years later I said "Wow, this AltaVista thing is great. I'm never touching WebCrawler again." And then I went "Wow, this Google thing is great. I'm never touching AltaVista again."