Slashdot Mirror
WebCrawler Turns 10 Today
Brian Pinkerton writes "WebCrawler, one of the first search engines on the 'Net, turns 10 today. You can read a short history of WebCrawler. When I wrote WebCrawler, one could do a credible job of crawling, indexing, and searching the Web from a single desktop PC. Today, the reality is a little bit different."
Ah the nostalgia of receiving search results via e-mail :)
(\_/)
(O.o) This is Bunny. (> <)
I remember when webcrawler was the only search engine I touched...
In 1996 it was nice and simple. Then as the time went on it got a bit too cluttered for my liking. Now looks like they're trying to googlize themselves with the current interface.
Holy crap!
... but do they have any relevance anymore? They're owned by InfoSpace. :P
I remember WebCrawler, but lost touch with it in around 1996, when I started religiously using AltaVista. They sure have changed a bit.
- oZ
// i am here.
Does anybody else remember getting a WebCrawler promotional CD 10 years ago? I didn't even have a CD-ROM then!
Steal This Sig
Was it 2001? The History states:
Oh, and if it is not being otherwise used, has the code for the WebCrawler spider been open-sourced?
Only Women Bleed (Sex, Sharia remix)
...to the days when the search engine market resembled the microcomputer market of the '80s. Several competitors, all with (roughly) the same market share, each with a certain number of hits that the others didn't have. I had to use at least a few of them to assure myself that I was getting something reasonably close to what the whole Web could offer on my search topic (even though no search engine comes close to penetrating all of the pages out there).
If I was looking for something, I'd query Lycos, AltaVistas, Infoseek, Excite, Webcrawler, and Magellan. And, later on, Google. Vastly different results, site designs, site objectives. I won't say it was the most streamlined, elegant experience, but it was kind of fun.
The coolest voice ever.
I remember using Webcrawler back when I got my first 14.4 Slirp connection back in 1994. It was the only way to search!
and then came the marvels of altavista.digital.com.
I'm so glad that google came along...
Do not look into laser with remaining eye.
http://web.archive.org/web/19961023234707/http://w ww.webcrawler.com/
:)
Presumably connects to the current crawler which still accepts the old format
--
Callas
I think I remember why I left WebCrawler.
WebCrawler was simple and effective. But then AltaVista emerged. I started using AltaVista.digital.com, and from there WebCrawler went down hill - lots of advertising and junk that kind of made me hate it. What was once seemless and simple became noisy.
I used AltaVista for a number of years, but once again advertising got the best of it. It turned super-sophisticated, with a lot of advertising fluff and "features". Altavista was becoming overly commercialized. They had a "simple" version that was better (I forget the name [begins with an "R"?]), but soon the result sets were scewed towards advertisers and abusers.
In 2001, I made the switch to Google. It was everything that WebCrawler once was in terms of ease of use and quality of results. I've been more or less happy with Google ever since.
It is scary to think that at one point I e-mailed the WebCrawler people to ask them how it worked. In response they sent me a copy of the source (Objective C for NeXT) so I could compile it up on my NeXT PC (I had a "black" NeXT - 68000 based) to index my intranet web server.
I doubt that someone like Google would send you a copy of their source these days - even if you asked nicely.
I could never get it to compile, and I deleted it long ago, but I kind of wish I had kept it now. An interesting piece of internet history.
I'd happily contribute cash to a publicly funded and publicly run search engine.
Anyone game ?
MP3 Search Engine
WebCraweler's Brian Pinkerton formerly worked at NeXT, and I remember being in the the NeXT kitchen when news arrived in 1995 of his sale of WebCrawler to AOL. The sale price was around $1 million, and everyone was absolutely awed that a software company could sell for so much. This marked the beginning for me of the dot-com era: Just a few month later, other companies started or run by ex-NeXTers sold for millions, then tens of millions, and at least one for hundreds of million. Soon after that, NeXT CEO Jobs took Pixar through an IPO, for a personal gain of about $1 billion!