Slashdot Mirror
WebCrawler Turns 10 Today
Brian Pinkerton writes "WebCrawler, one of the first search engines on the 'Net, turns 10 today. You can read a short history of WebCrawler. When I wrote WebCrawler, one could do a credible job of crawling, indexing, and searching the Web from a single desktop PC. Today, the reality is a little bit different."
...won't have an accompanying Google Doodle?
And tomorrow the stock exchange will be the human race
Ah the nostalgia of receiving search results via e-mail :)
(\_/)
(O.o) This is Bunny. (> <)
Happy birthday to Webcrawler AND Hitler! Hurray!
I remember when webcrawler was the only search engine I touched...
In 1996 it was nice and simple. Then as the time went on it got a bit too cluttered for my liking. Now looks like they're trying to googlize themselves with the current interface.
Holy crap!
... but do they have any relevance anymore? They're owned by InfoSpace. :P
I remember WebCrawler, but lost touch with it in around 1996, when I started religiously using AltaVista. They sure have changed a bit.
- oZ
// i am here.
Does anybody else remember getting a WebCrawler promotional CD 10 years ago? I didn't even have a CD-ROM then!
Steal This Sig
Heck, while reminiscing, I remember when excite was my start page, and when I used them for email. I remember they were the first "start" page to have groups. I stopped using them 4 years ago when their email stopped working.
I guess if anything, we can learn the web is not going to be the same in 5 years as it is today. My question is, "is it better"? Personally, I think it was better back in the day. I would like to see a search engine that does not display any spam or sales or sex sites as hits. I now do most of my searches on google doing "search parameters site:edu".
Rosco: "If brains were gunpowder, Enos couldn't blow his nose."
I can't believe it is even still around.
No kidding. Back then, one could serve a website from most any machine, and it would be there for all to see. Today only the largest websites can avoid a slashdotting with only 9 posts in the thread.
There is no sig, there is only Zuul.
Who uses webcrawler anymore? I didn't even know they still exist. Anybody remember opentext.com search?
Meh.
I'll be hosting my tenth annual WebCrawler birthday party tonight in the back of my Yugo.
Feel free to drop in, there should be plenty of seating available for those interested.
-- the only good thing the French ever did was two chicks at one time
Some guys are too cool for their own good. Brian Pinkerton has the domain 'thinkpink.com', AND he wrote his own search engine.
I bet he even has a 3-digit UID, a beowulf cluster of Xboxes running linux, and he sold all his stock options during the bubble.
"If you think you have things under control, you're not going fast enough." --Mario Andretti
Here is the google cache: www.thinkpink.com/bp/WebCrawler/History.html+&hl=e n&ie=UTF-8
http://216.239.39.104/search?q=cache:-vPR77Hq9OYJ
ah yes, i remember the old days, search engines like webcrawler, altavista, magellan, and infoseek. in those days wouldn't help you find what you want most of the time. now with google we need not worry :)
Marge, get me your address book, 4 beers, and my conversation hat.
Was it 2001? The History states:
Oh, and if it is not being otherwise used, has the code for the WebCrawler spider been open-sourced?
Only Women Bleed (Sex, Sharia remix)
...to the days when the search engine market resembled the microcomputer market of the '80s. Several competitors, all with (roughly) the same market share, each with a certain number of hits that the others didn't have. I had to use at least a few of them to assure myself that I was getting something reasonably close to what the whole Web could offer on my search topic (even though no search engine comes close to penetrating all of the pages out there).
If I was looking for something, I'd query Lycos, AltaVistas, Infoseek, Excite, Webcrawler, and Magellan. And, later on, Google. Vastly different results, site designs, site objectives. I won't say it was the most streamlined, elegant experience, but it was kind of fun.
The coolest voice ever.
I used webcrawler all the way up until the day I discovered google...it was a really good search engine, way better than yahoo/altavista at the time. I had forgotten all about it until this article...very surprised to see that it's still alive.
Oh yeah, and Yahoo as well. Forgot to include them.
Interestingly, their look has changed very, very little from their olden days.
The coolest voice ever.
I remember using Webcrawler back when I got my first 14.4 Slirp connection back in 1994. It was the only way to search!
and then came the marvels of altavista.digital.com.
I'm so glad that google came along...
Do not look into laser with remaining eye.
WebCrawler was the best search engine, back in the day when search engines were all new and Yahoo wasn't a search engine but a human-moderated list of sites.
--
Callas
http://web.archive.org/web/19961023234707/http://w ww.webcrawler.com/
:)
Presumably connects to the current crawler which still accepts the old format
--
Callas
Can't we celebrate their birthday too? :)
Or their death day... :devilsh_smiley:
1996 WebCrawler
I have NO idea how that space got in there...
--
Callas
Anyone else experience this? I click on the History of Web Crawler link (www.thinkpink.com) and my browser crashes - Mozilla FireFox, Netscape 7.1 and IE6...
So who remembers the first search query they typed into Webcrawler?
I was just crawling out of the gopher world, a short period where I was getting turned on to the web but there was no way to find links, almost everything came through the university homepage or word of mouth. Then someone pointed me to webcrawler.
What did I search for first? "fart jokes". No kidding.
"boobs" was second.
So, to read a story celebrating an anniversary about a search engine, we have to go through the cache of another search engine?
Go figure.
Engineer: ARGGHHH... IT'S ALIIIIIVE... <BANG> <CRACK>
<STATIC>
Simpsons #AABF06 Homer: Let's see, what's Marge's birthday? Barney is April twentieth, same as Hitler's, so Marge must be fifty ...oh, forget it. Flanders, what's your birthday?
(source)
It is scary to think that at one point I e-mailed the WebCrawler people to ask them how it worked. In response they sent me a copy of the source (Objective C for NeXT) so I could compile it up on my NeXT PC (I had a "black" NeXT - 68000 based) to index my intranet web server.
I doubt that someone like Google would send you a copy of their source these days - even if you asked nicely.
I could never get it to compile, and I deleted it long ago, but I kind of wish I had kept it now. An interesting piece of internet history.
Try http://209.24.201.206/bp/WebCrawler/History.html for the history.
n/t
I've reminisced before on slashdot about the beautiful geeky girl that introduced me to hotbot. Glasses, long blond hair, full breasts... cute sandals, short skirts... those silk panties.
Fuck WebCrawler. hotbot.
Anyone remember the WebCrawler Search Voyeur?
It was a little Java applet that sat on your screen and displayed the pseudo-real-time search queries of other people.
When I was a computer lab monitor at my college, we used to note in the log book any particularly amusing queries that we'd seen.
"hairy woman"... "squirrel torture"... "tom AND cruise AND foot AND odor"... "asian girl underage spanking"...
The coolest voice ever.
I'll never forget reading this entry in the monitor log book:
"Search Voyeur query of the day:
'Why does poop stink?'"
The coolest voice ever.
It was useful because it was one of the only options, not because it was good.. it returned way too much crap, many times the matches never contained even one of the keywords
I just did a search on webcrawler for "digital camera" and the results where 70% pay-per-click advertising with a small and hardly noticable "Sponsored by:" disclaimer. Worse yet the paid links are intermigled with the indexed hits.
Looks like Webcrawler is now more of a pay-per-click dispensor than a search engine... No thanks!
I think google has done a good job of clearly identified what is relevant and what is paid for.
WebCrawler is nothing more than a Gopher Ripoff. We knew it then, and we know it now.
I'd happily contribute cash to a publicly funded and publicly run search engine.
Anyone game ?
MP3 Search Engine
Hot from Google
Results 1 - 10 of about 7,660,000 for boobs [definition]. (0.14 seconds)
The exact day that I stopped using webcrawler. It happened to coincide with the day that AOL was announced as the new owner.
And the odd part is I don't even remember the interface being as cluttered as the very early one linked through the archive in an earlier post. I suppose I moved on very early, although I remember when as far as I was concerned they were the only game in town.
Wouldn't that have been hilarious? Every single search engine that it used would have had almost exactly the same market share!
Look again
I remember using WebCrawler on my very first SLIP dial up account and thinking "How cool is this?" I had used AOL for a couple years prior but was hoping trade in their UI (and limitations) for Netscape. The funny thing is that I wasn't sure if I could find enough content on the web.
Also a great testament to the original design and concept that search engines still look and work a lot like WebCrawler, 10 years on.
Happy birthday, and thanks for the walk down 32K memory lane
Someday a Slashdot ID of 177180 will mean something.
were...
webcrawler, mid 90's
altavista, late 90's,
google, even later 90's up to now
Meh.
But after AOL bought it I lost track of it, because it started sucking (returning lots more stale links than before), and altavista.digital.com burst upon the scene (anyone else remember "kayak sailing San Juan islands"?).
My guess would be that the meta-search switch initially happened when Excite bought them.
-- Old Man Kensey
Not your fault. Slashcode does that itself whenever there's a long enough unbroken string of characters, to stop page-widening posts.
I seem to remember that the WebCrawler site used to have a "Powered By NEXTSTEP" badge on it. I can't verify this with web.archive.org as it doesn't seem to go back that far (I started using WebCrawler in 1995). I can't RTFA at the moment, does anyone know what sort of hardware powered the WebCrawler site originally? Did it run on black NeXT hardware? White box PCs running NeXTSTEP? Did they ever utilize the WebObjects framework that NeXT (and later, Apple) used to sell?
Internet searching way predates 1994. Archie by Peter Deutsch (the one from Montreal, not the American one) was one of the most popular applications on the internet in the 80s. The http search engines like Webcrawler and Lycos came much, much later on internet time scales.
Has it been over a year since you last donated to the Electronic Frontier Foundation
Of course a few years later I said "Wow, this AltaVista thing is great. I'm never touching WebCrawler again." And then I went "Wow, this Google thing is great. I'm never touching AltaVista again."
I've found that my posts don't format quite right w/o a sig.
I think WebCrawler was my first search engine ever...
From there I graduated to MetaCrawler, which parsed WebCrawler and all the other currently popular web search engines at the time.
For some reason or another MetaCrawler started sucking and I used InfoSeek for quite some time... then they were acquired by Go.com and it went downhill from there.
I remember what I'd search the internet for back in those days tho. It was always "jedi knight" and "giga pets" (remember those cute tamagotchi rip-offs? =p)
I used to be one of the Excite@Home engineers who looked after Webcrawler. WebCrawler and the Excite front end all belonged to the same code base called My Excite Start Page (known internally as MESP at Excite).
The WebCrawler at Excite was pretty much an unsupported product when I was there. All I ever did were maintenance releases, never any new stuff for WebCrawler. WebCrawler was actually the Excite front end, except it had the WebCrawler logos instead of Excite.
The search engine was the Excite search engine as well as all links on the front page pointed to Excite media properties.
On a personal note, I was always somewhat saddened to see a piece of Internet history neglected the way it was but by December 2001, Webcrawler received very little traffic (I forget the numbers).
I remember when Yahoo was a stanford.edu site
WebCraweler's Brian Pinkerton formerly worked at NeXT, and I remember being in the the NeXT kitchen when news arrived in 1995 of his sale of WebCrawler to AOL. The sale price was around $1 million, and everyone was absolutely awed that a software company could sell for so much. This marked the beginning for me of the dot-com era: Just a few month later, other companies started or run by ex-NeXTers sold for millions, then tens of millions, and at least one for hundreds of million. Soon after that, NeXT CEO Jobs took Pixar through an IPO, for a personal gain of about $1 billion!
People, don't be stupid, don't send your emails to people you don't know.
I remember back in 1994 WebCrawler was running on three machines in the corner of Sieg Hall 433. They were rigged up so one could reboot the others via a serial line, but occassionally that machine would crash too. That was when Brian would call in and say "Hey, Webcrawler is hung. Could you go reboot it?". I'm guessing this doesn't happen much at Google...
Presumably connects to the current crawler which still accepts the old format :)
Ya, but I followed the link to get my free copy of Internet Explorer that is advertised on the page and got
Whatever that means.
I like one of Webcrawler's featured searches today: Camel Spiders.
Those things may have urban legends surrounding them or whatever, but they are GODDAMN SCARY!!!
Yup...
> Any tips?
:-)
Put your rage in it and then throw. Oh, and flick your wrist. Oh, and take the packaging off first!
138
- The number of people who have replied to this thread.
- The number of people who still use webcrawler
Anyone remember this one? I think it is older. IIRC it was not indexed. It collected links and you did a search on the page for what you wanted.
Q