Building a Bigger Search Engine

skreuzer writes "Wired is running a story about a distributed web crawler called Grub. People who choose to download and run the client will assist in building the Web's largest, most accurate database of URLs. This database will be used to improve existing search engines' results by increasing the frequency at which sites are crawled and indexed. Conceivably, Grub's distributed network could enable state information to be gathered on every document on the Internet, each and every day."

Comments

Will Grub take off or be smashed?

[Blaine Hilton]

I started to use grub, but then questions started cropping up. First we are using this to further a commercial organization. This is not research such as SETI or Folding At Home; this is doing the dirty work of a large commercial search engine. There is not even any potential reward such as with distributed.net.

Also the grub engine crawls everything, including adult content and other questionable content. They have a setting to turn it off, but it does not block it. With the current questioning of international law relating to accessing illegal websites this could have major consequences for the average user.

So for the time being I have stopped using the grub client until some serious questions are answered. It's an interesting concept and if it was being used in more of an academic setting it could be interesting. However I believe that search engines like Google are doing pretty good themselves.

Go calculate something

Re:Will Grub take off or be smashed?

[bcrowell]

This is not research such as SETI or Folding At Home; this is doing the dirty work of a large commercial search engine.
Actually, if I had a gun to my head, I'd choose to run Grub, because the client is open-source. I used to run SETI@home, but then the news came out that they'd been sitting on a potential root vulnerability for a long time. That really brought home to me the risks of running someone else's closed-source app on my box.

Re:Will Grub take off or be smashed?

[kaden]

Um, I think you're missing the point. This client could download highly illegal files, and make it look like I'm knowingly downloading them. Say I run it, and it downloads anything from kiddy porn to some Al Qaida webpage from an FBI sting server. I would quite possibly be arrested and charged, and while I wouldn't be convicted, it's quite an ordeal, and there's an ugly social stigma to even being charged with Kiddy Porn or conspiring with a terrorist. So that's a serious question that's posted by running Grub.

Re:Will Grub take off or be smashed?

[bcrowell]

Do you have any references? Please back up your claims.
here, and here

Actually I think the hole potentially gave the ability to run arbitrary code, which isn't the same as a root vulnerability.

Re:Will Grub take off or be smashed?

[dtfinch]

There are many ways to look at this. The idea is to install the client, set Opera to use the same useragent string, visit some of those sites, then blame it on Grub if the FBI comes busting through your door.

If you're a criminal, installing the Grub client might be a great idea.

Great idea, but will it pan out?

[dtolton]

LookSmart hopes to tap the altruistic nature of many Internet users.

That unfortunately seems like a naively optimistic hope. While the
vast majority of people may be altruistic, it only takes a few
unscrupulous individuals to completely undermine a fair result.

It's interesting that this idea is an extension to Google's model in
many ways. Essentially Google is able to index so much of the
interent by having 50,000+ servers. I don't think that's what makes
Google such a useful search tool, rather I think it's accuracy and
relevancy. If my search results started getting poluted with bogus
hits, I would stop using it almost immediately.

Unfortunately, by letting people run the client on their machine and
having it send the results back to the server, I think spoofed
results are inevitable. I don't think it will be possible to
safeguard the results either, it will be interesting to see how well
this project survives *when* people start spoofing results. It's
been a problem for SETI@home, and it's something that undermined some
peoples faith in the project as a whole. If the spoofed results are
more widespread and have a larger impact as they would in a system
like this, it may ultimately prove fatal to the project.

One factor that has been asbolutely critical to Google's success has
been their ability to remain resistant to spoofing attempts. It's
still a question mark how well grub will perform in that context.

Re:Great idea, but will it pan out?

[Nickilo]

"The General's Dilemma" would solve this problem. The story goes something like this: The general needs to get urgent information to one of his officers, however, he suspects saboteurs are present among his messengers. In order to insure the information gets through accurately, he sends the same message with several men. The officer on the other end collects all the messages and goes with the majority. (And, presumably, kills the others.)

Biiig questions to answer

[andy@petdance.com]

So Grub goes out, uses bandwidth, and then returns some results to the home base. It's really distributed bandwidth more than distributed computation.

I bet one of the big successes in Folding and distributed.net is that many people run the clients on work boxes, knowing that there's little actual overhead incurred to their work. How different that is for a URL sucker.

I wonder what broadband ISPs think of Grub.

Re:Biiig questions to answer

[friedegg]

I wonder what broadband ISPs think of Grub.

If it becomes a problem, I imagine ISPs will declare it a commercial bandwidth usage, and order users to stop or move to a business class plan for more money.

Haiku :-)

[Ignorant Aardvark]

Grub searches the web
Sniffing out all the good porn
Not just bootloader

I love being a Slashdot subscriber - it gives me fifteen minutes to figure out a good joke before anyone has a chance to post!

Seriously though, shouldn't they change the name? "GRUB" is already a bootloader. They should change the name ... and I have a suggestion. Has anyone written a program called "E-Coli" yet? No? I can just imagine my mom ...

"Agh! You have E-Coli on your computer!"

Re:Haiku :-)

[Anonymous Coward]

How about 'SARS'? Four letters, indicates something that spreads quickly...

Re:Haiku :-)

[Anonymous Coward]

Seriously though, shouldn't they change the name? "GRUB" is already a bootloader. They should change the name ...

I'm wondering if the Grub bootloader developers will throw a tantrum and flood the Grub crawler developers' e-mail addresses, claiming that this will confuse people and harm the bootloader project.

Hee hee.

Re:Haiku :-)

[Unoriginal Nick]

Seriously though, shouldn't they change the name? "GRUB" is already a bootloader. They should change the name ...

How about Firebird? I'm sure that won't cause any problems :-)

Re:Haiku :-)

[Chester K]

As time approaches infinity, the number of software projects named Firebird also approaches infinity.

It's ok though because they'll all still be different projects, so nobody will get confused.

Hrmm, I wonder how long...

[bergeron76]

until someone figures out a way to compromize their local client's results and "escalate" their fave URLS.

It still sounds like a really cool idea though.

Re:Hrmm, I wonder how long...

[CaptainMunchies]

Grub's clients don'tcome up with a ranking for each website they crawl; rather, they check to see if this website has changed since the last time it was crawled. For any website that has changed, the client notifies the server. The search engine asks the server which sites in its index need to be updated, and the server gleefully replies.

Clients artificially increasing their ranking isn't an issue, since the client has nothing to do with a site's ranking.

If previous results are any guide

[carl67lp]

1. Tech-savvy people will install this.
2. Tech-savvy people tend to be loners.
3. Loners most often search for porn.

C1. Tech-savvy people search for porn.

4. Items searched for most often reach the top of the list.
5. Porn is searched for often by tech-savvy people.

C2. Porn will be easier to find with this new search engine.

Count me in!

Re:If previous results are any guide

[anon*127.0.0.1]

You're having trouble finding porn now?

Grub

[squiggleslash]

Ok, so how are they going to store this giant search engine in the boot sector of an ordinary hard drive?

Oh wait, you mean it's not related to GRUB, the Linux/etc boot loader. *slaps forehead* But I guess this solves everything - we can call Phoenix "Grub" too, and just treat it as the generic name to call everything we're having problems thinking up a name for...

Firewalls?

[adam_megacz]

So if I choose to run this client, how do I know that it won't accidentally index content that is only accessible from behind my firewall?

Re:Firewalls?

[friedegg]

You can always put an entry in your robots.txt to block it.

Actually, the robots.txt issue is one they're still working on. Right now it doesn't check the file very often, which upsets some webmasters.

They're open to suggestions, so maybe you could suggest a list of blacklisted IP's/hostnames. I suggested they look into supporting gzip compressed web pages, and they said they'd look into it.

Google Toolbar

[petree]

Couldn't google do this anyways with the google toolbar? Cause with the advanced features version it tracks every page you visit. If they offered some incentive to install the toolbar, google could just beat them at this game. I actually use the google toolbar already by choice (it makes my web searching more productive) everyday, all they have to do is get lots of people using it and wouldn't that work just as well or better?

Re:Google Toolbar

[Kelerain]

This tracking is actually how a lot of important information leaks out. Security through obscurity has always been a poor mans system, and this busts it wide open. I wont post them here but there are several interesting searches you can do that give personal results for things that REALLY have NO place on a publicly accessable page. On a more positive note, google already uses distributed computing though thier googlebar http://toolbar.google.com/dc/offerdc.html However they donate the cycles to various worthy causes like folding at home (currently thier only benificiary), but it is concevable that if they came up with some secure and usefull search related thing to do with the cycles they could put it to use almost instantaniously. I think that there aren't segnificant benifits (plenty of discussion elsewhere here) for them to want to use it however.

Google's technology is superior...

[eidechse]

...those pigeons can't be beat.

What about the RIAA?

[One Louder]

So...let's say my instance of Grub crawls over a repository of .mp3s and supplies that information to the combined index.

What's the difference between my machine indexing them and the university students recently being hauled into court for indexing open shares? Why would I not be held liable for contributory copyright infringement?

No thanks.

Re:Not news for us webmasters

[Redwing]

Here is what slashdotters were saying about grub almost 2 years ago.

They realize they aren't the REAL GRUB

[anagama]

From the readme in the linux version - no idea what the other readmes might say. However, it appears that they are sensitve to the fact that bootloader grub pre-existed their program. They are requesting catchy names. Here is an excerpt:

Notice
======
The main executable has been renamed to "grubclient" out of respect for the GNU Grub bootloader, who's executable is named "grub". They were out first, so we decided to pick another name. If you have a catchy suggestion for a new name, please let us know.

A better use for my screensaver time

[Call Me Black Cloud]

I prefer grid.org to grub.org. There the cycles are going to cancer or smallpox research. Currently over 2 million machines are participating.

Altruism has its place, but since I'm more likely to die of cancer than of not having the complete www indexed I think I'll be selfish and work towards a cure for something that may affect me.

Indexor or Search Engine?

[digitect]

I expected some way to search... this looks more like a project to index the web rather than make the results available for public use via web interface. Did it strike anyone else odd that there was no web form on the home page with which to search?!

It seems like a good concept, but the availability of the information collected needs to be accessible without installing the client. I'm not game to install distributed computing apps without some freely available benefit. The "for the good of the world" motivation went out the window for me about a day after my first Seti At Home experience. (But now BitTorrent, there was appreciable benefit. I had RedHat 9 isos within 8 hours of their initial release!)

Re:search.msn.com is the future

[shibbydude]

In particular, the company has its own team of editors that monitors the most popular searches being performed and then hand-picks sites that are believed to be the most relevant.

You have to be kidding or working for Microsoft, or both! Have you ever searched for Linux on MSN? Try it - here.

Notice the third result? "Learn about the Microsoft alternatives and how to move to them from open source products." I shit you not! I don't think Google would ever use this kind of dirty, underhanded trick. Great "hand-picking", mate.

You can run both

[friedegg]

Grub isn't a heavy cpu users. Right now, on my Athlon (~2400+), it's using between 0-2% of the CPU at any given time. Grub is mainly interested in your excess bandwidth.

Re:You can run both

[rabidcow]

Grub is mainly interested in your excess bandwidth.

Unfortunately, so is my ISP. In fact, they've already sold it to other customers.

Looksmart

[Ark42]

Isn't Looksmart/Sprinks a big pay-per-listing deal? The looksmart logo in the upper right corner was enough to make me just close that page right away without any second thought.

Re:Search engine software and lack of A . I .

[zymano]

I didn't know that.

But it still kind of irks me that people think that a computerized 'dumb' search result could compete with a human rating system that filters spam,porn,and other garbage results. Google should hire some REAL PEOPLE that can do some sort catagorized intelligent directory so we can have QUALITY at the beginning of a search result. Some sort of HUMUN RATING system is needed to sort. The software is not up to par.

Altruistic?

[sulli]

That's the dumbest thing I've heard in ages. Why should I help out a for-profit company for free?

(Oh, I can't remember. Have I MetaModerated Recently?)

Re:Altruistic?

[eversunsoft]

Well, because web searching, to this day in age, has been a free service. Supposing that the index is built as the result of donated searches, it would be ethically in very bad taste to act against this trend.

Of course, I am the first one to question this trend. Has anyone else considered the possibility that one day we'll wake up, and notice that google is charging for access to it's basic searching services?

I for one, would probably pay. I have become so dependent on it. What price? That's a good question...

Re:Altruistic?

[R0]

Notice
======
The main executable has been renamed to "grubclient" out of respect for the GNU Grub bootloader, who's executable is named "grub". They were out first, so we decided to pick another name. If you have a catchy suggestion for a new name, please let us know.

I nominate "parasite".

Good Idea, Bad Implementation

[oaf357]

Yea. If you help Grub, Grub gives your web site a preferencial listing. Building the biggest search engine, sure. Building good search results, not so sure.

What _is_ a good project?

[bcrowell]

I have a FreeBSD server that wastes the vast majority of its CPU cycles (and most of its bandwidth, too). So what is a good distributed computing project to donate those cycles to? I'd like to find something that

1. makes me feel warm and fuzzy about my altruism
2. can run in the background on a Unix box
3. is open-source (so I don't have to run someone's closed-source app on my box and trust their security through obscurity)

Well, #1 rules out Grub, #2 rules out Folding@Home, and #3 rules out both SETI@Home and Folding@Home.

So what worthy causes are out there?

DDoS

[karlm]

So the idea is to DDoS the entire web? :-)

If this thing gets too popular without proper throttling, they could cause real havoc.

Legalities?

[cheshiremackat]

Alright, I have 3 major problems with this...

1) How different is this than the princton kiddies system? I don't know about you, but I don't want a 95 billion dollar bill arriving in the mail...

2) What if you local (cache?) contains a few links to kiddie porn? Not your fault, right? Software does it's own thing, you cannot control, BUT what will the FBI think? The FBI Scottland Yard, RCMP are currently heavily investigating Kiddie Porn cases (good work IMHO), but what if your the unlucky sap who getts stuck with a few sketchy URLs? Or Worse Yet, what if this GRUB keeps a cache of the website like google does? Then what?

3) What about material that is legal locally, but illegial somewhere else... eg. Nazi stuff in Germany, Falun Gong in China, etc... The last thing I want is to be refused to be given a travel visa cuz my PC has an illegial cache...

Good idea in principle, but with sketchy content on the web, I don't think I will be the one keeping track of it all. If there is a way to filter out the questionable stuff then maybe, but since the purpose is to be as inclusive as possible, it seems incompatible.

_CMK

Unlimited Use? Try Wishful Thinking.

[NeoMoose]

You can always use the Google API for more than 2,000 searches per day if you pay licensing fees for it. That's just Google ensuring that it can remain a viable company. Little text-box advertisements just don't cut it in this day and age where blatant pop-ups and colorful banner ads don't even have much turn-around. That's not the point though.

The point is that I wouldn't look anytime soon for LookSmart to allow unlimited usage of this API. It's too large of a project for them to just let people use it. It's simple economics. They may not be investing the computing resources into this projects web spidering software, but it's still using TONS of resources to keep this data catalogued and readily accessible.

The approach is inherently flawed

[oren]

It is too easy to send currupted information into the database. They have *no choice* but to trust the clients. Sure they could run spot checks on the results, but they would be very partial and it would be easy enough to fake responses for those as well.

So the more popular it gets, the more incentive people will have to promote their sites by feeding it fake index information. If this magically got to be very popular, within weeks search results would become meaningelss and it would drop back into obscurity. The more likely result would be that it will never become popular in the first place.

Besides, who wants to donate his CPU and bandwidth resources for a commercial company, anyway?

Re:Grub does NOT look for robots.txt

[Anonymous Coward]

Here it is on mine requesting it:

64.241.242.18 - - [18/Mar/2003:17:25:30 -0700] "GET /robots.txt HTTP/1.1" 200 222 "-" "Mozilla/4.0 (compatible; grub-client-1.07; Crawl your own stuff with http://grub.org)"
64.241.242.18 - - [19/Mar/2003:19:41:05 -0700] "GET /robots.txt HTTP/1.1" 200 222 "-" "Mozilla/4.0 (compatible; grub-client-1.07; Crawl your own stuff with http://grub.org)"
64.241.243.81 - - [30/Mar/2003:22:10:41 -0700] "GET /robots.txt HTTP/1.1" 200 222 "-" "Mozilla/4.0 (compatible; grub-client-1.07; Crawl your own stuff with http://grub.org)"
64.241.243.81 - - [01/Apr/2003:23:11:21 -0700] "GET /robots.txt HTTP/1.1" 200 223 "-" "Mozilla/4.0 (compatible; grub-client-1.07; Crawl your own stuff with http://grub.org)"

Notice those are LookSmart owned ip's and not just normal user crawlers. They seem to centrally crawl for robots.txt. They do know, however, that they need to crawl for robots.txt more often.