How Hard Is It To Write Your Own Search Engine?

kha0z writes "Anna Patterson, from Stanford University, overviews the difficulties that have to be overcome when attempting to develop and/or implement a search engine solution in this article in the ACM Queue Magazine. The article covers many issues dealing from data sources, to indexing, to ranking. How does Google make it look so easy?"

Comments

Search engine != entire web

[pauljlucas]

Not all search engines are designed or intended for indexing and searching the entire web, and not everybody needs such a search engine. Often, people want to search their stuff: their documents on their local disks, their e-mail, etc.

While writing a local search engine isn't trivial, it's a lot easier than writing a web search engine since all the scaling issues disappear -- I know: I wrote one.

Re:Search engine != entire web

[BigGerman]

exactly. A buddy of mine developed and now sells search appliance that can search network windows shares as well as web servers - EnterFind.
it is not complicated at all.

How does Google make it look so easy?

[stefanlasiewski]

How does Google make it look so easy?

Google has hundreds of millions of dollars. Google treats their 2000 (2500+?) employees pretty well, so those employees work hard and smart and put in 40-60 hour work weeks. Google started earlier then the other modern engines, and had some very good ideas.

They focus on a small set of goals-- make it easy to search through a ton of information.

Compare this to Yahoo and MSN, where search is really just one part of their business model (there is no Google Singles! or Google Games).

Re:How does Google make it look so easy?

[Gilk180]

They focus on a small set of goals-- make it easy to search through a ton of information.

Originally this was the case, but recently even Google has started forays into other services. Just click the "more>>" link above the search field on their front page.

Froogle(shopping), groups(news groups), blogger, and of course there is the even popular /. subject, gmail.

Unfortunately it seems they may be losing focus.

Re:How does Google make it look so easy?

[stefanlasiewski]

Well, most of the additional services still focus on searching and information. Information is also one of their core goals (I should have said this earlier). There still is nothing like Yahoo Travel or MSNBC.

Froogle is still a search product, but with a focus on shopping.

Groups is mostly still a search product (You can post also, so it's also about creating information). The service has been around for years (I think it's their second big project after web search). If I have a technical question, I often find the answers in Google groups. Blogger is new, but is similar to Groups in it's goals.

Gmail is also largely about search. With search they can place ads in your email.

Actually, I guess you can really say that Google is about using a good search technology to place highly targeted ads with the information.

Re:How does Google make it look so easy?

[armando_wall]

Groups is mostly still a search product (You can post also, so it's also about creating information). The service has been around for years (I think it's their second big project after web search).

Actually, Google Groups used to be DejaNews, and they bought the technology.

Re:How does Google make it look so easy?

[merlin_jim]

those employees work hard and smart and put in 40-60 hour work weeks

This is an excellent opportunity to point out that 8 hours of those every week, mandatory, must be spent on a personal project not related to google's line of business.

That one perk is the absolute best.

Picture here.

[mosel-saar-ruwer]

http://www-formal.stanford.edu/annap/www/annap-pic .gif

I mean, be for real - who gives a damn about the article?

Re:Picture here.

[HyperCash]

Anybody else notice the HARVARD binder in the picture?

--HC

Have you Read ast's Computer Networks Book?

[Phoe6]

Andrew.S.Tanenbuam's Book "Computer Networks" deals this topic in a very elementary and a very good way. Any person,who is wondered by the Search Engine,should read that book. In the Chaper dealing with Application Layer,ast describes the basic data structures which constitute a web-search engine. You could better look at the presentation here.

Ask Tim Bray

[grungeKid]

For a overview of the field that is full-text search (of which web search engines is an important, but not the only, part), you should read Tim Bray's essays on search. He's been working on full-text search for a long time, knows his stuff and explains it in a very readable manner.

Thanks for the link

[christopherfinke]

Thanks for the link to Google, submitter. I never would have found it otherwise!

Re:Thanks for the link

[Anonymous Coward]

Here's the FreeCache version in case of slashdotting.

not too hard

[TyrelHaveman]

I wrote a search engine once which was designed to index the entire web and some other basic Google-like features. It only took me a couple days (probably 20 total hours) to write the indexer and search engine. It searched quick and used very little disk space, but it was really slow to index due to CPU usage problems. If I had millions of dollars like Google, I could have bought a large number of machines to do the indexing, but I only had 3 machines. So after 2 months or so of attempting to index the web, I gave up... having predicted it would take almost 10,000 years at the current rate for me to index the entire current web. Oh well, it was fun :)

The crawl is hard, too

[blamanj]

...harder than she implies.

You have to deal with 404s, robots.txt, politeness (don't bring down someone's site by crawling too fast), redirects, content you can't handle (Flash, Javascript).

The list goes on.

Re:The crawl is hard, too

[merlin_jim]

politeness (don't bring down someone's site by crawling too fast), redirects, content you can't handle (Flash, Javascript).

I wrote my own crawler once. A just-for-fun-how-do-the-spammers-do-it kind of thing...

I got around redirects and content by going straight via the network socket and looking at the response as pure text. Anything that fit the regex pattern for an email address got harvested.

As far as politeness... I kept a circular growable queue (technically a linked list) of sites to visit. Each queue entry is current site - links deep - sites deep (I would ordinarily start with a google search and do 3-4 sites deep, 5-6 links deep)... if the crawl engine saw a link with the same server it had just visited, it would randomly change its link index. This kept it from hammering on a particular server too hard...

Nutch

[idiotfromia]

There is one open source search engine that seems to be up-and-coming. Nutch is now powering Mozdex, and it looks fairly impressive so far.

Now, instead of the previous free-will donations, you can support the project through purchasing very cheap sponsered listings that appear to the right of the results (similar to Google)

Author @ Google?

[sgauss]

Anna Patterson was quoted in the Newsweek article on Google as a Google employee.

4 parts to a search engine

[DukeyToo]

I had a go at writing one, and can summarize as follows. There are 4 essential parts...

1) The crawler - goes out and retrieves pages
2) The parser - parses the pages, finding links and text.
3) The indexer - indexes the pages.
4) The searcher - interprets the index in the context of a user's search request.

None of these are especially hard to do simple versions of, but all of them are hard to do well.