Slashdot Mirror

← Back to Stories (view on slashdot.org)

How Hard Is It To Write Your Own Search Engine?

Posted by timothy on from the can-barely-search-my-car dept.

kha0z writes "Anna Patterson, from Stanford University, overviews the difficulties that have to be overcome when attempting to develop and/or implement a search engine solution in this article in the ACM Queue Magazine. The article covers many issues dealing from data sources, to indexing, to ranking. How does Google make it look so easy?"

9 of 23 comments (clear)

  1. Search engine != entire web by pauljlucas · · Score: 4, Insightful
    Not all search engines are designed or intended for indexing and searching the entire web, and not everybody needs such a search engine. Often, people want to search their stuff: their documents on their local disks, their e-mail, etc.

    While writing a local search engine isn't trivial, it's a lot easier than writing a web search engine since all the scaling issues disappear -- I know: I wrote one.

    --
    If you reply, do so only to what I explicitly wrote. If I didn't write it, don't assume or infer it.
  2. How does Google make it look so easy? by stefanlasiewski · · Score: 3, Interesting

    How does Google make it look so easy?

    Google has hundreds of millions of dollars. Google treats their 2000 (2500+?) employees pretty well, so those employees work hard and smart and put in 40-60 hour work weeks. Google started earlier then the other modern engines, and had some very good ideas.

    They focus on a small set of goals-- make it easy to search through a ton of information.

    Compare this to Yahoo and MSN, where search is really just one part of their business model (there is no Google Singles! or Google Games).

    --
    "Can of worms? The can is open... the worms are everywhere."
    1. Re:How does Google make it look so easy? by stefanlasiewski · · Score: 3, Insightful

      Well, most of the additional services still focus on searching and information. Information is also one of their core goals (I should have said this earlier). There still is nothing like Yahoo Travel or MSNBC.

      Froogle is still a search product, but with a focus on shopping.

      Groups is mostly still a search product (You can post also, so it's also about creating information). The service has been around for years (I think it's their second big project after web search). If I have a technical question, I often find the answers in Google groups. Blogger is new, but is similar to Groups in it's goals.

      Gmail is also largely about search. With search they can place ads in your email.

      Actually, I guess you can really say that Google is about using a good search technology to place highly targeted ads with the information.

      --
      "Can of worms? The can is open... the worms are everywhere."
  3. Picture here. by mosel-saar-ruwer · · Score: 2, Offtopic

    http://www-formal.stanford.edu/annap/www/annap-pic .gif
    I mean, be for real - who gives a damn about the article?

  4. Have you Read ast's Computer Networks Book? by Phoe6 · · Score: 2, Interesting

    Andrew.S.Tanenbuam's Book "Computer Networks" deals this topic in a very elementary and a very good way. Any person,who is wondered by the Search Engine,should read that book. In the Chaper dealing with Application Layer,ast describes the basic data structures which constitute a web-search engine. You could better look at the presentation here.

    --
    Senthil
  5. Ask Tim Bray by grungeKid · · Score: 3, Informative

    For a overview of the field that is full-text search (of which web search engines is an important, but not the only, part), you should read Tim Bray's essays on search. He's been working on full-text search for a long time, knows his stuff and explains it in a very readable manner.

  6. Thanks for the link by christopherfinke · · Score: 2, Funny

    Thanks for the link to Google, submitter. I never would have found it otherwise!

    1. Re:Thanks for the link by Anonymous Coward · · Score: 2, Funny

      Here's the FreeCache version in case of slashdotting.

  7. The crawl is hard, too by blamanj · · Score: 4, Insightful

    ...harder than she implies.

    You have to deal with 404s, robots.txt, politeness (don't bring down someone's site by crawling too fast), redirects, content you can't handle (Flash, Javascript).

    The list goes on.