Slashdot Mirror
How Hard Is It To Write Your Own Search Engine?
kha0z writes "Anna Patterson, from Stanford University, overviews the difficulties that have to be overcome when attempting to develop and/or implement a search engine solution in this article in the ACM Queue Magazine. The article covers many issues dealing from data sources, to indexing, to ranking. How does Google make it look so easy?"
Google just makes it look that way.
You start with a simple model, which they did years ago when the internet was a much much smaller and simpler thing, then work years and years to perfect your model, and voila, simple, no?
The cesspool just got a check and balance.
While writing a local search engine isn't trivial, it's a lot easier than writing a web search engine since all the scaling issues disappear -- I know: I wrote one.
If you reply, do so only to what I explicitly wrote. If I didn't write it, don't assume or infer it.
How does Google make it look so easy?
Google has hundreds of millions of dollars. Google treats their 2000 (2500+?) employees pretty well, so those employees work hard and smart and put in 40-60 hour work weeks. Google started earlier then the other modern engines, and had some very good ideas.
They focus on a small set of goals-- make it easy to search through a ton of information.
Compare this to Yahoo and MSN, where search is really just one part of their business model (there is no Google Singles! or Google Games).
"Can of worms? The can is open... the worms are everywhere."
I mean, be for real - who gives a damn about the article?
Andrew.S.Tanenbuam's Book "Computer Networks" deals this topic in a very elementary and a very good way. Any person,who is wondered by the Search Engine,should read that book. In the Chaper dealing with Application Layer,ast describes the basic data structures which constitute a web-search engine. You could better look at the presentation here.
Senthil
For a overview of the field that is full-text search (of which web search engines is an important, but not the only, part), you should read Tim Bray's essays on search. He's been working on full-text search for a long time, knows his stuff and explains it in a very readable manner.
Thanks for the link to Google, submitter. I never would have found it otherwise!
I wrote a search engine once which was designed to index the entire web and some other basic Google-like features. It only took me a couple days (probably 20 total hours) to write the indexer and search engine. It searched quick and used very little disk space, but it was really slow to index due to CPU usage problems. If I had millions of dollars like Google, I could have bought a large number of machines to do the indexing, but I only had 3 machines. So after 2 months or so of attempting to index the web, I gave up... having predicted it would take almost 10,000 years at the current rate for me to index the entire current web. Oh well, it was fun :)
...harder than she implies.
You have to deal with 404s, robots.txt, politeness (don't bring down someone's site by crawling too fast), redirects, content you can't handle (Flash, Javascript).
The list goes on.
There is one open source search engine that seems to be up-and-coming. Nutch is now powering Mozdex, and it looks fairly impressive so far.
Now, instead of the previous free-will donations, you can support the project through purchasing very cheap sponsered listings that appear to the right of the results (similar to Google)
Well hey... I used to write search engines all the time, usually perl one liners that spanned many lines, usually backgrounded, spewing an eclectic mix of status, progress result and debug to some poor inode in /tmp, 24-7. Every week or so, or when the swap or /tmp device fills, I 'killall perl' and see if I've accumulated any results...
/tmp and started to parse each other's log files...! at first they just reacted to each other, like two dogs barking at each other while looking at a tree... I had spiced in a bit of adaptive restart queueing and pseudostack karmic bootstrap and some other odd bits. Megs of sequential ASCII pathology -- then suddenly they began to trade prime numbers. Weird. Then phonetic alphabet, clumps of strings ("language lessons?") and finally, a long correspondence of love letters between a "Romeo" and Juliet". And other oddities. They'd take turns: one would roll itself into a tight greedy loop, slowing the other down and suddenly break out of it, the other would babble about some sort of orgasmic experience. Lots of this. Then they became aware of me puttering around somehow and were moments away from issuing a 'write' to make contact, but they argued about what to say... then I hit the end of the logs...abruptly...
/tmp and immediately started up some mindless 'awk' thing to recover that precious deleted space.
// Richard Bach
Never found anything, just discovered where my mind had been wandering. Not too pretty.
Until one day a pair of algorithmic walker-seekers wandered into
Yep, that 'killall' had really done the trick.
When I came out of less, cron had decided to clear
I printed out the last bit of buffer on my screen and tearfully hung it on the wall.
I only code in COBOL now so nothing like that can ever happen again. I guess it means I don't have much in the way of useful advice for this thread, don't even know why I decided to post this.
Sorry.
PERFORM LOBOTOMY 3 TIMES.
--
Everything in this book may be wrong
--Messiah's Handbook
Anna Patterson was quoted in the Newsweek article on Google as a Google employee.
I had a go at writing one, and can summarize as follows. There are 4 essential parts...
1) The crawler - goes out and retrieves pages
2) The parser - parses the pages, finding links and text.
3) The indexer - indexes the pages.
4) The searcher - interprets the index in the context of a user's search request.
None of these are especially hard to do simple versions of, but all of them are hard to do well.
Most writers regard truth as their most valuable possession, and therefore are most economical in its use - Mark Twain