Slashdot Mirror
Understanding Search Engines?
An anonymous reader asks: "I guess by now we can be fairly certain that search engines are here to stay, and hence I'm trying to understand how the technology works. I'm not so much looking for a particular 'best' technology or implementation, but rather an overview of the different approaches and their trade-offs. Something that would teach me: which approach works in a distributed vs a centralized infrastructure; how different algorithms will perform on complete search words vs arbitrary sub-strings; or how mass storage (hard disk vs. solid state) affects implementation choices. For most mature technologies there is a host of 'overview' books and papers for my questions -- but I couldn't find anything on search engines. Where should I look? Are there any good books or papers?"
Same basic concepts apply today ... although they probably didn't anticipate the rise of Black Hat SEO which attempts to "beat" the algorithms.
Hulk SMASH Celiac Disease
SIAM Review had a survey article on different methods recently. You need to know linear algebra, combinatorics and probability
One of the founders of Google still has links to various publications (in PostScript format) about search engines, if that helps.
Take a class on information retrieval from your local university.
I've had this sig for three days.
Try the web. I have a short intro to search engines on my website. Many others exist. The basics aren't hard and are very effective.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
Managing Gigbytes author site Amazon
is a spectacular book on most of the underlying technologies. Although I've only read the first edition, I don't recall it talking about spidering/webcrawling. Instead it starts with building a simple index, and builds through all the refinements (ie stemming, etc) until you've built a serious workhorse for mining text documents. Its definitely at the core of what a search engine does,
Look up voting methods, with keywords like Kemeny, Condorcet, and Borda. A lot of search engine algorithms are like vote aggregation methods, where each site "votes" for other sites it has links to. There is quite a bit of stuff on spam page filtering and the like as well.
This tagline is copyrighted material. Please send $10 for an affordable replacement.
Comment removed based on user account deletion
Write in the language of the users you expect to use your site, and look at the server logs to see what terms people were using when they found your site.
If you sell poultry medication, for example, there's no point in only labelling products as being for cockerels if your visitors are more likely to use rooster as a keyword. You might also want to refrain from putting "cock pills" in your meta tags...
Other than that, write semantically valid code (header tags, etc) and don't put large blocks of navigation links first in a document -- you want search engines to concentrate more on the unique content of a page.
All common sense stuff, really.
Ph-nglui mglw'nafh Gates M'dna wgah'nagl fhtagn.
From the link I just clicked on I saw:
Ask Slashdot: Understanding Search Engines? 8 of 7 comments
It took a second glance to notice the subtle error in the wording.
Bug in slash?
liqbase
http://www.iprcom.com/papers/pagerank/
If the question was "I need to create a search engine for work, where should I start?" your answer would have been good advice. But the poster made no reference at all to creating his/her own search engine. Specifically: "...hence I'm trying to understand how the technology works." Even if the purpose is to create a search engine, your answer is still useless. Reinventing the wheel is a great way to learn.
Or perhaps you can explain how buying a search "solution" would teach him how a search engine works?
Maybe not
Well hell, guess everybody can go home. Nothing more to search for here, it is all figured out.
-- Crutcher --
#include <disclaimer.h>
That Google summary is useful, but is actually just a simplified version of their true ways.
So, aside from reading books on Information Retrieval and Data Mining, the other easily available reference are open source search engines. In particular, look at the Nutch project, which is actually a pretty high quality search engine implementation. Even better: start contributing to the project.
sigs are a waste of space
Although meta tags are largely thought to be a waste of time these days (from a search engine p.o.v.) you might still consider using "description". This is used by 'Google' and probably others to replace the snipet of your page with an actual description.
..." for every game it covers even when it's clearly not 'the most complete source', which just becomes annoying.
It depends on what you want to achive tho. GameSpy always uses "GameSpy is the most complete source for [insert game here] trailers, screenshots, cheats, walkthroughs, release dates, previews, reviews,
If you choose relevent descriptions that are are representative of your page they can be useful, although sometimes actually seeing the search word in the context that it appears in the page is better. See what your search results look like at the moment and if you don't like the summary you see in Google think about using the "description" tag.
If you really want to go behind the theory, you will want to start here. But be prepared to have some really good skills in math, statistics, computer sciences and system administration to understand the articles as they are not intended for general public.
:)
A brief intro of how classical search engines work goes as follows:
Grabbing: A crawler visits pages which it considers important, downloads them and parses them
Analysing: The document receives an identification string and is stored in a reversed index, which is simply a database table with culomns such as "word", "document", "possition", "importance". The "word" culomn is indexed and used for searching.
Searching: Say that you search for the phrase "ask slashdot". The search engine searches the lines with the terms "ask" and "slashdot", looks into the "document" cell and selects only those documents that both terms occure in. Then it looks into the "possition" cell which carries all the possitions of the searched word in each document and discards all the documents that do not have successive "ask" and "slashdot" terms possitions. The resulting documents are then sorted according to the importance cells of the searched terms.
This is how basically all search engine works. The only major difference is usually only in the math used to compute the imprtance. There are also some major optimisations done to speed up the responses. To discuss this would take too long. So if you have any questions feel free to ask. Currently I am part of a team developing a large scale search engine, so you have a chance to get some hot info here
If you need a search tool, look around for a solution that someone else has already wasted years of their life on rather than have yourself do the same. Why recode, when you can download?
I have never in my professional life run into a nontrivial production business application which was perfect. They all have bugs. They all need more work. So it doesn't matter whether he downloads an open source system, or inherits his employer's legacy system -- he will need to learn the principles behind the technology in order to move that system forward.
-- TTK
There is a website dedicated to search engines
Somewhere in all of the brain farts, lies a rosy bouquet.
-William Brendel
Why don't you "search" for one? LOL..... ok, bad joke
They tell the truth here: http://www.google.com/technology/pigeonrank.html Yep, PigeonRank(TM) is the way forward ;)
If I have nothing to hide, you have no reason to search me
Look at your page with Lynx.
Web crawlers can't see the text in your images and weird HTML constructions can make it hard to parse the text back out. If your page content can be clearly expressed in plain text there's a good chance a search engine will know what you're talking about.
As an added bonus, if a web crawler can read your pages so can blind users.
Ceci n'est pas une signature.
I found Mining the Web useful. It's written by academics, so you'll have to put in a little brain work translating it into implementable patterns, but it gave me a good jump start when I took on a new client that does a lot of crawling and searching.
To see Nutch in action... Mozdex
You might want to study "v7ndotcom elursrebmem" - the latest Search Engine competition. Just type it in your favorite search engine. If you enter it in Google, you'd even get strange ads.
Some are joining to get prizes from the competition, like v7ndotcom elursrebmem: Blogging for Charity
You might want to check out the paper Web Search for a Planet: The Google Cluster Architecture. It is 4-5 generations old, but provides some interesting information about Google's previous cluster setup.
There are a lot of papers on the underlying theories, but there is very little out there that will actually tell you how and when to implement them. Pagerank has really nothing to do with building a search engine, it is just one measurement that goes into determining relevancy. And it isn't applied the way most people think. I would say the best book on search engines is "mining the web" by Soumen Chakrabarti, but it doesn't really talk about implementations. But that's why information retrieval experts get paid the big bucks.
Read through Brett's 26 Steps to 1k per day. Great set of advice. And, the WebmasterWorld forum itself is quite a resource. Lots of experts in a lot of areas.
For print resources I would suggest:
Understanding Search Engines by Michael Berry and Murray Browne
as well as
Modern Information Retrieval by Ricardo Baeza-Yates and Berthier Ribeiro-Neto
For online resources I would of course direct you to the work of our Search Focus R&D Group
it doesn't matter whether he downloads an open source system, or inherits his employer's legacy system -- he will need to learn the principles behind the technology in order to move that system forward.
There are 3 basic approaches to solving any technical problem. Let's say, for the sake of argument, you want to cook a roast and have never done it.
Brute Force & Ignorance
Get 100 ovens. Stick a roast in each. Set them to all different combinations of time & temperature. When they're all done, sample them all, and go with the one that's given the best result.
The Scientific Method
Get a stove, and stick a roast in it. Using temperature probes placed at different depths in the meat, and possibly using some sort of thermal imaging, monitor the cooking process while varying the temperature over time. Make carefull notes and plot your results afterwards, and compute the optimal configuration.
The Engineering Method
Ask your mother
'nuff said