Slashdot Mirror
Building a Search Engine Using Open Technology?
cybrthng asks: "Mozdex.com is my attempt at building a search engine capable of indexing the entire web. Our goal is to provide a completely transparent system utilizing open technologies such as Nutch, Lucene and other systems to provide a search facility that is more scientific and 'protocol' vs the current propriety and almost 'faith based' search engine results and methods of getting listed. What do you look for out of a search engine? What would you look for out of this project? Should large commercial entities be the only way we find information and resources on the net? BTW, our beta index currently has about 50 million pages and we hope it shows what can be done using Open Source systems available today. We are seeking input on starting a developer & input community as well as getting concepts and ideas out and about, so we value your ideas and what you hope to see out of this project."
Support our index, sponsor mozAds keyword Advertising as low as 1/cent click
Is it different only because it runs on open source software? Hell google does that successfully already.
The thing I look for is a polite bot. Does it follow robots.txt fully? Does is hammer the server? Does it page modification headers?
An open source search engine is a great idea! I'll know exactly how to exploit the ranking algorithms to position my pages as #1!
NO CARRIER
buy an ad ;)
How about you work out some way to do this old saw:
c Hint("plain");n autical");
Searching for "Jaguar" the fighter bomber as opposed to "Jaguar" the comic book character.
But then, I'd want you get into natural language processing to determine what the real "topic" was that I meant. Of course I'm assuming a free form field. I'd like to just be able to put in "Jaguar the bomber" or "aeronautical: jaguar" or "plane jaguar" or even "plain jaguar" and have it do a Googlesque "Did you mean 'plane Jaguar'".
Hmm, and a fun API so you could build a search....
query = new Query();
query.setTerm("jaguar");
query.setTopi
or
query.setCanonicalTopic("aero
BTW, our beta index currently has about 50 million pages and we hope it shows what can be done using Open Source systems available today.
Uh, crawling & getting 50 million pages isn't the hard part. Searching them is.
Good luck. You'll need it.
Actually, there are a lot of open source porn-search projects. For instance, gnaughty, and the Porn Toolkit.
It's something to really consider, because I can see how an open algorithm would be beneficial, but it's very easy to see how it can be spammed into uselessness.
I think of the Dow and other financial indices and believe that the proprietary model may be the only successful way to provide useful, reliable information.
Then I look at encryption, and I see how the algorithms, being public, can be vetted without compromising the security of the communication through a proprietary, secret key.
I suspect that a succesful search engine, at this juncture in time, cannot have an open 'key' due to processing power limitations. Given infinite processing power and a reasonable language-understanding algorithm it could be possible to make the spammers work so hard at getting into the listings that they would actually have to provide useful information about a given subject just to show up in the listings.
However, a search engine cannot make enough money off of ads to even approach that power. We first have to create better natural language algorithms, because the web will grow as fast or faster than processing power.
At this time a good compromise would be to follow the encryption route. You use publicly vetted methods of rating pages and indexing pages, but you keep the key, the actual calculation that relates those ratings to a particular search, a secret and modify it to keep up with spammers who will brute force the key into the public.
Again, I disagree with the theory that it is possible to design
1) A fully public algorithm for page ranking and relating them to search queries
2) Keep spammers and keyword hijackers from ruining the index
3) Without infinite computer power and/or a very advanced natural language indexing system.
Don't let this dissuade you from trying. Search is not a solved problem, and there are many opportunities for small (and large) successes before we even come close to a complete solution. You may well find another important piece of the puzzle.
-Adam
What would you look for out of this project?
The only thing that matters is results. Is the answer that I need in the first three or four results? If you can do that, you win. If you can't, don't bother.
I'm skeptical about how realistic it is to develop an open source search engine. Wikipedia, although cool, has large gaps in content, and only a few months ago was begging for donations to survive. I'm betting that a Google sized operation would be even more resource intensive.
Three Squirrels
of such a search engine would be something akin to the philosophy of openness that is common to GNU etc, and free as in beer of course. It's OK to have open rankings, if the point of this is to index the more non-commercial side of stuff. And don't bother to cover the same base as google. What's the point?
What I have found REALLY interesting about MozDex is the "explain" button which I assume provides some insights into why MozDex decided to rank that web URL as whatever ... but the information as currently presented isn't understandable and/or explained.
For instance, I was interested where a Google Compute web page came up and was actually quite surprised that a MozDex Search shows it as #1. So I click on the explain button and I get a page with a buncha numbers ... but nowhere on this page (or anywhere on the MozDex site) can I find an explanation for what they heck they mean.
Since your claim-to-fame is open source/search, I think adding information on the internal algorithms would help you out. Keep up the good work - interesting stuff! ;-)
alek
P.S. Minor typo in the Corporate Info link from your FAQ
Hulk SMASH Celiac Disease
Well, it could easily assess a penalty to any website which was ranked above that which the user clicked upon in any given search, thus ensuring that "xxX Hot Teens Slashdot Xxx" is punted down to the bottom of the Slashdot searching pile. Of course, you would need an uncrackable way of summarizing the pages...
The ______ Agenda
- less importance given to commercial sites and blogs, more importance given to general informationw ords.html and their ilk)
- results not easily manipulated (eg. http://your-search-keywords.com/your-keywords/key
- fast discovery of new or updated sites
- features such as caching, view as X, spam reporting
the dead shall rise, from their graves, to destroy, geometry.
While browsing the Mozdex site, I learned they are using Nutch, an open source search enigine. So I started browsing the Nutch site. On their site I found out that they are sponsored by Overture Research ... The name seemed familiar. Clicking on the link I arrived at http://labs.yahoo.com.
Apparantly Yahoo is rather interested in this project. Browsing the Yahoo Labs site I found this page(which is also the third hit when googling for nutch): "Welcome to the Yahoo! Research Labs implementation of the Nutch open source search engine (www.nutch.org). This search engine is intended as a demonstration platform for a number of search related technologies that we are working on and is specifically not intended to provide a full and comprehensive search experience for the average user. If you do a search here, please do not be surprised or offended if your favorite site is not in the result set for your query.
With this in mind, please feel free to test drive the technology. Happy Nutch-ing.
A very quick test shows that the 50 million pages counting index of mozdex is indeed still far to small to really find something. The ranking system will also need some tweaking, but this is also clearly stated on the nutch site: "Nutch has not yet been tuned for quality. There are ten or twenty knobs that we can twiddle to adjust the ranking formula. We are developing software to do this tuning automatically, but the current code just contains guesses. With a little tuning we should be able to get results that are competitive with those of major search engines.".
Although it is currently not possible to do any real comparison due to the big difference in the number of indexed pages, it sure is nice to see both the Nutch project and the Mozdex project. I hope that both of these project will receive enough funding (and hardware) to continue, and maybe we'll see another /. post when they hit the 5 billion page count and we will be able to do a massive comparison ... and all change from googling to nutching or mozdexing!
One to watch
I'd love to be able to filter out all sites that are trying to sell something.
Searching on Google for things like reviews of mp3 players has become a nightmare these days. Any useful sites are drowned out in a noise of pricerunner/dealtime/kelkoo/shopping.yahoo/etc and other sites that are simply affiliate sites for Amazon etc.
...and a *HUGE* hard drive.
Download the internet with httrack and search it with grep.
An OSS search engine that actually indexes the entire web and is used by many people is at least a couple of orders of magnitude harder than the Mozilla project.
Writing the search code itself is not too hard (you still need a PhD in data structures and algorithms, but those can be found), the real hard part is the amount of bandwidth and CPU power that is required.
You need a name that is as easy to pronounce as google. As friendly sounding would be good as well.
You're "competing" on a number of different areas with google, including the name ofcourse.
The first thing that came to my mind when I read the name was: "Typical for geeks who are good at the technical side of things, but are bad at marketing and the human interface/psychology side".
- -- Truth addict for life.
Right now, there might be 50 million pages indexed, but right now it looks like I've got to go through 1 million of those to get to what I searched for.
My two tests were 1: "Hattrick" which is an online soccer management game, and is great. Google it and up it comes with some handy links to some sites about it. Using this engine, I got a bunch of crap. It may have been pages that linked to hattrick, but I didn't check.
2: "Buyer Agent Boulder Colorado" - Exclusive buyer agents are the preferred way of buying real estate. I know some in Boulder and wanted to see how they'd do. While the site I was rooting for didn't show up on the front page of either engine, a major EBA office did pop up as the #1 result on Google, no such luck on this other one.
Hell, with their bias towards open source, I would have expected decent results for a search on Linux, but no luck. This is a nice idea and all, but the interesting part of the search engine is the ranking algorithms. That's what they pay the PHDs to come up with in the proprietary world. That money seems to be well spent. Looks like you need to reach out to academia and beg for some algorithms.
Try this:
MP3 player review -buy -dealtime -pricegrabber -kelkoo -shopping.com -amazon.com -nextag.com -bizrate.com -moreover.com -celeb -porn -free -coupon -pimprig.com
I have it set to a hotkey for all the -'s and boom relatively valid search results.
A gift from a marketroid to you techies.
cat database | grep query
Completely Open Source!
how long until
Back to the drawing board.
when you search for google, the first page does not even link to www.google.com. I cannot see how a search engine that does not even mention the obvious results would do well.
Pretty cool, but much of it is redundant since you lose all but the first ten search terms.