Building a Search Engine Using Open Technology?

cybrthng asks: "Mozdex.com is my attempt at building a search engine capable of indexing the entire web. Our goal is to provide a completely transparent system utilizing open technologies such as Nutch, Lucene and other systems to provide a search facility that is more scientific and 'protocol' vs the current propriety and almost 'faith based' search engine results and methods of getting listed. What do you look for out of a search engine? What would you look for out of this project? Should large commercial entities be the only way we find information and resources on the net? BTW, our beta index currently has about 50 million pages and we hope it shows what can be done using Open Source systems available today. We are seeking input on starting a developer & input community as well as getting concepts and ideas out and about, so we value your ideas and what you hope to see out of this project."

Comments

Open source search engine?

[Chester K]

An open source search engine is a great idea! I'll know exactly how to exploit the ranking algorithms to position my pages as #1!

httrack and grep...

[Anonymous Coward]

...and a *HUGE* hard drive.

Download the internet with httrack and search it with grep.

Re:how is this different?

[Anonymous Coward]

Step 1: Create P2P search engine technology.

Step 2:

Also, the logistics of distributing the search across so many systems would need to be worked out.

Step 3:

Furthermore, there is the possibility that users may attempt to tweak the client handling their node to increase the score for various pages or decrease the score for others. These issues would have to be worked out, but it could be feasible. Frankly, I'm too lazy to implement it, but you are welcome to credit me for the idea when its all done.

Step 4: Profit!!

Easy!

[JamesP]

cat database | grep query

Completely Open Source!