Slashdot Mirror

with all this censoring going on and all, perhaps it's time for a distributed peer-to-peer open-source search engine...

interesting, I just found this project: http://www.majestic12.co.uk/

Join Majestic 12 and contribute to an alternative search engine. You can have your machines index a certain amount per day and contribute the result to the index.

Having alternatives is what keeps companies honest. Government regulation just makes the regulators a target to be corrupted.

This Grub seems to be a human-edited system not unlike Wikipedia. I'm much more interested in algorithmic search, which is why your YaCy link was most welcome :) Another distributed search I've come across is Majestic-12.

http://www.majestic12.co.uk/projects/dsearch//

http://www.aspseek.org/about.html//

http://sourceforge.net/projects/ebiness//

http://www.grub.org/html/documents.php//

http://lucene.apache.org/nutch/bot.html//

I really want to see one of these projects take off, I'd tap a vein at the local plasma center to donate funds :>

http://www.majestic12.co.uk/

There already exists a distributed, open source engine which has been around a while, which is called Majestic 12. It uses a client-based search engine, which crawls the web for hundreds of millions of URLS, and then sends the data back to central servers. The servers than compile the data and use user-based searching algorithms to perform the search. While the algorithms are still very much in alpha, it is still a very noteworthy project. Also, its URL base is currently around 30-35 Billion URLs.

It's a pity that the good folks at Majestic-12 choose to implement it in .NET instead of Java.

>Perhaps the most interesting engine to flock to would be http://www.majestic12.co.uk/, a seti@home style distributed indexing system.

Now that is an interesting concept. Indexing the web would seem to be the kind of parallel operation ideally suited for distributed computing. You'd still need a central server to search the index and provide results quickly. (Okay, I decided to RTFL rather than just speculate, and I see that's what they're doing.) My initial assessment is that this is the most credible medium term threat to Google I've seen.

>...the fairness of their ranking algorithms are open to view and discuss -- perhaps with time such closed algorithms could be viewed with as much dislike as Microsoft's closed OS sources

Another excellent point. I wish I still had mod points. The closed nature of Google's ranking algorithms has disgruntled some folks, and an open system could become popular. Robert Cringely did a series on the mysterious workings of the AdWords algorithm, and whether Google is using the algorithms to "unfair" advantage. "Unfair" being quoted because even if they are doing it, it is not illegal, and perhaps not even unethical. But they could be deceiving or "gouging" (another loaded term) their advertisers, and it could be seen as counter to "Don't be evil". Cringely includes Google responses.

The point is, the advantages of open algorithms are pretty obvious.

http://www.pbs.org/cringely/pulpit/pulpit20050922. html
http://www.pbs.org/cringely/pulpit/pulpit20051006. html
http://www.pbs.org/cringely/pulpit/pulpit20051013. html

And there are some interesting contenders out there. Google tends to forget that there are people living outside the US, UK and China. I'm sure they are getting there, but while they linger there are others moving into attack positions. The local ones mentioned before are eating away their user share, since they can take into account location & language specific things that Google may be ignoring while pursuing it's grand world domination plan. In some languages a simple word by word matching scheme may not be enough and conjugation needs to be considered as well. Google has also upset some folks with the China censorship thingy and wanting to move all the users' data on their centralized servers, in general displaying the sort of arrogant behavior that slowly makes people want to see them fail.

Perhaps the most interesting engine to flock to would be http://www.majestic12.co.uk/, a seti@home style distributed indexing system. Sure they're not to Google's index size yet, but they are getting there, faster and faster... and the fairness of their ranking algorithms are open to view and discuss -- perhaps with time such closed algorithms could be viewed with as much dislike as Microsoft's closed OS sources. I wonder if Stallman is using it.

I am not a Google hater myself, personally I feel their search engine is adequate for my needs and their goal of organizing the world's information a very appealing one (although so broad that they might as well have said "we'll do what we please"). All I am saying is that it would not be unthinkable that the public opinion might slowly change, not favorably for them.

> Does anyone know whether Majestic Reasearch has any
> connections to Majestic 12 (http://www.majestic12.co.uk/ [majestic12.co.uk])?

As the founder of the Majestic-12 project I can assure you that we are not related in any way, shape or form.

Does anyone know whether Majestic Reasearch has any connections to Majestic 12 (http://www.majestic12.co.uk/)? For those who don't know, Majestic 12 is a distributed search engine. The distributed part is in that they have a bunch of people donate CPU cycles and bandwidth to run a web crawler in a SETI at home fashion. Now i thought this was a good thing to join, because we kind of need some independent alternatives to google. But if it turns out i'm sponsoring some marketing firm, well... i'd feel pretty stupid.

I hardly believe it'll take 300 years to index all the world's information, take the distributed computing project i'm with at the moment. They're hoping to use distributed computing to index every webpage, which when you look at when United Devices started their cancer research project they managed to do millions of hours worth of work without it taking millions of years ;) Take a look at http://www.majestic12.co.uk/ and see what I mean. They haven't got it indexing yet, but they're getting there!

Disclaimer -- I run a distributed search engine project so my opinion is biased.

It was noted above that while there are plenty of CPU sucking projects they don't seem to have end results that can actually be used in daily life.

OK, d.net proved the point by breaking crypto that was thought to be too strong. Fine, done that, why waste CPU cycles further?

SETI@Home -- okay, its cool to search for aliens, but lets be realistic here -- its cool, but not exactly useable.

Lots of effort, heck, lots is too small of a word to describe amount of CPU that went into these projects! Cool scoreboards, teams etc, but what are the end results for millions of users after good 10 years of d.net's existance!?!?! Not much.

This is why I created my own project to build something that I use every day -- search engine. I can live without aliens or crypto, but I sure as hell can't live without a good WWW search engine. Can you?

Distributed projects have been around for a long time. Some of them are (SETI@Home) are cool, some were even useful to prove the point (Distributed.net), but what are the real-life uses of calculating Pi to the gazzillionth number? Come on - where is the benefit that you can take advantage today, tomorrow and every day after that?

There are none - in many respects those well known projects is waste of CPU time and electricity. But there are projects that aim to create something that will be of daily use to everyone... like a search engine. How many times you use WWW search engine every day? I use loads, so much that I could live with Internet Explorer 1.0, but I could not live without a search engine.

One of those projects is the Majestic-12 Distributed Search Engine project (http://www.majestic12.co.uk/) that aims to build a search engine contributed to and controlled by the community. We need help - if just a fraction of people who run other distributed projects joined us we could have build a major search engine in no time! You can run it with CPU intensive projects!

Disclaimer: I am the founder of this project.

Don't just measure - help build actual distributed search engine

And also neither msnbot nor Googlebot request pages with gzip compression, only semi-common bot that I know of doing that is Majestic-12.