Slashdot Mirror
Nutch: An Open Source Search Engine
Anonymous Coward writes "Someone forwarded me this site working to create an open source search engine called Nutch.
In the age of weighted rankings on search engines for profits, there's an obvious need for an unbiased search engine. After all, isn't a search engine supposed to be for finding relevant data, not as an indirect and sometimes slimy method of advertising?
Nutch is clearly in their intial stages, but it would certainly get my vote." You can find the project on SF.net, and also read the Business 2.0 article on it.
I hope the authours of this project do their homework. My impression is that most of the good search and indexing schemes have already been patented, which will make it difficult to release such a project without stepping on someone's toes.
Last i heard google still doesn't accept bribes for page ranking.
inobtrusive adverts on the right hand column nonwithstanding.
do() || do_not();
Also of note is that companies can still influence search engines in slimey ways - Google can be manipulated to make a page rank higher, although Google keeps an eye on this activity and works around it.
I hate liberals. If you are a liberal, do not reply.
I'm quite comfortable with how Google does this (present commercial links clearly marked to the side), and am not convinced a non-commercial (open source) alternative is needed.
I think that you absolutely have to have a closed source algorithm for ranking pages, because otherwise you'll get people who will simply tune their pages to be high on the list. I can see how making the majority of the search engine open source would be beneficial, but the algorithm itself? Its like saying "Here's the keys to my car" and thinking that, because everyone has access to the keys, no one's going to drive away with it. Sure, everyone has the opportunity to make your search engine better, but never underestimate the tenacity of a web-wanna-be-millionaire.
Here's what I expect to see on the webpage in a few months: "Currently Nutch is in the alpha stage- it doesn't index any web pages, doesn't return any results, and has no user interface. Programmer's needed!" Google has WON the search engine war, probably forever. Find some other mountain to climb, guys.
And I'm sure many Slashdotters would love a search engine dedicated to find pr0n and anti-Microsoft propaganda. Right?
To me, accuracy is the most important "Relevance".
The problem with Google is that there are errors in it: you ask for something and sometimes you get something else.
A search on "to be or not to be" produces an error (non-matching results) in three of the first ten results: a 30% search failure rate. It used to be worse, when most of the links were bad.
Since it seems like Google will never fix this problem, I'm looking forward to something with all of Google's great features, plus accuracy.
Don't blame Durga. I voted for Centauri.
Just use google. Search for "SEARCH-STRING site:slashdot.org"
Free and open code is good and all... but the one real cost of a search engine is RUNNING it. It requires a far from trivial amount bandwidth and hardware, and somebody has to pay for all of it. Unless someone comes up with a novel P2P solution (and many are trying) it just won't happen.
What they should be doing is pressuring the existing search engine companies for some integrity.
---If you can't trust a nerd, who can you trust?
google is already ideal... the weight of search results is not sold, just text ads.
people are already 'googlebombing' to try and get better rankings by signing up tons of domains and cross linking them all with the keyword that they want to be #1...
if the algorithm that determined how #1 is determined was public, then the best possible strategy to cheat the system could be demised... instead of paying for weight to the search engines you would be paying to web developers to make the search engine think you were #1. and as a web developer i feel that.... oh... wait, proceed.
MARIJUANA, SHROOMS, X: ONLINE?! - E
I think the idea is good in principle, but could it actually succeed? Google gets hit with millions of request each day. They've got hardware that can support thousands of slashdottings a day and a fat pipe to feed all of that info out. That takes alot of money. Financing an open source project is difficult enough, but financing an open source service such as that would seem next to impossible. Ideas?
The other major problem would be that, with the ranking criteria being available for all to see, it would be relatively simple to manipulate page rankings.
"Google has WON the search engine war, probably forever. Find some other mountain to climb, guys."
At one time, Oldsmobile won the auto company wars. Where are they now?
IBM ruled the PC roost. Hmmmm....
Command-line OS's were king. But now???
Altavista and infoseek and Lycos were search engine kings at one time. Whither this trio?
The point is, it is not over.
Don't blame Durga. I voted for Centauri.
One of the biggest issues with running a search-engine, open-source or otherwise, is that you can't eliminate bias in the results. No matter what scheme you put in place to handle rankings, someone will find a way to take advantage of it. It's a fact of any major system - there's always a way to twist it. Part of the challenge that Google and similar sites face is that they have to work constantly to protect themselves from systems designed to take advantage of their algorithm. While a completely unbiased search service would be nice, I think it would require the impossible. It would require that no one out here took advantage of it to further their own interests, be they political, commercial, or otherwise. That's fairly unlikely.
With most of the major engines today including Google, they make an effort to prevent horribly unbalanced results (recent controversy over blogs outweighing professional sites in the rankings due to linking and other factors). Some even admit (again, Google does) to manually messing with the rankings a little. If you search for suicide methods, they will bend the engine to make sure you get reasons why you shouldn't commit suicide before you get the how-to. That's in their own public docs. It's also discussed in Wired.
I honestly don't know if open-source could do a better job. The algorithm might be better (likely, given the manpower), but would it really be that much fairer?
"Be proud to be a fighter" - Martial Arts Adage
who knows... but as soon as they get it working, they can use it to search for a better name!
Don't worry. It is just a stepping stone to full project maturity reached when it is fully coded in Borland Turbo Pascal.
Don't blame Durga. I voted for Centauri.
I fail to see the point of such an endeavor. Without advertising Nutch can not possibly hope to become a serious contender with search engines such as google or overture. Advertising provides the money that enables search engines to have lots of bandwith to send those results quickly back to users, lots of computing power to quickly process each search, even the ability to hire people to research into new areas for better search results. Even if the search engine is selling its resources to other portals like google does with yahoo advertising would still be involved in the process. Yahoo would still need to be advertising on their site to bring in revenue to pay for the service. I think google's method is perfectly fine with small text based ads that are discrete. Why do we need to fix this?
Go Illini!!!
I think they're setting themselves up for something that will get too big and too expensive before it can get finished, and they'll have to figure out a way to (gasp) get some funding beyond donations.
I don't see a solution in one great open-source, independent search engine, but many individual specialized search engines, each mastering their own niche area of specialty stands a chance to compete, especially if run by people who focus on their areas of expertise. Alternative news search engines, music search engines, literary search engines, etc. each run by people who know what to filter in and out.
If Nutch.org could create the technology that would allow each of these search engines to exist autonomously, it could also be the hub/portal/start-page/blahblahblah that links all these engines and databases together.
Alex.
It'd be nice if they could make distributed. Kinda like P2P search engines, but for the web. That way, the main searching server farm wouldn't be tied to any company in particular. That would give Google a run for their money, and would keep Microsoft at bay for another while.
Being open an open search network, some peer servers could specialize in searching what they're hosting, making it possible to index otherwise dynamically generated content. These specialized hosts would act as "search plugins" for some otherwise hard-to-define content.
An authentication method (a la Freenet) would be needed, though. Some form of authority to prevent rogue peers from injecting too much crap in the results.
Overall, a good idea. If they make it, I'll run it.
-- Home is where you eat your heart out.
The answer: "What did Sean Connery say when he saw the reviews for 'League of Extraordinary Gentlemen?"
Trolls lurk everywhere. Mod them down.
Here you go.
Porn
Anti-Microsoft Propoganda.
Make America grate again!
Grub is another open-source search engine, I have the client running right now, its nice and distributed, I think this kind of idea is great.
i use linux and windows oh god how can i have an opinion
Ooh, what's this?
Overture Research has donated hardware and helped to fund development.
So, even an "open source," "unbiased" search engine is funded by a commercial search organization.
let's see where is the funding coming from. Project is funded by overture which is to be bought by Yahoo. More info is here. Hmm.. So i guess Yahoo needs a revival...
bin
look siG is kool
Don't worry too much. This is software, not a service. When available it may be implemented by someone and be the infrastructure of a company, which may then provide bugfixes and development to the original project. Or it may not. Who knows.
Nutch has four developers, one of whom is Doug Cutting who wrote several indexing engines. They count Alexa founder Brewster Kahle as a "friend" and are sponsored by Overture.
No, many comments don't end up getting indexed by Google, and recent discussions aren't indexed at all. I've tried that method in the past with little success.
I think having an open source search engine that people can modify and deploy would be an excellent thing, and here is why. Currently, google has the complete power to highlight or censor anything on the web. So far, they have used this power wisely, but that's no guarantee that it'll always be so. If they go public, you may find this power being used to increase the shareholders' wealth, rather than in the highest standards of fairness as it is today.
With that in mind, how would this project help? It would allow webmasters to quickly & easily modify it for their needs, and deploy their own niche engines; in other words, Google would be supplemented by 10,000 niche search engines, each focusing on a specific field (microsoft propaganda, for instance). This would create a balance of power, ensuring that no single search engine accumulates an insane amount of control over the web as a whole.
I made a PHP/MySQL library that prevents SQL injection & makes coding easier!
Nutch
E.
Never rub another man's rhubarb - The Joker
"In the age of weighted rankings on search engines for profits, there's an obvious need for an unbiased search engine."
Bias is inevitable -- we're talking about ranking, which necessarily means bias.
The question is: what bias do you want? What bias suits your purposes?
My ideal search engine would offer a variety of biases from which to pick.
-kgj
Lucene and Nutch are related:
http://scriptingnews.userland.com/2003/08/13#When
Paul Nakada, via email: "It appears that the coding muscle for Nutch is Doug Cutting, the author of Lucene, an Apache Project open source search engine. We use it here at salesforce and have a huge amount of respect for Doug's coding."
cpeterso
Why is it that when it comes to OS, everyone is bitching and screaming how bad monoculture created by Microsoft Windows is, but otherwise feeling warm and fuzzy and swear to god Google is and always be the only search engine they use?
:
The point is, are you really comfortable to have one, and only one, effective search engine? No matter how well it searches?
O'Reilly put it best
Actually, Nutch has no ambitions to dethrone Google. It's just trying to provide an open source reference implementation of search to help keep Google and other search engines honest, by letting people compare the results of an engine whose algorithms and methodologies are transparent and accessible. It also aims to give a platform for people outside of the search heavyweights to research new search algorithms.
That's nice that they want to open source the engine but that's the least of a search engine. They're going to need multiple high end servers to process the searches and plenty of bandwidth to get the results to the users.
How do they plan to pay for that? Apparently advertising is out. And we just had another monephobe complaining about lack of funds for his accounting software who expected people to donate because he couldn't figure out that maybe, just maybe he should find a way to sell his product in some form while also keeping one form free. I can get RedHat for free OR pay money to get a hard copy with some bonus stuff. Net result is that RedHat makes money and everyone is happy. Those who refuse to pay don't have to and those who are willing to pay have a reason to. Most people are not going to just give you money out of the goodness of their heart and accept nothing in return if they don't have to. Why do you think PBS gives you gifts with your donations?
I'd be more impressed with such undertakings if the owners weren't convinced the bandwidth fairy was real and that money will fall from the sky like mana.
When someone comes along who recognizes that the bandwidth fairy doesn't exist and that money needs to be aquired through marketing to get any real amount then I'll think twice before laughing it off.
Free is a pretty dream but free don't pay the bills.
Ben
Work Safe Porn
Nutch - Not Understanding The Capitalist Hegemony (I am just making it up
Without a sound revenue model they can't operate for more than a month. Google has indexed billions of pages and to operate at that level they have to spend a lot of money (Google recently leased an entire campus from SGI). To meet the Infrastructure costs alone you need some form of commercial revenue stream.
It seems like there would be a better choice than Java for the language when speed/efficiency is a must. Isn't the added overhead of the JVM going to decrease performance significanly?
Portability should be a mute point since the pages can be generated on the server, which could easily run an OS specific binary.
167 posts and no mention of ht://dig? It's a great open source search engine, and I've been using it daily (well, cron really uses it now, not me) to spider about 100 sites on my intranet, which has servers all over the world.
While not currently designed for massive whole-web spidering (it's aimed at single websites or intranets), ht://dig is a great starting point (and a lot further along than the Nutch 'nascent effort' mentioned in the story). Some database optimization to ht://dig seems easier than starting over with Nutch. Plus, the name 'Nutch' sucks.
everything in moderation
On Google.com, it is VERY clear what are paid ads and what are "real" results. With MSN, for example, they list Featured Site (you pay MSN), followed by Overture (you pay per click), following by the Looksmart Directory Listings (used to just pay for submission, for the past year, Looksmart charges $0.15/click for those results).
After the "paid" listings come the Inktomi listings. Those crawler based listings include PFI (pay for inclusion, you pay for daily spidering, but no "boost" in rankings) and the Partner Connect program, where you get free traffic for a week, then negotiate a PPC price for traffic.
If you search MSN, you would get the impression that Featured is editorial (which is kind of is), Sponsered is paid, and Directory/Page results are "real" search results, where the Directory/Page are often actually paid results.
The paid traffic from Inktomi involves an XML feed of terms and results, and your "fake" entries are treated as real entries with a boost for being a paid player.
In addition, for various adult terms, MSN tells you to use a third party adult "search engine," which ISN'T a search engine. It is a big player in the adult space that pays MSN for all the traffic and lists their sites, but does it in a "search engine" look and feel.
That is manipulating the rankings. If Google were to say, charge for the XML entry (either PFI or PPC) into Froogle.com, and then shot the Froogle results interspersed with Google results, that would be manipulating the rankings for money.
That is the manipulation angle.
Now, are paid results any better/worse from objective results if those are manipulated by SEO professionals (so you pay an SEO to get "free traffic" instead of paying the SE for the traffic)?
It's certainly more manipulated.
With a free engine, you could tweak the rankings and sell ads on the side, but not have manipulated editorial.
It's about maintaining a wall between advertisements and editorial, and the only engine that appears to have that wall is Google, and even Google pushes the boundaries.
Alex
For example, look at the results for the search 'convert wmv mpeg'. The first three results lead to the same exact search site. (Whether they have pop-ups or not, i can't tell, because i block them.) The fourth result is another search site. And then the last three are the same as the first three.
Of course, this obviously works with stuff you'd expect it to, like 'mp3s' and 'warez' and 'porn', but it works with legitimate stuff too. I wonder if there'll be anything to combat this trend, whether it be implemented by Google or by someone else....
I was looking over the site and a number of things concerned me.
Firstly the choice of Java, personally I have no gripe about this. And reading that a choice was made to use language-independent formats is a good idea. My main concern is for the larger scaling and distribution over multiple machines.
At present I make the educated guess that a project on this scale, in Java, would still be best run on a `hardware base as uniform as possible', like UltraSparc 450's with a fibre back-plain.
My second concern is that there is so much choice of indexing and searching technique that there are sure to be some problem due to Patent restrictions.
Just browsing the US patent office gave me a couple of possible Patent nasties;
6,463,428 or 6,278,992. (And about 10 others I glanced at...)
Lastly DB, in the short time I've been looking at the code it seems to me that a choice was made to implement a DB build for the problem. Although this could be a good thing, it is usually better to reuse existing products. I found SleepyCat (DB4) to match the requirements. And if the choice is final read this. [1]
I hope these comments are useful to somebody at least.
[1] http://www.xlnt-software.com/xml_dl.html
'I am become Shiva, destroyer of worlds'
I have a few comments on this development:
An open search engine application is a nice idea, but unfortunately it's one of those applications which are essentially useless without an enormous ASP architecture behind it. An earlier poster indicated that it might be useful for searching and indexing intranets and the like, analogously to the Google Search Appliance. This is indeed a valid potential application, but then, HT://Dig exists already. Is this dramatically better?
I've noticed that searching for Eric S. Raymond's home page brings up his actual home page third or fourth in the listing. I don't know if that means Google is on it's way to going downhill or what. The first listing it brings up doesn't appear to have anything to do with ESR. I don't even think his name appears anywhere on the page.
This is exactly like the problem the mice had one day. They couldn't come out of their mouse hole because there was a dangerous cat prowling around. One day, as food was getting scarce and everyone was afraid to leave the hole, the mice called a meeting to discuss the problem. One excited young mouse came up with the most wonderful idea: Let's put a bell around the cat's neck, so that when the cat is nearby, the mice would have advance warning and could escape! All the mice got excited at this proposal, until a very old, very wise mouse came over and asked, "And who will tie the bell around the cat's neck?"
What I'm trying to say is: If the search engine is free software and companies don't pay to increase their ranking... who will pay for the bandwidth to host the engine? I can tell you this much:
Proposed solution? Make it a distributed search engine, like SETI@home, or the DNS.
This is much easier said than done because:
- RAID-like distributed storage technology would have to be developed, so that the indexing database could be distributed among all computers worldwide that donate bandwidth and storage. This would have to guarantee statistically that all the data will be available at any point in time even if people turn off their computers for extended periods of time. However, this technology could make reliable clustered storage a reality, and the resulting free software implementation could be licensed for corporate use for an exhorbitant price, which would go to the EFF, FSF and other organizations that develop free software and/or support the development thereof.
- An efficient P2P-like protocol, along with a network topology of some sort (like the DNS system has) would have to be developed to support the searching; It would have to be damn fast and, like before, very resiliant to computers being shut off, chunks of data becoming lost at any moment, etc. Furthermore, changes would need to propogate at blazing speeds so that new items on the Internet could be found shortly after appearing.
- Bandwidth and disk quota would need to be managed at each participating host, so that limits set by the user are not exceeded.
Governments, companies, universities and individuals would likely support an effort like this by donating some bandwidth and storage, rather than money.In the spirit of worldwide computing on the Internet, I hope this makes some amount of sense.
If you reply, do so only to what I explicitly wrote. If I didn't write it, don't assume or infer it.
That's all they're teaching the kids in college these days. Seriously. At the school I go to (i'm not a CS major) you have to take C/C++ as an elective. The core CS curriculum is all Java. I don't think they even teach assembly there. Good schools are probably different of course, but who can afford good school anymore?
/. while high, it makes one ramble.
I met someone the other day who had an an associates in Computer Science from a community college and had never used anything but an AS/400 and a Mac. (Not even Windows! Seriously!) I think people saw the dollar signs from 4-8 years ago and went to school for something they really are only marginally interested in just because they thought they could make a few more bucks. It's a damn shame too, because these shitty little tech schools (or community colleges hawking their tech "degrees") are doing a disservice to these people by making them that 1) they'll get a good education, 2) they'll have a job waiting for them when they graduate, and 3) they'll be respected by their peers in the IT biz as professionals.
And then there's my niece who got a nursing degree in 18 months, had 4 interviews her first week out, and now makes more than your average 4-year college graduate. Go figure.
But anyway, yeah, I agree with you. Don't
This message brought to you by the Council of People Who Are Sick of Seeing More People.