Slashdot Mirror

I have seen also mentions of the Xapian which is written in C++.

Pleh!

Xapian and NLTK all the way!

For those who didn't know, Xapian, the search engine he used for this, is really awesome. Its very fast, stable, actively developed and packs some pretty impressive features. Its written in c++, but has bindings for Perl, Python, PHP, Java, Tcl, C#, and Ruby. If you need an embedded search function on a site, you should check it out.

I've used it for over 2 years on various sites and am really pleased with it.

The parent has a good point..

I'm currently messing around with turning Mediawiki into a basic CMS. Search has been a lot more effective at returning usable results since I changed over to Omega.

A year ago I was looking at search engine software and search query parsers. I didn't want to mess with setting up Lucene on Java. I found another tool called Xapian which compiles on Linux from C and has bindings for PHP, Perl, and other languages. I've found it to be fast and stable. The documentation is sort of spotty but the guys on the mailing list are great.

If Borland are being stinky, and poeople are starting to suffer from being "locked in" (even thought it was nice while it lasted) then its time to look at:

wxPython
and
Boa Constructor, a python IDE and RAD style designer. Its a bit harder than Delphi (or am I getting mentally stale) but at least its fully cross platform GUI and open source, so you get to increase your target market and never get locked in again.

No point in continuing with Delphi.net, it may only happen again in a few years when the fashions change.

I learned this lessen a few years ago when a nameless search engine salesman witheld some updates and we (Ananova/Orange) switched to the open source Xapian search engine and paid one of the original developers to do some more work on it for us. Xapian is now being trialed as the search engine behind gmane

Its the same lesson Richard Stallman learned years ago. Don't get locked in.

There's no need to learn that lesson twice. And, you may as well join the FSF while you are at it. You know it makes more sense than most political donations, and for less than the price of a night out each month! I got a copy of Lessigs "Free Culture" in the post today as part of my FSF membership.

Sam

It's the storage library that keeps an index of pages. You need a display front-end and a webcrawler to go with it (there's some code around). It's GPL and it has some clever features.

You may want to have a look at Xapian. It's an Open Source document indexing system, which in my experience scales very nicely. The Omega system, which is built on Xapian, is a nice not-so-little search indexing system. I've used it on multi-million (admittedly reasonably short) document indexing projects in the past and it's coped admirably.

The search refinement system it supports may seem a little strange, but to me it seems powerful and you don't have to use it if you just want 'simple search'.

Indeed. And if you want to experiment with relevance feedback, take a look at the Xapian Project for a highly scalable GPLed implementation of the Probabilistic Information Retrieval Model (which is derived from Bayes Theorem, the basis of all those Bayesian spam filters).

The only search demo up at the moment is the search over the site itself, which doesn't show off the relevance feedback especially well - the pages on the site cover a rather narrow topic area, which is fine for searching the site, but less good as a demo. But if you try it you'll see it suggests various related words and you can click on these to add them to the search. Additionally you can click a checkbox next to a hit to indicate that it's relevant - check a few and hit search and the suggested words will be based on those documents you liked.