Slashdot Mirror

There are many ways to skin this cat. I believe most of them have been mentioned, but I will outline my experiences anyway.

swish-e is a grand-daddy of an indexer. It can act as a robot, crawl your local file system, or get its input from STDIN. If indexing HTML, swish-e will index the document's metatags and provide field searching against them. Swish-e comes with a C, Perl, and PHP API. I don't think swish-e supports anything but ASCII very well.

kinosearch is my new favorite. Written in C but with a Perl API, this indexer works a lot like Lucene. Its resulting indexes (files) may be readable by Lucene. Kinosearch works by initializing a "document" with attributes, filling each attribute with values, and saving the document. Searching is fast an easy. It does not support wildcard searching, but uses extensive stemming instead. Kinosearch does not index files from your file system; you must parse your data and feed it to Kinosearch.

Ht:/dig is nice, but the last time I looked, it had no API. I found this to be too limiting. It indexes documents.

The Google Appliance is cool (and kewl) but also very expensive. This black box (well, it is really gold or blue) does a lot of the work for you. Configuring its output is dependent on your ability to do XSLT. You can feed the Google Appliance database dumps and other streams of data. Nice. I still think the price is steep.

There's Plucene, a Perl port of Lucene. Too slow, and seemingly unsupported.

Lucene and its kin seem to be the Gold Standard these days. I appreciate that, but alas, I don't have any Java experience. Increasingly people swear against SOLR, a Web Services-based interface to Lucene.

Zebra is an unsung hero. It has been around for more than ten years, actively supported and used extensively in Library Land. (I'm a librarian.) This thing can index just about any kind of document. It supports every type of searching feature (stemming, wild card, fielded, Boolean logic, relevance ranked, etc.). It can read files or be fed things from STDIN. Fast!

As an added bonus, I advocate readers explore abstracting their search interfaces with something like OpenSearch or Search/Retrieve via URL (SRU). These abstract layers allow you to create user interfaces to your underlying indexers without worrying what those indexers are. In other words, these abstract layers define the syntax for queries, the transport mechanism to the index, and the structure of the returned result. Given such a framework, you can write an OpenSearch or SRU interface to your index, but if you decide that Lucene is not what you want to use anymore but Kinosearch is, then you can change your indexer without the need to change your user interface. Very nice. OpenSearch is simpler to implement but is weak when it comes to expressive searches and search results. SRU is more robust but also more complicated.

I've worked IT at a Big-10 research library the past 7 years and GPL'd SourceForge project: http://libdata.sourceforge.net/. There's an excellent web site, dedicated to evangelizing open source software in libraries: http://oss4lib.org/. One progressive company that jumps to my mind in particular has bridged the open source paradigm with the basic necessity of earning an income. My hat goes off to these guys hailing from Denmark: http://www.indexdata.dk/.

That said, libraries exist mainly as government-funded entities here in the US. And, when you think about it, government agencies by-and-large don't actually produce things themselves -- they primarily exist as subsidizing entities: they have a mission and a budget, and "contract out" to the private sector, whether it's building spaceships, tanks, spying on would-be terrorists (or you and me), or stocking libraries.

One of many problems that libraries are encountering, I think, is that open source technologies -- and information outlets -- sort of violate the long-standing tradition of government=subsidizer. There have been some attempts (R-Santorum, as I recall) who tried to limit NOAA from offering any weather service that competed with the private sector (Google the specifics). I wonder if there's some political pushing that wants to prevent libraries from treading on their vendors' bandwagons also. This is very problematic, since we're in a post-industrial era, and practically any service you offer potentially treads on someone else's interest in offering the same service -- but with a price tag.

I'm now middle-age, and worked in public libraries 11 years before my current gig at a large university. I've seen (and assisted) libraries go from card catalog to fully automated, to (slowly but surely) private database subsidizers. It's the Y and Z generations that will need to really hammer this one out. Your chief challenge will be to change the nonsense model that requires tax/tuition-funded faculty to publish in closed venues, relinquish many of their rights, and the citizens/students are forced to buy back the same rights. It's dead model. The etymology of "publish" means "to make public". Today's dynamic is quite the reverse, sort of the anti-publishing industry, setting up protected access barriers more so than conquering them. Ponder this carefully.

The other thing to keep in mind is that academic is "one of the last great medieval institutions" as an IT consultant I once worked with at the University termed it. I worry that they are antagonistic toward sources like the Wikipedia for all the wrong reasons. If you think about it carefully, professors grade papers based on (a) the accuracy of the information the student presents and (b) how well the student properly cited his/her sources. If the information was correct, why should it matter whether it was his astrophysicist neighbor (personal communications are citable sources), textbook A, research paper B, a ridiculously expensive database that the university had to subscribe to, or some free source of information?

I think I know the answer, but simply knowing it won't help matters at all. It'll entail a change of guard -- so it's up to the under-40 crowd to figure this one out, and when they become the next generation of library managers, university administrators, and IT directors suggesting that libraries might become Wikipedia mirrors (hint, hint) and contributors, things may then begin to iron out on their own. :-)

To be honest, the widespread porting of the lucene engine(beagle backend) is the highest amount of forking i've ever seen.

There exists a port for every language, it just doesn't make sense. The basic algorithms for searching, and storing indexes hasn't changed for quite some time.

In the digital library space there even exists quite old (10 years) open source software such as zebra[1] which can handle large indexes fast. There are actually open standards[2] for information retrieval (IR), but nobody in the open source desktop space seem to know about it(?).
[1]http://www.indexdata.dk/zebra
[2]http://www.loc.gov/z3950/agency/zing/

You definitely want Koha (www.koha.org). For Koha support, and special flavors: LibLime (www.liblime.com) and for even more library goodness, Keystone (http://www.indexdata.dk/keystone/).

Both Perl and PHP already have Z39.50 support to connect to full-text search engines.

If you are looking for Z39.50 servers to search, you should try the Z39.50 Target Directory

If you just want to import citations, the Z39.50 search and retrieval protocol is the way to import from yor library catalog and many online databases. Indexdata has number of multiplatform tools that you can use, such as YAZ (a z39.50 client) and PHPYAZ. Three commercial packages import from Z39.50 sources nicely (Bookwhere, Procite and Endnote) both Procite and Endnot work well at managing your footnotes during workprocessing, taking care of numbering and layout (e.g. APA or Chicago Manual of Style, etc.).

If you want something under GPL and more oriented to managing web sites and other Internet resources, then you may want to try hypatia. You'll have to ask special for it, but it's available. Here are the parts I've seen so far:

• Web-based interface, both end users and maintainers.
• Fully multi-lingual, including both interface and content. (It is very easy to add another language to the interfaces. Right now English and Spanish are complete, Norwegian and Finnish are being translated.) Support for Unicode (Which means you're free to add interfaces in or ).
• Useable on many different platforms, including Linux, Unix, and Windows.
• Individual installations can exchange records, allowing federated content and service providers to work together seamlessly. (Haven't tried it yet.)
• Compatible with relevant standards, including MARC, Dublin Core, and the Networked Reference standard currently under development by NISO.
• Special features for digital collections, such as automatic URL checking.
• Authority control over names (e.g. People and Organizations).
• Uses perl/MySQL/javascript

If you just want to import citations, the Z39.50 search and retrieval protocol is the way to import from yor library catalog and many online databases. Indexdata has number of multiplatform tools that you can use, such as YAZ (a z39.50 client) and PHPYAZ. Three commercial packages import from Z39.50 sources nicely (Bookwhere, Procite and Endnote) both Procite and Endnot work well at managing your footnotes during workprocessing, taking care of numbering and layout (e.g. APA or Chicago Manual of Style, etc.).

If you want something under GPL and more oriented to managing web sites and other Internet resources, then you may want to try hypatia. You'll have to ask special for it, but it's available. Here are the parts I've seen so far:

• Web-based interface, both end users and maintainers.
• Fully multi-lingual, including both interface and content. (It is very easy to add another language to the interfaces. Right now English and Spanish are complete, Norwegian and Finnish are being translated.) Support for Unicode (Which means you're free to add interfaces in or ).
• Useable on many different platforms, including Linux, Unix, and Windows.
• Individual installations can exchange records, allowing federated content and service providers to work together seamlessly. (Haven't tried it yet.)
• Compatible with relevant standards, including MARC, Dublin Core, and the Networked Reference standard currently under development by NISO.
• Special features for digital collections, such as automatic URL checking.
• Authority control over names (e.g. People and Organizations).
• Uses perl/MySQL/javascript

- Zebra information server. Eats Marc (UsMarc, other local variants) as well as XML, mails, newsgroups, etc. You can add more input filters. Talks Z39.50
- Yaz Z39.50 toolkit for client and server side
- Zap web gateway and a PHP module for building easy search gateways to anything that understands Z39.50, for example our own Zebra
- and more. Even more to come later...

I am of course biased, but these tools are designed for library applications. All open source, at Index Data.