Slashdot Mirror
Lucene in Action
Simon P. Chappell writes "I don't know about you, but I hardly bother with browser bookmarks any more. I used to have so many bookmarks, back in the early days of Netscape's 4 series, that I would have to regularly trim and edit my bookmark file to prevent my browser from crashing on startup -- that's a lot of bookmarks, folks! Now, I go to my favourite web search engine, enter a couple of appropriate search terms and voila, there's my page! Search engines are so ubiquitous that we rarely give much thought to the technology that powers them. Lucene in Action by Otis Gospodnetic and Erik Hatcher , both committers on the Lucene project, goes behind the HTML and takes you on a guided tour of Lucene, one of a generation of powerful Free and Open-Source search engines now available." Read on for the rest of Chappell's review.
Lucene in Action
author
Gospodnetic and Hatcher
pages
421 (7 pages of index)
publisher
Manning
rating
9
reviewer
Simon P. Chappell
ISBN
1932394281
summary
Solid introduction to Lucene
Who's it for?
Lucene is a library and framework, rather than a complete application. It truly is an engine, around which you are expected to build and extend your own application. Like Lucene, the book is targeted at those who are looking for a tool to build their own search facility application rather than just "download and go." The book does include a number of case studies of Lucene usage (including at least one download and go search engine) but those are included to show how to use and adapt Lucene to fit differing environments rather than as ends in themselves.
The Structure
The book is sensibly divided into two parts. The first part looks at "Core Lucene" functionality, while the second part addresses "Applied Lucene".
Part one has six chapters, covering the central components and inner workings of Lucene. It's here that the book starts with a tutorial introduction, familiarising the reader with the concepts of Lucene as a search engine around which you wrap your own code. The other five chapters move steadily through good search engine fare, with indexing getting the whole of chapter two to itself The discussion of how to retrieve text from the documents being indexed is mentioned here but postponed until chapter seven, where it is dealt with exhaustively. Chapter three covers searching, and especially how Lucene ranks documents.
Chapter four examines analysis. In it's chapter introduction, the book explains that "Analysis, in Lucene, is the process of converting field text into it's most fundamental indexed representation, terms." This process is performed by an analyser, which tokenises text according to it's own built in rules; each analyser will have a different emphasis, some want only dictionary words, others might explicitly include acronyms and sometimes you'll want an analyser that will block stop words (those words in languages that are part of the structure, but that add nothing to the information being conveyed by the text; classic examples of stop words in English include "a", "and" and "the").
Chapter five looks at advanced search techniques; everything from sorting search results, searching on multiple fields to filtering searches. Many free or open source software tools are extensible, and Lucene is no exception. Chapter six addresses creating and using custom components within Lucene, everything from custom sort methods to custom filters.
Part two, the final four chapters, cover Applied Lucene. It is dedicated to practical uses of Lucene and answers the question "So, what can I do with a search engine?" Chapter seven covers ways and means to parse common, non-plain text document formats. The primary formats covered are RTF, XML, PDF, HTML and Microsoft Word. The ability to parse and index these file formats will cover the search engine needs of the majority of Lucene users. Chapter eight looks at a number of Lucene tools and extensions that are available; many of them being free and open source software. Chapter nine covers ports of Lucene. While for many users, Lucene being a Java library is not a problem, some users want its functionality in environments that do not have Java. The chapter looks at ports written in C++, C#, Perl and Python. Lastly, chapter ten takes a thorough look at seven Lucene case studies. Perhaps the "star" case study is the one about Nutch, a download and go search engine written by Doug Cutting , the original author of Lucene.
There are three appendices. The first offers installation advice for Lucene; a useful addition that those newer to working with Java libraries will surely appreciate. The second appendix has a very well explained description of the Lucene index format. This is the kind of information that can be hard to find, so it is welcome in a book of this sort. The last appendix contains a number of categorised resource references. The number and breadth of the resources provided could provide quite an incredible education in information retrieval theory if the reader was inclined to read them all. What's to Like? There are several things to like about this book. Let's start with the fact that the authors are part of the core development team of Lucene. This gives them both credibility and an excellent understanding of the internal workings of Lucene. Co-author Erik Hatcher is a fantastic writer, having previously been a co-author of the only Ant book worth bothering with, Manning's Java Development with Ant . (Full disclosure: I do know Erik personally.)
The structure of the book is well thought out and each chapter does seem to move your understanding forward when combined with what you learned from the proceeding ones. The division into core and applied Lucene is also helpful. While you'd hope that this was the case, it often isn't; hence I note it as a positive.
I especially appreciate that this book does not fill up page after page with API documentation. The authors appear to have grasped that if you have Internet access to download the software, you might just be able to access the documentation online; rather, they concentrate on the way to use the software. What a concept!
As a part of Manning's "in Action" series, the book has excellent layout and has obviously been thoroughly edited by both technical evaluators and copyeditors. This might seem to be a small thing to some, but a well-edited book stands out clearly from the crowd. What's to consider? If you are looking for a book on using and configuring a download and go style of search engine, this book would be less suitable. While the case study on Nutch is of good length, it would be too short to useful as a configuration guide. Conclusion I enjoyed reading this book. If you have any text searching needs, this book will be more than sufficient equipment to guide you to successful completion. Even, if you are just looking to download a pre-written search engine, then this book will provide a good background to the nature of information retrieval in general and text indexing and searching specifically.
You can purchase Lucene in Action from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Part one has six chapters, covering the central components and inner workings of Lucene. It's here that the book starts with a tutorial introduction, familiarising the reader with the concepts of Lucene as a search engine around which you wrap your own code. The other five chapters move steadily through good search engine fare, with indexing getting the whole of chapter two to itself The discussion of how to retrieve text from the documents being indexed is mentioned here but postponed until chapter seven, where it is dealt with exhaustively. Chapter three covers searching, and especially how Lucene ranks documents.
Chapter four examines analysis. In it's chapter introduction, the book explains that "Analysis, in Lucene, is the process of converting field text into it's most fundamental indexed representation, terms." This process is performed by an analyser, which tokenises text according to it's own built in rules; each analyser will have a different emphasis, some want only dictionary words, others might explicitly include acronyms and sometimes you'll want an analyser that will block stop words (those words in languages that are part of the structure, but that add nothing to the information being conveyed by the text; classic examples of stop words in English include "a", "and" and "the").
Chapter five looks at advanced search techniques; everything from sorting search results, searching on multiple fields to filtering searches. Many free or open source software tools are extensible, and Lucene is no exception. Chapter six addresses creating and using custom components within Lucene, everything from custom sort methods to custom filters.
Part two, the final four chapters, cover Applied Lucene. It is dedicated to practical uses of Lucene and answers the question "So, what can I do with a search engine?" Chapter seven covers ways and means to parse common, non-plain text document formats. The primary formats covered are RTF, XML, PDF, HTML and Microsoft Word. The ability to parse and index these file formats will cover the search engine needs of the majority of Lucene users. Chapter eight looks at a number of Lucene tools and extensions that are available; many of them being free and open source software. Chapter nine covers ports of Lucene. While for many users, Lucene being a Java library is not a problem, some users want its functionality in environments that do not have Java. The chapter looks at ports written in C++, C#, Perl and Python. Lastly, chapter ten takes a thorough look at seven Lucene case studies. Perhaps the "star" case study is the one about Nutch, a download and go search engine written by Doug Cutting , the original author of Lucene.
There are three appendices. The first offers installation advice for Lucene; a useful addition that those newer to working with Java libraries will surely appreciate. The second appendix has a very well explained description of the Lucene index format. This is the kind of information that can be hard to find, so it is welcome in a book of this sort. The last appendix contains a number of categorised resource references. The number and breadth of the resources provided could provide quite an incredible education in information retrieval theory if the reader was inclined to read them all. What's to Like? There are several things to like about this book. Let's start with the fact that the authors are part of the core development team of Lucene. This gives them both credibility and an excellent understanding of the internal workings of Lucene. Co-author Erik Hatcher is a fantastic writer, having previously been a co-author of the only Ant book worth bothering with, Manning's Java Development with Ant . (Full disclosure: I do know Erik personally.)
The structure of the book is well thought out and each chapter does seem to move your understanding forward when combined with what you learned from the proceeding ones. The division into core and applied Lucene is also helpful. While you'd hope that this was the case, it often isn't; hence I note it as a positive.
I especially appreciate that this book does not fill up page after page with API documentation. The authors appear to have grasped that if you have Internet access to download the software, you might just be able to access the documentation online; rather, they concentrate on the way to use the software. What a concept!
As a part of Manning's "in Action" series, the book has excellent layout and has obviously been thoroughly edited by both technical evaluators and copyeditors. This might seem to be a small thing to some, but a well-edited book stands out clearly from the crowd. What's to consider? If you are looking for a book on using and configuring a download and go style of search engine, this book would be less suitable. While the case study on Nutch is of good length, it would be too short to useful as a configuration guide. Conclusion I enjoyed reading this book. If you have any text searching needs, this book will be more than sufficient equipment to guide you to successful completion. Even, if you are just looking to download a pre-written search engine, then this book will provide a good background to the nature of information retrieval in general and text indexing and searching specifically.
You can purchase Lucene in Action from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Raise your hand if you have jettisoned all browser bookmarks and just use http://del.icio.us/ (and/or the wonderful bookmarklets or Firefox plugin for it).
bp
My home page is a nicely sorted webpage with all my frequently visited links in a password protected section of my web site. If something gets used enough in my bookmarks, it gets put on that page and gets deleted from my bookmarks. Then, no mater where I am or what computer I am on, I can access my links.
Thanks! I was looking for a good book on Open Source search engines. While I have never heard of "Lucene" I will definitely be looking into it now. Its probably a good opportunity to learn all about Search Engine Heuristics, methods, etc...
Also, I agree with the author that bookmark functionality has gone the way of the dinosuars... with the exception of the "open all tabs" feature found in many browsers today... that is about the only one that I use often.
Im just wondering how the "search" functionality will actually play out in the future. Apple has "Spotlight" and Microsoft is supposedly incorperating magick folders or something like that into Vista. Can anyone tell me more about Lucene and how it differs from say Google or other search engines?
Thanks,
keilinw.
Bookmarks are more secure than a search engine, since a search engine could provide a poisoned link, and if you're typing in the URL by hand, if you make a spelling mistake, you could find yourself at a pharming site, or someplace you didn't want to go.
I tend to use bookmarks in Firefox and the autocomplete about equally, and make use of the Quick Links toolbar for my most popular sites.
The Firefox bookmark all tabs feature is a breakthrough, since you can close your browser, and reopen it to the same set of tabs as before, which is great when installing extensions and you're forced to restart. The only drawback is that scrolling through bookmarks is too slow, but if you use your scroll wheel it speeds up considerably. That's a trick I didn't figure out until just last month.
Saskboy's blog is good. 9 out of 10 dentists agree.
Now, I go to my favourite web search engine, enter a couple of appropriate search terms and voila, there's my page!
You have a better memory than I my friend. Many times I only barely remember something I want to find again. Maybe I remember it was humourous, or maybe I remember it was an online game with pigs in it. Unless it's popular I doubt 'pig game' is gonna get me far. So bookmarks aren't so useless to those of us who don't keep everything in RAM.
Bookmarks, and a good hierarchy, also leverage the Associative aspect of our minds. Skim through your high level bookmark folders and you'll probably find what you were thinking of pretty quick. Additionally it reminds you of things you may have bookmarked yet forgotten.
"Not knowing when the dawn will come, I open every door." - Emily Dickinson
I might reference a search engine to tell someone how to find a site via word of mouth, but it has not replaced my bookmarks. If I am away from my system and all I can remember some common words maybe, but not so long ago I used to sync bookmarks with a firefox plugin. (Kept breaking after version updates, never went to reinstall it... though I think I will now)
Anyhow, I simply build to critical mass before I sort them into their respective folders. Some things are automatically tossed into temporary bookmark folders that are going to get washed away after they are no longer useful. (Think auction links)
Now I'll tell you why using a search engine as replacement bookmark concept is a bad idea. Page ranking changes. That particular combination of words you can remember... might just not produce the same results next time. Wonder why? The interenet changes! It is not Aol keyword search...
That said, I did something as foolish as to rely on google to get back to some website regarding video sync signals. It was an excellent page and then I went back to search for it again and I could not find it. (Eventually I did though)
Bookmarks good, search engine good... not mutually exclusive.
"You should always go to other people's funerals; otherwise, they won't come to yours." -- Yogi Berra
While search engines are great, bookmarks are not obsolete. I use RSS feeds to keep up on anything that is serialized that I might care about. FF is great for that.
I still use a few regular bookmarks (like the URL that logs me into /.). Or for development servers with obscene URLs. That is the kind of thing that a search engine won't find. Especially if you have to deploy to a few web servers (this is the WebLogic machine, this is the OAS machine, etc). I have even bookmarked LDAP strings for testing.
More to the point of TFR, I would be intereseted in learning more about OSS search engines. It would be great to set one up on my own net... hmmm. As an aside, can Lucene be used for local searches? It would be cool to make my own desktop search. What kind of licensing does it have?
the Lucene (http://jakarta.apache.org/lucene) indexer will be inplememtned within Hula the web and cal application (http://hula-project.org/Hula_Server) made from open sourced Novell NetMail code. Samples of the search engine have been comitted and should start functioning within weeks, just in time for the new cal UI, which you can now view a demo of here: http://nat.org/2005/august/hula.html That's looking to be an amazing app...
bad_outlook
--
Is this vague enough for you?
Does anyone find it a little funny that on the main lucene.apache.com webpage, there is a "Search this site with Google" textbox ? Kind of makes you NOT want to use their search engine if they dont' even trust it enough to work on their own site....
A million monkeys and this is the best sig they could come up with...
Typical search engines have licensing fees associated with them if you're embedding them in your application. This is basically an open source alternative. And you can customize the hell out of it. I've used it on several web-based applications and on SOA platforms, and it is fast, reliable, and easy to use. Did I mention it's open source? Take a look at the Apache site.
Some examples of customizable features are that you can index database entries and achieve quantum leaps in performance over that offered by Oracle, MySQL, PostGres, Firebird, etc. indexing. You can index formats that are not supported by the major search enginges.
It may not offer quite the performance of Google, Alta Vista, etc., but it's a FREE product, well supported by the folks at Apache, and many open source J2EE frameworks support it as well.
Check out this article for a good intro to Plucene, the Perl port of Lucene.
This is also a good link for all of you slashdotters who have no idea what Lucene is for and are posting rants wondering why people don't just use Google instead.
include $sig;
1;
Check out regain: http://regain.sourceforge.net/index.php Works as a computer indexer using Lucene.. Seems to do better on search than MS stuff :)
On a typical 2.97 GHZ system with 100 gig hd 70% full is about 6 hours to do first index..
It runs fine in backgroudn no noticeable slowing down of other apps while indexing..
It also come sin a server vesion as well for website searches to build search engines like google and yahoo :)
Fred Grott(aka shareme) http://mobilebytes.wordpress.com
Usual disclaimer: I have nothing to do with Sitebar or its development, just a majorly satisfied user.
"Powers. I have them."
Why not try delicious? It allows you to keep your bookmarks online so that they're accessible from multiple locations, while also allowing you to search your bookmarks and those belonging to other people.
If you use Firefox, there are extensions that allow you to view your bookmarks in a sidebar and sync your online bookmarks with your browser bookmarks.
Lucene is not like google (it's not a full application), it's a library focused on searching text based documents (you could use it to build a mini-google).
The basic idea is that you want to build an index, and then search it, to find some document.
A document has several fields (e.g. text, title, lastModificationDate, author, categories, summary, url, etc.) which may be indexed, stored, or both.
You usually build your lucene documents, based on some real documents (e.g. web pages, PDF, records in a database, etc.), and then add them to the index.
Once you have an index, you build a query to search one or more fields (lucene provides a QueryParser class, which handles the most common cases), and you get a Hits collection containing the documents matching your query in some order (this can be customized).
Before a document is added to the index, it is passed through an Analyzer which converts the text in the fields to terms, which are the basic internal concept that is indexed.
Another interesting feature of lucene indexes is that they can be searched while they are being built without noticeable loss of search performance, and that they are process-safe (many processes can access them for reading, only one for writing), this has the drawback that the indexes are append-only (actually a separate index is created if you modify an index), but periodical optimization of the indexes removes unnecessary entries and inefficiencies.
Hope this helps!
juancn
Scroll wheel? Thanks, that is a major helper.
What I've begun doing is using the "Bookmarks Toolbar Folder" for all of my bookmarks. I've got "Essentials" with links to Gmail, Adsense, my website, Distributed.net stats and so forth, basically all of the sites that I try to visit daily. Then I've got "Favorite sites" that holds Slashdot (even though now it's "home"), Woot, Craigslist, Free6.com (hehe), Assambassador.com, Myspleen, demonoid, you get the point.
Then I've got the essential one: "Functions" - that holds mostly Javascript links but other things like TinyURL, @nonymous, Wordpress Press-It, BlogThis!, post to del.icio.us, Ping-o-matic, Send SMS message, mailto: and whatever. Then there is "Junk" which isn't really used any more because del.icio.us is so sweet. I generally dump something I might want to read later there and categorize it later (like every 8 months). Then of course I've got a few drop-down RSS feeds, but since they are torrent sites I'll keep them to myself. (Oh, I almost forgot - a huge drop down of del.icio.us bookmarks with the help of Foxylicious)
This works well, and generally reminds me of a filing system. Since I'm never using the File, Edit, and etc menus this has become my new menu.
Get your Unix fortune now!
Also, I wrote a DevX article on Lucene:
:-) Just kidding.
http://www.devx.com/Java/Article/27728/0
Lucene is so well documented and simple to use that I am surprised that this subject would fill an entire book
Lucene can be used as is, or you can extend it with your own document type handlers, etc.
As a programmer, I way prefer dynamic languages like Common Lisp, Ruby, Python, Smalltalk, etc. However, one of the things that keeps me firmly in the "Java camp" is the great free infrastructure software tools (like Lucene, Tomcat, JBoss, etc.) As a programming language, Java is kind-of weak.
Lucene is a pretty amazing piece of software. Lucene is to text indexing what postgres is to relational databases. The API is simple, and though many people have reservations about java it is very, very fast. I've written lucene apps that could perform queries in 40-60ms that would take a relational database up to 20 minutes to perform on the same hardware. I've found it to be even a few orders of magnitude faster than Oracle text indexing.
And you can index pretty much anything you want, so long as you can get it into a string. Everyone knows you can index documents in XML, HTML, etc, but you can also index objects with strings, integers and dates, hash tables and lists. Just put a primary key value and a table name and you can retrieve the full object from your database. Very cool indeed.
Every programmer out there with skills in relational databases should take a look at a information retrieval library like lucene. It's a completely different field from relational databases and it will change how you think about storing and searching data. It's not a replacement for relational databases, but it does complement them very well and allows you to do things that you wouldn't normally be able to do, like allowing for results that partially match the search query, and being able to rank them by relevance.
You can combine the best of both worlds, bookmarks and search.
I find bookmarks slow to navigate, and it's hard for me to remember my own hierarchy when I've got enough bookmarks to organize. The problems with search have been expanded on by others in this thread.
So here's the solution: http://del.icio.us/.
You can create, edit, tag, describe, and search your own personal bookmarks. When you've done that, the world can see your links too. Subscribing to an RSS feed of some tags you're interested in ("python" for me) gives you a constant stream of interesting links other people who are into Python found useful.
If you're using Firefox, you'll probably want del.icio.us in your search box. Find that here . I find myself frequently using this when I'd normally use Google, when I know (or just suspect) that I've been somewhere before.
What I've been doing for about a year now is keeping my actual in-browser bookmarks an unsorted flat list with just about 20 sites I visit on a regular basis. Webcomics, blogs that don't have RSS feeds already, and the like. Everything else goes into del.icio.us. All my other bookmarks were of one of these categories: links I visit only occasionally for reference, sites I intended to visit just once later when I have free time, or sites that I don't even know if I'll find useful until I go back and reread them. Now I don't have to clutter my browser with those; I throw them in del.icio.us. As a bonus, del.icio.us tells you how many other people have bookmarked the page. Number of times a link is being submitted is a good first-blush indication of whether the information there is really interesting or useful.
It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
Much better than LuPy is PyLucene which uses the actual Lucene libraries compiled with gcj and wrapped with SWIG, thus giving you Python beauty with Lucene performance...
We moved most of our searching out of SQL and into Lucene for a variety of reasons; and this book has been useful, not just in figuring out good ways of writing those queries that maintain microsecond response times (which Lucene is absolutely brilliant at), but...
- Good ways of doing batch indexing operations
- The purpose of the compound document format
- How to generate explainers for searches
- Field-specific handling, and how to do it well
- Ideas like metaphone replacement (soundex) and use of WordNet to integrate a synonym database into search queries
- When to use CachedFilters to remember complex filters
- Ideas for how to build "Things Like This" lists
- Ideas on autocategorisation and geographic searches
- Named Entities and LingPipe - making the search system recognize "proper names" for things
- NGRAM recording to gauge word frequency and search terms to detect misspellings and offer alternate searches ("Did you mean XXX"?)
etc.
If you're building a search engine, this isn't just a useful resource on implementation - you probably don't need a book for that. What it is brilliant at is providing a lot of ideas that can take you to the next stage - how to build something really cool with your information, and not just a dumb text field search.
For that alone, the book was worth the purchase price, for me. It's now well annotated, and the back pages are full of references to ideas that can be used in our own implementation, and the page numbers to use to get there.
Highly recommended for anyone who needs something more than what a Google search of your site provides.
-- A mind is a terrible thing.