Slashdot Mirror

http://wiki.greenstone.org/wiki/index.php/Greenstone_FAQ

Sick bastards, first time I have ever seen that. Very unprofessional and a major strike against their credibility right off the bat.

I doubt I'll look any further into the product now.

How cruel is that? "Um, Ya, here's our FAQs... oh you want answers too!? We only have Frequently Asked Questions - no Frequently Answered Questions here!.

And the irony of using a wiki to deliver such a cruel joke is not lost on me.

The indexing part of your question can be well handled by Greenstone. While notoriously difficult to configure, it is capable of indexing and presenting a lot of information in the form of a digital library. Quite a mature project, I used it for my bachelor thesis back at the University of the South Pacific in Suva, Fiji. All of the OCR still has to be done externally though.

Is there a package that can read the mbox, the other box-formats, plain text, pull from pop, old tar.gz bundles, categorize (sorta), tag and make such things searchable?

Yes there is, check out

http://www.greenstone.org/cgi-bin/library

Try Greenstone
- unix/os x and win32, GPL'd, indexes quite a few filetypes.

Disclaimer: I've been a developer for this project

you could try Greenstone which does full text indexing of lots of different file formats, including email:

greenstone.org.

It's all covered by GPL or gpl-compatible licences.

Disclaimer: I am affiliated with them.

Great to see a project like this run on Free software. Read more at Greenstone's website.

A different approach, but highly adaptable.

Greenstone.

I use Greenstone (www.greenstone.org) but that not be what you are asking for.

My suggestion would be to 'clean' all your documents so they convert well with whatever converter you choose to use. It will probably have to be done by hand; a macro may help with some legwork.

Then, convert to html with which ever utility works best for your document set/version of ms-word (try the inbuilt one in addition to wvware etc.).

Why HTML? SGML/XML markups can record the document features you have asked for. HTML as a SGML Document type (and now an XML one too) has the Document structure features you requested.
If you need more sophisticated markup: Try TEI (or TEI-LITE) at www.tei-c.org or Docbook; it depends what your documents are and what you need.

If you really need plain text, then maybe you could work out some way of making use of the plain text web browser LYNX (http://lynx.isc.org/).
It will render the document features you require.
See [http://lynx.isc.org/current/lynx2-8-5/lynx_help/L ynx_users_guide.html#Tables] to see how lynx renders tables. 'Use the source, Luke' may also be good advice.

Greenstone is a digital library software designed for this sort of thing. Can manage massive amounts of text, build indices and thumbnails, etc. www.greenstone.org

See www.greenstone.org

Greenstone is a suite of software for building and distributing digital
library collections. It provides a new way of organizing information and
publishing it on the Internet or on CD-ROM. Greenstone is produced by
the New Zealand Digital Library Project at the University of
Waikato, and developed and distributed in cooperation with UNESCO
and the Human Info NGO. It is open-source software, issued under the
terms of the GNU General Public License.

Includes the document:

From Paper to Collection (224kb)
A document describing the entire process of creating a digital library
collection from paper documents. This includes the scanning and OCR
process and the use of the "Organizer".