A Grep-like Utility That Works on More than Text?

Nutria writes "This article got me thinking: What's a poor Unix-using guy to do, when he needs to grep text, compressed tarballs, OO.o documents, Debian archives, mime-encoded files, Evil Microsoft documents, PDF files, compressed AbiWord files, etc." Is there an extensible searching program for Unix that can handle a variety of different file-types? Search engines like ht://Dig can accomplish part of this task, however currently it doesn't index the whole file (just portions of the metadata). If you had to perform a substring search on a set of documents of different types, what tools would you use to accomplish this task?

Comments

Beagle

[Markusis]

Beagle will probably meet at least some of these goals. Beagle also aims to index things like your gaim-logs, browsing history, and email as well as your files. This is closely related to the dashboard project. Both projects can be retrieved from Gnome CVS.

http://www.nat.org/beagle/
http://www.nat.org/dashboard/

Are you sure you're a "unix guy"?

[ivan256]

What's a poor Unix-using guy to do, when he needs to grep text, compressed tarballs, OO.o documents, Debian archives, mime-encoded files, Evil Microsoft documents, PDF files, compressed AbiWord files, etc.

Um, why not pipe the output of your favorite program that interacts with the file type you're interested in to grep? Isn't that the "poor unix guy" way?

catdoc Blah.doc | grep foo
zcat compressed.txt.gz | grep foo
apt-cache show package | grep foo
pdftotext Blah.pdf | grep foo

etc...

How, indeed!

[Otter]

The same way everything else works!

1) Be had a half-decent version years ago.

2) Apple will have a reasonably robust version out soon.

3) Microsoft will have a more frustrating knock-off of Apple's version a few years later.

4) Four competing, incompatible open-source projects will copy the Apple and Microsoft implementations. When one of those companies sends a cease and desist letter to an open-source project that has shamelessly ripped off its trademarked name, Linux zealots will complain about how "intellectual property laws stifle our innovation"!

Use a pipe and untilities

[Matt Perry]

The unix shell and pipe are your friends:

grep text

grep text filename

compressed tarballs

tar zOxf filename | grep text

OO.o document

unzip -p filename | grep text

mime-encoded files

mimedecode filename | grep text

Evil Microsoft documents

strings filename | grep text

PDF files

strings filename | grep text

This would make a great shell script project. You could use file to detect the type and then filter and grep it appropriately. This sounds useful enough that I'll probably write this script this weekend. Thanks for the idea.

Re:Use a pipe and untilities

[robbkidd]

PDF files[?]
strings filename | grep text

I'm guessing you've never tried that search before. PDF stores the meat of a document in compressed data streams. strings would return a bunch of font names, headers and compressed garbage.

There are a few other tools available, at various stages of stability:

• PDFtoHTML
• PDFsearch
• CPAN PDF modules

Re:Use a pipe and untilities

[bunnyman]

less(1) already does this! Check out the $LESSOPEN variable on your Linux system, it points to a shell script that detects what type of file you are viewing, and runs a filter on it to get plain text from it.

Full text search engine?

[bentfork]

I like the Catfish namazu.org <-- google cache ( thats catfish in japanese ).

It has multiple 'filters' that allow you to index .txt .pdf .doc and various other file formats. It also has some grep like command line utilities that allows you to 'grep' for text.

Oh, and it has a perl front end for apache, and IIS. ( win32 installer available from here: www.namazu.org/windows

caveat: As with any full text search engine, the files should be reindexed on a regular basis, but this is worth it as the searches are very fast even with gigs of documents.

enjoy

Re:Beagle

[HRbnjR]

Seth Nickell has a blog entry discussing solutions in this area, including Gnome Storage, WinFS, Dashboard, Medusa, Spotlight, and Beagle.

Penetrator

[cbcbcb]

I've just implemented a Word/PDF/text search system using Penetrator.

It acts as a web search engine or a grep-like tool. You can either build an index or configure it to search on demand.

You can add filters to deal with any file type you like. I use xpdf and wvWare to handle PDFs and word documents.

A little bit of work to set up, but it's a nifty bit bit of software.

Half a solution ....

[gstoddart]

Um, why not pipe the output of your favorite program that interacts with the file type you're interested in to grep? Isn't that the "poor unix guy" way?

Because it involves the user explicitly using the right utility to decode each file type for every single file.

So you don't get the nice simple case the user is looking for like:

grep foo *

or even something cool like

find . -name "$pattern" -exec egrep -l -e "$1" {} \;

Grep is often used to hunt through large amounts of docs because you know something has that text, but you can't remember what.

What the guy is asking for is something that would take my second example, figure out the file type, and then do it automatically for you.

Cheers

Haven't tried it, but SWISH sounds good

[chjones]

Haven't actually tried it out (everything I write seems to be text or TeX), but I remembered reading this article a while back: "How to Index Anything", Linux J., July 1 2003.

Re:Haven't tried it, but SWISH sounds good

[bobv-pillars-net]

Actually, the last time I needed to index a large group of files (for disaster recovery purposes; the files in question were all from the lost+found directory) I used Swish++.

And yes, it works VERY well, orders of magnitude faster than Ht:/Dig, features incremental reindexing, and can be configured to auto-convert various filetypes to text before indexing. I'd say it's exactly what the poster ordered.