A Grep-like Utility That Works on More than Text?

Nutria writes "This article got me thinking: What's a poor Unix-using guy to do, when he needs to grep text, compressed tarballs, OO.o documents, Debian archives, mime-encoded files, Evil Microsoft documents, PDF files, compressed AbiWord files, etc." Is there an extensible searching program for Unix that can handle a variety of different file-types? Search engines like ht://Dig can accomplish part of this task, however currently it doesn't index the whole file (just portions of the metadata). If you had to perform a substring search on a set of documents of different types, what tools would you use to accomplish this task?

Comments

ever herad of 'strings'?

[szyzyg]

strings filename | grep

Re:ever herad of 'strings'?

[TheWanderingHermit]

But that doesn't pull out compressed strings -- only recognizable ascii strings.

Re:ever herad of 'strings'?

[JabberWokky]

Realistically, this should operate similar to text filters in less. I.e., a set of filters that convert files to text for grep'ing.

A more generic solution would be to have less stop using those filters and instead use strings, which would identify the file type and then subcall strings.text.plain or strings.text.html or otherwise the strings.mime.type. The fallback would be the normal strings behaviour. That way, anything that expected text (a la grep, less, etc) can have a flag to filter through strings to get the 'clean' feed:

grep -riw --strings "gold" *

Or just pipe it through:

strings mytext.html |grep -iw "gold"

That latter case could be handled by a strings.text.html that consists of a simple script calling lynx --dump.

--
Evan

Re:ever herad of 'strings'?

[Black Acid]

This also won't work with MS Word documents because they store text out of order.

Re:ever herad of 'strings'?

[Anonymous Coward]

Yep, the person who posted this one knows little of UNIX.

strings, od, awk, sed, pr, grep, egrep, etc.... give a very rich set of flexable utilities this poster knows nothing of.

I think they miss the whole point of UNIX... so something real well and pipe it.

Got to be better than XP's puppy dog, but

[Marxist Hacker 42]

the problem I see isn't searching compressed or tarballed files- where the text inside is still largely in plaintext. It's just a different file format- but it's still bytes and I've seen supergrep programs in the past for both Linux and Windows that do this (try Tucows or SourceForge before posting on slashdot in search of freeware and shareware.) The problem I see is searching *encrypted* files, esepcially ones with different keys. Now that would be *hard*.

file...

[runswithd6s]

file+[GLUE] and optionally, grep. GNU File is a nice utility to determine file types. From there, you could use some sort of glue language, such as Python or Perl, to perform the necessary actions when certain filetypes are found: decompress, index, etc. I'd be surprised if there wasn't a plugin style grep utility already out there.

Beagle

[heikkih]

Beagle

Re:Beagle

[HRbnjR]

Seth Nickell has a blog entry discussing solutions in this area, including Gnome Storage, WinFS, Dashboard, Medusa, Spotlight, and Beagle.

ht://Dig doesn't do this?

[dougmc]

Search engines like ht://Dig can accomplish part of this task, however currently it doesn't index the whole file (just portions of the metadata).

Um, ht://Dig WILL do it. As will any of the other search engine packages out there, if you can feed them the text that they work on. It's generally not difficult (on a *nix platform) to convert all of the formats you mentioned into straight text or html and feed those to your favorite search engine platforms.

Useful programs include wvWare, rtf2htm, pdftotext, and yes, even strings.

Beagle

[Markusis]

Beagle will probably meet at least some of these goals. Beagle also aims to index things like your gaim-logs, browsing history, and email as well as your files. This is closely related to the dashboard project. Both projects can be retrieved from Gnome CVS.

http://www.nat.org/beagle/
http://www.nat.org/dashboard/

Are you sure you're a "unix guy"?

[ivan256]

What's a poor Unix-using guy to do, when he needs to grep text, compressed tarballs, OO.o documents, Debian archives, mime-encoded files, Evil Microsoft documents, PDF files, compressed AbiWord files, etc.

Um, why not pipe the output of your favorite program that interacts with the file type you're interested in to grep? Isn't that the "poor unix guy" way?

catdoc Blah.doc | grep foo
zcat compressed.txt.gz | grep foo
apt-cache show package | grep foo
pdftotext Blah.pdf | grep foo

etc...

How, indeed!

[Otter]

The same way everything else works!

1) Be had a half-decent version years ago.

2) Apple will have a reasonably robust version out soon.

3) Microsoft will have a more frustrating knock-off of Apple's version a few years later.

4) Four competing, incompatible open-source projects will copy the Apple and Microsoft implementations. When one of those companies sends a cease and desist letter to an open-source project that has shamelessly ripped off its trademarked name, Linux zealots will complain about how "intellectual property laws stifle our innovation"!

Re:How, indeed!

[Carnildo]

Funny? Seems more "insightful" to me!

1) grep, ht://dig, find, etc
2) Apple's new search utility, due out with 10.4
3) WinFS
4) ????

Re:How, indeed!

[afidel]

Yeah except MS has had this in FindFast from Office 97 on and the Microsoft Indexing Service since Windows 2000 came out. It doesn't work terribly well out of the box (idle detection sucks) but if you put a little effort into it Indexing Service really rocks, especially in an AD environment where you can perform a search across servers.

Re:How, indeed!

[Anonymous Coward]

Stop using the string "linux zealots". That combination is my intellectual property. Seriously, intellectual property is Just Stupid. Why the FUCK should anyone have any right to stop me making a search engine for me and my friends files just because Be might have already done it?

Re:How, indeed!

[Anonymous Coward]

I think you'll find that grep probably predates the incorporation of Microsoft.

Throw in NFS, which predates Win3.0, and I'm not that impressed by what Microsoft offered in 97.

Re:How, indeed!

[AndyElf]

Parent's (1) had nothing to do with grep, ht://dig or find: it was more what will be coming in (2).

As for (3), putting an SQL Server onto each desktop to store all the meta-data about all the files was a wrong idea to start with (not learning from Be's lessons are we? Nothing to do with computers becoming more powerful then what they were in the age of original Be). A much better approach would be more similar to ECCO's organizer system, RDF, FOAF, what Chandler will (hopefully) one day become, etc. -- i.e. it must be flexible to accomodate any number of attributes, even the ones that might not be known at design time.

Lastly, searching for *content* will most likely have to be different from searching meta data. Here some sort of an FTS system that'll index your data in the background is required. And it would have to be intelegent enought o handle a variety of formats (any format if a plugin is available). There are quite a few of those that are available (for Win) even now -- check out Copernic, Lookout, etc. As for Lin/Un -- I guess that ht://dig is one's friend, but I never liked it much...

Re:How, indeed!

[arose]

4) Gnome Storage

Convert the files

[repvik]

I'm sure there are utilities to convert most of the formats to text. MS Word to text? No problems, there's an utility to handle that. PDF? Of course.
Basically, what you might do is something like this:

1. Figure out what kind of file it is (using 'file')
2. Select the correct converter based on file
3. grep
4. Profit? -- not quite sure about this one.

Use a pipe and untilities

[Matt Perry]

The unix shell and pipe are your friends:

grep text

grep text filename

compressed tarballs

tar zOxf filename | grep text

OO.o document

unzip -p filename | grep text

mime-encoded files

mimedecode filename | grep text

Evil Microsoft documents

strings filename | grep text

PDF files

strings filename | grep text

This would make a great shell script project. You could use file to detect the type and then filter and grep it appropriately. This sounds useful enough that I'll probably write this script this weekend. Thanks for the idea.

Re:Use a pipe and untilities

[Cyrus]

Why not just use zgrep?

Re:Use a pipe and untilities

[Matt Perry]

Oh you could. I just wanted to illustrate using the pipe and other commands with grep to achieve what the poster wanted.

Re:Use a pipe and untilities

[robbkidd]

PDF files[?]
strings filename | grep text

I'm guessing you've never tried that search before. PDF stores the meat of a document in compressed data streams. strings would return a bunch of font names, headers and compressed garbage.

There are a few other tools available, at various stages of stability:

• PDFtoHTML
• PDFsearch
• CPAN PDF modules

Re:Use a pipe and untilities

[alexandre]

there is also wordcat that displays only a txt version of the word document :)
(there is also yet another program but i forgot the name)

Re:Use a pipe and untilities

[bunnyman]

less(1) already does this! Check out the $LESSOPEN variable on your Linux system, it points to a shell script that detects what type of file you are viewing, and runs a filter on it to get plain text from it.

Re:Use a pipe and untilities

[Clover_Kicker]

catdoc.

Re:Use a pipe and untilities

[Spoing]

1. less(1) already does this! Check out the $LESSOPEN variable on your Linux system, it points to a shell script that detects what type of file you are viewing, and runs a filter on it to get plain text from it.

*blink*

$ less firefox-i686-linux-gtk2+xft.tar.gz

drwxrwxrwx cltbld/cltbld 0 2004-08-24 11:26:07 firefox/
-rwxr-xr-x cltbld/cltbld 30869 1999-10-05 22:14:51 firefox/LICENSE
-rwxr-xr-x cltbld/cltbld 177 2004-08-09 16:01:23 firefox/README.txt
drwxrwxrwx cltbld/cltbld 0 2004-08-24 11:26:07 firefox/chrome/
drwxrwxr-x cltbld/cltbld 0 2004-08-24 10:25:42 firefox/chrome/comm/

(Spoing does a happy dance.)

OK, here's one for you... tab completion in Bash for commands.

An example;

$ ls 1 [tab] [tab]

1file.txt 110012.tar.gz

$ gunzip 1 [tab]
[changes to...]
$ gunzip 110012.tar.gz

It works for other commands also -- and it's programable!

Re:Use a pipe and untilities

[gnu-user]

Evil Microsoft documents

strings filename | grep text

Use antiword instead of strings.

Works pretty well (It allows me to use mutt relatively painlessly in a corporate environment).

Re:Use a pipe and untilities

[Shadwell]

xpdf comes with a utility called pdftotext that dumps the text portion to a plaintext file.

Full text search engine?

[bentfork]

I like the Catfish namazu.org <-- google cache ( thats catfish in japanese ).

It has multiple 'filters' that allow you to index .txt .pdf .doc and various other file formats. It also has some grep like command line utilities that allows you to 'grep' for text.

Oh, and it has a perl front end for apache, and IIS. ( win32 installer available from here: www.namazu.org/windows

caveat: As with any full text search engine, the files should be reindexed on a regular basis, but this is worth it as the searches are very fast even with gigs of documents.

enjoy

Re:Full text search engine?

[shufler]

Just reindex at 9 in the moring, when you've stopped computing for the night.

Re:Full text search engine?

[bentfork]

Just reindex at 9 in the moring, when you've stopped computing for the night.

hmm so true, who goes to bed before 9 anyways... modifying crontab...

# reindex at 9 in the morning
0 9 * * * /pathto/namazu/reindex-foo

Ooh, goody, a "research assistant!"

[orangesquid]

You mean a personal slave^H^H^H^H^H^H^H^H^H^H^H^H^H^Hhapless grad student?

Oh, yeah, that works...

[Inoshiro]

Because strings will do things like: uncompress ASCII text from tarballs, recognize unicode font files, work with XML document formats to ignore markup and focus on content, search within images for strings, etc, etc.

The person is asking about a true file contents indexing scheme, where you have a database of file name, meta type, and keywords culled from inside the document -- something that'd work with PDF, JPG w/ EXIF, pictures of text, OpenOffice files, XML files, HTML files, etc!

Strings does none of that.

Re:Oh, yeah, that works...

[pete-classic]

A Grep-like Utility That Works on More than Text?

The person is asking about a true file contents indexing scheme[. . .]

They added file contents indexing to grep?! Sweet!

-Peter

Reiser4

[SpaceLifeForm]

Oh, you want it to work now? Nevermind.

But, down the road, maybe ReiserFS4 will do the trick.

Re:Reiser4

[ecloud]

Yep, I agree. But an indexing/search engine still needs to be written, after they get done arguing about who can stand the silent semantic changes and who can't. And then there will be those who use other filesystems wanting the same functionality, or at least the same interface to an inferior functionality...

Penetrator

[cbcbcb]

I've just implemented a Word/PDF/text search system using Penetrator.

It acts as a web search engine or a grep-like tool. You can either build an index or configure it to search on demand.

You can add filters to deal with any file type you like. I use xpdf and wvWare to handle PDFs and word documents.

A little bit of work to set up, but it's a nifty bit bit of software.

Re:Penetrator

[chez69]

with all the hidden goatse links around here I'll let somebody else hit a link with the word Penetrator in it.

Re:Penetrator

[GrumpySimon]

This looks very cool (and isn't a goatse link!), I'm just setting it up now. Trying to get html2text to kill things like font tags.

Also - your default pdftotext setting seems to barf on files with spaces in their names. I changed the line to '%s' and this seems to work.

Cheers,
Simon

Re:Penetrator

[cbcbcb]

w3m -dump does a very good job of converting html to text.

> Also - your default pdftotext setting...

I didn't write Penetrator, it just looks identical to the program I was planning to write before I found it :)

Re:Penetrator

[Twylite]

Also worth looking at is SWISH-Enhanced. It can deal with Word .docs (with a bit of configuration), PDF, HTML and anything you can filter to text.

Re:GOOGLE

[djsmiley]

this isn't such a bad as a stupid idea.

If the user was stupid enough to share their entire drive, and allow google to index it, it could find most things in most file types.

With more support coming all the time.

Half a solution ....

[gstoddart]

Um, why not pipe the output of your favorite program that interacts with the file type you're interested in to grep? Isn't that the "poor unix guy" way?

Because it involves the user explicitly using the right utility to decode each file type for every single file.

So you don't get the nice simple case the user is looking for like:

grep foo *

or even something cool like

find . -name "$pattern" -exec egrep -l -e "$1" {} \;

Grep is often used to hunt through large amounts of docs because you know something has that text, but you can't remember what.

What the guy is asking for is something that would take my second example, figure out the file type, and then do it automatically for you.

Cheers

Re:Half a solution ....

[AuMatar]

So write a script that runs each of those programs on all grepped files, ignore those with an error return, and pipes the outputs of the rest to grep

Re:Half a solution ....

[cortana]

This is what the less package is for. Specifically lesspipe(1).

find -type f -exec lesspipe {} \; | grep whatever

Re:Half a solution ....

[Matthew Weigel]

Funny... you'd think that with such a cogent explanation of the half of the problem that grep+find doesn't solve, file(1) would leap to mind as a reasonable start to the solution of it.

It doesn't handle everything, you still need utilities for most file types to convert them to text, but it's a damned good start.

Re:Half a solution ....

[El Buho]

You could add support to your script to many file formats by using some of the many PostScript converters that already exist. Pipe their output to an application that extracts plaintext from the PostScript file , then pipe to grep.

Re:Half a solution ....

[Anonymous Coward]

grep foo * works fine, quit inventing problems. Grep will say "binary file bar matches" and then you know which file has that string, and can go ahead and open it with the appropriate program.

doodle

[tenco]

Get doodle
or simply do a
#apt-get install doodle

Haven't tried it, but SWISH sounds good

[chjones]

Haven't actually tried it out (everything I write seems to be text or TeX), but I remembered reading this article a while back: "How to Index Anything", Linux J., July 1 2003.

Re:Haven't tried it, but SWISH sounds good

[bobv-pillars-net]

Actually, the last time I needed to index a large group of files (for disaster recovery purposes; the files in question were all from the lost+found directory) I used Swish++.

And yes, it works VERY well, orders of magnitude faster than Ht:/Dig, features incremental reindexing, and can be configured to auto-convert various filetypes to text before indexing. I'd say it's exactly what the poster ordered.

Speaking of the 'puppy dog'

[MarcQuadra]

I think the fscking puppy is the WORST thing about XP. Why on earth would I click 'search' in my 'file manager' if I wasn't searching for a FILE? I despise any 'improvement' to Windows that makes me use my mouse any more than I already do. If Microsoft knew what they were doing the search would START with files, with an option to do all the other crap on the sidelines.

Apple has a really good search tool, it's simple but lets you string together conditions intuitively. KDE seems pretty decent too.

Re:Speaking of the 'puppy dog'

[no soup for you]

if you need to search, you can press F3 on almost any application or the desktop. Firefox seems to be an exception -- F3 finds text on that page.

Re:Speaking of the 'puppy dog'

[airjrdn]

I agree that the dog should be shot, but it is an easy setting to change.

Re:Speaking of the 'puppy dog'

[Marxist Hacker 42]

Which just brings up the damned dog again- I WANT F3 to do what it used to, find text in the CURRENT document on a Find First, Find Next meme.

Google search appliance

[muon1183]

Google has a search appliance that is capable of searching files on a local network. This may be more than what you're looking for, but if this is a serious enterprise application, this seems just right. It's like having your own personal google.

I did this a while ago...

[Ramses0]

Source (such as it is) is linked below. Requires Namazu. Developed on Debian / Unstable. ~75 lines of decently commented bash.

http://www.robertames.com/index.sh.txt

Call it BSD-licensed by author, and sharing back is encouraged (of course). I downloaded all my mail from yahoo account (2 wks before they upped it to 100mb) and stuffed it in a directory so I could search it better. I have to agree that namazu rocks. :^)

Major operations are "create, update" for working with the index, then "search, list, file" for searching. "Search" does the google thing, "list" will list out the filenames (like for further processing), and "file" will dump the contents of each search result (ie: for further grepping)

$ ~/bin/index.sh search something

....snip...

3. Fwd: Re: get-edid... "something special happened" :^) (score: 18)
Author: Robert Ames
Date: Wed, 5 Jun 2002 21:04:39 -0700 (PDT)
Branden Had a minor buglet that I forwarded to upstream regarding "read-edid" (probably
a hardware database out of date thing, no worries). Anyway, I mentioned to him that I
was confused at first by
/home/rames/email/test/Sent/Fwd_ Re_ get-edid___ _something special happened_ _^).eml (3,335 bytes)

$ ~/bin/index.sh list something

/home/rames/email/test/Subscriptions/Signature Confirmation - Petition to ignore the _R
ename _The Two Towers_ to Something Less Offensive Petition_ - 191.eml
/home/rames/email/test/Inbox/Being Twenty-Something.eml
/home/rames/email/test/Sent/Fwd_ Re_ get-edid___ _something special happened_ _^).eml

...now mod me up! Where's mah karma? :^)

--Robert

unicode grep

[Anonymous Coward]

Does anyone know of a win32 version of grep which handles unicode files correctly?

At the moment I have to convert to ascii first, which while not that big a problem, is an extra step.

Glimpse

[polymath69]

No one has mentioned glimpse, probably because they moved to a non-free license a while back, but the 3.0 version is still gratis. And it allows you to configure filters to search files of whatever type you like.

Here's a snapshot of the source for Glimpse 3.0, packaged up from my system as I don't have the original tarfile anymore.

Re:GOOGLE Try POPsearch

[accensi]

Try POPsearch (http://www.popsearch.net/)

Use PostScript converters

[El Buho]

You could add support to your script to many file formats by using some of the many PostScript converters that already exist. Pipe their output to an application that extracts plaintext from the PostScript file, then pipe to grep.

Missing the obvious...

[Anonymous Coward]

..Kazaa