Slashdot Mirror
Simple Document Imaging for Unix?
andylievertz asks: "I have developed a logical system of directories for storing my digital documents (i.e. *.doc, *.mp3, *.gif, etc.), and can usually find any obscure document with relative speed. These 'must-keep' hardcopies include everything from bills and shipping invoices to brochures and chinese-food menus. I've tried applying my electronic filing techniques to an actual, real-world filing cabinet, complete with folders and labels, but such a system: requires a great deal of effort to maintain relative to the electronic system, especially considering the frequent influx of new hardcopy material; and doesn't address the greater issue of reducing the sheer paper bulk, organized or not. What solutions have you, the Slashdot Reader, employed to solve this situation for yourself? Are there viable Unix-based Document Imaging packages, similar in function to the Microsoft Document Imaging utility packaged with Office? Do you use a Unix-based Document Imaging solution personally or professionally? If so, what package, and why does it work for you?"
"So, step one is to find ways to reduce the influx of hardcopy (i.e. electronic billing, etc.), but for me, the second step is to find and utilize a [Unix-based!] system that will allow me to scan and file hardcopies electronically so they may be indexed, searched, re-organized, shared, and retrieved as easily as their electronic counterparts. Naturally, any such system would need tolerances for multi-paged documents, and would need to store its output in a non-proprietary file format."
If you want to track money, having the paper is not nearly as useful as entering the data into a financial program. Try GnuCash or something of that ilk.
Delivery menus are different story. I keep them under a magnet on the fridge. If you get a nice rare earth magnet that can hold a half inch stack of menus, that problem is easy to solve (get at least the half inch cubes).
Any solution that requires every document to be scanned is not going to work for you if you can't even file the documents. what are the chances you are going to get around to that stack of stuff to be scanned?
Invest in a magnet, a big box, and a good paper shredder.
I've wondered about this myself.
Seems to me that the solution would involve a scanner, a database, and a mechanical system for retrieving the documents.
1) Scan the document.
2) Slide document into doc protector with ID tag (UPC codes might work, but really it could just be sequentioal
3) Create DB entry for ID, BLOB of scanned image, (or perhaps a foreign key to keep the images out of the quesry, but realistically most DBs optimize this for you) and most importatntly, meta data about document.
The more I think about it, the more I realize a number system of 1,2,3,4...would work fine. The automated retrieval, which would be nice, is not really vital. The match between the doc ID and the scanned version is enough, so long as the document always goes back into the same folder.
Insertion O(1)
Search O(log(n))
Deletion O(log(n))
Note that garbage collection (compation is not really an option, which means to reaclaim discarded IDS (Reuse folders would crank insertion back up to O(log(n))
The question is whether the scanning process would be worth the time.
Open Source Identity Management: FreeIPA.org
What's wrong with the "folder in folder in folder" approach I use?
I don't really need a "system" for that... just make your "root" folders explicit enough, then file everything where it should go.
I even have a "temp" dir for every category.
I don't really see the need for such a tool, IF you can spare a few seconds to browse&dump...
By reading this signature you agree to not disagree with the post you just read.
I have found that a digital camera does a very good job of quickly capturing usable images of paper documents. A 5 megapixel camera provides over 200 ppi for 8.5 x 11 hardcopy and grabs the image faster than does most flatbed scanners. Given the scarcity of drivers for Unix, the only trick is finding a memory card reader that is compatible with your system.
;)
A good digital camera may seem like overkill for scanning in bills, but then the camera also doubles as a camera too.
Two wrongs don't make a right, but three lefts do.
Does Apple OSX count as UNIX? I use iPhoto which comes with OSX.
Ok, so it's not available on any other unix platform, but it employes a nice design for storing images that takes advantage of simple UNIX symbolic links. All images are stored in a hierarchy based on the import date. Then, Albums are created, which contain symbolic links to the real image files.
_______
2B1ASK1
i haven't used them in about 2 years, but it was a pretty slick service. i was one of the first X to sign up (i think 1000), so i got it for free
... eg, pay $50 of phone on 12/4, etc.
basically, you tell the peeps billing you to send your bills to pmb.com. pmb.com scans in the bills, and you could download the scan in pdf and do whatever with it. then, you could pay your bills
when i moved, i lost interest because i could pay all of my bills online (i couldn't when i was using pmb.com), but having it all in one place was definitely nice
i'm not sure exactly how they made $$ on the deal, but i think it was when you'd pay part of a bill, they'd stick that in a bank account somewhere, earn like $2 interest after 3 days, then pay your bill. not sure tho
vodka, straight up, thank you!
Get a label gun.
Put each document in white business envelopes, numerical labels on each white business evelope, put document inside as it comes in. Put envelopes in one of three boxes; never throw away, throw away in five years, throw away in a year. Maybe have additional box for documents that will only fit in big manilla envelopes.
Write a quick perl script webinterface that records one of several customizable options from a pull down menu (ie grocery receipt, gas bill, heroin expenses) along with the date, the numeric label you just put on, and maybe a sentence additionally describing content as needed. The web interface calls an SQL database and stores the information therein or according to the scheme for your electric system you seem to find awesome.
When you need to find a document, search by type and date or text note, find the label number and box number, and proceed as you might typically.
The only trouble you're going to run into is that evelopes are expensive and take up more space than you might like. Try only using envelopes for stuff not printed on 8.5 by 11; put labels directly on stapled folded versions of these. Alternately, divide files by size of paper and paper weight in a serious and merciless fashion. Alternately, reuse envelopes. Anyone have a better system for actually storing the paper assuming you like the sequential numbering + database scheme?
I was looking for something similar that was Open Source or Linux based. Unfortunately, I wasn't able to find anything to my liking.
I settled on PaperPort for Windows. It allows a folder hierarchy to organized your scanned documents that can be altered to your liking. In addition, you can use the application's basic OCR capability to search the contents of all your documents. The previous version, PaperPort 8, used a proprietary file format. But the new version, PaperPort 9, has changed the default file format to PDF. I'm happy with it, and it reduced the mountain of papers on my desk to a small, manageable pile.
http://www.scansoft.com/paperport/
QuiteInsane.
Its insanely good. I use it to scan in all my important documents. It useful multipage modes for... well, multipage documents.
Try it. It's actually been considerably revamped since I installed it, I will have to try a more recent version,
Oh, it comes in a nice debian package via apt-get.
Yours Sincerely, Michael.
I have developed a three tier solution for documents. I have a fax machine that I use to fax my documents to/from. This is connected to a documentum docimage system, that stores the images in an oracle databse.
I was thinking about installing a java front end to use webdav to connect to the db to allow me to access the documents through a webpage, but I'm not sure if I'll go through with it or not, I want to keep it fairly simple...
I'm kidding of course, I have a trash can in my office that my girlfriend loves to throw all my important documents in.
Dude, nothing is simple on Unix.
The rare earth magnet sounds good but what I need now is a refrigerator that I can ssh into and have it read me the contents of the menus ;o)
Seriously...I can think of a number of times where I have a simple piece of paper on my desk at home, and if I could only retrieve it remotely (i.e. from work) than I could save myself a lot of time.
A good, recent example: I just got an eye exam, and needed to fax a copy of the prescription to the [separate] optics store to get my new lenses. If I'd had a scanned image on my server (or some other universally-accessible storage solution), I could have pulled the image down and sent it over to the store in about 10 minutes. But because it was a physical piece of paper, I had to retrieve it from home and wait until the following day to get it to the optician.
Maybe I'm trying to be too speedy?!?! I'm sure there are better, more crucial situations out there anyway.
Andy
In Soviet Russia, the signature reads YOU!
I've tried Kooka and should be good for simple stuff.
There's always Omnipage for OS X, but it's 500 bananas. I wish they sold something that was more in the price range of the normal consumer.
Need Free Juniper/NetScreen Support? JuniperForum
A long time ago on an ancient Macintosh I used HyperCard to deal with this. My particular problem was with technical papers and marketing literature--no matter how carefully I set up categories for filing the categories would need to change over time. So I went to a system of simple numbered folders 1-n for the physical files, and an electronic "index" that consisted of 1 HyperCard card per folder with a title field giving the theme of the folder and and a description field with more info on the actual contents. This worked very well as long as all these materials were hardcopy; then the WWW came along and I don't want to print out everything just to be able to find it again.
I went back to trying to categorize physical folders and hate this system. I'm disciplined enough that I can manage to keep a coherent system and be able to find things, but it takes so much more effort than it should. I've considered using one of the many nice outliners available for the Mac to try to manage this (NoteBook, NoteTaker, Omni Outliner, Hog Bay Notebook, TinderBox, DevonThink, NovaMind, Boswell). Although this appeals to me because I really like outlines and keep a lot of project info this way, it would still be too much work. So I'm thinking of going to a system that's all electronic. Much of my material already comes off the web anyway, and so can easily be captured into PDF format. For the rest I'm considering going to the effort to scan the paper
I've thought of putting together a simple database: something very much like my old HyperCard stack, but with a web frontend--easily slapped together with PHP some weekend. But what's holding me back is that there is a pretty large quantity of documents that fall into an obvious folder organization, and I'm not sure I want to move those off into the serialized id 1-n organization and be required to use a database to find them, when they're already in an obvious (to me only, of course) location on my disk. Yet I also don't particularly want to have 2 different systems for finding different classes of documents, when I know that the classifications will overlap and shift.
As for scanning under UNIX, there's a company that makes for various UNIX platforms commercial tookits which support the kind of scanner you'd want for this (high-speed, no need for super-high resolution). I have recently developed partial (meaning I only support the scanner features that I needed: black & white scanning using the USB port) support under OS X for a perfect (if you can afford almost $1k) scanner. I hope to put up a web page and give away a command-line utility (multi-page TIFF output) to run the scanner some time soon.
I currently use Westbrook Technologies' filemagic it only runs on windows, requires msde, and probably is not cost effective for an individual, but I know filenet runs on solaris, and who knows mabe it costs less.
Theoretically, a webcam should be sufficient for scanning documents. So far, I haven't found a package able to do this. This package should combine multiple (still) frames into a single frame for noise reduction, or possible even stick multiple frames into a single frame, and rescale the whole images to the paper size. Would it not be nice to pick your webcam and move it over the document and have an instant scan of it?
and htDIG to solve all my document storage problems.
The Digital Sender is a wonderful toy. Stick a stack of paper in the bin. Enter an email address. Press the big-green button. And a PDF shows up in my mailbox in a few minutes. Even does double sided. Very simple device and it does most of what I need.
It doesn't do OCR. The Digital Sender outputs a bit-mapped PDF that looks very good. I usually use the full version of Adobe Acrobat to do optical character recognition and store the results in the background. That way I still see the good scan on the screen and when I print. But I can copy and search the text as I would normally.
I use htDig (http://www.htdig.org/) to index my archive. I store content in file folders that make sense (2002 taxes, pitch perception papers, etc). But I still find htdig useful. It indexes both HTML (my lab notebook) and PDF files. All is good.
PDF is a well-documented file format. I wish there was a good free-OCR package, but sometimes you have to pay for good performance. htDig and PDF work great on Windows and Linux.
In three years I have accumulated just over 1Gbyte of content. That represents all my lab notes (in HTML format) and all the papers I've read (in PDF). It's wonderful having my entire paper life with me on my laptop. (I also back it up to three different machines.)
Haven't tried it yet, but it looks like it fits the bill. Personally I just wish Google would open source what they use for their Catalogs site...but maybe DocMGR or something similar will do the trick...
It's at http://docmgr.sourceforge.net/
I've given this problem much thought. I have files I'd like to index such as articles collected off the 'net, po^Hrogramming mpegs, and images. I was looking for a system that is
(1) integrated
(2) flexible enough to let you set and query by keyword AND key/value pairs
(3) transportable (so that copying a file to another disk moves transfers its indexing info as well).
So far the best interface I've seen is Windows 2000/XP's index service. It's an extension to the Find utility that lets you query files very finely (actress=jenna, date>2002, keywords=physics lectures). You set these attributes through a dedicated tab in a file's context menu. I wish there was something like this for KDE or GNOME -- but then, this would not be easily transportable (not unless you're willing to consciously remember to manipulate the metainfo database or move the file's metainfo file along with it).
Roey
Someone mentioned .PDFs. This was my solution
I have a very simple script that runs scanimage, then processes the output through convert to make it a rasterized postscript output, then processed that output through ps2pdf (part of ghostscript).
My scanner (epson 1640U) has a document feeder so the command line options for scanimage reflect that. A simple loop in the script handles all the pages.
The net result is a script called "scan2pdf" that I just specify the output PDF file name (something helpful, like the name of the document and the date). I've processed over a decade of financial records, easily 1000s of pages, in a day with this simple setup.
Funny, not hilarious, we were burgled (sp?) recently - the insurance company wanted original documents for all the items taken, original receipts ... I had most of them, going back about 5 years, and even knew where they were ... this is probably 'helped' by the fact I don't have much disposable income.
Thing is the insurance folk probably wouldn't accept a scanned image (even if it was third party verified in some way!?).
I really need this type of system. By far the single largest amount of clutter in my home has always been bills, other USPS mail that I need to keep (like mail from my 403B advisor), and recipients for a wide assortment of purchases. I've been looking for ths type of system for years. What I really want to be able to do is sit down in the evening with my bills in hand, pull up a software app that lets me choose which predefined company the document is for, toss it on my scanner and have all the settings correct from the get go, and automatically archive the images in the appropriate directories/database. I don't think I need OCR to be frank. I don't see a need for it, at least for my needs. I'd keep my archive in definitely since it's only a little drive space now. I'd write it to CDR once a month or less and drop it off in my bank's free safety deposit vault. Then I could afford to trash the mountains of paperwork that plague every corner of my home. I really need this.
I envision a simple web or application frontend that lets me pull up my bill by company and month (or entry day). Full text search might be nice but I think I could get by without it easy enough.
I'd like to also answer the first poster's question:
"How often do you really need to look at old bills?"
This truly depends on each person's situation. I do consulting work on the side so I have my own personal business. I file expenses under my Schedule C. I can declare all sorts of things to be expenses, including a percentage of my monthly utilities. I've always been told by those people with businesses that you should keep all business-related records for 15 years, just in case you get audited. If you get audited and can't account for ever penny then you're seriously screwed. I don't know what the rules are at the IRS about how many years back you can be audited. For all I know they might not even have rules. Just to be safe I'll keep my stuff indefinitely. I don't mind a GB of data a year if it covers my ass come tax time. Hard drives are cheap after all. :-)
- one big folder
- use highly descriptive and standardized filenames (ie "12-03-water-bill.tif")
- ls -l | grep water | grep 12-03
VIOLA!!! Always works for me!
You can't get any simpler.
I worked for them around 1987, when they used to build their own Unix workstations. They have a nice product, but most of their customers seemed to be large banks and insurance companies.
"I can't give you a brain, so I'll give you a diploma" - The Great Oz (blatently stolen sig)