Slashdot Mirror
Batch Cataloging of Scanned Documents via OCR?
munwin99 asks: "I am looking for some software to process a batch of images (scanned forms). We want to use Gallery to view the images, and be able to search them by 3 or 4 attributes. We want to get these attributes from the form (date, name, etc). We want it to check a section of the scanned form, read the info from that section(s), and dump the retrieved info into Gallery (using OCR / ICR). Is there any (preferably) free or open source software that can do this? Supported OSes should include either Windows, Linux or Mac OS X. Even Gallery is optional, if someone has a better suggestion."
Many pieces of OCR software allow you to create a 'layout' for OCRing, that is, specify where images and textual data are. If your forms all follow the same layout, or you have just a few [relatively], you can set up these layouts and, in many pieces of software, reuse them. The only cavet is that you need to be sure that the forms are scanned the same way; if your forms have prepunched holes or markings in specific points on the edge, you can use animation software [like Bauhaus Software's Mirage] on a batch to 'pixel-track' the pages and align them based upon these marks, then export no-/low-loss TGAs, TIFFs, PNGs, or similar for OCRing.
Video Production Support
Pay a kid $8 an hour to scan the forms and process them. You will get much better accuracy and you can put them to other uses as well. A kid from high school will cost you... $5,000 a year or something depending on how many forms you've got. Hell, the kid could do the work from home if you set up the computer post the images in an accessible way. Hire a kid and give him a future in data processing!
While Im unsure if Gallery allows you to create, edit and query 'meta' fields with each image I do know that it reads, stores and can query the EXIF fields of all imported images. One way to be able to store (once)/query (many) your custom data fields would be to initially fudge those values into the EXIF fields of each scanned image. Yes it would be weird to search for 'last name' with a 'camera model' query, but it would work.
Anyway this is probably how you'd want to go about this:
1. Scan doc to file
2. use an app or library to OCR the fields you want
3. Add EXIF fields/data to the image with perl (CPAN EXIF modules)
4. dump image into gallery. Gallery parses out and stores your crap in query-able EXIF fields.
This is all conjecture though - good luck. Seems like a pretty shitty task if you ask me.
The OCR / document image layout analysis world is dominated by a handful of commercial companies. There is a dearth of OCR and document analysis code available in the open source community. That which is available on any sort of 'free' basis is not going to be of a lot of use other than as a starting point for some serious development of your own, I would suggest.
The big names commercially are:
Abbyy's Finereader
Nuance's (formerly Scansoft) Omnipage
and then a number of smaller players like SimpleOCR
In the open source world, some places to start looking are:
GOCR
and GNU's OCRAD
Both Nuance and Abbyy offer an SDK for OCR integration at a code level which might suit depending on your budget. Certainly the price (probably between $500 and $5000 for a license) represent a good deal if you look at the costs and time it would take to write anything that does serious OCR work yourself.
BTW, if anyone out there knows of any good document layout analysis code available to have a look at, I would be particularly interested. I am looking into document layout analysis for a personal project and although there is a fair bit of academic research available at Citeseer, I actually haven't found much in the way of good sample code that I can use as a starting point for some of my own ideas.
Gallery 2 does use an RDBMS. I have set it up with Postgres but I think it can do MySQL also. BTW, I have run gallery 1 with thousands of images, and while I do agree an RDBMS would have been optimal, it didn't really slow down to excrutiating levels.
Occam's razor is the blind faith in the natural selection of least resistance and in universal oversimplification. -- EF
Most commerical scanning engines allow a human to touch up the results, comparing the output against the originals. This is where your intern or entry level temp can be useful. Or you could just outsource the hole ting to summ one overseas.
My job involves integrating with OCR, and we've looked at quite a few options. Though there are some bargains, you get what you pay for.
The big players are Abbyy and Scansoft. Both have extensive feature lists, from handy GUIs to form/document layout to Asian language support. They also come with a hefty price tag. Their Windows support is best, but they have software for others. Single user applications are reasonably priced in the two-digit figure range. However, we decided not to integrate with either of them in part because of the price tag for high volume server-side processing. If you only have a hundred or two forms to do at a time, a workstation solution may be your best bet.
We chose to integrate with Transym, a cheap but pretty good engine. It does a good job at what it tries to do, which is recognize standard printed text. We then take that text and extract meaningful data, like dates and names, from the output text + position information. Pretty much every other cheap/free package we looked at had pretty lousy performance on our straight-forward documents (primarily typed paragraphs).
ICR (recognizing handwriting) and IMR (mark recognition) is another bag. There are very few players in this arena. They work best when the domain is well-defined (the U.S. Postal Service, for instance, does pretty well at recognizing zip codes). If you're trying to recognize dates and check boxes, the form definition software that Abbyy and Scansoft provide probably fits your needs best.
Finally, you need to consider how reliable you want your process to be and how much quality control you want. Even the best OCR engine makes errors, and ICR is quite a bit behind that. You can't blindly trust OCR output unless you're willing to deal with incomplete data. If you're going to have a human verify the computer's work for only a few fields, you may not be gaining significant efficiency.
(I don't claim to have evaluated every potential option. There may be software we missed, software we didn't evaluate because it didn't meet our integration needs, and software that's come to light after we did our search.)
Ceci n'est pas une signature.
I would reccommend taking a look of Abbyy's offerings. Particulary FormReader 6.5 Family which is intented for OCRing forms and semi-structured documents.
http://www.abbyy.com/formreader/
Not sure what you're doing, but this might give you some ideas. I scan all of my papers docs in using a Fujitsu ScanSnap in OSX. It can automatically pipe them to ReadIris Pro for OCR, and dump them in a save directory. I can search for whatever I want in spotlight, and it pops right up. "Hennepin county property tax 2002" bring up one document, and putting in my address and the words "purchase agreement" comes up with the purchase agreement for my house.
It's pretty insane how much time it saves me.
Need Free Juniper/NetScreen Support? JuniperForum
Not cheap, but good...can OCR and match/highlight the search terms in the scanned document.
Now we have PDFs with pretty accurate text ready for Google to index
What OCR software did you use? I haven't had real good luck with this. (The documents are already scanned into PDF's when I recieve them so I have no control over the quality.)
To ensure perfect aim, shoot first and call whatever you hit the target
Get them to put it on mturk.com at .03 per page ;)
Do not meddle in the affairs of sysadmins, for they are subtle, and quick to anger.
Docubase does everything you are looking for. (For real, call them and ask) Too bad it'll probably cost you mid 5 figures to do it. Disclosure: I work for a company that resells docubase integrated into our product.
I am an opensource guy, but even I ended up buying Abby for my mass scanning needs.
Here's what you do: Buy a 5.0 license of Abby Finereader off ebay. You can buy it for about 10$.
Buy the upgrade version of the latest version of Abby Finereader for $150.
It's still $160, but that's still considerably cheaper than paying the new price of $500-600. Abby finereader docs say specifically that the upgrade software will work successfully on ALL prior versions of finereader.
As far as feeding into a database, I'm afraid I can't be any help, but if any software has this functionality, finereader would.
Robert Nagle, Idiotprogrammer, Houston
Thanks. I have acrobat pro 6.0. I'll see if that has the OCR feature. (If it does I've never noticed it). I was using readiris pro, which did a horrible job, at least on the scans I get.
To ensure perfect aim, shoot first and call whatever you hit the target