Slashdot Mirror
Large-Scale Paper-To-Digital Conversion?
An anonymous reader writes "I've just been asked to digitize several dozen sets of lecture outlines at the university where I work. Basically, professors want to hand me a big (often 100+ page) stack of their handwritten lecture notes (with messy text, equations, and diagrams; sometimes double-sided) and expect me to post a PDF-or-something-similar to their course's web page. However, every desktop scanner I've ever used takes 1-2 minutes of user-attention per page and the resulting files end up Huge, impossible-to-read, or both. All I have at my disposal is my PowerBook, Acrobat, a couple hundred dollars of department funds for a new scanner (this maybe?), and, if I ask nicely, overnight use of the secretary's Win2k box. Any ideas? Sheet-fed scanner recommendations? Better file formats than PDF (or better PDF settings)? Do any of you students have usability advice?"
Uh. How about telling your prof. to get stuffed and get a real secretary.
The owls are not what they seem
Some Kinkos have those big goliath Xerox scanners which act just like copiers. Load a stack up papers, and it will scan the pages and load them up. Not sure about PDF export/etc though.
if I ask nicely, overnight use of the secretary's Win2k box
;-)
Plus, if you're lucky, you could also get other after-hours favors from the secretary as well
You need a high speed scanner. Fujistu makes a nice one that works pretty well.
Outsource the job to India.
This is my sig. There are many like it but this one is mine.
The large multi-function HP Printer/Copiers will scan and e-mail a PDF of an entire stack of papers just as you would use a normal copier. I'm sure that the other manufacturers have similar features, but it is the HP equipment that we use at work.
LibBT: BitTorrent for C - small - fast - clean (Now Versio
The HP Digital Sender series are really great for this stuff. You feed it a stack of paper and it scans it, 15 pages per minute, and can store the PDF on a file server or you can send an email with the PDF attached directly from the network sender! It's a bit expensive, but try to look around for one, maybe the local copyshop? Guan
While PDFs are pretty well supported, you'll still be storing it as raster data, so there won't be any size decrease over using an image format, such as PNG.
Are there any web-based packages for searching documents, based on OCR-extracted keywords? Obviously with messy hand-written notes, formulas, etc, OCR won't work reliably. For a similar project, I'd like to OCR the files and use the text data solely for keyword searching. Obviously not perfect, but better than just images.
PNG is your friend....
Just say 'No'. (If you're being told, it's a different matter, of course).
It sounds to me like a damned hard job to automate (which is the only way it's not going to be a constant drain on your time), and you're being given next-to-no resources to even come up with a creative solution. Sometimes the best answer is in fact 'No' - it forces people to re-evaluate what they're asking. It comes with the danger of being sacked if it's you that's being unreasonable, of course....
Simon.
Physicists get Hadrons!
Is to first make an exact copy (by hand) of all the existing documents. Its vital to have a full backup in case anything goes wrong with the scanning process you can always restore the manilla folders to their original filled state.
Do not try to read the dupe, thats impossible. Instead, only try to realize the truth
What truth?
There is no dupe
What I suggest and use is the HP 4C scanner. It's a SCSI-II only scanner that can be found on Ebay for under $10 usually. They also have an automatic document feeder option that can be found on Ebay. This scanner was originally designed for both Windows and Apple compatibility as well. It cannot handle 2-sided sheets.
The scanner has four different pieces of software you can choose to use, I'd suggest Precision Scan Pro as that makes multi-document scanning easier.
Definately keep clear of the Scanjet 5550c; there's a reason why it's the cheapest feed scanner out there. It will frequently jam if you a) load more than 5 sheets into the feeder or b) use any sort of paper that has been handled by human beings.
Our Engineering Society was trying to put up an exam archive with one of them and quickly gave up and started scanning with the flatbed.
Also the scanner has no sane support (one of the few HP scanners that doesn't)
Acrobat sucks ass for bitmap images. It doesn't display them very well, they don't print out well, and the files are huge. DjVu is a new image format that compresses extremely well (a few kilobytes a page -- actually comparable to ASCII text). It's somewhat proprietary, but it's probably the best solution here. There are free web-based services that can compress your images. You can try some of them and see for yourself.
Just fax the documents to a computer.
Do it the open source way.
Get several (dozen) other students to use their own equipment and time in echange for a copy/copies of the completed work.
I would hazard a guess that there are more than a few people who would like to have a copy of the complete series of the lecture outlines.
Get some students of the professor's course to type them into LaTeX. Give them some points they'd otherwise get for homework.
a) Publication quality DVI/PS/PDF files
b) The student can deepen their knowledge of the topic
Everyone happy. Used to work like this at the university I went to. And you may be even lucky that some student typed these notes in for himself.
I found that DjVu format produces substantially smaller file than PDF for the same scanned image.
There is an open-source project http://djvu.sourceforge.net/ that provides code for reading DjVu docs, but I have no idea where to get DjVu encoder.
"Ummm yeahhhh... if you could just do that..."
Faust7 is right about this one. Frankly, OCR is ok, but not great - on nice text on book-or-better paper. Handwritten notes? With equations? No. Not unless your profs have some damn fine handwriting and we all know that that is absolutely not the case.
My advice is the same as Faust7's with these additions: spend some of that money on a really nice keyboard, wrist-rest and/or maybe a nice monitor. You are going to be needing all three. If there are any left over funds, get some really nice tea. I suggest Twinnings English Breakfast or Prince of Wales, if you're going to go bagged.
Exocet Industries - Taking over the world, one computer at a
But the broader question is whether this is really a good idea. The result is going to be huge files, which will be messy, hard to read, and will lack an index or table of contents. Seems like a case of profs with too much ego and not enough willingness to put their own work into more useful form.
Find free books.
GIFs compress very well, especially with source material that's in limited colors. Try making a page into an 8-color or even 4-color GIF at about 150 dpi. The handwriting should be about as readable as the original.
Also, if you're scanning material with copy on both sides, you might get some visible bleed-through. Try scanning such pages with a sheet of black paper between the page and the lid of the scanner, then adjust contrast to ensure white whites and black blacks.
Q: What does the "B." in Benoit B. Mandelbrot stand for? A: Benoit B. Mandelbrot
Maybe he *is* the cheap manual labor / unpaid intern...
Electronic test-equipment manuals are pretty much worst-case candidates for scanning. In Tek's case, the schematic volumes often consist of hundreds of double-sided, nonstandard-sized foldout sheets (11x23" for example) with lots of fine detail that must be reproduced clearly. You can either scan the pages in segments and leave it to the reader to reassemble them, or you can take the manuals to Kinko's and have the foldout pages shrunk to 11x17" or 8.5x11" for scanning. Either way, it's a real hassle, and highlights a clear need for a "prosumer" duplex sheet-feed scanner solution.
A few years ago you could buy scanners like this one that could handle arbitrary sheet sizes, but I haven't seen them in stores lately. These may be easier to use than flatbed scanners, assuming the precision they offer is sufficient for your application. I don't know how well they'd work on densely-printed schematics.
Other than bitching about the state of the scanner marketplace, I don't have much to suggest. There are a few hints that will improve the quality and usability of your final document:
Dahlmann tightly grips the knife, which he may have no idea how to use, and steps out into the plain.
Outsource the job to India
"No, no, not my entire job, just this one part. No, I can do the rest. No, really. No! No... please..."
The coolest voice ever.
I do conversion for fun, at Distributed Proofreaders.
The problem is the mixture of graphics, equations, and text.
It's easy enough to turn a page of text into a smallish file. Get a good automatic-feed scanner ($3500 or so) and a copy of ABBYY OCR software. If the original isn't too speckly, tiny, or smudged, ABBYY will give you a 95% accurate text you can then correct. Best format to save in? Depends on what the school is going to do the files. If they're to be posted on web sites, perhaps XHTML. If it's just for preservation, plain text (if there's no Greek characters) or XML with UTF-8.
Equations -- well, there's supposedly a version of XML for math, but Distributed Proofreaders has ended up using TeX, as it seems to be the mathematical standard. While this would work for preservation, it wouldn't work for a web site.
For a web site, perhaps the best way would be to intersperse text with pngs of the equations and graphics. The pngs would still take a lot more space than text, but the files would be smaller than PDF versions of the whole page.
Unfortunately you don't have much use for something like Acrobat Capture because you have handwritten notes to deal with. To process the files, SANE and/or TWAIN interfaces are reasonably easy to write code for. The cool thing about SANE is that you can run the saned daemon on any Mac or Linux box, and with a couple of lines of config file changes, it's instantly available over the network from any Mac, Windows, or Unix box (there are TWAIN bridges for Mac/Windows so it even shows up in Photoshop and so forth); there are also standalone GUI clients like XSane.
I wrote a document management system in Python/wxWidgets (for Windows) in about a month part-time, and it works very well. Either on Mac or Windows, PDF makes sense because of the ubiquity of the viewers, even if you lose a bit in compression compared to more optimized formats such as DjVu. On Windows you can easily embed the Acrobat ActiveX control; on Mac OS X you have native PDF support, Panther's Preview kicks ass, and there are several open-source PDF browsing components such as the ones out of TeXShop or Glen Low's Graphviz port you can embed in your own app.
Given a choice I would probably pick the Mac to do this project, because of the wonderful Quartz/CoreGraphics Python bindings. You can just draw right to PDF, and place PDF files as if they were images; for example, here's a short script to rotate a bunch of PDF files (sorry, Slashdot destroys Python indentation):
You could also use ReportLab, but because a lot of the PDF processing code is written in Python it's somewhat slower and memory-hogging for high-volume use. (I used ReportLab on Windows for the above project, and use CoreGraphics Python bindings for my research, so I do know what I'm talking about mostlyMy father is an attorney,
he has a couple of high speed scanners from panasonic. They cost less than a thousand dollars (4-500) if I remember correctly, they scan at about 20 ppm, and the software that came with them will save each scanned group of pages as a separate document (pdf, tif, whatever). My dad uses this setup to scan all of the files that his cases generate (shrinking his document storage from about 1000 sq ft to 2 shelves in a bookcase). we are talking files that consist of 10,000+ pages, and normally he saves a years worth of cases on 3-4 cds. They can scan up to 500 pages at a time.
Here is a link:
High Speed Scanners
Is say "Sure. I'll get this done- when I can. Don't expect it to be done for at least a few weeks, maybe longer."
DON'T CLEAN UP THE SCANS. Don't even look at the scans. DO NOT RETYPE ANYTHING.
With the kind of volume you say you're receiving, the only way you're going to survive is to:
1. close your eyes,
2. load the documents into the feeder,
3. press 'scan'.
4. Make sure everyone knows this policy.
poorly set expectations. How did the professors get the idea that it was possible? It's not pssobile under the contraints that you are faced with. If money was not a limiting factor you could do this. But I'll assume money is a factor and time as well. So go back and tell them that it's possible but it's going to cost this much to automate the process and this much if I type it in by hand and this much if someone else does it but with poorer accuracy and so on and so forth. Put the burden on them to decide how they want to deal with this. Only then will the appropriate solution be found and chosen.
http://tinyurl.com/3t236
Cheap? Dunno. It was just there. In any sort of volume though, the cost drops precipitously (cheaper that you doing a flatbed scanner!).
Check out something like that (or indeed that) used, use it, resell it. Or new, then use/resell. Or get the school to buy it.
If this is a continuous thing, then all the better to own.
Kent State just announced thier FlashNotes website. I go to school there, email me at fiveonethree@yahoo.com I would be more than happy to come down and help you sort out your options.
A bit of opinion on the project. This is not a good idea. Its one more tool that studnets will rely on to memorize information isntead of taking time ti THINK about thier subjects and really LEARN the material.
Check whether any of the photocopiers around campus support scanning: we have a Canon ImageRunner in one of the labs which I support. It's extremely fast - ~1 second per page for a double-sided scan and the feeder is pretty robust - we have grad students who take handwritten lecture notes for an entire class and dump this stack of a couple hundred crumpled pages into the feeder and end up with a PDF a couple minutes later.
Comment removed based on user account deletion
I tend to scan lots of documents and setup a simple perl script that uses the 'scanimage' command line tool to do the scanning. Using my Epson Perfection 1650 scanner (pretty standard flatbed scanner) I can scan an 8"x10" page in black & white mode in about 10 seconds.
I actually added a button to the Nautilus GUI shell so I can move to the directory I want and hit the button to scan a page to that directory. Very convenient.
I scan to tiff and then use the convert utility (part of imagemagick) to convert to png. The resulting files typically run about 100K to 200K depending on the content.
If anyone's interested in seeing the perl script I've posted it to: www.ollies.net/scanscript.html
Steve
The quality of the scaning is obviously important; get or borrow the best scanner you can. The point made about putting a black backing onto a flatbed scanner is important. Also important is adjusting the scanner settings so that you get minimum noise (random black dots) without degrading the stuff you want to keep.
For this sort of thing you almost certainly want to do it bi-level/B&W/one bit deep (hopefully there are no shaded pictures, but you can use screening for those), and to my knowledge nothing has been developed that compresses these images better than CCITT Group IV (fax machines use Group III). You almost certainly don't want to use grey-scale, at least not for your final images.
You should see if you can find some post-processing software; we used to use ScanFix, which would straighten the image (which makes Group IV compression a lot better) and depending on settings clean it up as well. You also need to decide upon the size of the final images; you want to scan at 200 to 300 or even 400DPI, but you don't have to have final versions at those high resolutions.
The standard used to be TIFF images with Group IV compression, but not every image viewer can read them, or display them well (esp. if the image needs resizing, and I doubt you can assume everyone reading these has their monitor at a high resolution).
If PDF will accept and display images compressed with Group IV compression, you're probably best off with that, since Acrobat Reader is ubiquitous and fairly easy to use.
PNG is a nice format that I use by preference for > 1 bit deep images, but a quick check of some PNG documentation says that Group IV "often" compresses a lot better than 1 bit "greyscale" PNG; it was simply not designed for document imaging. And you also want to avoid JPEG, it's a lossy (will introduce artifacts) system that also wasn't designed for bi-level images.
Hope this helps.
If the lecture notes you're scanning don't contain any grayscale or color graphics, your best bet is to scan in black-and-white mode (as opposed to color or grayscale) for smallest file size. I'd suggest scanning at 300 DPI for sharp-looking printouts. Be sure to play around with the "threshold" value (or equivalent) in your scanning software until you figure out what looks best. If it's not set to a good level, text may look too thick and blocky, or thin lines might disappear completely.
Once you have a monochrome scan, you'll want to save in a lossless compression format that preserves the monochrome attribute of the image, such as compressed TIF, and not as JPEG. When exporting to PDF, you could experiment with both ZIP and fax (CCITT group 3/4) compression types -- both compress black-and-white images very well. If your PDF software doesn't have those options, the default should probably be good enough. Even at 300 DPI, most pages should fit into about 30K or so.
Most universities already have this service. The professor might not know it exists, but check the other departments to see if they have one (not the scanner - but the service at the school). It is usually somewhat intertwined with a Distance Learning center or department.
It takes away the cost of printing lectures/notes/required readings from the departments and tacks it onto the students who now seem to pay for printing above a certain limit in the labs.
At least this is the way at the universities I have worked at.
The professional approach is to go back to them and clarify the outcome:
(a) you can scan the documents in, and they'll take X amount of space, and Y time; and this doesn't include OCR;
(b) you did a few tests (using the supplied document) and these are the results for TIFF, JPG, PDF, etc;
(c) OCR is probably infeasible (or not, do some tests) because of the nature of the documents;
Include in (a) the option of purchasing an automated document scanner, and the corresponding reduction in time.
Based upon all the above, get a clear go-ahead, and make the purchase if new equipment is authorised.
You said "where I work": this is your job: it's a bit poor to do as the other posters suggest and refuse to do the work: you need to make sure that the customer (professors) understand exactly what they are getting, and give them a choice to buy into it or not - i.e. "clarify the expectations".
If you assess that it's 2 weeks worth of work, and the professors don't disagree, then you're supervisor just has to put up with it.
Xerox bundles OCR as a software add-on. It works well when you get it all set up at your company. By the time you get back to your desk, the document is open and ready to be OCR'd with a drag and drop.
It obviously wouldn't be so convenient if he had to go to Kinkos, but they might have it set up on one of their machines. (Yeah, I doubt it, too.)
I've been reading a few minutes and nobody seams to address your setting etc.
The you should scan in grey-scale or if there is high enough contrast (pen notes, not pencil) in Black and White. The grey-scale with a JPEG medium or even low compressions is going to be much smaller then the deafaults. A pure black and white with group four compression will be even better. At work we scan pages at 300 DPI that way and get 20 to 30 k files (I think, haven't done it for a while).
Also typically images for web viewing of even text are scanned at 72 dpi (all the scholarly journals at my university). This can make things hard to read but really shrinks the file (about 1/16th the size of 300 dpi).
Also if the scanner is set low res pure black and white it will scan a lot faster, but still be pretty slow.
The other option is to pay someone to do it. If you have all of the stuff ready at once and give the pros a week or so to do it when they aren't busy you can probably get as low as 50 cents a page.
Blah blah, I lost my train of thought 2 paragraphs ago
Wow, sent an e-mail as suggested when clicking on "use classic" banner, and got a fast response that addressed my msg
on a side note, if the professors are utilizing a lot of additional material which includes might include3 handwritten information, you might consider getting encouraging them to transcribe that material(hopefully your not the TA that has to do the transcription) into a digital for, be it text or WORD. this'll difinitely help in reducing the size of your files.
also consider looking into adobe's pdf service, if you're overwhelmed with just orginizing the material itself. probably not so kosher to suggest ity on /. but it could be something the school already has an agreement with adobe(taking into account the units of acrobat the school itself might be using). i know it's not rolling your own, but sometimes using an "out of the box" solution to get thing up and running so you can explore other solutions has it's merit as well...
three can keep a secret, if two are dead - benjamin franklin
It makes no sense at all to me, to have a PDF created of handwritten notes. Since most students will probably just download and print out the PDF anyway. The only adavntage is it may save a few trees not everyone will print them out.
It sounds like the school wants to shift the production costs (i.e printing) to the students. This seems inefficient because the old way where the instructor could go to the copy center and have the notes copied the at the schools expense (I know these expenses are often passed along to the students anyway), rather than at the students DIRECT expense of their time for downloading, then printing out on their own equipment or using their own printing accounts at the computer center.
If the notes were being OCR'd and then made available on-line, or post processed in such a fashion (where they are searchable, indexed, etc) where they were searchable, it would be useful. Otherwise this seems like a waste of time and money.
-MS2k
The company I work at scans large amounts of documents to PDF format on a daily basis. Depending on the volume some people do, we use either a Canon DR-3060 or DR-5020 document scanner. These will scan both sides of a page simultaneously, clean up the image (despeckle and deskew) and convert them into TIF or PDF all on the fly. They're fast too. Between 20 and 50 pages per minute. Only problem is that they're expensive.
For your budget, you may be able to afford the Canon DR-2080C which goes for around $600. It has all the features of the more expensive ones, but it's meant for smaller volumes like what you're dealing with. With that, you'd be able to scan 100 pages into a pdf document in around 5 minutes.
charge by the hour, at least 50 dollars an hour. That way you can hire 3 student at 10 bucks an hour to do the actual work.
The Kruger Dunning explains most post on
So, I recommend scanning to TIFF (or TIFF inside PDF). Even if you don't currently have the encoding softeware, you can convert to JBIG2 compression later as it becomes more and more ubiquitous in the future.
And definitely use a automated document feeder of some sort to keep from going crazy. Newer Xerox machines work pretty well for this (I use a DocumentCentre 440ST for this all the time) unless you have hundreds of thousands of pages to deal with, in which case you should either invest in industrial scanning equipment or outsource to a scanning center that does.
I know there are Adobe archival systems that store the scanned image, along with whatever text they manage to recognize. You don't expect near 100% OCR accuracy from an old, largely handwritten sheaf of lecture notes and transparencies. But hopefully enough is recognized to be of some use.
I think what your professor wants is not a bitmapped copy of his handwritten notes or some vector curves that resembles such, but actually a typeset version of the lecture notes. If that is the case, assuming that his handwritten notes are sparse (and hopefully without diagrams, since it takes more time to mess around with them), you can definitely do a stack of 100 sheets in a week, or, as someone already suggested, hire some typists to help you out.
I once had a signature.
If you want to do a good job, you have to type it, in LaTeX. It's the only way to get something nice and something the professors will be able to enhance in future.
If a digitized copy of the manuscripts will do for you, you can go the scan -> image enhancement -> OCR -> save to PDF way.
For scanning, you already got a lot of good comments how to automatise the scanning of dozens of scripts. If you lack these possibilities also a SCSI or USB desktop scanner should do the job (it's definitely less than 1 min per page), so you scan a script in 2 hours. No need to bother to outsource the job to India. Probably you can scan B/W and don't need greyscale or colors. I would scan handwritten scripts at 200 DPI and save the whole pictures in front of the OCRed text, so the user doesn't see the OCRed text and can only use it for selecting and copy&paste. It would be too much work to correct the OCRed text here. For machine written text I would use 300 dpi or more for better OCRing.
As image enhancement you only need to be able to automatically orient the page so that the text is horizontal. I don't remember if Acrobat does it, but for this job I would anyhow get a good OCR program.
As OCR program I recommend FineReader, but also Omnipage is ok. FineReader does better OCR than Omnipage and Acrobat. It also saves better to PDF (with retaining all of the paragraph structure) than Omnipage.
If you keep the image before the OCRed text in the PDF you can expect files of 10MB for 100 pages for B/W scan at 200 dpi. OCRing of machine written text has become incredibly accurate, so you can do real OCR there and throw away the bitmap picture. This of course gives much nicer output (and smaller filesize), but you need to spend a lot of time correcting the text. Here the best OCR program really pays off (you probably have a lot of words which are not in a dict, need custom dicts (does Acrobat have them?),...). A program with a single flaw (e.g. that recognized you formula as text, or code as paragraph text,...) will let you waste a lot of time correcting it on every second page.
The answer is simple. You are at a university. MOST modern photocopiers do inbuilt pdf conversion or OCR'ing to network drives or email. Find one of them.
They no longer make it but they can be found on ebay for a few hundred bucks and no I am not selling this one or one at all.
This claims hardwriting recognition. (Although it requires some sort of structure in the OCR'd page I think)/
Have it professionally done, like other people here have recommended. High-end sheetfed scanners are great, but you probably can't afford one, and it wouldn't make sense as a one-time expense for this small of a job. I'm a big fan of just handing someone some money and it's magically accomplished.
Alternatively, use a digital camera and well-lit copy stand. You can improvise a copy stand with a tripod or whatever, but make sure you have a lot of light. It's a lot faster than using a scanner, and the results are acceptable if you have a good camera. The more megapixels the better - don't use the old 1.3mp one you have lying around. 3mp will technically work, but more is better. Ideally a digital SLR pointed straight down at the page, a very well-lit area (a clamp light on either side of the page works nicely), and you sitting there sipping Starbucks while you hit a cable shutter release after you flip every page. You could get a few hundred pages an hour done this way--your only limitation is how fast you can turn the pages. You'd only have to stop to transfer images to your computer, and you only have to do that often if you don't have enough memory cards. After you get all the pages into the computer, feed them into Acrobat and you're done.
If you don't want to use acrobat you could make a web-page with thumbnails linked to the hi-res images. Then your end-users wouldn't need to download the Acrobat reader. I love Acrobat's ubiquity but hate the file sizes and the slow start-up time.
I scan and upload various land use and financial documents for a county and its townships to the internet on a shoe-string budget - actually, no budget - all volunteer, public service for fellow citizens. This is my prescription:
...", with each page number hyperlinked to a corresponding graphic file. Your graphic files will run 15-25kb each. The use of PDF graphics format is a waste of time and space unless a professor gives you a MS Word file of their lecture notes which you can convert directly into a PDF file with embedded text. That is the only case in which I would use PDF over PNG. Good luck.
Stay with your current flat-bed scanner. Do not waste money on a sheet-fed scanner. You do not have nearly enough money for a high-end Fujitsu or Bell & Howell sheet-fed scanner which will reliably get the job done without mechanically screwing up. The pros use high-end scanners because they never screw up and they go fast. Cheap sheet-fed scanners miss sheets or jam up too often to trust them with anything. Make a sign-up sheet for work-study or volunteer students in your academic department to sit down at your computer and scanner and scan the documents into the computer. Give them free pops and gummy bears (slur it so it sounds like "rum & beers") or something similar which won't transfer from fingers to documents. Just take a few minutes to set them up and show them what to do. Keep it simple. Let those empty minds waiting to be filled with knowledge (and beer) do the time consuming zombie work. You should focus your attention on how to put the files on the website.
The scan file format I use is Portable Network Graphics format or PNG format. On average, it compresses black and white graphics 20-25 percent smaller than the widely used GIF format. PNG format is also supported to a basic enough level to be displayed using MS Internet Explorer, Netscape, Mozilla, and other internet browsers.
I use free Xsane scanning software on a linux system to scan the documents. Xsane can be set to scan in line-art mode, also known as black and white mode. This software can also be set to save files directly to disk in PNG format and automatically change the file names using numerical iteration, i.e., file-01.png, file-02.png, file-03.png, etc. without the need for human intervention to change the file name each time. I use a 100 dpi scan resolution setting because documents do not need to look ultra-smooth; they just have to be legible. Anything beyond that is a waste of hard drive space. Using this resolution also means I do not have to spend time embedding the graphic file in html code to constrain its width so it can be viewed on the average 15", 800x600 resolution monitor. I just insert weblinks to the individual, one-page graphic files: "Page 1, 2, 3, 4,
I looked into this once for a client. Agencies charge around 5c a page but that is only to scan. Add more for OCR, manual verification and/or transfer to M$ Word or what-have you. I think I recall seeing 50c a page for such value-adds. Agencies are good because you dont get need to buy the kit (30K and up) or watch it run (they need feeding and jam quite a lot, especially if the paper is lower quality). Agencies also make sense for shops with nil/low expectation of producing more paper in the future. Get some quotes, references and examples of their work and start with a short trial run.
I wish at was Friday, but I dont want to wish my life away. So I wish it was last Friday.
We've undertaken a pretty large archiving job at my university. We're scanning every page of every newspaper we've ever printed (started in 1927) up to the time we have digital archives starting around 1993 or so. We're also scanning about 80 300 page yearbooks. Hopefully this can offer you some help or suggestions.
We have a dual-processor G4 and an Epson 1640XL large-format FireWire scanner with the optional auto document feeder. It's probably a bit out of your budget ($2899 + ~$1200 for the ADF) but it's awesome. It can scan at up to 1600dpi and the ADF can automatically duplex and scan both sides of the page. We're using OmniPage Pro X for OCR software.
Right now we're more concerned with scanning the documents and getting them online, so we haven't started OCR'ing everything yet. But the ADF is awesome. It can scan both sides of all 300+ pages of a yearbook automatically in about 2 1/2 hours.
The newspapers are a bit different. They're getting a bit fragile in their old age so we have to manually scan them. We scan them at 300dpi in full color, so the 12x18 pages are around 50MB per page. But the scanner takes less than a minute per page. It's impressive.
We use Photoshop's web gallery feature to generate the image galleries. Pretty simple really. Let me know if you have any questions.
Hopefully you'll get to read this one and hopefully it won't get modded down to oblivion.
Yes there are scanners out there that can work for you. I have a Canon DR-5020 which we just feed it a ton of paper and come back in a few and it's done. It can scan VERY quickly. PDF format would work just fine as well. It's the best option especially since it's hand written notes as well.
If this is a requirement which is going to be on-going then you will have to pony up the money and spend a few thousand. If you're not ready to do that, you may be in luck. Some places will lease it out to you and with that few hundred bucks I'm sure you can easily get a hold of one for about a week or 2.
Look up for people who do Document Imaging, and you should find a lot of business that come up. If you're in the washington dc area then maybe I can help you out quite a bit.
we've gotten a bunch of jobs like this - turning handwritten documents into searchable pdfs - and had a lot of luck sending them to firms in india, either by sending the documents snailmail or scanning with a sheet feeder and ftp'ing. the firm we got the best results from was called suntec, suntecindia.com I believe. I know outsourcing is a touchy subject these days, but they were all set up for this, we weren't, and their prices were quite good.
Well I hope someone develops something soon, I've been unable to read my own handwriting since 1995.
As other people pointed out, if you can get a couple of departments in on this, then you can more easily amortize the costs of really good equipment to do this...
One thing that I'll note is that I don't really like PDFs for this sort of stuff. If you really have a 100 page article, you're going to be looking at a 3 meg file and, perhaps, a 30 second startup time... That's fine for someone who's going to read the document from cover to cover, or print it... On the other hand, it's a pain if you only want to look at pages 37 and 38.
GrokLaw gets PDFs of court filings regularly, and I got so fed up with PDF's that I created a (semi-automated) batch system to split up the PDF's into separate PNG images and create a simple index.
You can see a sample here. Far easier to view a page or two there (IMNSHO) -- but not as easy if you just want to download and print it.
Before you go too far, you might want to get a good handle on how people are likely to use what you produce -- Use that knowledge to decide just how you want to organize the result. You may want to make it available in two (or more) different formats. It's not that difficult to bulk convert things between different forms (at lest, not if you can dual boot into Linux, or have OS/X).
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
Use the fairly user-friendly LyX to do the LaTeX-ing.
Heck, get the academics themselves using it to prepare their notes in the first place!
They might actually thank you for introducing them to this convenient and easy document processor.
Important info:
http://www.lifeaftertheoilcrash.net
http://dieoff.org/synopsis.htm
http://www.peakoil.net
Frankly, I've seen professors' handwritten lecture notes, and 90% of them add nothing to the educational process. Certainly not more than a quick note saying, "Read sections 2.1, 2.2, and 2.4, paying special attention to least-squares curve fitting and finding orthonormal bases." They're generally disorganized and difficult to follow because they usually take a lot of material for granted when they write.
The mere fact that it's handwritten means that it's basically a rough draft that was hastily flung together. Send them back to him, and have him type them in and rework them until he figures they're worth recycling for next semester. The prof will save time in the long run, and the students will have something nice, clean, and organized to peruse.
You want the truthiness? You can't handle the truthiness!
or just get the ftp storage and a DocuJob Converter, which converts DocuTech jobs to TIFF or PS, or just use a DigiPath instead.
Hardware for image acquisition:
Check to see if the department copy machine has scan functions... most built in the past few years do, even if they aren't used in most places for that. You'll get a decent sheet feeder and way faster scanning than most desktop sheet-fed scanners.
If you have to buy something and have to go *really* cheap, you could get a multi-function print / scan / fax thing. Most will handle legal size, because they're not actually moving the sheet fed paper onto the flatbed glass... the image element stays stationary while the paper goes by. But, of course, you get what you pay for... expect to spend time dealing with paperjams and skipped pages. However, it should be faster than hand-feeding a flatbed.
Software:
I mention this simply because nobody else has (that I've found): Scansoft Omnipage Pro is designed for highly repetitive, batch-oriented OCR. It has options for doing automated or hand-tweaked "area recognition" (separating text from graphics) and has the best proofreading UI I've seen... it flags "low confidence" recognitions automatically, and displays both it's best dictionary guesses and the actual scanned words. Not sure it will help much with hand-written work, but for printed material it works well.
Format: Your primary concern when looking for a destination file format should be longevity... will the files be readable 5 years from now? I've seen a number of people recommending highly efficient but obscure compression schemes, which are a terrible idea if you want the data to stick around. Saving a few bits doesn't do you much good if you can't figure out what they mean. I recommend that people scan to two formats, just for safety (Omnipage can do this automatically).
-R
ScanSnap may be just what you need if the notes are on a uniform-sized paper (e.g. A4 or letter). You need Acrobat (included) on a Windows machine, but you just set the notes on the scanner and click a mouse then it scans 50 sheets (both sides in one-pass) without human intervention and gives you an Acrobat file in a few minutes. It is small and weighs light so you can easily bring it into the secretary's office. The price is also reasonable ($495 with Acrobat 6.0), and it seems they are even offering a $100 rebate now.
The specified resolution is for a colored documents. For a b/w one, you will get a better resolution. You can obtain scan samples from a Japanese page (pdf files at the bottom).
Actually, a newer model, fi-5110EOX, has already been available in Japan, and I think that is why they are offering a rebate now. The new model have usb2.0 connection and a higher resolution mode (excellent) that is not possible with fi-4110.
I don't know what the specifics of your work is, but you probably have a huge supply of untapped workpower at your fingertips.
.TIF to word/wordperfect/Mathematica, whatever, up to three pages worth.
.TIF files in a class-accessible online folder, and accept the end result in an e-mail.
The students who are taking these classes could easilly be a source of tappable work hours.
See Project Gutenberg's proofreading site for an example of this type of effort. http://www.pgdp.net/c/default.php
If you could get the professors to offer a little bit of extra credit for proofreading or converting a page, the task could be much easier for you.
Envision this: You use and ADF to scan an entire stack of notes in order, but you don't worry about how the scanning goes on each page. Then you xerox the whole stack and place the copies in a binder in someone's office. The students are then offered 10 points extra credit per page translated from
The points are justified since the student is in the class and learning something by carefully duplicating, analyzing, correcting, and studying the professors notes for that class. (Can you imagine a more likely way to end up accidentally committing three pages of facts to memory?)
You can place the
If the file isn't legible, the student can check the xeroxed copy out from the binder. Since it's just a copy, you don't need to worry about losing it.
You could skip the scanning altogeather, and ask the students to return any pages they don't finish translating.
Obviously this works best for large classes where the student:pages ratio is large.
Make sure you number pages if you do anything like this.
we've gotten a bunch of jobs like this - turning handwritten documents into searchable pdfs
We had to do this, too. For a Court, which requires the reasons, decisions, etc. to be publicly available online.
*Thousands* of documents, hundreds of pages each. The responsible department got me, as the IT guy, to set it up for them (after they'd already bought the stuff to do it).
Basically, a couple of Ricoh Aficio series copier/scanners, a couple of ancient Fujitsu sheet-feed scanners, and a bunch of students sitting all day in front of computers running OmniPage Pro.
The Ricohs were great on paper - fast, networked, etc. but their scanner drivers were poor (reminded me of bad CD-ROM drivers - "Copywrite 1995 Behavior Tech Computer. All right reverse." [sic,sic,sic]), and their service (contract) involved having to call the Ricoh guy because the scanner portions randomly wouldn't appear on the network, then wait for him to appear while at least one of the students sat idle. 2 stars out of 5.
Ancient Fujitsu scanners, black and white only, don't remember the model number, required proprietary SCSI cards, no support under Windows NT/XP/2K. These were commercial-grade super-expensive scanners when new (about 1990). Installed Windows 95 on a bunch of relics with ISA slots for the SCSI cards and let 'er rip. Scanning was fast, feed was reliable like a good-quality photocopier or fax machine. Only issue was requirement for an old computer running an old OS; better overall than Ricohs - 4 stars out 5.
OmniPage Pro 12 - reading was *excellent*, far better than anything else I've ever seen. Handled French and English, simple monochrome diagrams, etc. with only very small occasional formatting problems. Print to a PDF using Acrobat on the file server. Only real problem was stability, frequently locking up and losing the scan and OCR on page 99 of a 104 page document. 2 stars out of 5, being punitive because of frustration.
As they got to be more proficient with OPP, and as OPP's dictionaries filled up, we were able to add more and more computers and scanners, so that they were running around, tossing files into the scanners, stapling scanned documents back together, and occasionally rebooting one of the Windows 95 workstations. Peak was 15 computers and scanners.
Task took 3 students 3 months full-time.
Fire and Meat. Yummy.
Basically, professors want to hand me a big (often 100+ page) stack of their handwritten lecture notes (with messy text, equations, and diagrams; sometimes double-sided) and expect me to post a PDF-or-something-similar to their course's web page.
After I stopped laughing, I realized this may be a serious inquiry rather than a joke. I've assisted local government agencies in converting clear, printed, 8.5x11" text documents into searchable text / pdf documents, and the cost for these is over 10 cents a page. (Tax and mill levy records have to be verified 100% correct, as I'm sure your prof's notes need to be.) That's with volume discounting (> 500,000 pages), using nearly perfect ascii text documents, not scribbled notes.
So my advice is to get a few bids from outside contractors, then submit a realistic estimate based on the average. Hint: Given those spec's, it's clear you/your management have no idea what's involved in this process. (Shows at least a modicum of IQ that you had the good sense to ask, however.) If you simply need to scan/save as pics (jpg/tiff -> pdf), you can do this yourself at reasonable cost/effort expenditure. Seems to be implied that you need OCR capabilities for handwritten text, as complicated as equations at that, so you're really pretty screwed. Even simply creating 100-200 kb jpg's & emailing them in an automated process is going to run into problems when the campus mail servers refuse to accept attachements larger than a Meg.
Good luck, BWAhahahahaha!