Slashdot Mirror

Yeah that's similar to what I was thinking about. Actually, what I was recalling was this thing, which seems to pretty clearly use off-the-shelf DSLR cameras (not sure on the lenses though, they're not visible). It probably costs a fortune because of the robotics and vacuum system necessary for the automatic page turning, but I think you could DIY something similar out of two copy stands for a lot less if you were okay with flipping pages.

The one you linked to seems like it would have more distortion of the pages because the cameras aren't being held constantly perpendicular to the page, but maybe it just corrects for that in software afterwards. (It wouldn't be hard, in fact I think all the code you'd need to do it is part of the Panorama Tools / Hugin package.)

What I think is a bigger problem for most libraries isn't the scanning per se, because that at least is a problem that most non-technical people can understand, but it's the storage and document-management that's the issue. Once you have the book scanned, you have a giant pile of JPEGs or TIFF files...unless you're careful about organization, it could become a real mess in a hurry.

So where I think the missing piece is, has to do with getting from raw images to an actual ebook. The hardest problem seems to be in the proofreading step; if you run each image through an OCR program, and then you want to proofread it, you need some way of distributing pages out to proofreaders, and letting each of them have a page of text and the image from that page, side by side. And then managing their edits and checking changes back in, etc. It's nothing really novel -- they're all solved problems in other areas (documents management, change management, remote access, web services) -- but I've never seen them combined.

If you had a software package that handled all the document management and proofreading (preferably something that your proofreaders could log into remotely and work, while storing everything centrally), then the hardware required is mostly off-the-shelf. It goes from being a $25,000 grant proposal, to some undergrad's thesis/semester project.

Actually lots of people do book "scanning" with digital cameras. In fact, you can sometimes get much better results off of a book using a digital camera than you can by pressing it down against the bed of a flatbed scanner (because if the page wasn't typeset with a wide gutter, you'll start to distort some of the letters as you get close to the binding). Plus, it's a lot easier on the books, which is important when you're talking about books that are all going to be 75 years old and some much, much older.

The best way to use a flatbed scanner to scan books is actually to run them through a guillotine first, chop off the binding, and then scan the loose pages; this produces good results but it's not something most libraries are going to be willing to do.

Here's a commercial non-destructive book scanner which uses cameras. Basically, what you do, is you have two cameras, each pointing at one side of the book. You use lights held at an angle to the paper with reflectors and diffusers so that it's evenly lit, and then you just flip the pages and fire the cameras once per page turn. You can build a setup to do this (with manual page turning) for a few hundred bucks plus the cost of the cameras. The auto page-turning is really what drives up the cost.

People were photographing text using cameras for a lot longer than photocopiers have been around. The standard way of reproducing photographs was by using a copy stand and a fixed camera in order to make an internegative, and prior to the introduction of all-digital typesetting, almost all offset printing was done by photographing a paste-up of the final product with a special camera, which produced the plate used in the press.

So in short, although you're correct that just holding a digital camera over a book and clicking the shutter wouldn't give great results, the issues surrounding lighting, lens distortion, and focus are all solved problems. (And if you really wanted to be slick about things like barrel distortion and dust, you could start each run by photographing a standard grey field and a checkerboard, and use that to remove dust and correct for distortion digitally, rather than mechanically/optically.)

The focuses of OCA and PG are really quite different: PG is most interested in preserving the essential information of a book (ie, its text), while OCA's interest is in preserving the form of the book (ie, its fonts, pages format, coloration, even down to the yellowing of the pages). That having been said, there's a lot each can do for the other (and has!).

The Archive has archived most of PG's material, because even though the Books department of The Archive is focussed mostly on preserving books, The Archive as a whole is interested in preserving just about any information it can, and the PG data is definitely of interest.

When the The Archive's Scribe software processes the book images into its various format (jpg, djvu, pdf, flippy, et al), it OCR's the book's text. This text then becomes part of generating some of the other formats. It will be really trivial for PG to obtain this text for any book it wants to incorporate into their dataset.

qv: intlepisode00jamearch. The interesting files here are intlepisode00jamearch.txt which is just the OCR'd text, and intlepisode00jamearch_djvu.xml which is the OCR'd text with layout information (which has been useful to me in developing software which auto-corrects some OCR errors -- where the text is on the page often offers valuable hints for choosing the right heuristic for guessing the right text).

A quick side note on the differences between Google's and OCA's efforts that I haven't seen talked about much -- Google's main advantages in their bookscanning efforts are their wealth and fame, while The Archive's main advantages are experience, familiarity, and scanning technology.

Traditional book-scanning technologies are expensive and slow (which makes doing a lot of books, fast, that much more expensive, because you have to hire more people to do more books in parallel), but Google has enough money to throw at the problem that this is less of an issue. Google's fame means they can bring powerful partners onboard with a smile and a handshake, including some of the most prestigious libraries in the nation.

The Archive has been involved in scanning books and making them available online for several years now (qv The Million Books Project). This experience has shaped the processes used in the acquisition and scanning of books, as well as the technology used in their storage, indexing, and presentation. Furthermore, libraries around the world have grown familiar with The Archive over the years. That, and The Archive's good track record, make it a powerful rallying point for partnerships and alliances, and have given it more experience in facilitating such relationships. Finally, partially due to the limits of existing book-scanning solutions, and partially due to The Archive's limited budget, it has facilitated the development of two independent low-cost, reliable, high-quality book-scanning systems: The Scribe (developed in-house at The Archive) and the Kirtas Robot (developed at Kirtas, a Canadian company).

Many of the books scanned for the Million Book Project using traditional scanning methods are really lousy, sometimes to the point of being unreadable. These new scanning systems dramatically improve the quality of the end product, while equally dramatically reducing the cost-per-page. This means that more scanning systems can be purchased for more libraries (avoiding the per-library capital outlay problem), and more books can be scanned more quickly within a given budget.

Obviously, Google and OCA can benefit from co-operation, as each has a lot to offer the other. I'd be surprised if Google didn't join the OCA, eventually, if for no other reason that to gain access to the books of the >100 OCA

Actually, you can buy a robotic book scanner (there's a demo video of it). No doubt it costs an arm and a leg although it may be worth it if you're scanning a large enough volume of books.

It seems a pity to use such a manual method. This... http://www.kirtas-tech.com/ is designed to scan books, especially old and fragile books, automatically. It handles the pages even more gently than a trained person. It's not cheap, but is does around 1,000 pages per hour, and the operator just loads books in and takes them out when they're done. I looked at the company a couple of years ago (I'm a VC) and get regular updates from them. A LOT of libraries are using them now.

Check this out.
http://kirtas-tech.com/

Kirtas Technologies http://kirtas-tech.com/ makes a bound book scanner that turns the pages, at a rate of 1200 pages per hour, and claims to be gentler than a human. I've seen it in action -- it's a pretty neat mechanism.

Book Digitizer

At $150k, probably cheaper than a prototype stack robot, but more expensive than a grad student to perform the same task.

*f*

Here is a movie of the Kirtas machine at work. Interesting technology used here!