Slashdot Mirror
Digitizing Your Dead Trees?
smart2000 asks: "I'm tired of lugging around dead trees. I've just moved offices and had to move over 100 pounds of 'essential' technical books. It is clear to me that the dead tree industry is never going to supply the books I want in electronic form, so it's time to do it myself. What hardware and software should I use?"
"The Plan: Take the binding of each book and cut it off. Feed into a scanner with duplex and cut-sheet feeder. Scan as a 300 DPI jpeg with compression. Then OCR them overnight. I don't expect the OCR to be perfect, just good enough to use as a searchable index.
What are the suitable scanner choices for Linux? Any recommendations for OCR software that will write in an open format? Has anyone done this before?"
You can find a wealth of PDF/PS/HTML/etc copies of computer texts online. Kazaa is a good place to start. Obviously, only download the books you have physical copies of. :-)
Josh Woodward
Lots of college students at $5/hour.
Slogan-free since April! We pass the savings on to you!
Kinko's offers high-volume scan-to-PDF solutions ... at low volume, it is usually a 10 - 25 per page and the cost of the media to copy it to, but in large volume, sometimes the cost can go down to 1 per page.
Call Kinko's. Ask for the Territory Representative. They'll help you out!!!
hire an infinite amount of monkeys on typewriters and... oh wait, that is for shakespeare
Lysergic Acid Diethylamide, not just chemistry, reality!
Quite useful and handy.
D
I havent used OCR in about 2 years, but the last time I tried it out, it sucked horribly, its acceptable for small documents, that arent that hard to proofreed/correct But for huge documents, like books, etc... Dont expect a huge ammount of accuracy
"The United States has no right, no desire, and no intention to impose our form of government on anyone else." - Bush 05
You can't grep a dead tree.
Alcohol and Calculus don't mix. Don't drink and derive.
Now the bookseller's will join with the entertainment industry. Nexty we will be seeing books that can't be scanned easily.
Remeber those passkeys for computer games in the 80's that were black on maroon paper? Or some dial thingy.
he doesn't want google matches retard, he wants help from people who may have already spent time experimenting with different scanner software. The gift of experience. The whole point of open source, and the "ask slashdot" section.
That's it? Jesus, what are you, a 12 year old girl? That's 2 armloads. Sounds like you need the exercise, fatass.
I'm getting tired of buying books only to find out that a LOT of the chapters are on CD in pdf Form.
What's even more annoying is when the PDF doesn't let you print!
My other sig is extremely clever...
Most of my technical books contain vast quantities of useful information in charts, diagrams, and illustrations... which are far more of a challenge to OCR than mere printed text.
I suspect that even were this sort of thing really possible, it's a major time investment. I have several dozen technical books I'd like to scan, each with four hundred or so pages... and I'm not sure I want to spend a week's vacation time doing it.
And even were it done... there is just something comforting about having a nice printed book that I can set on the desk next to the computer and consult, without having to read it on the screen. Print still looks way better than monitors.
People are never as simple as their stereotypes. This applies equally to Christians, Muslims, and Emacs-lovers.
I suppose this will be marked off-topic, since the poster is asking about digitization hardware. But whenever I see coworkers with tons of books on their desk shelf, I wonder to myself why they really need them. Do they actually have time to read them? Or are they more for show?
;-) ). The rest of the time, I get what I need off the web or USENET.
Personally, I have about 3 books I consider _essential_, and I've read them cover to cover (mostly while in the crapper
As far as I'm concerned, the most important quality in an engineer is not what you know but what search engine you use to look stuff up.
That's what Ghostscript is for. :-)
Check the hardware list for sane and then pick one of the fastest scanners you can afford. The DB on Sane's web site is your best bet. You will find that to get good scanning speed you will need scsi as USB is just too slow.
jpeg also sucks for this. Jpeg is best for full color images like photographs. Better off using tiff or png. Most OCR software will require tiff. Don't know of any OCR software for linux although you might get some windows app to work under WINE. Textbridge from Xerox isn't bad for the money.
.....we use a xerox DC265ST. This digital photocopier scans pages at 65 per minute and posts them to an FTP server inhouse. It can scan at 300 or 600 DPI and you can apply OCR after the scans are done. The DC265 is a workhorse and there are about a million of them out there. The scan back feature is a additional price on the device so not everyone spent the money on that feature....but about 1000 Kinko's have these in house and a Kinko's with a good DTP department might actually even know how to use the feature. (Good Luck!)
.
Ignored Since 1973
This has to be one of the dumbest questions I've seen in a long time. If you're ambitious enough to attempt to scan '100lbs of dead trees' you'd think you'd manage to do some research on your own.
I dont know HOW many times i've looked at a tech manual(or other paper book for that matter)trying to find something I read a while ago and thought " i wish i could just do a text search to find the 3 or so words i remember seeing..." Sure theindex and table of contents gets you part of the way there, but if the author mentions something off-hand in an 'unrelated' section of the book...
Canon's 90ppm high speed scanner - only problem with high speed scanning is that they need loose leaves. Any decent books you have and want to copy will need a Stanley knife taking to the spine.
Please remember to make decent backups on a long lasting madium with a high chance of recoverability. Failing that place the loose leaf versions with a document recovery firm and take their insurance for the full purchase value of the originals.
Matt Thompson - Actuality - Insert product here.
You mean the "do my homework" section?
Scanned images solve these problems, but have two problems of their own:
Perhaps a hybrid solution exists, but I suspect such a solution will require a lot of manual intervention and tweaking, something you'll want to avoid if your goal is to digitize several books.
Personally, however, I still like printed manuals. Using an online manual means either reducing some windows or switching desktops. With a paper manual I can keep the screen exactly as it is. Higher resolution screens, or the use of multiple screens, are making online manuals much more useful (anyone remember what a pain in the ass it was to try and figure out something with only an online manual on a 640x480 screen?). Occasionally I still manage to fill two 1600x1200 screens with a bunch of stuff I want to keep visible while still reading the manual.
if you are anything like the computer guys I know (myself included), you'd end up printing out
portions of the text whenever you wanted to read them anyway!!!
-- Adam
Looking at over 2000 books ( and magazines ) in my garage in boxes im faced wiht the same issues..
Like what sort of scanner, software, etc to do such a massive collection.
And how to rationally complete the project... am i looking at having to cut(!) the books for a sheet feeder, or squish them on a flat bed.. Never had much luck with 'page scanners'..
Am i looking at *having* to buy something like acrobat to make the scanned pages useful??
---- Booth was a patriot ----
Yup. There is quite a lot already scanned. The best places to look are usenet (at alt.binaries.e-book, alt.binaries.e-book.technical, alt.binaries.e-books) and IRC at #bookwarez and #bookz on undernet, dalnet, and irc.nullus.net (and most likely other irc nets as well.)
You could try making a request in abeb, but the biggest selection in one place is irc. So as long as you are not scared by the interface, that is where I would look first.
Think about it.
:)
People love books in dead tree format for the most part. You don't really want to curl up with a cup of coffee and a nice monitor. No, you want some good old dead tree.
But when you're coding, you don't want to curl up with a cup of coffee. You want to sit in a chair and hammer out code while quaffing coffee as if it were, well, coffee.
Most of the time when I look through books for reference, it's annoying. I'd rather be able to just grep for info.
Thankfully, at least O'Reilly's catching on to this.
that is all.
Dont use jpeg, its not good for text. Jpegs are good for photographs because photographs have predictable gradients. Use PNG/GIF for images with sharp/nongradual edges, you will get better compression/quailty that way.
I like the / character. : )
Im not here now... Im out KILLING pepperoni
Use PNG! It's lossless and gets compression ratios that are just as good (unless you are using ultra-lossy compression with your JPEGs - in which case they will be a pain to read anyway). Why do people even use JPEG and GIF anymore? JPEG is only good if you need ultra-high compression and don't care about quality, and GIF only has the animation thing on PNGs.
Sorry about the rant, but there are so many cool computer technologies that people just overlook. It makes me sad.
Fire trucks!! Start your engines!!
believing the big bang requires a certain amount of supernatural faith
O'Rielly (sp?) has many of their java books available on CD-ROM, although I only own the dead tree versions of the ones I have in that series.
.sig and buy some of my favorite books!) That a lot of weight for two books, and I usually haul around a couple smaller ones as well, O'Riely's perl book, and their EJB 3rd edition.
On a regular basis, I haul 2188 pages worth, I just added them up, of QUE's Using Java2 Standard Edition, and Enterprise edition, between home an the office. (Speaking of which, go to the link in my
Not only are all of these books heavy, but I have also yet to find an easy way to card them around, they don't all fit right in any of my bags.
I want all of these books on CD-ROM, but not just CD-ROM. Half the books I have INCLUDED a cd-rom, it just doesn't contain the texxt of the book. With O-Riely, I'd buy the CD-ROM version, but I want to dead tree version too. I want to use the dead tree version, unless I am working from home, I want to haul home the CD's. I don't think I should have to pay any more for it either, I bought the IP (in the property sense), and I am already paying the price for the wood slices, which includes a silver disk.
PUBLISHERS, GIVE ME THE BOOK ON THE CD TOO! I spend $100/month or so on tech books.
-Pete
Soccer Goal Plans
And put them into an inferior visual format you cannot read without the computer being working and on?
And you are going to spend about 100 hours to do this.. and the original books are going to be ruined.
All this just so you don't have to make 3 trips to move your books?
Mmmkayyy.. (backs away slowly)
Have you ever heard of a dolly?
Schools for the blind have been doing this for years, especially with technical books. Many of my V.I. friends would remove the binding and feed them through a high-speed sheet feeder to a scanner. Then, the books are proofed by seeing people for OCR perfection. Contact your local school and ask if they already have some of your works in pdf/jpeg/tiff/WordPerfect (yes, lots of Word Perfect). They may be willing to give you some legal copies of your books in exchange for you converting some of the books you have that they don't into blind readable format (which means, you'd have to proof your own book for accuracy - but you're doing that anyway). Basically, you're donating your time for a good cause and bennifiting yourself.
Exoskeleton
I think you may be underestimating the sheer enormity of your task. Getting sheets to all feed right (a little skew and you're skrewed) and in order (feeder issues, what happens when one page mis-scans/feeds, can you go back and insert it into it's proper location), handling front to back issues (though I would assume that decent scanning software would take care of this for you). Also, your plan to use jpg might be problematic. OCR is finicky enough as it is, back when we were scanning documents we always used 300dpi tiff (using group3 or group4 lossless compression) to get the maximum accuracy rates from the ocr package we were using. And speaking of accuracy, keep in mind that OCR software that has a 97% accuracy rate means that it will flub 3 out of every 100 words, in a book that might contain tens/hundreds of thousands or words, that is a whole lot of errors. Now it's been a few years (6-8) since I've done this kind of stuff, so who knows, maybe things are much better now?
I've been wanting to do something similar for years, but with technical magazines, not books. But the sheer amount of manual labor involved has turned me off considerably (not to mention the thought of destroying the original source).
Keep in mind that this is such a common need, that if it were pretty straight forward, much of it would be done already (perhaps someone out there has the time/hardware/software to have done some of this already?) Not to mention the issue that with the web, that much of the information contained in those books are now available online, makes you wonder if it's really worth the time and effort, esp. considering that a great many of the technical books are obsolete two weeks before they hit the shelves.
I just wanna be able to look at the dollar bills on my computer instead of having to carry them with me. Is that so bad?
/^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$/i
I dont believe he was reccomended plain text as the digital format...
What I read was to use JPG, IE images of each page.. OCR is just to provide indexing...
hard core geek-ware
Isn't what you are planning on doing technically a copyright violation?
If you really want to go through all this effort use both PDF and OCR.
OCR sucks royally for large documents, documents with images or diagrams, handwritten comments, etc. However scanning the pages to an image and then creating a PDF of the images does not care about any of that.
So, scan all of your books as images that your OCR software can process. Use the OCR output to create an index of pages. If a specific word on a specific page doesn't OCR well who cares. With typed and professionally printed books your OCR software should be about 90% accurate. Take the images and create PDF files.
Now you have your nice clean images but you still have a searchable index. BTW, when you get this done post your procedures, problems, and solutions to a web site somewhere so that you can share your experiences with the rest of the world.
Start with google. There is a lot of technical information online, and google will find it. Not as good as those dead trees, but if you can find it and it is accurate, google is often easier than searching indexes. Best of all, dead trees are limited to the ones you own, while google is limited to whatever someone found useful to put online.
Note the last line of the above: google is limited to what someone else finds useful to put online. So if you can't find it on google, take some time to put it online for the rest of us. If/when you find yourself going back to the same few sites often, link to them from your homepage so google knows you find them useful. In other words, google is interactive, make it work for you and it will work for everyone. The internet is not a one way street.
Finially, some things are just plan eaiser to look up in dead tree format. I would strongly recomend you keep your books intact. Put the information you need on the web (what you can do legally), and keep the books for the rest. If you find you are not using a book anymore because all the information is on the web (including you put it there), then throw it out. My monitor is only 19 inches, not nearly enough to hold all the information I have scattered about my desk.
No, he means the "can anyone please change my damn dirty diapers?" section, of course.
What? You haven't noticed this section yet?!?!
Blimey.
So along with his current books, he would have an infinite number of pages that contain some of the works of shakespeare?
What you need is an infinite number of monkeys with an infinite number of computers
...Silly old bear
Robots are everywhere, and they eat old people's medicine for fuel.
Tons and tons of e-texts. In multiple formats: text, pdf, lit, HTML.
Excellent resource!
Anders Borg wrote this FAQ from Project Gutenberg. Lots of field-tested advice there, such as a suggestion to scan at 300dpi or better.
ancarett, historian and zombie gamer
Heres someone else doing the same thing As goatse.cx!
you'd also need an infinite amount of bananas...
what a mess, all the flies... oh, the humanity...
um, wait...
nevermind.
Before doing this, though, we were thinking of scanning/copying all the documents to keep copies for ourselves. In doing so, though, we could use some advice:
What special steps must we take in scanning 150+ year old documents, some very yellowed and fragile?
What is the best format in which to store them (assuming we want them easilly readble in 20+ years for our kids)?
What is the best media upon which to store the data (again, hoping for readability in 20+ years)? (I'm thinking online storage to allow easy conversion to the media of the moment, but I still want something to stash in the safe deposit box)
Does anyone have experience with digital preservation/resoration of archival documents? Should I just try cleaning it up in photoshop or should I find a pro to help out? Maybe I can make it a term of the donation to the museum/library, for that matter.
Thanks in andvance for your advice.
"Prepare for the worst - hope for the best."
Listen, chump! Scanning books and storing them online is a waste of hard drive space! Space which would be better utilized for illegal MPAA and RIAA copyrighted material, porn, games, warez and other materials.
1. You can keep paper docs next to your computer while you work without having to juggle applications on your PC.
2. It's much easier to read paper docs while you take a dump or lay in bed. I know that you can use a laptop while shitting or laying in bed, but a book is easier. Save the toilet/bed laptop sessions for important IRC chats.
oh my god... people actually use that word any more? We are talking a word that in the computer world is worse than all the old and slang curse words multiplied by their cumulative power... Are we going to let him get away with it?
Erutangis ym si siht.
Donate them to your local library if they are still relevant.
I just donated a bunch of books myself.BR>
Another strategy may be to only scan the stuff you need out of the books.
I just wish I could get rid of all of the leftover records/reports/legacy app documentation in my office.
Robots are everywhere, and they eat old people's medicine for fuel.
Its very simple, just take 1000% of the recomended dose of ginsing, add in about 1 gram of caffine and then sit down to read the books, Just download it all into your brain, the caffine to go quickly and the ginsing to remeber it. If thant doesn't work then just hit yourself in the head about 100 times with each book, through the process of osmosis/diffusion you will absorb al the information that comes out. Both worked for me with Moby Dick and Great Expectations in high school.
Hmmm, I have 5 mod pts, its time to metamod, and on top of that I have to meta-metamod? When do I get to read slashdot?
Call Paul Bunyan. Cause he's a lumberjack and he's okay!
/^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$/i
Apparently, it's vapourware...
Have you tried contacting the publishers directly? Or maybe the companies that created any of your software documentation? I know that some companies have PDFs of their manuals and other books, but don't make it well known. They don't usually offer them for free download, but if you prove you have a hard copy some companies will tell you how to get a PDF version. This works especially well for lost instruction manuals, which you can always get for free.
One good, but old, example is Oracle. Back in the day my company had megs of PDFs of all of Oracle's documentation. There was a main index PDF with links to basically every other possible document. I don't recall Oracle leaving them open for download on the internet. We got them on CD. But it was easy to get since they new we were a customer.
Developers: We can use your help.
Right then. In 1993/4, this is what I did for a living. The company I worked for did quite a lot of this, and one contract in particular sticks in my mind - the digitising of all books in the French National Library.
No doubt the equipment we used has moved on in the intervening decade however. We used Bell & Howell scanners fitted with automatic document shredders. Err...feeders. Yes, automatic document feeders. Not shredders at all. No. Honest.
You see, these were high-speed scanners, and some of the books we received were qute old. Me and the other coder on the project got really quite good at doing "pit stops", or changing the rubber wheels that drove the ADF. What I'm saying is no disrespect to the scanner company - it was the quality of the paper we had to put through it that caused the hassle. Some books, like the 18th century Academie Francais records, were so thin we had to photograph them and scan the photos.
We then scaled, OCR'd, deskewed and indexed the results on decent machines - 25Mhz 486SX, 4Mb RAM and Kofax graphics cards. Everything was then tarred up to DAT.
Hardware moves on, but I'll bet the amount of work remains the same. Do not underestimate the preparation required, and also the ammount of QA.
Oh, and don't use JPEG. Lossy compressionon text? Use TIFF - the image processing industry standard.
Cheers,
Ian
to make acopy such as this, which wopuld be easily transmittable,seems to me you will run up against copyright laws. simply put its illegal, no grey issues there.
http://www.slack.net/~hermit/ebook/- 4-1.html
http://www.slack.net/~hermit/ebook/documents/page
I recommend first looking for them online. I use irc.bookwarez.net #bw and have found many computer books (of course, I only download the ones I own. ;-) )
Isn't an infinte number of computers enough?
/dev/random > ebooks
cat
Yours Sincerely, Michael.
"I'm tired of lugging around dead trees."
I'm tired of that stupid term. "dead trees". I challenge you to come up with one good reason not to call them books, that could be more important then, easily recognized, shorter typing, and not sounding like a jack-ass.
Everything does double every 18 months, you know.
Stock prices especially...
Considered harmful.
All articles from past 12 years.
DjVu format is Wavlet Based with OCR
I have tried the Solo version, file sizes were incredibly small. A nice feature is that the Tiff is compressed to a wavlet compression technology, and if you buy the full version you can add an OCR layer which means you can text search. The sample content demonstrates the systems capabilities.
Many libraries are converting rare books and manuscripts because it preserves the original image so well.
This sig is self referential.
I know girl, does that count... Sorry I lied, but I did know a girl, she was on the bus and she asked me the time and I said 2:30. The good old days.
Real men use the command shell and man() or google ;)
Seriously, most of the hard-core computer folks I know either open their copy of the ORA book on the subject, steal their neighbors copy and flip it open, or use some form of online docs w/o printing said docs off. The only reason I've ever known anyone to print anything resembling a doc is when someone I knew had assembled binder full of pages on tech specs for a project.
It's just a lot easier to sit at the screen arrowing up and down on the doc than it is to print it, reach over to the printer, pull it out, shuffle through it....and then eventually have to take it out with the trash. I've seen comments about paperless offices vis a vis paperless restrooms, but the fact is that for reference there really isn't a reason to print the online doc.
What is your Slash Rating?
From what i can gather there's something that needs to be cleared up: £100 or 100lb?
Andy
..an index of the book on my system. just a table with all the words and which page they appear. Pretty useless without the book, since it would be practically impossible to create the book from it, and it would be damn convienant.
The Kruger Dunning explains most post on
Meanwhile, I've got this nifty little device which is capable of storing more text than I'll read in my entire life....the most current programming books I'll keep in physical format, but I'd dearly love to compress everything else into that little box.
I have a hard time reading content on a screen, I much rather open a book, and make notes in its pages. Thats just my personal taste, I totally see the benefits of a digital book, and I suppose someday that might be all we really have. But until then, I will keep buying books.
I figure college students will catch on to this idea soon. A few students could go in together on a book, saw off the spine and distribute the chapters. Each one scans a small section then they reassemble the pieces, burn a stack of CD's and sell/give them to the class. I could have bought a cheap 7 pound laptop several times over for what I spent on hundereds of pounds worth of books.
Simple people talk of people, better people talk of events, great people talk of ideas.
Go check the #bookz channel on Undernet IRC.
It's like the equivalent of Napster for books - and it's still in it's early stages... everything goes.
They also have 'scanathons' where you all start scanning a book at the same time, and the person that finishes scanning first gets... er... the kudos for being the fastest scanner...
I do not have any experience with their products, but the solution offered by this company seems simple and functional. Their system consists of an apparatus that turns pages of your book automatically, scans, turns, scans, turns. The result you can naturally pass to OCR.
Now, if I was to digitize all my books, I would try to create te the 4DigitalBooks kind of solution myself. The only tricky part is to find a cheap enough way to turn pages automatically, see also Kris Mckenzie's automatic page turner, still the best start is this document which is a proposal and overview on how to create an automatic page turner from pieces, the total cost is $459.
...so he can read them on his wearable. Why don't you ask him? (Thad Starner)
Want to see every step I took to start my company? http://www.rowdylabs.com/blogs/pitchtothegods
both Fuji and Bell & Howell make some tremendous scanners - even on eBay the good ones go for a chunk -look for a Fujitsu 3096E 11x17 high Speed Scanner Item # 2022530016. You don't need to OCR everything if you're gonna take in the whole book(s) intact, just readable images, you'll have page numbers that accurately relate to the TOC and index... and if you absolutely must have the stuff OCR'd this equip can do that too, just takes longer...
just be prepared for the DTIAA (Dead Tree Industry Association of America) to come after you for violation of the DMCA. it's circumvention of the "ink in fibrous media" encryption scheme, don't you know.
heck, even just talking about how you would do it means everyone posting to this thread is busted now.
Used to do this for a while. Document managament segment. The software out there are kinda suck. It is better if you can write your own, on the "embrace and extend" direction of available components out there. Even better if you plan to do a lot of this, go all crazy with all the bills, credit cards statements and daily papers and whatnots :-)
... By all mean go crazy on it .. it is fun .. :-)
Kofaxsells compression board + APIs to help you with all the deskew, strengthen your images and stuff. The APIs that I used was for Windows only, altho it come with full support for C and VB.
Bell and Howell scanners work with Kofax product like a charm. Some model can scan like 30-40 pages/min for a moderate amount of money. Better scan as compressed TIFF. It handles multiples pages much better.
ORC batches can be scheduled to run as soon as you finish scanning the docs.. I forgot what the package is called. But the current technique prolly improves alot more than when I was doing it ('97)
Add in a databse and some indexing capability, you can actually build a business around this (we did).
Why all these trouble to make your own? Well if you are a good coder, you can extende and add any functionality that you might want. Cookie-cutter package ain't also good enough or cheap enough.
Now you might want books only, but what about billings? statements and even integrate other digital forms of documents? And indexing and searching
Annamite
Reading over these responses I realized what it is that bugs me most about having a reference manual in PDF or some other electronic format versus having a nice book in my lap: I don't have the screen real estate for both a document reader and whatever app it is I'm using the reference for.
The endless jumping between windows gets old real fast, especially if I need to copy a code snippet out of a document (like a PDF) that won't let me select & copy text.
But if I had a second monitor right there at eye level, I could just open up the reference doc there. No more switching between windows, and no more neck strain from constantly looking down at a book in my lap and then up at the screen.
There is a trick in windows to get around the PDF no print. You need the full version of acrobat. Open the disabled print pdf and select the acrobat printer, I forget the name. But anyways you will notice the print menu greys it out to disable printing, but if you switch back and forth between the acrobat printer and another printer the print button will be enabled, then quickly disable itself. You can actually use the arrow keys to switch back and forth then with the mouse click on print before it disables it. I did this on a mighty words e-text book and printed a copy of it to acrobat printer which saves it to a new file.
My current setup consisits of:
4 HP scanners with ADF ~$150 ea. (eBay)
4 Sparc LXs from a property contol auction $50
one flatbed scanner for covers and bad scans. $50 (eBay again)
Barebones System/w scsi from Compgeeks $80
(NFS server), An Amtren Device(courtesy of the office) and away you go. I've found the best way to cut off the binders is to use a box cutter and to use your previous cuts as a guide. Several shell scripts to scan various types of books. It's amazing the page numbering schemes some publisers use. With this setup I can scan approximately 2-3 college textbooks 1000 pgs.(grayscale) or 1 color in an 10 hour period. (including checking for bad scans, sane ain't perfect, so you better check em) also jpg isn't very good for OCR, I store as png, and convert a second set to jpg for web viewing. OCR under linux isn't quite there yet (unless you want to pay through the nose) So I am Archiving the pngs to CD until it is. This also allows me to regenerate the jpgs if I lose a webserver disk. Add a nifty little IMageMagick web viewer and viola! eBookshelf! Oh and a NSM CD changer is nice too get to the CDs nearline.You can pick these up on ebay for $200-$400
First, I'd use PNG (lossless) or Photoshop's format(lossless) over JPEG (lossy). PNG/PSD will be crisper and color pictures will not be degraded.
:-/ )
;-) But really the problem I've run into is that the back of the pages tend to show through sometimes. You can help to alleviate this by rescanning the pages by hand. Place a blank piece of paper behind the page. This helps to make the page seem whiter to the scanner. If the paper is too bright then use a darker colored piece of paper (like grey or black). This will help the scanner to tone down the bright white of the paper. Only trial and error can tell you what you will need on a book by book basis. This is because each publisher uses a different brand of paper.
:-) Plastic rulers can also lead to problems. Use a metal one! Save time! Save going to the doctor for stitches! Keep those hard to get out red stains from appearing in your books!
Second, I'd make them HTML/PDF instead of plain text. Mainly because then you can retain the fonts. (Of course, some of the OCR programs will do this for you if you want to save it as MS-Word file but that's another story.
Fourth, a well scanned book is just as easy to read as the book itself. Honest!
Last, use an exacto knife to do the cutting and a good ruler with a metal edge. Exacto + wooden ruler means lots of splinters, badly cut pages, and sore thumbs/fingers.
Nuf Said!
There's a very handy database program designed for large legal case management (cases with 100,000s of pages of produced documents) that enables one to link OCR text with a scanned image of the document. It comes in very handy when you're doing a keyword search of the OCR'ed text - you can automatically choose to go to a image of the page - circumvents quite nicely the problems that OCR has with images, tables, graphs, etc.
http://www.lcsweb.com/Software/concord.htm
But I have to admit that I would just check out Usenet or file-sharing programs for the titles you have - why duplicate work that someone's probably done already?
Tig
It sounds like your best bet is the 'hire a bunch of collage students' route.
What? You want to make fair use copies of our copyrighted work. We'll be sending the lawyers out immediately.
http://djvu.research.att.com/home.html
I think it is free for non comercial use.
Makes nice archives of documents.
...then you can spend you extra time in the Gym becoming capable of carrying 100 pounds of books if you really need them...
Want to see every step I took to start my company? http://www.rowdylabs.com/blogs/pitchtothegods
IRIS OCR is very good software for this. This is not a stupid question at all. Books should go this way.
I have also been wrestling this issue for a while, but with shop manuals for old cars.... Try finding a shop repair manual for a '73 roadrunner. Even after exhaustive searches on the internet to find a place to buy these books, I find one shady site and have to pay $$$ for them.
My thought was that if I could find them, buy them and then digitize them, I could save some poor sap that might not be so search engine savy time to get them a searchable digital copy...
Plus if I could do a keyword search for "holly carb" rather than thumbing through a three volume repair manual set.... man.. I could get my car up and running in half the time, and not ruin the $$$$ manual set with grease and muck in the process!
My main question would be, for discontinued or non-traditionally published books (like company distributed manuals) what is the legality involved with re-distributing (or even selling) digitized versions of them them??? Even if it is a chore to do, would it legaly be worth me doing it for the good of others???
She is reasonably cute in the face.
Well I couldn't agree more about lugging around dead trees ... I've got over 200 mathematics textbooks myself that I am in the process of digitizing.
:)
:)
Having done a fair amount of research on the topic, this is what I can share:
If you are willing to feed your books to the guillotine, then it should be enough to purchase a home/office level scanner that includes a automatic document feeder (ADF). My advice here is to buy an Epson product. Even their lowest end models are lightning fast for their class, and the quality of the scan is consistently high.
For more serious scanning of texts that you don't want to destroy I settled on the Ricoh IS450SE, because this model has a 11"x17" glass plate, which is ideal for scanning both pages at the same time. It copies two pages in just over one second, which makes quick work of manually scanning a 500 page textbook. It's even quiet enough to operate while watching your favorite TV program. The native resolution is 400dpi and it can handle grayscale, at a slightly slower speed. The scanner was my top choice because of speed and cost. At $3000 street price (including ADF), I could justify it because I'll make that back selling my textbooks used on the net, not to mention avoiding purchasing any new textbooks. Some of the other competing manufacturers in this category include Fujitsu, Panasonic, Kodak, and Xerox. Visioneer has recently introduced a sub $1000 scanner that is supposed to be competitive in this category of machines which might be worth investigating.
One last comment about hardware. Since my goal is to read my texts online, I purchased a laptop with a 15" 1600x1200 screen from Dell which I highly recommend. Apparently Dell has recently introduced an improved version of this UXGA screen called "UltraSharp" which supposedly fixes some problems with uniformity of contrast from top to bottom of screen, which might be of interest to someone considering purchasing a 133dpi LCD laptop. (IBM, Compaq, Sony and Hitachi I believe also offer models with UXGA screens.) I'm curious if anyone knows the current lightweight champion for laptops with UXGA screens. My Dell is over 9 lbs
When it comes to software, I had to write my own Python script using the freely downloadable TWAIN interface kit for Python since the driver interface to various software packages I tried was not optimized for doing bulk scanning. In particular, I wanted the software to automatically name/number the images as they became available. One of the Ricoh TWAIN drivers I tried was a little buggy but I was able to correct for that in the Python script I wrote by seeking out the TWAIN spec from www.twain.org and reading up on it. I can make my mods available to anyone interested in driving a Ricoh product from Ricoh's supplied TWAIN drivers.
For archiving/image processing of the raw scanned images there are two formats I've looked carefully at using. The first is DjVu available from djvu.sourceforge.net. This uses the open standard called jbg2 to efficiently compress images of scanned text (by indexing recognized glyphs). It can also automatically segment images according to text and image and compress the latter with wavelet technology. AT&T spun off a commercial venture called LizardTech that is based on the same software provided on sourceforge albeit a little faster and more sophisticated. The open source tools should be more than adequate however for non-commercial applications.
There is also of course PDF, the latest version of which supports jbg2 encoding. Unfortunately the linux viewers for pdf do not support this feature in the latest PDF spec, and neither the PDF viewers or the DjVu viewers support sub-pixel rendering of images. (Of course Adobe Acrobat 5 does so through its "CoolType" technology, but that is only available on Windows platform as far as I know.) I am currently working to upgrade the opensource DjVu viewer to support "pixel borrowing" on LCD screens since standard grayscale anti-aliasing (supported by the viewer) does not look so great on LCD screens.
Then there is the open source OCR project called claraocr. My ultimate goal is to add support to this package for recognizing mathematical formulas and producing equivalent TeX code. This is a HARD problem and I don't expect to make significant progress on it anytime soon.
Anyway, I hope the above the long-winded tour through my adventures in this area can provide some useful insights.
Good luck, and make sure to make available your handiwork on P2P networks like gnutella and Freenet
Or better yet visit the father of wearable computers at http://eyetap.org/mann/ (I think he's involved in our local linux group, but I'm not entirely sure.)
"I'm not impatient. I just hate waiting." - My Dad
I started with the easiest books. - Books that could be removed from the binding. Scans go smoothly with the ADF, but it is not as easy as you might think. I find that I spend most of my time naming the files because the default naming comvention is *01.jpg , *02.jpg , *03.jpg, etc.
It is a problem for two reasons:
most of my books are double sided.
My HP scanning software for windows does not let me name files with a 2,4,6,8 or 1,3,5,7 format.
If books contain more pages than the ADF holds, The first page scanned will still be named page 1.
If I knew a little perl, I'd write a script to rename the files between scan batches.
For scanning full bound textbooks, there are two main problems:
Scanning the side of the page along the binding requires carefully holding downward pressure on the book to keep it near the scanner glass.
You cannot scan the book using ADF, so you should expect to spend A LOT of time scanning.
Do not even consider manual scanning hundreds of pages with a parallel port scanner. WAY WAY too slow. USB scanners are cheap now, and will usually scan as fast as the scanner mechanism can move (assuming black & White scans).
Lastly, be realistic.
Know how much time you'll need to invest.
Rule of thumb: If you need to scan manually, expect to scan about 200 pages per hour at top speed. Is it worth investing six hours to scan that 1200 page book of yours? If money allows, I'd suggest purchasing a second book that you can afford to destroy. Cut the binding off with something like a jigsaw, then insert the pages into an ADF scanner. Hope this helps somebody.
is http://docs.rinet.ru:8080/ - I ran across this site a few years back. It almost looks like an online library for a Russian ISP's technical support staff.
They've got lots and lots of official books, all HTMLized a chapter or a section at a time. They're all a bit old or out of date, too - I know of one Perl book in particular that they have there was one edition behind what was being sold on the shelf at the time I saw it.
-----
Is Darwin an evolutionary OS?
Come to the University of Mars! Classes starting soon!
"Oh... I'm a litte school girl who can't carry her her books around and I take it up the butt.."
don't go to kinko's, they suck...but you should be able to find a good quik printer in your area, calll and ask if they have a digipath scanner, have them scan it(digi's have good feeder's) and save it as a pdf. it's only one copy, I think thats fair use...
I've done a few books with Omnipage 11. I'm very happy with it - it does an excellent job.
After its done OCRing, it brings up a proofreading dialog - presenting you with any words it wasn't too sure of. Depending on the quality of your scans you are usally looking at 0-10 corrections per page.
I find that OCR/proofing takes about the same ammount of time as scanning. As a final check, I read along with text-2-speech at ~400wpm - you can hear any missed mistakes very easy and fix them as you go.
The end result is near perfect.
The idea is simple: All documents (paper and electronic) are stored in a single repository. Retrieval is based on what you know. If you know what folder it was put in, browse around. If you know how it was categorized/indexed, do a db search. If you remember some words that occurred in the document, do a full-text search.
Documents can then be made available through your PC, over the network, over the web, via CD, emailed, etc.
Read this overview if you'd like to find a bit more about the basics of document imaging
Depending on your budget, you could either buy your own system, or hire a service bureau to scan your documents and give you the images/text/index on a CD.
If you've only got a couple books, use a service bureau. We have reps all over the globe that can offer this service.
As for the "build your own" option... Well... Let's just say there's a lot of subtleties involved in building a reliable system that might be overlooked at first glance.
Tom Wayman
Senior Technologist, LaserFiche
E-mail: twayman "at" laserfiche.com
Web: www.laserfiche.com
Document Imaging for the Real World.
with a Fujitsu 3092DG (Duplex-SCSI) scanner. I'd recommend either a Fujitsu or a Canon DR* scanner.
On the rare occasion that a book doesn't want to feed smoothly, I just stand next to the scanner and loosly put each page on the input. Doing this, I can still queue up about 30 pages, leave for a minute, come back and add more, etc. I've done a 500 page book like this once without ever stopping the scanner.
I chop the spine off at Kinkos, OCR it with TypeReader or FineReader (auto-straighten of skewed images, auto-split of an open-faced 2-page scan on smaller books, despeckle, etc).
I scan at 400dpi, which seems to give just an error or two less than 300dpi without too much extra disk usage.
The biggest thing you can do to adjust OCR result quality is to play with the contrast and brightness settings. I've scanned several books that had about 1 OCR error every 3-4 pages. No, I'm not kidding.
Finereader can output directly to PDF, doc, txt, html, etc.
Don't steal. The government hates competition.
See www.greenstone.org
Greenstone is a suite of software for building and distributing digital
library collections. It provides a new way of organizing information and
publishing it on the Internet or on CD-ROM. Greenstone is produced by
the New Zealand Digital Library Project at the University of
Waikato, and developed and distributed in cooperation with UNESCO
and the Human Info NGO. It is open-source software, issued under the
terms of the GNU General Public License.
Includes the document:
From Paper to Collection (224kb)
A document describing the entire process of creating a digital library
collection from paper documents. This includes the scanning and OCR
process and the use of the "Organizer".
Adobe Acrobat I believe allows searches (at least it does allow cutting and pasting, so maybe searching as well?)
Plus it's a fairly small file size with a multi-platform reader.
No ideas on what scanning software to use, Omnipage is okay, but I don't think it creates Acrobat files.
"Shared pain is lessened; shared joy is increased. Thus we refute entropy" - Spider Robinson
All this over 100 lbs of books? Just put em in 3 boxes and carry them! Even if you changed jobs every 6 months this would be no big deal. The exercise will do you good. Digitizing technical manuals sounds like a big waste of time to me.
Run, don't walk, to http://djvu.research.att.com/home.html . DJVu is a image-based competitor to PDF that is a feat of beautiful engineering -- 300DPI scans break down to about 10-30K a page, the viewer is about an order of magnitude faster than PDF, the format cleanly supports separate encoding of page texture/graphics vs. page text, there's significant amounts of open source for it, and more.
It's truly a brilliant format. Go check it out.
Yours Truly,
Dan Kaminsky
DoxPara Research
http://www.doxpara.com
A mathematician, a physicist, and an engineer are asked to find the volume of a red rubber ball. The mathematician measures the diameter and calculates the ball's volume. The physicist submerges the ball in a full beaker, and measures the amount of water that spills out to get the volume. The engineer turns the ball over until he find's it's serial number, then looks up the volume for that model on his Red Rubber Ball Table.
Half of the library in my office is catalogs and equipment data sheets for components. A lot of the rest is more generalized data like stress concentration factors for various object geometries and material characteristics; these are things that CANNOT be derived from theory. Only about 4 of my books (which, admittedly, I do use a great deal) are theoretical books. Physics, Advanced Math, Design of Experiments, and a Mech. Eng. Handbook. When you work with real objects, rather than just theory and pure numbers, you tend to need a lot more detailed reference materials. And I'm sure that at least one Engineer in the red rubber ball industry has himself a Red Rubber Ball Table.
Because the yellow highlighter looks like shit on my CRT.
There are hundreds of them here. Very few are the kind of dopey software manuals you're referring to. Is that a "dearth?"
Find free books.
Has everybody missed the obvious? I mean, have you even tried contacting the publisher to see if the books are available in electronic format? So what if they turn you down, it may get them thinking that their customers want these books in electronic format as well as dead tree format.
They have to have them in electronic format in order to re-release them to the printing presses so maybe they can be reasonable about it. Who's the guy from O'Reilly always posting to Slashdot (Andy O'Reilly?)? He seems quite reasonable about a lot of things, maybe he can get a precedent started in the tech book publishing field just like Eric Flint in sci-fi!
Cut off the spines (a hacksaw works well) and drop them in a page feed scanner.
Interestinglr enough this process works. A few years ago now, when I was in university, a friend of mone was looking for a large repository of english prose from which to draw conclusions about his hypothesis regarding aspects of natural language processing.
Anyway, there was a certain dictionary company that was willing to provide their five million word dictionary for some exorbitant cost. When I discovered that another friend of mine working at the dictionary published by my university had a twenty million word database of actual prose that he collected by "cutting the spiones of books and dropping them in a page feed scanner". Whilst omnipage was abou the only decent OCR at the time the field now is probably more extensive.
My friend got to use the 20 million word dictionary for nada, as long as he helped the dictionary guys out with a bit of coding from time to time. Nice huh. Sharing is such a beautiful thing.
"The first thing to do when you find yourself in a hole is stop digging."
Well, I am scanning my books on old Epson scanner
bought for $35 online as monochrome. I scan them into pnm, then use -g4 compression which gives me 300dpi resolution and 60kB size per page.
I can later printout a single page I need and
I get much better quality than xerox.
Kubus
Scanned text pages should be black and white.
Of course it won't scan this way due to shading, bits of wood chips on the pages, etc. Your image processing software can/should convert it to literally two colors-- black text + background (white). As you can imagine, this kind of "lossy" conversion cuts out a great deal of information and the file size reflects this.
Combined with a lossless compression algorithm which takes these huge areas of the same value and compresses them very tightly and you have a tiny, high-contrast, easy-to-read (or OCR) image.
Now with JPEG, it "loses" information by smoothing (forgive my oversimplification of a complex mapping process). With text you *want* unsmoothed (hard) edges-- it makes things easy to read. The JPEG smoothing process results in hard to read text, so you can't use as much of it before the image degrades too badly to read.
The result, the 2-color conversion with lossless compression gives you a smaller image size for the same relative viewing quality as a JPEG. (Or the flip side, for the same image size, the 2-color image is much more readable than the JPEG.)
Try this-- take a screenshot of some text. (Only text) From the GIMP, convert it to 2 colors and save as PNG. Then save it as a high-quality JPEG and a low-quality JPEG. Check the file sizes versus the clarity of the text.
(1) But the time you finish, those books will be obsolete. Hint: Don't buy any more.
(2) What will you read in the crapper? Get a wireless card and laptop?
It shouldn't be Kinko's business at all what you are copying. The copyright commies have struck again.
Fair use policy on this dictates that I can do whatever the hell I please with my books for my own personal enjoyment. Why would a publisher have a problem with that? Why should anybody have a say in my enjoyment of purchased material.
For that matter, the only really essential papers out there are the Bloom County and Outland collections, for which I'd get only $35 from the used book guy - but which are worth incalculably vast sums to me in terms of necessary mirth on demand. I'm putting the whole schmeer into PDF right now because it's totally 100% worth it to me. I don't care if it takes me 15 years to scan and all that, when I need my mirth I need it now. Dammit. And I don't want to search through 20 books to find it.
Consider all the times you've had to hunt through 400 million pages of generic-looking documentation for one essential spec. Think how many times in the future you'll hunt through those 400 million pages (we'll use 275 million for the sake of argument because your data will tend towards the later-middle pages rather than the front or back - a pseudo-fact proven out, in my experience, time and time again) and assign a rough estimate - say, 1,350,000 more times in your entire existence if you're not far past 30 with a life expectancy of 80 years. Paging through those 275 million pages 1.35 million times leads to 371,250,000 pages perused in pursuit of tidbits. If you're super-duper-speedy man, you'll visually audit about four to five pages a second (unless you've not read the book - but we're dealing with a best-case scenario here so forget about that for the moment) until you derive that special nugget, and finally you'll have to put away all those books (unless you're like my sister, but we'll hope and pretend that you're not so ignore that also) which will take approximately 83 hours (assuming 3 seconds per book for 100,000 books at 4000 pages per book).
Now, I'm no math major here, but upon adding it all up, it would appear that one might easily spend 92,812,500 seconds directly searching plus 83 hours per instance of garbage collection (we'll use another thin-air variable, in this case with a value of 300,000 because chances are that you won't be so fastidious about cleaning up as I am) which adds up to 24,925,781.25 hours of your life wasted because you couldn't spare a weekend or three scanning those books.
Think about it.
I recently instaled a pair of Xerox Document Centre 440s, which have a high-speed network scanning capability. Basically, you set up user templates, point it to an ftp server, place your documents on top of it and hit go. The end result is a .pdf file or multi-page .tiff, dropped onto your network. The .pdf is immediately useful, but the .tiff can be run through an auto-OCR, producing an editable document.
I just tested this by cutting the binding off of one of my AD&D Player's Handbooks, placing it on the scanner, and creating an OCR'd document out of it! Makes it really easy to extend a resource for personal use, eh?
Now, the down-side to this beast is that they cost about 45k. *grin* Still, if your company needs a great printer/scanner/copier/fax machine, these things are well worth the price tag.
~wmaheriv
"Shema Yisroel- Adonai Elohenu, Adonai Echad!"
I have scanned several books (in my case, Atari and other classic computing books) for atariarchives.org. The process takes time, but is worth it.
A scanner with a reliable sheet feeder is essential. This doesn't necessarily mean expensive -- I've seen a lot of reasonable-looking scanners with ADFs on ebay for less than $100.
I cut the pages off the books using a single-edge razor blade -- non-ragged cuts are essential. Then I scan then into TIFF format at 300 DPI, greyscale. If I want searchable PDFs, I use OmniPage X on a Mac to create image-over-text PDF, it's quick and easy.
But most of the time, I these books are for Web viewing. So I use a graphics conversion program with batch capability (GraphicConverter on the Mac) to a) increase the contrast dramatically -- near 100%; b) trim the whitespace from the edge of the images; c) scale the pages as necessary. d) scale them more to create thumbnail versions.
There are no hard-and-fast rules for choosing the final file type. Just got to balance file size and readability, and this varies from book to book. Sometimes I go with JPEG, sometimes 8-bit GIF, and sometimes 4-bit GIF. Sometimes I'll convert every page to GIF and also to JPG, then use a little script to select the smallest one for each page.
I'd be careful about trying this. Film is not an especially stable medium. Photographs require careful conservation as well.
ancarett, historian and zombie gamer
The digital representation of the "copyrighted" work as existed in a "page layout" program, using a technological means to prevent digital copying: Imaged to paper using digitally created "Plates".
By attempting to "recreate" the digital representation by using technological means to defeat the digital copy protection of a bound book, you are criminally liable to the owner of the copyright.
(Now if you were just copying this to another piece of paper, you may be ok under existing laws. But moving it to digital... Um, hands up scofflaw!)
Some sort of information is timeless, and won't go out of date. Some sort of information is only relevant to todays technology and can be discarded or moved to a museum when no longer used. Currently I can read books/notes that were written hundreds of years ago in my local library. I don't need any special skills or hardware apart from my ability to read and a set of eyes. It can always be accessed by me, my children and all future generations.Do people really think it is wise to put *important* material onto media that will be redundant (and unreadable)in 20 years time??This article is a nerd feeding frenzy, a whole lot of technical solutions which miss the big picture.
I know exactly what you mean about the National Geographic CD-ROM set. I was very excited about having the complete archives available and was deeply disappointed in the quality of the final product.
Much of the text is completely unreadable because of over-JPEGging. (Is that a word? It is now.)
However, it did teach me to be very careful before plunking down $200+ for online books in the future. Now, I insist on a preview before I buy. (And yes, this does mean that many electronic collections don't get purchased simply because I can't find them in any libraries to view...)
The old destroyed books. I'll use them to wipe my ass.
We, that is two of us, have been doing this since 1997. Our site Internet Technical Documentation Archive (ITDA) houses a lot of freely available Field Service Manuals.
;-)
We started with borrowing local scanning resources and manually page flipping. That's one page per every 5-6mins! Then we bought our first LPT scanner and it was a little faster but ate pages....
...ack depends on what you want to do. Like most people say do you want to totally destroy your books? How much do you want to spend? Are you ever going to use those physical books again?
If its just low cost and personal copy with reasonable quality and you have LOTS of time then.... just grab a copy of OmniPage OCR v11, a HP ADF scanner [ hp scanjet 5490cxi (C9863A)], a copy of Adobe Acrobat and get a professional company to despine your books.
We spent a total of $800 on software/hardware to do this. We spend, on average, about 50 - 200 hours per book to process it - thats scanning, OCR, OCR proofing and format rework and then final PDF output.. Some of the books we're doing I have given to students to work on. They'll do it for next to nothing
Its possible to outsource this to companies to do this work for you. For example Crowley do this and they also handle large documents. You have to be aware of how they are going to process your book and the copyright problems. However, as someone said, some don't care about copyright and some do (eg Kinkos). Again this comes down to do you care about the books and how much you wanna pay for a digital copy...
In our case we don't make money off this site so we can't afford to out-source. So our biggest problem now is how we are going to get the over-size PDP-11 documents into PDF. The Minolta PS7000 looks like the beast we need but its way too expensive for a non-profit. We'll probably be out-sourcing and eating the costs.
My suggestion is to either go the HP scanner+Omni+Adobe PDF route OR out-source it if you can afford. At least with the out-source option you get to keep your books intact.
ITDA Team
And then 3 weeks after you chuck it, go "Damn, I can't read this page!" when you go to look up something and it says, "It is extremely important that you fark dnf2 gib oefll or else you will damage your hard disk."
Stick with books. There's a reason why they are popular. They work really well. Besides, the trees are already dead so you're not doing them a favor. And you'll just have to kill more trees to get more books to scan more stuff.
There is no lumber cartel. Move along.
You haven't investigated all the transportation options for your collection. Wouldn't you rather put all that potential scanning time to better use? :-)
Jack
We want more, but we're getting Jack instead.
Put Cd in drive. Run Sad Old Easy CD Creator that came free with your cd burner, select "copy cd", select source and destination cd drive, click copy and follow on-screen prompts about changing cds over.
Just remember to search for a crack on the web too!
graspee
you slashdott peoplez ar ea bunhca fuckin lame asshole kocksukerz and u''s can up and suck my fat boner
What is the oldest file that I have?
and ask:
What is the oldest useful file that I have?
For most people their papers and books are much older than the data they keep and the paper version is always available and easy to read.
You are much more likely to lose or corrupt your data if it is on a disk or a tape than if it is in a book. Your electronic version is going to be of much lesser quality than the books you had and you will have a lot of "adventures" getting your ebooks to be as easy to read as your paper books. What happens to your portable ebook when your reader runs out of batteries? Ebooks have failed because ... THEY SUCK. Let us all know how much time you wasted tweaking your ebook setup and worrying about how to make them sustainable. Also, please tell us when you go back to the store and buy new "dead trees" copies of the ones you destroyed.
Wow. Mr. Moderator, you're so clueless that when you spend time with your friends they won't let you play Clue for fear of you causing the board to spontaneously combust.
Someone notify the authorities! Adobe's software is circumventing copy protection!
*click*c trl-p*
*drag*
*ctrl-c*
*alt-tab*
*ctrl-v*
*
*click*
*wait*
*read*
Why, exactly, is 2.6 chars per line considered too few? Is this the infamous lameness filter? The formatting was so much nicer the first time.
Sakhmet.
Ban the Nukes! Save the Whales! Screw it. Nuke the Whales!
For bi-level images, the standard to use is JBIG, comes from an ISO group similar to those that created JPEG and MPEG.
It generates much smaller files than GIF for printed text, with none of the inconveniences of JPEG. Grey scale pictures come reasonably well, if done at 300 dpi, dithered.
I don't know exactly why JBIG never caught like those other standards. There doesn't seem to be many JBIG programs around, but, if you are handy with source code, there's jbigkit, a library for reading and writing JBIG files. I wrote my own software with that, and converted a half-ton of old magazines into a 20-pack caselogic of CD's.
I am also faced with the task of converting thousands of pages from paper to text files. I suggest looking into using a high resolution digital camera in a custom docking station above a flat surface that holds the printed material. (a photo enlarger comes to mind). Then instead of waiting for the scanner carriage to pass downward over the page, you can take a snapshot of the page.
Send the image directly from the camera to the OCR program. I find that the Xerox TextBridge program can do OCR on a page almost as fast as I could turn the page were I not using a scanner to input the text. TextBridge is quite ackward to use and not very customizable for new types of applications such as this.
Using a high resolution digital camera to input OCR text is also a good way to get around the question of whether or not to cut off the binding of the book.
By the way, I assume that you're wishing to scan european language text. Doing OCR on Japanese, Chinese, or Korean I would assume is much slower than recognizing ASCII. Does anyone know of an available program that will do OCR on Chinese?
With our friends in the middle east obsessed with blowing the shit out of us, it might be time to develop an open-source program that will do OCR on Arabic and Farsi, along with a translation program companion. Would Arabic be much more difficult to OCR because all of the phonetic symbols are joined together? I sometimes wonder about these things when I'm bumming about not having a life.
Two questions.
One how many companies & publishers are putting their material in an electronic format?
Two are their any E-publishers out there? You know strictly CD,floppy,etc, no dead trees.
Couple this with one of those new dual-screen laptops and you're set. I'd love to do this myself actually, I probably have around 100 to 150 pounds of technical dead trees. Of course, I'd look and see what digital documents are already available. For example, dont scan that XML Blackbook, just download the official spec.
Question
http://www.ironfroggy.com/
I scanned one book (500 pages) into Word format in a weekend on a basic flatbed scanner, using the OCR Omini Page Pro 10 by Caere. It is very reliable for text and proofreading is very minimal. Maybe 10% of the pages require any human input. If you go with OCR, this is the best one out there.
For scanned documents, nothing can beat DjVu. Bitonal documents are 3 to 10 times smaller than with TIFF or PNG. Color documents are 5 to 10 times smaller than JPEG or PDF. There is a free online conversion service at Any2DjVu.
It sounds good in theory. However, when I want to get something like a pin-out description on an IC, it is actually easier pick up a vendor supplied tome for the info desired.
.pdf files. Their first catalog on CD was quite pathetic....
.pdf....
.pdf to text whenever possible.
BTW, a lot of the manufacturers and businesses are publishing their own CDROMs. Case in point:
Mouser sells electronic stuff. They did pretty much just as you recommended, cutting and scanning their paper catalog into
Having learned their lesson, they apparently found a way to clean up the mess and now they have both a decent CDROM catalog and also publish on paper.
Now, about
I HATE IT! AND I HATE ADOBE!
Let me explain. Why make a %$#%# CDROM in a format meant for PAPER?!?!? Don't get me wrong, I'm no busom buddy of Billy G, but Microsoft lets you zip thru text easier than anything Adobe ever did.
They can keep their silly little free reader. I translate
Yours,
Complust
What part of "no text" don't you understand...?
Fair use policy on this dictates that I can do whatever the hell I please with my books for my own personal enjoyment.
/., and even the most obvious cases of copyright infringement aren't considered copyright infringement.
Except that this isn't a fair use. It's definitely copyright infringement. First of all, fair use doesn't at all say what you think it says. You're thinking of first sale doctrine. First sale doctrine doesn't help because he's making a copy. Neither does the statutory exemption for backups of software. And although this copying (or derivative work, same result either way) isn't for commercial purposes (arguable, of course), and the copied work is factual and informational, the other factors actually considered for fair use weigh against it (i.e., the copying) being fair use.
But this is
"Goddammit, where is that June 1993 issue of Hustler's Barely Legal?! All I can find are dozens of Mayfairs and Club International..."
;-)
>>I believe Babe his big blue ox did the hauling.
No, Babe was a PIG and he did HERDING.
Here, you can download the Ruby source or the executables for your platform at http://www.ruby-lang.org/en/index.html
I was locked out because of their spidering filter, too. But I called up at like eight o'clock one night & someone unlocked it for me (& set it so that it wouldn't happen again).
Safari also has a very good search engine, althought it's wierd that they coded it in MS ASP.
The spidering filter seems intent on inhibiting the casual copier. I thought this was lame, but there's actually a certain logic to it. If you go to all the trouble to download & reassemble the books, then you've put enough work into it not to not just throw the book out there on Gnutella.
At it's most expensive, Safari books cost $2 per month. So I'm not impeding anyone's education, and I'd like to see this service stick around. In fact, I can save people a bundle if I get them to use it the way it's meant to be used.
The one lame thing is that OReilly pads their selection with multiple editions of the same book and also with books that are available for free on the openbook site--ok, that's like five books, but still... They're really starting to get a good selection now.
In college, I used a free (as in stolen beer) html copy of a textbook for a class, and realized at the end of the year that someone had purposefully altered the book so that a lot of information was horribly incorrect. They'd basically cut out the word "not" all through the book, and inserted it after "is" in other places. Most people would not do that, but some a-hole did. Ah, college, what a hellhole.
I got check irc.nullus.net tonight for my daily dose of bookwarez and I can't connect. I think it got slashdotted! I just hope this never happens to my server. ;)
This situation has been approaching for a while now.
What will the ALA, librarians, and book publishers around the world say, what insanely stupid legislation will they lobby to enact, when people begin to creating legitimate electronic duplicates of books and other printed material under their fair use rights?
I'll tell you what. The DMCA. All over again. And worse. Applied to literature, rather than music. You heard it here first.
But one thing still puzzles me. Typically, librarians are super cool people, full of common-sense, against stupid legislation like that of internet censorship. (See also one of my most favorite 'sites on the net.) I wonder what the reaction to electronification of information will be of the level-headed, pro-freedom librarians of the world. Will it be librarians vs. publishers and the ALA, side-by-side with programmers and technophiles vs. the MPAA and RIAA?
What needs to happen is a complete and total revolution and upheaval in the way we think about intellectual "property" and copyright law. But that will, of course, never happen in our corporate-ruled capitalist soceity.
I think it's time to pay a visit, and hit the information desk. It's been way too long since I visited the local library.
Depending upon manual type, none of the suggested methods will guarantee more than 90% accuracy when you attempt to turn them into grep-able text. (My assumption is that you're not going to spend a couple grand burning CD / DVD documents that you can't then search.) This effectively renders the output useless, particularly those containing code. Either commit to typing in the important docs in by hand, perhaps scanning the images, or face up to the fact that the technology does not yet exist at the user level.
Note 1: The above comes after months of attempting to turn The C Programming Language, Rev. 2 into searchable text as a test model for a large government project. (Don't worry boys and girls, your paper documents will not be turned into easily referrenced HTML anytime in the near future.) I did, however, do so by hand to run a diff on various Co's output. The aggregate result was closer to 87% accuracy, and the average error rate showed only 5% duplication, meaning that further refinements in the OCR would probably only bring the errors back down to a level of 10% -- 90% accuracy.
(Scanning method was left entirely to Co's interested in bidding, with the proviso that this would be used for counties up to 200,000 residents - records, time line of 6 months.)
Amen on that hypertext comment. The battle has not even begun.
Most folks aren't lawyers, but generally people have seen some texts of court opinions at one point or another. I was just going over some court documents related to the patent courts --AKA, the CAFC-- and I was struck by how computer code-like the text was. The only reason people think it's hard to read court cases, especially patent court cases, is because they're riddled with links to other cases. Since the system was developed in a book only format in a rather rag-tag fashion, the text becomes very difficult to read because of all the notations they've used to indicate varying types of links.
In my opinion, requiring the legal system to use electronic hyperlinked texts for court opinions and other legal documents is absoultely essential to any kind of IP reform. Until judges are benefitting from hypertext in an immediate way, they're going to fail to see the urgency of advocating its use or deciding in favor of electronic formats.
Law and court documents should be readable by anyone with standard high school level English skills. The same is true for patents themselves. The core of a patent isn't the drawings. In fact, the drawings are often intentionally misleading to avoid disclosure of importatnt information valuable to competitors. The important part of a patent is the references to other works, these are natural places for hyperlinks. I bet Bounty Quest would move a lot quicker if patents had hyperlinks.
http://www.claraocr.org/
"Clara OCR is a free (GPL) OCR for systems that support the C library and the X windows system (e.g. most flavours of Unix). The development platform of Clara OCR is 32-bit Intel running GNU/Linux.
Clara OCR is intended for large scale digitalization projects....."
Havent tried it, but it looks good.
Jeff Knox
So.. it isn't "fair use" to copy your -own- book for your -own- use?
/. readers doen't consider the 'most obvious cases' because /. readers know that some laws need to be rewritten.
oh-k
If that's legally the case, then maybe
I'm working on a home project involving motor control project using an Atmel MCU. All my Atmel data is on CDROM or their website. I'm just about ready to call a local Atmel rep and have them send paper books to my office cause it is sooo much easier to flip through pages than to page through pdf files. Not to mention all those times you want to quickly flip back and forth between two sections. having multiple browser windows open to do this just takes up valuable resources that makes the entire development system run slower. Electronic documentation is not always the best.
Now, my 50 issues of Circuit Cellar on CD: that's a whole other deal
"A man observed by the celebrated Dutch physician Hermann Boerhaave took his meals at a table that had been cut away in a semicircle to accommodate his circumference"
:)
No kidding. I never saw him, but my Grandmother has stories about this.
(But I weigh all of 170# without any flab at all.
Try http://www.claraocr.org/ I tried it a while ago and it worked well, but haven't had the need for ocr in a while. -ms2k
You have a valid point; why do you hide behind anonymity?
/.'ers only have a lot of experience with dealing with software and music, both of which have statutory exemptions. Section 117 lets you make a backup copy of software, and the Audio Home Recordings Act lets you do the same for, well, audio recordings. But there's no such exemption for books.
Think about it this way. You had one book. You scan it in to make a copy. You now have two books, one of which you didn't pay for. This is a very, very simple case of copyright infringement.
Problem is,
Look, I'm not saying there should or shouldn't be a copyright violation. I am saying, though, that it's important to know what the laws say. It's also important to be able to think about why we have exemptions for audio recordings and software. To start off your thinking, think about this: fair use DOES allow you to make partial copies of a copyrighted work (assuming some other factors). This you and everyone else have probably been doing for years. How often do you ever need to copy an entire book? Really, if you did that, given the nature of what you're doing, it should be (rebuttably) presumed that you're infringing. But with music and software, you almost always *have* to copy the entire work, if you're going to have some sort of fair use. (If you only copy a bit of a piece of music, then that, given other factors, is then a pretty simple case of fair use.) Since people who were trying to fairly use software and audio recordings kept running into the copyright infringement problem, Congress passed exemptions for those two cases.
That would be the backdrop. What do you think? Do the laws "need" to be rewritten? Should books have a statutory ban? (Realize, though, that would pretty much gut our meaning of "fair use" and a lot of copyright jurisprudence. Which may or may not be a bad thing, of course.)
2 points: ... Very little chance of any of them being on the web anywhere, particularly the "in-house" ones 25 years old.
Firstly many people are reading the questioner's comment about "100lbs of TECHNICAL books" to mean "100lbs of COMPUTER books". Just looking over to my bookshelves, sure ther's a good few computer books out there; also about 30 kilos of reference works on palaeontology, some with print runs that made it to 3 figures; also a few tens of kilos of mineralogy references; lots of oilfield structure and stratigraphy analyses and reports
Second point: a number of the learned journals are addressing this very issue because of library storage space issues. Go to Jstor and particularly to the process description to see how one industrial-scale program goes about doing this. Note in particular the twin parallel paths they use: OCR to produce searchable indices but delivering fax quality PDF images of the original journal pages to preserve complex images, typograpcy and editorial quality.
Another interesting source might be http://www.octavo.com , who amongst other things produce high-quality PDF distributions of historical documents, again linked to a searchable back end produced by OCR again.
The combination of batch, offline OCR and PDF'd images to automatically generate some sort of useable indices to the images seems to have been selected by a number of independant groups.
If you can't be bothered to go the whole hog to build a database, at least scan in the indices so you can do as much searching in the scanned books as you could in the originals.
Not JPEG images - TIFF or PNG. That's a no-brainer. Image contrast is the issue here (for the text sections at least), not absolute image size. My reference library stretches to 1.6GB and is growing steadily (and yes, most of it is copyrighted and legal).
Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"
Actually I *do* use jpeg for storage. I scan the originals at 150x150 dpi RGB.
d s
:-)
.txt versions so I can both grep and use the original document
Why? Because mostly I'm doing archival documents. There is more than text on most pages. Many documents have text over images, text over colours, and so forth.
I give the documents good long descriptive titles in a format
yyyymmdd-source-title-author-keywds-pageno
or
yyyymmdd-docname-pageno-articlename-author-keyw
so they all collate nicely and I can usually find the relevant item with locate.
A very small percentage of documents are pure black text on white paper unless you are doing plain text books, in which case you may well be right.
It all depends what you are up to. If you were a museum archive (or law firm), you'd probably do tiff at 600x600 or better so you'd have a good image of Einstein's coffee cup ring on his notebook page
jpegs do quite nicely for me. Over a period of 5 years I've probably done a 100K pages that way, mostly corporate or personal documents or research materials. I have no trouble at all reading them on screen and although open source OCR sucks at present, it will eventually reach a point of sufficiency, at which time I'll just back the images with
Dale Amon, amon from the vnl.com domain.