Slashdot Mirror
Human-Powered Internet Archive Book Project
Carl Bialik from the WSJ writes "A group led by the Internet Archive is planning a massive, ambitious effort to scan millions of old books and make them available for Web searching early next year. Behind that effort are about a dozen scanners, employees making about $10 an hour to manually scan volumes -- some more than a century old -- one page at a time, on special contraptions. The Wall Street Journal Online visits a University of Toronto library to watch one of the scanners in action: 25-year-old Liz Ridolfo."
Will the scans be added to the Project Gutenberg collection?
Last time I moved, It took many VERY HEAVY boxes to Move all my books. Maybe I'll scan them all..
:(
All though anything useful has to be illegal...
0xB315AA8D852DCD3F3DCA578FD2E0BF88
Stories over 75 years old don't have the same copyright protections..
anyone can do 'a christmas carol' because it's copyright has expired..
using however, someones PRECISE arangement of the text is not permissible however- that has it's own copyright...
so if I buy a current day copy from amazon, I cant scan it in... but if I buy a copy that's last edition/print was more than 75 years ago, it is fair game.
every day http://en.wikipedia.org/wiki/Special:Random
Project Gutenberg frequently makes use of the page scans for source material. What PG does is to run the images through OCR, proofread and post-process it. It's more useful than a stack of page images, but considerably more work.
If you look at the current books on Distributed Proofreaders, you'll see that some of them credit the Million Books Project for the page scans.
Laws do not persuade just because they threaten. --Seneca
Getting written works off of paper and stored electronically should be a priority--bits are much easier to store, preserve, and copy for future use.
In Stanislaw Lem's science fiction book "Memoirs Found in a Bathtub", all the paper in the world gets eaten by a virus and chaos ensues. Interesting read if you've missed it, has made me paranoid about how much the world still depends on paper.
Entropy just isn't what it used to be.
Why hello, Ms. Liz Ridolfo. I'm happy to see you are into computers (at least I'll tell myself that) and you like to put your pictures online.
Please email me at superdesperateteengeek@needtogetlaid.net
Project Gutenberg and the Open Content Alliance are working on two slightly different things:
The OCA is making available the images of scanned pages. That's fine for reading an entire book, but you can't search it, nor copy a section of text into a document of your own.
Project Gutenberg makes available plain text, usually illustrated HTML, and occasionally other versions, of public domain books, which can be used by anyone for no cost.
If you'd like to help prepare public domain ebooks, visit Distributed Proofreaders and proofread a page a day (or more!).
The good:
Old books prior to copyright laws are being scanned.
The bad:
Pay is roughly $10/hr. Now, I happen to be concerned that someone being paid so little should be handling rare books. Not to mention the college graduate getting paid so little.
The ugly:
The digital camera contraption costs $30,000!! There's a few scanner manufacturers left in the world and none of them have exploited this niche. Shame on them.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
That's a good question, but I can't help but wonder if this is the miracle of capitalism at work. Right now we're in the eeeearly stages of this sort of thing, and the copyright laws, the mechanics, et al are still rather unexplored. Besides, I have to think -- the scanned images themselves are probably copyrighted by those who scanned, but chances are the plaintext isn't (considering they're copying it already, and not reinterpreting it). So the more people who want to scan whatever, the better, even if they overlap. Consider it error checking.
The real test and business opportunity comes in the distribution phase. The first person to have a huge library of old books, and contracts with publishing houses for new books (with "purchases" by the end users, and DRM encumbered, of course) is the person who will win the market and define the (capitalistic) best way to scan and distribute.
And come the semantic web, things get really interesting. Already we have tons of sites that do cross-referencing between academic papers -- at least, the citations, as well as categorization by topic. When we can start doing this for books based not only on genre, but topic or specific references to persons, or general concepts ("Book X mentions technology Y on page Z. Click here for link!")... well, things will become far more informative. I suspect that in this field, the information -- the texts -- may become free, but the computerized (and human-assisted) analyzation, linking, value-added stuff will be the new commodity. He who has the best algorithm wins.
I guess information has always wanted to be free, but the analysis of said information lies firmly in the realm of economics.
Wow, that book scanner rig is just what I've been dreaming of for years. I've been thinking about mounting a couple of glass plates at a 90 degree angle, and then I could put the open book on apex of the glass, then photograph it with a couple of cameras underneath. This rig is just exactly what I was thinking of, but upside down and even cleverer, with a footpedal to lift the glass up and down onto the book. A very nice piece of design work.
The obvious advantage of this rig is that you don't have to open the spine 180 degrees and smash the books flat onto a single glass plane, you don't have to open the book up more than 90 degrees, so it's gentle on the spine of fragile old books. And the glass wedge is always self-centering against the spine of the book. The only way this scheme could work better is if there was a way to turn the pages automatically. But these are old and presumably valuable works, safer to let paid low-wage drones to do the work than risk mechanical damage.
From the Wikipedia article on the Open Content Alliance:
The Open Content Alliance is a consortium of non-profit and for-profit groups which is dedicated to building a free archive of digital text and multimedia. It was conceived in 2005 by Yahoo and the Internet Archive. It was conceived in response to Google Print's closed nature, and aims to keep public domain works in the public domain on-line. These results will then be used in the search results of participating search engines. You can see a sample of the open content at openlibrary.org
A large difference between the OCA's approach and that of Google Print is that the OCA intends to ask a copyright holder before digitising a work that is still under copyright, while Google Print will digitise any book unless explicitly told not to do so by November 1, 2005.
So, Google Print will almost certainly be better when searching for copyrighted material. For public domain works, we'll have to wait and see.
IMHO, it seems like a little cooperation here would make a lot of sense for both parties - they could save money trading digital copies 1-for-1 while remaining in (healthy) competition.
That probably means that the search index will be uncorrected OCR, which leads to some inaccurate searches. The problem with using raw OCR is scannos, words that may be recognised as a different word that "looks" the same, for example modem and modern, or an i might be recognised as a slash.
Your time might be better spent at the real Distributed Proofreaders, or DP-Europe, since Projekt Gutenberg-DE is not an offical branch of PG, and actually copyrights its output (unlike the real PG).
Here's a list of book scanning equipment. I've seen the one from Kirtas in action, it's fun to watch.
> bullshit
I too want to be modded Insightful!
being smart is exausting
employees making about $10 an hour to manually scan volumes -- some more than a century old
I think that if they hired younger people to scan the books, it might go a little faster.
Imagine a 100 year old at this job...
"...(mumble mumble) in my day we used priests to copy books (mumble mumble) oh dear, I tore another page, darn Parkinson (mumble mumble)"
Ah, arrogance and stupidity, all in the same package. How efficient of you. -- Londo Mollari
Interesting - I don't understand your line of thinking - interested to hear more. Is the argument that automated page turning is *cheaper* so it's a pity that the project spends a lot on labour charges (manual scanning)? Or is the argument that the automated page turning is easier on the fragile old books? I'd appreciate if you could offer more details about the technology - the company's demo video shows a vacuum device lifting pages, but both examples are with modern books. Honest question: surely the advantage here is a low labour cost method of scanning huge numbers of pages (like the telephone directory example they show). But if you have fragile books, surely the advantage of a human is that they can see that individual pages might be particularly fragile, maybe even needing support or repair to scan, while the pre-set vacuum device will plough on regardless, it won't be able to make a decision on the quality of the pages. Does it have any sensing devices built in? My experience of older books (e.g. nineteenth century) is that in some cases the paper can be very brittle.
The (Jack) Vance Integral Edition was a volunteer effort to produce a limited edition 42 volume set of the complete works of Jack Vance, restored to as close to the author's original manuscripts as possible.
(The project is complete, and an amazing success.)
The team scanned and edited many of Jack's early works for which there was no good clean manuscript. They developed software tools that would compare scans from different editions to automatically find errors. It turns out that even the best human editor still missed "scanos" (typos produced by the scanning process) that the automated tools found.
Even so, in the final books there were a handful of errors that slipped through, despite extremely careful editing by hundreds of volunteers.
I'd RTFA if the black text didn't overlap a black image. IE-only web designers should be shot.
This is when the 'remove this object' firefox extension comes in handy. Just remove the image and the text is readable. 'Undo last remove' to get the image back.
I don't think you should have been modded down.
'Intellectual Properties' are uncontrollable in the wild. To base an economy on them is just stupid.
At least partner up for the process of scanning even if they have different plans as to what to do with the scans
- My question is: Can Slashdot be Slashdotted? -
As a few others have mentioned, jump in to Distributed Proofreaders. We take the raw images (either scanned specifically for DP or taken from scanning projects like this) and produce checked, corrected text, which then goes to Project Gutenberg. A few hours a week can help a lot.
(1) The library paid for the copy you're borrowing. (Or somebody paid for it, in case the book was donated to the library.) Thus the author was paid for that copy. If you read a whole copyrighted book via a Content Display Site (CDS - Google Print, Amazon Search Inside, etc.) and never buy the book, the author wasn't paid. Copyright law is about creating new copies; you're not creating a new copy when you read in a store or from a library.
(2) Browsing in a bookstore is pretty inconvenient. You can't take the copy with you to look at any time you want. (Unless you buy it! That's sort of the point.) Bookstores know that few people really read entire books in the store -- else they'd go out of business. However, reading a book from a CDS doesn't have that limitation: You can take it with you, on your laptop, etc. This is particularly critical in light of digital paper, when the digital copy is the paper copy.
(3) Libraries and bookstore reading isn't anywhere near free: You have to move your physical body to the bookstore to read. For one thing, you can't likely do that at 3am. (And certainly not in your pajamas.) You can't do it from your bed, couch, or desk, without getting up. You have to spend time to move your body down there, which might be 10min-30min each way; 20-60min round trip, plus say 10min to find the book, a place to sit, etc; call it 30-70min. If you value your time at say, $10/hr, that's $5-12. Then there's the cost of transportation. If the library/bookstore is three miles away, 6mi. round trip, and gas costs $2.50/gal., and you get 20mi/gal., that's another $.75. The IRS figures driving a car costs $.405/mile in repairs, wearing it out, etc., so that's another $2.40. So you're at something like $8-15 to go read a "free" book.
Really -- if it were that free, people would do a lot more of it.
Yet reading a free copy from a CDS doesn't have those limitations. It is much closer to $0, actually and truly free. THAT's the problem.
(4) You can't pass on a "free" copy you read in the store or from the library. You have to leave the book at the bookstore (or buy it); you have to return the book to the library. Reading a book in digital form that was stolen from a CDS, you could pass that copy on to others by email, via a web page, P2P software, etc.
So, bottom line, bookstore/library reading isn't really free. CDS copies are essentially free, and that's the problem. They're too convenient to read free.
This is one of the reasons we formed the COCOA Association ( http://www.copyrightaccess.com/ ), to make more copyrighted work available. (Note, COCOA does not inhibit indexing and searching and returning text snippet search results -- just what page images can be displayed.) If you support this, please sign our petition at http://www.petitiononline.com/cocoa/petition.html -- thanks!
Dr. Andrew Burt,
Chair, The COCOA Association
Commercial or non-commercial use doesn't enter into it.
If the work in question is under copyright, you can't copy and redistribute it; if it's not, then you can. The only exceptions would be the fair use provisions, and I don't think that they would cover you reproducing an entire book, even if it was for non-commercial use: if you're a university professor you can't copy an entire textbook and give them out to your students. That's a non-commerical use, but it's still illegal. There might be some exceptions for purely personal use -- some type of "format shifting" perhaps, like OCRing it and running it through text-to-speech and putting the result on your iPod -- that you could make a good case for if you already owned the book in print form, but non-commerical use normally isn't an excuse for infringement. Despite public opinion to the contrary, there is no exception to a copyright holder's exclusive rights for "non-commercial" uses.
Also, if you made a derivative work from something that was out of copyright, and then went and tried to sell it, only the portions that you contributed anew would be protected, the existing stuff doesn't change. That's not to say that you couldn't sell it (if it's out of copyright you don't even have to change anything to sell it, you can go and print out anything you want from Project Gutenberg and try to sell it, if you think you'll get any buyers), but you wouldn't have any recourse against someone taking your changed version, editing out all of your changes, and selling it themselves.
There is a fairly good introduction to these concepts here. Or read it straight from the U.S. Code.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
The focuses of OCA and PG are really quite different: PG is most interested in preserving the essential information of a book (ie, its text), while OCA's interest is in preserving the form of the book (ie, its fonts, pages format, coloration, even down to the yellowing of the pages). That having been said, there's a lot each can do for the other (and has!).
The Archive has archived most of PG's material, because even though the Books department of The Archive is focussed mostly on preserving books, The Archive as a whole is interested in preserving just about any information it can, and the PG data is definitely of interest.
When the The Archive's Scribe software processes the book images into its various format (jpg, djvu, pdf, flippy, et al), it OCR's the book's text. This text then becomes part of generating some of the other formats. It will be really trivial for PG to obtain this text for any book it wants to incorporate into their dataset.
qv: intlepisode00jamearch. The interesting files here are intlepisode00jamearch.txt which is just the OCR'd text, and intlepisode00jamearch_djvu.xml which is the OCR'd text with layout information (which has been useful to me in developing software which auto-corrects some OCR errors -- where the text is on the page often offers valuable hints for choosing the right heuristic for guessing the right text).
A quick side note on the differences between Google's and OCA's efforts that I haven't seen talked about much -- Google's main advantages in their bookscanning efforts are their wealth and fame, while The Archive's main advantages are experience, familiarity, and scanning technology.
Traditional book-scanning technologies are expensive and slow (which makes doing a lot of books, fast, that much more expensive, because you have to hire more people to do more books in parallel), but Google has enough money to throw at the problem that this is less of an issue. Google's fame means they can bring powerful partners onboard with a smile and a handshake, including some of the most prestigious libraries in the nation.
The Archive has been involved in scanning books and making them available online for several years now (qv The Million Books Project). This experience has shaped the processes used in the acquisition and scanning of books, as well as the technology used in their storage, indexing, and presentation. Furthermore, libraries around the world have grown familiar with The Archive over the years. That, and The Archive's good track record, make it a powerful rallying point for partnerships and alliances, and have given it more experience in facilitating such relationships. Finally, partially due to the limits of existing book-scanning solutions, and partially due to The Archive's limited budget, it has facilitated the development of two independent low-cost, reliable, high-quality book-scanning systems: The Scribe (developed in-house at The Archive) and the Kirtas Robot (developed at Kirtas, a Canadian company).
Many of the books scanned for the Million Book Project using traditional scanning methods are really lousy, sometimes to the point of being unreadable. These new scanning systems dramatically improve the quality of the end product, while equally dramatically reducing the cost-per-page. This means that more scanning systems can be purchased for more libraries (avoiding the per-library capital outlay problem), and more books can be scanned more quickly within a given budget.
Obviously, Google and OCA can benefit from co-operation, as each has a lot to offer the other. I'd be surprised if Google didn't join the OCA, eventually, if for no other reason that to gain access to the books of the >100 OCA
What would be the equivalent local rate for scanners in Europe?
;-)
Probably about $35 an hour, they'd only work seven hours, three days a week, and they'd be on strike half the year anyway. And you can't fire any of them.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."