Slashdot Mirror
Just One Page a Day
Charles Franks writes "Two years ago I started building an online proofreading system as a way to help Project Gutenberg (PG) get more books online: Distributed Proofreaders (DP). The concept is simple, we scan books and load the image and OCR output for each page into the online system. Next, proofreaders compare the OCR text to the image making any corrections as necessary, each page gets looked at twice. Finally the output from the site is massaged into a PG e-text and submitted to PG for posting to the archive. Now, nearly 600 books and a lot of PHP code later, we have snuggled into our new home which is graciously provided by the Internet Archive and Project Gutenberg. Now that we have 'real' resources available to us (the original site ran on a Pentium 200 over my 128kbps upstream cablemodem) I would like to invite the online community at large to help us put even more books online. To this end I would like to ask everyone to do 'Just One Page a Day'. Thank you, Charles Franks"
And start reading a page!
After that come back and you may continue();
... which is renowned for it's spelling prowess? ;)
A)bort, R)etry or S)elf-destruct?
Sounds like Gary Condit's plan for extramarital affairs.
A. Rightmann
Is there any worth-while open source OCR software? How about reasonably priced closed source OCR software for *BSD or Linux?
I'm shure that buy askin teh Salshdot crowd (esp. the editturs) to help, yule improove jamatically teh kwality off you're output.
:-)
Try NetBSD... safe,straightforward,useful.
The only works that go into PG are works in the public domain. While publishers sell dead-tree copies still, they have no copyright over the original text contained within. (Which is why these works are typically available through multiple publishers.
XML is like violence. If it doesn't solve the problem, use more.
Project Gutenberg only publishes books that are out of copyright. That means Dickens is okay but you wont find the latest Stephen King
It helps if you read the FAQ list.
Due to copyright laws, it is only legal to do this with older books (copyrighted 75 or more years ago). As a result, Project Gutenberg is mostly comprised of the "Classics."
Imagine the kids 200 years from now reading |-|uc||_3b3rry F1|\||\|.
(That hurts my brain just trying to type it in...)
--- I wish I could hear the soundtrack to my life. That way I'd know when to duck.
I think a better use of time would be to have all these programmers here develop a better OCR. Then you wouldn't need the proofreading and could just feed books into the scanner. I mean there are lots of things wrong with OCR and reasons why it can't be absolutely perfect, but it CAN bet better. If we just write one line of code a day each we'll have better OCR in no time.
The GeekNights podcast is going strong. Listen!
Copyrights aren't perpetual. The Gutenberg project aims to publish books that are no longer, or have never been under copyright.
And you probably are. The best efforts of our duly elected Congressional representatives notwithstanding, copyright still does expire. After that, a work passes automatically into the public domain. That means there are hundreds of thousands of books available.
In fact, if you've previously seen the classics online, they probably came from this project, which has been around for almost as long as I can remember.
"Patriotism is your conviction that this country is superior to all other countries because you were born in it." -- GBS
Have each client do the OCR (if you can find GPL software). Or maybe there's a company willing to donate it. That way you could farm out most of the processing too.
While publishers sell dead-tree copies still, they have no copyright over the original text contained within.
What? You mean to suggest that you have an actual example of a publisher making money without tyranny over the content?
Gasp!
The books that are being converted are whatever people feel like contributing.
Don't think your favorite authors are being represented? Can you demonstrate that the work is out of copyright? Make the conversion yourself!
Doing the hard work yourself is the best way to guarantee your interests are represented.
teeker
Very good idea.
Will there be any support for proofing in other languages (french, spanish, arabic, etc...)?
What about books published in other countries. Would we be able to post those books if they're not copyrighted in the US but copyrighted in other countries? or vice versa.
What if they kept track of every time the human reader finds an OCR-error. Couldn't you then build a profile of what words/phrases/letters the OCR software has the most problems with?
Then, couldn't you just selectively have the humans review the highest probably error prone sections of a book, instead of every single word of every single page?
What do you think?
Nuff said.
Don't you mean run a compare tool in the background using CPU idle time right?
You don't actually want us to read a
page of literature do you?
In order to make the proofing faster, maybe you could OCR a document 2 or 3 times, and then have only the disagreements proofread.
We use omnipro here at work, and I'm surprised at how well it works, even recreating page formats.
Of course, it doesn't work 100%, but it sure does get about 95%. If you were to OCR a document 2-3 or more times, and most of it was identical, it would save a lot of time if you had humans going over only the parts that the different OCRs didn't agree on.
Steve Lefevre
Computers are useless. They can only give you answers.
-- Pablo Picasso
The new congress might extend copyright protection to Shakespeare's great great great great great great great great great great great great great grandson's nephew's out of wedlock kid's son whose paternity is in question.
---
When you come to a fork in the road, take it! --Yogi Berra--
You mean a more communal approach than an oligarchy of "editros" that can't spot day-old duplicates? Great idea!
These are humans comparing identical books to text.. if they have the IDENTICAL book they won't have this problem.
Gutenburg often has published the same 'book' but of different publications due to slight variations in the text.
Not to mention malapropisms!! :-)
http://www.dictionary.com/search?q=duplicity&d b=*
I like the first definition better!
Alison
"It is a miracle that curiosity survives formal education." - Albert Einstein
I've just proofed four pages, a mix of modern English, quoted Cockney and religious babble (Jonah 4:13, 9 etc.)
OK it's only four pages, but the errors I've corrected so far have been when the scan has been poor and the OCR software has had to make a guess.
Though the web page was last updated in July, I find several happy references (and some less happy) to "Clara," a GPL'd OCR program.
Here's the web page: http://www.claraocr.org/index.html
timothy
jrnl: http://tinyurl.com/c2l8yr / foes: http://tinyurl.com/ckjno5
Well, copyrights weren't perpetual. Whether they will be or not remains to be seen.
Liberty uber alles.
Someone needs to do a google search on " Public Domain". Public domain is there for a reason. Just as Copyright is available to give the artist a means of supporting himself, it was never ment to last his entire life. The purpose is to give the artist an incentive to work, current copyright law fails in this respect because an artist only needs to create one successful work and can immediatly switch to being a leech on society for the rest of his (and his childrens, and childrens childrens) life. Having the works pass into the Public domain is a good idea for two reasons:
1. It is for the greater good of society as other people build on earlier works.
2. It keeps the artist busy as they were supposed to have to keep releasing work to feed themselves as their early work passed into the public domain, just like any other job.
I read the internet for the articles.
How about this.... use an open source speech synthesis tool/API that can play these text books (especially as more get added) over a PDA, laptop, etc while cruising in on the way to work and home. Something like:
o plug, just did a quick freshmeat search)
http://www.cstr.ed.ac.uk/projects/festival/
(n
would be pretty cool to get some good novels read to you w/o buying the tapes.
Sure, it starts as just one a day. But, before you know it, you're doing two, then five, then ten.
You stop going out with friends or even returning their calls, personal hygiene takes a back seat and even Counter Strike and Warcraft III become unappealling. And, finally, after countless chapters and hundreds of pages you realise that you're friends were right: you're an addict.
Just one page a day, huh? Yeah, right.
Opium. Pot. Cocaine. Now pages.
It might not be your older brother's drug, or your Daddy's or your grandfathers, but, trust me, this stuff can be dangerous.
Do what I do. Just say no.
"Accept that some days you are the pigeon, and some days you are the statue." - David Brent, Wernham Hogg
Is there a list of books that are out of copyright and perhaps the status of those books on the Gutenberg Project website or anywhere else?
This a great project... But after doing my first page I found a couple of possible enhancements.
r oof_ / 1000))
Add a "Quality" stat for each person. Base it on the number of things that were missed(another words, the number of things that the second-string proofer finds).
Use more than just two proofers. Have one "First String" proofer, who could be anybody, but have two second string proofers (who both get the output of the first string proofer). If the second string proofers have any differences in their output(with the exception of white space), then another second string proofer should be used. Only proofers with a certain quality rating(slightly higher than what a newbie's would be) should be able to do the second string proofing.
The "User rating" should be a combination of the number of pages done and the quality rating of those pages. Note that quality rating would only be increased by doing first string proofing. Page count would go up for any proofing.
Quality could be a float, starting at 1.0 for newbies. Every page that is completed and has a second-string person check would then go into a calculation like:
_new_quality_ = _old_quality_ + (0.01 - (_num_differences_between_their_proof_and_final_p
Thus, for every page proofed that requires NO corrections by the second string the user's quality would go up by 0.01. ( 0.01 - 0/1000 = 0.01 )
if there were more than ten errors in the proofing, their quality would go down ( 0.01 - 10/1000 = 0.00 ), (0.01 - 20/1000 = -0.01)
Have a threshold of 1.10 or some such for second string proofers... That way it would require the user to do at least 10 perfect pages, or 20 pages with 5 errors, etc, before they could do the second string proofing.
Obviously, make sure that the second string proofer can't see who the first string proofer is.
The "User Rating" (mentioned above) could just be a multiplication of the Quality and Page Counts...
Sticks and Stones may break my bones, but copyright will always protect me.
charlz has a workflow diagram for the works that go through his site. As you see, each book has a project manager, who has final processing/proofing responsibilities.
Also, I'm not sure you get the idea of two rounds of proofing. They don't see different versions of a corrected page -- the first one sees the straight OCR output (or, sometimes the project manager will do some automated corrections on it first) and then the first round proofer edits the text. Then, when all the pages have gone through the first round, the second round proofer reads the text as it was edited by the first round proofer. This helps because it builds off the edits of the first round proofer and allows the second round proofer to perhaps catch things not caught in the first round.
When proofreading, you're never going to capture all the mistakes with one pair of eyes. A distributed proofreading effort is very beneficial to the goals and efforts of Project Gutenberg, and I applaud the efforts of all those who have proofed even one page.
Having said that, I've done over 300 (under a different name).
"The evil of the world is made possible by nothing but the sanction you give it." -- Ayn Rand
Reading the blurb at the page-a-day site, it says ASCII only where bold is converted to ALL CAPS, the English pound symbol is rendered as "L," etc. No preservation of figures, drawings, or photos.
This seems very short sighted to me. Devices that can only display ASCII are becoming rarer and rarer. Why not, instead, store docs in some sort of SGML format to handle the special markup (which must be rare) and then down convert to ASCII when needed.
I've tried reading these things on my Palm. Very difficult. But if I could get a nice typeset PDF version, that would be a whole different story (no pun intended).
How long before someone writes a script to hit "Save and get another Page" and they shoot to the top of the ladder claiming to have proofread 13,450,213 pages per day...
I've used both clara and gOCR. Both are not yet working well enough to actually use to scan books..
OCR Engines are not email programs. You can't just add a line of code and all of a sudden it works better. Usually you have to spend time developing a complicated algorithm. Usually this is more than a line of code. Then you have to test it against known text (ground truth) to make sure it's a benefit, rather than a problem over a broad selection of pages. It's quite often the case that something that improves one page makes another worse.
Actually, having people make verifications against the OCR results establishes the ground truth which someone could use to improve the OCR engine so by doing a Page a Day, you are helping to make future Open Source OCR engines better.
I am not a number! I am a man! And don't you
OK, here's mine:
#include stdio.h
next...
My beliefs do not require that you agree with them.
I have a few books that are old enough to be well out of copyright (and obscure enough not to be found online already), and for a while I have been considering typing them in. OCR would be a lot easier, but getting a good image from a flatbed scanner would seriously damage most of these books. Even a handheld scanner would be impractical in some cases, and a digital camera seems even less likely to work. Is there any reasonable way to scan in pages from something like a 100+ year old 1.5" thick wire-bound paperback book that only opens about 60 degrees before putting up a fight?
Copyright law is supposed to give incentive to create, for the betterment of society, and allow the creator to derive direct benefits as a reward. An artist who has created a work so successful that (s)he can live on it indefinitely has arguably provided a suitable level of betterment to society.
Saying that copyright law is an incentive to "work" is accepting mediocracy. Artists who produce works that society values more highly should (have the opportunity to) receive more benefits.
On the other hand, I don't necessarily agree that copyright should last the lifetime of the creator (although there are strong arguments for this in the case of a natural person). But what is a "fair" limit?
Is 5 years enough? Almost certainly not. Many authors only achieve popularity after 10 or more years, and then make a fair amount of money off increased sales of their older works. A good number accept this as a risk, and plan to use this phenomenon to their benefit - work up a good number of titles with varied content, and you'll pull more readers, who are then likely to try some of your other titles.
Is 20 years enough? Maybe. But some of our best-loved authors were 15-20 years ahead of their time in terms of what readers wanted.
Is life enough? Strangely, no. If an aging star has just completed his/her autobiography, concludes the publishing deal, and dies ... well, the family could well be screwed.
Maybe the answer lies in a compromise, rather than an all-or-nothing approach. Copyright over a work lasts for the greater of 10 years or the creator's natural life (which gets very interesting when we get eternal life medications ...). But some rights fall away after the LESSER of those two times, such as exclusivity over derivative works (but not translations).
This allows society to (culturally) enrich itself by building on a work after a shorter amount of time, while the creator (and/or family) can still derive value from the original work for a longer time.
In the case of books this is easily understood: author writes book; 10 years later other people can write preludes and sequals, extend the world and characters, etc; 30 years later author dies and original book falls into public domain.
i-name =twylite [http://public.xdi.org/=twylite], see idcommons.net
It seems like every few years I turn around and notice that some massive archive collection gets sued, goes out of business, has funding pulled, gets tangled in legal action, has a university board go into panic mode, etc. and suddenly it disappears without warning or notice to the frustration of many. I'm certain you also can name a number of services, collections, and resources that spontaneously vanished when hosted at friendly sites. History has proven that despite best intentions, nothing lasts forever unless we go out of our way to protect it.
So that work isn't lost or destroyed, are any of the mega-sized projects replicated elsewhere in the event that a "it'll never happen" situation crops up to this unsuspecting resource?
MY GOD! A story where nitpicking grammar and spelling is *ON* topic.
This'll be a fun one to read through.
Do you Gentoo!?
It looks like the texts01.archive.org/dp site is holding up fairly well! If you cannot get through today, though, please check back later. Slashdot effect aside, it's usually quite speedy and has a decent 'net connection. If you want to keep informed of current events, get on one of our mailing lists via (when it's not slashdotted) our subscriptions page.
Dr. Gregory B. Newby // 919-962-8064
Chief Executive and Director
Project Gutenberg Literary Archive Foundation http://gutenberg.net
A 501(c)(3) not-for-profit organization with EIN 64-6221541
gbnewby@ils.unc.edu
...Not many, but there are some Project Gutenberg books that are copyrighted and distributed with the author's permission.
Also, Project Gutenberg of Australia publishes a number of works that are out of copyright in Australia, but still under copyright in the U.S. It is a copyright infringement for readers in the U. S. to download these works, which include, among others, Hervey Allen's _Anthony Adverse_(1933), F. Scott Fitzgerald's _The Great Gadsby_ (1944), Khalil Gibran's _The Prophet_ (1923), D. H. Lawrence's _Lady Chatterley's Lover_ (1928), all of George Orwell's novels, most of Virginia Woolf's, etc. etc.
Not exactly "the latest Stephen King" but a lot newer than Dickens.
"How to Do Nothing," kids activities, back in print!
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
If they're looking for proofreaders here, the project is in deep trouble...
This page accidentally left blank
are actually the preferred way to proof text. A project to create "The Collected Works of Edmund Spenser" is headquartered here, and the English-types were looking for people to work on some software for them. The current most accurate way to create an electronic copy is to hire people without even a passing familiarity with the alphabet you are targeting, train them to identify the letters themselves (using the font you're targetting, which may be very much non-standard, esp. for work as old as Spencer's), and have them enter it in character by character. You then have another illiterate person do the same, and have 1 editor (English graduate student) check both copies. Then any differences have to be handled by another editor (English PhD), and the final copy signed off by yet another editor (PhD).
A very very expensive way to do it.
See, an illiterate person won't introduce any bias into the text. They will faithfully duplicate any spelling mistakes that they find. In the case of an English scholarly collection, the mistakes are amoung the most important part, since they can identify different print runs, and how language shifts over time.
As a side note, the software project is hopeless. The best that cann be managed is to automate the administration of their current systems--no OCR will ever meet the level of accuracy that their current system provides.
...but first read the Proofing FAQ on the site and save yourself some confusion:
http://texts01.archive.org/dp/faq/ProoferFAQ.htmlEspecially read section 5 for some of their typesetting-to-ASCII conventions which would be non-obvious otherwise.