Linux and OSS to Aid the Library of Congress

flakeman2 writes with a link to Linux.com article about Linux's new role at the Library of Congress. The national archive of books is looking to begin an ambitious digitization project, aimed at getting some rare and crumbling documents into the public record online. These will include "Civil War and genealogical documents, technical and artistic works concerning photography, scores of books, and the 850 titles written, printed, edited, or published by Benjamin Franklin. According to Brewster Kahle of the Internet Archive, which developed the digitizing technology, open source software will play an 'absolutely critical' role in getting the job done. The main component is Scribe, a combination of hardware and free software. 'Scribe is a book-scanning system that takes high-quality images of books and then does a set of manipulations, gets them in optical character recognition and compressed, so you can get beautiful, printable versions of the book that are also searchable,' says Kahle." Linux.com and Slashdot.org are both owned by OSTG.

Comments

Help the Library of Congress save American History

[JanusFury]

For the past few months Microsoft has been dispatching crack teams of special operatives into the past to alter the course of American History for their benefit, in hopes of eventually transforming the United States into the New Microsoft Empire. But little do they know, a world-weary Librarian and Ex-Marine at the Library of Congress won't stand for that shit. He's put together a team of agents in hopes of reversing the damage to the timestream before it becomes irreversable. Together with Agents Linus Torvalds (Technology Specialist - Special power: x-ray glasses), Donald Trump (Logistics Specialist - Special power: nuclear fusion comb-over) and Stephen Hawking (Quantum Physics Specialist - Special power: medusa glare), he just may be the only hope for American History's future.

more info

[wwmedia]

more info in this interesting article http://www.nytimes.com/2007/03/10/business/yourmon ey/11archive.html?ei=5090&en=9bf0874841a9d705&ex=1 331182800&adxnnl=1&partner=rssuserland&emc=rss&

Re:Hmm...

[rednuhter]

RaTFA (note the lowercase "a" for "all")
"the Internet Archive has migrated Scribe entirely to Linux, and Windows support has been dropped."
Seems focused on Linux to me.

It's only natural

[donglekey]

So many technologies have been made specifically to hold libraries of congress.

No matter how you look at it

[zappepcs]

this has got to cause some flying chairs in Redmond.

Arguably one of the most important repositories of information in the U.S. is about to be available via OSS software and not MS products. For all the efforts that MS put out in Mass. this has got to be a kick in the face! Just wow!

All copyrighted works should be held

[rtb61]

They should expand the role so that all copyrighted works are held at the Library of Congress. It would certainly save the confusion over who holds what rights to what content. At the same time, unless congress wants to hold and distribute material of questionable moral quality, the copyright law could be amended to limit the protections of copyright to those works that do actually further the arts and the sciences as defined in the constitution.

The revisions to the law would not be infringing freedom of speech, in fact by allowing the free copying of works that did not further the arts or the sciences it would be limiting copyrights impact upon the freedom of speech. If people are really concerned about the quality of content, they should remember that eliminating the profit motive will have a substantial impact upon the amount of questionable content that is out there including movies, music, pictures and literature. Most of the members of the RIAA and the MPAA have a total disregard for the harm their content cause to society, let them feel some of the pain, wipe out the copyright protections on some of their more divisive content ;).

Re:All copyrighted works should be held

[forkazoo]

At the same time, unless congress wants to hold and distribute material of questionable moral quality, the copyright law could be amended to limit the protections of copyright to those works that do actually further the arts and the sciences as defined in the constitution.

Sir, you pre-suppose that morally questionable articles do not serve to further the arts or the sciences. I protest most heartily. Every modern technology is served by serving pornography, as we all know. Let us firstly ponder the case of VHS and BetaMax to settle that matter in a stroke. As for the arts, I think it is clear to say that such works as the Seduction of Misty Mundae served to elevate the erotic art form in any manner of ways, at least so much as any diserotic work served to also further the diserotic arts. Or, should you feel a need for some further example, I might offer up the Lord of the G Strings as an additional point. Rather than endevouring the hold the plot to a dramatic study of youthful exploration, as in the case of Seduction, Lord served to offer up comedic commentary and parody against the institions of our society. In so doing it clearly also furthered the arts.

Fie apon you sir, that youw ould so readily dismiss Misty Mundae's films on mere moral grounds. Fie, indeed.

Now, I just need to torrent me a copy of Spiderbabe. I heard that shit is hot.

Re:All copyrighted works should be held

[vivaoporto]

"At the same time, unless congress wants to hold and distribute material of questionable moral quality, the copyright law could be amended to limit the protections of copyright to those works that do actually further the arts and the sciences as defined in the constitution."

Uh-uh. Let's repeat the same errors from the past, keeping what the current generation deems "of excelent moral quality", and censoring everything else, just like some works of Michelangelo were. People must to remember, what is of questionable moral quality for some is perfectly acceptable (and even desirable) for others, specially when the benefit of time is give, and that's the idea of archiving for posterity.

Re:All copyrighted works should be held

[jimicus]

At the same time, unless congress wants to hold and distribute material of questionable moral quality,

Stop right there.

When the purpose of your organisation is, to put it in very simple terms, "catalogue everything", you can't start making exceptions on moral grounds on the simple basis that what constitutes "questionable moral quality" today may be totally different tomorrow. Furthermore, who gets to define "questionable moral quality"? The closest anyone's ever come to creating such a definition is to say "Well, I can't actually come up with a concrete definition but I knows it when I sees it".

The most important part is not free software

[Ed Avis]

As the article says, the OCR itself is still done with proprietary software. I wonder if Google is using Tesseract for their digitization efforts. It would be cool if the original raw scanned images could also be archived and available for download - then you could print your own copy of the book, check the OCR for errors, or even do some weird genetic algorithm thing to make a LaTeX style that typesets the text in the same format as the original book.

Excellent project!!!

[2Bits]

This is absolutely cool. There are a lot more places where "brittle books" are laying around, waiting to be digitalized and distributed to the whole world. And as the technologies used in this project are going to be refined and improved, and eventually released, everyone will benefit.

The question now is: would they accept technical contributions from the public (I mean, OS geek communities), just like other open source projects? I know a lot of people would be eager to join. How about a SETI-like system to harness the power of desktop computers around the world to help with image processing and OCR? Hey, I got 4 decent desktop computers that can contribute at least 8 hours/day each.

The sad part of digitization.

[Lethyos]

Eventually we will have no physical record of these writings and may someday learn from the digital copies that Benjamin Franklin, George Washington, and others had offered enthusiastic support for wiretapping and other forms of electronic surveillance.

Re:Scribe?

[rs232]

I mean this one ...
http://sourceforge.net/projects/scribesw

Re:Scribe?

[Ceriel Nosforit]

Ah, thank you. Clearly your google-fu is superior to mine. ;)

Oh, it will be.

[Grendel Drago]

Project Gutenberg uses plenty of scans from American Memory to make their etexts--they do pretty much what you describe. At the lowest level, they make a plaintext copy, but they also do formatting and in-text hyperlinking: for instance, linking footnotes to their references, or index page numbers to anchors in the text. (See the HTML version of this etext to see what I mean.) Browse to a random book from this random collection, and you'll see what the LoC provides for their collections currently. As Brewster Kahle will be involved, you might want to see what projects he's done and how they're provided: a random book from the Million Book Project is available as a DjVu document, as well (badly) OCR'd text.

Quality as well as quantity, please

[Ankh]

The books I've looked at have been scanned at a resolution that's more or less adequate for OCR, but isn't really adequate for reproducing fine woodcuts, and is hopeless at metal engravings. I've found from my work on fromoldbooks.org that anything less than 1200 dpi generally produces pretty poor results for images, so that, for example, you can't read the signatures of the artist and engraver, still less compare engraving styles. It would be sort of like having a paraphrase of the text instead of the actual words.

It does, of course, vary a lot depending on the style of image. Bold illustrations for children's books, for example, do better at, say, 800dpi greyscale or colour. Fine steel engravings with lines at, say, less than a tenth of a degree from horizontal (they were done by hand after all) and that come out only a couple of pixels wide even at 1200dpi just turn into gray mush with weird banding artefacts until you go to a higher resolution (I use 2400dpi). There's a widely-cited study indicating that an "ultra-high" scan resolution of 400dpi is more than sufficient, based on an extremely small sample of images.

The damage that's done by poor quality digitization is that it makes it harder to justify doing a better job in the future.