Universal Ebook Format Debated

Amy Hsieh writes "A well-known ebook industry expert, Jon Noring, recently wrote an interesting article for eBookWeb, formally calling upon the ebook industry to adopt a single universal ebook distribution format. Right now there's a plethora of essentially incompatible ebook formats, and this format 'babel' is hampering the growth of the ebook industry. In the article, Mr. Noring proposes a promising open-standards candidate which appears to meet a list of basic requirements: The Open eBook Forum's OEBPS Specification. Andy Oram, a Linux programming editor for O'Reilly, wrote an interesting reply to the article that should also be read." On the other hand, Noring's proposal has also met with some skepticism elsewhere.

Comments

Sounds easy to me...

[WestieDog]

What about .txt?

Re:Sounds easy to me...

[DrXym]

Have you ever tried to read something from Project Gutenberg in text format? It's horrible! Try downloading something like the King James Bible and go mad as you slog through in 80 character monospaced print. Furthermore, say goodbye to an pictures or diagrams, e.g. illustrations in Alice in Wonderland.

A standardized rich text format is absolutely required, one which defines document structure so you get all the goodness like chapters, quotations, sidebars, footnotes, images etc., but doesn't impose how it should be laid for the most part, or the layout is specified by an accompanying style sheet.

Something like docbook might be suitable, but some of its more gross or esoteric things would have to be pruned or moved into different levels of support for the sake of simplicity.

Well

[The-Bus]

If the software pirating industry can all agree on plain text "NFO" files with ASCII-painted flames, dragons eating your group's logo, and pot leaves surrounding shout-outs to your boys on efnet, I think the slightly more professional and law-abiding ebook industry can agree on a standard format.

My ebook format

[RenQuanta]

Is Project Gutenberg and a Palm Pilot.

Better readers needed

[Baumi]

I don't think any format will get Ebooks to catch on until we have reader hardware that makes reading those books at least as pleasant as reading a paper book.

Here's hoping that all those e-paper efforts will produce something usable soon.

I dont know....

[AmoebafromSweden]

Why should i choose a format that have all possibilities to have DRM included in the future thus allowing only one read. And will require Electricity to read.

This is especially true for for factbooks who are often used as reference and not to be read just one time.

So far Ebooks cant beat the paper version in portability, convenience and ease of use.

Paperbook still seems more favorable to me.

Re:I dont know....

[RobotRunAmok]

Why should i choose a format that have all possibilities to have DRM included

Because you might want to read an e-book of something NOT in the public domain, e.g, a current novel, and few authors or publishers are going to render their wares into a format that is going to end up on free P2P. There needs to be some way to ensure that money changes hands.

You were planning on paying for the books you read, weren't you? Or is this all just an exercise in seeing how we can best Napsterize the publishing indutstry?

Like the word processing industry

[sould]

Right now there's a plethora of essentially incompatible ebook formats, and this format 'babel' is hampering the growth of the ebook industry.

Yup - just like there's a plethora of essentially incompatible word processing formats - hampering the growth of the office/word processing market.

But the industry doesn't matter to one player - only their market share does.

The only way to really win this sort of thing is to persuade all (or at least most) consumers to boycott products that deliberately break compatability with standards.

But how likely is that to happen?

Why propose a different standard? PDF!!!

[adzoox]

Why do we have to propose a different standard? Just because it's a slightly different industry than computers, most eBooks are going to be READ on computers. Wouldn't PDF be perfect? Doesn't Adobe have an eBook PDF format? If I'm not mistaken a PDF can be locked and encrypted. This is also the same as DVD+/- Minus is the better standard, but companies with deeper pockets and greedier "proprietary minded" philosophy back Plus. Standards make sales PERIOD!

I think this was the mistake of the iTunes Music Store. While not terrible (actually slightly better quality) AAC is not as universal a standard as Mp3 or even Ogg. There are WAYS to encrypt and secure those formats. Napster, just before its demise, had figured out how to secure MP3's that were downloaded from it's system.

Re:Why propose a different standard? PDF!!!

[Tsu-na-mi]

PDF only meets half the goals IMHO.

You really have 2 types of documents, and the chosen format needs to support them both well. PDF is well-suited to Magazines or other content with lots of graphics where the layout of the page is important. Such content could be standardized one or more standards of 'page size' for this kind of thing is chosen.

The other type of content is more like a novel, where it's just a very plain free-flow of text. Here, it would be nice to have the device render layout, allow the user to up the font size, etc. Something along the lines of plain vanilla HTML 1.0 would fit well. PDF explicitly positions everything, so it would be bulkier and less flexible.

As I see it, 2 formats are needed, one with set layout and positioning, and one for free-flowing text. PDF and a stripped-down HTML would seem to fit the bill nicely.

Survival of the best-marketed, I guess

[heretic108]

I had a taste of incompatible e-Book formats when I got my first colour Palm.

Sadly, there were better (open) formats using better compression and rendering, losing out to closed formats with big marketing push.

The format that ultimately prevails will not necessarily be the best. It'll be the format pushed by those with the greatest marketing skills/budget, and the one which gives them the greatest control over how their works are used.

It wouldn't surprise me if authors are already signing e-book distribution deals which forbid them from releasing in rival formats.

One of these days, the masses will choose software and data formats according to quality and freedom.

But something within me suspects that the Pope will convert to Islam, and the Jews will profess the divinity of Christ first.

FictionBook XML

[ironhide]

There also is this e-book xml format:
http://haali.cs.msu.ru/pocketpc/FictionBo ok_descri ption.html

I use his excellent HaaliReader as a text reader on my pocketpc (fullscreen, landscape mode). There are also html2xml and word2xml tools on his site.

The ONLY Universal EBook Format!

[Anonymous Coward]

The Project Gutenberg Etexts should so easily used that no one should ever have to care about how to use, read, quote and search them ...

This has created a need to present these Project Gutenberg Etexts in "Plain Vanilla ASCII" as we have come to call it over the years.

The reason for this is simple. . .it is the only text mode that is easy on both the eyes and the computer.

However, this encourages others to improve our etexts in a variety of ways and to distribute them in a variety of the available media, as follows:

Once an etext is created in Plain Vanilla ASCII, it is the foundation for as many editions as anyone could hope to do in the future. Anyone desiring an etext edition matching, or not matching, a particular paper edition can readily do the changes they like without having to prepare that whole book again. They can use the Project Gutenberg Etext as a foundation, and then build in any direction they like.

Thus any complaints about how we do italics, bold, and the underscoring, or whether we should use this or that markup formula are sent back with encouragement to do it any ways any person wants it, and with the basic work already done, with our compliments.

The same goes for media. We have had a long-standing work ethic of providing our etexts in any medium people wanted: Amiga, Apple, Atari. . .to IBM, to Mac, to TRS-80. . .

However, now that our etexts are carried in so many BBS's, networks and other locations, it is easier to download the file in a manner that puts them in your format than we can make and mail a disk, so we don't really do that too much.

The major point of all this is that years from now Project Gutenberg Etexts are still going to be viable, but program after program, and operating system after operating system are going to go the way of the dinosaur, as will all those pieces of hardware running them. Of course, this is valid for all Plain Vanilla ASCII etexts. . .not just those your access has allowed you to get from Project Gutenberg. The point is that a decade from now we probably won't have the same operating systems, or the same programs and therefore all the various kinds of etexts that are not Plain Vanilla ASCII will be obsolete. We need to have etexts in files a Plain Vanilla search/reader program can deal with; this is not to say there should never be any markup. . .just those forms of markup should be easily convertible into regular, Plain Vanilla ASCII files so their utility does not expire when programs to use them are no longer with is. Remember all the trouble with CONVERT programs to get files changed from old word processor programs into Plain Vanilla ASCII?

Do you want to go through all that again with every book a whole world ever puts into etext?

The value of Plain Vanilla ASCII is obvious. . .so is very much of the value of most of the various markup systems we have in the world. But until some real standards arrive-- we would be limiting our options a great deal if we do not keep copies of all etexts in Plain Vanilla ASCII as well.

We don't have anything against markup. Not vice versa.

Alice in Wonderland, the Bible, Shakespeare, the Koran and many others will be with us as long as civilization. . .an operating system, a program, a markup system. . .will not.

This includes the many requests we have for compression in particular formats. There are only two formats we know of that are suitable for transfer to a wide general audience: Plain Vanilla ASCII (.txt files) and ZIPped files of them, (.zip files). Requests for other compression formats must be ignored as they are appropriate only for small portions of our target audience. However, (programmers take note: we will need help) we are planning to put some compression links on our files so they can be transmitted in any of an assortment compression formats on the fly. i.e. we should be able to generate any kind of file asked for, but we can keep only one copy of each etext on our servers. . .as the .Z compression format does in a similar manner today.

Re:The ONLY Universal EBook Format!

[shockbeton]

I'm happy you brought Project Gutenberg into the discussion. PG is a wonderful resource. When the project was nascent in the 1970s and 80s reading ASCII on your TRS-80 probably seemed pretty neat. But now that the PG dream of preserving and distributing the printed word through digital technology has stagnated into a dogmatic cult with the goal of preserving ASCII it's time to reevaluate the meaning of Project Gutenberg.

Those of us who are literate and computer savvy and have seen places other than the USA recognize the harm that reducing printed material to chunks of ASCII does. And far from mere loss of formatting or typographical embellishments much of the meaning of a text is destroyed when run through the chunky sieve of ASCII conversion. Most accented Roman characters cannot be rendered in ASCII. Non-Roman characters cannot be rendered in ASCII. Typographical features such as relative type size, style, and formatting are either lost entirely or reduced to the low-res rendering capabilities of monospaced ASCII. ASCII has no provision for rendering traditional methods of communicating typographically such as small caps, ligatures, distinction between hyphen, endash, and emdash, etc. despite the fact that virtually every printed text makes use of these features.

Digital technology has progressed without our friends at Project Gutenberg. There is an alternative to ASCII which is now standard to all major computing platforms: Unicode. From the unicode.org website:

Unicode provides a unique number for every character, no matter what the platform, no matter what the program, no matter what the language.

Encoding the PG texts in Unicode would require no extra effort on the part of the PG volunteers (well, those who have moved on from their TRS-80s, anyway).

Why not use technology that attempts to accomodate the typographical traditions inherent in your source material rather than reducing that material to fit an obsolete technology?

And even if you still cling to your belief in the infinite beauty, timelessness, and universality of ASCII, please stop using linefeeds every 70 characters within paragraphs. WTF do you Project Gutenbergers imagine we read these texts on TRS-80s?

Re:Babel?

[aldousd666]

I thought we already had a standard: HTML

Shows what I know.

A couple of side notes: And how can you not know what babel is? Babel: Tower of babel: a story from the bible where King Nebekenezur (there is no correct spelling for that in english, just commonly accepted ones) wanted to build a tower to god, so god being jealous, put a spell on everyone, and they all ended up speaking a different language. It's how the christians believe that there came to be multiple languages.

Now the website babelfish gets its name from 'The Hitchikers Guide to the Galaxy' by Douglas Adams, where the characters 'stick a babelfish in their ear' to act as a universal translator.

but...but...

[corian]

Different readers, different platforms, and different applications have different requirements!

Some uses want a format which is compact as possible. Some focus on readibility (switchable fots, etc.) Others -- facimile-style releases -- emphasize that the copy should as closely mimic the original work as possible. Formats can emphasize the syntactic structure of the text (sentences, paragraphs), or the structural qualities (line breaks, pages).

Even in their paper forms, books have different formats for different uses. Libraries prefer hardcovers, with durable bindings. Travlers prefer paperbacks, with small and light pages. Collectors pay extra for special editions, with quality supplies. Some readers prefer large-print copies, abridgements, or books on tape (in a choice of cassette tape or compact disc!)

Any format makes assumptions, and deletions. It's perfectly fine to have a multiplicity of formats. If its useable, and reasonably priced, people will buy it.

For me, the major hindrance to e-books is the price. Since there is no associated cost of the materials (paper/cardboard), printing, physical transportation, stocking space, and delivery, e-books should be [i]cheaper[/i] than physical books. But many of them are priced the same, or even high (you can check this at Amazon.) what's up with that?

What ebooks really need to take off...

[TallEmu]

... is paper. Seriously.

The nice thing about a book is that it doesn't have a power switch - it's actually relaxing to sit there and read it.

If it were possible to obtain a high speed printer capable of printing out "e-books" in the same form-factor as a normal book (ie double sided pages, standard size, neatly bound) then I for one would pay for *lots* more books (and paper, and ink.)

eBooks and DRM

[FinnishFlash]

DRM is a hot-potato, and rightfully so...

DRM Capability: Although end-users prefer not to purchase ebooks protected with DRM (Digital Rights Management), publishers are certainly interested in the DRM capability of the universal ebook format. Thus, the universal ebook format must allow inclusion of DRM protection technologies as needed.

Take 2 minutes and read this article from RMS

Right to Read

PDFs and html

[Anonymous Coward]

You have to be careful. Half of you are saying "I won't use this until e-books are as pleasant as paper books" and half of you are saying "why not use the standards that are already there? Just make the device do everything."

Don't you see these are at odds?

To make e-books as pleasant as real books, you're going to want to make them thinner and thinner in profile. You're going to want to make them run on a single lithium cell battery or AAA. You're going to want to drop all of the interface but the forward, back, and bookmarking buttons. You're going to want the computing device to be as close to nothing as possible, so you can put weight into making the device indestructible like a real book. You want to go to the store, buy the title, and have it just work, or go to Amazon and *know* your desired title is published in that format. That's the ideal, in the near term. It isn't a device that will easily accomodate PDFs and HTML and a number of other standards.

There's a lot to be said for plain text

[gidds]

• It's universal. Everything supports it, from PDAs to supercomputers.
• It's versatile. If properly formatted, it's reflowable on different screen sizes, fonts, layouts, &c. And it's perfect for other access methods.
• It supports most characters you'd find in books. The de facto standard is the Windows Latin-1 encoding, which has all the punctuation as well as accented characters. (Yes, I know, I know. But it's not just on Windows -- both my Mac and my Psion use it, for example.)
• It's editable. There are tons of tools already available, from spell-checkers in editors to complex analysis. I've written some of my own, for instance; one converts from American to British spelling, which is how I like to read my books.
• It has conventions for /italics/, *bold*, _underlining_, &c. Yes, at first, these may look clumsy, but I actually prefer them in many ways, as they're more precise; for example, you can differentiate between *word* *by* *word* and *all at once* highlighting (see the Jargon File for the difference).
• It's compact. Plain text files are smaller than HTML, PDF, RTF &c, sometimes by a lot; and when compressed in formats like PalmDOC (pdb) or TCR, they can be made even smaller and still usable directly.
• It's future-proof. Plain text has been around for decades, and will be with us for many more, long after DRM keys have been lost and proprietary apps have died.

Yes, of course some spiffy new format will have other advantages. But it's unlikely to gain quick acceptance. Plain text documents are everywhere, as are readers and other software. There are even online publishers selling text files. In fact, ASCII text is arguably the most successful electronic standard there is!

My proposed format

[Alsee]

I have developed a higly sophisticated format for storing books in a computer file.

Each character of the book is to be enciphered to a byte. I reserve the first 32 codes (0-31) for various system function characters. The next 32 codes (32-63) encipher the space character, various punctuation marks, and numerals. The next 32 codes encipher the capital alphabet and a few more punctuation characters. With the simple use of 00111111 binary mask 'A' maps to 1, 'B' maps to 2, and 'Z' maps to 26. Quite clever if I say so myself! Naturally the next 32 codes encipher the lowercase letters in the same manner. Using the very same 00111111 bitmask you find 'a' mas to 1, 'b' maps to 2, and 'z' maps to 26! Ingenious, isn't it?

To ensure compatibility with legacy computer systems values above 127 shall not be used.

I call this encoding Advanced Storage Cypherment Input Ideal - or A.S.C.I.I. Any file utilizing this encipherment is a Tagged eXchange Template. These files may be identified by the use of a .TXT extention.

-

www.wapnovel.com

[evilandi]

Expro: I would like to put up a server to serve up Gutenberg, etc. a page or so at a time for low-end WAP phones

I got bored last Christmas and did this.

www.wapnovel.com (WAP or desktop)

There's also an as yet unused discussion group at:

http://groups.yahoo.com/group/wapnovel

Another perspective...

[borschski]

My wife has a small publishing/consulting company that has taken us 16 years -- and a lot of investment and pain -- to build. She works her butt off gathering the content which she then publishes as print products and CD-ROM "ebooks".

She is devastated when she hears from someone that they've copied one of her color newsletters; made a "backup" of the CD-ROM ebook and someone else "happens to be reading it so I thought I'd call with a question"; and otherwise copies illegally (no...we don't have the funds to pursue them). She had an opportunity to publish a digital product in Asia and another in Latin America but these markets are notorious for buying *one* and suddenly hundreds or thousands appear (I could digress with a personal story when I was at a software company and saw this first-hand...but it's too long).

PDF is the best standard right now. Platform support for everything out there virtually; security; but there is no meaningful method of DRM that would protect a small businessperson AND make it relatively easy to move ebooks from device-to-device (I know that I would hate to have to remember codes from dozens of publishers; be locked in to one machine for viewing; or other cumbersome methods).

However, no protection = no incentive. I don't care if you're an recording artist seeing your music ripped off or someone like my wife struggling to grow a business. Why should my bride travel to Europe and domestically gathering content; pay correspondents and photographers; and publish a product in ebook format that is super-simple to copy and distribute?

This is why I'm struggling so hard with the whole discussion about ebooks; copyright; DRM and fair use. So some how, some way, we've got to come up with a solution that offers some sort of universal ebook format that content producers can agree on and users can live with.

My $.02....

The gaping flaw in their argument

[Dagmar d'Surreal]

"Right now there's a plethora of essentially incompatible ebook formats, and this format 'babel' is hampering the growth of the ebook industry."

Bullshit.

The problem is that the few people who actually still read books are not likely to be stupid people. On top of that, the people who are reading electronic formats of books are even less likely to be stupid people.

However, it would take rather dim consumers indeed to not see a problem with paying the exact same cost for an eBook as one would in a brick and mortar bookstore for a paperback... and strangely when I go to these eBook sellers online, I see exactly that. "Oh joy! Instead of paying $7.95 for that paperback over an Barnes & Noble, I can pay just $7.95 to download an electronic copy in a format that I probably won't be able to read again in 10 years because the format and it's reader will have been declared obsolete!"

The unwillingness of eBook publishers to see eBooks as something other than a way to increase sales profits by cutting out the middlemen of printing and shipping expenses is what is hampering eBook adoption.

On Beyond ASCII

[Creosote]

I understand the support in a lot of the comments here for the plain-vanilla ASCII Project Gutenberg approach to ebooks. Paradoxically, however, a simple ASCII conversion from print to digital form provides less assurance of future survivability and usability of your book than rendering it with the structured XML markup specified by the Open eBook standard (where well-formed XHTML is the least common denominator).

Why? Well, an ASCII text version of a printed book is really more like an analog facsimile than is a version in XML that has been tagged for structural features. Leaving aside issues of non-English characters, illustrations, and unusual typography, ASCII does a relatively poor job of capturing all of the structural conventions that exist in printed books. Books have copyright pages, tables of contents, chapter titles, subtitles, bylines, epigraphs, block quotations, footnotes, running headers and footers, citation lists, etc. ASCII can provide rough format equivalents of some of these, very poor equivalents of others. With an appropriate XML tagset, however, it's a relatively simple matter to tag most of the structural features of a book and then use stylesheets for presentational rendering. That's the whole assumption of the Open eBook specification.

Suppose you're in a world where all printed copies of Huckleberry Finn have been lost. You have two CD-ROMS that somehow you've managed to decode so that you can read the files and interpret their character sets. One of them contains the Project Gutenberg etext of the novel, an ASCII transcription. The other contains an XML encoding tagged according to a DTD from the Text Encoding Initiative, the current best standard for encoding literary (and many other) texts. It has all of the textual content of the PG version, as well as some that's missing (like the table of contents and the copyright page from the transcribed edition, which the PG version unaccountably omits). XML tags mark all the line and page breaks of the original. In addition, there are tags to mark quoted speech, unusual typography, words in foreign languages, and other significant features of the original. The CD-ROM contains the DTD used along with documentation on the tagset.

In this imaginary scenario, even if all of the XML documentation were missing it would be pretty straightforward for 31st-century programmers to strip out the tags and recreate the ASCII transcription. But with the documentation, it's possible to reconstruct something much closer to the original than the plain-vanilla PG version allows. And suppose your 31st-century archaeologist found a trove of TEI-tagged books on CD: with all of the structural tagging and metadata about authorship, publication dates, etc., a 31st-century librarian will be able to plug all of the books into a cataloging system that allows sophisticated searching. If instead you had a trove of plain-ASCII books, the best you could do with the collection would be simple full-text searches.

Leaving aside the sci-fi scenario, the reality is that our documents, over the next few decades, will move from format to format and be used for purposes that we can only guess at right now. Of course plain ASCII, or even proprietary formats, will be better than no documents at all. But the work involved in converting them will be a lot higher than if they are tagged in a well-documented, structured markup language.

Incidentally, there's already at least one project underway to take Project Gutenberg texts and add minimal XHTML or XML markup to capture structure and make them more readable via stylesheets. The Open eBook specification is just a more sophisticated way of doing the same thing.

It really isn't the formats...

[Mac Degger]

Though it might help, a universal format isn't what's hampering ebooks. It's price. I refuse to pay full price (and sometimes more!) for an etext of something I can get on paper; especially when I only get the etext.

Halfprice, maybe even quarter price, compared to deadtree is what ebooksellers should be going for...but if I still have to pay fullprice and I don't even get my dead tree, I'll pay the same for something slightly more tangible.

Now I would pay a couple of bucks (ie $2) more for a deadtree book which includes the etext.

This says it all, really.

[arafel]

>Although end-users prefer not to purchase ebooks
>protected with DRM (Digital Rights Management),
>publishers are certainly interested in the DRM
>capability of the universal ebook format.

So although people would prefer not to buy books with this stuff, we're going to put it in there anyway. Whatever happened to listening to your customers?

both PDF and HTML

[bcrowell]

The article proposes two incompatible criteria: typographic richness (nice formatting) and adaptability (letting the end-user change font sizes, reflowing the text to display it on small and large screens, etc.). PDF is a perfectly good format for typographic richness, and HTML is a perfectly good format for adaptability. There is never going to be a single format that's good for both, because they're mutually exclusive.

What I mean by that is that for many books with complicated layouts (including my own free books), it's simply not possible to reflow the text automatically. Consider an illustrated science textbook, which is the kind of work I do. There's a lot of hand-tweaking involved in getting everything laid out on the pages in the best possible way. And my books' layouts aren't even that complex compared to a lot of the big commercial textbooks out there. Some slashdotters may have used LaTeX to write academic papers, so they'll know how LaTeX tries hard to flow the text correctly, but ultimately it doesn't always do what you want, and either you or the publisher ends up doing more tweaking.

The solution isn't that complicated: if a publisher wants a book to make an electronic book available in both a a typographically rich version and an adaptable version, they can create both a PDF version and an HTML version. Of course, this is really an answer to a question that the publishers never asked. Most publishers don't want open formats, because open formats won't allow them to continue to steal away the rights of end-users, such as the right of first sale.