Is Free Software Ready For E-publishing?

johanneswilm writes "Over more than 3 years I have been writing my PhD thesis on the politics of Nicaragua. Being the most professional system for PDF generation, I went with LaTeX, and, to make the text accessible for the editors, I used the LyX editor. Now that the publication date comes near, I found I had to spend considerable time creating a script to convert the manuscript to formats such as Epub as none of the available tools were quite ready to do it automatically. Is LaTeX only good for writers in the natural sciences? Is the open source community boycotting ebook formats, as Richard Stallman has proposed? Are there better tools to do the same?"

Comments

You should had compared

[zget]

Being the most professional system for PDF generation, I went with LaTeX

Now that the publication date comes near, I found I had to spend considerable time creating a script to convert the manuscript to formats such as Epub

It sure sounds the like most professional system!

The truth is, if you want your job done, you look at the merits of every possible program without considering if it's open source or not. There are good software like Apache that are mostly good for web hosting (unless you have certain requirements). Then there is lots of shit. The same is true for proprietary software tho. But if you want to get something real done, it's just stupid to limit yourself to only open source OR proprietary software. Pick the best tool for the job.

Re:You should had compared

[udoschuermann]

Actually it makes perfect sense if you do not wish to support the mindset of proprietary software, and the dependencies and liabilities that such an association creates (and not just for yourself, either!) Obviously there is a price to be paid for refusing to run with wolves, hence the posted question: Is there a way to accomplish what needs to be done using only FLOSS (Free/Libre/Open Source Software)?

Re:You should had compared

[dgatwood]

I'm in the process of preparing three novels, and I took a rather different approach. I'm using AppleWorks as my editing tool (because that's what I started with). From there, I'm:

• exporting the results as HTML,
• running a nasty piece of Perl (is there any other kind?) to turn AppleWorks HTML into valid HTML,
• running another nasty piece of Perl to translate that HTML into DocBook plus a bunch of custom tags, and
• running dblatex with some rather large custom style files and custom xsl to translate that into suitable code for xelatex.

The advantage to this process is that I have valid XHTML on the way into the process, and with minimal effort, I could go from there to usable ePub content.

If I were starting from scratch on a new document, I would be writing XHTML with some custom CSS as my source format. That would give me full semantic markup capabilities (which would give me slightly more flexibility than I have now, but not enough to convince me to ditch the convenience of editing in a WYSIWYG editor for this project). Then, I would tweak my XHTML to DocBook translation tools to handle that. So for ePub, it would just require containerizing the source material, and for nice PDF output, it would just require using the translator bits I already have.

Of course, none of this is a general solution. Novels and theses are rather different in the way you write them, and the former was made a lot more difficult by LaTeX being designed so heavily for typesetting things like the latter. There are also a lot of flaws in LaTeX stemming out of the core design that make for less than ideal typesetting.

For example, as far as I can tell, there is no good way to indicate that a section break (three stars, for example) cannot be the first thing on a page, and that at least two lines of the content above it must be pulled down with it. The closest you can do is to make it part of an unbreakable container with the previous whole paragraph, but that doesn't really do what you want most of the time.

Similarly, it does not support proper widow control. LaTeX supports widow line control—that is, saying that you cannot have fewer than the last n lines of a paragraph on a page/column by themselves. What it lacks is widow paragraph control—that is, treating a single-paragraph line as though it were the last line of the previous paragraph for widow calculation purposes. The result is poor typography if a page break happens to fall near the end of a chapter. You can fix this by hand-tweaking the TeX markup to force a page break earlier, but I assert that good page layout software should produce good layout by default without hackery.

And LaTeX does not handle UTF-8 very well at all. In my XHTML to DocBook translator, I've had to hack in extra markup (\hspace{0.001pt}) after em dashes, en dashes, and hyphens to force TeX to allow the line to wrap. Without that hack, I get serious overfull hbox problems.

I could probably go on for hours about all the problems I've encountered, but it suffices to say that I'm not impressed by TeX, and at several points, I was tempted to build my own PDF generator using WebKit and CSS styles, but I didn't want to spend the time. (Yet, in hindsight, it would have been faster than trying to force TeX to behave.) That said, if you started with something like the hyphenator project, someone could probably replace most of TeX with a few hundred lines of JavaScript, and that would almost inarguably produce better typesetting with a lot more flexibility (particularly given that pretty much every programmer already understands JavaScript and the DOM).

Re:Easy solution

[TheRaven64]

Going through PDF is horrible. LaTeX contains a lot of semantic markup. ePub is XHTML, which is a form of semantic markup. PDF is a presentation format. So, you start with semantic markup, discard it all, and then try to generate it again by magic.

You end up with something that looks vaguely like the PDF, but loses most of the semantic information (e.g. section / chapter breaks). Worse, you often don't want the ePub version to look like the PDF - they're aimed at different form factors.

Boycott? I Think the Tools Merely Lack Maturity

[eldavojohn]

Others have told me that the financial gain of publishing an academic book may be up to 700 USD. In comparison to current Scandinavian wages that really means very little, so I don’t think that earning another 700 USD should be a motive to restrict the access to one’s thoughts.

First of all I would like to commend you and thank you for this sentiment.

Is the open source community boycotting ebook formats, as Richard Stallman has proposed?

I don't understand, Stallman decries e-book formats that aren't open. There are many open e-book formats--including ePub. Granted, there are tools out there that allow you (to varying degrees of success like Calibre) to crack and convert to these formats but why bother? As you can see in that table, most everyone supports PDF. You are misunderstanding Stallman's gripe. It's not that we are boycotting e-books, it's that e-book makers are trying to carve out their own proprietary section of the electronic market, reader and creators included. So let them take their ball and play elsewhere. As you noted in your blog, this isn't the only problem:

Most ebook-readers out there so not implement the Epub-standard perfectly. That means that although one has an Epub that follows all the standards, one can be quite sure that it will not display properly on all the readers. Kovid Goyal, the creator of the Calibre ebook management software has done a good job in creating conversion scripts that create Epubs for all the different readers. Unfortunately they do this by breaking compatibility with the standard, and many distribution sites will only check whether your Epub complies to the standards and not whether the book will actually look good in the reader.

Most readers handle PDF, I would just stick to the output of LaTeX. I might suggest that your expectations are misdirected at the open source community and might be better directed at the makers of readers that apparently force you to break standards. It's the IE6 conundrum all over again.

Stallman didn't suggest boycotting ebook formats, just the DRM associated with them (big surprise there). The problem you are experiencing is that sometimes it's difficult to go from one open standard to another. The tools are lacking in maturity and I'm guessing that since my Android phone can easily display PDFs for me that there's not a lot of people demanding this ePub support that apparently needs multiple flavors for each device (and Calibre helps you with this). The tools exist but they'll only get you so far and I think the really special stuff that LaTeX does well is what you'll find yourself needing to fine tune in the end product. Look at how long it's taken LaTeX to get that beautiful and I think you'll discover that making a magical cure-all converter to ${random format} can be a non-trivial task.

If you start a kickstarter and get your university to donate hosting to making an open free market for any academic papers in any open format, I'd definitely throw in $20 (I've spent about $200 on kickstarter in the past two years). Either that or maybe throw your lot in with arxiv and work with them to fund more format support?

Re:...PROFIT!!

[Khan Fused]

1. Realise no scripts exist for problem
    1,1 Realize that someone writing a thesis on Nicaraguan politics may not know how to program
    1.2 Begin learning to program
    1.3 Spend more time learning to program
2. Write scripts
    2.1 Divert time from PhD thesis to write scripts
    2.2 Spend more time (diverted from PhD program) learning to program sufficiently to write workable scripts to solve stated issue
3. Release scripts as open source
    3.1 Fail to complete PhD thesis in time due to time spent programming

Re:Easy solution

[digitig]

The trouble is, PDF is a pretty rotten format for e-readers, because it's all page-layout oriented and so produces output that doesn't scale well for different screen formats and text sizes. It's the wrong format for the job. And DVI has pretty much the same problems. The problem isn't that free software isn't ready for ePublishing -- Calibre and Sigil do the job well. The problem is that there's a disconnect between the assumptions laTeX makes about a document and the assumptions that are valid for ePublishing, Sorry if it's restating the blindingly obvious, but you didn't want the best system for PDF generation, you wanted the best system for PDF and EPUB generation, and that probably isn't laTeX.

Re:When copyleft restricts which tools may be used

[Anonymous Coward]

Be careful: sometimes, especially in cases of works under a "copyleft" or "share-alike" license, a work's copyright license limits which tools for the job are lawful

What are you talking about? This would only be relevant if we were discussing templates and packages to be embedded as part of the document, it has nothing to do with software.

(If you still don't understand: GPL/GFDL/CC-SA only affect derivative works [derivative as defined by copyright law], a document made in MS Word is not subject to the copyright of MS Word unless you decompile Word and paste the code into the document. This simple fact is why you can (and many people, most obviously Apple, do) compile proprietary applications using GCC. DISCLAIMER: I'm talking about copyright law here, not contract law. If the MS Word EULA [which is a contract rather than a copyright license] says that all MS Word documents must be copyrighted a certain way then that may be a problem, fortunately I'm not aware of any applications that have such a clause)

Re:Easy solution

[digitig]

No, the results won't look better. I have poor eyesight, so I choose a large text size on my eReader. If the original is EPUB then the text reflows smoothly and it's all nicely readable. If it's PDF it doesn't, and the results look like crap. The solution you propose means anticipating the individual requirements of every potential user, and producing a customised PDF for that user. What's more, if I'm in bright light then I can move to a smaller text size to see more at one time, but doing it your way I'd need two copies of the file (and some way of synchronising the bookmarks and annotations). We've moved beyond the age of one-size-fits-all, but PDF hasn't. LaTeX doesn't seem to have, either. Essentially, you need to separate content from presentation, which neither PDF nor laTeX does, although there is work on moving laTeX towards that.

Re:Easy solution

[digitig]

Yes, you've pointed out the drawbacks with deploying as pdf, which I agree are real. And I think I've pointed out the advantages of pdf, and the drawbacks of ebook formats as they are currently implemented. If you're an author and you care or want to control what the document actually looks like, then pdf (or a bunch of images) is your only option.

No, it isn't an option. PDF doesn't do that unless you also control the device on which it is displayed. When I view a PDF with large text on my eReader I'm damn sure that what I see isn't what the author intended (not all authors can be that demented, surely). If you are trying to do that then you have overstepped your role as an author. If you think you have succeeded then you should try talking to your users (especially ones with visual impairment). PDF does have it's uses, but that isn't one of them.

But one of the main reasons for LaTeX is exactly to separate content from presentation, so I think you're misinformed about that

I used to use laTeX a lot, and was a member of the TUG. LaTeX is better than raw TeX, in terms of separation of content and presentation, but most raw TeX is still there in LaTeX, and LaTeX commands such as \textwidth, \baselineskip, \raisebox (everything to do with boxes, in fact), \vspace, \textbf and so many other laTeX constructs are about presentation, not content. You can write laTeX that separates content from presentation, but tools that claim to process laTeX can't assume that you have; they need to accept all legal laTeX, including all the presentation stuff.

and that point doesn't apply to pdf, which is for consumption only, not for writing.

Internally, PDF is quite like DVI in terms of how it structures a page, and the content and presentation have been well and truly merged. PDF puts blocks in defined positions on the page, and the order of the blocks doesn't necessarily match the order of the content. That's why when you select text in a PDF you often get bits you don't want. And it's why it's hard to go from PDF to EPUP; it's not a simple translation, the software needs to understand the significance of relative positions of blocks of text, which is very far from trivial. Yes, it's a presentation format, but that means that you have lost information needed to make a robust EPUB file from it. A far better option is to start with EPUB and generate your PDF from it and a stylesheet. The only downsides are that free EPUB editing tools are not well developed (unless somebody can point me to one that I've missed) and that EPUB enforces a linear reading sequence (but you're going to have to deal with that anyway if you're going to produce EPUB).

Re:PDF is fine

[Grishnakh]

No, it doesn't.

If all you want to do is download a file and print it on Letter-sized paper (or A4, assuming the PDF is in A4), then PDF is great.

However, if you want to view it on a screen, especially a screen that's smaller than letter-size, it sucks. Maybe you haven't noticed, but ebook readers are all smaller than letter-size paper, so it's physically impossible to view a PDF page on an ebook reader without either panning, or shrinking it. Panning around to read a page is annoying, and shrinking it will make it difficult or impossible to read (depending on the font size and the ereader's algorithm), plus it's even worse if the viewer has poor eyesight and prefers larger fonts.

This is the entire reason that ebook formats were invented, so that readers could dynamically resize and re-flow text, instead of being stuck with a fixed page size. Of course, with PDF, instead of defaulting to Letter size, you could format your document for a page size equal to the ebook reader's screen size, and make it look great on that ebook reader, but only that one. They don't all have the same size screen, so you'll need different PDFs for every single ebook reader out there, which flies in the face of the "Portable" aspect that PDFs tout. Plus you'd still need one in Letter size for anyone who wants to print out the document.