Is Free Software Ready For E-publishing?

johanneswilm writes "Over more than 3 years I have been writing my PhD thesis on the politics of Nicaragua. Being the most professional system for PDF generation, I went with LaTeX, and, to make the text accessible for the editors, I used the LyX editor. Now that the publication date comes near, I found I had to spend considerable time creating a script to convert the manuscript to formats such as Epub as none of the available tools were quite ready to do it automatically. Is LaTeX only good for writers in the natural sciences? Is the open source community boycotting ebook formats, as Richard Stallman has proposed? Are there better tools to do the same?"

Comments

You should had compared

[zget]

Being the most professional system for PDF generation, I went with LaTeX

Now that the publication date comes near, I found I had to spend considerable time creating a script to convert the manuscript to formats such as Epub

It sure sounds the like most professional system!

The truth is, if you want your job done, you look at the merits of every possible program without considering if it's open source or not. There are good software like Apache that are mostly good for web hosting (unless you have certain requirements). Then there is lots of shit. The same is true for proprietary software tho. But if you want to get something real done, it's just stupid to limit yourself to only open source OR proprietary software. Pick the best tool for the job.

Re:You should had compared

[TheRaven64]

My fourth book (Go Phrasebook) is due to be published soon. I send 3 copies to the publisher:

• Print, PDF, generated by pdflatex. Black and white with crop marks.
• eBook PDF, generated by pdflatex, with cross-referencing hyperlinks and colour for the syntax highlighting.
• XHTML, generated by some code I wrote, with hyperlinks and cross references and semantic markup in the code listings generated by clang for [Objective-]C[C++].

The publisher can then just tweak the CSS for the ePub (XHTML) version. A C code listing has lots of span tags marking words as keywords, typedefs, macro uses, variables, and so on. How these are presented is controlled from the CSS, as is all of the rest of the styling.

The important thing is to make sure you separate content from presentation. If you use a lot of TeX markup in your chapters, then it's hard to use anything other than [La]TeX to typeset it. If you use simple semantic markup with all of the macros defined in a document class, then you can parse the same markup easily with something else and then transform it into some other format.

You could use some sort of XML and generate TeX from it, but typing XML is horrible. I like to work in vim, and with a couple of macros entering LaTeX is really easy.

Re:You should had compared

[udoschuermann]

Actually it makes perfect sense if you do not wish to support the mindset of proprietary software, and the dependencies and liabilities that such an association creates (and not just for yourself, either!) Obviously there is a price to be paid for refusing to run with wolves, hence the posted question: Is there a way to accomplish what needs to be done using only FLOSS (Free/Libre/Open Source Software)?

Re:You should had compared

[dgatwood]

I'm in the process of preparing three novels, and I took a rather different approach. I'm using AppleWorks as my editing tool (because that's what I started with). From there, I'm:

• exporting the results as HTML,
• running a nasty piece of Perl (is there any other kind?) to turn AppleWorks HTML into valid HTML,
• running another nasty piece of Perl to translate that HTML into DocBook plus a bunch of custom tags, and
• running dblatex with some rather large custom style files and custom xsl to translate that into suitable code for xelatex.

The advantage to this process is that I have valid XHTML on the way into the process, and with minimal effort, I could go from there to usable ePub content.

If I were starting from scratch on a new document, I would be writing XHTML with some custom CSS as my source format. That would give me full semantic markup capabilities (which would give me slightly more flexibility than I have now, but not enough to convince me to ditch the convenience of editing in a WYSIWYG editor for this project). Then, I would tweak my XHTML to DocBook translation tools to handle that. So for ePub, it would just require containerizing the source material, and for nice PDF output, it would just require using the translator bits I already have.

Of course, none of this is a general solution. Novels and theses are rather different in the way you write them, and the former was made a lot more difficult by LaTeX being designed so heavily for typesetting things like the latter. There are also a lot of flaws in LaTeX stemming out of the core design that make for less than ideal typesetting.

For example, as far as I can tell, there is no good way to indicate that a section break (three stars, for example) cannot be the first thing on a page, and that at least two lines of the content above it must be pulled down with it. The closest you can do is to make it part of an unbreakable container with the previous whole paragraph, but that doesn't really do what you want most of the time.

Similarly, it does not support proper widow control. LaTeX supports widow line control—that is, saying that you cannot have fewer than the last n lines of a paragraph on a page/column by themselves. What it lacks is widow paragraph control—that is, treating a single-paragraph line as though it were the last line of the previous paragraph for widow calculation purposes. The result is poor typography if a page break happens to fall near the end of a chapter. You can fix this by hand-tweaking the TeX markup to force a page break earlier, but I assert that good page layout software should produce good layout by default without hackery.

And LaTeX does not handle UTF-8 very well at all. In my XHTML to DocBook translator, I've had to hack in extra markup (\hspace{0.001pt}) after em dashes, en dashes, and hyphens to force TeX to allow the line to wrap. Without that hack, I get serious overfull hbox problems.

I could probably go on for hours about all the problems I've encountered, but it suffices to say that I'm not impressed by TeX, and at several points, I was tempted to build my own PDF generator using WebKit and CSS styles, but I didn't want to spend the time. (Yet, in hindsight, it would have been faster than trying to force TeX to behave.) That said, if you started with something like the hyphenator project, someone could probably replace most of TeX with a few hundred lines of JavaScript, and that would almost inarguably produce better typesetting with a lot more flexibility (particularly given that pretty much every programmer already understands JavaScript and the DOM).

calibre

[Anonymous Coward]

calibre is a free and open source e-book library management application that can convert to and from most of ebook formats. And does a pretty good job at it.

http://calibre-ebook.com/

RE: Stallman link

[craftycoder]

Stallman complains about DRM and a lack of anonymity with eBooks. It seems to me that this story relates very closely to legally acquired music. While it is still difficult to legally acquire digital music anonymously, it is easy to get it without DRM. I suspect books will follow this same path if consumers value it as a feature. In practice there is in fact little anonymity in the purchase of real books as everyone wants you to swipe your "club" card and use your debit card to make the purchase but his point is well taken. The option to buy an unpopular book in secret is nice.

With time and interest from consumers we will have DRM free books.

Anonymity is dead and gone and I didn't even get an invitation to the funeral. We should all mourn it's passing.

Re:Easy solution

[TheRaven64]

Going through PDF is horrible. LaTeX contains a lot of semantic markup. ePub is XHTML, which is a form of semantic markup. PDF is a presentation format. So, you start with semantic markup, discard it all, and then try to generate it again by magic.

You end up with something that looks vaguely like the PDF, but loses most of the semantic information (e.g. section / chapter breaks). Worse, you often don't want the ePub version to look like the PDF - they're aimed at different form factors.

Pandoc

[bbk]

I've found pandoc (here: http://johnmacfarlane.net/pandoc/) to be very useful for generating PDF/ePub/LaTeX/etc from Markdown formatted text files.

Re:Pandoc

[metamatic]

Indeed. For a PhD thesis on the politics of Nicaragua, I'd have started with markdown and then converted that to ePub and LaTeX.

Boycott? I Think the Tools Merely Lack Maturity

[eldavojohn]

Others have told me that the financial gain of publishing an academic book may be up to 700 USD. In comparison to current Scandinavian wages that really means very little, so I don’t think that earning another 700 USD should be a motive to restrict the access to one’s thoughts.

First of all I would like to commend you and thank you for this sentiment.

Is the open source community boycotting ebook formats, as Richard Stallman has proposed?

I don't understand, Stallman decries e-book formats that aren't open. There are many open e-book formats--including ePub. Granted, there are tools out there that allow you (to varying degrees of success like Calibre) to crack and convert to these formats but why bother? As you can see in that table, most everyone supports PDF. You are misunderstanding Stallman's gripe. It's not that we are boycotting e-books, it's that e-book makers are trying to carve out their own proprietary section of the electronic market, reader and creators included. So let them take their ball and play elsewhere. As you noted in your blog, this isn't the only problem:

Most ebook-readers out there so not implement the Epub-standard perfectly. That means that although one has an Epub that follows all the standards, one can be quite sure that it will not display properly on all the readers. Kovid Goyal, the creator of the Calibre ebook management software has done a good job in creating conversion scripts that create Epubs for all the different readers. Unfortunately they do this by breaking compatibility with the standard, and many distribution sites will only check whether your Epub complies to the standards and not whether the book will actually look good in the reader.

Most readers handle PDF, I would just stick to the output of LaTeX. I might suggest that your expectations are misdirected at the open source community and might be better directed at the makers of readers that apparently force you to break standards. It's the IE6 conundrum all over again.

Stallman didn't suggest boycotting ebook formats, just the DRM associated with them (big surprise there). The problem you are experiencing is that sometimes it's difficult to go from one open standard to another. The tools are lacking in maturity and I'm guessing that since my Android phone can easily display PDFs for me that there's not a lot of people demanding this ePub support that apparently needs multiple flavors for each device (and Calibre helps you with this). The tools exist but they'll only get you so far and I think the really special stuff that LaTeX does well is what you'll find yourself needing to fine tune in the end product. Look at how long it's taken LaTeX to get that beautiful and I think you'll discover that making a magical cure-all converter to ${random format} can be a non-trivial task.

If you start a kickstarter and get your university to donate hosting to making an open free market for any academic papers in any open format, I'd definitely throw in $20 (I've spent about $200 on kickstarter in the past two years). Either that or maybe throw your lot in with arxiv and work with them to fund more format support?

Pandoc

[cyocum]

The solution to your problems is Pandoc which can convert LaTeX to EPUB if you like. Now, it will probably take some fiddling on your part with the output but it very much smooths the process.

Re:Open source solves problems programmers have

[TheRaven64]

I've not published anything in a journal for a couple of years, but in computer science every journal worth reading accepts PDF submissions and either provide a LaTeX style, asks for your LaTeX source to edit themselves, or tells you which standard LaTeX style to use. It's a good first check for a journal - if they don't encourage LaTeX submissions, they probably suck. Apparently the same is true in mathematics and physics, but less so in other subjects. In the humanities it's common for journals to require MS Word documents (and place insanely strict requirements on the formatting of the bibliography that are trivial with BibTeX and very hard with MS Word, from what I've heard).

Re:...PROFIT!!

[Khan Fused]

1. Realise no scripts exist for problem
    1,1 Realize that someone writing a thesis on Nicaraguan politics may not know how to program
    1.2 Begin learning to program
    1.3 Spend more time learning to program
2. Write scripts
    2.1 Divert time from PhD thesis to write scripts
    2.2 Spend more time (diverted from PhD program) learning to program sufficiently to write workable scripts to solve stated issue
3. Release scripts as open source
    3.1 Fail to complete PhD thesis in time due to time spent programming

Re:Easy solution

[digitig]

The trouble is, PDF is a pretty rotten format for e-readers, because it's all page-layout oriented and so produces output that doesn't scale well for different screen formats and text sizes. It's the wrong format for the job. And DVI has pretty much the same problems. The problem isn't that free software isn't ready for ePublishing -- Calibre and Sigil do the job well. The problem is that there's a disconnect between the assumptions laTeX makes about a document and the assumptions that are valid for ePublishing, Sorry if it's restating the blindingly obvious, but you didn't want the best system for PDF generation, you wanted the best system for PDF and EPUB generation, and that probably isn't laTeX.

RMS not boycotting e-books

[spf13]

While he states "We must reject e-books until they respect our freedom." He also outlines 7 things amazon's e-books do that violate this freedom. Fortunately epub is the most widely accepted e-book format and it has none of these 7.

1. Available anonymously.
2. Standard ownership applies.
3. License determined by vendor, but many have very liberal licenses including CC and public domain.
4. Open format based on html.
5. Lending rules same as physical book.
6. No inherent DRM (though Adobe has a version compatible with DRM).
7. No one can remotely delete it any more than any other file on your computer.

RMS isn't against e-books. He's against amazon's approach to e-books.

Re:...PROFIT!!

[dstar]

4. Realize this is exactly what happened to Knuth.
        4.1 Take consolation in the fact that at least it's just a thesis, not the next volume of TAOCP.

Re:through HTML

[gatzke]

There are some things LyX is better at than pure LaTeX code.

I can see the current version of my figures, not just rely on the file name.

I can add references from a list instead of trying to remember what labels I have used.

I can search bib items and add / order citations easily.

I can make complex tables without forgetting some damn }

I can generate and view a new version of my document in a single keypress.

I can see my equations without having to mentally render them, while still using most of my TeX knowledge (\alpha _12 in LyX is the same as \alpha_{12})

Students can make the transition from Word a little more readily. Remember, LyX is not WYSIWYG, it is WYSIWYM (what you mean) so the on screen representation is close to the final but not exact.

Plus you have access to tons of menu options that you may not be aware of. I learn more about LaTeX by using and exploring LyX. And you can always use pure code if you want, for any fancy stuff.

Re:When copyleft restricts which tools may be used

[Anonymous Coward]

Be careful: sometimes, especially in cases of works under a "copyleft" or "share-alike" license, a work's copyright license limits which tools for the job are lawful

What are you talking about? This would only be relevant if we were discussing templates and packages to be embedded as part of the document, it has nothing to do with software.

(If you still don't understand: GPL/GFDL/CC-SA only affect derivative works [derivative as defined by copyright law], a document made in MS Word is not subject to the copyright of MS Word unless you decompile Word and paste the code into the document. This simple fact is why you can (and many people, most obviously Apple, do) compile proprietary applications using GCC. DISCLAIMER: I'm talking about copyright law here, not contract law. If the MS Word EULA [which is a contract rather than a copyright license] says that all MS Word documents must be copyrighted a certain way then that may be a problem, fortunately I'm not aware of any applications that have such a clause)

Re:Calibre?

[ConceptJunkie]

Because PDF to ePub conversion generally gives you pretty awful results. Nothing against Calibre. I use it. But most PDFs I've tried to convert for my Nook Classic have had less than stellar results: readable if you're lucky, but not nicely formatted. And if there are embedded images, all bets are off.

Re:Easy solution

[digitig]

No, the results won't look better. I have poor eyesight, so I choose a large text size on my eReader. If the original is EPUB then the text reflows smoothly and it's all nicely readable. If it's PDF it doesn't, and the results look like crap. The solution you propose means anticipating the individual requirements of every potential user, and producing a customised PDF for that user. What's more, if I'm in bright light then I can move to a smaller text size to see more at one time, but doing it your way I'd need two copies of the file (and some way of synchronising the bookmarks and annotations). We've moved beyond the age of one-size-fits-all, but PDF hasn't. LaTeX doesn't seem to have, either. Essentially, you need to separate content from presentation, which neither PDF nor laTeX does, although there is work on moving laTeX towards that.

Re:...PROFIT!!

[jedidiah]

I don't think any of you twits realize how much work goes into any PhD thesis.

A little programming overhead is not going to be that much of a burden really.

This is why most stuff gets invented. It's really not that much of a tragedy when people who don't specialize in selling a particular technology to others have to develop solutions for themselves involving that technology.

If real people thought like you weenies then we never would have had the original killer app for the PC.

Transparent

[tepples]

This would only be relevant if we were discussing templates and packages to be embedded as part of the document

In the case of documents under the GFDL, the copyright license requires that those who distribute copies of the document also make copies available in a "Transparent" form, one editable using free software. So those who make derivative works have to make derivative works available in a "Transparent" form. Or perhaps I misunderstood "Transparent" in the GFDL; what am I missing

Re:Easy solution

[digitig]

Yes, you've pointed out the drawbacks with deploying as pdf, which I agree are real. And I think I've pointed out the advantages of pdf, and the drawbacks of ebook formats as they are currently implemented. If you're an author and you care or want to control what the document actually looks like, then pdf (or a bunch of images) is your only option.

No, it isn't an option. PDF doesn't do that unless you also control the device on which it is displayed. When I view a PDF with large text on my eReader I'm damn sure that what I see isn't what the author intended (not all authors can be that demented, surely). If you are trying to do that then you have overstepped your role as an author. If you think you have succeeded then you should try talking to your users (especially ones with visual impairment). PDF does have it's uses, but that isn't one of them.

But one of the main reasons for LaTeX is exactly to separate content from presentation, so I think you're misinformed about that

I used to use laTeX a lot, and was a member of the TUG. LaTeX is better than raw TeX, in terms of separation of content and presentation, but most raw TeX is still there in LaTeX, and LaTeX commands such as \textwidth, \baselineskip, \raisebox (everything to do with boxes, in fact), \vspace, \textbf and so many other laTeX constructs are about presentation, not content. You can write laTeX that separates content from presentation, but tools that claim to process laTeX can't assume that you have; they need to accept all legal laTeX, including all the presentation stuff.

and that point doesn't apply to pdf, which is for consumption only, not for writing.

Internally, PDF is quite like DVI in terms of how it structures a page, and the content and presentation have been well and truly merged. PDF puts blocks in defined positions on the page, and the order of the blocks doesn't necessarily match the order of the content. That's why when you select text in a PDF you often get bits you don't want. And it's why it's hard to go from PDF to EPUP; it's not a simple translation, the software needs to understand the significance of relative positions of blocks of text, which is very far from trivial. Yes, it's a presentation format, but that means that you have lost information needed to make a robust EPUB file from it. A far better option is to start with EPUB and generate your PDF from it and a stylesheet. The only downsides are that free EPUB editing tools are not well developed (unless somebody can point me to one that I've missed) and that EPUB enforces a linear reading sequence (but you're going to have to deal with that anyway if you're going to produce EPUB).

Re:PDF is fine

[Grishnakh]

No, it doesn't.

If all you want to do is download a file and print it on Letter-sized paper (or A4, assuming the PDF is in A4), then PDF is great.

However, if you want to view it on a screen, especially a screen that's smaller than letter-size, it sucks. Maybe you haven't noticed, but ebook readers are all smaller than letter-size paper, so it's physically impossible to view a PDF page on an ebook reader without either panning, or shrinking it. Panning around to read a page is annoying, and shrinking it will make it difficult or impossible to read (depending on the font size and the ereader's algorithm), plus it's even worse if the viewer has poor eyesight and prefers larger fonts.

This is the entire reason that ebook formats were invented, so that readers could dynamically resize and re-flow text, instead of being stuck with a fixed page size. Of course, with PDF, instead of defaulting to Letter size, you could format your document for a page size equal to the ebook reader's screen size, and make it look great on that ebook reader, but only that one. They don't all have the same size screen, so you'll need different PDFs for every single ebook reader out there, which flies in the face of the "Portable" aspect that PDFs tout. Plus you'd still need one in Letter size for anyone who wants to print out the document.