Is Free Software Ready For E-publishing?

johanneswilm writes "Over more than 3 years I have been writing my PhD thesis on the politics of Nicaragua. Being the most professional system for PDF generation, I went with LaTeX, and, to make the text accessible for the editors, I used the LyX editor. Now that the publication date comes near, I found I had to spend considerable time creating a script to convert the manuscript to formats such as Epub as none of the available tools were quite ready to do it automatically. Is LaTeX only good for writers in the natural sciences? Is the open source community boycotting ebook formats, as Richard Stallman has proposed? Are there better tools to do the same?"

Comments

You should had compared

[zget]

Being the most professional system for PDF generation, I went with LaTeX

Now that the publication date comes near, I found I had to spend considerable time creating a script to convert the manuscript to formats such as Epub

It sure sounds the like most professional system!

The truth is, if you want your job done, you look at the merits of every possible program without considering if it's open source or not. There are good software like Apache that are mostly good for web hosting (unless you have certain requirements). Then there is lots of shit. The same is true for proprietary software tho. But if you want to get something real done, it's just stupid to limit yourself to only open source OR proprietary software. Pick the best tool for the job.

Re:You should had compared

[TheRaven64]

My fourth book (Go Phrasebook) is due to be published soon. I send 3 copies to the publisher:

• Print, PDF, generated by pdflatex. Black and white with crop marks.
• eBook PDF, generated by pdflatex, with cross-referencing hyperlinks and colour for the syntax highlighting.
• XHTML, generated by some code I wrote, with hyperlinks and cross references and semantic markup in the code listings generated by clang for [Objective-]C[C++].

The publisher can then just tweak the CSS for the ePub (XHTML) version. A C code listing has lots of span tags marking words as keywords, typedefs, macro uses, variables, and so on. How these are presented is controlled from the CSS, as is all of the rest of the styling.

The important thing is to make sure you separate content from presentation. If you use a lot of TeX markup in your chapters, then it's hard to use anything other than [La]TeX to typeset it. If you use simple semantic markup with all of the macros defined in a document class, then you can parse the same markup easily with something else and then transform it into some other format.

You could use some sort of XML and generate TeX from it, but typing XML is horrible. I like to work in vim, and with a couple of macros entering LaTeX is really easy.

Re:Easy solution

[TheRaven64]

Going through PDF is horrible. LaTeX contains a lot of semantic markup. ePub is XHTML, which is a form of semantic markup. PDF is a presentation format. So, you start with semantic markup, discard it all, and then try to generate it again by magic.

You end up with something that looks vaguely like the PDF, but loses most of the semantic information (e.g. section / chapter breaks). Worse, you often don't want the ePub version to look like the PDF - they're aimed at different form factors.

Pandoc

[bbk]

I've found pandoc (here: http://johnmacfarlane.net/pandoc/) to be very useful for generating PDF/ePub/LaTeX/etc from Markdown formatted text files.

Boycott? I Think the Tools Merely Lack Maturity

[eldavojohn]

Others have told me that the financial gain of publishing an academic book may be up to 700 USD. In comparison to current Scandinavian wages that really means very little, so I don’t think that earning another 700 USD should be a motive to restrict the access to one’s thoughts.

First of all I would like to commend you and thank you for this sentiment.

Is the open source community boycotting ebook formats, as Richard Stallman has proposed?

I don't understand, Stallman decries e-book formats that aren't open. There are many open e-book formats--including ePub. Granted, there are tools out there that allow you (to varying degrees of success like Calibre) to crack and convert to these formats but why bother? As you can see in that table, most everyone supports PDF. You are misunderstanding Stallman's gripe. It's not that we are boycotting e-books, it's that e-book makers are trying to carve out their own proprietary section of the electronic market, reader and creators included. So let them take their ball and play elsewhere. As you noted in your blog, this isn't the only problem:

Most ebook-readers out there so not implement the Epub-standard perfectly. That means that although one has an Epub that follows all the standards, one can be quite sure that it will not display properly on all the readers. Kovid Goyal, the creator of the Calibre ebook management software has done a good job in creating conversion scripts that create Epubs for all the different readers. Unfortunately they do this by breaking compatibility with the standard, and many distribution sites will only check whether your Epub complies to the standards and not whether the book will actually look good in the reader.

Most readers handle PDF, I would just stick to the output of LaTeX. I might suggest that your expectations are misdirected at the open source community and might be better directed at the makers of readers that apparently force you to break standards. It's the IE6 conundrum all over again.

Stallman didn't suggest boycotting ebook formats, just the DRM associated with them (big surprise there). The problem you are experiencing is that sometimes it's difficult to go from one open standard to another. The tools are lacking in maturity and I'm guessing that since my Android phone can easily display PDFs for me that there's not a lot of people demanding this ePub support that apparently needs multiple flavors for each device (and Calibre helps you with this). The tools exist but they'll only get you so far and I think the really special stuff that LaTeX does well is what you'll find yourself needing to fine tune in the end product. Look at how long it's taken LaTeX to get that beautiful and I think you'll discover that making a magical cure-all converter to ${random format} can be a non-trivial task.

If you start a kickstarter and get your university to donate hosting to making an open free market for any academic papers in any open format, I'd definitely throw in $20 (I've spent about $200 on kickstarter in the past two years). Either that or maybe throw your lot in with arxiv and work with them to fund more format support?

Re:...PROFIT!!

[Khan Fused]

1. Realise no scripts exist for problem
    1,1 Realize that someone writing a thesis on Nicaraguan politics may not know how to program
    1.2 Begin learning to program
    1.3 Spend more time learning to program
2. Write scripts
    2.1 Divert time from PhD thesis to write scripts
    2.2 Spend more time (diverted from PhD program) learning to program sufficiently to write workable scripts to solve stated issue
3. Release scripts as open source
    3.1 Fail to complete PhD thesis in time due to time spent programming

Re:Easy solution

[digitig]

The trouble is, PDF is a pretty rotten format for e-readers, because it's all page-layout oriented and so produces output that doesn't scale well for different screen formats and text sizes. It's the wrong format for the job. And DVI has pretty much the same problems. The problem isn't that free software isn't ready for ePublishing -- Calibre and Sigil do the job well. The problem is that there's a disconnect between the assumptions laTeX makes about a document and the assumptions that are valid for ePublishing, Sorry if it's restating the blindingly obvious, but you didn't want the best system for PDF generation, you wanted the best system for PDF and EPUB generation, and that probably isn't laTeX.

RMS not boycotting e-books

[spf13]

While he states "We must reject e-books until they respect our freedom." He also outlines 7 things amazon's e-books do that violate this freedom. Fortunately epub is the most widely accepted e-book format and it has none of these 7.

1. Available anonymously.
2. Standard ownership applies.
3. License determined by vendor, but many have very liberal licenses including CC and public domain.
4. Open format based on html.
5. Lending rules same as physical book.
6. No inherent DRM (though Adobe has a version compatible with DRM).
7. No one can remotely delete it any more than any other file on your computer.

RMS isn't against e-books. He's against amazon's approach to e-books.

Re:...PROFIT!!

[dstar]

4. Realize this is exactly what happened to Knuth.
        4.1 Take consolation in the fact that at least it's just a thesis, not the next volume of TAOCP.

Re:When copyleft restricts which tools may be used

[Anonymous Coward]

Be careful: sometimes, especially in cases of works under a "copyleft" or "share-alike" license, a work's copyright license limits which tools for the job are lawful

What are you talking about? This would only be relevant if we were discussing templates and packages to be embedded as part of the document, it has nothing to do with software.

(If you still don't understand: GPL/GFDL/CC-SA only affect derivative works [derivative as defined by copyright law], a document made in MS Word is not subject to the copyright of MS Word unless you decompile Word and paste the code into the document. This simple fact is why you can (and many people, most obviously Apple, do) compile proprietary applications using GCC. DISCLAIMER: I'm talking about copyright law here, not contract law. If the MS Word EULA [which is a contract rather than a copyright license] says that all MS Word documents must be copyrighted a certain way then that may be a problem, fortunately I'm not aware of any applications that have such a clause)

Re:Easy solution

[digitig]

No, the results won't look better. I have poor eyesight, so I choose a large text size on my eReader. If the original is EPUB then the text reflows smoothly and it's all nicely readable. If it's PDF it doesn't, and the results look like crap. The solution you propose means anticipating the individual requirements of every potential user, and producing a customised PDF for that user. What's more, if I'm in bright light then I can move to a smaller text size to see more at one time, but doing it your way I'd need two copies of the file (and some way of synchronising the bookmarks and annotations). We've moved beyond the age of one-size-fits-all, but PDF hasn't. LaTeX doesn't seem to have, either. Essentially, you need to separate content from presentation, which neither PDF nor laTeX does, although there is work on moving laTeX towards that.