Is Free Software Ready For E-publishing?

johanneswilm writes "Over more than 3 years I have been writing my PhD thesis on the politics of Nicaragua. Being the most professional system for PDF generation, I went with LaTeX, and, to make the text accessible for the editors, I used the LyX editor. Now that the publication date comes near, I found I had to spend considerable time creating a script to convert the manuscript to formats such as Epub as none of the available tools were quite ready to do it automatically. Is LaTeX only good for writers in the natural sciences? Is the open source community boycotting ebook formats, as Richard Stallman has proposed? Are there better tools to do the same?"

Comments

You should had compared

[zget]

Being the most professional system for PDF generation, I went with LaTeX

Now that the publication date comes near, I found I had to spend considerable time creating a script to convert the manuscript to formats such as Epub

It sure sounds the like most professional system!

The truth is, if you want your job done, you look at the merits of every possible program without considering if it's open source or not. There are good software like Apache that are mostly good for web hosting (unless you have certain requirements). Then there is lots of shit. The same is true for proprietary software tho. But if you want to get something real done, it's just stupid to limit yourself to only open source OR proprietary software. Pick the best tool for the job.

Re:You should had compared

[TheRaven64]

My fourth book (Go Phrasebook) is due to be published soon. I send 3 copies to the publisher:

• Print, PDF, generated by pdflatex. Black and white with crop marks.
• eBook PDF, generated by pdflatex, with cross-referencing hyperlinks and colour for the syntax highlighting.
• XHTML, generated by some code I wrote, with hyperlinks and cross references and semantic markup in the code listings generated by clang for [Objective-]C[C++].

The publisher can then just tweak the CSS for the ePub (XHTML) version. A C code listing has lots of span tags marking words as keywords, typedefs, macro uses, variables, and so on. How these are presented is controlled from the CSS, as is all of the rest of the styling.

The important thing is to make sure you separate content from presentation. If you use a lot of TeX markup in your chapters, then it's hard to use anything other than [La]TeX to typeset it. If you use simple semantic markup with all of the macros defined in a document class, then you can parse the same markup easily with something else and then transform it into some other format.

You could use some sort of XML and generate TeX from it, but typing XML is horrible. I like to work in vim, and with a couple of macros entering LaTeX is really easy.

Re:You should had compared

[Anonymous Coward]

The truth is, if you want your job done, you look at the merits of every possible program without considering if it's open source or not.

I don't think you realize where you are.

Re:You should had compared

[mkkohls]

I agree completely. Though as a student the best way it seems to get full featured software without pirating or spending an arm and a leg is open source. The quality is there if you know what you're looking for.

Re:You should had compared

[udoschuermann]

Actually it makes perfect sense if you do not wish to support the mindset of proprietary software, and the dependencies and liabilities that such an association creates (and not just for yourself, either!) Obviously there is a price to be paid for refusing to run with wolves, hence the posted question: Is there a way to accomplish what needs to be done using only FLOSS (Free/Libre/Open Source Software)?

Re:You should had compared

[gatzke]

For mathy stuff, LaTeX is great. I am not sure how to separate that content from the TeX itself. And I never understood why MathML did not just use TeX.

And I am a huge proponent of LyX. Much easier to get students to use than advocating vi+LaTeX....

Re:You should had compared

[AndrewNeo]

Apache! Burn the fanboy!

Re:You should had compared

[gatzke]

What was wrong with using latex2html to generate hyperlinked files and such? What is the advantage of using your code? I did not see a readme or faq.

I have used latex2html successfully in a few cases, but maybe I am missing something.

Re:You should had compared

[impaledsunset]

Because it doesn't work? There is no generic LaTeX to HTML convertor out there, and all of them handle only specific use-cases.

In particular, all convertors either try to parse part of the LaTeX code or try to interpret the DVI that is produced as a result. With the first approach you'd have to limit your use of packages, because otherwise the convertor will fail unless it's a complete LaTeX implementation. With the second approach you won't be able to use XeLaTeX because it doesn't produce standard DVI files, and instead produces a new extended format which is not supported by any other tools yet. And you should really be using XeLaTeX for serious work.

latex2html uses the first approach, and it fails to work with documents that use UTF-8 international characters, and if it will fail particularly spectacularly if you use some more interesting packages.

The best approach is to use a subset of LaTeX that you can safely convert to another format, using tools written by yourself is actually easier than using the tools that are available, because the tools available also handle only a specific scenario, only that you don't know which one. I have my own personal simplistic format that I convert to XeLaTeX code and I plan to create XHTML convertors when I happen to need them. It's really simple, I have a few macros that I replace with XeLaTeX code and a template.

Re:You should had compared

[lee1]

You could use some sort of XML and generate TeX from it, but typing XML is horrible. I like to work in vim, and with a couple of macros entering LaTeX is really easy.

I type both xml and LaTeX with vim, and I find both are easy if you use the right plugins. Using the xml plugin vim will type all the angle brackets for you, complete your tags, format, and generally take the drudgery out of writing xml.

Re:You should had compared

[gatzke]

A quick google, looks like latex2html may work on files with UTF-8 chars in them:

After a lot of research I finally could find the right invocation of latex2html to generate HTML correctly with accents:

latex2html -html_version 4.0,latin1,unicode book.tex

From http://miguel.leugim.com.mx/index.php/2008/05/18/latex2html-and-utf8-encoding/

But you are right on use of more interesting packages, latex2html probably can't handle them. But I don't see the advantage of writing my own code and limiting myself to simple cases.

Re:You should had compared

[TheRaven64]

My code doesn't convert the math stuff yet, but it's pretty easy to do. There are existing libraries out there that will generate MathML from LaTeX, and it's easy for a parser to detect math mode stuff bracketed by dollar signs and just pass that off to a library.

MathML didn't use TeX because it does more. TeX is great for typesetting maths, but it doesn't give you any semantic information. MathML captures both - how an equation should look and what it means, although both parts are optional.

Re:You should had compared

[Alex Belits]

Microsoft.

"Use best tool for the job" is now a code word for "Hey, look, Microsoft has some shiny thing we want you to use -- we promise, it's better than what you use now!"

Re:You should had compared

[rwa2]

Meh, it worked OK for my thesis. I managed not to need too many international characters, I guess.

lyx (eventually I migrated it to emacs so I could stop wrestling with the GUI, which was having a cow with some of my larger figures) -> latex -> latex2html -> pluckr

also had a makefile to generate/update the .dvi -> .ps -> .pdf targets as well.

Re:You should had compared

[hedwards]

The truth is, if you want your job done, you look at the merits of every possible program without considering if it's open source or not. There are good software like Apache that are mostly good for web hosting (unless you have certain requirements). Then there is lots of shit. The same is true for proprietary software tho. But if you want to get something real done, it's just stupid to limit yourself to only open source OR proprietary software. Pick the best tool for the job.

The problem there is that you also need to factor in the cost of Windows or OSX if you want to use proprietary software that's designed for professionals. If you're using one of those platforms then the calculation is much easier, but if you're not interested in giving those companies money then your options are severely limited.

Re:You should had compared

[CODiNE]

XHTML, generated by some code I wrote[gna.org], with hyperlinks and cross references and semantic markup in the code listings generated by clang for [Objective-]C[C++].

Dude I am NOT clicking a GNA link on a Slashdot post. </oldskool>

Re:You should had compared

[tehcyder]

Microsoft.

"Use best tool for the job" is now a code word for "Hey, look, Microsoft has some shiny thing we want you to use -- we promise, it's better than what you use now!"

No, in the real world "using the best tool for the job" is just what professional people do. That includes using Microsoft products if they are indeed the best suited for the job, impossible as his may be for some slashdotters to believe.

Back on topic, why does the PhD thesis need to be published as an eBook anyway?

Re:You should had compared

[dotancohen]

XHTML, generated by some code I wrote, with hyperlinks and cross references and semantic markup in the code listings generated by clang for [Objective-]C[C++].

The problem is, with an URL like that no jaded /.er will click that link!

Re:You should had compared

[Anne Thwacks]

The problem is you have to factor the appaling consequences of "vendor lockin" and "proprietry format" on a document you may need access to for the rest of your life. Even in the unlikely event that an MS product is actually useable/fit for purpose, you would want the document saved in an open format, and I for one, have had very badTM experiences of MS's attempt at odt.

Re:You should had compared

[foobsr]

No, in the real world "using the best tool for the job" is just what professional people do.

And I always thought the professional domain would mention constraints and talk of 'optimal' instead of 'best'.

CC.

Re:You should had compared

[Anonymous Coward]

"And you should really be using XeLaTeX for serious work."

Fancy telling me why XeLaTeX is any better for me than PDFLaTeX, which is doing everything I want and producing professional-grade papers for publication? Or am I not doing "serious work", doubtless unlike your esteemed genius self?

Re:You should had compared

[Eunuchswear]

XHTML, generated by some code I wrote[gna.org], with hyperlinks and cross references and semantic markup in the code listings generated by clang for [Objective-]C[C++].

Dude I am NOT clicking a GNA link on a Slashdot post. </oldskool>

No, it's the other GNA, Gna's Not Axis, basicaly the home of the French FSF (starterd by some guys working for a company called Axis - gave me my first newsfeed in return for a 2nd hand v.32 modem a long time ago).

Re:You should had compared

[goose-incarnated]

Because it doesn't work? There is no generic LaTeX to HTML convertor out there, and all of them handle only specific use-cases.

AMEN!

You cannot convert TeX to anything other than DVI/ps/pdf without getting a mess. Embarrassing. While it does produce incredibly beautiful printed output, note that very few of us are reading printed anymore. Most of us are reading the TeX output on computers, not on paper

I decided to write my own, without all the disadvantages of TeX[1], but with as many of the benefits as possible[2]

I did a tiny prototype last September (was in hospital for a week, had nothing else to do) in Lisp that read in my special "eBook-format" files, handled the trivial cases and could spit out LaTeX, HTML, plain text (with as much of the original formatting as possible) and a dynamic HTML/javascript file that had the content as well as the ability to honour the interactive portions (like placing bookmarks, making annotations, etc).

[1] Namely, inability to convert to common document formats, no support for images (its an add-on), no support for output in popular formats, no support for dynamic content with "onclick" or "onkey" macros (Yes - this is needed - it allows the reader to annotate the book, place bookmarks, etc), an inconsistent language, etc... All of these were not an issue when TeX was conceived and created, they are now as we no longer use non-interactive paper to read our books.

[2]A macro language, focus on content not presentation, styles, etc

Re:You should had compared

[dgatwood]

I'm in the process of preparing three novels, and I took a rather different approach. I'm using AppleWorks as my editing tool (because that's what I started with). From there, I'm:

• exporting the results as HTML,
• running a nasty piece of Perl (is there any other kind?) to turn AppleWorks HTML into valid HTML,
• running another nasty piece of Perl to translate that HTML into DocBook plus a bunch of custom tags, and
• running dblatex with some rather large custom style files and custom xsl to translate that into suitable code for xelatex.

The advantage to this process is that I have valid XHTML on the way into the process, and with minimal effort, I could go from there to usable ePub content.

If I were starting from scratch on a new document, I would be writing XHTML with some custom CSS as my source format. That would give me full semantic markup capabilities (which would give me slightly more flexibility than I have now, but not enough to convince me to ditch the convenience of editing in a WYSIWYG editor for this project). Then, I would tweak my XHTML to DocBook translation tools to handle that. So for ePub, it would just require containerizing the source material, and for nice PDF output, it would just require using the translator bits I already have.

Of course, none of this is a general solution. Novels and theses are rather different in the way you write them, and the former was made a lot more difficult by LaTeX being designed so heavily for typesetting things like the latter. There are also a lot of flaws in LaTeX stemming out of the core design that make for less than ideal typesetting.

For example, as far as I can tell, there is no good way to indicate that a section break (three stars, for example) cannot be the first thing on a page, and that at least two lines of the content above it must be pulled down with it. The closest you can do is to make it part of an unbreakable container with the previous whole paragraph, but that doesn't really do what you want most of the time.

Similarly, it does not support proper widow control. LaTeX supports widow line control—that is, saying that you cannot have fewer than the last n lines of a paragraph on a page/column by themselves. What it lacks is widow paragraph control—that is, treating a single-paragraph line as though it were the last line of the previous paragraph for widow calculation purposes. The result is poor typography if a page break happens to fall near the end of a chapter. You can fix this by hand-tweaking the TeX markup to force a page break earlier, but I assert that good page layout software should produce good layout by default without hackery.

And LaTeX does not handle UTF-8 very well at all. In my XHTML to DocBook translator, I've had to hack in extra markup (\hspace{0.001pt}) after em dashes, en dashes, and hyphens to force TeX to allow the line to wrap. Without that hack, I get serious overfull hbox problems.

I could probably go on for hours about all the problems I've encountered, but it suffices to say that I'm not impressed by TeX, and at several points, I was tempted to build my own PDF generator using WebKit and CSS styles, but I didn't want to spend the time. (Yet, in hindsight, it would have been faster than trying to force TeX to behave.) That said, if you started with something like the hyphenator project, someone could probably replace most of TeX with a few hundred lines of JavaScript, and that would almost inarguably produce better typesetting with a lot more flexibility (particularly given that pretty much every programmer already understands JavaScript and the DOM).

Re:You should had compared

[Anonymous Coward]

You really should try to advocate emacs+LaTeX rather than vi+LaTeX.. ;)

Re:You should had compared

[Anonymous Coward]

Last time I looked latex2html 10 years ago, it produced images when it couldn't produce markup. If the source is indeed LaTeX it is somewhat unlikely to require very fancy formatting on the html side. The html version likely gets eaten by a content management system or is enhanced with some nice css. Creating an interesting formatting using LaTeX for the html version is wasted effort in these cases. Then there is always make.

Re:You should had compared

[aaaaaaargh!]

Being the most professional system for PDF generation, I went with LaTeX

I have used LaTeX for my thesis and the book resulting of it, as well as for typesetting another book for an international publisher and tons of papers, and I'm afraid I have to disagree with this statement. It's perhaps the most professional free PDF creation tool but nevetheless has countless flaws. Apart from the arcane underlying ad hoc language with global namespace the most severe flaw of LaTeX is the paragraph formatting algorithm, which simply does not work with fixed page height (i.e. \raggedbottom which is required by almost every publisher). Even using all the microkerning options in pdflatex it is a pain in the ass to get the paragraph formatting of LaTeX even halfway right. You have to tweak every second page manually and pray no contribution requires a last minute change.

It would be nice to have a free and more modern alternative. For texts with formulas I'm only aware of one, lout, but it doesn't have enough features.

Re:You should had compared

[Aighearach]

real nerds use netcat for their webserver

Re:You should had compared

[tehcyder]

Pick the best tool for the job.

Thank you, captain obvious.

No, it's not fucking obvious, because "pick the best tool for the job" does not mean "pick the best tool that fits into your ethical-political worldview".

Re:You should had compared

[Anonymous Coward]

In most cases it does, because people don't do jobs that don't fit into their ethical-political worldview.

Re:You should had compared

[nick_urbanik]

• XHTML, generated by some code I wrote, with hyperlinks and cross references and semantic markup in the code listings generated by clang for [Objective-]C[C++].

In the process of compiling this on Fedora 15, I raised bug 728744, and worked around those problems. The compilation of ETClassMirror.m failed with "incomplete implementation of class ‘ETClassMirror’ [-Werror]". I'm tempted to leave this for now, as this doesn't seem sufficient reason to learn Objective C, and I have other things to do.

Re:You should had compared

[TheRaven64]

For future reference, filing bugs in a project with an unrelated entity generally does not achieve anything...

This seems to be a change that Richard introduced in GNUstep while shuffling around the headers. The real fix for it is to stop using the GCC Objective-C runtime (which lacks features, is buggy as hell, requires a huge pile of hacky work-arounds for basic stuff to work, and is not supported by Etoile). That would mean you'd be getting the original version of that header, rather than the version in the compatibility lib. Ideally, also stop using GCC, which is a painfully archaic Objective-C compiler and lacks support for most parts of the language, and is also not supported by Etoile. You can't use blocks if you compile with GCC, which I'm pretty sure means EtoileText won't build.

Of you'd read the INSTALL file then you'd have seen this:

* LLVM/Clang 2.9 or higher is required to build Etoile
* libobjc2 1.4 (other ObjC runtimes such as the one packaged with GCC won't work)

So, trying to compile it with GCC and using the GCC libobjc is most definitely not something we support. We're a small project, and don't have the time to spend hacking around limitations of badly designed compilers.

Re:You should had compared

[nick_urbanik]

For future reference, filing bugs in a project with an unrelated entity generally does not achieve anything...

I expect the Fedora maintainer will probably do something sensible with this; why do you declare this authoritatively to be not the case? Upstream also seems to be in a half-baked state.

* LLVM/Clang 2.9 or higher is required to build Etoile
* libobjc2 1.4 (other ObjC runtimes such as the one packaged with GCC won't work)

Okay, I've only 2.8 of LLVM/Clang, and libobjc-4.6.0-10.fc15.x86_64. I'll move on. I'll write my own solution in Perl when I need it.

Re:You should had compared

[TheRaven64]

I expect the Fedora maintainer will probably do something sensible with this; why do you declare this authoritatively to be not the case? Upstream also seems to be in a half-baked state.

Based on past experience with the Fedora, Gentoo, and Ubuntu bug trackers. People file bugs there, and then we never hear about them. Then someone posts to the mailing list saying that a problem has been known for two years and asking why we haven't fixed it. Answer: because it wasn't known to us. Filing a bug report on a distribution's bug tracker is like posting it on your blog. Someone who can fix it may stumble across it, but generally they won't. It's okay for really big projects, because RedHat / Canonical probably employs someone who has commit access to these and can fix the bug or refer it to the correct person, but for anything else it's a complete waste of your time.

Okay, I've only 2.8 of LLVM/Clang, and libobjc-4.6.0-10.fc15.x86_64

Interesting. GCC 4.6 was released after LLVM 2.8, yet you get packages for GCC not Clang. I guess the fact that RedHat employs a few GCC contributors may have something to do with that. I don't think I'd stick with an operating system where they restrict your access to new software for political reasons, but to each his own.

I'll move on. I'll write my own solution in Perl when I need it.

Good luck. If you stick to simple markup in the chapters as I suggested, it's pretty easy to do. Most of the complexity of EtoileText is from the fact that the framework is designed for writing semantic document editors, not just converting TeX. The LaTeX to HTML conversion was just a simple test program that filled a need I had.

Re:You should had compared

[nick_urbanik]

I am sorry about the tone of my last post; I don't mean to belittle your excellent efforts. I am glad of your good work. I'm just a daddy with a child who needs my time, and so I must move on, as I don't have a pressing need yet, just a curiosity, and I know I will be able to solve the problem of LaTeX to epub when the time comes, probably using tex4ht and some post processing. I was interested because I have written plenty of LaTeX for many years, and know I'll want to do this conversion in the future.

Re:You should had compared

[TheRaven64]

When you do get around to it - drop me a mail (my username in the commit logs @gna.org should work). I'd be happy to help, and we may even have some Fedora packages by then...

through HTML

[fph il quozientatore]

The best way to go seems LaTeX->HTML->ePUB. I guess many of your problems do not come from LaTeX itself, but from the fact that the LaTeX code that LyX outputs is... well... not meant for human editing and for further work. (haven't worked with LyX in a while, though -- maybe the quality of the TeX it produces has considerably improved in the meantime).

Re:through HTML

[gatzke]

I have been using LyX for over a decade, and I feel it is a great tool for using LaTeX without the headache of LaTeX. The code it produces is not great, but it is reasonably readable.

Re:through HTML

[tehniobium]

Once you practice a little bit with LaTeX (we're talking using it for a couple of weeks) there really is no headache.

In fact, the lack of headache from LaTeX is what makes it better than any WYSIWYG editor out there (LyX is good, but still a headache imo)

Re:through HTML

[gatzke]

There are some things LyX is better at than pure LaTeX code.

I can see the current version of my figures, not just rely on the file name.

I can add references from a list instead of trying to remember what labels I have used.

I can search bib items and add / order citations easily.

I can make complex tables without forgetting some damn }

I can generate and view a new version of my document in a single keypress.

I can see my equations without having to mentally render them, while still using most of my TeX knowledge (\alpha _12 in LyX is the same as \alpha_{12})

Students can make the transition from Word a little more readily. Remember, LyX is not WYSIWYG, it is WYSIWYM (what you mean) so the on screen representation is close to the final but not exact.

Plus you have access to tons of menu options that you may not be aware of. I learn more about LaTeX by using and exploring LyX. And you can always use pure code if you want, for any fancy stuff.

Re:through HTML

[alambda]

Most the features listed are built-in in any decent modern LaTeX editor. A classic one (for Windows) is TeXnicCenter. Many Linux editors (for example the cross-platform texmaker and it's forks) have the same approach. They all have a list of labels, to name one of the features you listed. Then there's the TeXworks approach (originating from the Mac-world), which focuses on a constant preview of the final work, using syncTeX to easily hop back and forth the PDF and the TeX file. This is still much in the works, and doesn't have a list of labels etc. Once these are added, I suppose I'll migrate.

Indeed, I much prefer writing LaTeX with an editor than using LyX, as I do like the coding aspect and the cleanliness of it. I love drawing pictures with TikZ and pgfplots, although I'm sure this is easily done in LyX as well.

Re:through HTML

[Man Eating Duck]

The best way to go seems LaTeX->HTML->ePUB.

Absolutely. Once you're at well-formatted xhtml you can do a lot of boring structuring, TOC and so on with Calibre. Just import xhtml/css, put the right options and class names into the convert pane, and you're done. Epub is the best source format for converting to other ebook formats as well. Once you have a well formatted epub you get most other common formats almost for free, also with Calibre, depending on the complexities of your document and the limitations of those other formats.

I have no idea how his source document translates to xhtml, but if you start with a pure page-oriented format you will run into snags. A 'page' is a meaningless concept in a flowed format, so you'll need to convert your footnotes to bidirectional links in an endnote section. Headers and footers are also out. Any references to other sections or pages will have to be hyperlinked to make sense. If you have side boxes or other such elements you'll have to convert them to something that works in an xhtml setting. Vector illustrations might have to be rendered in something like svg which epub viewers generally support well. If you use any other glyphs than those found in the basic English ASCII set you should probably embed a font like DejaVu to make sure that your readers don't see just spaces or rectangles. If you can post it to Slashdot you're probably fine, though, they don't even support the pi symbol :)

All that said, it's perfectly possible to make a very usable academic epub, I do it all the time as I work with exactly this in an academic publishing house. A particular book on French grammar with 1132 (sub)sections and more than 5500 crosslinks actually became a *lot* more useable in a digital context due to hyperlinking, and it translated very well to epub, although it was a lot of work.

calibre

[Anonymous Coward]

calibre is a free and open source e-book library management application that can convert to and from most of ebook formats. And does a pretty good job at it.

http://calibre-ebook.com/

Easy solution

[theatreman]

LaTeX -> pdf then convert here: http://www.2epub.com/

Re:Easy solution

[TheRaven64]

Going through PDF is horrible. LaTeX contains a lot of semantic markup. ePub is XHTML, which is a form of semantic markup. PDF is a presentation format. So, you start with semantic markup, discard it all, and then try to generate it again by magic.

You end up with something that looks vaguely like the PDF, but loses most of the semantic information (e.g. section / chapter breaks). Worse, you often don't want the ePub version to look like the PDF - they're aimed at different form factors.

Re:Easy solution

[digitig]

The trouble is, PDF is a pretty rotten format for e-readers, because it's all page-layout oriented and so produces output that doesn't scale well for different screen formats and text sizes. It's the wrong format for the job. And DVI has pretty much the same problems. The problem isn't that free software isn't ready for ePublishing -- Calibre and Sigil do the job well. The problem is that there's a disconnect between the assumptions laTeX makes about a document and the assumptions that are valid for ePublishing, Sorry if it's restating the blindingly obvious, but you didn't want the best system for PDF generation, you wanted the best system for PDF and EPUB generation, and that probably isn't laTeX.

Re:Easy solution

[lee1]

But you can make your pdf with any page size and font. You could easily make a set of pdfs for all the common devices from a single LaTeX source, and the result would look much better, typographically, than the typical ebook.

Re:Easy solution

[ConceptJunkie]

This is fine. Then if I purchase an e-Book, I only need the PDF version specific to the device I'm currently using (a Nook Classic)... oh, and any device I might ever want to use for the rest of my life. A proper eBook format cannot be tied to a specific page format.

I like PDF for computer use, but the parent is right... it's definitely "rotten" for e-Readers. I've tried converting PDF to ePub to use on my Nook and it's a hit-or-miss proposition, with much more "miss" than "hit".

Re:Easy solution

[Dog-Cow]

Who defines the "common" devices? How do you handle something like an Android or iDevice, where the orientation can be changed? PDFs are not a good format for anything destined for a screen instead of paper. That computer monitors are (mostly) large enough to display most of the common paper size (letter/A4) is fortunate, but should not be relied upon.

Re:Easy solution

[digitig]

No, the results won't look better. I have poor eyesight, so I choose a large text size on my eReader. If the original is EPUB then the text reflows smoothly and it's all nicely readable. If it's PDF it doesn't, and the results look like crap. The solution you propose means anticipating the individual requirements of every potential user, and producing a customised PDF for that user. What's more, if I'm in bright light then I can move to a smaller text size to see more at one time, but doing it your way I'd need two copies of the file (and some way of synchronising the bookmarks and annotations). We've moved beyond the age of one-size-fits-all, but PDF hasn't. LaTeX doesn't seem to have, either. Essentially, you need to separate content from presentation, which neither PDF nor laTeX does, although there is work on moving laTeX towards that.

Re:Easy solution

[lee1]

There are really only a few sizes that are used in practice. And the pages don't need to fit the screen exactly. Make one for phones, one for iPad-sized tablets, one for computer monitors, and a large-type version for each of those sizes, and you're done. Trivially scripted. But why not just use an ebook format? Because the readers attempt to do typesetting on the fly, most critically linebreaking, to flow the text to fit the device, usually without hypenation, and the result looks like crap - even worse than html displayed by a browser.

Re:Easy solution

[lee1]

Yes, you've pointed out the drawbacks with deploying as pdf, which I agree are real. And I think I've pointed out the advantages of pdf, and the drawbacks of ebook formats as they are currently implemented. If you're an author and you care or want to control what the document actually looks like, then pdf (or a bunch of images) is your only option. Neither ebook formats nor html will do this. Sometimes this is important, sometimes not. But one of the main reasons for LaTeX is exactly to separate content from presentation, so I think you're misinformed about that; and that point doesn't apply to pdf, which is for consumption only, not for writing.

Re:Easy solution

[Inner_Child]

Congratulations, you've just earned the "I don't know what I'm talking about!" achievement! All modern e-readers, when using their proper formats (generally ePub for pretty much anything worth using) handle line-breaking and hyphenation just fine, and unless you're reading from some badly OCRed plaintext copy, will look as good as the paper version. PDF is a bad format for e-readers, and you're a bad person for suggesting it.

Re:Easy solution

[Oligonicella]

I think you are correct that it is rotten for the earlier devices, but that's not a long term argument. Probably not even much of a short term one any more. Sorry you're device is becoming obsolete, but that's the way with cutting edge. I seem to remember my first removable disk was only good for maybe six months before the company discontinued it.

I don't think it will be all that long before all new e-reading devices can view rather large books and therefore pdfs with no problem.

Re:Easy solution

[digitig]

Yes, you've pointed out the drawbacks with deploying as pdf, which I agree are real. And I think I've pointed out the advantages of pdf, and the drawbacks of ebook formats as they are currently implemented. If you're an author and you care or want to control what the document actually looks like, then pdf (or a bunch of images) is your only option.

No, it isn't an option. PDF doesn't do that unless you also control the device on which it is displayed. When I view a PDF with large text on my eReader I'm damn sure that what I see isn't what the author intended (not all authors can be that demented, surely). If you are trying to do that then you have overstepped your role as an author. If you think you have succeeded then you should try talking to your users (especially ones with visual impairment). PDF does have it's uses, but that isn't one of them.

But one of the main reasons for LaTeX is exactly to separate content from presentation, so I think you're misinformed about that

I used to use laTeX a lot, and was a member of the TUG. LaTeX is better than raw TeX, in terms of separation of content and presentation, but most raw TeX is still there in LaTeX, and LaTeX commands such as \textwidth, \baselineskip, \raisebox (everything to do with boxes, in fact), \vspace, \textbf and so many other laTeX constructs are about presentation, not content. You can write laTeX that separates content from presentation, but tools that claim to process laTeX can't assume that you have; they need to accept all legal laTeX, including all the presentation stuff.

and that point doesn't apply to pdf, which is for consumption only, not for writing.

Internally, PDF is quite like DVI in terms of how it structures a page, and the content and presentation have been well and truly merged. PDF puts blocks in defined positions on the page, and the order of the blocks doesn't necessarily match the order of the content. That's why when you select text in a PDF you often get bits you don't want. And it's why it's hard to go from PDF to EPUP; it's not a simple translation, the software needs to understand the significance of relative positions of blocks of text, which is very far from trivial. Yes, it's a presentation format, but that means that you have lost information needed to make a robust EPUB file from it. A far better option is to start with EPUB and generate your PDF from it and a stylesheet. The only downsides are that free EPUB editing tools are not well developed (unless somebody can point me to one that I've missed) and that EPUB enforces a linear reading sequence (but you're going to have to deal with that anyway if you're going to produce EPUB).

Re:Easy solution

[lee1]

Getting a little worked up, aren't you? I wish you were correct, but you're simply wrong. If you're actually interested in this subject you might want to start educating yourself by looking here for a recent rant about the terrible typography in e-readers and why, specifically, their output does not even come close to looking "as good as the paper version". I'm curious: what hyphenation and line-breaking algorithms do you think that the "modern" e-readers use? Do they all use the same ones? Well, never mind - maybe I'm just being mean now.

Re:Easy solution

[lee1]

Pdf is obviously an option. I read pdfs on my iphone and every other screen I have. That there are problems with this option is clear, but that's not the same as "not an option".

I'm not clear what you mean by "view a PDF with large text". Do you have a pdf reader that tries to increase the font size, or make font substitutions, rather than just zooming the whole page? Of course that will mangle the page, and create something that was not intended, but pdf readers should not try to do that.

You have some pretty rigid views about the proper role of an author. Some documents are typographically complex, or convey their meaning partly through layout and typography, and these elements will be destroyed by typical e-book software and are not preserved in e-book formats. Pdf is the only way I know of to handle this. I think if a book is distributed as a pdf then a large-type version should also be prepared, since, as I think you're saying, there is no way to increase the type size of the pdf for reading on the small screen without screwing it up or having to scroll sideways to read it.

Re:Easy solution

[cdrguru]

Absolutely, what is needed is a ebook reader with a 24" display so it can handle really big page sizes. Right?

Please put down what you are smoking. When your chemically-fogged brain has cleared you will understand that an ebook reader is designed to be able to be held in one hand and the text read easily. Neither of these happen with larger screen sizes, as can easily be seen by the sales numbers for te Kindle and Kindle DX - the DX doesn't sell very well because very few people want the huge form factor. Also, I haven't seen any other ebook readers other than the DX which have anything like that form factor.

PDF is a rotten format for ebooks and it isn't ever going to get any better for them, nor them for it. What makes sense for a printed page layout doesn't make any sense at all for a variety of devices with varying screen sizes and resolutions. Navigating around a page makes little sense when what you want is the text in an accessible form for the device you are using. While it is possible to achieve this with PDF, it isn't simple or convenient.

Re:Easy solution

[digitig]

Do you have a pdf reader that tries to increase the font size, or make font substitutions, rather than just zooming the whole page?

Yes, my eReader, because it only has the facility to turn the page, not to move around the page. And as far as I'm concerned, it's not the fault of the reader for trying to do that, it's the fault of publishers supplying content in a format that tries to stop me, which I view in the same light as DVDs that won't let me pause the main feature while I take a comfort break. I don't want to have to read a broadsheet newspaper through a letter box by asking somebody on the other side of the door to move it around.

Some documents are typographically complex, or convey their meaning partly through layout and typography, and these elements will be destroyed by typical e-book software and are not preserved in e-book formats.

And won't be displayed on my eReader, even if you use PDF. Yes, PDF is a good format for such documents, but not for an eReader because it won't work.

Re:Easy solution

[Inner_Child]

I specifically didn't mention the Kindle. Mobi is atrocious, and Amazon cares more about the bottom line than presenting an acceptable reading experience. You're right, not everyone uses the same algorithms to determine hyphenation and line-breaking, but in my experience, ePub and the readers that use ePub (at least the Nook - I really can't speak for others such as the Kobo or iRiver readers, not having used them) do a better job than the atrocity in that article. Of course, none of it matters if the publishers don't bother putting a little bit of care into their e-books. I may have come on a little strong, but I believe e-readers are not all as bad as you and the article you linked are making them out to be. http://www.the-digital-reader.com/wp-content/uploads/2010/10/nook.jpg is at least an improvement on the look of the Kindle's page, however I agree, it's not perfect.

Re:Easy solution

[lee1]

It seems tautological that you won't be able to read a pdf on a device without a properly functioning pdf reader. So, yes, if you distribute your document as pdf only you will be excluding readers who insist on exclusively using these devices, if they exist.

Re:Easy solution

[lee1]

"I specifically didn't mention the Kindle"

You said "all modern e-readers". Kindle is the most prominent member of that class.

ePub might be a little better than typical Kindle output, but it still sucks. I can see it, and I'm not even particularly picky about typography. The image you link to is a good example of how bad they suck. Look how nasty and gappy the lines are. Notice how, for some reason, there is no hyphenation, and notice how a little hyphenation could have improved the readability and aesthetics of the page.

Re:Easy solution

[radtea]

ePub is XHTML, which is a form of semantic markup

Just as an historical note, use of the term "semantic markup" in this way makes SGML purists and probably Babby Jesus cry.

The "semantic" in "semantic markup" used to refer to domain semantics, not document semantics. SGML languages were supposed to allow users to mark up documents with <PART_NUMBER> and the like, while DSSSL would handle the transformation to presentation format.

HTML is considered a fundamentally broken use of SGML by purists precisely because it is defined almost entirely in terms of presentation semantics. If Sir Tim Berners-Lee had been a purist he'd've created a browser that defined a presentation model like that of HTML, with H1 and so on defined, and implemented a processor to convert any SGML markup plus an appropriate style sheet to that presentation model. And we'd all be now viewing MS Word docs on the Web, because Microsoft would have delivered something that worked decades before Sir Tim did...

Never-the-less, while XHTML+CSS is better than the FONT tag, it still doesn't qualify as semantic markup in a purist's eyes.

Re:Easy solution

[TheRaven64]

That depends on how you use it. In the XHTML that I generated for my last book, for example, every code listing was marked up using libclang. Each token was in a span element with the class set to the token type. This meant that the XHTML version contained information about whether something was a macro instantiation, a language keyword, a reference to a variable, and so on. The styling for all of this was then done in CSS. I sent my publisher a rough version and they could then tweak it so that it matched their house style a bit better.

I initially tried using tex4ht, but it generated presentation markup for the code listings. They looked in the output how the listings package made them look in the PDF, but it was impossible to style them properly from the CSS.

Re:Easy solution

[swv3752]

It is more likely the huge price tag rather than the huge size. I wanted the DX, but could not justify the several hundred dollars more for the device.

Re:Easy solution

[Aighearach]

Gotta agree with that. The last ebook I had, I had to run it through a script to strip it down to ASCII, and I viewed it on a cheap "mp4" player. I never thought, "gee I need a new device so I can read this ebook." I only thought, jeeze, I need this document in a data format instead of a presentation format.

...PROFIT!!

[djsmiley]

1. Realise no scripts exist for problem
2. Write scripts
3. Release scripts as open source
4. Don't post pointless problem on slashdot
5. ???
7. PROFIT!

(We don't talk about point. 6)

Re:...PROFIT!!

[Khan Fused]

1. Realise no scripts exist for problem
    1,1 Realize that someone writing a thesis on Nicaraguan politics may not know how to program
    1.2 Begin learning to program
    1.3 Spend more time learning to program
2. Write scripts
    2.1 Divert time from PhD thesis to write scripts
    2.2 Spend more time (diverted from PhD program) learning to program sufficiently to write workable scripts to solve stated issue
3. Release scripts as open source
    3.1 Fail to complete PhD thesis in time due to time spent programming

Re:...PROFIT!!

[kravlor]

This is very true. I wish I had mod points!

Re:...PROFIT!!

[dstar]

4. Realize this is exactly what happened to Knuth.
        4.1 Take consolation in the fact that at least it's just a thesis, not the next volume of TAOCP.

Re:...PROFIT!!

[Anonymous Coward]

4. The scripts have a bigger audience than yet another thesis on Nicaraguen politics.

Re:...PROFIT!!

[Anonymous Coward]

One of the best comments I have read on here. I am a technology user, not a hard core technology lover. There is room for both so it is nice to see a recognition that not everyone needs to be a proficient programmer.

Re:...PROFIT!!

[Anonymous Coward]

Arghh! Beat me to it.

Re:...PROFIT!!

[jedidiah]

I don't think any of you twits realize how much work goes into any PhD thesis.

A little programming overhead is not going to be that much of a burden really.

This is why most stuff gets invented. It's really not that much of a tragedy when people who don't specialize in selling a particular technology to others have to develop solutions for themselves involving that technology.

If real people thought like you weenies then we never would have had the original killer app for the PC.

Re:...PROFIT!!

[Anonymous Coward]

Well... On the bright side, he is learning a useful skill rather than proving that he is an expert in a field nobody cares about.

Re:...PROFIT!!

[Oligonicella]

You're shitting me.

Someone so brilliant as to be so intimately versed in what it takes to get a PhD that he can with impunity berate all others who don't have his amazing insight into the process used the phrase

"the original killer app for the PC"?

Re:...PROFIT!!

[Anonymous Coward]

1. Realise no scripts exist for problem
2. Write scripts
3. Release scripts as open source
4. Don't post pointless problem on slashdot
5. ???
7. PROFIT!

(We don't talk about point. 6)

OOOOHHHH, you sure showed him! Slashdot idiocy at its finest.

Re:...PROFIT!!

[Anonymous Coward]

It is important to realize that by the time someone is WRITING their PhD, they are often near burn out from having spent years doing the actual research that is why they got into their field of interest to begin with. It is an incredible, time-consuming enterprise, that has a high drop out rate.

But it is just as ignorant to suggest that "a little programming overhead is not going to be that much of a burden really" -- at that juncture it isn't the time spent doing the coding, but the loss of focus and the idea that 4,5, or more years into a program, when one is finally ready to finish (and is likely broke because they are either about to lose funding, already have, or weren't getting anything or much to being with), that they might have to do programming work which (a) none of their readers is going to care about, and (b) has little or nothing to do with what they are trying to do.

It may be that many inventions happen when folks step outside of their fields -- but I would certainly challenge your statement that it is "why most stuff gets invented" -- do you realize how much work goes into a typical "invention"? Or is this simply a selection bias that causes you to focus on the couple of cases where an outsider makes a major contribution. Frankly, I don't know which is true, because I haven't tried to pull together stats.

Those "weenies" above read like someone who is going through this process.

(disclaimer - I am married to someone who is currently finishing their PhD 1st draft in a social science field, and I have a computer engineering background. Were I to write a thesis, would I write in in LaTeX or TeX ... maybe ... even probably... Would she, absolutely not. MAYBE typeset it for publication if under duress... but the loss of time to fiddling with it would be tremendous and interferes with what she has spent the last decade learning how to do at the PhD level and has little to no utility in her research)

Re:...PROFIT!!

[frank_adrian314159]

If real people thought like you weenies then we never would have had the original killer app for the PC.

Solitaire? I think Microsoft would have built it anyway...

Re:...PROFIT!!

[Khan Fused]

Unless you work for the State Department and are about to be transferred to the Nicaraguan embassy.

Re:...PROFIT!!

[Anonymous Coward]

Low user ID != brilliant.

He's probably an aspie and clearly is not firmly in contact with anything other people consider to be reality.

Re:...PROFIT!!

[Anonymous Coward]

If real people thought like you weenies then we never would have had the original killer app for the PC.

Space War?

Perhaps you should have

[Anonymous Coward]

considered what LyX/LaTeX were actually designed for. LaTeX is an interface to the TeX *typesetting* program, which does exactly that: it sets type on a page for printing (on paper) purposes. It just so happens that TeX can "print" to digital formats like PDF and DVI, which also just happen to support some "ePublishing" features like indices, search and table of contents. TeX/LaTeX/LyX are themselves NOT ePublishing platforms, they are means of applying visual styles to text. Since ePublishing itself doesn't even make sense as a term (is a website with information considered "ePublishing"? I think so -- so perhaps you should have authored your paper on a web CMS) when you consider the technical intricacies of everything involved, I think you'd have been better off investigating all "publishing" options before even starting to author your paper in a complex system. If it was me: I would have started to write text in a plain text file until I had decided the best route to go. If graphics and equations were required, I would have moved to a generic HTML + css method.

Formats are all over

[Anonymous Coward]

Work with Calibre if you need to convert formats.

epub is basically compressed html with some indexes, I think its even .zip format.

Calibre can handle most of this stuff for you if you can get it into an acceptable format from tex.

Calibre?

[anton.karl]

Why not just use Calibre to convert from pdf? It is very easy to use and supports all kinds of formats. See: http://calibre-ebook.com/

Re:Calibre?

[ConceptJunkie]

Because PDF to ePub conversion generally gives you pretty awful results. Nothing against Calibre. I use it. But most PDFs I've tried to convert for my Nook Classic have had less than stellar results: readable if you're lucky, but not nicely formatted. And if there are embedded images, all bets are off.

Re:Calibre?

[johanneswilm]

In my experience it cannot handle the textflow completely, as that information is lost when the PDF is created. Do you have some tricks for making it do that better?

RE: Stallman link

[craftycoder]

Stallman complains about DRM and a lack of anonymity with eBooks. It seems to me that this story relates very closely to legally acquired music. While it is still difficult to legally acquire digital music anonymously, it is easy to get it without DRM. I suspect books will follow this same path if consumers value it as a feature. In practice there is in fact little anonymity in the purchase of real books as everyone wants you to swipe your "club" card and use your debit card to make the purchase but his point is well taken. The option to buy an unpopular book in secret is nice.

With time and interest from consumers we will have DRM free books.

Anonymity is dead and gone and I didn't even get an invitation to the funeral. We should all mourn it's passing.

Open source solves problems programmers have

[Qzukk]

I went to arxiv.org and picked a dozen or so papers from the "new" list and clicked their other format links. They're available in pdf, ps, and dvi formats. This is hardly a complete analysis since I don't have any access to the "real" journals, but I have to wonder how many journals and universities are demanding papers in ebook format.

Open Source generally scratches itches. You may be one of the first people with the itch of converting theses to ebook formats.

Re:Open source solves problems programmers have

[TheRaven64]

I've not published anything in a journal for a couple of years, but in computer science every journal worth reading accepts PDF submissions and either provide a LaTeX style, asks for your LaTeX source to edit themselves, or tells you which standard LaTeX style to use. It's a good first check for a journal - if they don't encourage LaTeX submissions, they probably suck. Apparently the same is true in mathematics and physics, but less so in other subjects. In the humanities it's common for journals to require MS Word documents (and place insanely strict requirements on the formatting of the bibliography that are trivial with BibTeX and very hard with MS Word, from what I've heard).

Re:Open source solves problems programmers have

[Phillip2]

Short answer; none want epub formats as submissions. But this doesn't mean to say that there is not a desire to produce them from submissions. Lots of scientists and academics want to read articles on the go, without having to carry around lots of paper.

My own experience, however, is that the big move up is from PDF to HTML. This improves the reading experience enormously. EPUB on the other hand is limited. Many ebook readers don't work that well for academic content: mathematics is dealt with badly with non-scalable fonts, graphs and images are poor, citations are not well supported. I haven't see a huge use case for epub yet.

Re:Open source solves problems programmers have

[jank1887]

forget references. proper equation formatting and referencing in MS word for most physical sciences journals. still a bitch.

Re:Open source solves problems programmers have

[Anonymous Coward]

>(and place insanely strict requirements on the formatting of the bibliography that are trivial with BibTeX and very hard with MS Word, from what I've heard).

You are wrong. BibTeX is radically inadequate for humanities formatting requirements. BibLatex +historian is better, but the tools suites and tools support are inadequate. (This is from the last time I checked to see if BibLatex + LyX made sense for a Turabian but have to format MLA for journals culture)

Actually bothering to learn how to format citations manually is cheaper in the humanities than dealing with BibTeX/BibLatex because of the technological immaturity, the number of special cases, the data entry costs, and the lack of automated supply of data from publishers. In most cases citation formatting can be accomplished with superscript and basic rich text.

Boycotting? Hardly

[Clopy]

"Is the open source community boycotting ebook formats?"

Hardly. Calibre is an excelent converter, library manager and it's compatible with most of the readers out there for syncing. You could try converting from pdf to e-pub with it, although PDF is a lousy input format.

Re:Boycotting? Hardly

[pseudonomous]

That's linked in TFA, so apparently it didn't work in this case.

Re:Boycotting? Hardly

[geantvert]

The link to the Richard Stallman page is not against ebooks but about Amazon ebooks, or to keep it simple about DRM, proprietary formats and all other nice features introduced to "protect" users

There is nothing wrong with open ebook formats such as EPUB (XHTML+CSS+XML) as they remain DRM free.

         

Pandoc

[bbk]

I've found pandoc (here: http://johnmacfarlane.net/pandoc/) to be very useful for generating PDF/ePub/LaTeX/etc from Markdown formatted text files.

Re:Pandoc

[metamatic]

Indeed. For a PhD thesis on the politics of Nicaragua, I'd have started with markdown and then converted that to ePub and LaTeX.

Re:Pandoc

[pitkataistelu]

Wait, you're serious though right? I'm very much interested in small-footprint, non-WYSIWYG solutions for humanities text editing, but surely Markdown doesn't support, say, generating a table of contents? If you have to add that in at the LaTeX level, then you're dealing with two (sets of) source files, which is inconvenient.

I imagine I'll write my future scholarship in something along the lines of Vim + LaTeX, but when I was setting out on my PhD thesis I chickened out in the knowledge that some humanities journals demand copy in legacy proprietary formats, so I've drafted in .odt instead. I'm sorry to say oOo/LibreOffice formatting using styles is highly unreliable—totally should have gone with Vim + LaTeX.

Anyway, none of this concerns ePub--do carry on.

Re:Pandoc

[metamatic]

I used plain TeX for my dissertation. These days I use ConTeXt for documents. But actually, yes, there are tools to generate tables of contents for markdown documents.

Re:Pandoc

[datakid23]

+1 - pandoc and github combined make for an easy option - I do the same, hosting the github locally

Re:Pandoc

[EvanED]

This is related to what I'm looking into now. I don't need something quite professional-printing level, but with that in mind I'm looking at Asciidoc. (If you know about Markdown and not Asciidoc, they are similar-in-spirit markup languages. Both aim to be lightweight and something that if you have to read the source of, it's actually not too bad.)

Asciidoc has tool that output HTML and DocBook XML directly; that DocBook can then be compiled to EPUB or PDF or whatever (or as an alternate route to HTML).

I'm not as happy with the PDF output as I am with a Latex version of those documents, but it's hard to put my finger on why in many cases, and in many others it's probably just changing some of the typesetting preferences. But I'm much happer with the HTML output. I haven't yet tried EPUB.

Boycott? I Think the Tools Merely Lack Maturity

[eldavojohn]

Others have told me that the financial gain of publishing an academic book may be up to 700 USD. In comparison to current Scandinavian wages that really means very little, so I don’t think that earning another 700 USD should be a motive to restrict the access to one’s thoughts.

First of all I would like to commend you and thank you for this sentiment.

Is the open source community boycotting ebook formats, as Richard Stallman has proposed?

I don't understand, Stallman decries e-book formats that aren't open. There are many open e-book formats--including ePub. Granted, there are tools out there that allow you (to varying degrees of success like Calibre) to crack and convert to these formats but why bother? As you can see in that table, most everyone supports PDF. You are misunderstanding Stallman's gripe. It's not that we are boycotting e-books, it's that e-book makers are trying to carve out their own proprietary section of the electronic market, reader and creators included. So let them take their ball and play elsewhere. As you noted in your blog, this isn't the only problem:

Most ebook-readers out there so not implement the Epub-standard perfectly. That means that although one has an Epub that follows all the standards, one can be quite sure that it will not display properly on all the readers. Kovid Goyal, the creator of the Calibre ebook management software has done a good job in creating conversion scripts that create Epubs for all the different readers. Unfortunately they do this by breaking compatibility with the standard, and many distribution sites will only check whether your Epub complies to the standards and not whether the book will actually look good in the reader.

Most readers handle PDF, I would just stick to the output of LaTeX. I might suggest that your expectations are misdirected at the open source community and might be better directed at the makers of readers that apparently force you to break standards. It's the IE6 conundrum all over again.

Stallman didn't suggest boycotting ebook formats, just the DRM associated with them (big surprise there). The problem you are experiencing is that sometimes it's difficult to go from one open standard to another. The tools are lacking in maturity and I'm guessing that since my Android phone can easily display PDFs for me that there's not a lot of people demanding this ePub support that apparently needs multiple flavors for each device (and Calibre helps you with this). The tools exist but they'll only get you so far and I think the really special stuff that LaTeX does well is what you'll find yourself needing to fine tune in the end product. Look at how long it's taken LaTeX to get that beautiful and I think you'll discover that making a magical cure-all converter to ${random format} can be a non-trivial task.

If you start a kickstarter and get your university to donate hosting to making an open free market for any academic papers in any open format, I'd definitely throw in $20 (I've spent about $200 on kickstarter in the past two years). Either that or maybe throw your lot in with arxiv and work with them to fund more format support?

Pandoc

[cyocum]

The solution to your problems is Pandoc which can convert LaTeX to EPUB if you like. Now, it will probably take some fiddling on your part with the output but it very much smooths the process.

Asciidoc

[olafura]

You should try Asciidoc or docbook directly. I don't know if LaTeX has enough information to be faithfully converted to epub. But Asciidoc can reuse the LaTeX notations for a number of things.

I think it's the questions of right tool for the job, docbook is very widely used and is designed for working with books, and asciidoc and simular tools are for the non masochistic of us that prefer to edit text files and not raw xml. If you like gui there are plenty of guies for docbook, including LyX.

Re:Asciidoc

[Enry]

I agree with the DocBook recommendation. I was writing a lot of DocBook back in 2000/2001 and it would output to just about any format, including PDF and a few of the ebook formats that were available at the time (plucker? among others).

The thing that most people (including the original) is separating content from presentation. HTML tried to do this and failed miserably. When you start thinking about things like layout while writing, you're spending more time trying to make it look good rather than actually be good. Especially in the publishing industry, there's people that can do the layout for you and DocBook makes the layout portion quite easy even if you have widely varying output types.

Take a look at what the Linux Documentation Project was able to do.

RMS not boycotting e-books

[spf13]

While he states "We must reject e-books until they respect our freedom." He also outlines 7 things amazon's e-books do that violate this freedom. Fortunately epub is the most widely accepted e-book format and it has none of these 7.

1. Available anonymously.
2. Standard ownership applies.
3. License determined by vendor, but many have very liberal licenses including CC and public domain.
4. Open format based on html.
5. Lending rules same as physical book.
6. No inherent DRM (though Adobe has a version compatible with DRM).
7. No one can remotely delete it any more than any other file on your computer.

RMS isn't against e-books. He's against amazon's approach to e-books.

Re:RMS not boycotting e-books

[icebraining]

In fact, he specifically mention "Project Gutenberg" as freedom respecting ebooks, and they distribute EPUBs.

Re:RMS not boycotting e-books

[Enry]

No one can remotely delete it any more than any other file on your computer

Given you can copy the .amz file to your local system and save it (thus preventing Amazon from deleting it), this doesn't sound like much of a problem.

Re:RMS not boycotting e-books

[sourcerror]

They release books in other formats as well, like HTML.

Re:RMS not boycotting e-books

[Mysticalfruit]

This is why, while I own a kindle... I'm yet to actually buy a book from amazon. I've bought plenty of other books from reputable sources in epub format and converted them to mobi. In a nutshell, when I buy a book I want to actually *own* it.

Re:RMS not boycotting e-books

[Anonymous Coward]

Except in Germany. German "Projekt Gutenberg-DE" doesn't even let you download the ebooks, you have to scrape them from multiple HTML pages, and even then it might be illegal as they claim Imaginary Property on the public domain books, transcribed by volunteers, that they publish. It's totally crazy. I don't understand how they are still allowed to use that name.

Re:RMS not boycotting e-books

[manaway]

Until you restore the .amz backup to your ereader and Amazon deletes it again, before you can read it.

Does Amazon's ereader software for PCs also delete .amz files from your hard drive, or in some other way not allow you to read it? If so, and even if not, perhaps the problem of remote deletions becomes apparent.

Re:RMS not boycotting e-books

[Anonymous Coward]

Epub is to e-books what mp3 was to music in the 90's.

Re:RMS not boycotting e-books

[Enry]

They can only delete it if they can connect to your device. Turn off wifi or 3G and you're set.

I've never used the PC port, so I don't know how that works.

Re:RMS not boycotting e-books

[mvdwege]

Given that East Germans could hide their conversations from the Stasi bij running faucets or playing loud music, that means that persistent surveillance wasn't a problem then, according to your reasoning.

If you have to take active measures to keep control of something you paid for, I posit that something is wrong.

Mart

Re:RMS not boycotting e-books

[Enry]

False analogy is false.

Re: Stallman link

[pnewhook]

Yes, well Stallman is quite the crackpot, so anything he says should be questioned as to ulterior purposes.

A word processor?

[superdude72]

Is there some reason MS Word or OpenOffice + stylesheets aren't up to the task? It sounds like you might be overcomplicating things.

Re:A word processor?

[fartrader]

Is there some reason MS Word or OpenOffice + stylesheets aren't up to the task? It sounds like you might be overcomplicating things.

I totally agree. Wrote my computer science PhD in word with no problems. IMHO LaTex is the right tool for the job if you have lots of equations, and unless his PhD is on psychohistory I very much doubt he needs that kind of power.

Re:A word processor?

[gbjbaanb]

or you need it converted. IANAP (I'm not a publisher) but I doubt they use Word to generate the necessary typesetting for a print run, and Word is pretty bad at converting a document to anything else whilst keeping the formatting intact.

Re:A word processor?

[Anonymous Coward]

Word & OpenOffice documents look like ass. You'd need Adobe before it'll look as nice as LaTeX.

Re:A word processor?

[kravlor]

Setting aside the fact that LaTeX will perform typesetting, those word processing tools utterly fail for creation of documents with lots of (or complex) equations.

They are also very cumbersome for generating cross-references, bibliographic formatting, and management of figures/tables.

One killer feature MS Word *does* have over TeX-based solutions for now is excellent commenting, change tracking and shared collaboration features.

I know both worlds well, having used MS Word for collaborative proposal writing, and TeX for scientific publishing. I strongly prefer LaTeX.

Re:A word processor?

[serviscope_minor]

Is there some reason MS Word or OpenOffice + stylesheets aren't up to the task?

Have you ever written a long document in either word/OO and latex?

Basically everything that should be trivial, e.g. cross referencing, references and bibliographies, contents, figures, applying consistent styles (including bibliographic styles), separating content from presentation, quality typography is trivial in latex.

Also, since latex is in text files, you can use any version control system you wish.

It sounds like you might be overcomplicating things.

Not at all. The learning curve is such that for any even moderately long document, it is worth learning latex from scratch rather than using a word processor.

Once you're used to it, it is so much simpler to use.

Re:A word processor?

[serviscope_minor]

One killer feature MS Word *does* have over TeX-based solutions for now is excellent commenting, change tracking and shared collaboration features.

I have done some cooperative writing.

Latex supports comments trivially and always has. Anything starting with a %. There are also packages which allow you to insert them into the text as well.

For change tracking, word is OK if you pass the document around from one person to another in sequence. However, since LaTeX uses plain text files, you can use it with any VCS, which are vastly more advanced and capable.

Re:A word processor?

[Hatta]

Any text processing tool that requires you to use the mouse is overcomplicating things.

Re:A word processor?

[Anonymous Coward]

or you need it converted. IANAP (I'm not a publisher) but I doubt they use Word to generate the necessary typesetting for a print run, and Word is pretty bad at converting a document to anything else whilst keeping the formatting intact.

I do deal with publishers, and actually, it's pretty much the standard for manuscripts to be Word documents these days. Most technical books, at least, aren't so rigidly formatted as to require a PageMaker-type of program.

Re:A word processor?

[Phillip2]

I sometime Google docs these days for collaborative writing as it avoids the "pass the word doc" around nightmare. Although with dropbox the latter has got easier.
In the end, the proposal gets turned into a word doc though for final formatting, because it is what people expect.

In terms of change tracking, I find this only works in word for a view people. Otherwise, you end up with change tracks everywhere and it's just an unreadable mess. Tex/latex in a versioning system can also work, although again only with so many changes and only if everyone is geeky enough to be able to use it.

I wish their were a perfect workflow, but there really isn't.

Phil

Re:A word processor?

[superdude72]

I've copy edited books for the consumer market (mostly test prep for IT certifications) and we used Word for the job. It's not terrible, as long as you use stylesheets instead of physical markup. And no way was I going to force the author to use LaTex.

Re:A word processor?

[Anonymous Coward]

Well, the Word-for-collaboration approach isn't generally passing the document around -- you host it on some network filesystem so anyone can access it.

Still not as powerful as versioning systems, but also much easier for most people to set up and use, and nowhere near as bad as your presumed system of finding out who has the latest version and emailing them for it -- though that still occasionally happens...

Re:A word processor?

[reg106]

The real magic of Word's Track Changes feature is the way it allows you to quickly see what's been changed and commented on in the *formatted* document. Marginal comments and changes pop out visually when you scroll through the document, and the displayed comments can be filtered based on the user who wrote them. This is an especially nice feature for giving feedback to the main author on collaborative writing projects. I can imagine a Latex package that would allow you to place comments in the margin with an arrow to some highlighted text with coloration based on, say, a username embedded in a field of the comment. A flag could optionally suppress the comments in the compiled document (so that the comments don't *have* to be removed.) If such a package existed, it would go a long way toward matching track changes (when combined with a versioning system). To my knowledge, there is nothing quite like this for latex at the moment?

Re:A word processor?

[serviscope_minor]

The real magic of Word's Track Changes feature is the way it allows you to quickly see what's been changed and commented on in the *formatted* document.

I know how they work--I've used them. The whole thing fails miserably when several people make concurrent edits to a document, then you have a merge nightmare. Also, the whole attribution thing fails when people don't setup windows with their real name (noone I know ever seems to do that).

The visual display is nice, but it is way too brittle. Having done this on both systems on large documents, I'd take a proper versioning system any day. BTW: Latex does allow marginal notes and optional comments with the right package.

With the right specials you can click on a place in the DVI file and it jumps to the right line in your editor. If you editor has VCS integration, you can find out who wrote the line trivially.

The tools exist but they are less integrated, though superior.

Re:A word processor?

[Anonymous Coward]

Change tracking for normal people using Google Docs

http://docs.latexlab.org/docs

Re:A word processor?

[serviscope_minor]

near as bad as your presumed system

it wasn't a presumed system :(

I set up a webdav share so that the scattered project members could all access the document and edit it. What happened is that everyone would grab it and make changes then upload it, trashing everyone elses changes. Or they would get a copy, edit it and email sections (not even the whole document around). It was terrible. No amount of explanation would get people to use it in a remotely sane manner.

Basically, the main problem is due to concurrent editing. One really needs a full-strength versioning system for that.

Re:A word processor?

[johanneswilm]

I went away from Word and alike a long time ago as I really could automate making certain design choices, so it really didn't make a lot of sense. I wrote in Kile for a long time, but when I needed to give editors access to the file, I figured that this would be too complicated and that LyX was probably as far as I would be able to stretch it. And indeed, they had no problems working in LyX.

Re:A word processor?

[reg106]

I've used both systems for small, medium, and large documents as well. For documents that are written by large numbers of authors in short amounts of time (like grant proposals), I like Word with Track Changes. I find the visual display of changes in the formatted document focuses attention and speeds up the writing process. I'm fine with non-WYSIWYG editing in general, but the visualization of changes is just really effective.

I know how to return clicks from xdvi and yap to an editor. But it's the coauthors' edits that I'm interested in, not just finding my place in the source file or identifying who wrote a particular line.

I've used versioning systems with latex documents as a single author as well as with coauthors. It worked great as a way to keep a centrally located authoritative version of the document. One thing I didn't like was that versioning systems often pay attention to whitespace, so trivial changes in the line wraps would be reported as large changes in the document. (This was with CVS. Perhaps newer systems understand text better.)

With latex documents for which someone else is the primary author, I often end up writing comments on a printout, scanning, and returning. This generally ends up being a better use of my time than trying to teach a coauthor (often not a programmer) about versioning systems.

Re:A word processor?

[WillKemp]

Word & OpenOffice documents look like ass. You'd need Adobe before it'll look as nice as LaTeX.

Either that, or you need to learn how to use LIbreOffice/OpenOffice/Word properly!

Re:A word processor?

[nick_urbanik]

The learning curve is such that for any even moderately long document, it is worth learning latex from scratch rather than using a word processor.

Once you're used to it, it is so much simpler to use.

When teaching in Hong Kong, a fellow lecturer was asked to create a template for exam papers. He did so in Word, but using inconsistent spacing, manually created using the spacebar and other blue-collar methods. I wrote my own style sheet in LaTeX, and my exams conformed to the Departmental Standard better than his model did. One problem is that many users of Word and other word processors have not invested sufficient time in learning how to create actual style sheets that take care of formatting rather than using low-level manual processes that look inconsistent and amateurish. The effort required to understand how to do this well is not much less than that required to understand how to produce good results in LaTeX.

Re:A word processor?

[Anonymous Coward]

Disclaimer: I am a CS guy,. so computer technology and new programs don't scare me :-)

When majoring in CS at the University I started writing my thesis using MS Word, since I wasn't so willing to learn LaTeX.
That would be an extra effort which would have diverted my attention from my thesis work: I'd have preferred to keep things apart since there was so already little time for doing the 'Science' part of the thesis. AH, by the way, I needed to insert many images (graphs) and formulas.

But Word isn't IMHO suited for such kind of documents, and I needed to be able to reformat the document appearance easily, without having to fuss on every part of it manually, for example to add references. After the first chapters I understood I needed something else and I was about to surrender and start learning LaTeX.

Then one evening browsing I discovered LyX, which let me actually forget the LaTeX under it and concentrate on my thesis.
And the output quality, choosing the most suiting one between the different styles available, was very high and clean, tidy.

At the end I've use LyX even to import and typeset my code, where I was able to add references too.

I've compared my thesis outlook with the one of others who used MS Word: their was very ugly, with a mix of fonts, styles....

LaTeX the problem

[Anonymous Coward]

LaTeX is the problem, not the FOSS movement. It's horribly obsolete and makes almost everything harder than it needs to be. LaTeX and TeX are the wrong answer for everything, natural science included, unless you have to set large amounts of mathematics.

Re:LaTeX the problem

[hugortega]

It's clear that you've never written a document larger than 100 pages with multiple chapters and a lot of equations and figures. Yeah, I love LaTex.

SGML?

[Anonymous Coward]

I don't know if it meets your needs, but did you look into SGML and all the related facilities? That suite seems to enjoy good support, especially in Debian-based distros.

When copyleft restricts which tools may be used

[tepples]

But if you want to get something real done, it's just stupid to limit yourself to only open source OR proprietary software. Pick the best tool for the job.

Be careful: sometimes, especially in cases of works under a "copyleft" or "share-alike" license, a work's copyright license limits which tools for the job are lawful. For example, some licenses require works to be made available in an editable format that isn't Java-trapped.* See, for example, sentences containing "Transparent" in the GNU Free Documentation License and sentences containing "technological" in CC BY-SA. You can use proprietary tools yourself, but you also have to make sure that the work can be edited with free tools.

*Term's original is historical, prior to IcedTea.

Re:When copyleft restricts which tools may be used

[Anonymous Coward]

Be careful: sometimes, especially in cases of works under a "copyleft" or "share-alike" license, a work's copyright license limits which tools for the job are lawful

What are you talking about? This would only be relevant if we were discussing templates and packages to be embedded as part of the document, it has nothing to do with software.

(If you still don't understand: GPL/GFDL/CC-SA only affect derivative works [derivative as defined by copyright law], a document made in MS Word is not subject to the copyright of MS Word unless you decompile Word and paste the code into the document. This simple fact is why you can (and many people, most obviously Apple, do) compile proprietary applications using GCC. DISCLAIMER: I'm talking about copyright law here, not contract law. If the MS Word EULA [which is a contract rather than a copyright license] says that all MS Word documents must be copyrighted a certain way then that may be a problem, fortunately I'm not aware of any applications that have such a clause)

i heard you liked to describe descriptions

[Anonymous Coward]

"Over more than most of the timely use of my time of 3 years I ..."

I heard you liked to describe descriptions so I put an adjective next to your adjective to describe your description

CSS paged media

[tepples]

If graphics and equations were required, I would have moved to a generic HTML + css method.

Most web browsers that I've seen are based on the model of rendering a web page to a scroll that is 960px wide by infinitely tall. But in the real world of print, the codex has replaced the scroll. The paged media module in CSS3 is still only a Working Draft. So which web browser would you recommend that has thorough support for MathML and for CSS paged media?

Re:CSS paged media

[CronoCloud]

It doesn't matter to most people, that's an edge case. Practically every time you ask for "recommendations" it's for an edge case that simply isn't important to most people. Which is why there aren't any web browsers that I know of that have the support you're mentioning. So simply put, it's not an issue. If you want to be a zealot on open source issues go ahead...but its not practical.

Re: Stallman link

[Richard_at_work]

I don't think anonymity is dead, I think its quite alive and kicking.

What did die was the expectation that others should protect your anonymity for you.

Calibre?

[sheepe2004]

Googling pdf to epub turns up this: http://lifehacker.com/5509965/how-can-i-convert-pdfs-and-other-ebooks-to-the-epub-format It talks about an open source program called Calibre which can apparently convert pdfs to epub (and many others). So presumably pdflatex followed by this would give exactly what you needed.

The best tool is the one you already use

[Phillip2]

I have helped to create a site for scientists to post their articles on the web. One of the problems is that academics tend to love their tools and do not want to switch, often because they have relatively elaborate workflows and practices, which can cope with their lives; whether this involves writing lots of maths, spending lots of time offline travelling, collaboration or whatever.

We got around this just using Wordpress. Many of the tools out there can already communicate with a blog: this includes Word which, like it or not, is the main tool that scientists use. Others have mentioned things such as asciidoc (which I use). It's okay for short articles, but for a thesis, I would want to use latex. The support for editing in asciidoc is just not as advanced, particulary if you want to do crossreferences, citations, graphs and so on.

There is currently not a good latex -> HTML solution -- in the end, I used PlasTeX to create a tool unimaginatively called latextowordpress. Not perfect, but it works okay in most cases.

http://www.russet.org.uk/blog/2010/08/latex-to-wordpress/

Once you are in wordpress epub and PDF fall out for free, as there are standard plugins for generating these. Personally, I don't do so; I have not found any substantive advantage over HTML, but they are there if you want them.

The process of publishing in this way is not entirely slick, but the results are quite nice. See http://knowledgeblog.org/ and subdomains for examples. And even if the process could be improved, my experience suggests that it is easier than using a commercial publisher. In many cases, it is even less error-prone, as you can see the final published form as you are going, without human intervention in the way.

HTML and LaTEX are not semantic markup

[strangeattraction]

First semantic markup refers to enhancing the text by providing information about its meaning. HTML (CSS) and LaTEX specify the layout of the text regardless of its meaning. Secondly the open source packages are a little behind the curve regarding ePub and support for MathML in browsers. Having a coherent tool that publishes to PDF, HTML , ePub and supports equations well would be a great boon to scientific and technical publishing. BTW it also needs to be scriptable. That said if people know of something I don't please fess up I do scientific publishing and you would save me considerable time if solutions already exist.

Re:considering if it's open source

[TaoPhoenix]

Sorry, I resoundingly disagree.

Looking at your brand new user name, some members would call your post a broad shill for all proprietary closed programs.

The entire point of Open Source is that it can be moved to new innovative uses. Open Source will be slightly-to-much harder to use in many cases! But that is not the point of Open Source! The point is that a valid computing experience can be made out of open components. Yes, someone will have locked down the "1-click" version of a feature with a patent. So it takes you three clicks. Three clicks is way better than spending an hour kludging it.

What rock did you just crawl out from under.

[bobs666]

Richard Stallman is one of the geniuses of our time. All he wants is to share software with his friends. and to have his friends share it back.

johanneswilm also wants to share. Its clear by his wanting to use FOSS in his process.

Perhaps you need to rethink who is the crackpot here? The answer is clear here.

Re:What rock did you just crawl out from under.

[Anonymous Coward]

I really value RMS's work, but most people can't make a living by being the Free Software Pope.

Re:What rock did you just crawl out from under.

[ConceptJunkie]

I think it's fair to say RMS is both a genius and a crackpot. His goals are laudable, and I support them, but what he's willing to sacrifice to achieve them isn't necessarily so.

Re:What rock did you just crawl out from under.

[pnewhook]

Richard Stallman is one of the geniuses of our time. All he wants is to share software with his friends. and to have his friends share it back.

No. If thats all he was doing then fine. But he wants everyone to ship free source and to not do so is an attack on freedom. He wants the word GNU to be used in every project that uses GNU. Thnks GNU is the primary architect of Linux. Likes socialist regimes and has worked with Chavez. Used to live in his office. Thinks Sept 2001 was an inside job. He scans everything he buys with an RFID scanner just in case someone is tracking him. Wants to boycott books like Harry Potter.

Basically an extremist crackpot.

Don't follow RMS.

[jellomizer]

You should listen to him, then you need to judge for yourself if what he is saying is Insightful or just Dribble.
RMS is very intelligent, but an Utopian idealist, who looks at situations more academically then practically, has a strong distrust of opposing ideal, so he is closed minded.
If you just follow RMS then you are not thinking for yourself. RMS has some good points and great ideas, and other just mindless rants to rage against "The Man".
Being that RMS calls for a boycott of something doesn't mean one should follow it.

Re:Don't follow RMS.

[Anonymous Coward]

"Dribble" and "mindless rants" you say. Got a link to prove your hostile claims?

And nevermind the fact that rms never asked anybody to boycott ebooks, like others have pointed out above.

Re:Boycott? I Think the Tools Merely Lack Maturity

[strangeattraction]

There is an open free market for Scientific publishing called PLoS http://www.plos.org./ PDF's suck on eReaders mainly due to the fact the text does not reflow for different size readers. The reason eReaders don't support ePub as well as they should is because most eReaders are not sold for profit but to hook you into the distributor's DRM'd products ala Amazon. It is not their priority. Converts just suck. Enough said.

Is there a compelling case for EPUB?

[The Famous Brett Wat]

My version of e-publishing was, "write the thesis in LaTeX, output in PDF via pdfLaTeX, and upload the PDF to Google Books." Instant global accessibility for anyone that wants it (well, instant after the processing period) -- certainly a heck of a lot better than any exposure my University can offer, although I gave them the PDF too, and they supposedly make it available somewhere. It's not EPUB, sure, and I would convert it to other formats if I felt that the effort was worth it, but maximising availability was more important to me than making it convenient for small form-factor e-book readers. I considered EPUB, but I feel that PDF is good enough, particularly given the effort that went into making it look nice in its published dimensions.

If I were going to write another book, however, I'd finish my half-baked "writer's mark-up language" project first. It's a markup language designed to be writer-friendly, medium-agnostic, and readily translated into other forms like HTML and LaTeX for actual rendering. I don't have any immediate plans to write another book, though: writing the thesis has taken the edge off my enthusiasm for the subject for now.

PDF is fine

[evanh]

What more do you want? PDF works well as an e-document.

Re:PDF is fine

[Phillip2]

Actually, it doesn't. The results don't look that great, don't work in a web browser, often fail in screen-readers, are harder to archive, very difficult to text extract from. PDF is really pretty much a legacy format.

Re:PDF is fine

[grumbel]

The results don't look that great, don't work in a web browser,

Neither does ePub. With PDF the situation is actually a little better as Chrome will get (or already has) native PDF support and Plugins for other browsers are also rather widespread. That ePub is internally just HTML in a Zip doesn't really help when the browser doesn't have a way to deal with that.

This is one of the weird things with browsers that I never really understood: They are essentially the primary tool for consuming text these days, yet they are incredible shitty at actually handling something that is the size of a book or heck, they even fail at rendering something nicely that is just raw HTML, violating every rule of good typography with their default style (Readability and other third party hacks however help).

Re:PDF is fine

[Grishnakh]

No, it doesn't.

If all you want to do is download a file and print it on Letter-sized paper (or A4, assuming the PDF is in A4), then PDF is great.

However, if you want to view it on a screen, especially a screen that's smaller than letter-size, it sucks. Maybe you haven't noticed, but ebook readers are all smaller than letter-size paper, so it's physically impossible to view a PDF page on an ebook reader without either panning, or shrinking it. Panning around to read a page is annoying, and shrinking it will make it difficult or impossible to read (depending on the font size and the ereader's algorithm), plus it's even worse if the viewer has poor eyesight and prefers larger fonts.

This is the entire reason that ebook formats were invented, so that readers could dynamically resize and re-flow text, instead of being stuck with a fixed page size. Of course, with PDF, instead of defaulting to Letter size, you could format your document for a page size equal to the ebook reader's screen size, and make it look great on that ebook reader, but only that one. They don't all have the same size screen, so you'll need different PDFs for every single ebook reader out there, which flies in the face of the "Portable" aspect that PDFs tout. Plus you'd still need one in Letter size for anyone who wants to print out the document.

Re:PDF is fine

[Overzeetop]

Yeah, I'm always worried about using a browser to read...WTF?

PDF is about the only format where you can mix a lot of elements, get it to look like you meant it to, and has a huge array of readers on nearly every known platform. No, it won't reflow well, but if you want you formatting preserved you don't want reflow anyway. It's being used in more and more industries as an archival format beacuse it is so easily readable. Text extraction is simple; my reader/editor of choice can easily pull text out of a document - just highlight and copy.

Re:PDF is fine

[Captain Damnit]

PDF is about the only format where you can mix a lot of elements, get it to look like you meant it to

Ah, but therein lies the problem...if you meant for the document to be printed on an 8.5x11" piece of paper, and I instead try and read it on my 7" Nook (rooted and running CyanogenMod, of course...), then we have a problem. Namely, that I need to zoom in and scroll around, when the text should just reflow.

Re:PDF is fine

[CronoCloud]

No, it shouldn't reflow...PDF is intended for documents that KEEP their formatting the same on all platforms/viewers no matter what, so that it's the exact same document with everything in the same place every time.

Once e-readers get higher resolution (and larger) screens, it won't be a problem.

RMS against ebooks?

[Compaqt]

While it's true that ebooks present the possibility of digital restrictions management, Smashwords, a ebooks distributor site, doesn't use DRM, AFAIK.

pandoc is your answer...

[Lumpy]

http://johnmacfarlane.net/pandoc/

Unless you do some really wacky latex stuff, pandoc works great

org-mode exports to pdf, html and OpenDocumentText

[complex_pi]

It is based on plain text and allows cross linking, references, equations. http://orgmode.org/

Why Not Just...

[Greyfox]

Write a new DVI outputter for epub?

I really like TeX and LaTeX (But don't google on the latter without some additional modifying keywords...) and used to maintain my resume in it. Turns out most contracting companies don't want a static document they can't modify, so I ended up dropping the whole thing into a big E-Lisp data structure which I serialize into eieo objects and then emit to some other markup language. I wrote emitters for HTML and Plain Text, but really you can do anything. I have it on my to-do list to rewrite it in another object oriented language one of these days, and possibly change the storage format to XML. It wouldn't be hard to get it in XML format -- I'd just have to change my HTML markup emitter a little bit.

What else you want?

[Faisal Rehman]

All you need is proficiency in latex and that's it. There is no shortcut. Be patient and become expert. The more expert you are the more fast and productive you will be. Hifi tools make you lazy.

WHY CONVERT!? There's a New Kid on the Block.

[rjbradlow]

Living in a Winblows world with all the brainwashed money and fanboys behind it, its hard to fight mass ignorance. Enter FastPencil.com... http://www.fastpencil.com/company/media http://www.fastpencil.com/company/learning_center#TB_inline?height=648&width=800&inlineId=importing-pdf

Wrong format

[frisket]

LaTeX is a formatter: it does an excellent job of typesetting and can produce publication-quality PDF (my company uses it all the time for this).

But it is not a reusable file format. The only processor that really groks it is TeX, and that can only output PDF or DVI. If you want multiple formats of output under program control, you need to use a output-agnostic file format like XML, from which you can generate LaTeX and any other kind of appropriate source code to create PDF and other kinds of output (eg HTML, XHTML, and whatever else you need).

I very strongly suggest you use TeX4ht to convert what you have into an intermediate format of XML, and then postprocess it and clean it up to create a master copy in a well-known and robust XML format like DocBook or TEI, and start from there. There are then plenty of good tools that will let you create multiple outputs, including eBooks, web sites, topic maps, even Braille and voice output...and of course LaTeX, to regenerate your original, if needed.

<plug>I am chairing the session on XML and Publishing at this year's XML SummerSchool in Oxford next month. Come and find out all about it.</plug>

Re:Wrong format

[nick_urbanik]

I very strongly suggest you use TeX4ht to convert what you have into an intermediate format of XML, and then postprocess it and clean it up to create a master copy in a well-known and robust XML format like DocBook or TEI, and start from there. There are then plenty of good tools that will let you create multiple outputs, including eBooks, web sites, topic maps, even Braille and voice output...and of course LaTeX, to regenerate your original, if needed.

Among all the noise in this "discussion", this is the first post that actually addresses the problem.

Re:Wrong format

[johanneswilm]

Indeed, this is what I did. If you check the tex4ht email list, you'll see a ton of my posts over the past month. My script mainly cleans the stuff up that tex4ht could not do on its own. Btw, that script is here: http://www.johanneswilm.org/download/compile

Re:Wrong format

[johanneswilm]

well, actually, I just converted it directly to xhmtl/html, which I can import relatively easily to Epub. The issue is getting acceptable output from tex4ht in the first place. I needed something that could do it again and again, so that I could test the epub-output while the editors continue to fix errors. Also, I hope to be able to use my script for similar projects in the future. I just found it strange that I needed to do this -- it seems fairly simple.

Re:Wrong format

[nick_urbanik]

Btw, that script is here: http://www.johanneswilm.org/download/compile

Thank you very much; you have asked many good questions on the tex4ht list, and the answers have also been illuminating. Your script is very specific to your "SANDINISTA DISSERTATION" (copied and pasted from the script). It assumes a very specific file layout, but does not have defaults that cope with the basic simple case of a LaTeX file that includes some images. But thank you for making it available; I think it (and your discussion on the tex4ht list) will help me write a simple program to cope with the basic case.

It may be impertinent of me, but I offer this gratuitous advice: the additional effort to write a generally more useful tool is not great; you can simply write a script to call it with the options that apply to you. Your code can also create directories if they are not present, and give more specific help for the first time user of your program rather than exception backtraces. I guess this is the difference between code written for a specific purpose by a researcher and that by a programmer who wants to make a generally useful tool. I appreciate that your main effort is your research, and that it is generous of you to provide your useful code, which others such as I can benefit from. Your code, after suitable modification by a persevering programmer, is the solution to the question posted in this ask.slashdot article.

Re:Wrong format

[nick_urbanik]

Your code, after suitable modification by a persevering programmer, is the solution to the question posted in this ask.slashdot article.

Since you are the original asker of the question, I see you are after a better answer than that you created yourself. (Silly me!). Okay, I'll see what I can do, even though I will write Perl (but good Perl!)

Transparent

[tepples]

This would only be relevant if we were discussing templates and packages to be embedded as part of the document

In the case of documents under the GFDL, the copyright license requires that those who distribute copies of the document also make copies available in a "Transparent" form, one editable using free software. So those who make derivative works have to make derivative works available in a "Transparent" form. Or perhaps I misunderstood "Transparent" in the GFDL; what am I missing

Re:Transparent

[Anonymous Coward]

A life.

Re:Transparent

[CronoCloud]

Sigh, Aspies.

Look, just because one uses GPL tools, it doesn't mean that any document created by such tools is automatically GFDL. That's what you're missing. One has to CHOOSE to use GFDL for their documents. And most people don't, it's that simple.

Yeah, do it the hard way!

[Paracelcus]

Even grandpa can do it!

CutePDF writer (www.cutepdf.com) Just print from your application.
Calibre to convert PDF to any other format (calibre-ebook.com).

Done something similar last year

[godrik]

I wanted to have an article readable on my kobo on the go. That was not so easy but not so difficult as well. epub is mainly compressed xml (or xhtml can not remember).

The procedure to obtain the proper html goes through compiling the latex with pdflatex and bibtext so as to have a proper pdf AND intermediate latex files. Extract the bibliography information from the intermediate file. Regenerate the bibliography in html format using bibtex2html. Play with head and tail to cut header and footer.

Then, generate the main document in html using a latex2html tool. I think one of the best one is hevea. It leaves some header and footer that can be once again removed with head and tail. I did not had chapters since it was an article, but if you do, you probably want to cut your latex source in independent chapters to have one html document per chapter.

Now you have everything you need in an html format, converting to epub is a few sed away to go from html to whatever fooML format epub is composed of. Finally write some TOC (or extract it from the latex intermediate files) and compress the whole thing.

Congratulations, you have an epub.

That's a pain in the ass to have to do that, but that is not very difficult as well. I'd love to see a tool do that automatically though.

Re: Stallman link

[Oligonicella]

Cash is hard to use why? I walk into a store. I pick out a CD, a DVD and a book. I proceed to checkout and pull out a fifty. I get change and leave.

Wherein lies the difficulty?

There was a gap so I filled it a bit.

[Anonymous Coward]

I write with Scrivener so my entire tool chain isn't open source but I wrote XHTML2EPUB to create an EPUB from my project.

Instead of thinking about the toolchain per se it goes from MMD -> XHTML -> EPUB with working directory for tweaking the CSS and whatnot.

Re:There was a gap so I filled it a bit.

[johanneswilm]

Ah, thanks a lot! My main problem was actually getting it into HTML. From there it was not that difficult, using Calibre.

Re: Stallman link

[Anne Thwacks]

Wherein lies the difficulty?

The store has 87 CCTV cameras, all linked to face recognition software.

Re:Wordpad

[JordanL]

Or you could build yourself a PHP tool that does it, as I did here. (Disclaimer: intended for my personal use, so basically no user-friendliness at all.)

Creating ePub's is surprisingly easy from a programing perspective.

Open Document Format works fine

[dwheeler]

I wrote PhD thesis using Open Document Format (ODF) and it worked out very well. I used OpenOffice.org, but I expect that LibreOffice would work at least as well. You can translate to many other formats (e.g., with "Save As" or external tools).

As with any big writing effort, one key is to separate formatting from content. You should FIRST set up a template with that does all the formatting, including all the paragraph types you'll need and the right format for them. Then write your document, selecting the paragraph types for each paragraph as appropriate. Do NOT embed formatting commands in the document itself - paragraphs should NOT have font settings, etc., but instead these should be controlled by the paragraph's paragraph type. In my case, I created an OpenDocument template for George Mason University (GMU), and gave it to GMU so others could share it. If you create a template, please share it with others.

Calibre?

[Anonymous Coward]

Calibre [http://calibre-ebook.com/] is open source and doesn't a decent job of converting between various e-book formats. I use the PDF to EPUB conversion frequently for my nook.

epub is XHTML

[Arancaytar]

LyX already supports output to XHTML. Merely generating XHTML output and renaming the file extension to epub has a good chance of working.

Some things might have to be tweaked, however, and that would require changes to the LyX source code.

Re:epub is XHTML

[johanneswilm]

For the humanities/social sciences you need to use many of the features which currently only are accessible in biblatex -- which still is not supported by Lyx. Also other needed packages, such as pgfplots are not supported.. That's why I needed to go via a script. Btw, that script is here: http://www.johanneswilm.org/download/compile

Re:Boycott? I Think the Tools Merely Lack Maturity

[PuckSR]

PDF is great for a lot of things, but not for readers. Sure you can read your PDF on your Android, but you better be prepared to zoom like a mad-man.
If I am reading a paper on the geo-political state of Belize, why would I need to be zooming so much? Just let the text be reflowed to my device. This is why epub and similar are the way to go, because they allow a great deal more portability. Why does the author of the paper care if I maintain his margins or his font? He doesn't.

I get so many different types of information each day via PDF when it really should be in another format that I am sick of it. If you are sending me a document that is about to be published and I am proof-reading the layout, then send me a PDF. If you are sending me an architectural drawing then send me an Autocad DWG!!!

Re:Boycott? I Think the Tools Merely Lack Maturity

[swv3752]

Pretty much only the Kindle does not support ePub. Everything else pretty much supports ePub well.

Before my current career (mid 00s) I was an editor

[aussersterne]

and worked, over several years, as an editor in technology publishing, then as an editor in history publishing, then as an editor in academic publishing (journals in the social sciences). In all cases, we worked in Word and sent Word files to the production team/department/contractor. Their processes upon receipt of Word files varied, but the fact is that MS Word is a major standard in many areas of publishing, certainly in mass-market books.

A document for which someone chose to use GFDL

[tepples]

Look, just because one uses GPL tools, it doesn't mean that any document created by such tools is automatically GFDL.

I know that. But all derivative works of a GFDL work are GFDL and therefore must be made available as a Transparent copy. I apologize for not making it clearer in the first place that I was referring to GFDL works.

One has to CHOOSE to use GFDL for their documents.

And if I'm tasked with converting a document for which someone chose to use GFDL, I have to abide by the GFDL. This means, as I understand it, that the toolchain must support starting with a Transparent copy.

Re:A document for which someone chose to use GFDL

[CronoCloud]

And if I'm tasked with converting a document for which someone chose to use GFDL, I have to abide by the GFDL. This means, as I understand it, that the toolchain must support starting with a Transparent copy.

Your understanding is wrong...the GFDL only applies to the distribution...not the toolchain. So you can convert said document to an opaque format, as long as you include a link to a transparent one. You're interpreting things as being more complex and involved than they really are.

I did solve it.

[johanneswilm]

Hey everybody, most of the comments seem to assume that given I write about the politics of Nicaragua, I surely cannot have solved this. That is incorrect, indeed I did wrote a monster spaghetti code script that does what I set out to do. It works with graphs made with pgfplots, etc. . I just wondered why I needed to do something that I saw as a fairly basic task. You can find it at: http://www.johanneswilm.org/download/compile

LaTeX and PDF are not ebook formats

[Anonymous Coward]

You are suffering from a fundamental misunderstanding of what an ebook is. An ebook is not any old digital file that contains text. For a digital file to be an ebook, it has to be reflowable. That means that a computer has to be able to relayout its text and images intelligently at different screen sizes, automatically. In order for a computer to do that, with a level of understanding, short of actually understanding the text, the text in the digital file has to be marked up semantically. That is paragraphs, headings, caption, etc have to identified as such and there must be no drawing commands like, "move to this position and draw this character".

PDF is a page layout format (I'm ignoring PDF tagging which is a bolted on after thought). It is designed to represent text and images at a *fixed* page/screen size. LaTeX is also a page layout format, which is why you use LaTeX to generate PDF, but you cannot use it to generate HTML (without ugly hacks that dont work for an arbitrary LaTeX document).

So if you want to publish in both PDF and an actual ebook format, you need to start with a semantic format like XML/markdown/etc. Then use tools to convert that semantic format to ebooks and to PDF. There will never be a tool (opensource or closed) that can convert a PDF that has non trivial non-formatting into a decent ebook automatically.

Re:Boycott? I Think the Tools Merely Lack Maturity

[Anonymous Coward]

I've owned several generations of eBook readers (MS Reader/Acrobat on a Casio EM-500, Franklin EB-911, eb1100, Sony PRS500, PRS300, PRS600, and a first gen Kindle DX). Almost all of them notionally supported PDF. Out of all of those, only the Kindle DX does a decent job of displaying PDF documents. In fact, I had a first gen Nook that I didn't keep for more than 2 days because PDF support was so crappy.

One size does not fit all

[tepples]

[Styling for printing] doesn't matter to most people, that's an edge case.

But it's an edge case important to the recommendation that Anonymous Coward made: preparing a document that will eventually be published in a paged medium using "a generic HTML + css method".

Practically every time you ask for "recommendations"

I'll confess that I ask for recommendations to avoid "just Google it". I've found Google useful for learning more about something that I already know exists, not so much for discovering that something exists (gets tripped up on synonyms) or for finding what is the best practice (no indicator of a source's reliability).

it's for an edge case that simply isn't important to most people.

Part of what makes us individuals is that our respective situations each have different edge cases. One size does not fit all.

If you want to be a zealot on open source issues go ahead

You're right: The comparison of CSS layout engines at Wikipedia claims that IE and Opera are ahead of Firefox and Chrome in support for paged media, though I have no idea how up-to-date that is.

My fault: Transparent is looser than Source

[tepples]

I guess that's one way to go about it. I must have misremembered the extent of "Transparent" in GFDL, having confused it with "Corresponding Source" from the GPL. It appears that a "Transparent" copy need not be the ultimate source document that is processed into the copy that is sold, just something machine-readable that preserves the semantics of the original source document. It is a form for making modifications rather than the preferred form for making modifications

Re:UTF-8 and LaTeX

[mcswell]

"And LaTeX does not handle UTF-8 very well at all." Use XeLaTeX, it comes with the TeXLive distro.

I use DocBook XML, and convert it to XeLaTex (alternatively, to plain LaTeX) using dblatex. But if you don't like XML tags, I suppose you won't like DocBook. (There are programs that effectively hide the tags, like XMLMind.)

Write in Markdown

[Anonymous Coward]

Markdown or one of it's derivatives like Multi-Markdown is much easier than LaTex for any writing without formulas. Gruber's Perl script will turn the text into clean HTML. There are several sources that provide the needed XTML templates to turn the HTML into ePub or mobi formats. Add a little CSS and you're golden.

Its Open Source, step up and code your needs

[Anonymous Coward]

Do not wait for anyone else to do a work that does not benefit them. If you want someone else to give you a professional solution where you only need to ask for features to be considered go for a commercial solution. Thats how it works.

Re:Boycott? I Think the Tools Merely Lack Maturity

[Anonymous Coward]

>>Most readers handle PDF, I would just stick to the output of LaTeX

I assume you don't own a e-ink reader. PDF support tends to be poor to nice-but-better-it-have-been-formatted-to-be-printed-in-a-postcard. And most of user preferences cease to work for most PDFs out there, or break badly. When I have to read a PDF, most of times, i read it in my laptop. Which sucks. I would never do the same with a EPUB, a FB2 or any other sane e-book format. It is a printing format, describes presentation in paper; any real ebook format describes content.

PDF is a printing format. Don't let Adobe and their DRM-happy friends convince you it's a web or ebook format. It is not.

Go LaTeX-XHTML

[Anonymous Coward]

There are dozens of converters out there, I would personally recommend LaTeXML (http://dlmf.nist.gov/LaTeXML/) and if that fails try TeX4HT (http://www.tug.org/applications/tex4ht/mn.html)

  I think people have already done the work for a full TeX->ePub translation with LaTeXML, but you should drop a line on their mailing list to make sure.

careful you don't get a latex allergy

[Anonymous Coward]

because sheepskin is expensive. and saran wrap is risky.

Word Processor and Adobe InDesign?

[Anonymous Coward]

I have not produced any large books, but the above worked very well for a 40 page book. Create the content and basic text formatting in Word/Text/LibreOffice/etc copy and paste into arranged text areas in InDesign.

Re:org-mode exports to pdf, html and OpenDocumentT

[nick_urbanik]

It is based on plain text and allows cross linking, references, equations. http://orgmode.org/

Let's see:

Org-mode is like a Swiss army knife. People use it for Getting Things Done (GTD), as a Day Planner, as a Notebook, for Web and PDF Authoring, and much more.

Now why isn't dissertation and thesis there in that list? I wonder. Hmmm. Let me think now....

Re:Wordpad

[damn_registrars]

Microsoft Write then Wordpad for all my assignments. Those are free

No.

Microsoft Write is not free. Just because it comes with the operating system you paid for does not make it free.