Slashdot Mirror

← Back to Stories (view on slashdot.org)

Ask Slashdot: What Is the Best Open Document Format?

Posted by timothy on from the when-plaintext-just-won't-do dept.

kramer2718 writes: I am working on a project that requires uploading and storing of documents. Although the application will need to allow uploading of .docx, doc, .pdf, etc, I'd like to store the documents in a standard open format that will allow easy search, compression, rendering, etc. Which open document format is the best? Since "best" can be highly driven by circumstances, please explain your reasoning, too. Have a question for Slashdot's readers? Take a look at other recent questions first to see if someone else has had a similar question. And if not, ask away! The more details and context you include, the more likely your question will be selected.

35 of 200 comments (clear)

  1. can't you search the current doc types? by alen · · Score: 3, Informative

    if you use the API's supplied by their creators?

  2. PDF/A by thechemic · · Score: 5, Informative

    http://www.pdfa.org/2011/08/pd...

    --
    Let's make like a bird... and get the flock outta here.
  3. Don't convert needlessly by PSVMOrnot · · Score: 5, Insightful

    I would suggest, unless you have a pressing need to convert them, that you should store the documents in the formats they are uploaded in.

    Whenever you convert a document you run the risk of completely messing up the layout, style, etc.

    1. Re:Don't convert needlessly by Anonymous Coward · · Score: 5, Interesting

      Or store both the original, and a standardized format. The place I work stores everything from engineering drawings, meeting minutes, purchase records, to manuals of old equipment in a central document library. It retains the original file, and makes a pdf of every file, and a link to both is listed in each entry. We've already had some older CAD formats no longer supported by current software we have easy access to, but the old pdfs are still readable and it is cheap enough to find some intern to re-create the document from the pdf if need be.

    2. Re:Don't convert needlessly by darkain · · Score: 2, Interesting

      All of the "X" variants of MS Office documents stand for "XML" - that is, the documents are stored in a series of XML files inside of a ZIP file that is renamed to formatX (docx, xlsx, etc). There is no real need to even have Windows or Office installed to index these documents. Just write up a basic script to extract the ZIP file and parse out the related XML documents. Note: this isn't as trivial as it sounds at first, though. This would assume that Microsoft's XML structures (yes, plural), had an easy to comprehend standard that was logical to work with. It'll take a little digging but totally doable.

      TLDR: not by choice, my company heavily relies on Excel documents, and this is how I ended up managing them, importing their contents into a SQL database for indexing and other purposes .

    3. Re:Don't convert needlessly by AthanasiusKircher · · Score: 4, Interesting

      Or store both the original, and a standardized format. The place I work stores everything from engineering drawings, meeting minutes, purchase records, to manuals of old equipment in a central document library. It retains the original file, and makes a pdf of every file, and a link to both is listed in each entry.

      THIS.

      PDFs (or some similar standard) will ensure that the original documents can be read by everyone and viewed with the original formatting intended by the person creating them. Any differences in the version of Word or whatever is going to tweak the formatting in unpredictable ways.

      But the originals should always be retained, since it may make future editing easier. And people also won't be stuck trying to undo whatever unpredictable reformatting or editing (e.g., loss of certain features moving between formats) might go on in your conversion process.

  4. .txt by Anonymous Coward · · Score: 5, Interesting

    .txt. If you need pretty formatting, fill it Latex tags.

    1. Re:.txt by jythie · · Score: 5, Insightful

      And here I am without mod points...

      Generally when I have to worry about integration or longevity, it is still hard to compete with ASCII & LaTeX. While they do not have the every day visibility of various office document types or pdfs, renderers, search tools always know exactly what to do with them. They can even interact with version control systems cleanly since the underlying tools do not need to know anything about the formatting to manipulate it.

    2. Re:.txt by TechyImmigrant · · Score: 3, Insightful

      How is it impractical?

      --
      I should use this sig to advertise my book ISBN-13 : 978-1501515132.
    3. Re:.txt by ShanghaiBill · · Score: 2

      How is it impractical?

      It is impractical because the average end user will have no idea what to do with a .txt file containing Latex markup. It will look like gibberish. Txt files also have no clickable table of contents, or index, or hyperlinks to other documents.

    4. Re:.txt by Desler · · Score: 4, Informative

      Then you end up with Microsoft inserting garbage characters at the start of each text file to make their job easier, breaking scripts and confusing both users and other editors alike.

      It's not a garbage character. It's a BOM and it's part of the Unicode standard. If your scripts and text editors can't read the BOM in 2015 then they are the things that are horribly broken.

    5. Re:.txt by TechyImmigrant · · Score: 2

      Does latex support MS Office? If not then it's very impractical for 95%+ of users.

      If I had this question, I would google before asking Slashdot and exposing my ignorance. There are lots of tools.

      --
      I should use this sig to advertise my book ISBN-13 : 978-1501515132.
    6. Re:.txt by ClickOnThis · · Score: 3, Insightful

      If you are in publishing - like in writing or editing books - you need MS Word.

      Well, use whatever you want to write the book. But if you are printing it, I'd definitely use something other than MS Word. It just doesn't produce publication-quality documents.

      And as far as Latex is concerned, it would be even more work.

      That depends on what you are writing. If your document contains lots of equations and you're using MS Word, then God help you.

      Latex and other formats are great if you are in complete control from start to finish of the publishing process. But working with other people that are scattered all over the World? Nope.

      Again, that depends on who the other people are. Many academics, particularly scientists, use LaTeX.

      --
      If it weren't for deadlines, nothing would be late.
    7. Re:.txt by Darinbob · · Score: 4, Insightful

      MS Office is also impractical for 95%+ of its users.

    8. Re:.txt by Yaztromo · · Score: 4, Insightful

      It's not a garbage character. It's a BOM and it's part of the Unicode standard. If your scripts and text editors can't read the BOM in 2015 then they are the things that are horribly broken.

      This is one of those sticky situations. For UTF-8, the Unicode standard discourages the use of a BOM, unless you're converting from a different Unicode format that requires a BOM. The whole purpose of a BOM is to describe the byte order used to generate the file data, however UTF-8 data is broken up into 8-bit code units, and thus endianness doesn't play a role. You simply read the stream one byte at a time.

      Indeed, using a BOM is discouraged (by both the Unicode standard and the IETF) precisely because it breaks backward compatibility with ASCII text processors. Unfortunately, Microsoft seems intent on adding an unnecessary (and, in the case of UTF-8, badly named) BOM to virtually every UTF-8 file created on their platform. This is done to make it easier for them to detect the encoding; however there are reliable, published heuristics which do the same job without the need for the BOM. That's what every other platform in existence does to detect UTF-8 streams. Microsoft's BOM use is purely to make their processing easier, even if it means that it breaks backward compatibility with older tools.

      Thus, you are technically both correct. It's technically not a garbage character at the beginning of the stream, however it is unnecessary, and contrary to the way every other OS on the planet handles the situation.

      (I've run into this more than once in my professional life, dealing with people who are supposed to be technically minded who use Windows Notepad to try to figure out what encoding a file is using. I've had them come back claiming my files weren't UTF-8 because Notepad claimed they were 'ANSI' (never mind that there is no character encoding standard called 'ANSI' in the first place). I've had to explain to more than one person that standard ASCII is valid UTF-8, even going so far as to providing them chapter and verse of the Unicode specs to prove that what Notepad says shouldn't be treated as gospel.)

      Yaz

    9. Re:.txt by Yaztromo · · Score: 2

      And yet I use a multitude of text editors and have scripts that can handle UTF-8 text files with a BOM just fine. Your programs and scripts are broken if they can't.

      Or they're legacy tools. There are a large number of such tools out there that do various jobs, where having an unnecessary BOM is a liability.

      If you're compiling for some legacy embedded hardware, for example, I have little doubt that its compiler would choke on BOM characters, and you may not have access to the source to fix it. And just because YOU don't need or use such tools hardly means that nobody out there does.

      Yaz

    10. Re: .txt by billDCat · · Score: 2

      Perhaps fine for Roman characters, not so fine if the document contains Kanji, Hiragana, Katakana, Hebrew, or any of the other character sets that don't play nice with "plain text" formats. For something that you would think would be pretty straight forward, plain text character handling is surprisingly maddening to work with.

  5. Forget the Universal Format crap by xxxJonBoyxxx · · Score: 5, Informative

    1) Forget the Universal Format approach - your users will kill you for messing up their formatting, and you'll never get complete feature parity
    2) Store the docs in their original format
    3) Get Apache Solr to search your content
    4) You'll be spending a lot of time on #3, so leave time to tinker

    1. Re:Forget the Universal Format crap by Anonymous Coward · · Score: 2, Informative

      I work at a typography, and I get a lot of documents from a lot of different people. Those "documents" come as MSWord files with missing fonts, pdfs made with some shoddy software, strange ODTs, many more different types of doc files, the mysterious lnk files that work perfectly fine for them, but not for anyone else and my personal favorites, jpg files (not png or some other lossless format, because that would imply actual thinking).
      Strange enough, I've yet to receive any plain text files.

      To index everything, I use calibre, just something simple like changing the file name to "Project name - tag 1, tag 2, tag 3 - Client name.pdf" and it imports it automatically from a folder I drop it in, adding that info to the metadata.
      It doesn't actually index or search inside the files, but they are easier to find and handle.

  6. Oldes are the bestes by Anonymous Coward · · Score: 5, Funny

    Word Perfect Document, because it's been consistent for nearly 20 years. it has a simple underlying format, it's more finely granular than HTML and because I just like obsolete things.

    1. Re:Oldes are the bestes by Megane · · Score: 3, Funny

      I think WordStar has problems with Unicode support. But then again, so does Slashdot.

      --
      #naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
  7. Need more information by nine-times · · Score: 4, Insightful

    As an IT person, I hate questions like this. There's not enough information to give a solid answer. For example:

    * What kinds of documents are you talking about? Text? Photos? Spreadsheets?
    * What is the source of the documents? Are these currently printed out documents that need to be scanned back in? Are they currently digital, and in a particular file format?
    * What will people need to do with them when these documents are retrieved? Do they need to be able to edit the documents?
    * How much does formatting matter? If someone retrieves the document in 5 years, will it be important that all the line breaks and page breaks are in the same place? Does it need to have all of the correct fonts? Or are you more interested in being able to have access to the information itself?
    * When you say that the application will need to allow ".docx, doc, .pdf, etc", what formats are in "etc"?

    There may be many other relevant questions, my point is that there just isn't enough detail here. In general, if the most important thing is that you have a printable document that you want to be able to print out from any machine, maintaining the formatting as much as possible, then PDF is a pretty good choice (be sure to embed the fonts and include searchable text!). If you already have a bunch of Word documents and you want the formatting unchanged, and would like the capability to edit the document after it's retrieved, then I'd typically just recommend keeping it as a .docx. It keeps things simple, will be widely supported, and prevents the risk of something going wrong while you're converting to another format. If you like the idea of using .docx because of what I just said, but want something more "open", then ODF is probably worth looking into.

    Really, there are only so many choices, and each have advantages depending on your specific needs.

  8. For Two-Millennia Durability... by nightcats · · Score: 4, Insightful

    ...you can't beat bamboo strips. The oldest original versions of Lao Tzu's Tao Te Ching are written on rolls of bamboo strips. Not sure how they scan electronically, and you will have to keep your pet pandas away from them, but for document durability, you can't beat that format...

    --
    Development is programmable; Discovery is not programmable. (Fuller)
    1. Re:For Two-Millennia Durability... by jfdavis668 · · Score: 2

      We carve 16 bit Unicode into stone slabs for long term backup storage.

    2. Re:For Two-Millennia Durability... by gstoddart · · Score: 5, Informative

      Nonsense, bamboo can't touch papyrus for longevity, and you don't need to worry about pandas.

      Damned bamboo shills.

      And don't anybody go suggesting cave paintings, it's a completely dead platform.

      --
      Lost at C:>. Found at C.
  9. And a pony too? by gstoddart · · Score: 3, Insightful

    Although the application will need to allow uploading of .docx, doc, .pdf, etc, I'd like to store the documents in a standard open format that will allow easy search, compression, rendering, etc. Which open document format is the best?

    Lets' see ... you want to allow uploading in a large number of formats .. you want to magically turn it into a universal format ... while retaining all of aspects of the original ... and will be easily maniuplated ... and you want it in an open, and documented format? And all for free?

    I want one of those too. And a Red Rider BB gun with a compass in the stock and this thing which tells time. And a new skateboard. And a pony.

    Honestly, you're asking for the holy grail of document management systems ... the universal, lossless document format.

    I'm not sure it exists. And I'm not sure companies like Microsoft or Adobe would allow it to exist.

    --
    Lost at C:>. Found at C.
    1. Re:And a pony too? by gstoddart · · Score: 3, Informative

      English idiom connoting yet another impossible thing in a child's unrealistic wishlist ... typically placed at the end of a series of outrageous demands: " ... and a pony".

      Now, please, don't make me pedantic you again to explain the cromulency of phrases. ;-)

      --
      Lost at C:>. Found at C.
  10. "Best" depends on intent by Phoenix+Rising · · Score: 2

    On the conversion side... If you're taking in PDFs created by a layout/page design program, then you're not likely to get good satisfaction converting them and storing them as something other than PDFs. OTOH, if you're taking in a lot of documents created in an office suite, and they have collaborative notes, and you need to retain the documents for legal purposes, then converting them to PDF is going to lose data.

    On the future use side: PDFs are slower to render and search than most formats; they're harder to alter, but they're more reliably rendered than any other format. Office documents offer richer content and easy editing; their layout may vary depending on the output device (good and bad), and office document formats seem to change a bit more than other document types. HTML with CSS is good, and probably now stable enough that future clients will render something similar - but it's not PDF for reliable formatting, nor office docs for feature richness; editing tools for HTML aren't all that intent on preserving what came before. LaTeX is a reliable formatter wrapped around text-centric documents, but it's not something most people will be able to use and edit.

    Each document type has its reasons for being - you'll need to decide why you need to store your documents and what you need them for in the future. Retaining the original document along with a text conversion stored and indexed in a search engine may be your best bet - or not.

    --
    Let us live so that when we come to die, even the undertaker will be sorry -- Mark Twain
  11. Docbook? by Enry · · Score: 3, Funny

    Docbook allows you to separate out the content from the presentation. You write in XML and define paragraphs, chapters, images, etc. and then leave it to the various stylesheets to drive how it looks like when it comes out the other end - PDF, HTML, Word, whatever, and the stylesheet makes sure that if some features are supported (hyperlinking from the table of contents to the chapter) it'll be included in there. Since the content is in plain 'ol XML you can use any kind of XML processor to go through it..

  12. Re: Starlight Glimmer 2016 by gstoddart · · Score: 2

    Dude ... can I point out to you that you got the reference and that many of us wouldn't know WTF it was?

    So maybe your question is how many other Slashdotters are Bronies besides you? ;-)

    And, for the record, I included that link because I had to google it to find out what it meant.

    Now if you will excuse me I need to go apply brain bleach. The images which came up in that google search are terrifying.

    --
    Lost at C:>. Found at C.
  13. How about by pahles · · Score: 2

    Markdown? It's easy to write, read, render, compress, search.

    --
    Sig?
  14. LaTeX CoNDoM by tepples · · Score: 3, Funny

    LaTeX is the CoNDoM that protects you from Microsoft Office ViRuSeS.

  15. Burning Bush by Etherwalk · · Score: 4, Funny

    ...you can't beat bamboo strips. The oldest original versions of Lao Tzu's Tao Te Ching are written on rolls of bamboo strips. Not sure how they scan electronically, and you will have to keep your pet pandas away from them, but for document durability, you can't beat that format...

    Chisel it into stone tablets, then find an ignorant local. Set up a natural gas line to a nearby bush and hide behind a rock. Cub your hands to add a slight reverb effect and tell him to preach the chiselled word, then break the tablets and hide them in a box and trick nazis into looking at them.

  16. Save Space, Switch to ODT by BrendaEM · · Score: 3

    I've written several books. Because ODT's have standard compression, they are usually much smaller. For a 109,683 word book, with styles and formatting:
    ODT: 271,090 bytes
    Docx: 300,057 bytes
    Word 97: 1,379,328 bytes
    PDF: 1,050,788 bytes

    If bytes cost money to store ODT rules.

    Imagine yourself sticking with Word 97 because it's a reliable standard: imagine buying three times as much storage, as well as the backup for the storage,

    --
    https://www.youtube.com/c/BrendaEM
  17. Silly Rabbit, Trix are for kids.... by David_Hart · · Score: 3, Interesting

    No, just no....

    Store the documents in their original format.

    There are many possible reasons why you shouldn't mess with the originals such as formatting, legal implications, loss of content because one format supports stuff that the other doesn't, etc.

    The only way that I could see this working is if you converted everything to an open format but kept copies of the originals and linked to them. But if the plan is to dump the original documents, then it just isn't worth it....