Slashdot Mirror
Stephane Rodriguez Dismantles Open XML
Elektroschock writes "Stephane Rodriguez, a reengineering specialist who became popular for his article on MS Office 2007 binary data, now comprehensively debunks Microsoft's new Open XML format. With small case studies he demonstrates the impossible challenges third-party developers will face. His conclusion: it is 'defective by design.' Next week members of the International Standard Organization are likely to approve the format as a second official ISO standard for office documents, even though most nations have submitted comments. Rodriguez claims he is 'not affiliated to any pro-MS or anti-MS party/org[anization]/ass[ociation].'"
This is not proof of OOXML being defective by design. It only shows that apparently MS's software isn't able to handle OOXML properly.
-- Cheers!
But that's still a problem. Microsoft's implementation becomes the de facto standard and all others must (attempt to) conform to the behavior of that implementation or be judged defective. This is what happened when MS published the MAPI (Mail API) spec and then released an implementation alongside it. Lotus and others could never fully mimic what the MS implementation did, so they eventually languished.
"by design" is of course about motivation which we can know in OOXML from emails, quotes, obtuse or brittle design, and lack of specification.
The document contains all of these. I suggest that you read it.
By the way -- there's newly discovered undocumented Microsoft tech present in OOXML, such as SSPI ("Security Service Provider Interface") which is a proprietary Microsoft developed protocol for security providers, and OLE ("Object Linking and Embedding") which is for embedding (eg, taking an Excel spreadsheet and putting it into a Word document). This is undefined in OOXML only available on Microsoft Windows.
Um, isnt the fact that not even Microsofts own software can handle OOXML which btw. is designed by Microsoft themselves, proof enough that something is seriously wrong with the design of OOXML?
I mean if not even the maker of OOXML can get it to work properly in its own products, how are third parties supposed to do it? And if no one is able to implement OOXML correctly, what is this "standard" good for besides being a great smoke-and-mirrors tactic by Microsoft themselves?
OOXML is a theoretically perfect standard that just happens to have no implementations whatsoever.
http://rocknerd.co.uk
Stéphane is a French male name. The female version is Stéphanie.
It's deliberate. The standard is just a distraction, to keep competitors busy trying to implement it, while documents are actually being created in the Office 2007 variant of OOXML. A few months of legacy almost guarantees a transition to the real OOXML would be an uphill battle, especially with no real documentation of how *either* format works. So even with a supposed 'standard' and a near-enough implementation, the vendor lockin is just as strong as it was with the binary formats.
Sam ty sig.
Sent: Saturday, December 5 1998
To: Bob Muglia, Jon DeVann, Steven Sinofsky
Subject : Office rendering
One thing we have got to change in our strategy - allowing Office documents to be rendered very well by other peoples browsers is one of the most destructive things we could do to the company.
We have to stop putting any effort into this and make sure that Office documents very well depends on PROPRIETARY IE capabilities.
Anything else is suicide for our platform. This is a case where Office has to avoid doing something to destroy Windows.
I would be glad to explain at a greater length.
Likewise this love of DAV in Office/Exchange is a huge problem. I would also like to make sure people understand this as well.
I'm not saying this as some linux nut job but its things like that which just drive me nuts. Regardless of which ever os I prefer that kind of thinking just boils my blood.
How can any committee deciding on open standards seriously take a company which has been proven time and time again to play by its own rules and whenever it offers something labeled OPEN its about as open as the doors to Fort Knock are to the average person.
This "OpenXML" stunt is just a smokescreen covering Microsofts controlled retreat in the office format battle. It only needs to keep parties distracted until Microsoft has reclaimed the control over business content by means of vendor lockin v2.0 aka Microsoft Office Sharepoint Server.
/ 2007/04/while_you_were.html
http://weblog.infoworld.com/openresource/archives
http://www.itbusinessedge.com/blogs/mia/?p=198
This is not proof of OOXML being defective by design. It only shows that apparently MS's software isn't able to handle OOXML properly.
OK, lets have MS have their choice either way on this one.
If their office tools work well but are not using the OOXML spec, they must be using some other spec, perhaps MOOXML. In which case they are not OOXML compliant.
On the other hand, if they want to be OOXML compliant then I guess Redmond programmer can't read their own spec and thus are having problems being compliant.
Either way, and for whatever reason Microsoft is not compliant with their own spec. Shall we call this MOOXML? And while I have only read a part of the spec, it is far too "undefined" and thus ambiguous to be reliable used by itself. A standard needs to be defined enough, that 2 or more parties could take the standard document specifications, run off and program it from scratch. And have a reasonable chance that their code will inter operate on the same data sets.
Trouble is, if Microsoft cannot do that, how is anyone else?
But might I submit, Microsoft wrote office and then wrote the spec. A poster child of why you think about and write the spec before the software is a good practice.
Still it's better than the original DOC format.
A DOC is actually a FAT12-like filesystem (called OLE) that has files and clusters. Clusters can be lost and files can be fragmented. One of the files is the document's text; it's not plaintext but rather another obscure binary format, with text chunks seperated by some kind of metadata (my brain nearly exploded when trying to understand how to separate text from the metadata and I gave up). Images, videos and embedded objects are stored as separate files in the OLE file.
Instead of a simple *.zip file with an HTML-like text file they invented a completely fucked up format that gives people nightmares. The only point is making third-party compatible applications is extremely difficult, but the plan seems to have backfired because even Microsoft's own Word Mobile doesn't work well with native *.doc files (and ironically, Documents To Go for PalmOS works better with DOC than Word Mobile!)
Interesting experiment. However, I suggest you do not title your posts "Can anyone repro?" on Slashdot. The answers you get may be, well, .... exciting and very, very scary.
The relevant code from an ODF spreadsheet:
<table:table-row table:style-name="ro1">
<table:table-cell/>
−
<table:table-cell office:value-type="float" office:value="123456.123456789">
<text:p>123456.12</text:p>
</table:table-cell>
</table:table-row>
If Office can't read OOXML files produced by other tools, and other tools can't read Office OOXML files, where do you suppose end users will place the blame?
And what do you suppose users will do when faced with incompatibilities?
It's a brilliant strategy: Define a new "standard" but don't quite implement it yourself, ensuring that no one can implement a competitive office suite that is compatible with yours. Further, make the standard complex and weird enough that you can always blame inconsistencies on the other implementations. Voila! You get to proclaim to the world that your de facto standard office suite supports an open, ISO-blessed international standard format -- but with no worries about losing your lock-in.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
So? It's still perfectly valid XML even without the BOM. XML it's a real standard and I suggest you read it, it's not Notepad.
And don't even start talking about malformed UTF-8 since he only used characters in the ASCII subset, so even saving it as Latin-1 would have generated valid XML.
There's a hidden treasure in Python 3.x: __prepare__()
Uh, UTF-8 files do not need a BOM. What the fuck is the point of a byte-order-mark on an encoding that is byte-order neutral?
One of the advantages of UTF-8 for text files is that you don't need a BOM. With XML it's even easier because, as you point out, the XML declaration ("XMLDecl" in the spec) header can contain the "EncodingDecl" to tell explicitly you the file is in UTF-8. If the EncodingDecl says UTF-8, and the file is encoded in UTF-8, then if an XML parser cannot handle that, it's seriously fucked an needs to be fixed.
You might also want to go read STD-63 at some point. It points out that there are a few problems with using BOMs in UTF-8, and that if there is a way for UTF-8 to be determined in a way other than with the use of a BOM, that should be used instead. Given that XML specifically includes support for an "EncodingDecl" in the "XMLDecl", it is clear that best practices dictate that you *shouldn't* use a BOM when working with UTF-8 encoded XML files. Even if your tools _insist_ on writing BOMs to such files, they had *better* still be able to work if the BOM is missing.
Heck, with OOXML, you could also use the ZIP's manifest file to keep track of file metadata like the character encoding.
Why doesn't the gene pool have a life guard?
"..Next week members of the International Standard Organization are likely to approve the format as a second official ISO standard for office documents.."
Err.. Next week news called, they want their draft story back.
There is no certain outcome of next weeks vote; and the fact that we even are discussing the defects of OOXML are proof that the ISO body will have much problems just waiving this through. Please refrain from taking sides just because this is an 'Microsoft-standard'.
I'd say it's possible that OOXML will NOT be approved next week. It will probably have to take the long road through the ISO as a real standard proposal, not just a fast-tracked 6000 page gorilla.
"-Who said sit down?!"
-- S. Ballmer @ MSDC 2003.
Oh, I just love being schooled by AC's who don't know what they're talking about.
So, there are numbers that floating point formats do not represent well. However, the world is not floating point numbers. And computer math is not just floating point numbers.
The number is stored in the XML as an ASCII represented decimal real number. They're not stored as binary floating point numbers and they shouldn't have the kind of brain damageness that floating point has.
Let's look at what's going on here.
User enters a number in a decimal format. User sees the number in a decimal format displayed on the screen. Excel apparently does not use floating point or it's got a lot of compensation because if you do things like multiple 12345.12345 * 100000000 you get 1234512345000 and not some weird approximation. I would guess that the XML output routine is using floating point (and why would be a good question).
Why is this a problem? Well, we don't know how many digits of precision to work with here or how to round things. If I write an app to work with the spreadsheet I'd probably use something like a Java BigDecimal to handle the numbers. But, I don't know how to round things out so that I get the right numbers. If I use a BigDecimal, 12345.123449999999 is going to be 12345.123449999999. If I multiple by 100000000 I will get 123451234499.99999 instead of 1234512345000 as I would expect from looking at the values that were put into the spreadsheet.
Excel should be putting the proper values out in the XML or the standard should define the form of rounding/conversion to be applied.
No, I don't think so. It will serve Microsoft's purposes better if they too cannot properly implement the OOXML standard. Then their fully proprietary file formats would continue to be used since no one could trust that an OOXML document hasn't been corrupted by the OOXML save process.
This is how Microsoft destroyed the nascent RTF standard that the US Navy wanted to use: they implemented it, but gee there were problems in getting it to work right so maybe all you sailor boys should use Word's native file formats until we get things worked out (which never happened).
Windows just don't belong on a battleship or aircraft carrier. You would have thought the US Navy would have known that, but no, they had to go and try it anyway.
Don't forget the delicious language. Instead of the legendary "syntax error", we now get a "catastrophic failure". Do it yourself FUD!
(Scene at office)
ComputerGuy: "Sure, let's open that with GoogleApps."
Colleague: "Why am I getting a catastrophic failure? Maybe I better use Excel."
My first Journal Entry ever, in 8 years! http://slashdot.org/journal/365947/aphelion-scifi-fantasy-horror-poetry-webzine
Stephane has for a long time presented a weak case against OpenOffice XML.
"1) Self-exploding spreadsheets"
His top issue "1) Self-exploding spreadsheets" has been discussed on Brian Jones' weblog:
http://blogs.msdn.com/brian_jones/archive/2007/08/ 15/why-there-s-no-microsoft-in-open-xml.aspx
It boils down to: the fact that is XML does not mean that you can modify it in any way you want; There are rules for modifying the schema and Mr Stephane is not happy with that. Had he followed the actual rules he would have had no issue.
This is a case where two locations must be updated per the spec; He can avoid updating the two locations by removing the chainCalc.xml file (which is optional, and Excel will reconstruct). He later gets upset because if he does that, he claims performance on load will be slower.
"2) Entered versus stored values"
His second point in "2) Entered versus stored values" in an interesting distinction between entered values and stored values. It reflects the way that Excel works (and so does Gnumeric) by storing the values instead of the data that was entered by the user. This responds to the need of the spreadsheet to do something interesting with the data, for example when you enter a date, it is stored as a number with a format applied not as a string. This allows computations on dates to happen based on the underlying numeric value. The featured is used extensively by spreadsheets.
In the Excel/gnumeric case you have to generate a single value, in the ODF case you must generate and update the two values (which just a point before, Stephane was having a seizure about).
The precision issue that he brings up, I suspect is merely an issue with double format precision. He claims that the data is unusable and there is a loss of precision, but handing that out to a C compiler will produce the expected result with no loss of precision. I do not know how "atof" or the compiler work internally to cope with this issue, but at least my libc/gcc combo does not have this problem.
I would not be surprised if this is an artifact of floating point, someone with more background on doubles and floating point math could probably answer the question with more authority, but a cursory read of "What Every Computer Scientist Should Know about Floating Point" seems to validate that there is no error in the floating point representation for the values that he uses: http://docs.sun.com/source/806-3568/ncg_goldberg.h tml
3) Optimization artefacts become a feature instead of an embarrasment
His 3rd point is open for debate, like the 1st case, we have a case where he has to handle things differently. Stephane sells a commercial product to handle Excel files and I suspect that his product has to cope with the same patterns in different ways, which has naturally upset him. OOXML might be inspired by Excel's needs, but it does not mean that it has to be a 1-to-1 match.
4) VML isn't XML
VML is labeled as "deprecated" in the OOXML documentation (Section 8.6.2, page 25) and it states: "The VML format is a legacy format originally introduced with Office 2000 and is included and fully defined in this Standard for backwards compatibility reasons. The DrawingML format is a newer and richer format created with the goal of eventually replacing any uses of VML in the Office Open XML formats. VML should be considered a deprecated format included in Office Open XML for legacy reasons only and new applications that need a file format for drawings are strongly encouraged to use preferentially DrawingML."
So the standard basically says "VML is still in use, but its better to use DrawingML". Stephane misconstrues the above statement and tries to portray this as evil
Yep. Brilliant, isn't it. Given a horribly complex and incomplete specification, Microsoft can easily blame any problems on the other tools -- and they can do this with a straight face because they'll be right! (Quietly ignoring the fact that their own tool produces non-compliant OOXML). Even better, they can smugly point out how their tools fix the "errors" caused by other crappy tools, even as the text of their messages frighten users away from trying any tool that doesn't come from Microsoft ("catastrophic failure", no less!).
If MS weren't trying to pull a fast one, they'd have designed a more reasonable format, one that does make it practical to make small edits to the XML and expect reasonable results or, even better, used an existing standard like ODF. If ODF can't fully represent all facets of Office documents, the format has a well-defined technical and procedural path to add any necessary extensions.
By way of comparison, try the same series of experiments with a .ods document, using any of the handful of available applications that supports it, and you'll quickly see how a format that is designed to be straightforward, accessible and specifiable in less than 500 pages compares to the brilliantly-executed monstrosity that is OOXML.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
-Q(1) What does Rodriguez's article show?
-Q(2) is OOXML in and by itself flawed?
-Q(3) What's the practical relevance of the question whether OOXML is flawed?
-Q(4) So what's in it for Microsoft? Why do they bother?
-
- Q(1) : What does Rodriguez's article show?
- A(1) : Rodriguez's article show that the OOXML format written by latest Microsoft Office applications, among them MS Excel, is:
- sorely defective in that you can't be sure to get your original data back after saving it to OOXML
- impossible to change outside MS Office applications
- tied to the MS Office way of representing internationalised versions of documents because "of the way Microsoft chose to store XML using the US English locale, no matter how good your implementation is, you have to retrofit it to work just like Office does" in order to accommodate internationalised documents
- MS Office legacy formats supported throughout, greatly (and unnecessarily) contributing to the size and complexity of the 6,000 page standard.
- Q(2): Is OOXML flawed in and by itself?
- A(2):Yes, I think so, partly because of Rodriguez's article, partly because of flaws documented elsewhere: see http://www.noooxml.org/petition The points 2,3,4,5 listed there seem especially crippling to me:
(2) There is no provable implementation of the OOXML specification: Microsoft Office 2007 produces a special version of OOXML, not a file format which complies with the OOXML specification;
(3) There is information missing from the specification document, for example how to do a autoSpaceLikeWord95 or useWord97LineBreakRules;
(4) More than 10% of the examples mentioned in the proposed standard do not validate as XML;
(5) There is no guarantee that anybody can write software that fully or partially implements the OOXML specification without being liable to patent lawsuits or patent license fees by Microsoft;
- Q(3): What's the practical relevance of the question whether OOXML is flawed?
- A(3): Enormous. We currently see that Microsoft is trying to convince the world to accepted OOXML as an ISO "standard", whereas it's no such thing. It's too loosely defined, and opposed to the existing Opendoc standard there is no open-source reference implementation. So there will be a morass of possible implementations, of which only Microsoft's own implementations will be guaranteed mutually compatible. That's a polite way of saying that Microsoft simply aims at continuing its format lock-in, only this time the under the name of OOXML.
- Q(4) : So what's in it for Microsoft? Why do they bother?
- A(4) : Well ... Microsoft has a policy whereby it quite explicitly does not want other people's software, let alone Open Source software, to render MS Office documents correctly.
For reference, see this email, (cited from Rodriguez's article):
Is that
My guess is, yes, it occurred to the poster you were responding to, since I highly doubt that when he wrote exactly that, it was in his sleep. Did it occur to you that reading his post all the way to the end might have resulted in slightly less of your foot being inserted into your mouth?
San Francisco values: compassion, tolerance, respect, intelligence
I don't think you intended it that way, but you should be aware of the vast number of people you just insulted. US English and US dates are only "canonical" in the minds of US citizens. If not for Microsoft purposely and determinedly screwing up the implementation of anything but US standards in their software the usage would have no traction at all.
The majority of the "English speaking" world still uses the English language and English formats and standards, not US variant ones. The fact that the USA has seen fit to re-invent English, still refer to that as English, and then foist it on the rest of the world doesn't make it "canonical."
As the author of this article so aptly describes, date formats and language implementations are a multi-stage nightmare in Office. To the point that the majority of users even in English speaking countries like Canada, Australia, New Zealand and the UK itself, often end up using American English and American dates simply because Office is the only game in town and you cna only bash your head against the wall on these things for so long. That doesn't make it right, and that doesn't mean that those users wouldn't be happier and more productive if they were not forced to use a US standard when they may have not even traveled to the US.
Any kind of English except the US variant, is severely broken in Office and always has been. Your answer sounds to me a lot like: "So what, they should all be using our standards and language anyway." Not helpful at all, and illogical as well.
> But that's still a problem. Microsoft's implementation becomes the de facto standard
> and all others must (attempt to) conform to the behavior of that implementation or
> be judged defective.
It's worse than that. Since MS defines a number of aspects of the specification solely
in terms of compliance with MS application software, the MS implementation is not only
the -defacto- standard, but the very explicit standard. Not only can no one conform
to a sufficient level to be judged compliant in the marketplace, for all contractual
specifications, -nothing- but MS software can -ever- be 100% compliant.
This means on big, contract driven projects, such as many government projects, MS
and vendors using MS tools are effectively the only possible competitors, unless
the contracts and specifications specifically waive vendor compliance with those
parts of the spec.
And I strongly doubt anyone would ever write a contract like that.