Slashdot Mirror
Stephane Rodriguez Dismantles Open XML
Elektroschock writes "Stephane Rodriguez, a reengineering specialist who became popular for his article on MS Office 2007 binary data, now comprehensively debunks Microsoft's new Open XML format. With small case studies he demonstrates the impossible challenges third-party developers will face. His conclusion: it is 'defective by design.' Next week members of the International Standard Organization are likely to approve the format as a second official ISO standard for office documents, even though most nations have submitted comments. Rodriguez claims he is 'not affiliated to any pro-MS or anti-MS party/org[anization]/ass[ociation].'"
This is not proof of OOXML being defective by design. It only shows that apparently MS's software isn't able to handle OOXML properly.
-- Cheers!
But that's still a problem. Microsoft's implementation becomes the de facto standard and all others must (attempt to) conform to the behavior of that implementation or be judged defective. This is what happened when MS published the MAPI (Mail API) spec and then released an implementation alongside it. Lotus and others could never fully mimic what the MS implementation did, so they eventually languished.
"by design" is of course about motivation which we can know in OOXML from emails, quotes, obtuse or brittle design, and lack of specification.
The document contains all of these. I suggest that you read it.
By the way -- there's newly discovered undocumented Microsoft tech present in OOXML, such as SSPI ("Security Service Provider Interface") which is a proprietary Microsoft developed protocol for security providers, and OLE ("Object Linking and Embedding") which is for embedding (eg, taking an Excel spreadsheet and putting it into a Word document). This is undefined in OOXML only available on Microsoft Windows.
Um, isnt the fact that not even Microsofts own software can handle OOXML which btw. is designed by Microsoft themselves, proof enough that something is seriously wrong with the design of OOXML?
I mean if not even the maker of OOXML can get it to work properly in its own products, how are third parties supposed to do it? And if no one is able to implement OOXML correctly, what is this "standard" good for besides being a great smoke-and-mirrors tactic by Microsoft themselves?
OOXML is a theoretically perfect standard that just happens to have no implementations whatsoever.
http://rocknerd.co.uk
Stéphane is a French male name. The female version is Stéphanie.
It's deliberate. The standard is just a distraction, to keep competitors busy trying to implement it, while documents are actually being created in the Office 2007 variant of OOXML. A few months of legacy almost guarantees a transition to the real OOXML would be an uphill battle, especially with no real documentation of how *either* format works. So even with a supposed 'standard' and a near-enough implementation, the vendor lockin is just as strong as it was with the binary formats.
Sam ty sig.
Sent: Saturday, December 5 1998
To: Bob Muglia, Jon DeVann, Steven Sinofsky
Subject : Office rendering
One thing we have got to change in our strategy - allowing Office documents to be rendered very well by other peoples browsers is one of the most destructive things we could do to the company.
We have to stop putting any effort into this and make sure that Office documents very well depends on PROPRIETARY IE capabilities.
Anything else is suicide for our platform. This is a case where Office has to avoid doing something to destroy Windows.
I would be glad to explain at a greater length.
Likewise this love of DAV in Office/Exchange is a huge problem. I would also like to make sure people understand this as well.
I'm not saying this as some linux nut job but its things like that which just drive me nuts. Regardless of which ever os I prefer that kind of thinking just boils my blood.
How can any committee deciding on open standards seriously take a company which has been proven time and time again to play by its own rules and whenever it offers something labeled OPEN its about as open as the doors to Fort Knock are to the average person.
I tried to repeat the cell changes experiment but I do not see the Excel error.
I bet Mr. Stephane is not saving the sheel xml in utf-8.
The header of the xml file says its utf-8, but he might be saving it without the UTF-8 BOM header.
This "OpenXML" stunt is just a smokescreen covering Microsofts controlled retreat in the office format battle. It only needs to keep parties distracted until Microsoft has reclaimed the control over business content by means of vendor lockin v2.0 aka Microsoft Office Sharepoint Server.
/ 2007/04/while_you_were.html
http://weblog.infoworld.com/openresource/archives
http://www.itbusinessedge.com/blogs/mia/?p=198
You are correct.
That's why the title says "Microsoft Office XML Formats? Defective by design"
not "OOXML defective by design"
He is dissing the Microsofts claims of transparency and openness of Microsoft Office XML
This is not proof of OOXML being defective by design. It only shows that apparently MS's software isn't able to handle OOXML properly.
OK, lets have MS have their choice either way on this one.
If their office tools work well but are not using the OOXML spec, they must be using some other spec, perhaps MOOXML. In which case they are not OOXML compliant.
On the other hand, if they want to be OOXML compliant then I guess Redmond programmer can't read their own spec and thus are having problems being compliant.
Either way, and for whatever reason Microsoft is not compliant with their own spec. Shall we call this MOOXML? And while I have only read a part of the spec, it is far too "undefined" and thus ambiguous to be reliable used by itself. A standard needs to be defined enough, that 2 or more parties could take the standard document specifications, run off and program it from scratch. And have a reasonable chance that their code will inter operate on the same data sets.
Trouble is, if Microsoft cannot do that, how is anyone else?
But might I submit, Microsoft wrote office and then wrote the spec. A poster child of why you think about and write the spec before the software is a good practice.
Still it's better than the original DOC format.
A DOC is actually a FAT12-like filesystem (called OLE) that has files and clusters. Clusters can be lost and files can be fragmented. One of the files is the document's text; it's not plaintext but rather another obscure binary format, with text chunks seperated by some kind of metadata (my brain nearly exploded when trying to understand how to separate text from the metadata and I gave up). Images, videos and embedded objects are stored as separate files in the OLE file.
Instead of a simple *.zip file with an HTML-like text file they invented a completely fucked up format that gives people nightmares. The only point is making third-party compatible applications is extremely difficult, but the plan seems to have backfired because even Microsoft's own Word Mobile doesn't work well with native *.doc files (and ironically, Documents To Go for PalmOS works better with DOC than Word Mobile!)
The relevant code from an ODF spreadsheet:
<table:table-row table:style-name="ro1">
<table:table-cell/>
−
<table:table-cell office:value-type="float" office:value="123456.123456789">
<text:p>123456.12</text:p>
</table:table-cell>
</table:table-row>
No, this is a pretty reasonable thing to point out. It wasn't a value that was undisplayed. When you look at the cell it shows it (in decimal) as 1234.1234 (without the cell rounding). So it shows you that on the screen but doesn't store it properly in the XML file. I would say it's a problem. If it were stored as a binary floating point number in the XML I'd say you might have a point, but if it's displayed on the screen in decimal and then the decimal value in the file is different, that's pretty broken. And it's not just broken, it's now damned hard to work with. What happens if you pull the value from Excel using VBA and then try to change a value in the XML? They're not going to be the same.
Oh nice! So you mean the W3C took it over?
If Office can't read OOXML files produced by other tools, and other tools can't read Office OOXML files, where do you suppose end users will place the blame?
And what do you suppose users will do when faced with incompatibilities?
It's a brilliant strategy: Define a new "standard" but don't quite implement it yourself, ensuring that no one can implement a competitive office suite that is compatible with yours. Further, make the standard complex and weird enough that you can always blame inconsistencies on the other implementations. Voila! You get to proclaim to the world that your de facto standard office suite supports an open, ISO-blessed international standard format -- but with no worries about losing your lock-in.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
For example, the part about "Entered versus stored values" is certainly valid (though I wonder if that's not a problem with Excel itself, and not the format). The complaint about the date format is also on the money.
However, other things seem either wrong or have a bias towards hand editing of the files, e.g. "International, but US English first and foremost". He complains that it uses U.S. English settings. He may not like the U.S., but it's called picking a canonicalized format. Consider the alternative for implementing this in software, parsing of the values in the XML would now depend on settings also found in the XML. That would be insane.
"..Next week members of the International Standard Organization are likely to approve the format as a second official ISO standard for office documents.."
Err.. Next week news called, they want their draft story back.
There is no certain outcome of next weeks vote; and the fact that we even are discussing the defects of OOXML are proof that the ISO body will have much problems just waiving this through. Please refrain from taking sides just because this is an 'Microsoft-standard'.
I'd say it's possible that OOXML will NOT be approved next week. It will probably have to take the long road through the ISO as a real standard proposal, not just a fast-tracked 6000 page gorilla.
"-Who said sit down?!"
-- S. Ballmer @ MSDC 2003.
Oh, I just love being schooled by AC's who don't know what they're talking about.
So, there are numbers that floating point formats do not represent well. However, the world is not floating point numbers. And computer math is not just floating point numbers.
The number is stored in the XML as an ASCII represented decimal real number. They're not stored as binary floating point numbers and they shouldn't have the kind of brain damageness that floating point has.
Let's look at what's going on here.
User enters a number in a decimal format. User sees the number in a decimal format displayed on the screen. Excel apparently does not use floating point or it's got a lot of compensation because if you do things like multiple 12345.12345 * 100000000 you get 1234512345000 and not some weird approximation. I would guess that the XML output routine is using floating point (and why would be a good question).
Why is this a problem? Well, we don't know how many digits of precision to work with here or how to round things. If I write an app to work with the spreadsheet I'd probably use something like a Java BigDecimal to handle the numbers. But, I don't know how to round things out so that I get the right numbers. If I use a BigDecimal, 12345.123449999999 is going to be 12345.123449999999. If I multiple by 100000000 I will get 123451234499.99999 instead of 1234512345000 as I would expect from looking at the values that were put into the spreadsheet.
Excel should be putting the proper values out in the XML or the standard should define the form of rounding/conversion to be applied.
I already know how this is going to turn out.
OOXML will be voted in as an ISO standard.
Third party vender's trying to implement the "standard" will waste time, money and effort and accomplish nothing of import.
MS will continue as normal, claiming support for open standards while locking anyone they can into formats/software they own.
ODF will continue as a marginalized format used by people on the "fringe".
No, I don't think so. It will serve Microsoft's purposes better if they too cannot properly implement the OOXML standard. Then their fully proprietary file formats would continue to be used since no one could trust that an OOXML document hasn't been corrupted by the OOXML save process.
This is how Microsoft destroyed the nascent RTF standard that the US Navy wanted to use: they implemented it, but gee there were problems in getting it to work right so maybe all you sailor boys should use Word's native file formats until we get things worked out (which never happened).
Windows just don't belong on a battleship or aircraft carrier. You would have thought the US Navy would have known that, but no, they had to go and try it anyway.
No. What it means is that Office has so much legacy code that they can't rewrite it all to be conformant. Think of OOXML as a target that MS feels they can eventually meet with office, not necessarily what office will actually meet today. After all, much was changed in OOXML after Office 2007 went to bed. One would expect the next version of Office to be much closer to the spec, since they will have had a full design cycle to conform to it.
If you need web hosting, you could do worse than here
Separating the value and the display solves the problem. As long as the value stored is preserved, other programs can work with it without introducing arbitrary changes. That M$ does not store the exact value and relies on the reader to make the same rounding error is crazy. It's a trap for every system that is not M$, and might not even work across different processors for M$.
I've run into this problem in my own work, where it did not matter. A data acquisition system I used required Winblows. It could write to either text or some nasty binary format. I chose text with a sufficient number of digits to avoid the binary conversion. This blew up my file size, but made it easy to read. In my case, the extra digits were noise anyway and it only gets read once by other programs. In a bank this clearly would not work. In a place where the values must be read and saved multiple times, this would not work. As a programmer, I'm a relative zero but even I can see how broken the M$ way is.
Value storage was only the beginning of OOXML problems. The formula and binary inclusions are even worse. Hopefully, ISO will reject this mess.
Friends don't help friends install M$ junk.
Don't forget the delicious language. Instead of the legendary "syntax error", we now get a "catastrophic failure". Do it yourself FUD!
(Scene at office)
ComputerGuy: "Sure, let's open that with GoogleApps."
Colleague: "Why am I getting a catastrophic failure? Maybe I better use Excel."
My first Journal Entry ever, in 8 years! http://slashdot.org/journal/365947/aphelion-scifi-fantasy-horror-poetry-webzine
But that's still a problem. Microsoft's implementation becomes the de facto standard and all others must (attempt to) conform to the behavior of that implementation or be judged defective.
.doc as if nothing happened? I guess they will do the latter since it's the most economical option for them. If that happens I'm curious what the EU will think of that, and how long it will take before MS is forced to use ODF as standard, if it ever comes to that.
I wonder what happens if OOXML is not voted a standard. Will MS simply discard it, and embrace ODF, or will they continue to use
-- Cheers!
Stephane has for a long time presented a weak case against OpenOffice XML.
"1) Self-exploding spreadsheets"
His top issue "1) Self-exploding spreadsheets" has been discussed on Brian Jones' weblog:
http://blogs.msdn.com/brian_jones/archive/2007/08/ 15/why-there-s-no-microsoft-in-open-xml.aspx
It boils down to: the fact that is XML does not mean that you can modify it in any way you want; There are rules for modifying the schema and Mr Stephane is not happy with that. Had he followed the actual rules he would have had no issue.
This is a case where two locations must be updated per the spec; He can avoid updating the two locations by removing the chainCalc.xml file (which is optional, and Excel will reconstruct). He later gets upset because if he does that, he claims performance on load will be slower.
"2) Entered versus stored values"
His second point in "2) Entered versus stored values" in an interesting distinction between entered values and stored values. It reflects the way that Excel works (and so does Gnumeric) by storing the values instead of the data that was entered by the user. This responds to the need of the spreadsheet to do something interesting with the data, for example when you enter a date, it is stored as a number with a format applied not as a string. This allows computations on dates to happen based on the underlying numeric value. The featured is used extensively by spreadsheets.
In the Excel/gnumeric case you have to generate a single value, in the ODF case you must generate and update the two values (which just a point before, Stephane was having a seizure about).
The precision issue that he brings up, I suspect is merely an issue with double format precision. He claims that the data is unusable and there is a loss of precision, but handing that out to a C compiler will produce the expected result with no loss of precision. I do not know how "atof" or the compiler work internally to cope with this issue, but at least my libc/gcc combo does not have this problem.
I would not be surprised if this is an artifact of floating point, someone with more background on doubles and floating point math could probably answer the question with more authority, but a cursory read of "What Every Computer Scientist Should Know about Floating Point" seems to validate that there is no error in the floating point representation for the values that he uses: http://docs.sun.com/source/806-3568/ncg_goldberg.h tml
3) Optimization artefacts become a feature instead of an embarrasment
His 3rd point is open for debate, like the 1st case, we have a case where he has to handle things differently. Stephane sells a commercial product to handle Excel files and I suspect that his product has to cope with the same patterns in different ways, which has naturally upset him. OOXML might be inspired by Excel's needs, but it does not mean that it has to be a 1-to-1 match.
4) VML isn't XML
VML is labeled as "deprecated" in the OOXML documentation (Section 8.6.2, page 25) and it states: "The VML format is a legacy format originally introduced with Office 2000 and is included and fully defined in this Standard for backwards compatibility reasons. The DrawingML format is a newer and richer format created with the goal of eventually replacing any uses of VML in the Office Open XML formats. VML should be considered a deprecated format included in Office Open XML for legacy reasons only and new applications that need a file format for drawings are strongly encouraged to use preferentially DrawingML."
So the standard basically says "VML is still in use, but its better to use DrawingML". Stephane misconstrues the above statement and tries to portray this as evil
I agree with all that, but I think you're missing something very fundamental: the purpose of a document format is to encode what the user did and what it means. This is the reason why the details of binary floating point arithmetic are irrelevant in this context, and their use in the file a flaw: if the user typed "1234.1234" in the document, the user meant 1234.1234, and the file better guarantee me, author of a program that reads it, that I can find out for sure that the user meant 1234.1234. The trivial way to do that, of course, is to store precisely what the user is shown on the screen, because it is the thing that the user manipulates until it looks right to them, i.e., until they judge that the thing that they see in the screen is what they mean.
This doesn't apply just in spreadsheets, of course; it applies everywhere.
Are you adequate?
Yep. Brilliant, isn't it. Given a horribly complex and incomplete specification, Microsoft can easily blame any problems on the other tools -- and they can do this with a straight face because they'll be right! (Quietly ignoring the fact that their own tool produces non-compliant OOXML). Even better, they can smugly point out how their tools fix the "errors" caused by other crappy tools, even as the text of their messages frighten users away from trying any tool that doesn't come from Microsoft ("catastrophic failure", no less!).
If MS weren't trying to pull a fast one, they'd have designed a more reasonable format, one that does make it practical to make small edits to the XML and expect reasonable results or, even better, used an existing standard like ODF. If ODF can't fully represent all facets of Office documents, the format has a well-defined technical and procedural path to add any necessary extensions.
By way of comparison, try the same series of experiments with a .ods document, using any of the handful of available applications that supports it, and you'll quickly see how a format that is designed to be straightforward, accessible and specifiable in less than 500 pages compares to the brilliantly-executed monstrosity that is OOXML.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
-Q(1) What does Rodriguez's article show?
-Q(2) is OOXML in and by itself flawed?
-Q(3) What's the practical relevance of the question whether OOXML is flawed?
-Q(4) So what's in it for Microsoft? Why do they bother?
-
- Q(1) : What does Rodriguez's article show?
- A(1) : Rodriguez's article show that the OOXML format written by latest Microsoft Office applications, among them MS Excel, is:
- sorely defective in that you can't be sure to get your original data back after saving it to OOXML
- impossible to change outside MS Office applications
- tied to the MS Office way of representing internationalised versions of documents because "of the way Microsoft chose to store XML using the US English locale, no matter how good your implementation is, you have to retrofit it to work just like Office does" in order to accommodate internationalised documents
- MS Office legacy formats supported throughout, greatly (and unnecessarily) contributing to the size and complexity of the 6,000 page standard.
- Q(2): Is OOXML flawed in and by itself?
- A(2):Yes, I think so, partly because of Rodriguez's article, partly because of flaws documented elsewhere: see http://www.noooxml.org/petition The points 2,3,4,5 listed there seem especially crippling to me:
(2) There is no provable implementation of the OOXML specification: Microsoft Office 2007 produces a special version of OOXML, not a file format which complies with the OOXML specification;
(3) There is information missing from the specification document, for example how to do a autoSpaceLikeWord95 or useWord97LineBreakRules;
(4) More than 10% of the examples mentioned in the proposed standard do not validate as XML;
(5) There is no guarantee that anybody can write software that fully or partially implements the OOXML specification without being liable to patent lawsuits or patent license fees by Microsoft;
- Q(3): What's the practical relevance of the question whether OOXML is flawed?
- A(3): Enormous. We currently see that Microsoft is trying to convince the world to accepted OOXML as an ISO "standard", whereas it's no such thing. It's too loosely defined, and opposed to the existing Opendoc standard there is no open-source reference implementation. So there will be a morass of possible implementations, of which only Microsoft's own implementations will be guaranteed mutually compatible. That's a polite way of saying that Microsoft simply aims at continuing its format lock-in, only this time the under the name of OOXML.
- Q(4) : So what's in it for Microsoft? Why do they bother?
- A(4) : Well ... Microsoft has a policy whereby it quite explicitly does not want other people's software, let alone Open Source software, to render MS Office documents correctly.
For reference, see this email, (cited from Rodriguez's article):
Is that
My guess is, yes, it occurred to the poster you were responding to, since I highly doubt that when he wrote exactly that, it was in his sleep. Did it occur to you that reading his post all the way to the end might have resulted in slightly less of your foot being inserted into your mouth?
San Francisco values: compassion, tolerance, respect, intelligence
I don't think you intended it that way, but you should be aware of the vast number of people you just insulted. US English and US dates are only "canonical" in the minds of US citizens. If not for Microsoft purposely and determinedly screwing up the implementation of anything but US standards in their software the usage would have no traction at all.
The majority of the "English speaking" world still uses the English language and English formats and standards, not US variant ones. The fact that the USA has seen fit to re-invent English, still refer to that as English, and then foist it on the rest of the world doesn't make it "canonical."
As the author of this article so aptly describes, date formats and language implementations are a multi-stage nightmare in Office. To the point that the majority of users even in English speaking countries like Canada, Australia, New Zealand and the UK itself, often end up using American English and American dates simply because Office is the only game in town and you cna only bash your head against the wall on these things for so long. That doesn't make it right, and that doesn't mean that those users wouldn't be happier and more productive if they were not forced to use a US standard when they may have not even traveled to the US.
Any kind of English except the US variant, is severely broken in Office and always has been. Your answer sounds to me a lot like: "So what, they should all be using our standards and language anyway." Not helpful at all, and illogical as well.
Then we could judge if his example is reasonable or not. I realize we could all do this ourselves, but I for one am not going to go out and buy Excel 2007 just to do that!
He wanted to remove a formula from a given cell. His first attempt was to simply remove the formula and change the value.
Instead, he has to go update all the reference and dependency information, which programs have to generate and update all the time anyway. I can't really think of a good reason this information needs to be saved to disk, and I certainly can't think of a good reason that Excel deletes the cell, rather than updating the dependencies itself to reflect the physical document.
In fact, I can't think of a good reason to store the value alongside the formula, except as an optional cache, which a program can recalculate if needed.
They are using XML in the first place. The point of XML is interoperability and human-readability/editability, not performance.
Don't thank God, thank a doctor!
> But that's still a problem. Microsoft's implementation becomes the de facto standard
> and all others must (attempt to) conform to the behavior of that implementation or
> be judged defective.
It's worse than that. Since MS defines a number of aspects of the specification solely
in terms of compliance with MS application software, the MS implementation is not only
the -defacto- standard, but the very explicit standard. Not only can no one conform
to a sufficient level to be judged compliant in the marketplace, for all contractual
specifications, -nothing- but MS software can -ever- be 100% compliant.
This means on big, contract driven projects, such as many government projects, MS
and vendors using MS tools are effectively the only possible competitors, unless
the contracts and specifications specifically waive vendor compliance with those
parts of the spec.
And I strongly doubt anyone would ever write a contract like that.
Call me crazy, but unlike Bush I do not divide the world in "them" and "us" I like to live in a world of colors, a world of Pantone if you will and abandon the black and white mentality.
There are good and bad things about Microsoft. When they do something bad, I point it out, when they do something good, I do not see why I would not point it out. I also try to judge everyone with the same metric, I do not use one metric to judge Microsoft, and another one for us.
Stephane's article touches on a subject that I have plenty of experience on (I originally wrote Gnumeric, and later worked with Sun to open source StarOffice and over the years worked to grow the OOo team at Ximian and later at Novell).
Stephane's criticism lacks meat. If someone had done a review of Linux with this level of quality, we would have rightfully called it bullshit.
Miguel.
OO.o also support OLE.
Also, Mac Office supports OLE as well, so it's not "Windows-only".
And you claime that OLE is "newly discovered"? It's been around for over 13 years, and was present in the very first OOXML specs.
I don't know about SSPI, but given that your OLE knowledge is so woeful, I feel safe in assuming that your SSPI complaint is FUD as well.
-- "I never gave these stories much credence." - HAL 9000