Slashdot Mirror
Microsoft Releases Office Binary Formats
Microsoft has released documentation on their Office binary formats. Before jumping up and down gleefully, those working on related open source efforts, such as OpenOffice, might want to take a very close look at Microsoft's Open Specification Promise to see if it seems to cover those working on GPL software; some believe it doesn't. stm2 points us to some good advice from Joel Spolsky to programmers tempted to dig into the spec and create an Excel competitor over a weekend that reads and writes these formats: find an easier way. Joel provides some workarounds that render it possible to make use of these binary files. "[A] normal programmer would conclude that Office's binary file formats: are deliberately obfuscated; are the product of a demented Borg mind; were created by insanely bad programmers; and are impossible to read or create correctly. You'd be wrong on all four counts."
Joel's articles are a joy to read. No matter what time I receive the email about a new article by Joel, it will be read on the spot.
Except... we all don't have this, OLE, thing on our computers nor do we all walk it easier than the languages we deal with now.
But let's say you do. Now you have to find an API to do it for you. As an every day guy, I can write my own HTTP parser, IP connection manager and so forth, w/o requiring special API to do it. As a smarter guy, I'd look for the libraries that can do some of the heavy lifting for me. It's flexibility. The document structure is going to affect how I write code to work with ti.
W/ office docs, Joel is arguing, I have to know the one way to interact with them. There's no TIMTOWTDI about it. There's no intuitive way to do it either. Were the format to be simple, be it "sanely" constructed CSV, XML, RTF, etc, I have more choices. I'd rather use the most well known, bestest of the best, but sometimes it's not intuitive and just hamper's work. It shuts out programmers who would think, open(file); readSomeData(); construct_a_structure();. Now it's, structure = oneOfAHandfulOfParsersThatWillEverWork().
The worst part of that is, since I have no way *I* can choose how to mess with documents. I have to either a) spend more time figuring out the native format unless I'm a genius or have an MS crone behind me, or b) parse it incorrectly, and then have to go back and fix any number of things, including my methodology. Remember how the various encodings affected document format? I.e. UTF-8, 16, Latin-1, Unicode, etc etc etc..
Joel, you're not right.
One may wonder, why release the documentation now?
If you read Joel's blog you'll see the formats are very old, and consist primarily of C-structs dumped to OLE objects, dumped directly to what we see as an XLS, DOC and so on files.
There's almost no parsing/validation at load time.
Having this in a well laid documentation may reveal quite a lot of security issues with the old binary formats, which could lead to a wave of exploits. Exploits that won't work on Microsoft's new XML Office formats.
So while I'm not a conspiracy nut, I do believe one of Microsoft's goals here are to assist the process of those binary formats becoming obsolete, to drive Office 2007/2008 adoption.
I'd assume it has something to do with the antitrust action the EU was taking. Didn't they order that Microsoft had to open all their protocols/formats?
As PJ pointed out over on Groklaw, MS are giving a "Promise" not to sue but this is very very far from a license. Careful analysis suggests that any GPL'd software using these binaries could easily fall foul of the fury of MS lawyers.
Nothing to see here. Move along.
How to look nice and offload some work in one shot.
With this M$ can shut off critics that say proprietary formats are evil, especially those using the long-term viability argument.
Now that the formats are documented, hordes of open source hobbyist can develop (for free) code and tools to read / convert the old Office formats. Then M$ will tell "See, we do not lockout anybody, there are myriads of ways to read our old crap".
Smart indeed. And anyway these format do not hold any competitive advantage anymore since most users are coping with the new ones now.
Just as OOXML files and WMF make references to Windows or Office programming APIs, I think it would come as no surprise to anyone that Office binary formats would also make similar references. The strategy behind it would be obvious -- to tie the data to the OS and to the software as closely as possible.
Spolsky's advice explains that the format code is extremely bad code from the POV of a programmer picking it up to use starting now. Because it grew like a coral reef, starting so long ago that interoperability with anything else but the app's codebase at the time was not in the designs. And every new feature was thrown in as a special case, rather than any general purpose facility for kinds of features or future expansion. The Microsoft legacy that leverages every year's market position into expansion the next year.
But we're not Microsoft, and we don't have the requirements MS had when making these formats. So we should by no means perpetuate them. We should do now what MS never had reason to do: upgrade the code and drop the legacy stuff that makes most of the code such a burden, but doesn't do anything for the vast majority of users today (and tomorrow).
That's OK, because Microsoft has done that, too, already. The MS idea of "legacy to preserve" is based on MS marketing goals, which are not the same as actual user requirements. So that legacy preservation doesn't mean that, say, Office 2008 can read and write Word for Windows for Workgroups for Pen Computing files 100%. MS has dropped plenty of backwards compatibility for its own reasons. New people opening the format for modern (and future) use can do the same, but based on user requirements, not emphasis on product lines if that's not a real requirement.
So what's needed is just converters that use this code to convert to real open formats that can be maintained into the future. Not moving this code itself into apps for the rest of all time. Today we have a transition point before us which lets us finally turn our back on the old, closed formats with all their code complexity. We can write converters that can be used to get rid of those formats that benefited Microsoft more than anyone else. Convert them into XML. Then, after a while, instead of opening any Word or Excel formats, we'll be exchanging just XML, and occasionally reaching for the converter when an old file has to be used currently. MS will go with that flow, because that's what customers will pay for. Soon enough these old formats will be rare, and the converters will be rare, too.
Just don't perpetuate them, and Microsoft's selfish interests, by just embedding them into apps as "native" formats. Make them import by calling a module that can also just batch convert old files. We don't need this creepy old man following us around anymore.
--
make install -not war
When Excel started importing 1-2-3 documents, the right way to do that would be to create an importer to your own native format. Not to munge a new slightly different format into your existing structures. Yes, you'd have had to convert some dates between 1900 and 1904 formats (and maybe, detect cases where the old 1-2-3 bug could have affected the result) but at least you wouldn't be trying to maintain two formats for the rest of time.
If this is an example of programmers throughout history always doing exactly the right thing, I'd hate to see an example of code where the original author regretted some mistakes that had been made.
Joel is usually spot on, but the advice he gave in the article is actually pretty terrible if you are going to have to generate any volume of Excel reports. Automating Excel is slow and unwieldy, and should not be hooked up to a server. You will be limited to a few workbook generation requests per second, and if you need to handle more, buying another Windows/Office license and load balancing is pretty awful. The only way that this might be workable is to set up a process that sits in the background with a "pool" of automated excel instances launched and waiting for work, so that when there is a high volume of requests, they get forwarded to different instances. Still not very scalable.
There are companies out there that have reverse engineered the file format (the one I have experience with is SoftArtisan ExcelWriter, which is buggy), but overall there will be no clean, scalable solution for this until Excel 2007/the Excel 2003 compatibility pack are more prevalent you can just generate the XML to represent the workbook.
It's interesting you give a nicely egotistical critique of a well-regarded expert's article, but don't suggest a single alternative to how M$ could have met their design goals, nor explain why the no-interoperability assumption was unreasonable at the time. If you can't appreciate the design goals, nor suggest a way to meet them, what's the point of the rest of your post?
Then what's with that 2Gb limit ? Or what's with the decision to use such formats for mail-storage and databases ?
Religion is what happens when nature strikes and groupthink goes wrong.
Wait, I thought you were trying to convince us that this doesn't reflect bad programming... Wholly out of context, Batman! They made a design decision to ignore interoperability and optimized towards small memory space. What part of that is hard to understand? You think everything should be designed up front for interoperability, regardless of context? In the mid to late 80s, there just wasn't a huge desire for this feature, as Joel states. but then again I was 15 years old (in 1995) and still learning C. Ah, now your post makes sense. You completely lack perspective. The Word/Excel doc formats were around 10 years before you. You lack the knowledge about why dumping C data structures directly to disk was necessary--even though Joel spells it out. You don't understand what OLE truly solved (not just embedding spreadsheets inside of word, by the way). And most importantly, you seem to lack the ability to understand design trade-offs.
That is almost the the stupidest thing I've read today (RTFA with respect to development costs to figure out why), except for this:
We can ignore the shockingly poor logic inherent to this statement and just take it at face value: doing something just because M$ wants you to would easily make the Top 10 Stupid Things To Do In IT list. It's particularly bizarre to hear it on Slashdot.
1. You have a web-based application that's needs to output existing Word files in PDF format. Here's how I would implement that: a few lines of Word VBA code loads a file and saves it as a PDF using the built in PDF exporter in Word 2007. You can call this code directly, even from ASP or ASP.NET code running under IIS. It'll work. The first time you launch Word it'll take a few seconds. The second time, Word will be kept in memory by the COM subsystem for a few minutes in case you need it again. It's fast enough for a reasonable web-based application.
2. Same as above, but your web hosting environment is Linux. Buy one Windows 2003 server, install a fully licensed copy of Word on it, and build a little web service that does the work. Half a day of work with C# and ASP.NET. So if you are on a Linux system, you are screwed . I think this article is written by some M$ fanboy. Nothing wrong here. But saying that Linux user should just dump their software, and go for Microsoft stuff , just because It's very helpful of Microsoft to release the file formats for Microsoft and Office, but it's not really going to make it any easier to import or save to the Office file formats. I think it's wrong wrong wrong.
You're kidding right? That's been exactly Microsoft's marketing strategy for the last ten years. Remember the Win9X BSOD ads for Windows XP? Microsoft is in the difficult position where their only real competition is their own previous products.
Support Right To Repair Legislation.
"Apps Hungarian", which adds semantic meaning (dx = width, rwAcross = across coord relative to window, usFoo = unsafe foo, etc) to the variable, not typing, is what is good and what he is advocating.
What is the justification for putting that semantic meaning into a variable name, instead of incorporating it into class definitions?
For example, if a string can be "safe" or "unsafe", why not have "SafeString" and "UnsafeString" classes that extend String, and use instances of those, instead of having instances of the base String class names 'sFoo' and 'usFoo'?
Actually, when possible, you should do both. Hungarian notation is a grammar. In the same way that English has rules for writing which include capitalizing the first letter of a sentence, proper names, and so on, Hungarian notation provides visual cues to programmers that make certain types of semantic errors "sTanD oUt." There's nothing particularly unusual about the text "sTanD oUt," and it's meaning does not change by writing it that way, but it violates the English grammar and your brain's pattern recognition identifies it as an outlier. So too with Hungarian notation. Code that does not use at least some form of Hungarian notation looks devoid of the meta content I expect my follow programmers to provide, namely what decision they've made, and whether the code conforms to those decisions. To someone accustomed to Hungarian notation, finding "double fValue;" or "if (uCount < 0)" in the code prompts the eye to linger, the brain to reparse. Ultimately, many conceptual errors are identified and resolved this way, even if the compiler fails to catch them.
Also, like any grammar, the rules depend on the circumstance and should be followed in order to resolve an existing problem or ambiguity. Fully qualifying a variable name "caiIndex" to imply "constant array index" is silly. That is cargo cult mentality. Any of the following would be fine according to the guidelines at my company and each reflects a different decision by the coder: "int nIndex;" "unsigned int uIndex;" "index_t index;". The first works best if the index will be used backwards and the loop constraint is that the index is positive. The second works best if the index is random access, so that functions that use it can check the range with one comparison rather than two. The last case indicates that the semantics and nature of the index could be dependent on a variety of factors including processor architecture, and care should be taken. Therefore, the code "--nIndex," "++uIndex," and "next_index(&index)" look correct while "for (uIndex = 4; uIndex >=0; --uIndex)" looks very bad, and "++index" should make one immediately recognize that any of the following are possible: 1) the ++operator has been overridden, 2) index_t is typecast to an integer type, or 3) this won't compile as would be case if index_t was a struct.
And so, after 28 years of programming, dealing with all different styles of C and C++, I've come to recognize that understanding and using Hungarian notation correctly is a skill. Your productivity increases as you use it, eventually you don't even notice it, and the benefits come later, particularly when refactoring, or making changes to older code, especially if written by someone else. Like syntax highlighting for your brain, if you use it long enough, you'll know when there's an error in the code without having to compile it because it will look wrong. Supposedly for lisp programmers, the same epiphany comes when you no longer see the parentheses.
Happy Programming,
-Hope
And I think your ability to assess another's work is flawed courtesy of an over sized ego. That was my point.
You have yet to provide an alternative solution to the problem. Given that one constraint is memory, your inability to be concise suggests you're not capable of coming up with one either. Certainly your "squeeze out a few extra microseconds" comment suggests you have absolutely no clue what you are talking about. Yet you persist in calling it bad design. You are strangely smug about what was quite possibly an implicit assumption forced by tough constraints, with no actual interoperability requirements, at a time when they were rarely offered let alone expected. I would stop using "IMHO" - clearly there is nothing humble about your opinion.
Why the bit about metadata, out of interest? It's as if you think the more irrelevant things you can fit into the post, the more we're supposed to be impressed.
Before jumping up and down gleefully, those working on related open source efforts, such as OpenOffice, might want to take a very close look at Microsoft's Open Specification Promise to see if it seems to cover those working on GPL software; some believe it doesn't.
From MS's own mouth - and mind you that these quotes probably had to be vetted by a billion lawyer-types to ensure that MS wouldn't incur any sort of bizarre liability fifty years down the road by saying them. Based on what is said here, the only other thing that MS reserved is the ability to sue anyone who sues them for violating the patents that they already own, and are releasing to the public. That would be kind of like placing a legal disclaimer on your Halloween candy bowl: "Attention: You can all take as much candy from this bowl as you want, and I legally give up my right to prosecute anyone taking candy from this bowl of Theft, forever. But if any of you accuses me of Theft for eating candy from *my own candy bowl,* then I reserve the right to accuse that person (and *only* that person) of Theft, too." Here's a few pertinent excerpts:
Q: Is the Open Specification Promise intended to apply to open source developers and users of open source developed software?
A: Yes. The OSP applies directly to all persons or entities that make, use, sell, offer for sale, imports and/or distributes an implementation of a Covered Specification. It is intended to enable open source implementations, and in fact several parties in the open source community have specifically stated that the OSP meets their needs. Moreover there are already a significant number of implementations of Covered Specifications that have been created and/or distributed under a variety of open source licenses as well as under proprietary software development models. Because open source software licenses can vary you may want to consult with your legal counsel to understand your particular legal environment.
Q: Is this Promise consistent with open source licensing, namely the GPL? And can anyone implement the specification(s) without any concerns about Microsoft patents?
A: The Open Specification Promise is a simple and clear way to assure that the broadest audience of developers and customers working with commercial or open source software can implement the covered specification(s). We leave it to those implementing these technologies to understand the legal environments in which they operate. This includes people operating in a GPL environment. Because the General Public License (GPL) is not universally interpreted the same way by everyone, we can't give anyone a legal opinion about how our language relates to the GPL or other OSS licenses, but based on feedback from the open source community we believe that a broad audience of developers can implement the specification(s).
Q: I am a developer/distributor/user of software that is licensed under the GPL, does the Open Specification Promise apply to me?
A: Absolutely, yes. The OSP applies to developers, distributors, and users of Covered Implementations without regard to the development model that created such implementations, or the type of copyright licenses under which they are distributed, or the business model of distributors/implementers. The OSP provides the assurance that Microsoft will not assert its Necessary Claims against anyone who make, use, sell, offer for sale, import, or distribute any Covered Implementation under any type of development or distribution model, including the GPL. As stated in the OSP, the only time Microsoft can withdraw its promise against a specific person or company for a specific Covered Specif