Microsoft's OOXML Formulas Could Be Dangerous

hill101 writes "According to Rob Weir's blog, Microsoft's 325-page OOXML specification for spreadsheet formulas is deeply flawed. From basic trigonometric functions that forget to specify units, to statistical functions, to critical financial functions — the specification does not contain correct formulas that could possibly be implemented in an interoperable way. Quoting Mr. Weir: 'It has incorrect formulas that, if implemented according to the standard, may cause loss of life, property, and capital... Shame on all those who praised and continue to praise the OOXML formula specification without actually reading it.'"

Comments

Proof that open formats are a good idea?

[moosesocks]

I don't know about you, but I view this as being a very GOOD thing.

Because the format was an "open" standard, the serious flaws present in the format were quickly and correctly identified by third parties outside of Microsoft.

If it had been a trade secret, it could have been bundled into a product, and assumed to be reliable by its users. Instead, it's been exposed for what it is.

If anything, this proves that open formats are a good idea.

Re:Proof that open formats are a good idea?

[Rakshasa Taisab]

MS pretty much seems to have cut-n-pasted their MSOffice help files and decided to call that a 'standard'. Only thing good about it, is that it will make ISO be so much less willing to ratify their standard. If you look at their CEILING definition, as linked in the article's comments, it is so unprofessionally written you'd wonder at the size of EMCA's checks.

Re:Proof that open formats are a good idea?

[jkrise]

If it had been a trade secret, it could have been bundled into a product, and assumed to be reliable by its users. Instead, it's been exposed for what it is. Exactly! Imagine a Hospital implementing OOXML for it's entire IT needs, and a prescription reads: 1 tbsp Terramycin, twice daily. If a patient sues the hospital for wrong dosage, lots of red faces will be guaranteed.

FTA:

The CONVERT function (Part 4, Section 3.17.7.48) converts from one unit to another. Some conversions explicitly allowed include liquid measure conversions such as from liters to cups or tablespoons. But whose cup and whose tablespoon? Traditional liquid measures vary from country to country. In the US, a cup is 8oz, except for FDA labeling purposes when a cup is 240ml. But in Australia a cup is 250ml and in the UK it is 285ml. Similarly a tablespoon has various definitions. OOXML is silent on what assumptions an application should make. I guess I won't be using OOXML to store my recipes, and certainly not to calculate medical doses!

Re:Proof that open formats are a good idea?

[mikeb]

"You could find similar problems in virtually all specs" Well I would like to see your evidence of that! Having worked on the the original C standard extensively and done a fair bit of work on the C++ standard, I find it rather annoying that an unsubstantiated statement like this is trotted out.

In the standards committee it was typical to find 50 people in a room reading *each* *single* *word* of the draft standard and arguing for hours over a single line - 8 hours a day for five days at a stretch. Immense attention to detail was spent on considering every possible interpretation of the words and wrangling over the best and most precise, unambiguous way to define what the standard was supposed to mean. The fact that the original C standard passed through almost unmodified (though slightly extended) in its later version is testament to all that work. Typically the people who work on standards committees put in vast amounts of effort to avoid precisely the lameness that TFA's article refers to. Seriously - not to specify whether SIN uses degrees or radians is inconceivable for an ISO standard.

Bill Plauger in particular did Trojan amounts of work on the C libraries to avoid dumb mistakes of those kinds.

Re:Proof that open formats are a good idea?

[bjourne]

Well I would like to see your evidence of that! Having worked on the the original C standard extensively and done a fair bit of work on the C++ standard, I find it rather annoying that an unsubstantiated statement like this is trotted out.

If you have been involved in drafting the C standard then you should be aware of the list of defect reports. You should know that it is almost impossible to precisely specify every single detail that a normal working human would naturally assume.

The standard is far from complete in the sense that a literal and mechanical interpretation of it won't produce a working C compiler. You still need a human to fill in the gaps and to guess the trivially sane assumptions. Also remember that C standard is a much simpler thing to specify than the file format for an office suite. And yet it takes, as you say, 50 people in a room reading every single word.

And for the record, I work with implementing and testing JSR:s. They also contain goatse-sized holes, which is why the JCP requires reference implementations because specs never specify everything they need to specify.

Re:Proof that open formats are a good idea?

[mikeb]

At face value, the list of defect reports might be assumed by a naive reader to suggest that the C standard is full of holes and would metaphorically never float. In fact on rereading it, I think that in the main it supports my point :)

Any standard that's intended for human readership will suffer precisely because it's written for humans. Attempts to use formal specifications (perhaps denotational semantics or something like 'Z') haven't really caught the public imagination though it would have been interesting to try. I'm sure I remember that being discussed tangentially during one of the boozy degenerations of an ANSI meeting after a long day of wordsmithing circa 1985.

Maybe it was a typo that italicised the 'If' in 'If you have been involved' .. my track record in involvement in the ANSI standard for C is well known for those who care to look. For those who don't, ISBN 978-0201544336 and http://publications.gbdirect.co.uk/c_book/ may assist.

Sadly I'm also caught up in the mire of OOXML fast-track reviewing as a member of the British Standards Institute's panel. There are some very serious questions to be raised about just what can be done with a document that's so big. It probably took hundreds of staff-years of work in total to produce something as short as the C standard. Where does the effort come from to review and QA something so very much bigger?

Re:EULA?

[setagllib]

Precisely because it's so likely with Microsoft products. If they didn't disclaim it they'd be in serious trouble. Disclaiming doesn't make it a non-issue though.

And proof that single-ownership is bad

[Anonymous Coward]

Now that MS are the only ones who are allowed to change the standard, we must

a) wait until MS change the standard
b) then progress it through the "approvals" procedure
c) find out again if there are any problems (and go back to a)
d) implement these changes

And when it comes to WordSpacingLikeWord95 or whatever, how has this being "open" helped? People have asked what it means and been told nothing useful.

Oh, and doesn't this show that if MS had opened up the standard for perusal BEFORE filing it (like ODF did), wouldn't we have avoided this problem?

Surely we all saw this coming

[bersl2]

While I think that the "loss of life, etc." part is a bit overboard, since nobody builds a mission-critical system on top of Excel (or do they...), I do think that the criticism is appropriate.

Anybody keeping a comprehensive and up-to-date list (or list of lists) of specific things that are wrong with OOXML? I see a bunch of scattered ones here and there. Of course, I've also wished there were a comprehensive list of specific "bad" things that MS has done; it would make demonstration of their unscrupulousness that much easier.

Re:Yeah, I'm sure this guy is objective

[topham]

Parroting the party line is promoting the fact that it has formulas as showing it is superior to ODF when the formula specification is next to useless because it wasn't reviewed properly.
If you read the article it isn't a cople of minor mistakes which can be corrected; it's a number of mistakes which have already made it past a review stage.

Implied warranty - fit for the purpose

[QuestorTapes]

> ...in general:
>
> * We trust all hand tools like wrenches and sockets to be exactly the size on the label
> * We trust all of our doctor's opinions whether or not a second opinion is recommended
> * We trust our math applications to do math properly
> * We trust our spell checkers to check properly
>
> In general, we trust the things we by to work as expected... as advertised.

http://www.oandp.com/edge/issues/articles/2006-08_ 06.asp

http://www.brajeshwar.com/finance/insurance/Liabil ity-Insurance.html

These links refer to the concept you're talking about. The second refers to the UK Consumer Protection Act, but the concept is general and fairly well accepted. From the first link:

"...any product that is sold comes with an implied warranty of merchantability and fitness for a particular purpose; and, just by selling a product, a seller is implicitly promising that: (1) the product is merchantable, i.e., fit for the ordinary purposes for which such products are to be used, provided that the seller is in the business of selling products of that kind; and (2) the product is fit for a particular purpose, provided that the seller, at the time of sale, knew the particular purpose for which the product was required, and the buyer relied upon the seller's skill or judgment in selecting a suitable product for that purpose."

This hasn't been successfully applied to software cases like this, but the issue hasn't be ruled out either. But it's hardly a stretch to expect that software such as a spreadsheet comes with an implied warranty that ordinary financial and statistical calculations are properly performed.

Microsoft can't code

[BillGatesLoveChild]

> From basic trigonometric functions that forget to specify units

Amazing. That's the sort of mistake you'd expect from a First Year Computer Science Major, but not from a Second Year. This isn't the first time Microsoft have done this. Even for the Windows API, the code trumped the documentation. The best way to find out what a feature did was to write test programs to poke at it. Heck. Until recently DirectX needed three pages of goobleydo-gook to start up. These people just don't get APIs, period.

In Microsoft Visual Studio when you press F1 Help it comes up with a list that includes "How to Write Good Code". Yes, by Microsoft. Even in the early hours of the morning, it gets a smirk if not a gufaw or a laugh. Microsoft are not good programmers. Haven't been for a long time. Anyone worth their salt will launch a Start Up, or at least join a company offering reasonable growth and prospects. Microsoft is like a Pyramid Scheme. The people that joined at the start did very well. As for the people that joined late... not a chance. Which makes you wonder about the ones that joined anyway. Read the Book "Microserfs".

> Ecma

Why didn't Ecma pick it up? These Standard Bodies are in-name only. When a "Member" wants to push something through, it gets pushed through. Then the Member's sales reps can go to the Government body and say "Look! We have an Ecma approved Standard" and t he Government worker ticks the "Uses Industry Standards" box on the tender.

One of the funnier "standards" was a simulation standard called HLA. It was approved before anyone had built a proof of concept. People bet their careers on it and the whole government was ordered to embrace it. The only problem: When they finally built it, it didn't work. *OUCH!*

Re:EULA?

[jeffasselin]

Well, here in Quebec at least, such provisions are illegal and software manufacturers can and have been held responsible for the reliability and functionality of their products.

This is to be expected...

[rsmoody]

After all, they did not BUY this from someone else. They came up with it on their own. We all know, Microsoft's best products were purchased from someone else. Excel for example.

Re:nCr mapped to AveDev?!!

[LiquidCoooled]

You think thats bad?

In the MSDN documentation for .NET Compact Framework Data Providers there is a small note:

Note This feature has been designed to be used in conjunction with a prerelease version of an anticipated successor to Microsoft Visual Studio .NET. Check the documentation accompanying the commercially released version for any updates.

Now it took a while to parse that and I decided that I might possibly in the future write some code for it but only if I cannot find a better IDE/ and dev system.

MS have really started to come apart at the seams.

M$ statement at Beuro of Indian standards

[anivararavind]

As a part of M$ ECMA fast track process ISO/IEC JTC1, Bureau of Indian Standards(BIS) a P-member of ISO has to vote on OOXML. The Discussions for this is going on now. You can see the Documents at http://www.odfalliance.in/OOXML.html

Here (Page 4, item 9b) M$ repeats again the same affirmation: "Ecma 376 contains full documentation for spreadsheet formulas"

F

ull documentation for Microsoft and ECMA = copy and paste of Excel formula Online Help

From Minutes of Meeting of BIS working group on wordprocessing ML held on May 07, 2007, with comments

Two beautiful points in MS response:

OOXML does not have a large number of features but is "feature rich".

"The statement was not that the size is due to the large number of features but "feature rich". The size of the document is also due to the fact that it is a fully defined specification." (note, OOXML is incomplete, inconsistent, and lacks semantic, ie, it is a street directory without a map)

6000+4000 pages of OOXML specifications are needed because MS couldn't be bothered to ask Oasis to define spreadsheet formula's

"As an example, which was cited in the meeting but has not been captured in the minutes, is the specification for implementing formulas in Spreadsheets which is not present in ODF. In this case what would spreadsheet formula specifications be considered as if not a point of standardization."

Re:now arriving at Dallas-Fort Worth...

[node 3]

Actually, doing a job to too high standards can kill people That's not what we're talking about. We're talking about malpractice insurance. If the standards are set to high, that's the problem with the standards, not the insurance itself.

I believe the rights of the accused trump any right of the alleged victim We're not talking about the accused. We're talking about the guilty. If you're found *guilty* of malpractice, you must pay, and there should be no artificial cap--you pay what you deserve to pay. Until you're found guilty, you most certainly deserve rights to protect you from malicious or unmerited persecution. It seems to me you are focusing on the wrong part. There's (seemingly) a flaw in the first half (the part *before* you're found guilty), yet you think that, due to this, you should apply the fix to the second part (the part *after* you're found guilty).

I am all for capping prison sentences for everything but murder and rape... saying that you can only be punished so far sounds like a very fair and sensible thing to do for all but the most extreme crimes. So do I. Unfortunately for you, that's not what I argued against. I argued against limiting damages (punishment) before even taking into account the nature and severity of the wrongdoing.

If people want protection, let them go to a doctor who is bonded. They can do this now. This is also the case with auto insurance. You can be bonded, or you can buy insurance. But mandatory insurance (or bond) makes us all safer.

Yeah... cause you know, places that aren't lawsuit crazy, like France, or Germany, or Canada, are just sooooo much more dangerous than the United States. Please! No, I said, "If it weren't for trial lawyers, the US would be a much more dangerous place to live." Are you saying France, Germany and Canada all do not have trial lawyers? Interesting. How do they settle civil disputes?

*ALL* cases go to trial. No they don't.

[about whether you'd pay for a trial lawyer's services] You are missing the point. No, that's *exactly* the point. You decry them as an evil, yet you'd not hesitate for one moment to avail yourself of their services.

If trial lawyers are so despicable, you should vow to never, EVER, use them. EVER. But you know you would. That either makes *YOU* evil, or them not. Which is it?

Additionally, if trial lawyers are so awful, doesn't that make trials awful, by association? I mean, the whole point of a trial lawyer is to argue a trial. This seems a fairly fundamental requirement for a free and civil society.

The vast majority of medical lawsuits involves cases where someone never suffered any significant loss of health, and the doctor wasn't responsible. First off, did you just make that up? Second, even if it's true, do these cases result in the plaintiff winning? Third, isn't that fraud? And again, this does not indict insurance, it indicts flaws in the rules.

Instead of fixing the legal system, you'd rather just do away with that portion of it altogether? What's this got to do with whether insurance itself is good or bad? Or trials are good or bad?

Re:MS Office approx. Reference Implementation

[gnasher719]

'' I agree with the general point that the spec. should be more carefully defined.
However, in practise does MS office not act as a reference implementation to clear
up ambiguities? ''

That's how Microsoft works. That's not how standards work.

The right way to handle this would be to take the whole thing away from Microsoft, who clearly doesn't have people who can do the job, and give it to people who have experience with standards, and let them create a workable standard. Then Microsoft can try to create an application that follows this standard, and they can try to translate old office documents to the standard.

On the other hand, you could save a lot of work by throwing away this whole nonsense, and let Microsoft use an existing, well-designed and carefully reviewed standard like the OpenDocument Standard.