Massachusetts Adopting 'Open Format' Software

XopherMV writes "A Massachusetts state senator who had complained about the state government's effort to promote open-source software at the expense of proprietary software has hailed the state's effort to reach a compromise over future software purchases by the state. The latest iteration of the state's policy emphasizes 'Open Formats' such as TXT, RTF, HTM, PDF, and XML." And if file formats for state use must be in truly open and free formats, then it matters much less what OS or application is used to create or open them. (On the other hand, XML and other TLAs don't always mean free or open formats.)

Comments

True, but...

[professorhojo]

>On the other hand, XML and other TLAs don't
>always mean free or open formats.

This is true, but XML documents themselves are also considerably more open than their binary counterparts. Anyone can parse a well-formed XML document, and validate it if a DTD is provided. While companies may still create XML that behaves in a specific way bound to their application, the data in the XML document is available to any application. While developers could create obfuscated DTDs or encrypt their data in a proprietary manner, they would lose most of the benefits of using XML. XML doesn't bar the creation of proprietary formats, but its openness is one of its greatest advantages.

Re:True, but...

[SilentChris]

"Namely, that they could create obfuscated DTDs and encrypt their data in a proprietary manner while still using it, thus convincing the masses that they're using an open format while not actually using one."

But they won't. They can't. Microsoft has a history of sticking with the original file format they created along with 1.0 of the application. Today's Word docs have a lot "tacked on", but they still have the basic structure openable by the original Word.

WordML (Microsoft's XML structure for Word docs) is fairly clear-cut. They can "obfuscate", but they won't, because people'll will want those original files openable in 10-15 years. Backwards compatibility is a huge goal at MS.

Re:True, but...

[fireboy1919]

Today's Word docs have a lot "tacked on", but they still have the basic structure openable by the original Word.

Word documents are not backward compatible, except in a few lucky cases, despite the fact that most of the functionality is the same. Have you even tried this? Word XP documents don't work in Word 97; 97 don't work in 95, and I would assume it goes back even farther.

I think a better claim would be "backwards compatibilty is a huge thing to avoid at MS" considering that almost no new functionality has been added to word in the past 10 years and yet the document format has changed.

Re:True, but...

[10am-bedtime]

you can pour a gallon of perfume on shit, but it is still shit.

if you accept "open on the outside but proprietary on the inside" you do not understand what it takes to be truly open. you have lost to the marketeers and spin artists, and given up your only true possession -- your mind and its ability to think critically.

Ploy On Price Negotiation?

[Manip]

This is not the first time we at /. have seen states and countries go this route but they almost always end up back with Microsoft but with a discount on their licence.

I don't know about you guys but I won't believe it until I see office workers using it, before then it is just a negotiation ploy to save some money with Microsoft (Why else announce it early?)..

PDF

[DeepDarkSky]

Adobe certain has done its job in making PDF so common place that it's become an "open" format, hasn't it?
I think that for specific purposes proprietary formats are ok, but for interchanging and for storage purposes, the open formats are important.

Great but

[BlueYoshi]

I think it's good because it will permit people/company to interact and be able to exchange document without to be forced to use some particular software.

But I think if you take XML, we need to have some effort to produce some standards DTD or XML-schema to be sure to have a real interoperability

Also it will be neccesary to have some kind of validator for each format to force that everybody is using the real standard and not some fancy extension that could ruin the all idea

if they are serious about it, that's enough

[idlake]

If they are serious about enforcing open document formats, that's good: open source can compete and win if formats are open. The big concern is that companies like Microsoft will try to portray their proprietary formats as "open". For example, the DOC format has been documented by Microsoft, but it isn't truly open because it keeps changing and because it is under Microsoft's control. In particular, XML is not an open format--it isn't a format at all; XML is a standard in which people can define formats, both open and proprietary.

A format isn't open until it has actually been standardized by an independent body that can guarantee that it is free from patent or other claims, and until it has been demonstrated that it can be implemented independtly by actually doing so.

This is bad news, not good news

[OblongPlatypus]

Most commenters seem to be missing the fact that this news is unequivocally bad. There were efforts to adopt open source instead of closed source software, but this senator (probably sponsored by Microsoft) managed to talk them into focusing on open formats instead. This coincides nicely with Microsoft's new XML formats for their office products, and lets Massachusetts continue using Microsoft products while paying lip-service to the fans of "open" solutions.

Re:This is bad news, not good news

[OblongPlatypus]

To clarify: I'm not saying open formats are bad. But in this case we went from an administration pushing open source products (which incidentally automatically means open formats), to one that's going to stick with Microsoft products. They wanted to change what software they were using, but were deflected into some vacuous statement about "open formats", and in reality no change at all.

XML does not autmoatically mean "open format", at least not in the way you seem to be thinking. Even if everyone in the Massachusetts administration starts using exclusively WordML for their documents (including converting all old documents), any open-source product would still have the problem of relying on a format defined by the very same monopoly they're trying to compete with. WordML is patent-protected specifically to prevent the equivalent of a "fork" of the format, so anyone using it is completely at the mercy of Microsoft's whim on where to take the format in the future.

Re:This is bad news, not good news

[Tim C]

You couldn't be more wrong if you tried.

One of the main complaints about closed source software is that the proprietary, closed file formats keep you locked in to using the product, and that changes to said file formats tend to push you to upgrading because everyone else has, so you have to to be able to read their documents.

Opening the file formats removes this restriction - now *anyone* can write software to create and edit them flawlessly. You're no longer tied to a single editor. How many .doc editors are there, compared to .html or .xml?

Mandating open source software, while appearing good, would be a bad thing. Software should be used based on fitness for purpose; if the open source solution is superior, then use it. But don't use an inferior open source app just because it's open source if a superior closed source one is available and affordable. Mandating open file formats increases the likelihood that an appropriate open source solution will become available.

Re: This is bad news, not good news

[tyen]

Policy, Not Mechanism.

They are very close, but need some additions to nail this right. Everyone freaking out over XML being cited should read the article. Reading the original article, I note that they defined "open format" by policy and not mechanism:

specifications for data file formats that are based on an underlying open standard, developed by an open community, and affirmed by a standards body; or, de facto format standards controlled by other entities that are fully documented and available for public use under perpetual, royalty-free, and nondiscriminatory terms.

This means they really don't care about the actual format, they care about the terms of access to the format. Microsoft can't drive a DTD with encrypted blocks through a mechanism-based loophole simply by declaring, "Hey! Look! XML!".

However.

It is said that even the largest companies bear the imprint of their founders. Gates was raised by lawyers, and his company operates like one. Unless you adversarially test this legislation before it passes, I guarantee you Microsoft will find a perfectly legal way to protect their crown jewels if it passes. There are other big players who will fight tooth and nail against this legislation, too. Oracle. IBM's DB2 folks.

It is unfortunate that I could not find on their web site a full explanation of what they meant by "open format". However, going by that small excerpted blurb, if I was thinking of legal and marketing workarounds, here are some things I can come up with off the cuff.

1. Dilute or pervert one of the definitions of "open standard", "open community", or "standards body". No definition legislated, easy enough to do. Control the standards group, control the standard.
2. Note that the clauses separated by a semicolon (;) stipulate access terms for the latter but not the former. Sure, place it with a standards group, but make it expensive to obtain the standard "to cover distribution costs". The EIA standard for racks for example, costs over $50 to obtain an electronic copy. Perfectly open, perfectly standard, but certainly not "royalty-free".
3. Play the Internet Explorer bundle game again, on a different playing field. Make the default format of the application a proprietary format, and allow saving as the standards format as long as the user takes additional steps to configure it or specify the standard format. By default, the vast majority of users will deploy with the default setting, killing any standard format in the crib through sheer inertia.
4. Sure, there is an open format. It just doesn't support all the features of the application.
5. Twist the definition of "fully documented" because that term is not nailed down. Yup, it's fully documented. "The 'dynamic_index' field stores dynamic indexes". There. It's fully documented. What? You want to know what a dynamic index is? Oh, but that's a trade secret. Or here is the full format of the dynamic index data structure, "fully documented". Leave out enough adequate description of the semantics, and you can bamboozle nearly everyone, including yourself. Why do you think Microsoft themselves can't get their own Word format consistent across versions? You can take the Microsoft-is-Evil theory that they do this to "entice" their customers to upgrade, but I tend to think it is because the format is ambiguously documented enough that even their own smart programmers trip up on the specifications.
6. Supply an open standard, but your implementation of the standard is different from the outside world's implementation(s). Hey, bugs happen. No reference implementation that everyone standardizes upon, not a problem to be just barely incompatible enough (without any need for evil conspiracies) to annoy users enough to make them stick with the original application. Coders who hack EDI systems can sympathize with me here; even when everyone agrees upon an implementation standard, a "data format dissonance" tend

Re:This is bad news, not good news

[Anonymous Coward]

E.g., MS Office is the best thing out there, nothing comes closer in terms of productivity.

Precisely how is is better than OpenOffice.org. Or even cheaper proprietary choices--Corel, Lotus, etc.?

I need support too. At least with a bit of software sold by some company, in most cases I have an 800 number that I can call and get help with.

You can buy such support for F/OSS too. I've worked with SuSE, RedHat, and MS. Guess which had the longest hold time & which gave the least satisfactory result. Hint: it is the one that cost the most too.

With OSS, the best I can do is send an email and wait a while (maybe get ignored, maybe get told to RTFM even though I have, maybe ge told to write a patch, maybe even get something useful) or pay lots of money for a support contract.

I'm sorry you've had negative experiences. I've found the F/OSS community to be VERY helpful for the most part. I have had some of the same negative experiences, but then that is also true of asking strangers to support MS or other proprietary solutions. The cost of a support contract is often still less than with a proprietary provider.

Mind you, the former case really doesn't work for Suzie Secretary. Here again, closed source almost always wins.

Suzie Secretary relies on your company's IT for support. So the winner is what your IT guys can cheaply and quickly support. If the problem is with Suzie, perhaps Suzie should be replaced. If she couldn't use the commercial product, you wouldn't even think twice about doing this. This is a bit mean, but the Suzie Secretaries _I_ have supported can use F/OSS AT LEAST as well as the commercial products.

Re:This is bad news, not good news

[Renegrade]

If the format is open, then an open sourced solution could be developed to deal with any issues with the proprietary software.

If the file formats used are truly open (as in a decent standard that's well documented and actually works as it's described, which allows other applications to read and write in the same format without legal encumbrance), then the customer can take their data and have a new application or data converter written, allowing them to easily migrate to a new platform.

That addresses the following issues thusly:

>Other valid complaints are the per-seat costs, upgrade costs

These are limited by the fact that the customer can now walk away. Something they cannot easily do if they were locked into Word's *.doc format or such. If the proprietary vendor tried strong-arm tactics or charged too much, the customer could simply write, or have written, a replacement application (or a file converter to convert the data files to a format used by another application).

This effectively caps the maximum cost that the vendor can impose. Company X cannot charge a billion dollars for a given product if Company Y can offer the same thing for $59.95 that works on the same data, or if Open Source Programming Group Z offers a totally free version of said program.

Also, the bigger the vendor, and the more pervasive their product, the more likely some open source group is to pick up the challenge of making an open sourced solution which works with these same data files.

> limited effectiveness of outside support,

Well, only with regards to the application itself. The data, properly documented and open, could easily be handled by any competent programmer.

> architecture lock-in

What architecture lock in? Vendor screwing you over? Take your data and walk away. Have someone else write a replacement. Write it yourself! Use Y or Z's product! Suddenly Corp X decides that the 3.2 update will NOT cost a trillion dollars, afterall.

>and a slow, costly route to get bug-fixes and new features implemented.

Bug fixes to the original application would depend on the vendor. New features, however, could be implemented by utilties that work with the data created by the application. And if the original application was too buggy to work with at all, why was it purchased to begin with?

Open sourced solutions can be costly to bugfix as well. If your package is abandoned, you'd have to _hire_ people to fix it.

> And, of course, the threat of being left out-in-the-cold if the company stops offering the proprietary program (or if said company collapses).

How do you figure this? Take the data and _WALK AWAY_. This would be no different if an open sourced group collapsed or grew bored of a package.

---

Basically, having open file formats is like a foot in the door for open source software: If free/open source software can work with the same data, and be better and cheaper, then why wouldn't some conversion happen? It also takes away one of the tools in the arsenal of big corporate evils: the hostage-holding of your user-created data.

This could also revitalize a rather stagnant commercial software industry by forcing more interoperability.

What, can't open sourced software compete then? Is it like some big, free version of Microsoft? Forced to rely on legislation to get it's place in the world? I'd hope not!

I'd say that it's probably better to allow for free and proprietary to compete openly, than to have one side legislated over the other. It could easily go the other way, you know. "All government agencies must use Microsoft(TM)(R)(C)(Patent Pending)(TM)(C)(R) software AT ANY COST!".

How could this not be win-win?

This of course, all hinges on the definition of an 'open' format:

1. It must be readable and writable by other applications or utilities withou

Re:This is bad news, not good news

[Noksagt]

If the format is open, then an open sourced solution could be developed to deal with any issues with the proprietary software.

And, rather than throwing good money after bad to upgrade and maintain closed source software, public agencies should fund efforts to bring free open source software to the public. Perhaps they should keep proprietary software that has already been paid for, but as a taxpayer I want the most bang for my buck. This would be a strong argument against any commercial product that could charge a lot of money to MA & then to IL & so on, when a Free (in both senses of the word) program could be used instead.

If the file formats used are truly open (as in a decent standard that's well documented and actually works as it's described, which allows other applications to read and write in the same format without legal encumbrance), then the customer can take their data and have a new application or data converter written, allowing them to easily migrate to a new platform.

So why not do it now? The argument against doing it later will be "we've already invested so much money and time in closed product X from company Y, that it would cost an unreasonable amount to build F/OSS program Z from scratch."

These are limited by the fact that the customer can now walk away. Something they cannot easily do if they were locked into Word's *.doc format or such. If the proprietary vendor tried strong-arm tactics or charged too much, the customer could simply write, or have written, a replacement application (or a file converter to convert the data files to a format used by another application).

This is short-sighted. First of all, the vendors can charge much more than an upgrade is worth--they just have to keep it cheaper than a replacement. Furthermore, the "annual upgrade tax" to the company will add up quicker than F/OSS over the lifetime the files need to be read.

Well, only with regards to the application itself. The data, properly documented and open, could easily be handled by any competent programmer.

And how often do you suppose problems are with the file format? Support is almost always inherently for the particular product.

What architecture lock in? Vendor screwing you over? Take your data and walk away. Have someone else write a replacement. Write it yourself! Use Y or Z's product! Suddenly Corp X decides that the 3.2 update will NOT cost a trillion dollars, afterall.

Why should the government feed Corp X if this is a for-seeable problem?

Bug fixes to the original application would depend on the vendor. New features, however, could be implemented by utilties that work with the data created by the application.

So you have a dozen applications to do what you thought the one you first bought should have done? End users will love that.

And if the original application was too buggy to work with at all, why was it purchased to begin with?

The bugs might not have been discovered. What if a gaping security hole is discovered in your closed source application after you bought it? It also doesn't have to be "too buggy to work at all." Any discovered bugs should be fixed if possible. A patch to a minor bug can prevent major headaches.

Open sourced solutions can be costly to bugfix as well. If your package is abandoned, you'd have to _hire_ people to fix it.

I agree that specific applications can be more expensive, but with open source you'd ALWAYS have the option of hiring someone to fix it. Not so with closed source--the vendor has a monopoly on the ability to fix things. Good luck trying to negotiate with them by saying you'll hire someone else who is cheaper.

Basically, having open file formats is like a foot in the door for open source software: If free/open source software can work with the same d

Um

[DOS-5]

Since when do plain text files count as an "open format"? Is it just because someone hasn't tried to patent it yet? (probably) Just seems a bit weird to me.

A strong position

[ContractualObligatio]

There's a lot of (good) commentary on the detail of what is and isn't an open format. And it would be good to get the detail right, because there are many ways to abuse the phrase "open format" and there are companies that will take advantage of them.

Nonetheless, requiring the use of open formats is a strong, defendable position in practise. like it or not, mandating the use of open source isn't possible, or at least highly unlikely. The reason for this is that open source might be good but it's not *necessary*. Not in the short term, and never in the minds of people with votes and money for lobbyists. A lot of good things have been done with proprietary systems (I'm an Apple fan) with a lot of openness. Open source is therefore a difficult argument to win in terms of *requiring* its use. Again, as a Mac fan, I wouldn't agree to it myself as a *requirement*.

Open formats (*real* open formates) produce a level playing field. Open source could win its argument in a fair fight in public, not a dogmatic argument conducted in courts between various zealots on both sides that many people fight it hard to really bother with.

I'm not interested in forcing organisations to drop Microsoft. Dictatorial approaches to solving problems never appeal to me. I'd be far happier with a situation where by legal documents and government documents (as some important examples)must be in an open format so that full featured editors could be available on a number of platforms e.g. Windows, Mac OS X, Debian, Red Hat, Suse, Solaris, HP-UX, AIX.

That list is deliberately composed primarily of commercial companies because a) that's the way the world still works - make use of it to your own advantage, and b) to get all those platforms (including a good open source distro) sharing some new equivalents of .doc, .xls and .ppt formats would really open up the desktop marketplace.

I think this is the correct way to go

[fitten]

I don't think that government should mandate open source vs. closed source code purchases. This is unfair. The government should not mandate against valid, legal business models.

I think the government should mandate that output from any software be an open standard format (XML or whatever) and then they choose, based on a competative bid process like they are supposed to do, the software that will do what they want (which may include adding features at some point). If some OSS group wins, so be it. If some proprietary group wins, so be it.

Allowing only OSS is both wrong and bad, IMO, for a number of reasons.
1. It is straight against capitalist economy to require one business/development model. In capitalism, you specify the product and whoever can do it best/cheapest/easiest wins. Only an OSS zealot would think that OSS would always win.
2. The government should not dictate the "right" business model for people to follow. As long as they are legal under the laws (both criminal and financial) of the country, they are valid. The government should not dictate that some valid models are not valid for the government.

TXT? HTM?

[Ed Avis]

Since when were 'TXT' and 'HTM' the names of document formats?

Please, this isn't MS-DOS, and even if it were there's no need to resort to such barbarisms. You mean plain text, and HTML.