Slashdot Mirror
MN Bill Would Require Use of Open Data Formats
Andy Updegrove writes "A bill has been introduced in Minnesota that would require all Executive branch agencies to 'use open standards in situations where the other requirements of a project do not make it technically impossible to do this.' The text of the bill is focused specifically on 'open data formats.' While the amendment does not refer to open source software, the definition of 'open standards' that it contains would be conducive to open source implementations of open standards. The fact that such a bill has been introduced is significant in a number of respects. First, the debate over open formats will now be ongoing in two U.S. states rather than one. Second, if the bill is successful, the Minnesota CIO will be required to enforce a law requiring the use of open formats, rather than be forced to justify his or her authority to do so. Third, the size of the market share that can be won (or lost) depending upon a vendor's compliance with open standards will increase. And finally, if two states successfully adopt and implement open data format policies, other states will be more inclined to follow."
It is not hard to reverse engineer a proprietary format - several word processors can save in competitor's formats. So this is a symbolic gesture, but it will be good in the future when Microsoft and others copyright their file formats to try and extort money from OpenOffice.Org and others.
Does anyone think it's significant that the two efforts are in states with "blue" histories (at least in election years)?
I keep seeing these stories about governments deciding to legislate the internal usage of ODF. Is the standard really ready for prime time? I can't say I know much about the details of ODF (if anything), but it seems like it's such a new standard that there are likely issues that would need to be worked out before it's so widely adopted.
Having said that, I think it's high time our public offices stop feeding monopolistic practices by continuing with document format requirements that more or less pre-determine software choice.
The Spoon
Updated 6/28/2011
We know a hell of a lot more about ODF than we do about .doc, and that hasn't slowed it down.
I am glad. Open Standards should be, well, the standard. If it passes there, and similar ones in other states, everyone will be able to read important gov't files without surrendering their freedom of choice, and the files will not become locked in an abandoned format. Hopefully the Federal Government will see the light.
it's nice to see the guys and gals in St. Paul talking about something other than that same-sex marriage ban. Now if the house, senate and governor could agree on a bonding bill thing might be looking up.
"You can see I know very little about pimp policy." George McGovern.
But this isn't about Open Source, it's about Open Standards, two orthogonal issues. Of course, Open Source is preferable, but it's not required to have Open Standards. Microsoft could add ODF support to its next version of MS Office (which they'll of course try to resist for as long as possible, as it'll kill their market lock-in), and it would be viable as a software supplier, but it'd have to compete on ease of use, price, robustness, etc.. It'd have to compete on its merits for once, instead of being the mandatory choice because of the current platform lock-in (even though OpenOffice.org does an excellent job interoperating with MS Office files).
I suffer from attention surplus disorder.
The real key to any 'open standard' is that it be implementable without payment of royalties or encumberments of any kind. That is what makes ODF or Ogg Vorbis 'open standards' and MS Office formats and MP3 non-open standards. Open standards are great for consumers and voters because it means they can buy which ever standard conforming product best suits their needs, and that encourages true competition. Vendors like Microsoft will off course complain loudly, but it isn't supposed to be about what is best for the vendors, it is supposed to be about what is best for the citizens, and Minnesota seems to understand that better than most. I expect that Microsoft will go in with all guns blazing to derail this proposal.
ASCII text is an "open data format".. and all word processors support that.
I am the maverick of Slashdot
The more important point is that governments deal in records that are meant to be more or less permanent, or at minimum long-lived. Proprietary formats exist at the whim of a single supplier -- a software company -- and those suppliers are subject to the whims of the market like any other company.
If I buy some paper from the Bienfang company and write a report on it, that report will still exist and be readable possibly for hundreds of years after Bienfang goes out of business. If Microsoft stops making a word processor or (god forbid) goes out of business, the situation may be different.
"So what," you say, "just reverse engineer it." But what if, in the intervening years, Microsoft has successfully lobbied for laws that make that a criminal offense? We're talking about future-proofing data here; whether it's implausible is not really the point. The point is that using a closed format introduced risk.
Another, more likely scenario: Microsoft subtly changes its format, or changes the way that newer versions of its software interprets the older format files. The government is forced to upgrade because Microsoft stops supporting the older version of the program, but the newer version does weird things to all those old records when it opens them.
There are various reasons to choose true open formats and standards beyond ideological ones.
Breakfast served all day!
Thousands and thousands of bills are introduced in legislatures around the US and worldwide it is probably a million or more. Far fewer pass and even fewer of those make it th erest of the way into becoming a law. (For the civics challenged/non-US readers: In most US states it will also need a sponsor in the other legislative body, passage in both bodies and either signing into law by the Governor or another vote to override a veto.)
It is good that such an idea is starting to bubble up, but it has yet to pass into law ANYWHERE at this point. The political power, wealth and proven tendency to resort to outright illegal measures of the Foe is going to make this a long difficult struggle.
Democrat delenda est
This is a lovely gesture. But ultimately I'm not sure it means a lot. This is a loophole the size of an 18-wheeler:
where the other requirements of a project do not make it technically impossible to do this
The thing is, 90% of government purchasing is steered by very, very careful tailoring of the claimed "requirements". The way government purchasing works, what this bill is very likely to do is just make it so all procurement requests are written up in such a way that it is "impossible" for a format to meet the "requirements" unless it can fully interoperate with Microsoft Word...
...it is the responsibility of a democratic government to be as open and transparent as possible, whenever possible. Saying that the tool chosen should be "the best tool for the job" must take openness into strong account when that tool is for use by the government.
"Destroy science and religion. Science would re-emerge exactly the same; but not religion." - Penn Jillette, paraphrased
Government assisted-funding software could again have a positive impact as MECC did. OOo, Mozilla, etc. SE Linux even...
This is a big bonus for more open standards, but what is not mentioned is the reality of landing the PO/contract for all of those potential Microsoft licenses will still strongly favor incumbents because they can come up with a triple-special licensing bundle for OS+Office Productivity that can't be beat.
Sadly enough, it puts the burden on WordPerfect to support ODF. Which isn't coming real soon. http://www.linux-watch.com/news/NS7758948461.html I think Wordperfect is the preferred word processor in the legal industry. They'd be wise to start now, but I have a feeling they won't.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
As for "de-facto standards" - even de-facto standards are supposed to be documented and open to competitors. Standards exist for more than one entity to adhere to. Otherwise it's a "de-facto monopoly".
I live in Minnesota, too.
So, you're saying you prefer the original, bloated office application that doesn't work properly... but costs a lot more?
--
Get out of my state, Red.
"...objectivity resides in recognizing your preferences, subjecting them to especially harsh scrutiny." -Gould
You're not being told that you cannot use Microsoft Office. The Government is also not being told it cannot use Microsoft Office. The subject here is how the _data_ will be _stored_. You can use Microsoft Office, but the files must be saved in an open format of some sort.
This is certainly a Good Thing.
Install COX in your backend today!
And finally, if two states successfully adopt and implement open data format policies, other states will be more inclined to follow.
This will be great, until, just as they're trying to do with food labelling standards, the federal government makes a law with a new standard that specifically invalidates any state laws that are more restrictive.
But I guess you can't blame the lobbyists... would you rather bribe..err..payoff..err, I mean, lobby a handful of people in one town, or have to spread your efforts across 50 towns and a lot more people?
Still, if enough states and countries do this
Engineering is the art of compromise.
I don't see why these bills even qualify as news.
There is nothing GNU about these.
If this signature is witty enough, maybe somebody will like me.
I'd love to see it become more than just Open "Formats"... it should also include Protocols. e.g. MS Exchange. Exchange essentially locking people into using MS Outlook.
Minn, home of Wellstone, Gopher, and MECC. I have to find out who'se sponsoring this bill and give them my vote. >>baci of duluth
Some people in the MN goverment think its a good idea to try to tack a Gay Marrage Ban on every bill. So I dont think it will pass.
Guys, you aren't looking at this correctly.
This is not about some OpenSource community effort versus Microsoft.
This is about a Sun/IBM alliance versus Microsoft. IBM and Sun are both quite capable in terms of political efforts. I'd put IBM way above Microsoft, for that matter. Sun's StarOffice for smaller organizations, IBM's Workplace for enterprise class, and OpenOffice.org to fill in various gaps.
Fortunately, this is not David versus Goliath. This is more like clash of the titans, and in terms of wooing government contracts my money is on IBM.
WhiteWolf666 an exBush supporter. All you new-school,compassionate,save the children Republicans can rot in hell
All this will accomplish is MS opening it's wallet, encouraging large "contributions" to varois politicians it never hear of before.
;-)
OTOH, might be a way to eventually bleed MS dry, eh?
(Might take 1000 years...must take the long view)
Lacking the time to actually look this up (college student with a ton todo for tomorrow)...what are the bill numbers so those of us who are MN residents can write to our reps. and voice our opinion?
Thanks
With the latest MS Office XML license, I don't think there's any chance royalties might be required. The only real remaining issue is that the license doesn't make much of a guarantee about future versions, but IIRC MS released a binding statement addressing this. I would have to go and hunt up the details now to be 100% sure. Just be aware that there's a LOT of outdated information and misconceptions about what they're doing (and they didn't help by ignoring many questions for long periods, giving roundabout answers, etc).
The only thing I'd really be worried about these days is who controls the format into the future.
The beautiful flip-side to this is a play on words. They will use open standards unless it's technically impossible to do, which really just means a lot of people will say "It's impossible to do function Y in format Z, I'm sticking with MSWord". It's not so much about impossibility, as it is about ignorance.
-Billco, Fnarg.com
...requirements.
I recall, back in the days when IBM was the enemy, being told by someone that his boss handed him a copy of the specs for a 360/370 clone and a copy of Principles of Operation and told to find some difference betweeen the two. I imagine that difference was written into a requirements specification so that only the IBM iron would satisfy it.
Being a Packer fan, I don't generally pay much attention to Minnesota. But if this passes, it might just be enough to get me to acknowledge that Minnesota is a state.
And to think the federal government thought a transfer of wealth to lawyers was necessary to combat Microsoft's monopoly. Good to see the M-states realize that they own their data, not their software vendor.
"Tempers are wearing thin. Let's just hope some robot doesn't kill everybody." --Bender
You can find out who represents you here. If they hear from a lot of people, they will get a theoretical warm fuzzy knowing that the issue is important to their constituency, and that they're doing the right thing. More importantly, they'll get a very down-to-earth political message, that if they don't vote for this thing it could be their ass in the elections coming up in November.
This post expresses my opinion, not that of my employer. And yes, IAAL.
Most gvt agencies are required by law to keep all the old records. Many of wich were created during the days of windows 3.1 and 3.2. For some odd reason documents created back then (this is all assuming they were made with microsoft office) can not be oppened with openoffice or any other present day software excluding microsoft office. Because the agencies will want to keep all their records uniform they will simply say that they prefer to keep using using microsoft office, and nothing will happen because nothing will ever force them to switch.
It is strange how our ideals affect information.
In the olden days, people concentrated quite heavily on open formats. Many programmers saw the data as separate from the program. In this environment, one would expect multiple programs to be dinking with the contents of a file. In such a world, maintaining and adhering to published open formats was the key to success.
One of the ideals of OO revolution was that object would own the data. Taken to extremes that means that one object should own the data through its entire existence. Early ideas on the problem of persistence was that OO would just save the internal state of the program to the disk, rather than going through the complex task of converting the data to an open format (risking the potential that other programs would be tempted to modify the data). It seems to me that OO ushered in proliferation of proprietary formats. It definitely provided an excuse for creating proprietary formats.
It seems to me that open data formats is contrary to the ideals of object oriented programming. However, when dealing with data that last longer than the computer, it seems naive that one object will be able to own the data.
The actions taken as a result of characters from 0 through 31 are specific to the implementation and circumstance. (An embedded motherboard is unlikely to have a bell, for example.) This means that although the syntax is defined, the semantics are not. The semantics are part of what makes a format "open". Thus, ASCII is only an Open Format between characters 32 and 127. All others may have proprietary interpretation.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
I'm a big proponent of Open Systems, and have been for years.
If you look at this solely in terms of word processing, yeah it makes sense.
But consider a database product like say Oracle. Since they don't document the file format, they're in violation of this bill. Yet it's relatively trivial to extract data from an Oracle database and use it elsewhere. And if your app uses ANSI SQL you can move relatively easily between Oracle and other database servers. Hence the historical definition of Open Systems, as opposed to other uses of the word Open.
I understand the intent, but I think this is going to cripple the government's ability to procure technology. From my experience working with the state, further crippling is not something they really need.
Good intentions. Bad Bill.
Not to mention that if we put it in an open format, citizens will not be able to open it and will call up complaining and we'll have to talk them through it.
You haven't lived until you've tried to explain to a cranky 80 year old what acrobat reader is, nevermind open office.
The real problem is that the open standards aren't useful for people and the standards that are used aren't open.
Yes, believe it or not. You obviously haven't ever read many state statues, I suggest you do, it can be both educational and entertaining. They always start off by defining terms. In this case:
"Open standards" means specifications for the encoding and transfer of computer data that:
(1) is free for all to implement and use in perpetuity, with no royalty or fee;
(2) has no restrictions on the use of data stored in the format;
(3) has no restrictions on the creation of software that stores, transmits, receives, or accesses data codified in such way;
(4) has a specification available for all to read, in a human-readable format, written in commonly accepted technical language;
(5) is documented, so that anyone can write software that can read and interpret the complete semantics of any data file stored in the data format;
(6) if it allows extensions, ensures that all extensions of the data format are themselves documented and have the other characteristics of an open data format;
(7) allows any file written in that format to be identified as adhering or not adhering to the format;
(8) if it includes any use of encryption, provides that the encryption algorithm is usable on a royalty-free, nondiscriminatory manner in perpetuity, and is documented so that anyone in possession of the appropriate encryption key or keys is able to write software to unencrypt the data.
ASCII is only an Open Format between characters 32 and 127. All others may have proprietary interpretation.
Unicode UTF-8 text excluding control characters other than tab, return, and linefeed, is an "open data format", and all internationalized word processors support that. Even Notepad in Windows 2000 and later.
Microsoft still refuses to support the proper \n line break.
O rly? Do Start -> Run -> edit to open MS-DOS Editor, which has been part of Windows since Windows 95. When opening a text file, MS-DOS Editor treats both \r\n and \n as newline sequences, but it always saves with \r\n. Opening a UNIX text file in MS-DOS Editor, saving it, and reopening it in Notepad will work correctly.
I mean we don't need open standards -- we need we need competition, er, innovation, er, securit... uhm... you know what I mean!
Free Software: Like love, it grows best when given away.
I don't think the intent of this bill has anything to do with the Open Document standards, MS Word, or OpenOffice. I work for a vendor that provides systems for MN state agencies (law enforcement) and there is big hoop-la with XML based web services and otherwise data sharing. There are several state run systems in place and many more up and coming to facilitate and even mandate XML based data sharing (with a state defined schema) for law enforcement agencies. It seems to me this has more to do with data-bases than data-documents.
The law wouldn't require ODF, but open data formats (of which ODF is just one; or is on it's way to being one, it's in the EMCA and/or process right now as is MS Office 2007 formats).
I commend MN for being consistent, and not throwing in PDF has MASS did, which showed MASS utter hypocrisy (only a small subset of PDF is recognized by EMCA as a standard; the full PDF is proprietary controlled by Adobe, yet MASS blessed it).
-- "I never gave these stories much credence." - HAL 9000
I certainly support the idea that Govt and public-domain agencies should use file formats that are royalty free, published, and ideally, with a source-code reference implementation for reading them (assuming they aren't self-describing) and a bunch of sample documents to use as test data for said reference implementation.
What might be surprising is that I hold this view inspite of being a Microsoft Employee. While I certainly want as many people as possible using MS software, I want them doing so because it's the best choice for their situation. I'd like to think that Word can deliver more value to its users than the ability to open Word files, so if govt agencies want to mandate that documents be created, shared, stored, etc in published, royalty-free formats, that's fine with me. Government agencies are a large customer of ours, so hopefully government action around requiring open file formats will push us to make our tools best-of-breed for dealing with those formats, or may even push us to open some of our own.
I don't use WMA for my music files, even though I could just email the guy that designed it if I have a problem. Just because I can today, doesn't mean I can 5 years from now. And there won't ever be a supported WMA player for something like OpenBSD, which might otherwise be a perfectly good audio appliance.
Now here's where I explain the title. As much as I am for the idealistic POV that open formats should be used where possible, I also beleive that govt is amazingly effective at turning a good idea into a bad law. (See also: 99% of current US laws). Another comment suggested that this story is more about Sun/IBM fighting MS via legislation, as opposed to some ideological position that is being done for the best interests of the people. If that's true, it's unfortuneate that our govt is continuing to do these sorts of things that are allegedly in the best intersts of "the people" but no one can explain exactly how, and ultimately, other businesses or politicians seem to derive the most benefit.
My opinions are my own, and do not necessarily represent those of my employer.
No, no, no, and no. However, there's no need for any of those. There are thousands of database programs out there. Quark Express is a substitute for competence. The GIMP can do most things Photoshop can, and more than enough for government purposes. RTF is open, and that's even supported by Word.
In so many ways and so often government has picked up the ball where few others would. Where ideas that aren't popular, that are too expensive, or that appear to risk all, government is regularly mandated to take on the challenge we would not as individuals, groups or businesses.
Whether building dams in Nevada or sending Americans to the moon, it is the one entity that can face risk of failure and not flinch. In part this is because it represents our collective courage to attempt the impossible.
For this reason it is entirely logical to me that it is a state, in this instance the great state of Minnesota, that would take on the the challenge of ensuring that its technological infrastructure is solid and, if not, fix what needs fixing.
Indeed, what other company or organization in recent memory has challenged itself with so introspective and potentially damaging a question in order to arrive at a better place.
Open Formats are a difficult proposition to be sure. No one denies this. But it is becoming increasingly clear that without the safeguards associated with open formats the State abdicates control of the very information it is suppose to protect.
Regardless of derived the system a solution must be constructed that returns balance and moves control of State information back to those with whom the trust has been placed.
I for one will be writing my representative to support the Bill.
Off Topic: Election Judges can't read software.....therefore elections must not be dependent on software for certification of the main balloting.
I Say that as Citizens, with the "common man" as Citizen election judges, each group of election judges must, with their existing capabilites (say, high school education, in International Falls MN), shall not be precluded from personally certifying the ballots and the summation of counts up to the state level.
Therefore, all software, Diabold, opensource, or otherwise, should be disallowed on the face of it.
It could be that a shrink-wrap spreadsheet, self programmed to help with addition, and first tested by the local judges with known test data (and no vists from the "vendor") would fall within their capability and use.
It is the common man (gender neutral) citizen judge, not some State (appointed) administrator that is RESPONSIBLE and therefor must have method and tool decision power.
We don't need results before the polls close.
We do need results that we, as citizens, can trust.
All ballots should be paper.
All intermediate sums should be printed (hand, adding machine, or personally coded spreadsheets).
Paper Ballots:
How about the Check printing business?
Quick turn of anti-counterfit, serialized, personally labeled documents.
Standard smallish size, easy to handle,
Each precient can order their own color/background.
Printers can know that they only take orders from the right person.
Printer and precient and third parties can assure a single printing of the right number of ballots...
It's not often my home state gets Slashdotted about. Despite U of M being quite techie, there doesn't seem to be much of that kind of news originating from the area.
Both "major" parties have a vested interest in protecting the status quo. Nearly all states went to that sort of "all or nothing" EC allocation in the early mid-19th century, you know. It's not like partisanship is a new phenomenon. Geo. Washington warned us it.
We need a system that reduces the dependency on partisanship. I don't have anything against partisanship, per se, but I do not like some of its effects. Making the system nonrewarding to those effects would help diminish those aspects. I believe the solution is making the political machine more multipartisan, and I think the way to do this is using Condorcet voting instead of plurality voting. When the electorate can vote honestly, rather than strategically because of "wasted vote" concerns, then you'll get true representation in office.
BTW, you're more likely to find the minor parties more likely to work "for the people" than the major ones, because they are not (yet) in a position of needing to protect themselves. Their whole appeal is that they reach out to the masses not being served by self-serving parties. I used to live in MN and that's where I first got active in partisan politics - but it was a "third" party. The experience on the whole was great. Do make sure to pay attention to the "character" of the party though...any small party can be "for the people" when small, but you want to support one that will continue to be so when it grows, a "principle over politics" approach.
Constitutionally Correct
Fist off: I agree with you totally. It's a sad way the world works, lobbying and all. Not only IBM or Sun, but MS also. Just look at the pro-software patents camp here in Europe.
:(
Anyway, the question: Why doesn't Microsoft just support ODF? It's in their interest, just as supporting wordperfect's formats were. I'm confident they have the manpower to compete on open standards and remain the market leader for a long time ahead, as much as I personally don't support them. It's a simple solution to a simple 'problem' these states brought up. They would most likely win any government contract either way, because they have 'better legacy support' for the doc format and they have the advantage of not having to make the government retrain existing users, or to have citizens buy or download another office suite. It would also be a tremendous PR boost, even if some of us would be careful in embracing microsoft again. The only good reason I can think of is MS's vendor lockin. Which is really sad in the same way lobbying is. IMHO, with the assets MS has, and with the bragging about innovation they do, they should be the first to embrace open standards to "Show people who the boss is". Today they just think and work the way a 'small player' would... It's embarrassing for a company their size...
I wholeheartedly approve. For years, one of my biggest and most complicated tasks has been the transfer of data from source A to target B. Usually A is in a format that is undocumented and inconsistent. Reverse-engineering can be a nightmare when formats are badly designed (although that explains why they are often considered propietary) and used randomly. You would be surprised by the vast number of programmers who neglect to start a file with an identification tag and format version number.
I would happily go a step further than the MN proposal. I would require by law that all files produced by commercial software have to be in a formats that are documented and open, so that they can be read by any programmer skilled in the art and in possession of the documentation, which the supplier of the software has to give to you.
Actually anyone who creates a new format should be required by law to deposit its description in a public, easily accessible facility. A kind of Congressional or British library of file formats.
I am not totally opposed to patents on formats, but I would require any company that patents a file format to give any of its competitors a license for its use at small cost.
Goes to show: The real Green Party is flat and rectangular and comes in wads.
It's becoming increasingly hard to reverse engineer formats. Why?
Encryption
Legal protection for the encryption, the DMCA
Software patents
The relatively new concept of being able to own a file format.
NO NO NO S3730 IS NOT THE RIGHT BILL. it frankly looks like something put together quickly and almost has the FEEL of something submitted by MS to create confusion. 3971 IS THE CORRECT BILL AND THE LEGISLATION SUBMITTED TO SLASHDOT. It is more sophisticated and techncially sound. AGAIN 3971 IS THE CORRECT BILL AND THE LEGISLATION SUBMITTED TO SLASHDOT. It is more sophisticated and techncially sound.
this is about data in formats that will withstand the test of time and market churn. not slamming MS. it is also not about OS.
You're exactly right!
I'm so glad this is a "we must use open formats" bill, and not a "we must use open source" bill. Open Source software can lock you in to closed formats almost as badly as proprietary software can, simply by putting an undocumented format/interface/protocol behind a complex API.
It's a depressingly common situation to find oneself in: Reverse-engineering a protocol or format by playing with inputs and looking at outputs because that's easier than decoding (hah) the source that generates it.
if they're using G.729 to save bandwidth on their Cisco Call Manager based VoIP deployments?
Unlike the other ITU-T G.X voice standards, G.729 requires licensing fees to the patentholders.
Microsoft is trying to get their .doc format certified through ISO. But will it be truly open after that? Or are there patents on it that will prevent open source projects from utilizing it?
The bill was sent off to Rep. Tingelstad's Operations' Committee originally, but now has been joined into a larger bill, according to my state representative. Tingelstad's district neighbors mine - so I inquired about the bill to both Chairwoman Tinglestad and my representative.
Within minutes, Chris DeLaForest, my (wonderful) state, rep replied to my inquiry as follows -
===============
Michael,
I believe the text of this legislation was incorporated into the State
Government Finance committee's omnibus bill. There was no controversy
regarding this language. Although it's touch to tell fortunes around
here, I do not anticiapte any hostility to this language and believe it
will become law.
Sincerely,
State Rep. Christopher J. DeLaForest
House District 49A
503 State Office Building
Saint Paul, MN 55155
Web: www.house.mn/49a
================
HF3971 is the real deal. hf3982 is a very different bill. combining the two just confuses the situation. i'd suspect the two bills were written by very dissimilar people with very dissimilar objectives.
The bill's definition of "open standards" is laughable. It's all "open" and no "standards."