The US Government and Open Standards: a Tale of Personal Woe (thevarguy.com)

An anonymous reader writes: This article details a Linux user's struggles to submit a grant application when the process requires finicky, proprietary software. It also covers familiar ground made timely by the upcoming elections: the U.S. should prefer open source software and open standards over proprietary alternatives. The grant application required a PDF created by Adobe Acrobat — software Adobe no longer supports for Linux. Once the document was created, attempting to submit it while using Ubuntu fails silently. (On Windows 7, it worked immediately.) The reader argues, "By requiring Acrobat the government gives preference to a particular software vendor, assuring that thousands of people who otherwise would not choose to use Adobe software are forced to install it. Worse, endorsing a proprietary, narrowly supported technology for government data poses the risk that public information could become inaccessible if the vendor decides to stop supporting the software. Last but not least, there are privacy and fairness issues at stake. Acrobat is a totally closed-source program, which means we have to take Adobe's word for it that nothing sketchy is going on in its code. ... It would seem to be in the interest of the public for the government to prefer an open source solution, since it is much harder to hide nefarious features inside code that can be publicly inspected."

Comments

Re:who here can fix that?

[pz]

Hmm. I use Firefox / Fedora to access both the NIH and NSF web sites without any problems.

I also use Adobe Reader / Windows to fill out the SF424 forms because, well, if it screws up because you've got your panties in a twist about not using one company's software versus another's, and you don't get the grant because the form was unreadable or inconsistent, you have no one to blame but yourself.

Indeed, I was just submitting the JIT (Just In Time) information for a DARPA award and the PDF form wasn't working correctly despite having been recently downloaded. Whom do you think gets the sharp end of the stick if I were to submit a wonky form? You go ahead and be pendantic and self-righteous and blame the government; I want to keep doing science. So the old copy was deleted, and a new copy re-downloaded. Fortunately, it wasn't some hidden, Fed-sponsored pro-Adobe conspiracy, but likely a simple TCP error during the first download, as the newer copy worked just fine.

Moreover, when it comes down to it, grant applications to the US government are likely accessible by the public through FOIA requests, so it's not like the information is really private or protected in any deep sense. What sort of nefarious activity does the OP suspect Adobe will commit with the data in the application anyway?

The current use of a PDF-based application is phenomenally better than it was before when the applications required a specific program to be downloaded in order to fill them out. That was frustrating to say the least, highly non-portable, and full of bugs. The present PDF-based mechanisms are great, simply great, in comparison. They also work very, very reliably.

There are battles that are worth fighting, and those that aren't. I'm always pleased when the US government allows me to use my Linux box (and I do that preferentially), but as a realist, I also have a dedicated Windows box on my desk for exactly the times when the assumption has been made that Windows is the computational substrate. That the government no longer requires .DOC files in its grant applications (at least the ones I see), and takes PDFs instead is a huge, huge win.

Re:Horrible Summary: Some clarifications

[nmb3000]

XFA PDF's are different in that there are no actual postscript commands in the PDF and they do not use the AcroForms technology. The layout and form inputs are defined in an XML document embedded into a PDF container. Adobe Reader then dynamically generates the postscript to render the document on the fly when the PDF is opened. If the PDF reader being used doesn't understand XFA (for example. pdf.js), then they get the generic "Please open in Adobe Reader."

Good god. Just when I started thinking Acrobat couldn't get any worse... What ever happened to the Portable part of the Portable Document Format? :(

Adobe needs to stop riding on the coattails of the PDF standard and just create their own damned document format completely separate from PDF. They've been shoving more and more of this kind of stupid shit into PDF files for years, all under the guise of PDFs being a "standard" -- just to encourage the spread their bug-ridden malware by making the files unusable in other programs. It's gotten worse than the ActiveX webpages from the early 2000's.

Re:The "Trust us" aspect is intentional

[serviscope_minor]

No, in this case I disagree. I'm not usually a fan of Hanlon's razor, but I think it applies here. I recently had an experience with the US Govt in this regard submitting an application to the NIH.

It was a second stage grant so a chunk of the proposal was how you did on the first stage. And they let you submit a video. So far so good!

What about the formats, well, not only did they allow wmv and mov, they also allowed the industry standard, open (if not unencumbered) and widely supported h.264 in an mp4 file.

Woah! That's amazing. Open standards are great, that should work anywhere, easy to make, etc etc nice happy flowers and bunnies and rainbows and unicorns yay!

Oh and the file has to be embedded in a PDF.

er, what? I mean, u wot m8? I mean WHAT THE EVER LIVING WHAT WHY WHY WOULD YOU DO THAT WHAT DO YOU EVEN MEAN???

I am not kidding that was a requirement. So this comes with about a billion problems. First, "embedded" is ill defined: some versions of PDF support video playing in the PDF, but they can also hold files you can simply download. In the former case, Adobe (tm) decided to do it two different ways in two different versions. The first (older) way is to embed the video file and use the system's video player to play the video.

That's moderately sane. Was PITA before every platform supported MPEG4, but even back then I had a PDF which would play on Windows, OSX and FreeBSD (probably Linux too---sis not check). These days it should be easy---just use MP4.

Except it doesn't work that way any more. No, the newest version which not everyone has will only play stuff using flash. So, you have to find a flash player for the video and convert the video to flv and embed it that way. So far, so bad. Flash player is getting somewhat rare now, at least the standalone flash plugin not bundled with a browser (chrome?). And it ain't bundled with acroread.

Well that's all pretty obnoxious. Firstly the methods are mutually incompatible, of course. Naturally because one is for older acroread, one for newer. The file size is strict so you can't embed it both ways and hope for the best. Actually we couldn't get the flash version to work on anyone's (windows) machine. Well, fuck you very much Adobe.

So what I did was the third method which is to have it as an attached file. Double clicking on it invites you to save or open it.

Naturally of course NONE of these things work in anything other than acroread. None of the other PDF readers---the sort everyone seems to have now, like the firefox and chrome ones, the mobile ones or the one embedded in newer versions of windows---work with these methods.

And thankfully someone figured out how to do this in LaTeX. Scott Pakin of course---anything sufficiently obscure in LaTeX always ends up there. Anyone else noticed that?

So there it was, I had the nice, standard works anywhere video file embedded in a uh... PDF where you had to piss around to open it. It was still accessible to submit for anyone using open tools, but WTF?

Oh and of course I tried including a youtube link for when it didn't work and the PDF got bounced with a snippy message pointing out angrily that of COURSE links weren't allowed (heaven forfend!) because then someone might CHEAT by linking to a longer video than is allowed!

This is one of the cases where I think only incredible incopmetence and not malice describes the situation.

Re:Not that crap again

[TheRaven64]

Except that the "open" PDF standard you're talking about is only a small subset of the oldest, most primitive image/text drawing features of said file format

That's not even remotely true. Read the PDF 1.7 specification (chapter 8, specifically) and you'll see all of that stuff documented. JavaScript has been part of the spec since PDF 1.3. The fact that some viewers don't implement features that have been part of the spec for over 10 years is not the fault of the spec.

You might be thinking of the PDF/A family of standards. These are ISO standards for long-term document archiving and specify an intentionally restricted subset of PDF features to ensure that it will always be easy to implement readers for them.

PDFs are not PDFs

[rickb928]

At work, my clients use PDFs to submit to us regularly. We immediately convert this to a TIF for our use...

This is not so good as it seems.

We have one client that uses a custom font. Yeah, really. Being not just custom but copyrighted, and they do not include it in the PDFs, when they submit, and our converter makes the best choice it can to make this into something we can use internally. Sadly, the mapping is off by one character code. The original word 'carrot', for instance ends up being 'dbsspu'. Really. They could not change this. We could not change this. They submit using PNGs now. 'Solved'.

Another client uses some third-party PDF software to send those to us. Their solution results in perfectly readable files that our converter refuses to recognize as a PDF. I looked at the data, and it looks ok to me with an unuusal qualifier in the header. Seems their software creates PDFs with version numbers that can't really exist... Solution? Open the PDF before they send ti to us, save it, and magically somehow it changes things.

Another client sends us PDFs that often convert perfect images, hidden behind what can be described as zebra stripes. Except for when it looks like black & white leopard stripes. Solution? Send us JPGs.

PDFs are a lot more complex and difficult than people think. So many third-party apps that generate almost-compatible PDFs, Adobe probably trying to kill these by modifying the file format, adding features that just don;t come out so well, it's not bliss with PDFs.

But to the OP, what open document format would we want the government to use? It should be first, read-only when needed, for instance for applications and submissions, though read/write as an option, of course. Signable. Able to secure, probably via certificate. Forms capability of course. Does this readily exist?