Slashdot Mirror
The US Government and Open Standards: a Tale of Personal Woe (thevarguy.com)
An anonymous reader writes: This article details a Linux user's struggles to submit a grant application when the process requires finicky, proprietary software. It also covers familiar ground made timely by the upcoming elections: the U.S. should prefer open source software and open standards over proprietary alternatives. The grant application required a PDF created by Adobe Acrobat — software Adobe no longer supports for Linux. Once the document was created, attempting to submit it while using Ubuntu fails silently. (On Windows 7, it worked immediately.) The reader argues, "By requiring Acrobat the government gives preference to a particular software vendor, assuring that thousands of people who otherwise would not choose to use Adobe software are forced to install it. Worse, endorsing a proprietary, narrowly supported technology for government data poses the risk that public information could become inaccessible if the vendor decides to stop supporting the software. Last but not least, there are privacy and fairness issues at stake. Acrobat is a totally closed-source program, which means we have to take Adobe's word for it that nothing sketchy is going on in its code. ... It would seem to be in the interest of the public for the government to prefer an open source solution, since it is much harder to hide nefarious features inside code that can be publicly inspected."
When opened in Acrobat Reader it had a form with a button at the bottom to submit the information. He tried to process it using the most recent version of acrobat for each of the following operating systems:
The takeaway is this: a government process used a supposedly open format but ruined it by using a proprietary extension that only worked on a recent version of proprietary software running on a recent version of a proprietary operating system.
Except that the "open" PDF standard you're talking about is only a small subset of the oldest, most primitive image/text drawing features of said file format, and the aforementioned government website is not only requiring use of a PDF document that used some of the newer (massively insecure) JavaScript-enabled interactive form input/validation features not included in said "open" PDF standard or implemented outside of Acrobat, but apparently they even then used said features to code the document such that it blocks you from even trying to read the document without Acrobat.
Go ahead. Go download it and try to open it with Xpdf, let us know how that works out.
In order to get this working under Linux, you have to install the (Ancient) Adobe 9.5.5 Reader and its associated npppf module. Then it will work. I have alot of experience with this. While Okular, Evince, and XPDF can fill out forms, there is no support for submitting an XFA Form under anything other than the real Acrobat Reader.
I am an Administrative Official for a large organization. Uploading grants is literally a major part of my job. (As a research scientist, I also write my own grants - so I understand this from several angles.)
The argument that open standards should be used is a fair one, but it is missing the bigger picture here. The vast majority of grants (NIH, NSF, Veterans Affairs, DoD, etc.) are SF-424 NIH standard packages obtained through Grants.gov and submitted by an AO such as myself, not by the applicant. Very few grants require the person authoring them to be the signing official who agrees on behalf of the organization to administer funds if the grant is successful. The vast majority of the applicants therefore route grants through a corporate or University network, where Windows (and to a lesser degree OS X - I'm a Mac user myself) predominate. In all of these cases, the organization will be providing the tools necessary - Acrobat is handed out like candy in my organization. It's part of the corporate image for all computers. Using Acrobat forms streamlines and simplifies submission for 99% of the applicants. The government is not going to change this to address a few edge cases.
The suggested alternative - web forms - is laughable. It might be good for one person, but in an average submission cycle I am sending 10-15 grants with widely varying requirements including esoteric formatting issues, hard-coded naming conventions, and etc. - not to mention that the typical grant includes dozens of required components and attachments, each with set formatting restrictions. It is hard enough to comb through an assembly SF-424 package to check for errors prior to submission as it is. If I had to manually upload each of these grants, one at a time, one piece at a time, into a web forms system, I would not be able to do my job. Period.
Post-submission, forms are processed by a clunky system in eRA Commons, then get referred to Grants.gov for eventual routing to the reviewing agency. The system has a series of automated checks built in to verify that the package is complete before it is assembled. This requires the various bits and pieces to be separate documents, as they are in an Acrobat package (and it is a package, with embedded attachments, not a flat PDF). This process is flaky and fragile enough as it is. Web forms are not going to improve the process, but they certainly would increase the workload for the AO by about 1000% and would definitely increase the error rate. This is also ignoring the fact that the forms are modular, in that some sections (like the budget) are only inserted as needed, and the necessity of being able to assemble and pre-check these things offline precludes any kind of web form system. The article writer is being intentionally obtuse and a bit naive here to make a shallow argument in favor of open standards. Heart is in the right place but reality is being ignored here.
Tl;dr version: it's hard. We do the best we have with the tools provided. Just be glad Grants.gov didn't decide to use InfoPath instead of Acrobat.
HTML forms are a bad idea for proposal submission.
I've written quite a few grant submission systems (I have a grant cycle running right now, with a deadline of this Friday...yay...). It's a pretty standard deal- web based system that allows for a fair amount of meta data (PIs, co-operators, institutions, name of grant, funding request, etc.). These of course are all part of the HTML forms.
BUT- the proposals themselves- the 2-20 page document where they explain the project- is always a complete mish-mash of stuff that could never go into an HTML form. Formulas, images, etc. Tons of formatting. And typically it is a document that has been shared/edited with other researchers. I ran one system about 15 years ago that was HTML only, and the number of projects that had 8 different PIs, who all wanted edit rights at the same time was way too high. This was pre-Google Wave, and the idea of 8 people simultaneously editing the same text on the web was insane then...as it is now.
Plus, the way that researchers/PIs handle these submissions is to turn everything in at the last possible minute. Any complication on the receiving system will just cause you to get your ass chewed out in the hallway at the next big conference.
I absolutely, 100% never ever want to hear someone say, "I tried to submit my proposal, I typed everything in, then there was an error." Because really, these people will open the page, then sit on it for 3 days as they dink around. When they finally hit 'submit' they're surprised that there was an error. Yes, there are technical ways to mitigate this problem...and the very best way is to have the applicants submit documents.
But, in the case of this article...I usually provide support for these systems. I've been doing this for about 20 years, so I'm fairly good at it. And the absolute quickest way to provide support to someone having problems is to say, "Just email me the document, and I'll submit it for you." 90% of the time I get an email that says, "I figured it out...thanks for your help." 8% of the time people say, "I tried to email the document, but it failed...my file was corrupt, so I re-saved it and then submitted...thanks for your help." The last 2% send me the file, I convert it if necessary, and we move on. (that's 2% of the problems, not 2% of the submissions)
There is no reason for me to make a 100% bullet-proof, all-inclusive system that will handle every single different scenario perfectly. It would take too much time. For the very small number of people with a problem, I just do it the old fashioned way. So if somebody told me, "I'm on Linux, and I can't convert my file to PDF, and I don't want to use one of the billion on-line PDF conversion tools, why is the government supporting Adobe and Microsoft!!!, blah blah blah" I just tell them to send me the file. In about 3 minutes I'm done and they are happy. Once upon a time I even hired temps to do this work- but these cases are really about .5% of submissions, and it just isn't worth it.
The article wasn't about the practical aspects of using PDF, it was about the (crap, can't think of the word...) aspect, where someone got their panties in a bunch because the government doesn't facilitate their worst-case-scenario approach to proposal submission.
Source: Been doing this for 20 years for the gub'ment. Yes, there is a guy like me behind most of those systems. See the part of the submission site that says, "For technical assistance...". Yeah, call me or send me an email and I'll take care of it for you. That's why they pay me, and good service is how I make the system look good.
***On the other hand, when you send an email to me, my boss, the funding organization and the overarching agency describing how the system does not function properly, and you were not able to submit your proposal...yes, I will send back a very detailed screenshot laden email pointing out step by step how you failed, and probably send the logs showing that you logged on one time 3 hours before the submission deadline. Goddam I hate it when people blame their failings on the system.
No reason to lie.