Obama Orders Federal Agencies To Digitize All Records

Lucas123 writes "President Obama this week issued a directive to all federal agencies to upgrade records management processes from paper-based systems that have been around since President Truman's administration to electronic records systems with Web 2.0 capabilities. Agencies have four months to come up with plans to improve their records keeping. Part of the directive is to have the National Archives and Records Administration store all long-term records and oversee electronic records management efforts in other agencies. Unfortunately, NARA doesn't have a stellar record itself (PDF) in rolling out electronic records projects. Earlier this year, due to cost overruns and project mismanagement, NARA announced it was ending a 10-year effort to create an electronic records archive."

Comments

This shouldn't cost too much.

[Anonymous Coward]

There are only about 1,300 federal agencies. Each will need it's own resources to come up with a specialized method that will work for them...
A couple of trillion should do the trick.
Think of all the jobs this will create! Can you feel the stimulus?

Re:This shouldn't cost too much.

[forkfail]

Questions worth considering:

What are the savings for going digital? (Without a doubt, they exist; if not, we'd still all be filling out forms in triplicate at work.)

What is the up front cost to convert?

How long will it take the up front cost to be absorbed by the savings?

I suspect that it will pay for itself faster than you might think. Paper records searches are expensive to say the least. And they're extremely personal intensive, not to mention inefficient and error prone.

I realize that there are people out there who will condemn anything this administration does out of hand, but at least try to pretend that you think about things before you make a judgement.

Re:This shouldn't cost too much.

[NotQuiteReal]

There are only about 1,300 federal agencies

Ok, mark this as troll...

Only a few dozen that should even be, per the Constitution...

Then, why a blanket "save everything" vs a "save nothing (e.g. age worthless junk)" edict?

Although, if you save stuff, e-copies do take less space... but can be digitally manipulated... so I say don't bother keeping it, if it isn't real... (you know, the old, if something is worth doing, it is worth doing right")

Re:This shouldn't cost too much.

[hairyfeet]

An even better question, what is gonna keep them from just changing the old "we lost the files" FOIA bullshit to "Ooopsie. somebody wiped a drive!" FOIA bullshit? If they are gonna go digital we need a ROCK SOLID way to make sure they don't pull shit like the last one did with sending all their email through the RNC so they could be conveniently "lost".

Re:This shouldn't cost too much.

[WindBourne]

Major savings in real estate since the space will be freed. And oddly, much of the paper is kept local, rather than warehoused
Printing will plummet.
Labor will become greatly productive. Even the worst search engine is faster than a human search.

This has an added advantage of allow citizens to search through it.
Now, one thing that most are missing is that this will ALSO allow the feds to search it quickly. TOTAL INFORMATION.

In spite of the last, I would still do it quickly. I am amazed that he did not do this sooner.

And yeah, there will be idiots that will blast him just because he is a dem, without a single care about real costs.

Re:This shouldn't cost too much.

[TheCarp]

> What are the savings for going digital? (Without a doubt, they exist; if not, we'd still all be filling out forms in triplicate at work.)

Will save a lot of time for people looking to leak documents to wikileaks. On those grounds alone, this is my favorite Obama decision to date.

Finally we may see some real freedom of information acting.

Re:This shouldn't cost too much.

[datavirtue]

What is the up front cost to convert?

How long will it take the up front cost to be absorbed by the savings?

Irrelevant accounting speak, it has to be done, so just do it.

Re:This shouldn't cost too much.

[datavirtue]

"I am amazed that he did not do this sooner."

You have never worked in government have you?

Re:This shouldn't cost too much.

[CrimsonAvenger]

What is the up front cost to convert?

How long will it take the up front cost to be absorbed by the savings?

Irrelevant accounting speak, it has to be done, so just do it.

Not really. If it's going to cost $200 billion to do, and take 150 years to recover the cost, it's probably not worth doing.

Note that above figures were pulled out of my ass, and are not intended in any way to be realistic. Though US Government policies will tend to make the process far more expensive than you might expect, and cost recovery times far longer than you might expect.

Re:This shouldn't cost too much.

[WindBourne]

Just CDC, NASA, and several TLAs, but otherwise, no.

BTW, I was meaning, that I was surprised that Obama did not do this earlier.

Re:This shouldn't cost too much.

[Teancum]

What keeps government bureaucrats from doing that same thing is some laws with real teeth to keep it from happening. Those who try that kind of stunt can find themselves losing retirement benefits, spending time in prison, and certainly out of a job with just cause in such a way that they would be unemployable in any similar kind of job position.

The reason the RNC or any other political organization can get away with doing that kind of act (including the DNC or even the Constitution Party) is that the same laws don't apply to "private organizations", which instead have internal policies and contracts that do pretty much the same thing if the organization has any real merit. The problem with political organizations is that setting any kind of policy is going to be.... a political process that usually has strings attached and other sorts of problems including a desire within those organizations to deliberately lose many records.

Yes, it happens in the government service, but the people at the top usually pay more attention and there have been at least a few people in some positions to establish policies that keep such sort of "accidental" erasure or deletion from happening. Some of those laws were written in response to the Watergate Scandal, but there have been other laws written as well, some of them nearly 200 years old in some form or another in terms of preserving records.

Unit Of Measurement

[wideBlueSkies]

So, how many Library of Congress equivalents worth of material are they intending to scan??

Re:Unit Of Measurement

[Anonymous Coward]

At least 1.

Re:Unit Of Measurement

[ron_ivi]

Brings up a great point -- why not have the Library of Congress manage it.

The LoC is very good at managing digital content -- and making it searchable/available through partnerships with open source projects like the Univ of Michigan's Hathi Trust project:

http://www.implu.com/federal_contracts/listing/LC-HathiTrust

Could the National Archives and Records Administration outsource this project to the LoC?

Re:Unit Of Measurement

[Anonymous Coward]

Just think of all the government work this can create. Perhaps it can save the US economy and give the laid-off postal workers something to do.

Re:Unit Of Measurement

[arkenian]

So, NARAs problem is that there really are no standards, as of yet, for truly archival data storage mechanisms. For NARA to go purely digital, it has to be able to guarantee PERMANENT availability and accessibility of records. That's not, as has been discussed here before in other contexts, a trivial matter.

Re:Unit Of Measurement

[sorak]

Does the LoC have adequate security? I may be wrong here, but I believe part of the NARA's job is to keep classified records (espionage and wikileaks fodder) so they can be released at a later date. I'm not sure if the LoC does anything with classified material.

Re:Unit Of Measurement

[operagost]

The LoC's mission is pretty much the opposite of keeping information secret!

seriously, how hard is this?

[Anonymous Coward]

Slashdot community, why is this so hard? I've worked for several organisations, all have failed to impliment a simple electronic document store. Nothing fancy, just a database of scanned forms in pdf format and the like. Has anyone worked at this sort of field? Is there some complication I don't understand?

Re:seriously, how hard is this?

[XDirtypunkX]

The company I work for has successfully implemented many such systems based on the HP TRIM software (previously Tower software), which seems to do a pretty good job for both paper and electronic records keeping. It doesn't require any custom code to do most things an organisation wants out of the box, which seems to be a key point in its success.

Re:seriously, how hard is this?

[kiwimate]

Is there some complication I don't understand?

Yes. More than one.

Nothing fancy, just a database of scanned forms in pdf format and the like.

There's the first problem. It's never simple.

First issue - if you're going to put documents in, you're going to want to get them out. How do you search for them? You're going to want to define the metadata, and that's a headache. Got lawyers? They'll want client and matter. But those fields are just about meaningless to anyone else. How do you resolve the incompatibility? Do you use different forms for different groups of users? How will the engineering department find the subpoena papers that the lawyers filed?

What fields are globally useful? Are they so generic that any search will retrieve hundreds of documents? Conversely, are they so specific as to make your metadata field selections horribly long and therefore ambiguous? (Free text metadata? Let's not go there.)

Remember that you've got to fill in that metadata any time you add a document. What's the balance between useful and annoying? Too many fields and nobody will want to fill it in. Too few, and you won't be able to find anything.

That's for new documents. When you first implement a DMS, you have a truckload of documents to be imported. You're not going to do it manually, you're going to use an auto-import. But how do you define the metadata for all those millions of documents you're importing? What if you have client/matter, for instance? Hopefully they're all already sorted, and you can use something like Kofax Capture, a seriously powerful and fast scanner, and separator sheets on which you can do forms recognition to define the metadata fields. But there's a lot of work involved up front to get that import working properly.

Don't forget the OCR. Hopefully all your paper documents are clean and will OCR nicely, so you can do full text indexing.

Security. Better get that set up right. Profile level security? It's more secure, but people will complain that they don't know if a document is there and they just need to request access because profile level security means if you don't have permissions to access a document it won't even show up in your search results. Groups. And by the way, remember to define the permissions on all those millions of documents you're importing.

Version control. How do you control check in and check out? Do you control check in and check out, or just audit it?

I've only just scratched the surface of a document management system. Then there's records management. You'll want to make sure your system is DoD 5015.2 compliant. Setting up the retention schedules...hopefully you've got a records retention policy already, otherwise that's months worth of work to define those policies and ensure you comply with all regulatory requirements while still balancing your need to purge/archive old records.

How does something even become a record? Hopefully you've already got knowledgeable librarians (yes, that's what they're called), and you just need to train them on your new RM system.

Are all your boxes already barcoded? Your RM system should be able to register where a record is - building, shelf, box.

You're probably getting the idea. The technology is easy. The processes are complicated, and they get exponentially more complicated as the size of your client base grows.

Re:seriously, how hard is this?

[Capt. Skinny]

Why is it hard? Too many people have influence in the process. Put one person in charge who will (1) actively be involved in the project and (2) have final say on decisions. No committees, no one-off directives from politicians or bosses who don't know the day-to-day details, no approval process. Just one guy calling the shots. A lot of people will be disappointed because it doesn't do X, Y or Z, or because it uses platform P instead of platform Q, but the project will be completed and will serve its purpose.

Re:seriously, how hard is this?

[Capt. Skinny]

Free text metadata? Let's not go there.

Google and it's users seem to be doing a pretty good job of utilizing free text to locate documents.

Re:seriously, how hard is this?

[kiwimate]

Very different problem space. Google doesn't need to have a high precision score in its results. A DM system, on the other hand, needs to have really good precision because its corpus will contain thousands of very similar documents. Content searching isn't going to work very well there - you need specific metadata (e.g. delimit by date of filing with the federal agency).

* If you want to get technical, your tf.idf score is going to be well nigh useless in this case. It's about precision, not so much recall.

Re:seriously, how hard is this?

[mcavic]

There's no off-the-shelf software I've seen that makes document search and retrieval easy. Even for pictures. Sure, there is software that will do it, but it's usually silly or clumsy.

It's sad, though, that the federal government can't hire people to do a job right.

Re:seriously, how hard is this?

[ganjadude]

or just expensive.

I used to work for a DMS software company at the corporate level and while the systems are on everything from elementary schools to health care providers to governments, the retrieval is pretty damn nice IF the system is set up properly. A properly set up system for a small pizzashop takes an hour or 2, a gov agency could take weeks or months to perfect. But the user side of things was a breeze.

Re:seriously, how hard is this?

[Kjella]

For all the complications you claim, how the hell do you find and manage paper records? Just mapping that 1:1 into an electronic archive with scanned PDFs would be a good start. At least then you have an archive that can be backed up easily and people can access it by opening the PDF rather than requisitioning a paper record which would be extremely much faster and require no manual labor. Then you could start getting fancy with OCR, metadata and such based on cost/benefit. And if it comes down to making a hard copy, then archival grade microfilm is supposed to be good for 500 years which is at least as good as actual paper forms.

Re:seriously, how hard is this?

[SirGarlon]

No committees, no one-off directives from politicians or bosses who don't know the day-to-day details, no approval process.

Yeah, software would be easy to design if it weren't for all those pesky stakeholders.

A lot of people will be disappointed because it doesn't do X, Y or Z ... but the project will be completed and will serve its purpose.

Whose purpose, exactly, does it serve if the stakeholders are disappointed?

Re:seriously, how hard is this?

[Anonymous Coward]

No it isn't hard because of project structure. It is hard because it is a hard problem to solve.

Re:seriously, how hard is this?

[Anonymous Coward]

Also, one must keep in mind that digitizing record management doesn't always mean digitizing all of the records- for a system of historical documents, etc, it will still be necessary to keep those old documents (they wouldn't scan them burn a copy of the declaration of independence). But the meta data used to locate those documents would be important.Systems where all of the data could be digitized would probably take several years anyway, and the process would also be improved while digitizing it.

I work for a consulting firm contracted with a government branch digitizing a large identity and record management progress, and can personally attest to it being slow and tedious, but all of the government workers we work with are blown away at the improvement.

Re:seriously, how hard is this?

[Anonymous Coward]

NARA is simply storage. They already have a system in place to ID records by what box they were shipped to them in. For NARA the task is simple, they don't need OCR, or really any meta data. They just need to make sure when they scan a doc that they give it the original ID. Hell, a SAMBA share would be sufficient if each PDF had it's file name as the original ID. Its the responsibility of the originating agency (the one that sent the box to NARA) to maintaine records of what is in each box, and the boxes ID. This is how records get lost, they're boxed, shipped, and then the guy that had the excel spreedsheet quits.

Re:seriously, how hard is this?

[Anonymous Coward]

Very different problem space. Google doesn't need to have a high precision score in its results. A DM system, on the other hand, needs to have really good precision because its corpus will contain thousands of very similar documents.

You're spot on with that. Google actually was a part of the trade study for NARA's electronic archive, but underperformed for the required application.

Re:seriously, how hard is this?

[Eevee]

Google and it's users seem to be doing a pretty good job of utilizing free text to locate documents.

Not quite. Google does a good job of utilizing free text to locate some documents. If all you want is instructions on how to fix your car or the date that an event happened, getting one good result is all you need. If you need to find every document an agency produced about the toxic effects of a chemical, then finding one good result sucks--you need (within certain error levels) every document.

Re:seriously, how hard is this?

[kiwimate]

People still have to be able to locate those scanned PDFs. Now it's electronic, you need to know where to go to get it. Is it on a network share in a well-organized directory structure? At some point it gets so close to a taxonomy that you get past the simple hierarchical mapping limits.

The traditional way to handle paper records is the method I referred to; you have them stored in a traditional vault and your RM system tracks by building/room/shelf/box. Everything is barcoded to make it quick/efficient to check in/check out.

These are not insurmountable complications. I'm really just pointing out that there's a lot of details to think about, and it has to scale - both in terms of size of company and in terms of longevity of system. It's requirements gathering for a new paradigm in a company's traditional records keeping processes (which includes the workflow, by the way - I didn't even start to get into that). Again, not insurmountable, and the technology is simple. It's the processes that are crucial.

Re:seriously, how hard is this?

[kiwimate]

Yes, exactly. If I'd thought a bit more before writing my reply above, I'd have commented that Google does information retrieval. That's different from document management and VERY different from records management.

Re:seriously, how hard is this?

[Capt. Skinny]

This is a US government project, so all US citizens are essentially stakeholders. All government agencies are stakeholders. You just can't please that many people, and it's the process of trying to do so that does projects like this in.

I didn't mean to suggest that the guy calling the shots would ignore stakeholders. He's a project manager. This person is in a position to consider ALL input and make fair compromises, instead of trying to create an amalgamation of whimsical directives by those who "outrank" him but don't have the complete picture (e.g. politicians responsible for project funding, the guy he works for, the gal he owes a favor to). While I did say a lot of people would be annoyed with the details of the end result, they wouldn't be the majority. Those people truly willing to compromise will be happy with a working system that meets the goals set out for the project.

Re:seriously, how hard is this?

[Anonymous Coward]

You're probably getting the idea. The technology is easy. The processes are complicated

This is a great post. I think kiwimate has a very good understanding of digital archiving.

There really is a lot more than scanning documents and putting them into mass storage. In fact, most of the DM stored by NARA has far more metadata associated with it than actual content. There are several forms, requests, approvals, verification, etc that is involved in the process of ingesting data.

One big thing the kiwimate didn't mention is the legal aspect. The authenticity and accuracy of the information in these documents needs to hold up in a court of law. If information is just slapped in there without the extensive process, NARA would find themselves up to their eyes in legal trouble.

Regardless, it wasn't the technology or the process that led to the difficulties of NARA's system (which is falsely reported as a failure). The reasons had more to do with NARA's inexperience/inability with developing a government-funded tech system - which the GAO was very critical of.

BTW, archivists hate when you call them librarians (even if that's what they are).

Re:seriously, how hard is this?

[Geoff-with-a-G]

Google and it's users seem to be doing a pretty good job of utilizing free text to locate documents.

Or, to put it another way, the problem you're expecting each of these government DBAs to routinely solve required a 100 Billion dollar company which makes a point of hiring geniuses in order to tackle.

I don't think "but Google can do it!" is synonymous with "that problem isn't a big deal"

Re:seriously, how hard is this?

[SirGarlon]

So what you're saying seem to be simply that the project needs capable management and some independence from political meddling. In that case I certainly agree. But I would also say that's another way of phrasing the answer to why this is hard: to manage a project well, and to resist political meddling, is inherently hard.

Re:seriously, how hard is this?

[k6mfw]

Hard, really hard. Especially when you have racks of D and E size engineering drawings, numerous calibration printouts, stress and analysis reports (handdrawn and typed) for a facility, i.e. NASA wind tunnels, and with not enough staff to do everything (and buyout notices sent to employees). First you need some scanners (big ones for for those large drawings) then some people to spend tons of time to scan, convert, store and organize all that material. By the time they are done, we will all be dead of old age!

Re:seriously, how hard is this?

[Anonymous Coward]

I am currently doing business analysis for a content management system at a motor vehicles agency. Currently 100% of research and processing is done with paper. The entire bureaucratically controlled process (RFP, contracting, etc.) including implementation is under 24 months. This is for an agency that handles 5 million documents per year. It is fairly simple and straight forward even in government. The insurance companies and banking already crossed this bridge and the available systems are nicely designed and proven.

This will be good news for ECM vendors.....

How Can The US Construct a Big Brother Database

[Jeremiah Cornelius]

When all the records are locked in 8x11 filing cabinets, sealed in Manila envelopes?

And the FOIA headache!

Destroying those records is hard, and some turn up - years after they were declared not to exist!

Re:How Can The US Construct a Big Brother Database

[Anonymous Coward]

When all the records are locked in 8x11 filing cabinets, sealed in Manila envelopes?

And the FOIA headache!

Destroying those records is hard, and some turn up - years after they were declared not to exist!

This is a logistical nightmare. It would be cheaper to assign a Secret Service agent to guard every file! Also, electronic filing is not as good as paper, unless you back it up many times and take it all offline.

Re:How Can The US Construct a Big Brother Database

[rhizome]

They don't even want the garbagemen to know how much they're shredding.

I'd say once Lockheed can actually implement this, you'll start seeing "now that we don't have to worry about paper records anymore" retention laws flowing through with YEAs.

Re:How Can The US Construct a Big Brother Database

[mirix]

On one hand you've got images of the stasi shredding everything at headquarters as fast as they can... On the other, they never got hacked.

I feel I should elaborate but I can't put it in words right now, so hopefully you get my drift.

So Palantir can work better?

[Anonymous Coward]

http://yro.slashdot.org/story/11/11/26/1823200/palantir-the-war-on-terrors-secret-weapon

orwell
1984
panopticon
big brother

Re:So Palantir can work better?

[siddesu]

Not only, but so that all records can be updated in real time in case of need. Remember all those movies from Men in Black to Enamy of Amurrika? Now they are one implementation away ;)

Congressional Record

[ChrisMaple]

I'd like to see 220 years of Congressional debates in digital form.

And he will receive

[Aerorae]

(the number of federal agencies)-odd number of completely incompatible digital records systems proposals.

Distributed object stores

[inKubus]

This is actually the perfect place to incubate distributed object stores (e.g. Hadoop on one end, something like Zimbra on the other). One namespace .gov, with sub-namespaces. With a CMIS interface. Anyone see VMWare Project Octopus yet? Well, take that times 10,000 and you have a pretty nice records management system, platform independent. There's also Alfresco which is using the JCR spec which I believe can be moved to some type of distributed backend. But it implements CMIS, has a DoD spec records management system.. So the general spec would be a CMIS framework, each department/branch/whatever makes available a service for document retrieval, central .gov listing of the services, basically what Amazon does for literally everything it does. Do not compromise, executive order Jeff Bezos style, everything is a service with a public interface. I think it is possible, but it would take a lot of just plain buying in and our government (the bureaucratic, non-political side) has gotten really really good at dragging their feet and doing nothing. The cuts are coming though, and they will have to improve efficiency just like we all have in the private sector. Of course Defense is the worst, but education can use some work as well.

Re:Distributed object stores

[oneiros27]

You mean like OODT ( ) ? or something more like iRODS ? Both are used by various 'big data' groups (NASA, NIH, NOAA, NOAO, super computing centers) to share data across multiple sites.

As for the indexes .... well, if science.gov and data.gov are any example, they could use some work. Although, hopefully in this case, you're describing bibliographic records, so the necessary metadata is a little more standardized.

In some cases, I'd be better to just put the records out there under standardized open APIs, and let interested parties make interfaces to the stuff they're interested.

since President Truman's administration

[swell]

Now I am truly shocked- no paper records before Truman admin?!!?

No wonder Americans are so ignorant of the past.

Re:since President Truman's administration

[siddesu]

Remember, this was when Communism tried to invade America, and so to counter it the need for a comprehensive system of records for everyone arose.

Ye$!

[Chewbacon]

Let's spend some more cash. Balance my fuckin' budget, B!

Re:Ye$!

[forkfail]

Because, you know, having to do paper record archive searches is so much cheaper than going digital. That's why all the big corporations insist that all records be stored in triplicate in properly filing cabinits... oh... wait...

Re:Ye$!

[forkfail]

So, you condemn Obama for things he doesn't do (e.g., reduce costs), then condemn him for doing things (e.g., reducing costs).

Gotcha.

Re:Ye$!

[Chewbacon]

I think there's better places to start. Elect to stop spending money on something, rather than spend money to save it an indefinite amount of time down the road.

Re:Ye$!

[artor3]

No, he condemns Obama for being black and a Democratic president, both unforgivable sins, and together they make him the anti-Christ.

Re:Ye$!

[forkfail]

Yeah - as noted, the man can't win. Ask any corporate bean counter about the cost savings (that is, stopping spending money) by going digital.

Also - remember - he's the President. He doesn't make the budget. (That's tied up in the Super Committee.) And unlike the previous President, he hasn't been ruling by fiat, executive order and signing statement.

Re:Ye$!

[ChipMonk]

Right, because all of Slashdot knows the anti-Christ will be a white Republican.

Re:Ye$!

[WindBourne]

Actually, no. There are plenty of time where corps are MUCH MORE WASTEFUL than gov. For starters, look at the shear number of bankruptcies by corps. That alone should tell you that our corps are VERY inefficient and ran by short-term idiots.

Re:Ye$!

[Anonymous Coward]

You're fucking retarded. That goes double for your next reply to forkfail, and triple for the one after that.

Now, you can put Rush's dick back in your mouth whilst you await your next talking point.

Re:Ye$!

[Anonymous Coward]

Scanning documents in no way reduces cost.

Re:Ye$!

[CrimsonAvenger]

And unlike the previous President, he hasn't been ruling by fiat, executive order and signing statement.

I'm assuming this was supposed to be sarcastic?

Re:Ye$!

[operagost]

And unlike the previous President, he hasn't been ruling by fiat, executive order and signing statement.

Did you type this with a straight face? And I wonder whether the moderator was LOLing as he modded it up. Please read this. The numbers are quite the same. And W's weren't quite so... "fiat". How else can to describe ordering the wells in the Gulf to shut down, being told by a federal court that's illegal, and then just resubmitting the order anyway?

Inevitable Release

[Anonymous Coward]

I would think that after the wikileaks debacle the government would have learned that you are at higher risk of losing control of digitized records than physical ones. There's a corollary to Murphy's law here. Digital records, given enough time, can and will become public.

Re:Inevitable Release

[siddesu]

You say it like this is a bad thing. What has the downside of Wikileaks been so far?

Lockheed Martin

[dg41]

Looking at the NARA article, as soon as I saw that some big IT contract was given to Lockheed Martin I saw all I needed to know about this initiative.

Re:Lockheed Martin

[flappinbooger]

Looking at the NARA article, as soon as I saw that some big IT contract was given to Lockheed Martin I saw all I needed to know about this initiative.

how much money did LM or someone closely associated with LM give to the present administration or someone closely associated with the present administration? Like "they" always say, follow the money.

New Republican candidate talking point:

[forkfail]

We must save our children's heritage. President Obama obviously hates America and it's legacy, otherwise, why would he be trying to destroy all the paper records? Undoubtedly, he'll claim that his long form birth certificate was destroyed during the digitization effort. It's obviously an Islamic socialist fascist communist ACORN black panther George Soros funded plot of some sort. Also.

Re:New Republican candidate talking point:

[Anonymous Coward]

It's the Michael Moore Chicago machine politics apologizing for the United States of America "in all 57 states".

Re:New Republican candidate talking point:

[sorak]

Since this helps Lockheed Martin, I'm pretty sure the GOP will let it slide. They may make casual references to "Obama increasing scrutiny of US citizens", trying to portray it as an attempt to implement Orwellian Telescreens, but that will die out pretty quickly.

Archaeology

[GrahamCox]

In 1000 years or more, they'll have no idea what we were up to at all. At lease some paper records have a chance of surviving.

Re:Archaeology

[Grave]

In a thousand years, if archaeologists cannot gather sufficient data from other observations besides paper records, then it really wasn't that important anyway.

Re:Archaeology

[Anonymous Coward]

Everyone wants their legacy to survive... I don't. Once the people who knew me personally are gone, I hope nothing of me remains. The future needn't care about my shit. I don't care about some random nerd from 1011.

Re:Archaeology

[earls]

And hence you are repeating his or her follies!

Re:Archaeology

[c0lo]

In 1000 years or more, they'll have no idea what we were up to at all. At lease some paper records have a chance of surviving.

I wouldn't worry that much... think of it... USofA has some pretty extraordinary archaeologists: Indiana Jones, Lara Croft, Rick O'Connell, Benjamin Gates... should I continue?

Re:Archaeology

[AgNO3]

You are so right. Homer's works didn't help solve anything. Hieroglyphics didn't help use out at all. Cave paintings that where used to record historical events did nothing to help ups understand. Yup writing and graphical representations have done nothing to help the present understand the past what so ever.

Re:Archaeology

[Anonymous Coward]

In 1000 years or more, they'll have no idea what we were up to at all.

Oh, noooes! Not only that we don't have a clue right now what we are up to, but even our descendants won't ...

Re:Archaeology

[Anonymous Coward]

Hmmm, I believe ms Croft is actually British.

Re:Archaeology

[Anonymous Coward]

They will be more confused when the electronic records DO survive 1000 years. If they do, chances are that they will be floating around on "futurenet 2.0", freely available for everyone. Problem will be that there will be many different versions, all tampered with a long time ago by someone to rewrite history to their advantage.

That is also the problem with purely electronic archives. It is too easy to alter them without leaving traces. Especially for the owner of the archive who may have other priorities then the people whose data is in it.

Re:Archaeology

[Dr_Barnowl]

That's Lady Lara Croft to you, peasant.

Re:Archaeology

[gl4ss]

oh really?

what we are living is the _start_ of ridiculously accurate history, with video and images and schoolbooks all in neat packages for the people of the future to examine. 100 years ago contrast between year and year on that isn't going to be huge, but the contrast between now and just several decades ago is huge.

never before has been so much information printed and recorded, never before have so many people lived who are doing their best to record information so that it's available later. never before has "confidential" information about statespersons and what they've been up to been recorded as much as in the past 50 years. the funny thing about that is that the people who lived 50 years ago had no fucking clue about what was going on - though even they knew a lot more than what people 100 years ago knew - ignorance might have been a bliss though. cross referencing between multiple sources is now much more viable than before too, where you might only get the official records and historians have just guessed the rest.

there's more books that will get preserved by pure random chance than ever before and those books accumulate information from the past to present. as well as the rate of change has now slowed down considerably.

Re:Archaeology

[sorak]

It's just as easy to destroy a written document as an electronic one. The only way this information will be lost is if the powers-that-be intentionally destroy it, or if something so catastrophic occurs that the internet becomes a historic fad. In the internet age, information is a virus. The media may come and go, but the data will live on, so long as there is another remote system somewhere to copy it to.

Re:Archaeology

[Anonymous Coward]

Data on a digital format such as a hard disk only lasts about 20 years. If the data doesn't move and go somewhere else, every 20 or so years you need to copy that data, otherwise you lose it. Paper on the otherhand has the potential to last thousands of years. People that are putting too much faith in electronics keeping this stuff around are being naive (and this is coming from a computer scientist).

Re:Archaeology

[sorak]

My point was that as long as that information is freely available and someone, somewhere finds it worth keeping, it will be copied.

Re:the fix is in

[forkfail]

Trollish troll is trollin'.

George Orwell had it wrong

[paper tape]

While there is a certain amount of (justified) paranoia that the government would use digitizing records as an opportunity to engage in revisionist history, I have to say that despite a desire to do so, the odds are against the government being able to pull it off.

In order for something like 1984's Ministry of Truth to function, the government would have to be far, far more competent and efficient than is ever to be likely.

Re:George Orwell had it wrong

[Capt. Skinny]

So, we're relying on security through obscurity?

Re:George Orwell had it wrong

[AHuxley]

Thats why they can hand it over to the private sector. What the US gov cannot not collect, the private sector can share and build on.
What the private sector cannot link over time, the US gov can do, medical, other govs.
Any laws that stop the US gov, use private contractors or friendly govs outside the US e.g. Canada, UK.
Databases are now very efficient, data entry is in place in most states in a shareable form.
http://en.wikipedia.org/wiki/Information_Awareness_Office showed the vision before it was lost into other new projects.

I hate to say it...

[koan]

Why not just pay Google to do it, the have the infrastructure experience and coding talent.

Re:I hate to say it...

[Anonymous Coward]

And have even DONE several projects like it...

But being a gov project it will be given to one of their existing big money guys....

Re:I hate to say it...

[Anonymous Coward]

Why not just pay Google to do it, the have the infrastructure experience and coding talent.

Because a search engine is only about 10% of what is involved in a true digital archive. It's not just a searchable file repository, the majority of the work has to do with processing the vast set of information that must be gathered and accompanied with the materials present within the archive. This makes for highly structured, rapidly changing data - not Google's strong point.

Not to say that your suggestion is a bad one... it actually was considered. Google was a part of NARA's initial trade study when they were selecting a search engine, but underperformed when faced with the application requirements.

ALL paper documents?

[webdog314]

Does that include the Declaration of Independence? I suppose it would be much easier to change in digital form...

Re:ALL paper documents?

[Anonymous Coward]

You mean the constitution?
Or do you think they'll want to rejoin the British empire?

Re:ALL paper documents?

[FrootLoops]

do you think they'll want to rejoin the British empire?

That wouldn't be all bad. We'd at least be able to pawn off our debt on someone else.

Re:ALL paper documents?

[gtbritishskull]

The Declaration of Independence has no legal standing under the US Constitution (and hence US laws in general). For some reason conservatives cannot fit that idea into their heads.

Re:ALL paper documents?

[forkfail]

And yet that comment gets rec'ced up as being "insightful".

Even more ironically, it's the same folks who love to talk about "life, liberty and the persuit of happiness" the most that seem to forget the bit about the next clause, "That to secure these rights, Governments are instituted among Men" more often than not.

But then, and speaking of editing the Declaration of Independence, Texas did drop Jefferson from its textbooks:

http://www.nytimes.com/2010/03/13/education/13texas.html

Re:ALL paper documents?

[Anonymous Coward]

All those are written on sheepskin, not paper. :)

Re:ALL paper documents?

[Anonymous Coward]

Does that include the Declaration of Independence? I suppose it would be much easier to change in digital form...

Why would they bother digitizing their toilet paper?

Re:ALL paper documents?

[JesseMcDonald]

Even more ironically, it's the same folks who love to talk about "life, liberty and the persuit of happiness" the most that seem to forget the bit about the next clause, "That to secure these rights, Governments are instituted among Men" more often than not.

It's not forgotten, although whether that approach is effective or even justified if effective is debatable. Either way, the nature of government is in conflict with liberty (which incorporates both "life" and "pursuit of happiness"); it can only be justified to the extent that it does more to secure these rights than its existence detracts from them. Every action it takes which is not directly in support of these rights increases that conflict and serves to undermine its legitimacy.

They'd better do something

[actionbastard]

So there isn't a repeat of this:
http://www.archives.gov/st-louis/military-personnel/fire-1973.html

David A. Powner???

[Anonymous Coward]

seriously? the guy who signed the report is mr powner.
is this some sort of 4chan joke?

Here we go...

[Anonymous Coward]

Not going to work. Half these agencies probably have no idea how to accomplish this. Managers will bring in consultants etc... and have no real idea of what needs to be accomplished. Designs will be worked up and constantly revised, no work will actually get done, because no one who has any clue about how to accomplish this will be on the payroll for this project. Requirement creep will set in, more useless people will be thrown at the project, deadlines will be missed by months, then years, budgets will be exceeded by millions, and 5 years from now we will have nothing to show for it...Great idea though.

Private Industry Can Do This Better

[laing]

Why not outsource the whole task to somebody like Iron Mountain? They could get it done quickly and economically. It might even create a few jobs.

Re:Private Industry Can Do This Better

[SydShamino]

It was given out in a contract, so you are already getting your wish.

Though I think we could save money by having the government do something itself instead of having to pay for Lockheed's profit and overhead.

Re:Private Industry Can Do This Better

[Anonymous Coward]

It was given out in a contract, so you are already getting your wish.

Though I think we could save money by having the government do something itself instead of having to pay for Lockheed's profit and overhead.

You do realize that you're referring to the America government, right? How's that postal service working out for you?

Re:Private Industry Can Do This Better

[Anonymous Coward]

If the government could EVER do anything efficiently that might be something to consider.

Re:Private Industry Can Do This Better

[kiwimate]

Sorry, but I have to disagree. As I pointed out in another post, there are a lot of factors to think about when you do something like this, and if you don't have the experience you'll make mistakes.

It's kind of like saying (my favorite distro) Linux/Windows 7 is so easy to set up these days that anyone can do it. If it's just a matter of clicking Next->Next->Next, then yes. And that might even be sufficient for a home computer (ignoring things like backups). But most people reading this will know there are a lot of things you have to know to do when you set up a server. Otherwise, it'll work, but chances are it's inefficient, insecure, and prone to the first failure that comes along.

Or - a home wireless router. You could just plug it in and be on your way. Or, you could take a few minutes to do it right and set up encryption, turn off SSID broadcasting, perhaps MAC filtering, etc. Both ways will get you up and running, at least initially. One way, however, is at the very least resulting in a risky unsecured configuration.

Re:Private Industry Can Do This Better

[surgen]

You do realize that you're referring to the America government, right? How's that postal service working out for you?

Pretty well actually, too bad our congress is needlessly squeezing it dry to make up for gaps in the budget.

POLITICALLY INCORRECT!!!111!

[Anonymous Coward]

Don't you realize making fun of Republicans around here is Politically Incorrect and will cause offense and hurt feelings?

Re:POLITICALLY INCORRECT!!!111!

[Anonymous Coward]

So you think Slashdot has a GOP bias? That's a good one.

Re:"Web 2.0 capabilities"

[Anonymous Coward]

So these systems will encourage the general populous to give them all their personal info (i.e. more than the government already has) in return for free use of some service?

Exactly, except it won't be free because it's your tax dollars.
They will however: store all your data for you in "the cloud" so it can be hacked by the cyber-terrorists which will prompt an all out internet war creating more jobs for patriotic americans such as yourself.

It's all about the formats!

[Anonymous Coward]

You missed the most important question worth considering - in what formats will these records be maintained?

And Obama missed it, too. I don't see anything in his directive about it.

Good archival practice entails preserving original documents, not just scanned copies.

And if the purpose is to place documents on the Internet, then it's a GIGO situation. If you allow garbage, closed formats like .doc or .docx or .xls or .xlsx to be put on the Web, you're not serving transparency very well, and you're defeating your whole purpose of wanting to make data accessible for Web 2.0 mashups and the like.

Why won't government ever "get" it? The prerequisite question is ALWAYS, what formats? If the formats aren't truly open, then the data isn't open, either.

Re:It's all about the formats!

[forkfail]

If you look at the executive order itself:

http://www.scribd.com/doc/74042394/Managing-Government-Records-November-28-2011

you'll find that while formats aren't called out explicitly, it basically instructs the archivist to come up with a comprehensive system within a limited amount of time. It's a pretty high level set of business level requirements; basically, these business level requirements translate to, "give me the system level requirements docs and specifics within four months." I can't imagine that such a system wouldn't include the proposed formats.

Re:It's all about the formats!

[Anonymous Coward]

LOL Seriously, you think the PREZ would be involved in deciding which FORMATS to be used??? Agreed, its an important question, but HARDLY a question that the president needs to make. Get real.

Re:It's all about the formats!

[tqk]

You missed the most important question worth considering - in what formats will these records be maintained?

No. The most important bit is they'll be available. Conversion from one format to another isn't all that difficult, though it may be time consuming to do so, and lossy.

Just look at the output of "man -k 2" on a Linux box and see which ones of those are for converting a proprietary format to a more open format.

Re:It's all about the formats!

[517714]

Formats - not so much. I would like to know which "solutions provider" made generous campaign contributions though.

Re:It's all about the formats!

[Anonymous Coward]

ISO and open formats would make sense (maybe xml for textual content), but i reckon there are a lot of software corporations out there coming up with pretty slide shows to prove their format is best. could you imagine if they chose something like flash, or worse... Microsoft SQL Server. gah!!! IBM might be a good chance though (DB2).

Re:It's all about the formats!

[datavirtue]

My guess, they will all end up in different formats none of which are text based, which is what it should be.

Re:It's all about the formats!

[sorak]

I'm not so worried about SQL server, as it is a database format. Now, I am concerned about JPEGs of written papers scanned at 36dpi. Considering the dirty tricks politicians use to be less transparent (giving out printed copies of emails to meet FOIA requirements), I would consider SQL to be an improvement*.

* Unless it is a SQL database accessible only via a web front end. I'm sure any enemy of transparency could make this unusable.

Just send them to WIKILEAKS...

[Anonymous Coward]

they'll do all the work for free!

p.s. WHERE is all that imaging going to be stored with the shortage of hard drives?
Oh, right, the cloud, it's trustworthy.

They're between a rock and a hard drive.

Re:"Web 2.0 capabilities"

[Anonymous Coward]

What do you think Facebook has become?

Will the file formats be publicly documented?

[MichaelCrawford]

This would be a good time to write your congresscritter to point out the problems with undocumented file formats as well as Apis and network protocols.

There are plenty of formats that could be used that are open and vendor neutral.

If congress doesn't require that in it's funding authorization, many of our public records will be stored as word dos or in ms SQL databases.

Re:Buffoon Obama has no Money

[Anonymous Coward]

Buffoon Obama has no Money

Because George W. Bush and the Republicans didn't leave him any.

NARA

[Rinisari]

IIRC, NARA didn't end the effort, it just stopped further development because it considered it complete.

Re:Buffoon Obama has no Money

[forkfail]

Two wars on credit combined with high end tax cuts do tend to drain the coffers with a quickness.

You've been smoking the hope

[Anonymous Coward]

You've really been smoking the hope, huh?

Ask any corporate bean counter about the cost savings (that is, stopping spending money) by going digital.

Here's the thing: the government? It's already all-digital in the places that make sense. Also, you apparently missed the "with Web 2.0 capabilities" bit in the fucking summary, which is a buzzword meaning "giant waste of money." But, hey, it's /., the summary is probably a bit off, right?

plans for improving or maintaining its records management program, particularly with respect to managing electronic records, including email and social media, deploying cloud-based services or storage solutions

Oh, so not so much. That's from the order itself, mind you!

No, this is an order that's going to waste a ton of money to not accomplish anything. The government already uses electronic records where it makes sense. Where it doesn't, they don't.

This tries to force government agencies to move "to the cloud" by executive fiat. It's a recipe for disaster and government waste.

And unlike the previous President, he hasn't been ruling by fiat, executive order and signing statement.

Yeah, not so much, actually.

Re:You've been smoking the hope

[forkfail]

Actually, I went and read the executive order here:

http://www.scribd.com/doc/74042394/Managing-Government-Records-November-28-2011

which itself says nothing about Web 2.0 itself. Nor about moving to the cloud. The requirements laid out there are business level, and basically translate to the following: "You have 120 days to come up with system level requirements to move our data from hard copy to soft copy."

With this said, the section from the order that you're quoting is 2-b-i. It refers to the need to have a unified solution for archiving all existing electronic communication. Would you prefer that every department and agency have its own? And here I thought you might be in favor of cutting costs and efficiency.

Finally, your link shows that Obama has issued 17 signing statements in 3 years. That's about 6 per year. Bush issued 161 over 8 years. That's 20 per year. The number of executive orders is similar. And honestly, the Democrats in congress didn't play the cloture games that the Republicans play now. They made a huge stink about the ONE appointment that the Democrats tried to block (remember the chants of "up ur down! up ur down!"). Now, the Republicans won't let a damn thing to the floor of the Senate for a vote that doesn't explicitly further their causes. In other words, false equivalance fail.

Re:You've been smoking the hope

[Anonymous Coward]

So, basically, you're agreeing with the AC about Obama being the reincarnation of Bush re: ruling by fiat, and then changing the subject to Obama's ineffectiveness when it comes to dealing with Congress as if that some how makes him better than Bush. Bush and Obama both suck, but your extreme eagerness in comparing the two (and bizarrely seeing Obama as the *better* of the two) just shows you as a partisan hack. Wow, one turd floats better than the other! Hurray team blue!

This order by Obama is like using a bandaid on your scraped toe when you're femoral artery is cut. The best that can be said about it is it probably won't make anything worse.

Re:You've been smoking the hope

[forkfail]

Let's see. A difference of an order of magnitude in number of signing statements. The difference between putting the war costs in the budget - and insisting that they all be by special appropriation or would veto. The difference between starting multiple wars of occupation without a declaration and not. The difference between following the law as created by congress and accepting what congress passed (or didn't as law).

Bush was effective towards his goals. Because Obama doesn't play Bush's games, but the Republicans no longer play be the rules, Obama is not effective. That's part of my point.

No, I'm by no means happy with what Obama has (and hasn't) accomplished. But I'm sick to death of the Republicans and their Rovian games and of the charred earth policy of passing nothing that will help the country (see also abuse of cloture) and blaming Obama. The Republicans declared in 2008 that they had exactly one goal: to make sure that Obama failed. And everything that they've done during these years of crisis has been aligned with that goal, while America rots.

Finally, if you've something to say, say it for yourself as opposed to trying to spin what I'm saying into the opposite. You aren't very good at it.

Re:You've been smoking the hope

[codepunk]

The republicans are doing exactly what I voted them in to do. "block stupidity" and they are doing a fine job of it.

Re:You've been smoking the hope

[WindBourne]

Please quit feeding the troll.

Re:You've been smoking the hope

[jmac_the_man]

Bush had Congressional authorizations for those wars. Unlike Lybia, I might add.

Re:You've been smoking the hope

[Xphile101361]

If they were blocking stupidity then they would pass a law to ban straight ticket voting, as it is the PEOPLE not the PARTY that matters. Oh wait, they'd probably lose power then *thumbs up*

Seriously?

[Anonymous Coward]

Web 2.0?

Holy buzzwords batman. What does that have to do with data records?

Why do government records need any web functionality at all? Do they WANT to be hacked?

Re:Seriously?

[Anonymous Coward]

The citizens demand the right crowdsource legislative responses to LOLcats KTHXBYE.

Re:the fix is in

[TapeCutter]

This is how governments control the masses in "1984".

So your saying this project is putting a camera in everyone's bedroom and tying rats to their faces? Fear and omnipresence was how the fictional government controled the masses in 1984, the masses knew the official history was manafactured bullshit in much the same way as people today know that Fox is a right-wing bullshit factory.

Dunder Mifflin

[Temujin_12]

Dunder Mifflin is gonna be pissed...

"I want a plan in four months"

[Culture20]

"And I want it to be implemented in less than four years. Then we can change all government records to show that presidents can have four terms in office. Then we'll change it to four decades. I call it my 4-4-4-4 plan. Get out your little red kindles, children. We're going to read about democracy."

In the Archival Trenches...

[jlaprise1]

As a professional historian who has worked in the National Archives in College Park, MD and at four different presidential libraries, which incidentally are also managed by NARA, I need to interject that this is an immense costly but valuable project.

Remember "the warehouse" from the Indiana Jones movies? NARA is a little like that in terms of size but are better organized. Aisle upon aisle, shelf upon shelf, row upon row, room upon room, floor upon floor, building upon building of neatly indexed banker's boxes with labelled folders of documents. The labels may have been checked by the archivists at NARA, but they may also simply be the labels affixed to the records by the source federal agency. The individual documents in folders are almost never labelled. In the course of my work, I gathered 30k digital pictures of documents over the course of two months. The acquisition process sounds deceptively easy. Look in the index, find key words and request boxes from the archivist. Then you look through folders to locate individual documents. In point of fact, I probably visually scanned 3M pages to see if they were "interesting" and photo worthy for future research, usually taking only a few seconds per page to make a snap judgement. My decisions on which boxes of documents to request were far more time consuming. What is the right keyword for talking about computers in government in 1970? If you said "information automation" then you would be right. A few presidential (Ford especially) libraries have updated electronic files for indexing which is a huge advantage.

On my trips to the archives, it was interesting to see both professionals and amateurs using a range of technologies. I saw really old school researchers using 3x5 note cards and taking notes on legal pads. They sometimes supplemented their work by photocopying really important documents at $.75/copy. Some researchers avoided this cost by using flat bed scanners which they carried in with them. Still other researchers brought in high end digital cameras and tripods. I used a digital camera freehanded. All of these people still need to find a way to actually get to physical proximity with the records. Digitalization would open up a new era in research.

On the metadata issue, most of these records already have copious amounts of metadata recorded in well-established fields that are used by NARA.

On the OCR issue, some documents have hand-written notes on them which would not be machine readable and sometimes are not human readable. It is likely that the documents will have to be digitally scanned and flagged if handwriting is detected.

Making these records available to the general public would be a huge advantage to anyone interested in government and US history. Come to think of it, in terms of size and complexity, it would be a worthy challenge for Google. U.S. government documents run back to the founding of the country and the number of documents only increases over time.

Re:In the Archival Trenches...

[Dachannien]

Remember "the warehouse" from the Indiana Jones movies? NARA is a little like that in terms of size but are better organized.

Does it play the music when you go in there? That's what really sets the mood, you know.

Re:In the Archival Trenches...

[Anonymous Coward]

I'm an amateur historian who has spent hundreds of hours in the National Archives researching, and sometimes copying, World War II unit records. When I started, almost 20 years ago, I quickly learned that one of the best ways to learn what a particular military unit was doing on a certain day during the war was to look at the map overlays that accompanied its daily reports. These overlays were big sheets of translucent onion skin paper that were placed over printed maps, with the locations of friendly units marked on them, along with their planned movements and the locations of known or suspected enemy units. These overlays were rolled up (or more often, folded up) and sent up to higher headquarters along with the typed-up reports that were required by regulations. Eventually both the typed pages and the overlays made it into the National Archives.

50+ years later, these onion skins (and many text records too, which were often typed on whatever paper was available at the front, much of it very low quality) have grown so brittle that many can no longer be unfolded or examined, since doing so would damage or destroy them. The copious and valuable information they hold is essentially lost. If they could be salvaged by qualified archivists and digitized, it would be an incredible boon to historians for generations to come.

Re:In the Archival Trenches...

[Anonymous Coward]

...some documents have hand-written notes on them which would not be machine readable and sometimes are not human readable. It is likely that the documents will have to be digitally scanned and flagged if handwriting is detected...

Have the LDS (Mormon) church handle the handwriting transcription. Their genealogical records (census, etc) records transcriptions are widely recognized in the field, and they are sure to have an extremely efficient (if not completely automated) method to handle tagging and metadata.

Agreed, sad it won't happen... unless Obama is...

[LostMyBeaver]

Let's be blunt... this is 2011. The task he set forth will be tied up in bureaucracy for a minimum of a year. There will be arguments such as "Where will we get the budget to do this?" and there will be arguments like "Who will do it?" and such. Even if the program gets started, the company who will provide the obviously custom system will underbid the others involved to land the contract and once the contract itself IS started, then whoever won the contract will then stop part way and claim "The agency misinformed us as to how much would need to be digitized and therefore we need more money." at which time the project will be placed on hold pending an audit to which time it will be made known that there was corruption involved in choosing the given vendor.

Agencies who have thus far opted to NOT digitize their records have done so for many reasons. And even though they're being forced to digitize now, they'll find many different methods of making the process cost substantially more than it should have and drag the process out over extended periods. Let us not forget that most of these documents can only be handled by certain staff with high enough clearance given their confidential nature. If the expose writers are to be trusted, there are entire rooms of records of paper where only one highly trusted person is allowed to enter.

Let us also point out that many of these records have been written in cursive which unlike block is a screaming nightmare to handle automatically. That means that the people who hold the clearance to view the records will need to manually enter these records themselves. There will be issues of encrypting the records so that only certain individuals will have access to them. While Obama would like to make it so that there could be some central database per organization, I'd imagine that there will be many individual, sealed networks to guarantee security.

With all these issues, let's be blunt...

1) The agencies will fight it... outright AND through bureaucratic means.
2) The agencies will say "Sure... we did it" and since many of the records are highly classified, no one can actually contradict the statement... so it most likely won't happen. When a given record is asked for they'll claim "oh...we must have missed that box"
3) It will take decades to complete as there are rooms of records where only a single individual is likely to have access and I'm guessing their typing speed isn't 100wpm.
4) Obama is on his way out. Even if he survives this coming election by some miracle (he sucks as much as the next guy, but people know he sucks and are more likely to trust someone else with less of a known suckage) by the time the project is likely to start, it's almost certain whoever takes over will pull the funds from that budget within hours of getting into office.
5) For data security sake, the agencies will most likely have to design the systems themselves using whatever crap engineers they manage to find with high enough clearance that's willing to actually code document management systems. And truthfully... this isn't a TV show... if the agencies have "Super Hackers" on staff, they're probably just as lame as the self promoting idiots you find everywhere else.

So, I'm willing to say... this will cost a tremendous amount to talk about... but will go nowhere. Sad :(

Re:Ye$! (But not so much in the real world)

[Required Snark]

Let's list some real world examples of "efficient" corporate behavior.

Enron

Lehman Bros.

BP Gulf Oil Spill

Exxon Valdez

Fukushima

Bhopal (Union Carbide)

AIG

WorldCom

Washington Mutual

General Motors

CIT Group

Not to mention all the "too big to fail" financial companies that got bailed out on the backs of the taxpayers. It was just revealed this week that the amount of assets back up by the US Treasury was about 77 Trillion $US.

Efficient Business

PS. You're a fucking racist slug.

firesale!

[Anonymous Coward]

better improve the self-sufficiency of my command centre. lets see... lightsaber; check...

Security?

[David-D2]

Open formats are great and all, but why isn't anyone asking about security?

Web 2.0?

[Anonymous Coward]

Does this mean the content will be driven by web users and we can write comments on each other's records and rate them? Wow... this is going to better than 4c**n!

Digitizing is great but keep most offline.

[Karmashock]

Who wants to bet they spend about .0 seconds figuring out how to secure this data? For the love of god, if we have no hard copies and these digital copies stand as good as the paper record... protect them. Keep the sensitive stuff offline. If someone wants the file, they can make a phone call, get authorization, and get it patched over.

Please tell me the government learned something from wikileaks.

Re:Buffoon Obama has no Money

[Anonymous Coward]

75 cents of every dollar of the "Bush" tax cuts go to people making less than $250K/year.

Hell, they're not even the "Bush" tax cuts any more. The originals were passed via reconciliation and were subject to a 10 year sunset. That sunset has passed, and the onus for the extension of said tax cuts is squarely on the current occupant of 1600 Pennsylvania Avenue.

Start with the classified documents

[raymansean]

If a commercial vendor can manage to write the required code in the time given, the budget given, and meet all interface demands from the various perspective users. There is still no way that certain TLA (three letter agencies) will let all their documents be indexed. Thus, the project is DOA.

Technology determines information solution? Dumb

[OldHawk777]

IMO
Objective: Information Determines Social Change and Technology Application.
Legacy: Technology Determines Social Change and Information Application.

Yes, a paradigm change. Decision makers (.com/.gov/.mil...) are legacy mind-locked on technology always defining and providing the "Information Technology" (IT) solution.

Yes, a paradigm change. Decision makers (.com/.gov/.mil...) must go to academia to help define the new "Information Management" (IM) market place. IM must determine the required IT architecture. Fitting information into an IT solution is always foolish/simple. Fitting an IT solution to information requirements is presently dismissed by decision makers (.com/.gov/.mil...) as foolish/simple; Hence, money, failure, and time wasted.

Technology sector marketing can hook most decision makers (.com/.gov/.mil...) their bait is still seen as the food for career success. Technology sector marketing ever knew IT and are totally unfamiliar with IM.

Re:the fix is in

[TheCarp]

Well it is a fictionaly story based on observations of actual stories. Clearly it represents an exaggerated world, but one that is so believable when we look at our own. Oceania wasn't just a place with cameras and torture with rats. Look at Wilson's own job, sitting at a desk, taking in little scraps of paper, processing them, and then disposing of them. It was very much about control of information flow, control of the historical record.

The masses knew the official story was BS, people know it today. And by know it today, I don't just mean Fox News. I mean the official story. Many comments on the embassy cables that were released basically just confirmed what everybody already knew. Sure, a lot of specifics came out, and some of those were significant and made a difference. However, was anybody shocked at the kinds of things that were found? Nobody that I have heard comment on them. Hell a lot of people believe much wilder and crazier versions of reality than all that! Some even believe fox news.... scary.... and some people really did love Big Brother.

Going back though, I think it has always been true. There were not genuine supports of BB or even masses knowing they were lied to in 1984 because Orwell just dreamed it up from his imagination, this is how it always is.

NARA? Digital?

[oneiros27]

So I was at a data.gov meeting in the spring, and got to talking to someone from NARA ... he said their digital archive was um ... I can't remember the exact size, but I want to say it could all fit on a single disk, so given the time, 2TB or less.

Some of the government agencies have PB of storage already ... we'd love to turn it over to NARA for long term archiving, but there's no procedures in place, and I don't think they currently have the infrastructure or personnel to deal with it.

(note, I'm taking a broad definition of 'record' here; I help to manage an archive of solar physics images; our discipline's data is growing at a raw, compressed rate of ~1.5TB/day due to SDO ... some of the earth science groups have multiple satellites with those sorts of rates).

I'd say bring in the group from PDS (planetary data system) who built OODT, which is now an Apache project, the iRODS folks from UNC, the folks who did LOCKSS, and all of the other large-scale distributed data networks, and have them discuss the benefits / flaws in each one, and come up with a good solution. If they pass this off to yet another standard government IT vendor, it's going to blow up on them again.

Re:Agreed, sad it won't happen... unless Obama is.

[datavirtue]

"these records have been written in cursive which unlike block is a screaming nightmare to handle automatically."

Bull, the post office has been processing chicken scratch (in milliseconds) since the 80's. IBM has hand writing recognition solutions that would blow your little mind.

Good Move

[lunatic1969]

We did this at our office some time back. There's more to it than you might think, and I wish we'd done it sooner. First, the cost savings is pretty significant. You've no idea how much paper, files, file cabinets, and sheer storage space for all this paper that's involved until you don't have to use it anymore. Add to that the labor cost of constantly running somewhere to hunt down a paper file, or the labor cost of having someone file away a stack of papers into that paper file. It really is pretty significant if you're in an office type environment that creates paperwork. The problem is going from a hard copy environment to a soft copy environment. What do you do with all your existing hard copies? What mechanisms or hardware do you use for going from hard copy to soft copy? We opted to implement our change on a going-forward basis. Basically as of a certain date all future paperwork would be soft copy. The idea being that (at least in our case) eventually the hard copy files would age into being obsolete and destroyed. There's other issues. What kind of a system do you use to store it? Do you run your own server solution? Do you farm it out to a cloud-type solution? In our case, there was excellent proprietary management software geared to our agency, but what happens if that company goes under, or is sold? All in all, it's an excellent idea but the solution isn't as simple as one might expect.

Laserfiche

[hesaigo999ca]

I heard that Laserfiche is a great tool for document management. As it stands they are on the fore front of the anti-piracy movement, and seem to have a stable version to avoid security issues. Maybe this is what they need?

Re:Laserfiche

[neminem]

Hey! I work there, I totally wasn't expecting to see someone else mention it. Glad to see people talking about it (either that or you're a shill and actually also work there, in which case, I'm totally -not- posting this from work. :p)

It was certainly my first thought when I saw a thread with this title, whether a mandate like that would increase interest in our software. But you would expect me to think something like that, given that I work there (even if I am several layers removed from people actually purchasing the software, and quite happy about that.)

Re:Laserfiche

[hesaigo999ca]

Well it is a great product and a bit underrated, but they are coming along for sure....also the potential for improvement on the overall new functionalities keeps me very excited

Can You OCR? Do you know XML? Are you a minority?

[ProppaT]

If so, I suggest creating your own business and get ready to bid on some work. No one is going to do this in house, they're going to take bids on conversions. I used to work at a company that made quite a bit of money off of paying people, per page, to OCR patents, correct OCR errors, and tag the document in XML. And I can assure you that, because of the way the government works, the majority of the work will go to minority owned small business. The work is easy and you can get college kids to do it for peanuts.

I'd swear I've been in this guy's mtgs

[jacksdl]

Seems to be the parent's objections are a classic case of "perfect being the enemy of the good." Of course the metadata definitions won't be complete enough when these documents are scanned. This leaves the user with having to digitally search through the records (if they OCR'ed decently). How is this not an order of magnitude better than going through the paper?

Re:I'd swear I've been in this guy's mtgs

[kiwimate]

You do make a valid point, but here are my counterpoints.

One - a paper library is hopefully categorized and indexed properly. If you go to an old-fashioned book library, everything is arranged according to its appropriate category, filed according to the Dewey decimal system. It's easy - you look up the number in a card catalog (or the electronic equivalent) and it tells you the number is 378.143. You go to the appropriate section of the library and there it is. Why? Because a librarian took that book when it first came into the library and filed it properly.

A DM system doesn't work that way. Everything starts off as just a bunch of documents in a flat system. The only way to distinguish and categorize them is by metadata.

Which leads to...

Two - you're moving from a structured filing system (almost certainly under the control of a librarian) to an unstructured filing system. Leaving aside the instantaneous confusion, remember it's going to grow in size over the next several years. Not only will the quantity of documents increase, but even if the initial import was well structured and tagged you're now most likely implementing a system where anyone can insert a new document (not a new record, hopefully - that had better still be under the control of a librarian or you're facing compliance nightmares). I really don't want to sound elitist, but librarians know how to categorize and file things in an organized fashion; users often don't, and even if they do there's a good chance Joe is filing documents according to Joe's categorization, and Marsha is filing documents according to Marsha's categorization, and Sandy is...etc. Unless you take the time to very clearly define those categorizations/metadata up front.

Re:I'd swear I've been in this guy's mtgs

[Anonymous Coward]

Seems to be the parent's objections are a classic case of "perfect being the enemy of the good." Of course the metadata definitions won't be complete enough when these documents are scanned.

Unfortunately, the system is far more complicated than that. The metadata isn't just there to improve search, it contains a whole lot of information dealing with the schedules, approvals, verification, certification, and custody decisions of the material.

When all is said and done, NARA accepts legal responsibility for this content. If something isn't perfect, NARA can end up in a lot of trouble with the GAO. Keep in mind that these records can be used as legal evidence, so accuracy and accountability are very strict requirements.

Even the current paper archives aren't just stacked in a room with a card catalog like an old public library. Each item has been several NARA forms associated with it. This process and information must exist in the digital archive as well.

Simple Records Last, Complex records Not so much

[Anonymous Coward]

We're able to see and use census records from the 1800s today because they used simple recording methods.

I only have a few concerns about electronic records.
* Illegal uses
* Long term migration of data
* Access 30 ... 100 ... 500 yrs later when the computers and programs that created the records are not longer used.

I've deployed electronic record conversions for 2 Fortune 50 companies. Obviously, these were just departmental, not corporate-wide. Start with the paper forms and keep it simple. 3rd normal form for DBMS is your enemy.

Paper is the best long-term storage method available today. It is possible to keep the most important data in a few tables, then export those as CSV and store them to paper in digital format. It is possible to create the simple table output from 3rd normal form, but that needs to be ensured.

Being digital is important, but losing the ability to easily access the information in future generations is also a concern, especially for government documents.

Not Quite

[Anonymous Coward]

Earlier this year, due to cost overruns and project mismanagement, NARA announced it was ending a 10-year effort to create an electronic records archive

Ummm no.... source please?

The truth is that NARA's Electronic Records Archive was not scrapped. This year, the final increment of development came to an end (on schedule), the system was formally accepted (on schedule), and the program entered the O&M phase (on schedule).

NARA wasn't "ending a 10-year effort", a 10-year effort was coming to an end.

What about medium lifespans, EMP protection, etc

[tomweeks]

I hadn't heard anyone discuss the limited lifespan of optical and other eDoc mediums. They have very short/finite life spans (5-20 yrs) and have to be remastered or transferred off to another medium. Also.. would such a system be "safe" from technology attack like EMP, etc? Paper is cheap, and good for a couple hundred years.

Tweeks

Music

[jones_supa]

Reading the title, I for a microsecond thought this was some kind of culture preservation project, digitizing all the music on analog media...

Re:Agreed, sad it won't happen... unless Obama is.

[Teancum]

While there may be some agencies that will try that "highly classified" BS story, there are inspectors and people who have security clearance which can go in and verify that even the classified documents are archived in a responsible manner. Some of those inspectors answer only to members of congress (usually something like the CBO or perhaps accountants/inspectors tied to specific committees) and are fully cleared to view any classified material as their need to know is usually within the scope of their official duties with oversight.

So yes, there are "3rd parties" that can contradict whenever somebody says "sure... we did". And if they claim compliance and it hasn't happened, those folks will find their ass nailed to the wall or possibly find themselves in prison for making a false statements like that when it isn't true.

Keep in mind here that the need to store classified materials may be made in various means, including complete secondary networks (physical layer separation on the OSI model, not mere VPN separation) or even computers "off grid" that only use SneakerNet when data needs to be shared between computers with couriers... and a stack of protocols for sharing that information that would make your head spin.

American Indian Records Repository

[Tekfactory]

I used to work on a project that was colocated with the AIRR, we imaged millions of pages, created JPEG images for people to use in our application due to network speed issues and the application we used for users to make notes on the documents wouldn't work with the TIFF images we created for NARA.

We were auditing over a hundred years of accounting for how the US government dealt with Native Americans and had Historians and Accountants both working on the program. We had our own little Indiana Jones warehouse.

Several times our leadership recommended we just scan whole boxes and digitize them, due to budget restrictions we had to pull and digitize relevant information (thousands of documents, millions of pages).

I also was part of a group outing when we delivered a TB or so worth of our records to College Park, fun times.

FOIA headache.

[Anonymous Coward]

Once everything is "digitized" then Big Brother can much more easily search for and retrieve any documents that are requested via FOIA.... almost as easily as Big Brother can then digitally "adjust" the contents of those very documents to make them say whatever Big Brother wants them to say, depending on whoever is requesting them, of course.

Re:FOIA headache.

[Jeremiah Cornelius]

"...within easy reach of Winston's arm, a large oblong slit protected by a wire grating. This last was for the disposal of waste paper. Similar slits existed in thousands or tens of thousands throughout the building, not only in every room but at short intervals in every corridor. For some reason they were nicknamed memory holes. When one knew that any document was due for destruction, or even when one saw a scrap of waste paper lying about, it was an automatic action to lift the flap of the nearest memory hole and drop it in, whereupon it would be whirled away on a current of warm air to the enormous furnaces which were hidden somewhere in the recesses of the building."

Good idea, extremely expensive

[jweller13]

I work for a federal agency trying to implement this. It is a wonderful idea with many benefits but it is very expensive to implement. It's not just a matter of scanning documents. The scans have to be verified error free and a lot of meta data has to be manually input on the document. Mandates like this are so often passed down with out giving the agencies the resources needed to carry them out. So we so often end up getting half assed implementations.

Re:What about medium lifespans, EMP protection, et

[Anonymous Coward]

Paper fades/discolors, is subject to flood damage, and burns at the slightest spark - which weapons far simpler than EMP can readily provide. You can't attach a search engine to paper. Storage in large quantities also becomes an issue.

Disk storage data can be accessed quickly and indexed for optimal search. Most SANs have RAID5 built in and fire off a variety of bells and whistles when a drive goes bad, so lifespan isn't really an issue. Futhermore, data can easily be copied across several geographic locations (some of them buried deep underground) to pretty much eliminate any threat of EMP or natural disasters.

As a last point, you are asking the /. community to argue the use of paper over technology... you would have better luck going on the Top Gear website and asking people if they would prefer a bicycle over a sports car =P

consultantants

[Anonymous Coward]

Deliotte and Accenture were heard celebrating!

It's not about technology, it's about policy

[Anonymous Coward]

What you decide to digitise it a policy choice not a technology one.
Whenever some analogue-world artifact is digitised - and the original artifact is destroyed - there is a danger of information loss.
Examples: what paper stock was used? What hand-written annotations are made? by whom? when? says who? what does the paper smell of? (think I'm kidding? check out this post ); has the digital copy been modified since original analogue information was captured?
See also "In praise of paper" to understand more of the arguments - it's not a simple technology issue, ever... see 'In Prasie of Paper'

LOL

[TheSkepticalOptimist]

For years they thought digitize meant feeding it into machines that turn it into tiny bits....of paper.

Best effort

[PeterWone]

So, unable to achieve godlike perfection, we ought to do nothing? Besides, why is this an XOR thing? A high degree of precision is easy in dimensions that are curated (DM'd) and Google performs extremely well at searching in dimensions that weren't anticipated. In any given domain it is reasonable to expect the people currently filing bits of paper to know how documents need to be tagged, simply because they already do the curation. In many cases I fail to see any value in transcribing records; apart from matters of ownership, contract or engineering, bureacratic records are best forgotten.