Slashdot Mirror
Researcher Warns of "Digital Dark Age"
alphadogg writes "A assistant professor from the University of Illinois at Urbana-Champaign is sounding a warning that companies, the government and researchers need to come up with a plan for preserving our increasingly digitized data in light of shifting document management and other software platforms (think WordPerfect and floppy disks). Jerome P. McDonough, who teaches at the Graduate School of Library and Information Science at the University of Illinois at Urbana-Champaign, says there exists about 369 exabytes worth of data, and that includes some pretty hard to replace stuff, including tax files, email and photos. Open standards could play a key role in any preservation effort, he says. 'If we can't keep today's information alive for future generations, we will lose a lot of our culture,' McDonough said. Even over the course of 10 years, you can have a rapid enough evolution in the ways people store digital information and the programs they use to access it that file formats can fall out of date.'"
In 100 years, I won't care.
It's only because people are so anal these days. Who gives a shit? It's not like anyone in the future's going to miss anything. Even today with items like the Rosetta stone it's not worth much more than a Trivial Pursuit question - we'd not be any more educated or intelligent if stuff from 2000 years ago hadn't gone missing. Sure, there's a certain entertainment value in it all, but the idea that in 2000 years time anyone's going to be remotely bothered about the loss of websites, games and so on from the late 20th century is just ridiculous.
"I often ask, 'Everyone in the audience who thinks they're going to be using the same word processor in ten years, raise your hand.' No hands go up. 'Everyone who has data around that's going to have value in ten years?' After a minute's thought, every hand goes up. The lesson is clear: information outlives technology."
- Tim Bray
Parity: What to do when the weekend comes.
The cultural loss isn't something that should be overlooked, some can bemoan it but the value of culture is that it exists, and that different ones existed in the past. Culture changes from moment to moment but without some action the real meat of the early 21st century will be lost forever. That is the big thing here, and that is justification for working for truly readable digital archival methods. There is a project of making minisuce indentations, but that requires a lot of technology to see much less decode. Continuous duplication, by transfer of all old data across all mediums as they rise and fall, by printing content and storing it in climate regulated warehouses, etc. We relish seeing things from thousands of years ago. This is humanity, that is our legacy. We need to leave a legacy for our grandchildren.
The only motivation for a company to invent new ways to preserve data long term is to provide it as a service so they can profit from it. Other than that, a companies main goals are deleting everything it legally can. Anything that no longer exists can't result in a lawsuit.
Everything that is preserved is a potential liability. For items requiring indefinite retention because they are critical to the business... They will be stored, redundant, and backed up appropriately. As the systems that provide those qualities age, they will be replaced in regular maintenance and upgrade schedules as economics and timing come together in the right proportions. In that way, reliability and long-term survivability are maintained - nothing stays on ancient systems that are unmaintainable forever. When systems go out of support, everybody has already been looking to the next solution to migrate to.
So what's wrong with this approach? Its essentially what all "big" companies are currently doing. I don't believe in this proprietary format FUD either - if the proprietary format is no longer supported, you migrate. Potential of future cost to migrate is the only concern, not survivability.
Migration is todays solution to long term storage and I see no reason it should be ignored. Like security, data retention is an ongoing objective that requires maintenance - its not some end-state. Dreaming of a solution that will just last forever seems archaic, no?
Overclockers
OPEN file formats and OPEN hardware, well documented.
Even if no program exists anymore to read your data, as long as you have the specs you can rebuild it. And I mean hard- AND software. If you know how to build it, you can build it provided you have the means. And I'm pretty confident that our future cousins will be able to build a current computer with their future technology, as long as they know WHAT they should build.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Most of the text in most word processing documents are easily available to be parsed out even without the specs. The formatting would be lost, as would any embedded objects or images.
Open formats would improve it, but I would be more concerned about encrypted documents and media loss than not being able to recover data (text/images/video/music/etc) from available files. There are a lot of clever people that can do amazing things with deciphering proprietary formats.
I rarely read replies, it's my opinion and if you thought about your opinion a little more, I'm OK with that.
Amazing as it sounds, I still have very VERY old data that goes as far back as 7th grade when I started using computers. I know of no converter for Professional Write that will convert Professional Write documents into ODF, or even MS Word 97/2000/2003.
The only hope I have is that I can use strings to extract the text elements of the data.
From the article -
âoeIf we canâ(TM)t keep todayâ(TM)s information alive for future generations,â McDonough said, âoewe will lose a lot of our culture.â
Hardly.
Apparently none of our culture is stored in books anymore?
Sure if every piece of data was wiped out the world would lose a lot of information... but a lot of valuable and useful information is still put on paper. I don't think that is our biggest cause for concern.
However I do agree that the world really needs to agree on more open / non-proprietary ways of storing data. Sure, I can open a .wav of Blackadder talking about 'sticking a Christmas tree' somewhere from 1992, but I have a bit of trouble opening .ra (real audio) video files from a few years ago.
And working in government everywhere I go the electronics file storage is just a discordant mess. Anything important we have to print and store hardcopies because our electronic systems are just unreliable.
Everyone thinks of changing the world, but no one thinks of changing himself. -- Leo Tolstoy
And I'm not immune of course, there's a lot of shitty software out there and it's easy to trivialise the value of Custers Revenge or Giana Sisters but remember that historically archivists want to know about tasteless/racist video games or tributes/Mario-ripoffs just like they want to know about 1980s comedy shows and magazines.
This article is saying that libraries and archivists had a blind-spot when it came to software. It took them decades to realise that people expressed themselves artistically in this medium. Archivists didn't know that they should preserve it like we do other media.
I know how easy it is to mock these efforts (Eg, the tag "!nothingofvaluewaslost") but please consider supporting and justifying this digital culture as part of a wider effort to justify software expression.
It's easy to pick out dumb software but closing
Open standards could play a key role in any preservation effort, he says
The way I see it there are two approaches to the problem. The Quixotic fight consisting in changing the world and forcing in a dictatorship of openness regarding file formats, which doesn't solve the problem for the past 50 years of computer history.
Or let a few hundred people around the world worry about file format parsing or, in the worst case, even emulators to do whatever old computers did. In a hundred years from now, you'll have very complete emulators for our modern PCs. Considered that a 1994 PC is quite comparable to a 2008 PC (and presumably a 2015 PC) from an emulation point of view, you know that's a given, and even then, in case there was no such emulator, you know you could find a good such emulator for machines from the 2040s, which themselves would be well emulated by machines from the 2070s, and so on.. that's what we already do. There's hardly any program you used 20 or 30 years ago that you couldn't use today.
You just got troll'd!
Most of the garbage that we have now just isn't worth keeping. The biggest problem is filtering out the junk we have so that we know what is really valuable. That would be things like great music; writing; the origins of software freedom; works of history and biography etc. Then we could store that, but the problem is we mostly store SOX inspired lies for compliance audits. This garbage takes away from any effort to store serious stuff long term. Who could we trust to do the filtering? The govt? (no please don't answer that :-)
=~ s,(.*),<sarcasm>$1</sarcasm>,g if any_point_you_wish();
...the more they stay the same. Here's something I posted back in 2006 about this same issue: http://slashdot.org/comments.pl?sid=207582&cid=16922754
Quantum mechanics: the dreams that stuff is made of.
Historically, things that have been very uninteresting at the time, have been hugely valuable to researchers later on. We may not care about the countless people talking "crap" on bebo right now, but in a few hundred years it might be a different story. When people can easily analyse all those posts for meaningful psychological profiles that aren't currently understood never mind modelled and easily detected, all of that could tell a lot about our society. Even rubbish tips from thousands of years ago are hugely valuable to paleontologists.
This goes more so, for important government records, etc. Peter Quinn did a great job of explaining that, with his Sovereignty talk.
Except you can explain what a painting is, no one can clearly define what the cloud is. Mostly becasue it's a marketing term looking for a technical design it can adhere to.
The Kruger Dunning explains most post on
Try loading up an image form Dr. Halo. That was a pretty popular paint program on the PC back in the day. Depending upon your perspective, it wasn't that long ago, 20 years. I think that the format has been published. Maybe it's skewed because there isn't that much really desirable data in the Dr. Halo format but it was a pretty popular toy.
Even if the format is published (which I believe it is to some extent) it's a bit of a chore to go write a decoder. Go back a few more years, say pull some PDP11 files or EBCIDIC files, it's not impossible by any stretch to decode them but the benchmark goes up just a touch more. EBCIDIC may never die simply because of the size and might of IBM, but PDP11? It doesn't seem that hard to imagine a world in a few more years where people don't really know what middle endian is. (not that they'll forget exactly but It just makes the chore that bit more complicated. How often to you go back and reverse engineer a video game for the C64 of original PC or Apple II? How come nobody is modding them? (maybe there are a few folks doing it but it's not like it's a really popular hobby) It's not a matter of possibility so much as the benchmark to accessing the data, it cuts off the common person who wants to casually look at history. Imagine if we make a fairly radical change in the way we process data in the next 50 years, this isn't a stretch, maybe organic computing or quantum computing where some of the "fundamentals" as we know them change. Maybe in 100 years, binary data on drives will look like punch cards look today, how often have to captured data off of a punch card on a modern computer?
Open specs are part of it, coming up with some intelligent ways to develop more timeless document formats without over engineering the hell out of them is part of it too. Maybe the most diligent thing to do would be to contrive a format for that and as part of the specification as it grows over time migration is included in it. That's one thing no format really "takes care of" for you, if all your documents are in Word it's all well and good but what you want is every time a new version comes out, you want to freshen all your docs to the new format and maybe resave them without the legacy stuff. Try doing that to all your jpeg digital photos. The data either has to be kept alive as part of the specification or something more intelligent has to happen. Fast forward say 100 years, your great great great grand son finds a way to pull our ODT Resume off of a DVD, what's the liklihood of him building an ODT viewer to crack open the data? Even with the specs, that's a somewhat involved task.
Make copyright last 5 years. Then everything worthwhile will be backed up by someone who cares about it.
Everyone here seems to be missing the point -- Businesses don't need help preserving data. Anything that's really valuable and needs to be preserved will eventually be put on a laptop and lost in an airport. But what about your wedding photos? What about that book you've worked for three years on, and saved it in word doc format?
The problem of data preservation is not one business needs to address -- there's a million geeks (hi slashdot) that will be eager to earn their pay coming up with washing-machine sized solutions for business, in black cases with a stylish logo on the front. But what about me -- the person who makes less than $30k a year, keeps all my files on a laptop and an external drive, and doesn't have a lot of cash?
Let's say I want to put it in a safety deposit box and forget about it for 10 years? 20? 50? What are my options for preserving photos, videos, and text cheaply? And by cheaply, let me say less than a grand, since "cheap" seems to be relative here.
This is one of those fairly bogus, highly overblown stories that keeps cropping up every so often. A similar one is the supposed shortage of scientists and engineers in the US, which has never existed, and is always supposed to be coming Real Soon Now; in fact, the data to support this claim are always either nonexistent or wrong. (E.g., they compare Indian college graduates with US college graduates, but the Indian degree they're comparing with a U.S. bachelor's is more equivalent to an AA degree in the U.S.)
First off, the concern about incompatibility of physical media was valid 30 years ago, but it's a false analogy to try to apply it to today's situation. Thirty years ago, I had data on a mixture of 8-inch floppies and 9-track tapes. I can't read an 8-inch floppy anymore, and although 9-track tapes still exist, most 9-tracks from that era are no longer readable due to physical deterioration of the media. But that was all in an era when hard disks were expensive, and the internet didn't exist. Today, I have all my data on hard disks of various computers, and I use file synchronization software to keep them all in sync. If one of my hard disks dies, I replace it, and I haven't lost any of my data. (I also have backups on optical media, but I basically never need those.)
There's also the concern about formats. People tend to bring up, for example, the image of rooms full of physically deteriorating 9-track tapes with data from old NASA space probe missions. The formats are often not documented. The thing is, most of our data isn't at all analogous to the raw data from Mariner or Voyager or Viking. Those were unique historical events, and the only way to get more data like the data they collected is by sending another space probe. (People also tend to vastly overestimate the value of scientific raw data. It's extremely uncommon for raw data to be of interest decades later.)
Most of the world's data isn't in some obscure NASA format, it's stored in formats that are used by tons of people, and are extremely well documented. Sorry, but I just don't believe that the knowledge of how to decode Adobe Acrobat format is going to be lost to future generations. Ditto for html, jpeg, and mp3.
Another thing to keep in mind is that nowadays you can emulate old computers with excellent performance. For instance, my first home computer was a TRS-80. I can still run my old TRS-80 games on my linux box, using an emulator. Sure, emulation isn't perfect, and some information may be lost. But the claimed threat of data loss is vastly overblown.
The biggest threat to the preservation of information isn't technological change, it's copyright. The most likely reason that I wouldn't be able to get back an old piece of digital data is that the people who tried to preserve it and put it on the web got sued by the people who own the copyright -- the same people who let it go out of print. The economic incentives are to hold on to your copyrights (because that doesn't cost you any money) and send out DMCA notices to anyone who puts it on the net (because that doesn't cost you any money either), all in the hope that your content will be worth eleven cents fifty years from now. This is exactly what we see happening, for instance, with ROMs for old video games, which you can play in MAME, except that you have to find an illegal source for the data, because the owners of the copyrights aren't willing to sell you a copy.
Find free books.
this means you have to bypass access keys or encryption
This is going to be a big problem. I have CAD files, code manuals and other engineering data that cannot be accessed with anything other than the proprietary CAD apps or browsing software. Some of these apps have been 'orphaned', in that the applicable versions are no longer supported by the vendors. Activation keys are locked to a particular machine, so trading in that Windows 98 machine for a nice new XP system is out of the question.
I make sure that none of my contracts oblige me to maintain electronic versions of deliverables or that any delivered to the clients will be accessible beyond the completion of the contract. Its rolls of blueprints or nothing.
Have gnu, will travel.
"Dark Age" is kind of an exaggeration. Presumably it's a reference to the period right after the Fall of Rome (475 AD) when most classical literature was lost because existing information technology (hand-transcription of documents) got too expensive for what passed for an economic system. This time around, if we lose much more, it's because we have a lot more to lose. But how much of it matters? If my USB drive dies and takes the last surviving copy of Debeee Does Dingos or the collected bloggings of Joey Joey, it's not that big a deal. But anything that really matters (the complete works of Shakespeare, the Beatles, the user's manual for Ultima IV) is going to be saved in multiple places in multiple formats, and it just not going to get lost.
I think the big problem is the exact opposite of what TFA warns about: too much preservation of stuff that isn't worth preserving and doesn't really represent our culture. Future generations wading through the digital crap we leave behind — blog rants, porn, advertising, spam, internet rumors, Star Trek flame wars and fan fiction — will be hard put to sift out our serious accomplishments.
Classical Greek civilization is probably the most influential in all of human history. And yet you can buy a single CD containing every single surviving work from the entire civilization! It's quality, not quantity, that defines a cultural heritage
My concern is how the components depicted on the schematic are going to be made out of rocks and nuts and berries, even if the schematic is readable.
There is a thick complex web of technologies needed to replicate said drive.
And I say this as somebody who has the complete Technical Reference information for the IBM PC. (that means I have the schematic diagram of the 10mb hard drive, and the schematic diagram of the hard disk controller, along with the source code for the BIOS extension on said controller)
I do agree that we will see some 'cultural disintegration', but not for the reasons cited in the article (which I, of course did not read). The reasons? New media models that require "monthly access fees" (yes Blizzard, Sony, I'm looking at you), and DRM protected media. Sure, some companies will 'do the right thing' and open their media to the public once they are not actively using it as a revenue source, but they will be in the tiny minority. My kids will probably never be able to dust off the World of Warcraft DVD, insert it into their holo-reader and find out what our generation did for fun. Likewise with the millions of songs that are stored precariously on iPods throughout the world. Once the iPod breaks, and the iTunes servers are switched off for the last time, that music is lost forever to the people who loved it dearly, but were foolish enough to accept a 'limited rights' version of their media. Looking back, we can still enjoy art from the entire history of humanity - cave paintings, books, canvas and sheet music, just to name a few. Apart from the physical disintegration of the medium, little can destroy these expressions of our culture. With our new encoded, protected and limited DRM-riddled media, there will be very little to look back on from an individuals point of view. I expect that organizations will spring up to restore these lost works of art, and efforts will be made to make our current culture accessible in fifty or a hundred years. But where does that leave the young kid who finds the suitcase full of DVD's, or Blue-Ray discs in his attic, left to him by his grandfather? Will he or she be able to take a glimpse into history, in the way that our generation has been able to dust off the old vinyl record player, and reverently remove that piece of vinyl from its weathered cardboard cover, to listen to a crackly rendition of Muddy Water's 'Baby Please Don't Go' I doubt it.
Garbage isn't the problem.. the problem is that we have millions of copies of the same data. Think of the 50gb of video games you may have installed.. 10 million people have the same games as you. Music? Unless you performed it yourself or it's sub-underground, chances are millions of people each have multiple copies of it. The anime you've torrented has 10,000 downloads. .
No, see.. actually I'm just keeping a back up for the RIAA in case they lose their copy. PLus I keep it all transcoded to the next generation formats at no charge. And on top of that it's forward deployed for easy re-distribution without bottlenecking their servers. I even paythe lectric bill on the disks and internet connection. So copies are a good thing.
Some drink at the fountain of knowledge. Others just gargle.
It's a Digital Supernova Age and they're bitching. Think of your parents' or grandparents' generation, and try figuring out how much information exists about them. Sure there's the basics like birth certificates, marriage certificates, property records and other big things, there's probably some pictures and maybe they're mentioned in some books but I doubt there's any real record of how their daily life was and what they were doing. I know I have chat logs and such from my youth that are probably way, way more accurate and uncensored records than anything my parents have, even if they kept a diary which they didn't. If I get over how immature I was at the time, that's easily something I could release for research in 50 years time. With blogs and myspace and twitter and facebook and whatnot you can do a lot more, in a lot more detail with pictures and whatnot today and capture a large part of that as it happens.
The only thing happening here is that a few historians look at all this trivia which was always there, but never in a form to be captured and go "We should preserve ALL of it!" in a historygasm. If you preserved 0.001% you'd still preserve more than any generation of humanity to date. It's a case of diminishing returns, we don't truly need 24/7 live footage of 8 billion people as an historical record. It's certainly important to catch some sample of daily life and not just the big historical events and mainstream media, but I have no doubt that more than enough of this will be preserved anyway. Maybe we're in deep shit if humanity nukes itself out of existance but otherwise I'm sure it'll be kept as collectables or antique information from hundreds of years ago. Can you imagine in 2544 saying "It's a original (=bit exact) 2008 CD by [Artist]"? That's not going away no matter how crappy it is. And if we do nuke ourselves out of existance, I'm not REALLY concerned with what alien archeologists think of us anyway.
Live today, because you never know what tomorrow brings
Can I ask why you would want to restrict the ability of two consenting adults to enter into a contract together? It seems rather ridiculous.
You should be able to enter into a contract to share benefits with whomever you wish, X- or Y-chromosomes, I'm not sure I understand the difference.
It's called "Civil Unions" and in CA they have the same rights as marriages. The point is that "Marriage" would be not be used to describe these Unions in the same way as "heterosexual" would not be used to describe a homosexual person -- it's simply counter to the definition. It's not based on hatred or hope for inequality -- simply concern for a word that would quickly lose 100% of its meaning if we start tampering with the definition. Or do you go around calling homosexuals "straight" because it's bigotry to only call heterosexuals straight?
... to share benefits" -- but it's starting to.
And if you don't understand the fundamental difference -- biologically, socially, or purposefully, it's because the word "marriage" doesn't simply mean "two consenting adults to enter into a contract together
Mod -1 Offtopic
I am the richest astronaut ever to win the superbowl.