Avoiding a Digital Dark Age

al0ha writes to recommend a worthwhile piece up at American Scientist on the problems of archiving and data preservation in an age where all data are stored digitally. "It seems unavoidable that most of the data in our future will be digital, so it behooves us to understand how to manage and preserve digital data so we can avoid what some have called the 'digital dark age.' This is the idea — or fear! — that if we cannot learn to explicitly save our digital data, we will lose that data and, with it, the record that future generations might use to remember and understand us. ... Unlike the many venerable institutions that have for centuries refined their techniques for preserving analog data on clay, stone, ceramic or paper, we have no corresponding reservoir of historical wisdom to teach us how to save our digital data. That does not mean there is nothing to learn from the past, only that we must work a little harder to find it."

Comments

Won't matter

[countertrolling]

Our landfills will provide all the info they need.

Re:Won't matter

[Tubal-Cain]

Newspaper is not exactly the most durable parchment we've ever used.

Re:Won't matter

[Anonymous Coward]

But what my digital pictures of my family? It's kinda cool to dust off the old photo albums and check out my family history. Plus I can't imagine how P.Oed my Mom would be if I lost all the family pics (digitized from slides before the slides fade away to nothing).

Every time I upgrade hw or sw, no matter how careful I am, I seem to lose a file or two. I've taken to planting a couple of cheap NAS at my friends place and we all share the cost and benefits of off-site storage.

Re:Won't matter

[rubycodez]

right on, and for that matter it is silly to say we don't have paper records any more. We have even more of them than ever before. Receipts, leases, mortgages, contracts, invoices, manifests, packing slips, explanation of benefits (EOB), licenses, warranties, guarantees, manuals....fuck, if anyone thinks digital age means less just order a single piece of software on Amazon and by the you take everything out of the box you'll have generated at least eight items on the list I just mentioned. God damn!

Re:Won't matter

[biryokumaru]

Tubal-Cain (1289912)

You know, I said that to a judge once and he threw out my case and went to go paddle himself...

Re:Won't matter

[HBoar]

And just as well too. Most journalism these days is best forgotten.

Re:Won't matter

[Third Position]

Our landfills will provide all the info they need.

Well, I'm not entirely sure of that. If you pick up a stone or a paper with characters on it, you at least have an idea what it's purpose was. But 5000 years from now, how does someone interpret a shiny little disk? It might be a long, long time before someone is able to discern it's purpose, let along figure out how it's encoded and how to un-encode it. And that's even before getting a look at the language, and learning how to translate that.

That's one advantage of paper, stone and parchment - they don't assume a technical infrastructure in order to use them.

I have heard that some of the braided ropes left by Mayans might actually be a "written" language. But consider that it's taken us over 500 years to suspect these braids are a form of media, let alone learned to read it, and you can imagine what a future civilization might be confronting trying to figure out our digital media.

Re:Won't matter

[JustOK]

They wouldn't be able to use that stuff because of copyrights and DRM

Re:Won't matter

[y4ku]

That reminds me just a little bit of Wall-E. Here's to hoping it never comes to that. I've spent plenty of time working in archives for a museum in Chicago and this is an interesting point that I've never really thought of. Here I am spending hours a day archiving manuscripts and scanning letters and signatures into digital to have for the future and safe-keeping but who will upkeep it in the future. Looking into it, this doesn't seem like that big of an issue. Digital media is so easy to transfer and copy at such high speeds... Its almost a wonder how we manage to know so much about those before us. Especially the way they used to transfer information (in candle-lit monasteries on flimsy paper). The way things are now, the future will know way more about us than we will ever know about those before us.

Re:Won't matter

[Smallpond]

newspaper buried in a landfill will easily outlast unmaintained digital data. I'll send you some 8" floppies if you don't believe me.

Re:Won't matter

[Anonymous Coward]

"(in candle-lit monasteries on flimsy paper)"

If you really think so,

"I've spent plenty of time working in archives for a museum in Chicago"

You didn't take too much benefit from the time spent on it.

Re:Won't matter

[Redlazer]

While I agree to some extent, an advanced culture is advanced not just with it's technology, but also in the way it thinks.

I would suspect that, in the future, our ability to understand and figure things out will be far higher than it is today. Especially since the question of what a DVD is for is clear - not just to us now, but I would imagine even to someone who had no preconceived notion would be able to piece together the clues into what it might have been used for.

A reflective on one side, perfectly round disc? Looking at it under a microscope would no doubt show the presence of the "peaks and valleys" of digital data, and I think it would shortly fall into place.

Of course, a thought experiment such as this is nearly impossible to do, as I don't know anyone intelligent enough that also has never experience optical storage.

It's just that, as time passes, and our perspective of the world zooms out (coinciding with our understanding of the world), it becomes much easier to see how things are connected together. In the above example, part of the trouble with the Mayan civ is that we know so little about them and their world. It is not that they were complicated, or smarter than us, or were able to figure out things better than us (Yes, I'll see you all in 2013); the real issue is that we do not know enough about their fundamental culture in order to deduce what they were using things for. Certainly, using rope as a form of writing is an incredibly unusual way to write.

However, time marches on, and someone figured it out. Just like they will in the future.

In a way, it would be interesting if, in the future, someone did confusingly stumble across a shattered DVD, and, having analysed the data, finds a young man's porn collection, relentlessly locked down with encryption, it takes an unusually long 15 seconds to decrypt, and his reward is just the disclaimer:

"If you're reading this, I'm probably dead."

The researcher can't help but be gripped by the strange coincidences that must have lined up to bring this to him here.

Sorry, sometimes I like to write fiction.

-Red

PS. Certainly, the obvious counter to my thoughts is the human ability to look at ONLY either the forest or the trees - certainly, I've missed the "Tree" part of the "forest" when learning and figuring things out in the past, and will continue to in the future.

Re:Won't matter

[Nefarious Wheel]

I like the method shown by the George Pal version of "The Time Machine". (I think it was George Pal's - it's the one that debuted a young, red-haired Michael Doohan). Little rings on a round table top. You can't help but want to spin the first one you see. Brilliant piece of UI design, I think, although difficult to accomplish without an opposable thumb (it's anthropocentric, of course - our robot overlords might not have the knack of spinning a ring on a table) but I doubt that's a problem if we limit our scope to homo sapiens.

Re:Won't matter

[Tibia1]

I remember when I first started recording guitar when I was younger. Now all of the old records just serve to showcase how bad I was, while it still serves for some good comparison, making my new recordings look better.

Re:Won't matter

[CharlyFoxtrot]

It's not about the texts. Archaeologists learn a lot from trash :

"The unusable or unwanted remnants of everyday life end up in the garbage. By studying what people have thrown away, archaeologists can learn a great deal about a culture. This is true not only of prehistoric peoples who left no written record about their lives, but also of people today. Archaeologist Bill Rathje studies the garbage of Americans. He has learned many things about the relationships of human behavior and trash disposal, information useful in studying people of the past and the present. Rathje has found that people will often tell an interviewer what they believe is appropriate behavior, but their garbage tells another story. For instance, people frequently say they eat lots of fruits and vegetables, yet their garbage shows they do not. Another example is that people say they recycle more than they actually do (Rathje 1984, p. 27)."

Re:Won't matter

[Nefarious Wheel]

..."in candle-lit monasteries on flimsy paper"

Dude, parchment is leather. Nothing flimsy about it. The Book of Kells (9th Century), the Lindisfarne Gospels (8th) both written on parchment, are still around today.

I think that if you want perfect information retention, put the information on something people can read directly - tattoo a dead sheep.

Re:Won't matter

[Antique Geekmeister]

Unfortunately, as someone who's tried to challenge a tax audit for old expenses, I can tell you that a lot of those records do not keep well. Many poor quality printouts are not likely to last even a few years due to the poor quality of the paper and the ink. This is especially true of receipts, which are on the cheapest printers possible.

Re:Won't matter

[ObsessiveMathsFreak]

A related but more pertinent point is that no-one right now is able to archive or in most cases obtain anything because of copyrights and DRM.

I work in academia and I can tell you that future researchers are not going to be able to get their hands on 90%+ of the papers written today because the private companies that own them will lose the data when they inevitably go bust (Or just lose it). It will be one of the huge ironies of history.

Re:Won't matter

[Anonymous Coward]

I recycle most of those when they're no longer needed. They're not exactly the kind of thing I'd expect people to keep around for millennia.

Re:Won't matter

[countertrolling]

...It might be a long, long time before someone is able to discern it's purpose...

More likely they'll be studying the perfectly preserved 2,000 year old Twinkie...

Re:Won't matter

[Rennt]

I took the GP to mean all our non-biodegradable trash will still be here, even if our data isn't. 1000's of years from now they will be able to determine our diet, occupations and pastimes without the need for digital data.

Re:Won't matter

[BrokenHalo]

A reflective on one side, perfectly round disc? Looking at it under a microscope would no doubt show the presence of the "peaks and valleys" of digital data, and I think it would shortly fall into place.

I would have thought it more likely that a DVD or CD would be assumed to be a cheap, shiny thing used to scare off crows. It would just be too time-consuming to reverse-engineer and build a DVD reader from scratch.

Not too long ago, I threw out some 8" floppies I found in a cupboard. Readers for these are now almost non-existent, even though they're only about 30 years old. And the 5.5" and 3.5" floppies will soon become equally obsolete.

The only way to keep information preserved other than maintaining vast archives of acid-free paper or stone tablets is to *actively* maintain all of it online on multiple servers which are backed up verifiably. That means it either has to be stored in a format that doesn't become obsolete (ASCII) or it has to be routinely re-archived in other formats as they become fashionable. I still have a number of files created by WordPerfect 4.x which are (presumably) still readable by OpenOffice/NeoOffice, so I haven't worried about them, but that's not something I would count on forever. Sooner or later, if I want to preserve the content, I'm going to have to take the time to convert them to something more current.

Re:Won't matter

[dcollins]

"That's one advantage of paper, stone and parchment - they don't assume a technical infrastructure in order to use them."

Written language is itself a technical infrastructure. Your Mayan ropes are an excellent example of that.

Re:Won't matter

[Redlazer]

These people are talking about on the scale of thousands of years, the point where recovering data is a matter of "What were our ancestors like?" rather than "Oh look, a zip disk!".

Whose to say ASCII won't be outdated in the future. The very concept of digital data may be completely obscured by something rediculous like quantum data, with data being stored any other being like... receiving a million punchcards with Call of Duty 4.3, instead of a DVD.

Re:Won't matter

[LocalH]

Welcome to the modern world of production cost trumping all. Back in the old days, stuff was built with quality in mind. Nowadays, it's built as cheaply as possible, with no regards to longevity. Look at modern TVs - I recently had a TV built in early 1986 that just lost it's vertical deflection. 1986 to 2010. 24 years. Most cheap TVs (especially the fairly recent $79/$99 CRTs) die within four or five years. I lucked out in early 2002 when I bought a 20" JVC CRT for around $250. I still own this set to this very day, and it still has just as sharp and crisp a picture as it did the day it was purchased. The set was built in December 2001, which means it's just over 9 years old. If I had bought a $99 set at the same time, I would bet any amount of money that it would be dead by now (in fact, I bought my uncle a $99 set within a year after purchasing my JVC, and it has already been replaced). There are people who have sets from the 60s and 70s that still function (at most, with minor repairs of a few capacitors). There are even a few of the original RCA color sets that still function (albeit such sets have required a bit more restoration).

Re:Won't matter

[LocalH]

Gah. Bad math. The JVC set is just over 8 years old, not 9. /. really needs an edit function that expires within 10-15 minutes.

Re:Won't matter

[HungryHobo]

This is an example of sampling bias.
The crap your grandmother bought which broke right away you never see. The handful of items which were unusually well made and which lasted are all all that survive to get included in your experience.

People were complaining about the exact same thing long before you were born because *item they bought in the last year* broke while *Item they inherited from their great great grandfather* is obviously extremely well built and obviously they build things better in those days.

There has always been cheap crap which breaks right away.

Re:Won't matter

[umghhh]

Funny that. I think our forefathers did just that i.e. they (unknowingly?) created backups when knowledge was written by hand - the monks copied them by hand creating 'backup' copies - from dozens or hundreds only few survived and now we have them. If it is important (pr0n?) then it will survive because there will always be one place where there is a reading facility for the old stuff. So yes - back up stuff, keep it in many places and some part of it will survive. OC copyright may be an issue if technical means used to prevent piracy will make it difficult to decode and use if you have no contact with the server etc. but I think the important stuff like pr0n will do just fine. The rest if obviously not interesting enough....

Re:Won't matter

[jc42]

I have heard that some of the braided ropes left by Mayans might actually be a "written" language.

Can you give some references for this? I'm familiar with the Inca knotted-rope notation, but I've never read of any instances of it in Mayan areas. I did a quick google check of "Mayan braided rope writing", which gets a few thousand hits, but none of them seem to actually talk about such a Mayan writing system. They just use those words in different sentences, such as talking about braided-rope decorations in one paragraph and writing in another. I also substituted "Inca" for "Mayan", and google's first page linked to descriptions of the familiar Inca notation.

Also, the Inca knot notation has been understood as some sort of notation all along; it's just that nobody knows how to decode it. This was also true of Mayan writing, but it was finally cracked in the 1970s and 1980s with the help of native speakers of the modern Mayan languages (and there's a fair amount of scholarly research on the subject now).

So where can we read about Mayan knot notation?

Re:Won't matter

[jc42]

But 5000 years from now, how does someone interpret a shiny little disk?

They wouldn't be able to use that stuff because of copyrights and DRM

You're probably right; copyright extensions will probably eventually last that long.

But historically, this whole issue is nothing new.A number of historians have pointed out that, for example, our "scientific methods" have been independently discovered by intelligent people in most societies. The "scientific revolution" in the Western world wasn't based on discovery of scientific methods of investigation. Rather, it was the result of the new concept of open publication. Before that, technical knowledge was generally closely held by "guilds", organizations that controlled and restricted access to their specialized knowledge. This is clear in medical areas, where there is all sorts of "folk medicine" that typically turns out to include knowledge of a lot of naturally-occurring drugs. Western medicine surpassed the "medicine men" of other cultures in the late 1900s, because the Western medical people started publishing their results. This slowly produced a distributed body of knowledge that was available to all the practitioners.

But even in modern Western societies, there's a major exception: Knowledge developed by "private" organizations such as corporations is still closely held, and protected by the legal system. Even the patent system, which generally requires publishing the details of a patent, limits the usability of the information to people who have paid the appropriate license fees. The motive is profit, but the actual effect is very often to block use of the information by others. Trade Secret is, of course, the old guild-style secrecy in a modern form. Information handled this was is routinely lost when the single owner dies or loses interest due to low profits.

In any case, what's happening with digital records fits right in with the historic secrecy that most societies have had. The Western scientific community is one of the few known cases of free and open sharing of information. You'd think its spectacular success over the past few centuries would have taught us something. But this seems to not be the case. Even the computer industry, which is especially dependent on modern science for its very existence, tends to keep most of its records in closed, proprietary forms that are protected by trade secrecy. We do have a significant "Open Source" subculture whose results have been in line with the rest of the scientific world's. But most of the commercial part of the industry treats the Open Source approach with contempt and does extensive PR to prevent its adoption by the wider culture.

So it shouldn't be surprising that digitized records are a sinkhole of information. Such data has been and will continue to be quickly lost in the same way that the traditional Guild system lost most of its information within a few generations. If you want information saved, you have to put it into a form that will survive and be readable in the future. Most of our current digital media won't even outlive the people who write to it.

Re:Won't matter

[jc42]

These people are talking about on the scale of thousands of years, the point where recovering data is a matter of "What were our ancestors like?" rather than "Oh look, a zip disk!".

We have had a very recent success story of a case like this. After the Spanish destroyed most of the civilization of Central America, the Mayan writing system became unreadable. There are thousands of "stelae", stone columns covered with writing, all over the area, and they were generally understood to be historic markers. They did contain dates that were readable (since any mathematician will recognize the numbers as such and will understand the notation), but nothing else could be decoded. The writing was cracked in the 1970s, with a lot of help from speakers of the modern Mayan languages. Scholars are now busy decoding all those historic markers and the inscriptions on ruins of old buildings. Most of the original Mayan libaries were burned as "heretical", and only a handful of books survived (ironically smuggled to Europe by Catholic priests who understood their value). One of them is an astronomical reference text that tells us a lot about the capabilities of Mayan astronomers, but we don't have much else of what could have been thousands of other technical works.

So in that case, we've gone from "Oh look, a carved stone historical marker; I wonder what happened here" to "Hmmm ... On July 27, 1147, there was a battle here in which the forces of so-and-so city won. I wonder where that city was? Does anyone recognize the name?"

A funnier example: A few years ago, someone published recipes for drinks made from ground chocolate beans and hot peppers (and no sugar), translated from some old writings. There are several companies in Central America now selling the ingredients and instructions for making such potions, which were apparently quite popular with the Mayan upper crust a thousand years ago. They're really macho drinks, guaranteed to fry your tonsils.

Both of these do tell us a lot more about those societies than just "Oh look, a stone pillar covered with writing!"

(In case anyone wants a good scholarly project to work on, people are also tackling the pre-Mayan writing systems, e.g. the Olmec writing. They're all related, so cracking Mayan writing is helping decode the others. There are images online, so interested people anywhere in the world can get involved. That should satisfy just about anyone's nerdy desire to get involved in decoding obscure encoding systems. ;-)

Re:Won't matter

[Redlazer]

And you raise an important point - storing data encodes it in some way, and we must always first learn how it is encoded before we can make real progress, and there is no way we can predict the future well enough to leave our data encoded in a future-proof way.

Re:Won't matter

[rubycodez]

receipts can be a mixed bag, good idea to make copy at the time for any that are used for record of taxes or in business dealings. especially true for thermal paper

Re:Won't matter

[rubycodez]

that reminds me of light bulbs that have burned continuously for a couple decades, people might think "wow, they really made them well back in the day", but of course it's just the very very tail end of a bell curve that's been hovering a nanometer above the time axis for years, most all the other millions of units popped after the normal 1500 hours.

American cars don't rust out after five years with the piston rings worn away blowing oil...even with the cracked iron rods and sleeves in aluminum blocks and disposable modules it's actually higher quality. And the damn stuff starts when it is below zero degree F at turn of key, used to be trying to get engine to start without flooding or draining battery after sitting in extreme cold for two days could be a real challenge.

Re:Won't matter

[icebraining]

If you don't have on site and off site backups, you're not being careful enough. And if you do, how the hell are you losing files?

My problem is that I don't have anything worth saving:|

Re:Won't matter

[toddestan]

On the other hand, can you even buy a new TV nowadays that's not made in China? Sure, in the past there were plenty of TVs made that are now landfill fodder, but you could also spend some more and get a quality product that still survives today. Judging by what's built now, I would say having a new TV last 25 years would be a fluke, no matter how much you spent on it.

The question remains

[Anonymous Coward]

How much of it is really worth saving? Except The Goatse image and a good RickRoll video I mean...

Re:The question remains

[Tubal-Cain]

"Those that fail to learn from history are doomed to repeat it."

The more context you have, the easier it is to learn.

Re:The question remains

[giampy]

actually ... not at all.
After a certain level, too much information is even worst than too little.

Re:The question remains

[Anonymous Coward]

We'll soon find out. If it ain't worth paying a tiny yearly fee to preserve a file on an off-site storage, it ain't worth saving.

The only problem is what happens when the off-site company goes bankrupt. New legislation needed?

Re:The question remains

[YrWrstNtmr]

How much of it is really worth saving?

Pictures of my grandmother from the '20s? Priceless. Mostly useless to anyone but our family, but there ya go.

Re:The question remains

[Anonymous Coward]

Your grandmother is from 2020? So we DO invent time machines in the future!

Re:The question remains

[CharlyFoxtrot]

How much of it is really worth saving? Except The Goatse image and a good RickRoll video I mean...

And do we want to save everything ? We've never been as close as we are now to a society with perfect recall, we'd be nearly there now if it wasn't for copyrights. What happens to culture when things don't get to fade away and be rediscovered or reinvented in a different form, a perpetual remix culture ? Interesting times.

Re:The question remains

[Anonymous Coward]

actually ... not at all. After a certain level, too much information is even worst than too little.

[citation needed]

Re:The question remains

[Foobar of Borg]

How much of it is really worth saving? Except The Goatse image and a good RickRoll video I mean...

Yes, I can just see it now. An archeologist 5000 years from now uncovers an ancient CD-R perfectly preserved in an old, discarded refrigerator. After years of painstaking study, he finally figures out how to decode the information. He uses his specially created device to read the disk and places the first picture on his display.

A split second later, the other researchers at the university are treated to a blood-curdling scream of "My Eyes! My beautiful eyes!"

Re:The question remains

[presidenteloco]

Most brilliant artists, composers, and revolutionary scientists were scoffed at or ignored during their lifetime, and their work was only recognized as genius some time after their death.

How will we manage to avoid losing all that good stuff, when no-one knows its worth saving, until later.

The fight is lost

[MeNeXT]

How many of you have digital files from 15 years ago that you can read today? 20 years? There was no DMCA back then, now just imagine the future....

Re:The fight is lost

[Bluesman]

I do. The cool thing is that they all fit in a tiny portion of my portable hard drive.

It's amazing that the entirety of everything I've produced since high school can fit on a single $100 device, with plenty of room to spare.

Re:The fight is lost

[NotQuiteReal]

I have files from at least as far back as 1982 (Z80 source code). Of course they are not on their original media (which, might of been 8" floppies, I really don't recall.)

Re:The fight is lost

[Anonymous Coward]

I still have my programs from the C64 era and I can still run them, thanks to emulation. Most games from that time still exist only due to crackers who removed the copy protection which would otherwise be a challenge for emulators.

Re:The fight is lost

[pookemon]

I still have all the floppies and HDD's from my Amiga (and the ones I've bothered to look at in the last couple of years do still work). The question is more how many of us have files from 15 or 20 years ago that we want/need to read? I undoubtably have assembler code on the Amiga, and my C code from my Mac development at UNI. I might one day look at it and revisit the "good ol days" - which is exactly why I still have my Amiga tucked away in a box. But I doubt it.

Of course I am probably in the excessively rare situation where code that I wrote 10+ years ago is still code that I work with. Granted it was originally written in VB 4/5 and it's now written in .NET 2k8 - and that specific code is rarely modified - but i could probably go back and look at the original code if I wanted (I've certainly got enough copies of it lying around between backups to CD/DVD/Subversion/VSS HDD's etc.

But for now I will keep my copy of my UNI code on it's floppy secured to the filing cabinet with the Neodymium magnet I bought specifically for the purpose. That way I'll know exactly where it is when I find a 15 year old Mac that'll read it...

Re:The fight is lost

[peragrin]

I do. while only 12 years I have tax forms from 1998 in PDF format that open just fine. I efiled those taxes too.

Of course I worried about that 8 years ago, and switched all my files out of excel wnd word formats to text, rtf, and at the time OpenOffice. Now they are in ODF. Since ODF and PDF formats are easy to ready by many programmers, and thus open. Emails are stored in mbox. All of it is stored in multiple locations, with encryption(and decryption software) used as needed.

I moved all my data because i was tired of being tied to any one platform. I can access everything i have electronically on all three platforms now.

Re:The fight is lost

[iggymanz]

And what parts of those digital records would be *important* information? c'mon, you are talking about personal crap. Important records (birth certificate, medical records, academic records, insurance, account statements) will be on paper

Re:The fight is lost

[Synthaxx]

My Atari ST disks are all archived on my raid5 with backups.

And guess which ones made the cut, it's not the ones with codewheels and junky color swatches. It's the ones with the catchy introtunes and the scroller texts that read "elite" or "automation".
The guys liberating these media are gonna be remebered for a damn long time.

Re:The fight is lost

[Kenshin]

Not much of note. A few old websites I made, a few 3D renders (can't find the models, though.) I didn't produce much worth holding onto back then.

But pretty much all my stuff from 10~12 years back until now stays on my hard disk, and moves from new disk to new disk as I upgrade. All of my music and photos are managed by library apps, and I have automatic backups at least weekly. (Backing up is more convenient now, since I recently moved to a Mac and have Time Machine set to do it when I plug in my external drive.)

Re:The fight is lost

[jtownatpunk.net]

/raises hand.

I still have the "utils" directory from my '286 even though the programs have been obsolete for a 'coon's age. (A 'coon in captivity, that is.) Every company I've worked at has had a "data refresh" plan of some sort where we move old archived data to new media. And only one of those companies ever mined that data for a useful purpose. The rest kept it "just in case".

I have a feeling the problem isn't going to be that we retain very little important information but that, of the vast mountain of crap we retain, a shocking small percentage will have any real value. 2 tons of chaff for every grain of wheat.

Re:The fight is lost

[hldn]

"personal crap" could be just as interesting and as important to future researchers. think of all the things we could know now if we had more "personal crap" from the peoples of times past.

Re:The fight is lost

[turbidostato]

"And what parts of those digital records would be *important* information? c'mon, you are talking about personal crap."

What do you think History is but a lot of "personal craps" tied together?

Re:The fight is lost

[Locklin]

Exactly. It's not a technological problem it's a legal problem. The institutions that preserve printed material (mostly libraries) would be happy to help preserve digital information -if it was possible. Unfortunately nearly all digital content cannot be copied legally, and thus cannot be effectively archived by libraries. It takes a team of lawyers to do what archive.org does with web pages, and forget anything multimedia.

Re:The fight is lost

[Locklin]

You are talking about files that you have personally retained a couple decades. I can go to the library and find transcripts of Darwin's, Galileo's or Da Vinci's notes and letters.

Re:The fight is lost

[khellendros1984]

I still have a working Atari and a few dozen game cartridges, if those count...

Re:The fight is lost

[peragrin]

my insurance, account statements, and tax forms are all in PDF's.

I do have some paper copies as proofs, but only a handful of actual paper is needed for most of that stuff.

Re:The fight is lost

[ildon]

I have some .mp2 files from 15 years ago.

Re:The fight is lost

[Anonymous Coward]

I do. I've got source code I wrote in microbee basic, old assignments from early highschool... you name it. Takes up only an insignificant portion of my hard-drive and still reads fine in the emulator.

Re:The fight is lost

[Antique Geekmeister]

Wow. The Linux emulaors are available over at the Penguin Liberation Front, if you can ever extract the data from those cartridges.

Re:The fight is lost

[BrokenHalo]

I have code and documents dating back to 1976 on a HDD on this machine. Until 15 years ago I had it all stored on 800BPI mag tapes, but before I left my last serious "big-iron mainframe" site I transferred it across to floppies. I doubt if I'll ever need the files again, but since they don't make any significant dent in my storage, there's no reason to throw them away.

I know many historians (in fact my wife is one), and one day someone might be more interested in a perspective on '70s and '80s programming than I am right now. If I throw it out, that information will be gone forever.

Re:The fight is lost

[simplu]

All of it is stored in multiple locations, with encryption(and decryption software) used as needed.

I moved all my data because i was tired of being tied to any one platform. I can access everything i have electronically on all three platforms now.

Do you think that you encryption/decryption software will run on any operating system 10 years from now? What about 20 years?

Re:The fight is lost

[Ravenger]

I used to be a c64 artist back in the 80's and I've still got my artwork from that era. Most have been converted to modern file formats, but I still have some in the original Koala Painter format as well - That's files 24 years old. I also have some print-outs of basic programs from a few years before that.

I did lose some of my artwork over the years, but I managed to get nearly all of it back (bar a couple of my really early pictures) via the internet from various C64 archives or from individual users sending me files. In one case someone sent me one of my pictures I'd totally forgotten about. So archiving digital digital data in a usable format may be a concern, but without the internet I'd never have been able to retrieve my lost work.

Re:The fight is lost

[Anonymous Coward]

Do you think that you encryption/decryption software will run on any operating system 10 years from now? What about 20 years?

I'm not sure what you're getting at - why wouldn't it? Algorithms don't change - they can just be implemented elsewhere.

Re:The fight is lost

[Anonymous Coward]

"Mr. Pepys, we've found a diary. Let me read you a sample extract. [reading] December the third. Whilst the King was away, did sport madly with Nell Gwyn."

Re:The fight is lost

[kaiidth]

Your case study seems to reinforce Torvalds' theory of data preservation:

Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ;) -- Linus Torvalds

Re:The fight is lost

[quanticle]

Given that the commonly used encryption algorithms are open source (e.g. AES, Blowfish, etc.) there's no reason to suspect that encrypted data will somehow become "obsolete". All the key needs is a tag stating which decryption algorithm to use.

Of course, if the key is lost, the data is unreadable, but isn't that the whole point of encryption in the first place?

Re:The fight is lost

[sgage]

"How many of you have digital files from 15 years ago that you can read today? 20 years? There was no DMCA back then, now just imagine the future...."

I do. In fact, I have some files going back to my CP/M days - very early 80's. They got transferred along the way to every computer I have owned, and are now sitting on my 250 G drive in my current computer, and also backed up on an external HD. (I finally had a scare that convinced me to have a more or less formal backup regimen.)

I can read the old WordStar and WordPerfect files, no problem. Even some old SuperCalc spreadsheets.

In all that time, the files were never really "backed up", they were just live on my current system.

Re:The fight is lost

[Convector]

I recently came across a 3.5" floppy disk while rummaging though the home office. I'm pretty sure it's no more than 12 years old. Curious as to what was on it, I discovered that none of my (working) computers had a floppy drive. It looks like I lose. Or I have to finally replace the power supply in the older machine that does have a floppy drive, but I'm far too lazy for that.

Re:The fight is lost

[Colin Smith]

What do you think History is but a lot of "personal craps" tied together?

What they victors tell us it is?

 

Re:The fight is lost

[MightyYar]

Isn't that the ultimate in sampling bias? You can go to the library and find their transcripts because they survive... how many lesser artists' notes and letters have been lost?

I'd bet that people will find a way to preserve, say, Linus's old code and notes as opposed to a handful of geeks who read Slashdot :)

sadly lawyers are working tirelessly

[Anonymous Coward]

to ensure this never happens. This is the same reason why DVD's and Bluerays will never work in 100 years time.

DRM will destroy any record of our current culture, but looking around at the abyss, I really have to say its for the better.

But I already feel bad for the eventual people that will spend far too much time trying to recover "scary movie part 15" or some other 'gem' from our time. But much like 'abandonware' and other areas of trying preserve machine code, lawyers will always race in to make sure all copies are lost forever.

Support things like SIMH while you still can!

Re:sadly lawyers are working tirelessly

[secondhand_Buddah]

I'm willing to bet money on the fact that our sophisticated DRM of today will seem like a stone age toy on a few years time. Its only purpose is to control the market right now. Frankly the DRM thing in terms of preserving culture is a red herring. How many movies to you have in an unencumbered format like DivX? or songs in MP3 format? These are the artifacts that future archeologists are most likely to examine.

Re:sadly lawyers are working tirelessly

[Anonymous Coward]

Hey, look at the upside, at least future generations will never be subject to the punishment of listening to Britney Spears CDs.

The Middle Ages didn't have the DMCA

[MagikSlinger]

The main way ancient writing reached us is because someone copied it. Lots of copies. Sometimes translated into another language and back, for example, a lot of Greek learning went into Arabic and came back out into Latin or Greek. With all the copy protection and encryption on our media today, can we ever copy the data and be able to decipher it again?

Re:The Middle Ages didn't have the DMCA

[a whoabot]

"a lot of Greek learning went into Arabic and came back out into Latin or Greek."

Can you give me an example of a significant text? I'm pretty sure it is a myth that lots of Greek learning has gone through this process. I see the claim made a lot, but I've never come across a text which has done this (the philosophers, the dramatists, the historians, the lyric poets, all seem to come from the original Greek).

Re:The Middle Ages didn't have the DMCA

[Anonymous Coward]

How about the Rosetta stone? That's a pretty significant use of multiple languages to extract meaning from an even older one.

Re:The Middle Ages didn't have the DMCA

[Anonymous Coward]

With all the copy protection and encryption on our media today, can we ever copy the data and be able to decipher it again?

Luckily, the strength of most DRM is shit. If it's so easy to crack today, imagine what the space-historians of tomorrow will be able to do with their positronic computers and whatnot.

Re:The Middle Ages didn't have the DMCA

[Anonymous Coward]

Ah, and therein lies your problem, artificial selection by genre. A lot of the science and mathematics work was lost except in Arabic. Hero of Alexandria, for example, had to be recovered. The monasteries weren't interested in that kind of stuff, you see.

Re:The Middle Ages didn't have the DMCA

[lennier]

I don't know about Greek, but it's a good thing that at least the authentication server for the Epic of Gilgamesh is still online.

Re:The Middle Ages didn't have the DMCA

[reverseengineer]

Probably the most significant texts to undergo this process were Ptolemy's Almagest and Euclid's Elements; both had been lost to Western Europe, and were thus translated in the Middle Ages to Latin from Arabic by Gerard of Cremona and Adelard of Bath, respectively. I believe in both cases the original Greek texts were eventually recovered by the West used for later direct translations, but for a while Western Europe knew Hipparkhos/Hipparchus as "Abrachir."

Re:The Middle Ages didn't have the DMCA

[Ltap]

The problem is that very few identifiably Greek writings survive. In ancient times, copying was a bit like playing telephone - writing at the time was very politicized, so scribes would often alter works while copying them, mostly to give a local slant or simply changing the names. This makes it frustrating to trace things like legends (see: Noah's Ark/Epic of Gilgamesh and its infinite variations with every other culture that existed nearby). A lot of Greek and Roman writings are now quite simply lost for good, but almost certainly inspired works that aren't lost. For instance, the Odyssey and the Iliad were originally just two parts of the epic story of Troy (out of, AFAIK, four or five parts in total), and the set of works that we derive most of our knowledge of Rome from, Ab Urbe Conditum, are only partially preserved - it was a set that chronicled the history of Rome from its founding to when they volumes stopped being produced, and there were hundreds, enough to fill entire libraries. It was only in the Renaissance that anyone tried to assemble a collection, and we've only been able to come up with about 30 - if we had the full set, we would know a great deal more about Rome than we do now.

Re:The Middle Ages didn't have the DMCA

[MagikSlinger]

I see other people have responded, but a lot of the mathematical texts came that way. For example, Euclid's Elements was the most famous of these. There were a lot of books on the geometry and mathematical knowledge of that age, as well as most of the ancient astronomy. If I still had my text book from the "History of Mathematics" class I took, I could give you specific names and titles.

Almost all of today's surviving texts of Archimedes came via Arabic: http://en.wikipedia.org/wiki/Archimedes#Writings

From what I gather, we are lucky to have what we do. A lot more of that preserved Ancient learning was lost when the library of Baghdad was sacked and burned.

Re:The Middle Ages didn't have the DMCA

[Anonymous Coward]

The DMCA does have archival exemptions and it doesn't apply to everything. We don't know what the popular songs were in the 1600s that street musicians played but we still know a remarkable amount and there are family histories and such that fill in a lot of those gaps. Plus the data that is covered by the DMCA is usually owned by an entity that has a vested interest in making it playable to paying customers. It's funny, you can get blu-rays with old movies on them but you might have a really hard time playing super8 films your grandfather made of your mom or dad 40 years ago.

It's startling how quickly things can change in technology, my wife and I have a VHS recorder somewhere in the basement or garage but we've not used it this century... The media for storing data, the formats the data is in, those are some hard problems on the generational time scale. I just did an exercise a couple months ago, copying some CDRs I made in 1993 (from data what was on floppies before that, probably spanning back to the early 1983 or so) on to newer media. For reasons I don't know, I used 6 or 7 different DOS based archivers to save space, some of which are completely dead and gone (but the compressed really well in 1993...) I scoured the web to find copies, then I had to figure out how to make some DOS software work, then I had to figure out how to actually get the data in to my DOS virtual machine (no networking in DOS and I have long since forgotten how to make a CD-ROM work in DOS...) and one of those programs happened to do enough exotic crap that it just wouldn't run in Win2000... This was only 17 years and fortunately I'm the one that assembled the data and I gradually started to remember all the stuff as I recovered it, someone else might not have figured it all out. The CDR will worked fine though.

Even with opensource stuff there is a half life, I had xiafs based Linux disks at that time, that's long been yanked out of the kernel. GIFs might not be readable in 15 years, it's completely possible. You might laugh but take ZIP, it has to be one of the most ubiquitous formats for data out there, it won't work with files over 4GB in size and files that big are becoming remarkably common... So xar or tar or something else starts to generally replace ZIP. Fast forward 10 more years and you could have some challenges compiling unzip and then using it on your files.

I think the best thing is to have a policy of sorts of going back every 5 years and keeping your data living. The media grows so it's not hard to store it, that's cheap, but the formats and everything need to be updated along the way.

Re:The Middle Ages didn't have the DMCA

[martin-boundary]

The ancient Egyptian hieroglyphs were not even DRM'd at all, and still the only way to read them was to use a cheat sheet (a.k.a. Rosetta stone), and that was not for lack of trying other methods.

Re:The Middle Ages didn't have the DMCA

[Anonymous Coward]

Aristotle.

Re:The Middle Ages didn't have the DMCA

[Anonymous Coward]

> If I still had my text book from the "History of Mathematics" class I took, I could give you specific names and titles.

Did you left it at the library of Baghdad?

Re:The Middle Ages didn't have the DMCA

[a whoabot]

I don't really see how your link supports your assertions. It says: " Archimedes' work was translated into Arabic by Thbit ibn Qurra (836-901 AD), and Latin by Gerard of Cremona (c. 1114-1187 AD). " That does not say that Archimedes' work comes to us through Arabic, it just says that it was at one point translated into Arabic. Indeed, this source seems to contradict your position: a collection of Archimedes works in 9th-century Constantinople form the "basis of the texts we have today." Why would Greek scholars in the 9th century who didn't know much Arabic, if any, make a collection of Archimedes in Arabic, instead of Greek, when the Greek manuscripts were clearly still available in the 9th century (how else would bin Qurra have translated Archimedes into Arabic?)? I think I'd believe that that collection (which "is the basis of the texts we have today") which they are is mentioned here was in Greek, unless you can provide some evidence that it was not...

This page seem to contradict your statement about Euclid's elements. If a 4th century edition formed the basis of texts until the 19th century, then the texts until the 19th century were not based on Arabic, because there is no way that 4th century text was in Arabic. And then that text was improved with another Greek version, to form the Heiberg edition, which, so it says (in 1971), "still stands". There's no mention of Arabic.

Other responders have similar problems (Ptolemy's Almagest: http://www.ibiblio.org/expo/vatican.exhibit/exhibit/d-mathematics/Greek_astro.html). But one of the other responders mentions Hero of Alexandira, and appears to be partly right: of his 14 works listed in this article [PDF], 1 comes to us from Arabic (the Metrica), and 1 comes from Arabic with some Greek fragments (the Mechanica). Both seem to be significant works.

So I still think there is this myth that "lots" of Greek works are transmitted to us through Arabic, although I'm happy now to have some examples of significant texts (2 of Hero's 14) that do actually fit this model.

Re:The Middle Ages didn't have the DMCA

[agrif]

I'm not sure where I heard this idea, but it bears repeating:

Future historians will hate us, with a passion, because we encrypt even the most banal things. We encrypt movies, for God's sake! Where's the justification in that? We're robbing the future of our culture, even from things like movies with talking hamsters!

Re:The Middle Ages didn't have the DMCA

[Anonymous Coward]

The producers are making lots of copies themselves. Not everything will survive. The library of Alexandria is a good example of great intentions, lots of care, and insane loss. It will happen again and again, and yet enough will ooze out of the cracks to inform the future generations.

Re:The Middle Ages didn't have the DMCA

[Guido von Guido]

This isn't a specific text, but see Colin Wells' Sailing from Byzantium. Wells discusses the legacy of the (poorly named) Byzantine Empire, including how knowledge from the ancient Greeks and Romans was transmitted to the west and to the Arabs.

Re:The Middle Ages didn't have the DMCA

[a whoabot]

Aristotle.

I know that's wrong and I don't even have to look it up, but since others may:

Source: "Modern editions of the Greek text of Aristotle are based on the Greek manuscripts copied in the Byzantine period (mostly during the tenth century and later) from manuscripts derived indirectly from the edition of Aristotle works produced by Andronicus in the first century B.C."

Re:The Middle Ages didn't have the DMCA

[Anonymous Coward]

Um... Livy's Ab Urbe Condita - The History of Rome ?

http://en.wikipedia.org/wiki/Ab_Urbe_Condita_%28book%29

Which would have filled shelves, not libraries, and was basically full of repetition, propaganda and the occasional funny anecdote. And this is coming from a man with a decent section of it. And as far as I know it was his work, with probably a few scribes tossed in to ghost write.

Would have made Certamen a lot more fun.

Re:The Middle Ages didn't have the DMCA

[Dreaming in R'lyeh]

Yes, now, (by and large). But it wasn't always that way.

For a long time (say roughly 5c to 13c), western Europe did rely on second- and third-hand translations.

See http://en.wikipedia.org/wiki/William_of_Moerbeke regarding translations of Aristotle, for instance, or http://en.wikipedia.org/wiki/Galen for another significant author who was widely read in translations from Arabic, and some of whose works are indeed still only known from Arabic translation.

Re:The Middle Ages didn't have the DMCA

[gnapster]

What if they were DRM'd? IANA(ancient Egyptian)L, but I can imagine that those Pharaohs were pretty stingy about people copying their words from one pyramid to the next. Can you imagine what the penalties for breaking the Ancient Millenium Copyright Act would be? Death by crocodile? Live mummification? Or just having to lift a finger in the building of your own tomb?

No wonder they took us so long to decipher!

Re:The Middle Ages didn't have the DMCA

[Caue]

math.

Re:The Middle Ages didn't have the DMCA

[Anonymous Coward]

off the top of my head, Galen is a good example (although the value of an author as significant is, obviously, subjective. I consider Galen to be significant.

http://en.wikipedia.org/wiki/Galen

Re:The Middle Ages didn't have the DMCA

[quanticle]

Given that no form of copy protection in use today has lasted more than a couple of years before being broken, I'd say that copy protection won't be much of an obstacle to future generations. Remember, they'll have hardware that's beyond our current capabilities by a few orders of magnitude. Cracking copy protection should not be a technical problem.

Re:The Middle Ages didn't have the DMCA

[aBaldrich]

Maybe today we have the original Greek texts, but you have to consider the world in the middle ages. Manuscripts were precious. Greek texts were kept mainly in Byzantium. The Greek language was not widely studied in the West until the renaissance (specifically, after the Ottoman victory a lot of Greek scholars traveled to Venice and from there to the universities of Florence [Medici], Ferrara [Este] and Milan [Sforza]). Of course many people had notions on Aristotle's Logic many centuries before. That's because a Bizantine Emperor (whose name i don't recall right now) had gifted the Caliph of Cordoba some very rare copies of Aristotle, and then Ibn Rushd (aka Averroes) wrote the famous Commentary (see wiki). In it, he explained Aristotle's philosophy, -specially the logic- and criticized Al Farabi, another Arab philosopher. Lots of Arab writings, including Averroes' Comment, were translated into Latin and then spread through Europe.

Many centuries later (during the renaissance), philology emerged as a science. Credit Lorenzo Valla and Nicola Cusano. After that, everyone searched and compared every copy of each work until they could reconstruct the originals.

Re:The Middle Ages didn't have the DMCA

[Anonymous Coward]

Sure - Aristotle. And whole lot of mathematics, particularly geometry.

Quick...

[eegad]

Everybody print out all their emails!!!

Re:Quick...

[biryokumaru]

Oh god, why doesn't Gmail have a print all function!?

Re:Quick...

[Anonymous Coward]

I assume you mean this?

Does anyone learn from history anymore?

[mrbene]

Those that forget history are doomed to repeat it - but these days, it seems that there's more and more effort put into actively avoiding learning from history.

Or maybe I've just hit that age when The Kids ought to get off My Lawn.

Re:Does anyone learn from history anymore?

[N!k0N]

guess I'm there too... except i still live in a flat, and have no lawn for the damn kids to get off of...

Re:Does anyone learn from history anymore?

[MrEricSir]

History is like software, it needs maintainers or it's doomed to disappear in the next version.

Re:Does anyone learn from history anymore?

[iggymanz]

you mean "needs rewriters and revisionists."

"History" is written by the winners to appease their benefactors, as they say.....

Re:Does anyone learn from history anymore?

[Anonymous Coward]

It's not your age. You are right. Excuse me, there's someone on my lawn...

Re:Does anyone learn from history anymore?

[Anonymous Coward]

Today, class, we are going to learn about the primitive bipeds that once lived on our planet, Omicron-Persi 9, only they called it 'Dirt'. Not much is known about these primitive lifeforms. Before we colonized this planet, we sent giant degaussing probes to scan for signs of intelligent life. After the degaussing probes found no signs of intelligent life, our archeological teams were only able to find some stone tools and spongy yellow thingies with cream filling...

Perhaps the way we think..

[malkavian]

About storing data will change. Historically, we've stored on paper, stone, or whatever could be inscribed. The 'backups' for data has been more about attempting to 'inscribe' media with the digital info.
Perhaps we're entering an era where we'll be trying to keep information 'live' perpetually, with the internet the first attempt at having an active library (though there are currently lots of cracks for information to be lost).

Many of the laws that overly stymie information flow (DMCA etc.), I think, are just a knee jerk reaction in the way printing presses were suppressed, and controlled until everyone realised the benefits of having them opened up.

Still, having the long term offline stores is no bad thing..

Re:Perhaps the way we think..

[CharlyFoxtrot]

Many of the laws that overly stymie information flow (DMCA etc.), I think, are just a knee jerk reaction in the way printing presses were suppressed, and controlled until everyone realised the benefits of having them opened up.

Barbarians have always burned down libraries. No reason to think they'd stop doing that just because they wear ties these days.

perfect example: Geocities

[Eravnrekaree]

It is indeed a big problem. The problem was illustrated recently when Yahoo suddenly pulled the plug on Geocities, wiping out a vast cultural archive that went back to the early days of the internet, a lot of valuable information was lost as a result of that. Yahoo's blatant arrogance caused me to refuse to ever use any of their products again. Geocities was actually a fairly nice service, often people criticised it because of the ads, but how do you pay to continue to offer a free service. The loss of geocities was a perfect example of the need for a permenant store or online archive of information, personal websites and so on that can be maintained as a cultural legacy and informational resource.

Re:perfect example: Geocities

[biryokumaru]

If only there were such a thing...

Re:perfect example: Geocities

[jaavaaguru]

You mean like archive.org? I actually went there recently to look at old Geocities, and was shocked that they don't have it all backed up there. Archive.org has pretty much everything else I've looked for. Any idea why geocities is not there?

Re:perfect example: Geocities

[Eravnrekaree]

I checked archive.org backups of geocities. half of the sites are not backed up correctly. Mine was never backed up, it seems, at all. With most sites 90% of the files are missing. Is archive.org the solution? Apparently not.

Re:perfect example: Geocities

[Eravnrekaree]

Interesting that you mention that. I havea geocities site so i knew of the Archive.org thing, that was supposed to back up the site. I checked archive.org backups of geocities. half of the sites are not backed up correctly. Mine was never backed up, it seems, at all. With most sites 90% of the files are missing. Is archive.org the solution? Apparently not.

Re:perfect example: Geocities

[Anonymous Coward]

And someday they'll disappear too, taking their entire archive with them.

The point isn't that someone is backing up the Web sites, the point is that there aren't a bunch of copies of Web sites spread around everywhere. Sure, the Internet Archive would probably try to transfer their archives to some other organization if they folded, but it's still a single point of failure.

In short, who archives the archivers?

Re:perfect example: Geocities

[lennier]

Perhaps because others were doing it. A number of independent projects tried to back up Geocities, and may have between them recovered most of the data.

* http://geociti.es/
* http://reocities.com/
* http://www.archiveteam.org/

Re:perfect example: Geocities

[Anonymous Coward]

Most likely because of the bandwidth limits. When I tried wgetting my Geocities page it failed about halfway through because I had gone over my allowed bandwidth for the time period(I had quite a few pictures on there).

Re:perfect example: Geocities

[Ltap]

Archive.org is the solution, and this is just one of those problems where throwing money at it actually works - give them more bandwidth, more contributors, and more disk space, and they could work wonders.

Re:perfect example: Geocities

[Ltap]

Part of the problem is of manpower - geocities was just so massive, and Yahoo gave them very little time to archive anything properly, so most of it was simply a dash to copy as much as they could before it was deleted. When you look at public domain audio, video, and texts, you'll see that things have been done much better.

Re:perfect example: Geocities

[Eravnrekaree]

Well, unless there was a directory that was provided by yahoo which archive.org could crawl, of all sites, possibly some could be missed. Mine was on google, I know that. Plus there was over month of notice. Archive.org is just not a comprehensive archive solution.

Re:perfect example: Geocities

[Nimey]

Ah, you must be a young'un. Geoshitties was famous for having a tiny bandwidth cap, something on the order of 10MB per day (or week?) per account. I'm sure the good folks at archive.org didn't want to be constantly DoSing peoples' webpages, so they didn't bother backing up that site.

Re:perfect example: Geocities

[antdude]

http://archive.org/ has the copies of GeoCities.

Re:perfect example: Geocities

[burningcpu]

Oh boy, this got me interested in my old geocities site that I hacked together in I was ~10. I just saw it at archive.org. This is like a checklist of crap that noobs put on geocities.
I had a repeating animated GIF of a spinning playboy bunny that I made with some shaddy program I downloaded off of cnet.com

The main page played low rider midi and there were least 3 frames going on. Top, left, right.

My clan's name was AFKMFERS, which was obvious code for ass kicking mother fuckers, and I had an animated GIF of it spinning around

Some lame guestbook full of spam

Under Construction signs

All that is left at archive.org is the carcass, some badly lain frames and broken images. If you really must see, and you have a morbid imagination, here you go.
http://www.geocities.com/Area51/Labyrinth/3381/

Re:perfect example: Geocities

[RMH101]

I hear a sound: as if a million [BLINK] tags and "Under Construction" animated gifs were extinguished at the same time

Re:perfect example: Geocities

[Ltap]

Considering the volume of stuff on geocities and the manpower available to archive.org, a month wasn't enough notice. Six months, maybe.

Also, I think their approach is good and they've already got a lot of stuff archived - what exactly would you suggest as an alternative?

Re:perfect example: Geocities

[carvalhao]

Actually most of it was backed up http://www.geocities.ws/

ffs.. the "zomg how to preserve" story -again-!?

[Animaether]

Seriously, Slashdot.. until there's a revolutionary insight into this matter.. quick posting these stories ad nauseum.

For further commentary, see previous stories... here's one.. it's from september 2009 and -nothing has changed-.

http://ask.slashdot.org/story/09/09/29/1646251/Archiving-Digital-Artwork-For-Museum-Purchase

Uncompressed

[Singularity42]

You'll have to go to .wav (not FLAC)--just straight bits. This does away with both copy-protection and compression.

One Site to Archive Them All

[enoz]

http://archive.org/

They've already got a copy of your Geocities sites from the first Digital Dark Age.

Re: One Site to Archive Them All

[indeciso]

...One Site to find them, One Site to bring them all and in the Darkness bind them...

Re:One Site to Archive Them All

[JoeMerchant]

They aren't archiving everything the way they used to... I've been trying to convince them to archive a couple of sites for 2 years now and they still haven't shown up. I think they're allergic to Wordpress weblogs...

Re: One Site to Archive Them All

[Anonymous Coward]

and One SIte to censor them

To forget is good

[Anonymous Coward]

IMHO we'll find that our problem is that we drown in a sea of useless information because we can't find the islands of relevance. Trying to archive everything will only lead to failing to archive anything. On the other hand I doubt that we'll lose much important information despite failing at organized preservation attempts, because important information is copied all the time, which is the only way for information to survive quickly changing technologies and file formats anyway.

In a more philosophical light, I think that forgetting is good for us. It frees us from the constraints of our past and makes way for new ideas. Archives are backwards-facing, but we all live in the future, all the time.

Re:To forget is good

[presidenteloco]

Yes but the essence of human cognition is to apply our memory of the past and the patterns we have detected in it to our understanding of the present and the future.

That said, you are correct that a "relevant and significant memories retrieved first" algorithm and storage architecture is necessary, to make sense of it all.

Re:To forget is good

[donleclaire]

... but we all live in the future, all the time.

No, we all live in the present, all the time. 'The future' is just an idea in your mind, a mental projection. Besides, the future isn't what it use to be ;-)

Re:To forget is good

[Anonymous Coward]

Vint Cerf talked about this kind of young chap who had problems understanding why only relevant information is not the relevant in future.

Start watching this http://www.youtube.com/watch?v=oTnAKUMFvKw from around 2:20 mark...

"It took about an half hour to take librarians off the ceiling"

Forecast: Cloudy forever

[presidenteloco]

I think that many people are failing to appreciate the longevity of information preservation
that cloud computing (more specifically, redundant, geographically distributed network storage) can bring.

If we get the protocols right, and insist on open standards for data interchange, we can obtain
properties such as:

Data bundles that know how to move themselves to more recently commissioned, and/or more
reliable hosts.

Data bundles that know how to check in with copies of themselves, to make sure there are enough of
them alive, and that they are adequately geographically distributed, at every given moment.
If not, then more baby copies of the same data would be produced and stored elsewhere automatically.

There are other issues to longevity of course, like maintenance of software that understands different
versions of data etc. Not trivial but very doable.

How long an individual disk or SSD or stone tablet lasts is COMPLETELY IRRELEVANT to
the prospects for information longevity, given the network, and new levels of automated distribution
that will take place on it going forward.

Re:Forecast: Cloudy forever

[value_added]

How long an individual disk or SSD or stone tablet lasts is COMPLETELY IRRELEVANT to the prospects for information longevity, given the network, and new levels of automated distribution that will take place on it going forward.

I don't know that I agree with that.

Compare, for example, letters written during the Civil War, with email messages sent and received by those involved in either of the Gulf Wars. Which do you think had, at the time they were written, a better chance of being available to future historians?

I'd suggest we are in danger of losing our history. What's odd is how blithe we are about it.

Re:Forecast: Cloudy forever

[lgw]

Information "in the cloud" will disappear the moment you stop paying for it. Corporate information in the cloud will come with a destruction date (as do paper corporate records in a storage facility, so no real difference there).

Re:Forecast: Cloudy forever

[Ltap]

It depends on how the senders and receivers think about the information. I know people who kept every postcard and letter they'd ever received - I doubt you could say the same about that with email. People still just don't consider email a serious medium.

Re:Forecast: Cloudy forever

[presidenteloco]

Well, one thing that will have to happen is the establishment of "public library" clouds. I guess you could imagine these being funded by various governments or non-profit associations.

Or you could go with the massively P2P model in which the data is stored, in little fragmented encrypted chunks, on millions of edge devices on the net. i.e. peoples' personal computers, en masse, each contributing a tiny bit to the perpetual storage cloud.

Re:Forecast: Cloudy forever

[JoeMerchant]

Yeah, what could go wrong...

Cloud is nice, and I have been very appreciative of Yahoo and Google archiving my e-mail for the last 10+ years without losing any (as opposed to my local copies), but a massive redundant system is no guarantee of future service in the face of bankruptcy, war, etc.

The post Roman era dark age was brought about by the collapse of society - the next time we manage to do that (and, in geologic time, it's coming soon), it's going to be quite the mess. At least there wasn't much difference between the Romans and the Barbarians, just picture today's world with no organized energy (electricity, oil, etc) distribution systems...

Re:Forecast: Cloudy forever

[guspasho]

This assumes that the information age lasts forever. Technology continuously improves and human civilization survives without interruption. And most importantly, that information survives brief or even extended periods of irrelevance.

Basically, your answer to how to survive a digital dark age is to assume that one will never occur?

Re:Forecast: Cloudy forever

[value_added]

I know people who kept every postcard and letter they'd ever received - I doubt you could say the same about that with email. People still just don't consider email a serious medium.

Which is my point. If digital files aren't considered "serious", how can elaborate backup schemes and distributed network storage amount to anything more than transient (albeit reliable) storage?

The irony is that those letters and diaries of days gone by may not have been considered "serious" either, but they did outlive their original owners.

Re:Forecast: Cloudy forever

[presidenteloco]

"Basically, your answer to how to survive a digital dark age is to assume that one will never occur?"

No. I'm stating that the best way to avert loss of the information is to improve our techniques for massively redundant and well distributed storage of that information. This by the way is the strategy that life uses to preserve its core (genome) information, so I'm in good company.

It wouldn't hurt to improve the sustainability and self-sufficient power generation of many more independent sites, either.

I'm basically saying that the redundancy and distribution solution is going to be more effective, and more probably successful, than a "stone tablet" solution.

If we experience a nearby supernova or a massive meteor, neither is going to be of any use, so hopefully we're agreed that we're talking about less extreme scenarios than that.

What we need are "fractally self-sufficient" nets-within-nets.

Re:Forecast: Cloudy forever

[Staniel]

I keep thinking about the parallels of this with the older digital media of hand-copied books. Computers are much better than humans at copying data exactly, but much worse at understanding data once a small part of it has been corrupted.

We don't have to trust one neurotic monk anymore, but it's still possible for small corruptions to leak in. I wonder which has better data integrity: paintings inside a dark, windless cave, or a bunch of computers yelling ON and OFF at each other.

Re:Forecast: Cloudy forever

[clickclickdrone]

>I'd suggest we are in danger of losing our history. What's odd is how blithe we are about it.
Possibly because we back up stuff we think is important today - news, science etc. However, historians in the future will be far more fascinated by shopping lists, crochet magazines and TV listings.

Re:Forecast: Cloudy forever

[Yvanhoe]

The most problematic obstacle to data preservation, IMHO is the current legal and copyright system : A lot of copyrighted data is worth saving, but it raises a lot of problem when it comes to distribute backups of these. Right now, some data created before WWII are still protected by copyrights and saving/distribute them might be illegal, but their copyright holders might also not care for it.

This is a serious legal problem. We might lose a lot of the 20th century data because of unadapted copyright laws.

Re:Forecast: Cloudy forever

[lgw]

I can't see the government funding what would inevitably become everyone's pr0n repository.

Re:Forecast: Cloudy forever

[zero0ne]

Somewhat exists already:

http://allmydata.org/trac/tahoe-lafs

Open software, open standards

[gwern]

"Movable Type is a dead end. In the long run, the utility of all non-Free software approaches zero. All non-Free software is a dead end."

http://diveintomark.org/archives/2004/05/14/freedom-0

PLASTIC PUNCH CARDS

[Marxist Hacker 42]

Far outlasts stone, and if you did it right I'll bet you could get nearly 1Mbit per card without running into the problems of Lace Cards

Re:PLASTIC PUNCH CARDS

[lgw]

The army had a program to design a means of storing data in case of really being nuked back into the stone age. They chose punched metal tape. Most plastic doesn't last long whn exposed to sunlight or weather, and the downside of a card deck is obvious the first time you drop one down the stairs. It's a clevel idea really, since you can read punch tape manually if you have to, and it's far faster than cards.

Great! Now I'll have to buy the White Album again

[BluBrick]

We will naturally make multiple copies of everything we consider important, continually transcribing important data onto the latest generation data storage media. (Consider what was the very first publication printed on Gutenberg's big invention.) Unfortunately, that's not necessarily what will be considered important many generations into the future.

I have every confidence that, far into the future, we will have or be able to develop the capability to read any media we preserve today. The problem then becomes how to determine what data we should should preserve now rather than how to preserve it. What do we know now that will be important and useful to someone 10^n years from today?

Not so hard

[T Murphy]

Just put a massive data server in a spaceship and accelerate it near the speed of light. Data loss would be slowed enough that it would be negligible, and if we have to retrieve anything it should have a fast enough processor to respond to a request in a timely fashion and send off a pre-made copy of the needed data (as it may take too long to copy petabytes at near light speed).

This should work out perfectly- by the time we have the technology to do this, today's worthwhile material should finally be coming out of copyright.

Re:ffs.. the "zomg how to preserve" story -again-!

[iggymanz]

especially that the main insight is that 99% of digital records are useless crap. Just like it won't matter if archaeologists never find 99.9999999% of our cities, when you've seen one Starbucks next to a McDonalds next to a Walmart, you've more than seen them all. The ditto mark will be the most used character recording our drivel... don't even get me started on our mostly devoid of talent "music" and "art" (is a frontal lobotomy prerequisite to being a rap star?)

924 Years and nothing has changed

[rudy_wayne]

The Domesday Book was commisioned in December 1085 by King William (aka William the Conqueror, who invaded ngland in 1066). The first draft was completed in August 1086 and contained records for 13,418 settlements in the English counties south of the rivers Ribble and Tees (the border with Scotland at the time). It is a detailed statement of lands held by he king and by his tenants and of the resources that went with those lands. It records which manors rightfully belonged to which estates, thus ending years of confusion resulting from the gradual and sometimes violent dispossession of the Anglo-Saxons by their Norman conquerors.

In 1986, at a cost of £2.5 million, the UK compiled the contents of the Domesday Book into electronic form that was stored on laserdiscs. The information stored on the laserdiscs, which is the equivalent of several sets of encyclopedias, is now unreadable because the equipment needed to read the discs is no longer available. Meanwhile the original book is still readable after more than 900 years.

Re:924 Years and nothing has changed

[Anonymous Coward]

Really? The last functioning laserdisc reader I saw was in the late 90s... around the time when DVDs were becoming available. There is really no excuse for not transferring this data to DVD. Are you certain there are no functioning laserdisc readers? I have working floppy drives and other media from the mid 80s (although the plastic on the connectors is pretty fragile now).

Re:924 Years and nothing has changed

[tomhudson]

We can scan in the surface pits of the laser disk at high-enough resolution to decrypt the bit patterns - we no longer need the original readers.

Re:924 Years and nothing has changed

[Lucidus]

Except that laser discs are analog!

Re:924 Years and nothing has changed

[Bob Cat - NYMPHS]

I invite people to read more about your inaccurate statement. Copy paste much?

http://www.si.umich.edu/CAMILEON/domesday/domesday.html

It's a NEW Domeday, not a scan of the old one. Emulating old computers is not that hard. COPYRIGHT seems to ne the real problem.

Re:924 Years and nothing has changed

[QuantumG]

Meanwhile the original book is still readable after more than 900 years.

Sweet, got an url?

Re:924 Years and nothing has changed

[adavies42]

pioneer still sells a laserdisc (video) player. how hard it would be to use this to read a LD-ROM (or whatever these were), i have no idea.

Re:924 Years and nothing has changed

[elronxenu]

Because they forgot key parts of the process:

• Keep it simple
• Make lots of copies which are readily available
• Keep converting to new formats over the years

The UK fouled up by inventing new proprietary storage formats which needed custom hardware and software to read and process the data. The laserdisc needed a special laserdisc player and a BBC Micro. The BBC who produced this were years ahead of their time and had to invent a lot of stuff. Unfortunately the rest of the world invented a lot of different stuff, which is what we use today.

And how many of these systems were produced? I don't know, but they cost 4000 pounds each which is a significant investment for a school and certainly the high price reduced the number of items which were sent into the community.

Even though we have extracted the data from the original formats (and also obtained improved images by re-mastering original video footage) it seems that one of the main impediments to putting this data online is copyright - the contents of the 1986 project won't be out of copyright until 2090!

The above two points come together with "keep converting to new formats". If your stuff is all proprietary, it may be hard to convert to new formats. If your stuff is copyrighted, you may be able to convert it but you can't distribute it, and widespread distribution is one of the requirements of effective data preservation.

The data which was produced in 1986 wasn't lost and won't be lost. People are working with it and upgrading it. However, you won't be able to see it, primarily due to the shortsightedness of the original project.

So loss of digital data is not so much a technical problem, more a social problem, of shortsightedness in creation, distribution and copyright.

Kinda like the BBC's lost videotapes of Monty Python (or was it Dr Who?) ... priceless recordings were allowed to degrade and become unusable, were thrown away, or were overwritten ("media re-used"). I don't mean to point the finger only at the BBC - NASA did it too. Lack of foresight, folks.

Re:924 Years and nothing has changed

[tomhudson]

Doesn't matter - so are those old shellac 12" records, and the technique works for decoding them.

Re:924 Years and nothing has changed

[imakemusic]

Wasn't it that Dr Who was recorded over the top of NASA footage?

Re:924 Years and nothing has changed

[Going_Digital]

Actually the BBC domesday project was LV-ROM disc, this is not what you might think it is. Basically it is an Analogue Laser Video disk where the Audio tracks have been used to store digital data to go with the analogue video frames. So the only part of this that is digital is the text that goes with the pictures. The low quality video frames and text data have been extracted from the laserdisk. The reason this has never been re-issued has nothing to do with technology, but everything to do with copyright. The BBC are not able to re-issue the material as they would have to seek permission from almost 1million contributors.

Re:924 Years and nothing has changed

[Going_Digital]

http://www.nationalarchives.gov.uk/domesday/

Re:924 Years and nothing has changed

[Going_Digital]

Oh please you haven't a clue have you. In 1984 when the Domesday project was conceived there were no standards for mass storage devices, CD-ROM was nothing more than an idea without any standards. But here is the point that you are missing at the core of the BBC Domesday project were thousands of photographs showing life in Britain in the '80s. There was no system that could handle the data required to hold thousands of digital pictures, not to mention the fact that no computers had the graphics capability to display digital pictures. It is all well and good for you to criticise what was done then now that you have the benefit of hindsight and can see how technology has progressed now but here are some of the problems. If the project were to be re-run today with the benefit of technologies such as JPEG picture compression and storage technology like DVD-ROM it would still be a struggle to fit all the video, pictures and data on to a single DVD. Back in 1984 a 400k floppy disk was state of the art as was 16 colour graphics. LV-ROM was a new technology developed by Philips that offered a viable solution by offering a hybrid system that was able to overlay video frames and computer data and it was a technology that philips expected to be a successful standard. However philips not having the luxury of time travel could not see that people would prefer the CD-ROM format due to it's small size and that computers were now able to store images digitally making CD-ROM the format of choice. The BBC did the best they could with the technology they had at the time, sadly the technology moved in a different direction and left them behind. If the BBC deserve any criticism it would be that they made some mistakes along the way that have prevented this resource from being converted. The main mistake is that they were liberal with their legal approach and did not get every contributor to sign away entitlement to everything including their brain cells so now it is a legal problem that the BBC can not fix with todays modern copyright laws. The second is they lost the film slides of the photos so the only source for the pictures is now a second or third generation video frame.

Re:924 Years and nothing has changed

[elronxenu]

You say I haven't a clue and then you agree with everything I said and I agree with everything else you said. What the hell?

Let me try to put it in simpler words that you can understand.

The BBC did great work in 1984 to get those images, video, text and etc onto a system which they could sell to consumers. But they stuffed up by thinking that it was a "do once" project and not realising that 20+ years later, people would still want to see/use this thing.

For longevity it is necessary to migrate to new technology over time. This is particularly important for the BBC since they invented the technology they used - as JPEGs and HTML and so on became popular the BBC should have progressively converted to newer formats.

And the main point of my post - that we're dealing with a mostly human problem - is explaining that BBC should have obtained enough copyrights to reuse and reissue the images/video/etc later. Copyright law wasn't that much different in 1986 to today (certainly shorter durations). And BBC didn't preserve their originals very well as you said. Those are human problems, not a failure of technology.

Re:924 Years and nothing has changed

[elronxenu]

Actually the NASA footage showed Dr Who and the Tardis and aliens, so it was incredibly important that the videos never see the light of day.

Re:924 Years and nothing has changed

[Overzeetop]

I think you're mistaking "state of the art" with "common desktop" capabilities. In 1984, 400k floppies and 16 color graphics was mainstream for desktop computers. Hell-Linotype was already producing machines which were used to manipulate full color photographic images and John Lasseter, who would head Pixar (to be formed in two years) created the fully animated 3D computer short "Wally B" while at Lucasfilm. State of the art was quite a bit more advanced. That's not to say they didn't have challenges, however it is particularly short sighted of them to not transfer the data to new media as it became available.

Lots of other things to consider

[syousef]

In my own quest to preserve my digital photos, I've created multiple backups on hard disk including a remote backup which gets updated every few months. I use different disks created by different manufacturers and buy new disks every couple of years (but do not throw away old copies).

I've recently come across another aspect that isn't addressed by the article. Data that is in use in an online copy can be modified (including corrupted).There is no point in copying/propagating data if the data you are copying is damaged. Typically this has happened when I've tried DAM software like Lightroom which will modify the original file despite claiming to be non-destructive I have no proof that photos were re-encoded or quality was reduced but I do know original files were altered, and I want an original unaltered file preserved

Most people when they backup files do very little verification to ensure the files they are copying today are the same files that were created 5 or 10 years ago. They rely too much on backup software to do this for them, with no attention paid to what's happened to the data between copies. To keep this under control I've started putting checksums on all my photo files, which I check when I create a fresh copy.

Of course where my photos are captured in a proprietary format I copy to an open or at least well documented format (typically jpg, sometimes also tif). This is done as soon as I transfer the photos, which are not removed from the camera card until i have 2 additional copies. So I shouldn't have the same issues that the author had assuming jpg can still be read throughout my lifetime.

--
Sammy

Re:Lots of other things to consider

[Anonymous Coward]

Tip for maintaining files unaltered: Use disk image files and mount them read-only. Create checksum files for the files in the image, then create a checksum file for the whole image and make copies. If you also store your data on CDs or DVDs, use ISO images instead.

Re:Lots of other things to consider

[rubycodez]

bad choice! jpeg is lossy format, information is deliberately dropped to make an approximate reproduction!

you're like the guy in the India Jones movie who drinks from a fancy chalice, has the flesh and guts dissolve and burn from his bones: "...he chose....poorly....."

really, if you value your work do a little research, maybe standard such as "TIFF Revision 6.0 Final" or similar should be used, and perhaps with widely known and well documented lossless compression.

Re:Lots of other things to consider

[syousef]

Tip for maintaining files unaltered: Use disk image files and mount them read-only.

Try doing this on Windows XP for a USB hard disk. Official method is to make all USB drives read only. There are clunky kludges out there for getting past this but nothing usable long term.

Also I do make edits of some files, so I need a "working copy", and once I've made these edits I have to keep them. Checksums on the working copy on individual files seem to be the best way to go.

Re:Lots of other things to consider

[swillden]

bad choice! jpeg is lossy format, information is deliberately dropped to make an approximate reproduction!

That's the SECOND bad choice!

The FIRST bad choice is taking digital photos! They're inherently limited approximations of the actual scene; numerous issues with optics, internal filters, sensors and digital post-processing lose tremendous amounts of information.

Actually the ZEROTH bad choice is taking photos at all! Film has its own problems, and not only that, any single-lens image capture device automatically discards all depth information!

Seriously, the tiny amount of information discarded by JPEGs is negligible in nearly all cases. And I speak from the perspective of an advanced amateur photographer who shoots everything in RAW to ensure that I have every possible bit of information available for me to use in post-processing (a big part of which is the process of deliberately discarding information that I don't want in the image).

If the JPEG looks good enough on your screen that it shows what you care to see and it's what you want to keep, by all means keep it. Bothering yourself about putting everything into an "archival, lossless" format will just bloat your image store and perhaps make the whole thing too bothersome to worry about.

If you're going to put a lot of effort into image preservation, what's more important is to focus on preserving the contextual information that NEVER WAS capture by the camera. Dates, locations, people, events... that contextual data will add a lot more value to future users of the data than what gets discarded during JPEG compression.

Re:Lots of other things to consider

[cpghost]

Instead of just putting checksums in your backups, why not add (block-level) reed-solomon error correcting codes? Like in parchive for files, and dvdisaster for whole DVD-R images...?

Re:Lots of other things to consider

[syousef]

Instead of just putting checksums in your backups, why not add (block-level) reed-solomon error correcting codes? Like in parchive for files, and dvdisaster for whole DVD-R images...?

Those algorithms are suitable if you plan to recover partially corrupted data. If data is corrupt, I'd prefer to restore a fresh copy of the file from elsewhere, and would not trust error corrected files. Also I'm not familiar with the tools for doing so on a hard drive. I would of course be using such algorithms implicitly if I was using DVD storage, but I've found it to be unreliable and packages of 4.7GB are inconvenient.

Re:Lots of other things to consider

[Kjella]

As someone that's collected 100,000s of pictures and verified them against CSVs, I can just say that this "bit flip" paranoia is in fact mostly paranoia. 99% of the time it's a complete media or process failure like say a broken network connection or computer freezeup or whatever. And even on the few occasions it does happen it's often not very visible, and normally a single picture is not the big deal. It's losing the whole collection with all photos of some family member or event that is a disaster. If it was really such a big problem as some make it out to be, applications would randomly stop working all the time, as they're very sensitive to the smallest change. That just doesn't happen.

Re:Lots of other things to consider

[Anonymous Coward]

Should not any decent file system do it itrself?

Re:Lots of other things to consider

[rubycodez]

"If the JPEG looks good enough on your screen"

no way, screen only has one or two thousand pixels on a side, anything more than 4 megapixel image can't be accurately represented.

Re:Lots of other things to consider

[swillden]

Zoom 1:1 to see the detail. I do a lot of big prints of high-resolution images and I have no problem using a screen to get them to look the way I want.

Honestly, for most consumer cameras there's no reason to have anything higher than 2-3 MP anyway. The optical resolution isn't much higher than that anyway, and cramming 10 MP into teeny sensors uses means loads of noise.

Self-correcting problem

[drDugan]

we are generating data far, far faster than we can save. We have for some time, and while trends for storage are catching up, we will always be able to generate more than we store, as a function of how computing and communications work.

So what to save? The Director of the NLM had a unique insight on this exact question: [paraphrasing] "What is used, is saved." Basically, its the utility of information, that information that people find useful and actually use is the best proxy for long term value. The good thing is that all people are motivated to store and maintain the data they find useful, or their constituents or customers desire. As long as people keep wanting data, it will be stored and available.

This is a very different situation to real-world archeology. In the digital, connected world we can access data today once it's publicly available, evaluate it and use it if we want. There is no dust that covers old data, it does not get buried...

Re:Self-correcting problem

[Anonymous Coward]

There is no dust that covers old data, it does not get buried

Unfortunately that is not true. Bit error rates are staggering when data is copied unprofessionally (i.e. uncontrolled). Digital decay is real. For example, every copy passes through RAM, which is not perfect. I once had a system which passed every test in the Memtest suite but one, where one bit flipped about 50 percent of the time. That system introduced about one bit error per gigabyte copied. Since it was otherwise stable, I didn't notice the problem for months. When I finally did, I found dozens of files with invalid data structures and there must have been many more where data was changed in places that didn't create a structural error. Unfortunately the Windows world doesn't have a file system with automatic checksumming, so I'm doing that manually whenever I archive files. Still doesn't protect against bit errors while I'm working with the files. I do test new systems more thoroughly now though.

People also tend to work with their data, but since they are not educated in the underlying technology or have other priorities than preservation, lossy formats are used over and over again, accumulating compression artifacts. Another source of unwanted cruft is format conversion: Data which is in use is constantly converted into the preferred data format of the person who uses the data. Since two formats hardly ever support the same features, every time data is converted, something is lost.

Point being, even if we rely on data being archived implicitly by those who use it, we should still work on ways to make their copies more true to the original.

Re:Self-correcting problem

[guspasho]

What we preserve based on our present biases may not be what interest future archeologists. It may skew their perception of us as well. Imagine if all they found of us was the porn.

Re:Self-correcting problem

[Anonymous Coward]

You've said it correctly. Important stuff will get copied and transfered, of course we'll loose some things but the argument of a dark age in knowledge is plain idiotic.

Hell they print more books now than the entire history of man kind, we're talking about data on a level that would be impractical to save in any other manner so lets get over the fear of it and embrace the future already.

Besides, I'm sure in a million years they will find all they need to know about human anatomy and mating habits from thrown away hard drives alone.

Re:Self-correcting problem

[Buelldozer]

There are two problems with this:

1) Wanting gaps. What happens when knowledge lies on a shelf, unwanted and unused, for two decades and then someone needs it again?

2) Intentional destruction for political, economic, or personal gain. Lots of people would LOVE to recall every history rebook and make a few edits to "improve their accuracy".

Re:Self-correcting problem

[Anonymous Coward]

Yep. People used to lose all of their information instantly. Now we save it for a while before we lose it.

Silicon Is the New Stone Tablet

[rebelscience]

Forget CDs, DVDs, magnetic media, etc. All data should be stored in solid state devices. Google knows.

Loss of all technical knowledge

[Punctuated_Equilibri]

The bigger danger is there is some major event like a plague and not enough people are left to maintain a technical society. Who would know how to make a microprocessor, or even refine gasoline? Tan leather? Grow crops? We could be back to the stone age in a single generation.

Re:Loss of all technical knowledge

[LWATCDR]

I think we would go no farther back than the steam age. Really most of that information is still in paper books.

It's more than just web pages.

[OrionSeven]

Archives.org isn't enough. The digital problem is more than just web sites. Every state records legal files with various methods, platforms and formats. Tiff's, JPEG's, WAV's, PDF's and more are the real heart of this issue along with web sites.

The Washington State Digital Archives (http://digitalarchives.wa.gov/) is already taking on this issue and has been for five years. Hopefully, more than just a few state will get serious about this issue.

We may have already lost files from 15 years ago, but that doesn't mean we have to loose files from this year.

(Full disclosure, I work for there.)

Burn them all

[gmuslera]

At how much Farenheit the digital records combust? Wont be so dark that ages, at least while the fire last.

A century later, we will still will find buried snapshots of wikipedia on devices like WikiReader.With paper books making copies is expensive,to one kind of device usually, and takes a lot of space. Digital records,in the other hand,could be put in a lot of ways, but what must be preserved is how to decode or interpret it (using open formats for it could help a bit there).

Cranberry DiamonDisc: 1000 years

[Bleek II]

The Cranberry DiamonDisc is a 1000 year option already on the market. They aren't cheap but they should come down in price if they're able to get a enough costumers to bring the supply up.

Re:Cranberry DiamonDisc: 1000 years

[turbidostato]

"The Cranberry DiamonDisc is a 1000 year option already on the market."

Are you sure? Do they come with a 1000-year durable reader?

What use would you give now, less than twenty years away, to a 1000 year durable 8" floppy disk?

Um?

[drDugan]

This is what counts for science nowadays?

http://www.americanscientist.org/include/popup_fullImage.aspx?key=vo50G9YwnF6SwlOk2usL5R9EyqRLsNX+YiPzweX/0ZsH0IeSOOXIBip7qwN2/ZRY

Look carefully at the 'digital encoding' of the "simple tone" sine wave. ??? Really? What encoder is that?
cf. http://en.wikipedia.org/wiki/Fourier_Transform

Re:Um?

[imsabbel]

Glasshouse, meet stone.

Or in other words: Are you functionally retarted?

Look up the meaning of "digital". Hint: it has nothing to do with all the stuff happening if you _compress_ digital signals.

Re:Um?

[Pence128]

Pulse-Code Modulation

Re:Um?

[drDugan]

As I understand it, all modern audio encoders encode in the frequency domain. The graph and description
don't mention this, but imply amplitude quantification/digitization, then damage, and reconstruction only
in audio amplitude. Not an expert though...

"During encoding, 576 time-domain samples are taken and are transformed to 576 frequency-domain samples."
http://en.wikipedia.org/wiki/MP3#Encoding_audio

"The signal is converted from time-domain to frequency-domain using forward modified discrete cosine transform (MDCT). This is done by using filter banks that take an appropriate number of time samples and convert them to frequency samples."
THEN... "The frequency domain signal is quantized based..."
http://en.wikipedia.org/wiki/Advanced_Audio_Coding#How_AAC_works

Best way to avoid a dark age...

[Anonymous Coward]

Best way to avoid a dark age: prevent a fundamentalist cult from exercising its fantasies of domination / cultural supression.

Includes: Christianity, Islam, Scientific Materialism, among others (these happen to be the most successful to date).

Re:Best way to avoid a dark age...

[strangelovian]

Tell that to the Romans. Another dark age is definitely coming (in fact it's probably already started), but it ain't gonna be the fault of religion any more than it was in the past. I suggest you study some more history if you think religion causes dark ages. Here's a good place to start: http://www.darkage.fsnet.co.uk/Manuscript/The%20Coming%20Dark%20Age.pdf

File formats vs physical media

[Chuq]

This topic involves two vastly different things:

- File formats - easy - just make sure everything is stored in an open format, or something so ubiquitous its as good as an open format (odt, txt, jpg, pdf, csv, ogg, etc) and it will be readable forever.

- Physical media - this is the risk - most new machines these days can't read 3 1/2" floppies, let alone anything older, but so long as you migrate contents of your old physical media onto new media formats - AND you have multiple copies of important stuff - that shouldn't be a problem.

Re:File formats vs physical media

[enoz]

- Physical media - this is the risk - most new machines these days can't read 3 1/2" floppies, let alone anything older, but so long as you migrate contents of your old physical media onto new media formats - AND you have multiple copies of important stuff - that shouldn't be a problem.

I get your point but you've used a terrible example. If you NEED to read a 3 1/2" floppy, then you can go and buy a new or used floppy drive for under $10. Most mobos still come with a floppy interface, just the drive is not bundled because most people don't use them.

Re:File formats vs physical media

[mattpalmer1086]

You are right about the issue being file formats and physical media, but you are dead wrong about the risks. You have to distinguish between rescuing something which is already obsolete vs. keeping something "alive" as you go along.

If you're talking about rescuing obsolete stuff, then it ranges from hard to impossible, mostly because it generally isn't economic to recreate older media readers. For file formats, it's also expensive, but for some formats will be possible.

If you're talking about migrating information, then physical media is essentially irrelevant, and the process is a no-brainer (large volumes may pose problems, but they aren't media issues). However, migrating unstructured information with complex content (e.g. documents with macros) to new formats is extremely hard (if not impossible), as each format has different capabilities. It just isn't possible to transform the information losslessly. Over time, this inevitably results in information loss.

Terrorism is our savior!

[Jah-Wren Ryel]

With all the pushing by law enforcement for permanent archiving of everybody's web use the problem will solve itself!

Rah! Rah! for terrosists - they hate our freedom but they have saved our culture from fading from history!

Re:Terrorism is our savior!

[Sulphur]

We've got top people on it. Top!

Ehhh... not so much.

[CODiNE]

Virtual machines really eliminate a lot of those concerns. But what we really have to worry about is silent bit rot. I've found a few old files of mine that are corrupted. Not cool. ZFS and drobos... I don't really see a good end-user ready backup system that verifies data integrity.

Please don't troll

[syousef]

bad choice! jpeg is lossy format, information is deliberately dropped to make an approximate reproduction!

Many cameras only capture in a lossy format such as jpg. Even those that have RAW sometimes use lossy RAW. Losses only occur once per save. So to mitigate you don't modify files repeatedly. If there is a need to do this, go back to the original, save to TIFF and edit from there. So long as you have the original preserved you can always reapply any edits.

you're like the guy in the India Jones movie who drinks from a fancy chalice, has the flesh and guts dissolve and burn from his bones: "...he chose....poorly....."

I can't tell if you're trolling or just being melodramatic.

really, if you value your work do a little research, maybe standard such as "TIFF Revision 6.0 Final" or similar should be used, and perhaps with widely known and well documented lossless compression.

TIFFs are a poor choice unless multiple edits are going to be made. They slow down current hardware immensely and aren't widely supported. Try flipping through 12MP tiffs using Windows fax and picture viewer on XP.

I think you ought to have some idea what you're on about before you criticise so floridly.

Re:Please don't troll

[rubycodez]

was being melodramatic for fun,but I've had photos butchered by subsequent moving from one jpeg software to another.

Yes, Windows XP will choke on your little ~52 megabyte TIFF. Real operating systems won't, Mac OSX with sufficient RAM and good software and there's no problem at all. Linux does ok too, though available free software not as high quality.

Re:Please don't troll

[syousef]

I've had photos butchered by subsequent moving from one jpeg software to another.

If you're continually re-saving the work, you should be working in TIFF. Simply opening pictures up in software isn't going to ruin them. Going back and forth between lots of programs and saving in each will. Any more than 3 saves required, and I'm using TIFF. 1, 2 or 3 saves and the deterioration is neglible if the files are decent to start with.

Yes, Windows XP will choke on your little ~52 megabyte TIFF. Real operating systems won't, Mac OSX with sufficient RAM and good software and there's no problem at all.

Pity about vendor lock in and "rights management" (where the user only has the right to do anything if Apple deems it wholesome) on the Mac platform. I'll stick with windows.

Linux does ok too, though available free software not as high quality.

Old Sierra Games

[AnotherAnonymousUser]

Anyone who remembers some of the old Sierra games knew how they occasionally had glitches or bugs that would cause the game to crash completely. Though Sierra would sometimes drag their asses about support, at one point or another the fans would patch the games themselves and publish the unofficial patch to the web. Quest For Glory IV was a perfect example of this; a fantastic game, but it had a few very memorable glitches that would cause the game to crash. The problem was that these fan patches were most often hosted on Geocities or Angelfire sites, most of which have since vanished from the world, leaving the games unplayable, unable to be completed, and unfixed hosts to whatever problems plagued them at the time of their publishing. Something that would have been a "popular" fix even just a few years ago is now irretrievable; forums point to dead links, users have long-since abandoned the posts, and the files themselves are nowhere to be found because of the death of the hosts. Obviously it's minor in the grand scheme of the history of the nation, but it *is* a good indicator of just how much can be culled within a very short period of time.

Re:Old Sierra Games

[JoeMerchant]

Arguably, if the community cared, they would have kept the patches alive elsewhere - Geocities and Angelfire weren't the only possible hosting sites.

Sometimes, a hack like that is interesting when it's done, and hopelessly boring later, not even worth the time to preserve. Other times, copyright laws grind slowly away at "illegal works," maybe not suppressing them when they're popular, but getting them in the end anyway.

Re:Old Sierra Games

[TandooriC]

With the fast pcs nowadays you could only finish QFG 4 with a thief and you can't complete the initiation in QFG3 with a fighter.

Anal Grannies, Weapons of Mass Destruction...

[DVD9]

Gonzo, War on Terror, First Person Shooter...

It would indeed be a tragedy for civilization if such data were lost and the mind of the early 21st century American went unrecorded.

VERY interesting topic

[adosch]

This is something that I've seriously taken a look into on the personal side of things. I look at all the digital data I've collected (and lost due to a drive failure, virus, corruption, disaster, ect.) over the years and it really makes your head go foggy. I only hit this realization putting together a wedding anniversary party for my parent's together in the last few months. My parents brought over bucket loads of photos and keepsakes that I have to rummage through for an overhead slideshow. On top of them being (thankfully) highly organized with their personal keepsakes, it far superseeds what I have for my own family. My wife and I went back and we literally have a 'digital divide' in the last decade for any tangible photos. Most of our memorable moments were done with a digital camera, which is great, but we have SO many dribblets of photos here and there on this burnt CD or that external storage device, ALL of which can get lost much easier, broken or misplaced FOREVER than a big ass, heavy rubbermade toat of pictures my parents have (negative included, I might add).

So I ask myself, what if my copy of my copy of my copy is corrupt? I'm screwed. What if I have something in an unsupported format that I can't find any support for? I'm screwed. What if I have a photo at 320x240 resolution and I want to make a 8x10 photo of it and put it on my wall? I'm screwed. We've successfully stove-piped ourselves for a high rate of non-reproduction of our valued items along with a staggering rate for failure on the mediums we've chosen for them.

I've come to the conclusion that tangible is becoming an obsolete word when it comes to anything I like anymore: music, movies, photography, books, news, conversations, ect. I don't think there is a way getting around it that I can see.

Re:VERY interesting topic

[CharlyFoxtrot]

So I ask myself, what if my copy of my copy of my copy is corrupt? I'm screwed. What if I have something in an unsupported format that I can't find any support for? I'm screwed.

What if you lose all those paper pictures and negatives in a fire, a flood, lose the book when moving, get burgled, ... ?

What if I have a photo at 320x240 resolution and I want to make a 8x10 photo of it and put it on my wall? I'm screwed.

Tell that to all people who had polaroid camera's.

Re:VERY interesting topic

[adosch]

What if you lose all those paper pictures and negatives in a fire, a flood, lose the book when moving, get burgled, ... ?

Burgled? Maybe if they had, mixed among their child's 4th birthday party, the photo of who shot Kennedy.

Re:VERY interesting topic

[CharlyFoxtrot]

Burglars could destroy them for kicks while tearing through your stuff looking for valuables or you could happen to have them in a bag because you took them to some fiends and they swipe the bag, or you left them in the car and they take the car.

They will not care

[kikito]

The future generations will just not care about us. Just stop thinking about this already.

Consumer demand will fix this.

[Seor Jojoba]

I'm not worried. We are pretty soon going to have a bunch of people that are heartbroken about their data from 10 years ago being lost. The travel photos, the e-mailed love letters, the brilliant blog posts. And these people will create demand for longer-term storage and data collection techniques we don't have now. Why should it happen in the near future if it hasn't already? Because we first needed a generation of people that use computers and the internet as the primary way of expressing their life. Nobody was in that boat ten years ago. Now anybody reading this is. So consumer-grade "lifetime" storage options will enjoy a more prominent place on the market. And if you can get some old data to stick around for a half century or more, the value of it bumps up to "time capsule" status. Which means somebody might think to archive your mess of media around the time you die. Maybe some younger cousin of yours will take care of it. Heck, funeral parlors might offer data archival as a service 20 years from now.

Re:Consumer demand will fix this.

[turbidostato]

"consumer-grade "lifetime" storage options will enjoy a more prominent place on the market."

Long-lived consumer-grade gadgets, no matter if logical or physical, are at odds with the corporate-grade greed for benefits: you won't sell a device that won't be replace in fifty years if you can find the way to sell fifty one-yeared devices in that time span, do you?

Look around: which one do you think is the best bet? Consumer or corporation interests?

Don't trust cloud computing.

[Lazarian]

If you want to preserve your data, backup your data yourself, and keep it on its own storage medium. There seems to be a growing impetus where "cloud computing" and "thin clients" are envisioned to replace traditional architectures where data is stored and decoded by the individual who owns/created it. I'd rather store my data myself than ask permission to access it through the equivalent of a 1980's green screen dumb terminal from some corporation who's interests run contrary to mine.

Practical example : Classics emulation

[DrYak]

The main way ancient writing reached us is because someone copied it. Lots of copies. {...} With all the copy protection and encryption on our media today, can we ever copy the data and be able to decipher it again?

(And as another example of copies being important for preserving : Fritz Lang's Metropolis got recently another 30 minutes of its missing part recovered from a copy located in Argentina)

After a long enough time, virtually any DRM measure end-up being broken. What only matters is time, resources and some clever tricks (to avoid waiting until universe heat-death while bruteforcing a 4096bit key).
So DRM has only 2 direct effects :
- it annoys legitimate users everywhere with no practical reason.
- it forces the basement-dwelling teen with too much free time on their hand to wait until 2 weeks before official launch date, instead of 3 weeks before, because it took 1 week to the pirates to find a way to break the DRM.

This implies 2 results :
- That the 99.99% of pirate users, will never ever interact with the DRM nor be affected by it in any way.
- The important part : DRM protected piece of data will get copied, eventually and a lot. Lots of copies will exist and virtually 99.99% of these copies will be the "pirated" copies. Be it legal backup or unlicensed copies.

So in the end, the DRM-protected data will survive, only not the DRM version itself, but the DRM-free version as found on The Pirate Bay and similar. Case in point : Classics emulation.
Most of the companies which produced the game we played as children are now belly. Of the few remaining, few of them have kept the assets of their old production. Few of them are interested in doing anything with these old assets. The few who do, generally do modern re-imaging and re-interpretation, rather than re-issuing the old.

So in short, if you ever wanted to pull back some of your children memories out of the grave, don't count on the original companies.
Some time you can find still working vintage equipment and media - but these will eventually break.
Today, the biggest part of these oldies are available ... as image of pirated disks. It's practically sure that, if in 2010 you want to play the same game as in 1985, you'd probably see a cracktro in the beginning.

All your Commodore C64, Amiga, etc. favourite games are currently best sourced from download site which contain warez copies that were carried over back from that era, while at the same time the companies went belly up and/or let their assets rot.

So, in 25 years, when most of the current media companies have either disappeared, or completely forgotten about today's media, your children's best way to find a copy of them to remember fond memories, would be finding a copy which will be the digital descendant of what's today on pirate bay.
Yes, **AA, today's EVIL pirate, might be tomorrow's heroic archivist.

In 25 years, when the current maker of

Sponsored by?

[Anonymous Coward]

Was this article sponsored by HP, Epson, Canon, Lexmark, and Xerox perchance?

there's not enough stone...

[Anonymous Coward]

...to chisel all my ascii porn

Making a hard problem harder

[MpVpRb]

Preserving digital data is inherently hard.

Not only do you need to preserve the bits, but you also need to preserve the knowledge about what the bits mean.

So...instead of addressing this issue as important, the content owners have decided to add another layer...

Now, they encrypt the data, to prevent copying.

This makes the problem A LOT HARDER!

The content owners are the ones to blame if we lose entire decades of art and culture.

"We need copies. Lots of copies."

[sfled]

Seriously. Many copies. Multiple, ad-nauseum uber redundancy. And, so what about that DRM crap? Is it _that_ important to preserve pop music? If so, when did DRM ever stop us? Burn a CD/DVD/Blu-Ray and then RIP it, upload the thing. Put it on a hardened RAID, what...ever.

True, technology is an ever more complex cycle. I guess we should try to get the info/code down to lowest common denominator. Text? If so, what language? Boggles the mind..but if the unthinkable happens then maybe we can assist the great minds millenia hence rediscover what we did. What we did right, and how to avoid what we did wrong.

So I repeat, whatever the content is, tech records can be fragile. To paraphrase, "We need copies. Lots of copies."

Microfilm lasts 500 years...

[BlueScreenOfTOM]

Last I checked, Microfilm (and Microfiche) was guaranteed to last 500 years by Kodak. Unlike JPEGs, you can read Microfilm with a magnifying glass, and speaking from experience both writing and reading, the quality isn't bad. It's also not horrendously expensive once you have the writer (I believe a Kodak i9600 Series Archive Writer sells for somewhere around 35-40k, depending on the model you get).

I wouldn't fool myself into thinking Microfilm is some magic solution to our digital storage problems, but it does go to show that there are ways we can save really important data. Given current technology, we could make something similar to Microfilm that didn't require professional development...

idea: opt-in mirror as you browse option,encrypted

[Anonymous Coward]

what of this idea: an opt in mirror as you browse option for web browsers
allowing anonymous mirroring via tarballs sent to a database and hashed.
some type of plugin could offer this so everything you browse is mirrored
automatically. there could be an option for the web browser's plugin to
also include this feature in the background, similar to how distributed
computing projects work, using a user set predefined amount of cpu time.
this could also be done through a screensaver, the plugin could work
through a fork of tor or something like it, where the plugin works in
such a way where the data is encrypted from the user and anonymized without the user or the collection bank at the end knowing who it came from, only hashes are generated and stored from each of the browser's anonymized submissions. the central databank for all this information could also work in such a way somehow offering torrents of hashed tarballs, allowing for further spread and storage of the data

You people dissapoint me

[Anonymous Coward]

This story was posted an hour and a half ago, and not a single one of you has made a snarky reference to Foundation.

I see it is easier...

[blai]

... but not by much. You'll still see

10101000101010010101010101100100100010010101000101001

and have no clue.

Gmail Paper

[illiteratewithdrawal]

Google does one better: Gmail Paper

What software are you using for checksum?

[Hyperhaplo]

... and how are you storing the checksum information?

It would be nice to have checksum information, eg MD5, on *all* files in any OS (in the file metadata).. but given that this is not a critical requirement for today's desktop systems I don't see it happening any time soon.

What I'd like is to have all files hashed natively by the OS and have that information available when required.

Re:What software are you using for checksum?

[syousef]

... and how are you storing the checksum information?

My checksums are md5 hashes stored in a text file alongside the image files. I am not expecting a malicious attack on the data. If a copy is compromised I expect to fetch the data from a copy that isn't compromised.

Re:ffs.. the "zomg how to preserve" story -again-!

[bsDaemon]

me too!

Re:ffs.. the "zomg how to preserve" story -again-!

[EdIII]

when you've seen one Starbucks next to a McDonalds next to a Walmart, you've more than seen them all.

The exception will be the Adult Superstores. I expect a lengthy and protracted archeological excavation at each site with all of the items meticulously cataloged and considered for their 'function'.

Global warming already lost the data

[Anti Cheat]

Well according to that last round of climate data falsification there has already been a digital dark age. The claim is the original source data has already been digitally lost. So no one can prove their numbers wrong, we just have to take their word for it, even though these same people have been caught fudging data that can be proven and not victim of digital dark age. I wonder if the tax man will believe me if I said it was a digital dark age so please take my word for those huge tax deductible donations I made.

This is a longstanding problem

[mikein08]

And it's a big one. I've spent the better part of 30 years working administrative applications in IT. Every time you change from, say, one accounting system to another, the problem of how to preserve and be able to look at meaningfully old data become significant. Yes, you can have backup takes and disks running out your ears: but you also have to have the formats for this data, you have to have all the progrmas which accessed and massaged this data, you probably have to have the OS these programs ran on, and you have to have the compiler(s) used in the old system. Sounds pretty daunting, doesn't it. Or you can convert all the old data from old system(s) to new one. Do you know how much resources management will allot to that? But all of a sudden, someone has written an article about it, so it's a hot topic again. Sigh ... there's nothing new ... MK

Blast from the past

[aktzin]

I thought this sounded familiar. I found a comment I wrote over 2 years ago for a similar story: http://slashdot.org/comments.pl?sid=207582&cid=16922754

I Can Still Play Bionic Commando On My C64

[fyrie]

I can still play the copy of Bionic Commando I bought for my C64 20 some years ago, but the ability for me to legally play the copy of Bionic Commando Rearmed I bought two years ago is in jeopardy since the company folded and the software needs to phone home on install. They never released a patch to clear the DRM. If a clearing of DRM on "dissolution" is not in the software license, then don't expect their to be one.

Re:I Can Still Play Bionic Commando On My C64

[Anonymous Coward]

Let that be a lesson to you, pirate everything. Irony is when DRM actually /encourages/ piracy.

I know, I know!

[Anonymous Coward]

I know the answer to this one! Put librarians in charge of IT!

Re:I know, I know!

[tobiah]

mod parent up :-)

This is a real problem

[ParanoiaBOTS]

As someone who has done quite a bit of work in the genealogy line of software development I know what a problem this can be. The best alternatives we have currently for storing digital media last
There is however a company out there that is working on a project they call the "Millenium disc" apparently it can hold roughly as much data as a CD but it has an expected life of ~1000 years. It works by meshing the ideas of hieroglyphics and a traditional CD. This was an idea that was presented at the Family History Technology conference at BYU a few years back. I am not sure where the project is now, or even if it is still alive. But either way, we need a way to come up with a long term storage solution.

Re:This is a real problem

[cenc]

Yea, if someone is going to solve this problem it will be the Mormons. They got to be throwing millions at it.

So who's curating their own pr0n archive? Anyone?

[Dr. Spork]

I think that some intrepid information preservationists on slashdot are doing their best to preserve a copious volume of our pr0n for future historians! You just know that it what they'll all want to study!

Speaking as a historian focused 1950-1990 Western

[Anonymous Coward]

Significant amounts of data have been lost due to:
# Loss of physical mechanisms to retrieve data from media
# Programs and Operating Systems that will not retrieve and convert for use stored data
# Inability to convert documents as they were produced, used, read and received into a modern or permanent archival format (what, it lost the Italics / Umlauts)

But the main problems are:
# Good records keeping policy not applied to data bases / electronic data
# Good records keeping policy not applied to electronically generated documents
# A general decline in the quality of filing and records keeping policy with the computerisation of many office correspondence tasks and with the decline in pay and skill levels for white collar menial labour employees.

At the heart of the problem is a general decline in the quality of filing from about 1965 onwards. As far as I can tell from documentary records, the increase in office duplication technology encouraged this as records no longer became difficult to copy individual items, but became roneo'd and xerox'd pieces of disposable paper.

Re:ffs.. the "zomg how to preserve" story -again-!

[xactuary]

And that story had a comment just like yours linking to an even earlier story, and so on. Slashdotters LOVE recursion.

Won't matter - there ain't no more do-overs

[Anonymous Coward]

If human race is alive 5000 years from now it will be in either of the following scenarios,

  1. we learned to live within means of this planet and most likely colonized other planets and solar systems, or
  2. we failed and a few handful of surviving humans live on a world devoid of easily accessible resources (eg. copper, nickel, silver, oil, etc.)

In first scenario, most of today's knowledge would survive as an interesting history project(s). In the latter, today's knowledge will mean very little.

So yeah, there ain't going to be any future civilizations. This is IT! This is the only shot we have. It is too late to think of "do overs". The last do-over possibility happened before humans discovered industry and antibiotics propelling our population to what it is today.

Apple

[Anonymous Coward]

Apple is the Roman Catholic Church of the Digital Dark Age.

It's called a backup.

[he-sk]

That, and wide-spread usage of open standards and free software, of course.

Online Gaming History

[Tickenest]

It's by no means the most important topic out there (and there's a personal plug here, of course), but online gaming communities have thrived for well over a decade (even longer for some forms) but most of that information has been lost and forgotten. I decided about a year and a half ago to post my old demos of my days playing QuakeWorld Team Fortress (QWTF) on YouTube, a little out of vanity, a little out of preservation, and also because there just wasn't any real footage of QWTF on YouTube. After I got done with my old demos, I decided to start posting other demos that were publicly available on the internet. Some old players found my site and made their own contributions, but I was saddened that a lot of guys would say that they lost their old demos on a hard drive crash or something. Again, not the most important stuff ever, but I think QWTF was a pretty significant part of gaming history because it pretty much defined class-based multiplayer FPS (or at least popularized it), and, as far as I know, there isn't any sort of archive like mine anywhere else. If you're interested, http://qwtf.digitaljedi.com/ or http://www.youtube.com/user/Tickenest.

like an odd sock

[timmarhy]

this is the same shit story that keeps popping up on /. ever 6 months or so.

typically kdawson posts it, what a tard.

Re:like an odd sock

[Lunzo]

Give kdawson a break. He's just trying to preserve this story for future generations by making a lot of copies.

Here's an example...

[BrokenHalo]

"personal crap" could be just as interesting and as important to future researchers.

Indeed it is. My great-grandmother was left living in the Channel Islands throughout the German occupation during WWII. Over those years she kept a series of diaries, handwritten with a fountain-pen on paper that is now disintegrating.

My mother has since transcribed those diaries, so although they are not yet available online (my mum is for some perverse reason opposed to that), they are at least preserved for posterity.

Print it

[FaytLeingod]

My Wife is very suspicious of all technology so all our photo's, important Doc's are printed This helped alot when my hard drive crashed

Future Generations

[TandooriC]

I would really love to see the expression on their faces when they find that Rick Roll video.

Tyler's Advice

[Anonymous Coward]

"Always backup in hard copy."

This is old news

[rfc1394]

I tried posting the following reply on the website the original article appeared on, but their comment system kept having an error so I'll post it here and expand upon what I would have said.

There is nothing new here; the problems of electronic data deteriorating or becoming unreadable because of proprietary lock in of various closed-source applications is well-known going back more than 20 years that I'm aware of, and certainly a lot longer than that. The use of wire recorders, player piano rolls, 78-RPM records, phonorecords now, 8-track tapes, laserdiscs, 8" floppies, 5 1/4" floppies, Jazz discs, Zip discs, and now 3 1/2 inch floppies, (and lots of other media I know I've forgotten) are all now obsolete storage media, some of which may have data which can no-longer be retrieved because the hardware and/or software to read them is unavailable, lost or forgotten.

RMS on Digital PDP minicomputers running RSX and RSTS and VAX machines and OS . ISAM and PAM on Univac VS/9 OS on 90/60 /70 and /80. VSAM on IBM mainframes (except the few places continuing to run z-System). The Control Data Cyber systems and their data file formats. Gould, Goodyear, Harris and RCA mainframes. All of these are basically obsolete, most if not all are gone, and data stored on media from those systems, if developed by a proprietary application, is probably, for all intents and purposes, lost forever even if the data is still present. The media may have deteriorated, and the systems to read them are essentially nonexistent.

Mechanisms for regular conversion as technology changes have to be provided for. This, however, requires that as the older media ages, that there be budget and personnel available to provide the conversion while both old and new media types are available. As the case of NASA cited in the article (an employee scrounged equipment and tapes on her own in order to keep the data alive until a means to retrieve it could be found), sometimes either or both may not be available.

Libraries have mentioned how their resources are stretched thin as it is, they may not have the funds or trained personnel to export old data to new media. And at the rate media keep changing this is happening more and more frequently. 30 years ago is 1980, 250K 8" disks are still in use. The 5 1/4" 360K disc is popular because of MS DOS machines. 20 years ago is 1990, and then, the 5 1/4 was still and 3 1/2 inch floppies were becoming popular, 15 years ago a reasonable medium for high-capacity storage were 100-meg zip disks. Now I don't even have a 5 1/4" drive, my computer still has a 3 1/2 but I don't have any floppies or use them, because I have a 4 gb jump drive I wear on a lanyard around my neck, and cost ten bucks.

We've gone to digital storage because it's orders of magnitude cheaper than analog. I've pointed out in previous articles that with a digital camera and 4GB SD cards, I can take thousands of pictures at an effective cost per picture that effectively rounds to zero. A single photo might take 1/2 to 1 meg, which means, without changing media, I can take upwards of 3,500 photos. Net cost is $10 when the media is bought; nothing more unless I print an image. When I take pictures, I don't take one, I take 3, or 5, or 20 because the extra pictures are essentially free and I can delete the ones I don't want later. When I was using 35MM film, each photo, with film and developing, was about 30c. A couple hundred pictures would set you back over US$50. Today, for $50 I can take more than 20,000 images.

But my sister still has an older digital camera that uses Smartmedia, She has to be careful to copy her images to hard disc when she uses it because you basically can't buy smartmedia any more and even when you could, the maximum size was 128 meg. Her photos were in the 150K size range so she can still take more than 500 photos on a 64M chip, and also the cost is effectively zero.

And for current media it's still near-zero per image. I bought a 1 te

Big Brother...

[Anonymous Coward]

Big Brother will certainly enjoy this lack of history. It's much easier to fill an empty space than to replace an existing history.

Corollary: old media wiping

[LocalH]

Back in the old days, entertainment media was not seen as something worth preserving. Many media companies junked, trashed, or wiped content because they saw no need to preserve it for the future (or indeed were even contractually obligated to do so). As a result, there are hours upon hours, nay weeks upon weeks, of material that simply does not exist anymore. The story of the classic Doctor Who junkings comes to mind, where if a certain person had been stopped merely a year or so earlier, there would be many more existant stories on film than currently exist.

Nowadays, the problem is quite similar, although on a different scale. Many media organizations currently have excellent archival policies, such that even if material is not currently aired or sold, it is at least maintained in an archive somewhere. The problem lies with smaller-scale content (publically available and private), which doesn't demand the same archival treatment. Admittedly, a vast majority of this content is shit. However, for the benefit of future generations, we should strive to retain as much data as possible, and let them determine what is shit and what is worth keeping.

I'm all for letting future generations sift through my data. They'll mostly find crap, but if they see a few gems in there, all the better.

Solution might be keep making backups + VM

[Anonymous Coward]

When I started using computers I had 1 5"1/4 floppy drive, when I got a 3"1/2 drive I made backups of my old floppies, then I got a HD, then a ZIP drive, archived all my floppy and HD inf on ZIP disks, at the same time as I kept getting bigger HD, I kept a backup of my old HD on the new one including backups of my ZIP drives, cds/dvds. It's true today I have ~6TB of data where ~1/3 of them are backups (of backups of backups), and I got those 6TB backuped as well, but I still can get my 25 years old floppies! Virtual Machines & emulation should also ensure being able to access/use those informations, it may require in the future to run emulators through emulators....
Only problem I see: in case of strong EMP/solar flare, all the word's data may be destroyed (would optical media be destroyed as well like when you put them in the microwave?)

This is hardly a problem

[Anonymous Coward]

Classic stupidization of society. What to do? How about organizing yourself? How about solving your own problems? How about thinking what data you really want to preserve and keep accessible until you die, and organize yourself accordingly? It doesn't have to be digital to be lost: NASA lost the Apollo 11 recordings even if they were in a really quirky and analog format, they could be read today, only they 'misplaced' them in their own terms.

Sheesh people. Build a ZFS array, OpenSolaris is out there, keep everything online 24/7 for 60 years until you no longer need them, let the next generation decide what to keep. If a disk dies, replace it promptly. Have backups. What more do you need besides getting off your ass?

Re:ffs.. the "zomg how to preserve" story -again-!

[xded]

Still, in previous stories I didn't find any reference to PaperBack.

It just lacks a textual description of the matrix format to attach at your centuries-lasting data.

Google's Servers

[cenc]

5,000 years from now people or aliens can just Google it.

I am only half joking. Google does (or is capable) doing what in some respect government offices use to do like the library of congress. The only problem is that it is not clear just how long Google keeps its data. Does it really not have a backup of what it crawled and cached of any sort from say 2 years ago or 5 years ago somewhere in a bunker under a mountain?

To much Data / Information

[Pec]

The problem nowadays is the excessive amount of data, there is so much that we have no information at all.

Lame sometimes

[hesaigo999ca]

The first part I laughed at was that he had forgotten to make a backup of the program he used to make and read the backups...
then I sort of forgot to read the rest, ......wonder why.... : P

Samizdat...

[BrokenHalo]

Funny that. I think our forefathers did just that i.e. they (unknowingly?) created backups when knowledge was written by hand...

Certainly not unknowingly. Many, if not most literate monks would have been perfectly aware of the fate of the Library of Alexandria. There is also evidence that there were also scribes who were not really literate in any language, but who were capable of reproducing a text without fully comprehending it. The only possible reason to do this would be to create a backup copy.

But in any case, it is worth mentioning that copying texts by hand has also been done in recent times in the form of Samizdat literature, a means of publication not uncommon even in the 1980s.

Re:Samizdat...

[cmarkn]

No, copying a book was rarely, if ever, for the purpose of creating a backup copy. It was the only way to make another copy of a book, so that it could be, for example, sold. Every time a new church opened, it needed a bible. Somebody had to run off that copy.

There was a law in Alexandria that all travelers had to turn over their books to the library so they could be copied. These weren't backup copies, these were copies that belonged to the library.

Re:Samizdat...

[BrokenHalo]

There was a law in Alexandria that all travelers had to turn over their books to the library so they could be copied. These weren't backup copies, these were copies that belonged to the library.

Actually, the library kept the original and left the traveller with a copy.

laserdisc players are still available

[vaporland]

they are available on ebay, craigslist, etc. I have bought several and they work fine!

It's not just a technical problem

[CoffeePlease]

It's a legal and social issue as well. Unlike something written on paper, works on the web are prone to disappear when payments to the service involved stop. Proposal: Advance Directives for our digital legacies

You need a paper briquette maker

[Colin Smith]

And a fireplace or wood burning stove.

Junk mail is great. Free heating.

 

Re:Great! Now I'll have to buy the White Album aga

[mattpalmer1086]

LOCKSS - Lots of Copies Keeps Stuff Safe:

http://lockss.stanford.edu/lockss/Home

Digital encoding onto the longest lasting medium

[Colin Smith]

Paper.

http://ronja.twibright.com/optar/

Stone ain't a medium, it's a hard.
 

oblig

[Lunzo]

We're robbing the future of our culture, even from things like movies with talking hamsters!

And nothing of value was lost.

Re:Great! Now I'll have to buy the White Album aga

[Lunzo]

Consider what was the very first publication printed on Gutenberg's big invention.

Hallo, Welt!