The Digital Dark Age

zygan wrote to mention a Fairfax Digital article about the possibility of a digital dark age, as a result of the increasingly short-term lifespan of digital storage. From the article: "It is 2045, he suggests, and his grandchildren are exploring the attic of his old house when they come across a CD-ROM and a letter, which explains that the disk contains a document that provides directions to obtaining the family fortune. The children are excited. 'But they've never seen a CD before - except in old movies - and, even if they found a suitable disk drive, how will they run the software necessary to interpret the information on the disk? How can they read my obsolete digital document?'"

Comments

this should be soluble.

[yagu]

Scary article. But probably too true.

In my opinion data archival screams to be handled in as simple an lowest-common-denominator a way as possible. For me, that means text for documents, and picture formats that would seem guaranteed to be around for a long time, if not forever. I'm guessing a good candidate for pictures would be something like jpg. I can't imagine jpg going away or ever being a non-decipherable picture format. Video might be a tougher nut to crack but I would guess some flavor of mpg.

Note that none of these flavors: text; jpg; nor mpg, include or imply any reliance on vendor proprietary formats (yes, I know there's a certain proprietary tinge to the picture and video forms, but they're pretty universal). So, storing and archiving for historical purposes rules out Microsoft and all of their formats. This would especially make sense considering there are already huge compatibility issues with Microsoft documents among their various versions of their products.

Also, for retrieval assurance it no longer makes sense to me to use "dead" or "inert" methods for storage, e.g., tapes, cds, dvds, etc. Instead, at least for my purposes I maintain multiple physical and current storage devices for all of my important data. This has been a recent (last three years) development for me when I started reading about early failures of the supposedly rugged storage.

So, that being the case that introduces (introduced) the need to devise a strategy for forward migration of all of may data so nothing got left behind. Fortunately, this has been mostly easy since right now the "active" storage du jour seems to be hard disk drives, and the capacity has grown sufficiently with each new generation of drives I have been able to simply roll my data forward onto the new drives with the new data with plenty of room to spare.

This shouldn't be an approach foreign to comapanies with reasonably competent data shops either. But maybe a philosophical change. All is not lost, and hopefully all will not be.

Just my $.02. ~

Re:this should be soluble.

[Anonymous Coward]

this should be soluble

That could be a problem. At least a CD won't get damaged by water.

Re:this should be soluble.

[Cruciform]

Even resorting to paper these days you want to make sure you've got archival quality equipment.

Some inkjet pages fade considerably in just two years. After a decade they may just be yellowing pages with no discernible content.

Re:this should be soluble.

[merreborn]

I'd think bmp would be preferable to jpg. bmp is to images what .txt is to text (and while ASCII is arbitrary, it's a single substitution cypher, and therefore easily crackable) -- the simplest, uncompressed format. I've written 1-bit (black and white) bitmaps by hand. I couldn't ever hope to do the same in jpeg.

Re:this should be soluble.

[jd]

I would personally opt for PNG for images, to avoid loss of data. Video almost has to be MPEG, as neither MNG nor APNG have really gone anywhere at this time and the BBC's high definition format isn't getting much adoption yet either. For audio, MP4 would seem the best choice - less loss of data, but more likely to be readable in the far future than Ogg Vorbis (which is a shame) or AIFF (yay! AIFF's gonna die!)

No matter what form you store the data in, if you want it readable in the far future, you've got to remember two things - there's no guarantee ANY specific technology will exist, and there's no guarantee ANY specific timeframe for the reading to take place.

What you want, then, is to do the reverse of the language decoding that has taken place over the years. Imagine yourself faced with a puzzle every bit as baffling as Egyptian Hyroglyphics, only stored at a vastly greater information density and probably in an electronic format. What would you want/need, to be able to recover the data?

Well, there would seem to be a few things that are essential. First, the explorer in the future will need to know the data is there and in what form. So, if you're using optical storage, make that clear (along with frequency). If you're using N-state logic, make it clear what N is. If there are M layers, tell them the value of M. You don't need to know all of the technical information, because all they need is where to start looking.

Secondly, the information needs to be correctly indexed. Languages are broken because types of information can be grouped and identified. The same will be true here. So, produce a contents list with corresponding data formats and/or MIME types, along with the offsets within the medium.

Thirdly, a key is a REALLY good idea - something analogous to the Rosetta Stone. Let's say you're using binary logic and a fairly rudimentary FS on the storage medium with text-based directories. The key would be a printout of the root directory in binary, again in ASCII and a third time as a set of records describing the logical layout. The printout would also need the offset of the directory. From this, it would be trivial for someone in the year 3000 to determine how offsets were calculated, how the data was laid on the disk and how the data is connected.

If physical storage is going to be used, ensure the various media used will last about the same length of time. So, if you're aiming for a hundred years, CDs may just about work. But you must NOT have the CD in contact with sulphides or anything else which will destroy the surface. The CD must be kept cold (but not so cold it is damaged) to slow decomposition. It should also be kept somewhere where accidental exposure to UV is impossible.

If you're keeping paper notes with the data, as I've suggested, the paper must be acid-free and the inks must be long-lasting. Most modern paper is of very low grade, as are most modern inks.

If you're looking more at a time capsule that is for the FAR future (we're talking something that happens AFTER Star Trek), then you've got to be extra careful but it should still be possible. I see no reason why you couldn't have physical storage under ideal conditions which could be retrievable after a thousand years or so. You just have to be very careful on what you choose to use. Same with paper. If you're looking to produce the next Beowulf (no, not the clustering technology), then you're probably going to want to look at vellum or some other extremely high-quality medium. I'd also look up early inks on the Internet and modify a recipe that could be used as a refill for a printer ink cartridge. Many early inks are highly stable (iron oxide is one example) and fade more by damage to the medium than decay of the ink.

Re:this should be soluble.

[Coryoth]

Thirdly, a key is a REALLY good idea - something analogous to the Rosetta Stone.

Not exactly replying to your post as simply having my memory spurred with regard to something relevant: if you're really interested in storing information for future generations then The Rosetta Project is an interesting on. They seek to have as many distinct languages as possible printed on a small disk, beginning in large print but decreasing in size as it spirals inwards to the point where it is micro-etched. It's easy enough to figure out how to read it, and as long as you cna build tools to magnify it you can read everything on it.

Jedidiah.

Re:this should be soluble.

[Pharmboy]

I have heard the same for photographs. Today's photographic paper isn't the same as older stuff, with less silver, and it tends to fade quicker. While we can rely on 100 year old photographs, our decendents may not. Most paper nowdays is relatively acidic as well, so it breaks down faster with any exposure. This would mean books as well. While there is good paper that is better than the old stuff, most is made to be cheap, not high quality.

Re:this should be soluble.

[noyren]

heh, if they look in my attic they will find a bunch of cd's, but they'll also find a bunch of computers that can read them ;)

Re:this should be soluble.

[ZorinLynx]

I dunno, I doubt it'll be impossible to read a CD even in 90 years.

Even today, you can find places to convert old 8mm home movies into a more modern format.

CD-ROM drives are resilient devices; I'm sure millions of them will survive in working condition for many decades. Some will eventually be owned by data conversion services that will do this for you.

You can still readily find equipment to play 78RPM records, reel-to-reel tape, 9track computer tapes, TK50, and other dead formats. It may be difficult, but not impossible to recover your old data in 50-100 years.

Whether the CD media will hold up that long is another story. Polycarbonate is pretty stable, but we're talking a long time during which an accident could happen. An accident that merely tears or crumples paper can destroy a CD!

-Z

Re:this should be soluble.

[Kesh]

Yep. I worked selling printers for Epson for a while. Even their good (consumer) archival paper is only rated 80-100 years, when using their good archival level printer (about $700 USD). Supposedly their cheaper printers would get 50-80 years when using archival photo paper and the normal inks.

Of course, that's their own ratings, so I dunno how accurate it is.

Re:this should be soluble.

[kabz]

Yeah, I can just imagine ...

You find the CD buried in a box in the garden.
You see the Microsoft logo. An old, long-dead company.
You scrape some dust off the CD.
You read through the logos and fine print on the CD.
You see the logo 'PlaysForSure' (tm)
You groan and throw the CD in the trash.

Re:this should be soluble.

[Lumpy]

probably true? how about IS true.

I have a side business as a data mage. I have a nice collection of very VERY old dat astorage reading/writing devices that sit un-used for 99.997% of their life now. but every once in a while I get a call from a friend of a friand's colleague. and I make a very tidy sum reading the files off of that Bernulli disk or 9 track tape in ebcdic format and but it to a modern format like CD.

There are several of us that exist, and there always be some that will have the ability to read long forgotten data formats. Hell I bet I have a working paper tape reader here somewhere.

My ex wife called me a fool for buying and keeping all this "crap" as she calls it until the day that I dragged a 9 track tape reader into the living room, set up an old XT that was able to use the isa card that connected to the drive and spent the afternoon reading 5 tapes to the drive and then transferring via rs232 to my laptop to burn to CD.

I made $8000.00 in one afternoon because I was the only one able to help them.

She finally shut the hell up when I returned with that check and threw it on the table next to the bills.

There are businesses that make their good living off this exact need.

My biggest customers are small businesses wanting 5.25 disks read or the popular bernulli's or the odd TRS-80 business compter disk (Model III and Model VI types typically) that needs to be read. I do turn away business, some balk right away at the price I charge, but I knw they will not find someone else locally and certianly not near my low price nationally. This help seperate the curious from the real customers.

Re:this should be soluble.

[russellh]

My advice is pick the really important things to preserve and save them well. eg- print it out. Everything else: just do the minimum, and throw away the crap. Most of it will probably be fine anyway, for our lifetime, but the important stuff you've already taken care of.

(btw, the specific problem with burnt CDs is the decay of the organic dye, iirc. the blue ones last the longest.)

Re:this should be soluble.

[NanoGator]

"Not to be rude, but for the love of God, don't you have something to do besides write 1-bit bitmaps by hand?"... said the guy being needlessly critical on Slashdot.

Re:this should be soluble.

[Kadin2048]

I agree. But there should be something available to store stuff that's not quite important enough by itself to warrant printing out, but might be worth storing anyway.

Also, there are those things which gain importance by being a complete record. For example (and this is a weak example, I admit) take all my email. It's far too much to print out, and it wouldn't be worth the paper anyway. However, that's not to say it's unimportant: if I could keep a complete record of every email I ever wrote, for my entire life, that repository in toto could be important or worthwhile. It's important though only because it's a complete record, and useful really only if it's searchable and indexed (or sorted). Very few messages are by themselves important, but it would be neat to have the ability 30 years from now to go through and see what was on my mind, at any given point in time.

For things like this, you really need a storage medium that's both random access and supports a high data density. To go with that, it's probably acceptable for it to have a shorter lifespan than the "time capsule" applications.

Re:this should be soluble.

[Green Salad]

...they will find a bunch of cd's, but they'll also find a bunch of computers that can read them...

I'm pretty sure the computer's broke or something. I tried plugging the computer into a regular hydrogen outlet and the hydrogen receptacle claimed in a very insistent voice that device didn't meet code for an approved hydrogen appliance. I figured I was Foobar.

Then, I remembered a story about how energy used to be distributed. After extensive research, I sucessfully built a 120 volt sine wave generator, plugged the computer in, and spoke into the mouse. Guess what? The computer didn't say a thing back.

Still more research made it clear that sound would come from the silver platter things. I held one tight against my ear when I spoke into the mouse...still nothing.

After still more research, I connected the big vacuum tube thingy (which also runs on 120 volt sine waves) to the computer. After a few microblicks, I saw some strange white symbols glowing on the surface. It that looked like this "C: Drive Error: No Bootable Device."
BR> Now where getting somewhere. I'm sure the pattern of glowing dots is tightly compressed data being shown on the screen. I've got a pattern recognition expert looking at the pattern of glowing dots on the screen. It's probably some sort of ancient code.

Yesterday, someone said that feelings of surprise, anxiety and disbelief was represented by a glowing pattern of dots that looked like this =:O

Re:this should be soluble.

[thejynxed]

I develop photographs for a living. I can tell you right now that matte finish paper is much better than this glossy crap that everyone seems to like nowadays. Glossy paper is not only more acidic, but it can "smudge" if handled too soon out of the developmental process. There is also the fact that glossy paper retains skin oils more than even regular writing paper... In essence you get tons of fingerprints all over your pictures, and the oils from those prints eat into the paper over a short period of time. This is not a problem with matte paper, as it contains a higher silver content, and has a more textured surface that doesn't retain skin oils to the degree of glossy paper. Matte paper photos also don't fade as fast as glossies when exposed to direct sunlight. Matte photographs also produce better results when scanned, and require alot less color, hue, brightness and contrast manipulation in Photoshop when creating copies from a print. I've noticed that with glossies, you get colors that are overly bright, and the reflective shiny surface of the paper (the same shiny surface that makes the photos hard to see at certain angles in bright light) can throw off digital scanners a bit and make the colors off-balanced. For instance: If you scan a black & white glossy and a black and white matte photo of the same subject matter, the results from the glossy make the black appear as a greenish color while the results from the matte photo are identical to the original. The supply cost for the each paper is the same (At least up until you get into the 11x14 - 12x18 range, then matte cost slightly more since it isn't as common as glossy), so I encourage people to use matte. BTW - Almost all of the professional photographers in my area (Jamestown/Erie/Buffalo/Niagra) choose matte photographs for their personal collections. They get glossy prints only for customers who want them.

The equipment?

[Dogers]

Nevermind the equipment needed to read it, what about the rights they'll need to read it?

And even that's ignoring the fact the CD will long since have self destructed, decaying away..

(From TFA: "Dark age ... the Powerhouse Museum's Matthew Connell with an ancient clay tablet that will probably outlive the 1980s tape in his right hand.".. Probably? Definitely more like!)

ebay

[truckaxle]

Maybe they can buy all the necessary components on ebay!

Seriously archeologist have decoded all sorts of dead languages, decoding digital (assuming you can still pick out the bits) would be easier.

...and

[Stanistani]

In the second box is a player, if the fellow had any real fortune.

Besides, I'd have drawn the map on parchment, and tied it up with a string.

Arrr! Ye Mateys...

Re:...and

[$RANDOMLUSER]

OK - let's examine this concept - I have a family fortune - and the only person I'm going to tell the location to is a lawyer???????

Can't I just blow it on hookers and cocaine before I die?

dark age

[foxhound01]

They'll take it to that crazy old guy in the corner house with uncut grass in his lawn, for he was once a great programming guru and has a ton of still functioning archaic equipment that requires insanely large amounts of power.

Re:dark age

[Frogbert]

I think its safe to say, with the number of people who showed up on Qlink using real Commodore 64s reciently, that there will be no shortage of "Crazy old guys" in the near future.

Re:dark age

[shokk]

Sounds like a mission from some new MMORPG. Where can I download a demo?

easy

[DrSkwid]

perhaps the same way I would read a wax cylinder today

visit a specialist

a good place to start would be here :

http://www.bl.uk/collections/sound-archive/wtmcyli nder.html

Re:Easy

[michael_cain]

Seriously, if you want to think in terms of 100-150 years, this is a solved problem, and without the need for stone tablets. Pigment-based inks on acid-free paper. Silver-based black and white photo chemistry on acid-free paper. Stitched bindings, not glue. Store in a trunk where there's neglible light. Put the trunk in the attic of a house where it's reasonably safe from large amounts of water (rain or flood). Civil War documents using these techniques have survived nicely to the present day. The Bell Labs archives have Alexander Graham Bell's original laboratory notebooks, still easily legible. To date, there are no reliable archival media for this length of time for audio or moving pictures. Write it down. Sketch it (as silver-based photographic materials are getting harder and harder to find). And you can be the source material for the historians of 2155 :^)

Yeah right

[Anonymous Coward]

Whatever is worthwhile to keep will be migrated to new media. Even if 90% of it is lost odds are 10 times more information will be preserved from this decade than the last. Digital media is cheaper to own and operate.

Easy

[joe_bruin]

How can they read my obsolete digital document?

The same way we do it today: emulators. Of course, your cdrom is not going to survive that long, so there's no need to worry about that. Have you considered leaving your legacy carved into stone tablets?

The times they are a changing

[Orionetheus]

Hopefully someone isn't stupid enough to store their will on a CD rom...would you?

Re:The times they are a changing

[dfjunior]

Hell no!
Zip discs are the *only* reliable way to archive digital data indefinitely

Re:The times they are a changing

[the real darkskye]

Thats why I store all my important stuff on LS-120 discs.
That way even if someone does steal them they'd be hard pressed to find out what it is and finding a drive for it!

This is a touchy subject.

[empvirus]

Really, who knows what the future holds? And who says we won't be able to trace history back to these days and even further? And just because we don't use a media anymore means it is forgotten and no one will ever be able to read the media again. I mean, if one did some digging, I bet he/she would find information to be able to read punch-cards even. Just my 2 cents.

Huh?

[tktk]

...they come across a CD-ROM and a letter...

\/\/H47'$ 4 L3773r?

a lesson on impermanence

[puzzled]

  Each moment arises out of the moment before - call it 'dependent arising'. No object exists in perpetuity - even black holes evaporate over long time spans.

  This being said, our digital storage systems, in a collective sense, are becoming more like a brain and less like an archive. 'Memories' of some importance are in multiple locations and accessible via different search methods. They're also being changed, just as memories of our pasts acquire a patina as we age. Someone took something I wrote in the early 90s on Usenet and added it to their humor site. My flickr content is spreading if the hits are any indication, as are my contributions to YouTube.

  Public records are an important thing, but understand the other, positive things that are happening in the background as the the internet acts less like a database and more like a neural net with each passing day.

Doesn't take that long ...

[angst_ridden_hipster]

I have a bunch of old DSDD 40-track hard-sector TRS-80 5.25" floppy disks (NEWDOS/80v2 format) that I'd love be able to read.

Unless I want to build custom hardware, I don't believe it can be done...

And those are only ... uh ... well, OK, twenty to twenty-five years old.

Re:Doesn't take that long ...

[Murphy Murph]

Are you suggesting that your TRS-80 had 1% of 1% of 1% the market penetration of CDs?

Apples to oranges my friend.

Besides, what is stopping you from reading that data on an ebayed machine, printing it out and OCRing it?

Re:Doesn't take that long ...

[jcr]

Unless I want to build custom hardware, I don't believe it can be done...

There are service bureaus that will read those disks and stick the data on a secured server to download. Hell, you can even get paper tape and hollerith cards read for you.

-jcr

SESSION #18 - SPEAK LIKE A CHILD

[infonography]

Subject of a Cowboy Bebop episode. This is why I watch anime. They actually take some time to examine an idea like where to find a Betamax player 150 years from now. http://rfblues.aaanime.net/Sessions/session18.htm

Give it to me

[fumanchu32]

Just give me the document. I'll print off a hard copy today, that new fangled paper technology looks promising (Assume acid free paper, no sunlight, etc, for you picky individuals). Just leave them a cd with my contact info. I will give them the directions to the family fortune, I promise. You can trust me, I'm a [insert political party of choice here].

The format is probably not relevant

[hungrygrue]

as the CD probably couldn't be read regardless. CDs do not last forever. http://www.warehousephoto.com/How_Permanent_is_you r_CD-R.htm In fact many will be unreadable in as little as 2 years. If you want to archive, print it with good ink on acid free archival paper.

Interesting - historians' concerns

[BitterAndDrunk]

I read an article about 10 months ago about the "death of history" due to the electronic age.
In a nutshell, as we've moved to more digital forms of communication (phone and email), one of the primary methods historians use to piece together older eras is going extinct - the written correspondence from one person to the next.
It was an excellent article; my google-fu sucks apparently because I can't find hide nor hair of it. Curses. No +5 Informative for me.

I think that..

[slapout]

..a more likey outcome is that patents and DRM will lead to a digital dark age.

Not really a problem

[geekoid]

people migrate their data to new technologies all the time.

Think about it. A person gets a new computer with the latest technology, then they transfer their data to the new machine.A contant upgrade cyscly.

Same with lerge businesses, they may be using a tape library, but they upgrade there tapes regularly. And if some came out with a 1000 terabytes in a cubic inch of crystal storage device, they would also ahve a way to migrate there clients data. If they didn't they would have a hard time selling any.

CD Rot

[Malicious]

Don't forget about CD Rot. While you'd like to believe that if you put that Treasure Map on a CD so you can find the treasure years from now, chances are... your map will have disspeared on you.

This is why I still get my digital photos developed. Last thing I want is all my treasured memories to become suddenly un-readable someday.

Let Google worry about it

[rarewire]

Make your problem Google's problem: Mail yourself all your archive files to your Gmail account

An interesting drawback to digitalization

[kerohazel]

Reminds me of a discussion I once got into about analog vs. digital storage. Some of the people on the analog side argued that the myth of digital media being everlasting is false -- which it is. Digital media, on their own, should be seen as temporary storage. The true virtue of digital media isn't even the media itself -- it's the content. Content is what can be copied over and over again with no degradation.

Like oral traditions, the chain of copying needs to remain unbroken for any information to truly last forever, outliving "mere mortal" media. As long as P2P networks continue to exist, I can die happily knowing that the sum of mankind's knowledge will be floating around there somewhere... even if it is buried under millions of terabytes worth of lesbian porn. ;P

Who cares?

[Chysn]

So we're going to lose our information. Who cares? Proton decay will eventually destroy all of it. Sure, that's a long time in the future. You know how things go: it's 10^1032 years away today, but before you know it the kids have moved out and the end of the universe is right around the corner.

Just try and keep those bits in line.

Re:The tools are not the problem.

[limabone]

Why do people keep saying CD's die in 2 year/5 years/x years? Has anyone actually had a CD die on them? I have CD's in front of me at this very moment that are over 10 years old and still work great (yes I did in fact test them). Is there some conspiracy by the blank CD manufacturers to make you think all your CD's are going to die so you need to keep transferring the contents from one disk to another forever?

Similar issues with old movies

[Alien54]

All too often these are literally rotting away in storage, because the originals are decaying, and the movie companies are unwilling to invest money to rescue them, even though they would sue you for millions if you published these on your own.

Re:Similar issues with old movies

[Cylix]

That's why we have companies that will track down a reel that is in public domain and recover it.

Sure technology that is even 10 years old gets lost.

It's the nature of the beast.

There are ways to store data so that it lasts. It's just a little expensive.

Someone should burn a cd, lock it away and come back and tell us how it works in 5 years. Do it again in 10. I bet you can get 5 or so mod points out of it.

Re:Similar issues with old movies

[Steve Franklin]

Interestingly enough, despite the commonly accepted wisdom that the loss of the material stored at the Library of Alexandria was the result of successive burnings, an analysis by Luciano Canfora ("The Vanished Library") shows that it simply crumbled to dust because the ongpoing process by which it was continuously copied and recopied was interrupted. If you really want data to survive, you need to put it on something that will physically last, like the clay tablets from Mesopotamia. The notion that civilization somehow began at Sumer is a direct result of the fact that their documents survived burial in the ground for thousands of years. This is the standard to which you need to aspire.

Media Evolution and Digital Photography

[TFGeditor]

Media evolution and subsequent obsolescence is what keeps may photographers from adopting digital cameras. Slide film images, though not "forever," are certainly more enduring and readily adaptable via scanning to whatever digital storage medium is the current state of the art.

there is going to be no digital darkage...

[3seas]

... except for stuff that has copy protection on it...

why? because anything anybody wants to preserve they will either copy it over to newer larger space media or the archiologist will build the device to read the old media.

if there is any concern its with teh ability of the media to hold data... but we were all told how much better cds are to tape and floppy at holding information....

so its on the media industry to be sued when the truth is exposed....????

cd's are to last at least 100 year???

of course there is always writing it out and storing it in some cave at the dead sea site...

Re:Paper

[antifoidulus]

Heh, thats something I didn't understand in the scenario mentioned in the summary, why would someone create a paper explaining a document on a cd, but then not bother to print out the document itself? Seems a bit weird to be combining "formats" like that if you will. More than likely what would happen is that the grandchildren would find a spindle of cds that may contain old family photographs and throw them out not knowing what they contained(priceless family memories or they could just be leisure suit larry games)

In 2045 grandad knew about this stuff called. . .

[kfg]

paper. Use it. Maybe give a copy to the family lawyer who handles grandad's will and shit, also on paper. The stuff isn't going away any time soon. Like, ever.

In 2045 there will still be functional CD drives around, because they are a mass market item. There are still Edison wax cyclinder "record" players around too. Data loss due to format changes are the biggest problem where rare or custom machinery is used, such as at NASA.

There is a huge difference between losing some data and losing all, or nearly all, of it, as in a Dark Age. Yeah, losing the key to the family fortune is a bit tragic for the family, but it already happens all the time without any reference to digital storage, and nobody declares a "Dark Age" over it.

In any case, dark ages aren't even defined by the loss of data, per se. They are the defined by the loss of data because reading and writting itself is lost and/or denigrated. It was not so much the burning of the library at Alexandria that created a dark age, it was the lack of social importance placed on recovering and preserving what had been lost.

People ceased to backup.

At the time backing up was labor intensive and expensive. Now it is quick, easy and cheap. Even, comparitively, for obsolescent/obsolete data storage devices. If the family fortune is really anything substantial it will be recovered because the knowledge of how to recover is maintained and the CD itself still exists.

Nothing has actually been lost, it's just a cost/labor issue to recover.

KFG

Yeah, but so what?

[ottffssent]

The example's contrived. I don't like contrived examples unless they illustrate an important principle, which this one doesn't really do. Such data loss has already started happening even in my own life, but I don't think that's a bad thing. The fairly minimal effort required to keep data up-to-date is a natural impediment to a policy of keeping everything. Data which isn't worth a new hard drive and an rsync dies. Data which isn't worth the effort of importing and re-saving in a newer format dies. This isn't bad. It's not new either.

Data goes the way of the dodo not because of technological obstacles, but because of a decision made or not made to preserve it. We don't know how the great pyramids were built, the obelisks shaped and erected, etc. not because there was no way to preserve that information, but because it wasn't important enough to justify the effort. The same is true of 10-yr-old WP documents I made to bill people when I mowed lawns for spending money, or a million other things that get saved or trashed every day.

If you're serious about the problem, then it's not a technical hurdle. Data storage is cheap. Emulators are good. Batch document conversion is possible. The problem, if you're willing to call it that is that the benefit has to outweigh the cost. Lowering the cost of data preservation only increases the cost of data searching and real information retrieval. And very quickly it becomes a philosophical argument about the value of preserving irrelevant knowledge in a world that has moved on. Yet the argument is couched in terms of data storage and manipulation which is really the tiniest corner of the issue.

Old problem

[phocuz]

It's not something new. There are lots of projects around the world trying to solve it. As with all issues of preserving knowledge, you'll find that even the simplest things can turn to great pains. For instance - you have your great hole in the mountain for storage of nuclear waste. Now, you don't want people in the future walking down there. Assuming they dont speak your language, what do you do? Paint something, like pic 1: human outside, pic 2: human going in, pic 3: human dead. Sounds good eh? Try reading it backwards, as some cultures do. Human dead, goes inside, comes out alive? Not really the same thing, but it's an interesting problem. What will we leave behind? Not many things are written in rock any more.

All this makes me think of Stanislav Lem's "Memoires found in a bathtub", which is a grat book for those of you who haven't read it.

Old news with Analog

[Ironsides]

This is already happening with analog recordings. The Piano Paper that you put into a piano and the piano plays the music. The old drums that were originally used to record sound. Records (45s, 33s, 78s), 8 Track Tape, Reel-to-Reel, dictaphone, Cassettes (becoming this way).

Want picture/video? My father has some negatives that are 3 inches by 5 inches. Back before the days of 35mm film. Then there are those old home movies that predate VHS.

The only difference between that and digital is that digital is newer.

if you expect to have to reverse engineer it

[toby]

bmp would be preferable to jpg

Only if you expect to be in the situation of having no software to read JPG, and no specification. That's a slightly extreme scenario? Since your data has been, obviously, carried forward. You could always carry forward source code or specifications too, along with your JPG corpus. Or am I missing something?

Re:if you expect to have to reverse engineer it

[Kadin2048]

This is a pretty good point.

If you're thinking that your data will be carried forward electronically, then there's no reason why a set of specifications or source code in a commonly-understood language (I can't imagine that any reasonable programmer of the future wouldn't be able to at least puzzle out some well-commented Pascal or C) showing how to decode your data. However, you'd have to hope that whoever is 'carrying forward' your data isn't lazy or cheap, because this would be the kind of thing that would get cut off or removed for being redundant if you started running short on storage space. It would be unfortunate, but not hard to believe, if some space-consious bureaucrat somewhere were to keep only the data and not any of the sample decoders, especially if it was in a repository with a lot of other people's data, and each one had its own set of example decoders. I guess the decision of whether to trust the archivists in question would vary with each situation.

In the case of physical media, I think it's important to bundle each set of data-containing artifacts with either an actual reader device which produces some kind of easily understood output, or schematics for same. For instance if you were going to bury a vault of CD-Rom type discs, it would make sense to put at least a CD-Rom drive in (actually in a vault, you might as well put a whole computer in too). In something smaller, at least include a decoder schematic, or at least some kind of minimal diagram. I'm thinking the bare minimum would be something like what NASA put on the top side of the Voyager probes' "Golden Records."

Actually if you want a good example of a data-storage 'time capsule,' I think the Golden Records pretty much are the 'best practice.' Engraved into metal, built to last practically forever, each one comes with a cartridge and some minimal schematic instructions. If I'm not mistaken too, the very beginning of the recording is a triangle-wave test tone, which is represented on the front, so you know you've gotten it right. I've also read that they included analog slow-scan TV images on there too, although how they expected people to puzzle those out I don't know.

Re:if you expect to have to reverse engineer it

[drownie]

And don't even think about it if this is in a post-technological or post-apocalyptic scenario. That's when you want hardcopy! Old-fashioned printouts and photographs... with all their attendant preservation headaches. That should be in the bunker too. The basic data that defines our civilization along with a lot of technical data is in fact stored on microfilm, in metal containers in a salt mine. The german Bundesarchiv does this, the Swiss archive does it and I think there are some other countries with these kind of national archives.I think the library of congress has some of these kind of storage capacities. It's absolutely safe, there has never before been a safer storage for information. The microfilms are produced to last some thousand years. A microfilm reader is essentially some kind oflense with a light and the salt mine was here for the last 50.000.000 years. It will survive a nuclear war, an asteroid impact, it will probably survive human civilisation. I wouldn't be too worried.

Answer - document custodian daemons.

[jlseagull]

Idea #1
What about a semi-intelligent expert system daemon that, given two document formats, could figure out how to convert one to the other?

Consider this: I would like to archive a set of CAD documents, but they're in archaic format X. Modern CAD formats are A, B, and C. CAD programs typically have ancestors that can convert from past versions for migration purposes.

So consider an interlinked set of CAD converters:
#1 can convert formats F, G, H to formats D and E.
#2 can convert formats W, Y, X, and Z to formats I, J, K, L, and F.
#3 can convert formats D and E to formats A, B, and C.

Consider then a daemon that continuously monitors a filesystem looking for documents that aren't in a current format. It then fires up the converters and performs the conversion while archiving all past versions.

So in the example, the daemon fires up converters 2, then 1, and finally 3.

It could also cryptographically sign the files to provide a chain-of-custody.

It also maintains a set of applications and an emulator for different operating systems. When one needs to open an archaic dataset, one can either look at the converted files or call the daemon directly to seamlessly pass an emulated application session to the user if you want to look at it in the original form.

Idea #2
Documents could contain their own viewers. Yes, I know that's a bad idea making document objects executables, but hear me out. The document custodian daemon could also maintain a sandbox for document viewers to run in - it could even be a standardized virtual machine written in something like Java. This is getting a little out of my area of expertise, but I'll ask my girlfriend about it. It would get interesting after several levels of emulated virtual machines.

This year, hard drives became cheaper than tape for the first time in terms of $/GB. RAID with NFS should be way better than tape backup in terms of retention and nearline access, but I'm not really an IT guy.

I'm sure there's a business model in there somewhere.

No problem

[HungSoLow]

The way I see it is, in 2045, they'll have computers powerful enough to look at any binary data and accurately assess what the data represents, and how to extract any useful information from it.

Easier

[Sycraft-fu]

Copy it to a new format. That is the real beauty of digital. Since it can be perfectly duplicated easily and quickly it's no problem to move it to a newer format. I have data on my drives now that was orignally on 5.25" floppy. It has just been recopied many times. Some of it has been converted to new formats, some of it is unmodified. Either way, it's still here despite being decades old.

I don't know where this silly idea comes from that somehow digital is really fragile and we'll just lose all of it later. Sure, we lose tons of it all the time, but it's worthless, by and large. The by product of the information age is that we produce so much of it, it is not only impossible to archive all of it, it's undesirable. To have more information than you could ever sift through would be almost as bad as having none at all.

Also what's the this stupid notion that we'll forget how to read things? That's like saying that we'll forget how to build sailing ships, now that we have motors. Of course that's not the case, the knowledge is preserved, in the case of sail boats, they are still made.

This is even more clear for computers since emulation is a major protect for many people. We have emulators for all kinds of old systems. Means if you find data for one of them, you just load up said emulator and it'll get at it.

Digital actually seems to be the ultimate prevention against a dark age. The ease of copying information and archiving it in multiple spots means that it's difficult for a single catastrophe to wipe out large amounts of data forever. There was a lot of work in teh past, for example the Mayan Codexes, that was destroyed and is totally unrecoverable. It was fragile precisely because it was hard to copy and thus there wasn't much of it around. Now, of the orignal hundreds of thousadns of Codexes, we have but 3.

I think it's just a bunch of alarmism.

Re:I totally agree

[Kadin2048]

I was thinking of how you could store data that would really stand up to the test of time. History provides us some examples: things cut into stone seem to do pretty well. Paper isn't bad, providing you store it well. Animal skins, not so good. Celluloid isn't either (evidenced by the old movies and cartoons that are degrading).

However glass is really good, and while it might not have the proven track record that stone tablets to, it can also support a much higher data density. For example, Ansel Adams original glass plate negatives are in some cases just as sharp as the day they were shot, and they should stay like that for the foreseeable future providing they're well taken care of. But even they are dependent on the chemicals used in processing -- whether the silver sticks to the glass over time, etc.

So here's what I was thinking: what if you used some sort of photographic process to physically etch a pattern of bits into glass: use a fairly strong acid and get the etching pretty deep, or maybe etch the bits at the bottom of phonograph-like grooves so that light surface touching wouldn't destroy them. If you could make something like this that could be read with a regular CD Rom, that would be even better.

I think some sort of process like this is used on metal (or is it actually glass?) to make the dies for stamping CDs. Basically I'm suggesting just make and retain the masters, but don't degrade them by stamping anything.

Until the universal eternal format is developed ..

[RustyPelican]

Periodically rolling over data from its existing format/media to the latest new fromat/media before the media or format life expires (whichever comes first) is the only way to perpetually save data. The problem is not in having a reliable method to preserve data. It is an unwillingness on the part of the guardian of the data to spend the resources (operational overhead) necessary to do it. The perceived value of historical data is often viewed by how often it is accessed as opposed to what future insights might be derived from it or what commercial value it might have in markets that do not yet exist. In other words we lose data because operational management doesn't see its contribution to the bottom line on the profit and loss statement. The reality is that these judgements evintually do have to be made because we just don't have enouth bandwidth to store all of the data that ever existed even if we wanted to. Until everyone recognizes that maintaining your data is no different than maintaining any other valuable asset such as highways and city buildings the situation wont change.

So, if you want to have the map to the family fortune readable in 50 to 100 years, store it in the highest linear (uncompressed) resolution available to you and every time there is a paradigm shift in digital media or formats transfer the data again to the new format linearly at the same resolution is the original. If you compress the data and then transcode to a new compression format some data will be lost in each iteration and eventually the data will be unusable.

Check this link regarding the Electronic Records Archive http://www.archives.gov/era/index.html .......Just my 2 cents worth

hardware is much, ah, *harder* than software

[toby]

I think it's important to bundle each set of data-containing artifacts with either an actual reader device which produces some kind of easily understood output, or schematics for same. For instance if you were going to bury a vault of CD-Rom type discs, it would make sense to put at least a CD-Rom drive in

Yep. That's my worry. It's going to be much tougher to actually find the data and read it than interpret the data. Imagine trying to read a CD-ROM, or hard drive, or NVRAM, anything! in a world where the complete integrated systems aren't available. Even if you had specs to say, and IDE interface, you'd have to do man-millennia of engineering to get at the data. We already know this from the impossibility of recovering data stored during the last 40 years (NASA's Viking probe data is the famous example but there are thousands of other cases). The hardware gets decommissioned and scrapped; and even if the media survives, it has a limited shelf life.

And don't even think about it if this is in a post-technological or post-apocalyptic scenario. That's when you want hardcopy! Old-fashioned printouts and photographs... with all their attendant preservation headaches. That should be in the bunker too.

Writing software or even reverse engineering formats looks much easier by comparison.

DRM & Out of Control Intellectual Property Law

[salesgeek]

While file formats and media have presented a problem, there is plenty of evidence to suggest that digital information has an extended lifetime, and the most valuable information will be converted into newer formats as well as more simple and fundamental formats. Simple formats like ASCII text have handled the test of time. I'm more concerned by the potential lockdown of information through overzealous use of DRM technology backed by overbroad intellectual property laws. Just like the last dark age, the next one will be the result of people trying to control other people.

For optical media, it's very easy...

[tlambert]

For optical media, it's very easy... assuming the media actually survives, it's the same way this guy plays vynil LP's using a flatbed scanner:

http://wired-vig.wired.com/news/digiwood/0,1412,57 769,00.html

Obviously, in the future, ultra-high resolution optical input will put the current scanning/video technology to shame; they will just need to scan the thing in and run a program against the data to get the contents of the media back.

It won't be that dark

[kilodelta]

I work with a bunch of library science and archvist types who worry about this all the time.

It's such a pain taking care of books that are a few hundred years old. But they miss the point when it comes to digital.

For example, data I had on 5.25" floppies was moved to 3.5" floppies, then to a 20MB hd, then to a CD-ROM, then onto my current system.

If it's that important you transition it to new media.

WordProcessor Recovery Possible

[Kadin2048]

I'm not sure about Wordstar, but I'm fairly certain with MS Works that if you open up that file in your favorite text editor that your data will be in there, and you should be able to just copy it out. Naturally you'll lose all the formatting, and probably have a lot of crappy unreadable characters introduced, but I'll bet you'll recover the bulk of it.

I had a similar issue once with a very nice (but very dead) word processing program that I used to use called WriteNow -- where the developer has stopped work and sales on the product but refuses to release it even in binary form to the public -- I had a bunch of disks of documents, but none of the original program disks and no computer with it installed anymore. I was pretty stuck until I just opened up the files one day and searched through them until I found the plain text. A few minutes of cleanup in BBEdit later, and I was ready to go.

Needless to say, they're all stored as .txt files now. Anyway, I wanted to say that I can definitely feel your pain with a problem like that, but that there are solutions to at least one of the problems you described.

Hey, I've read this before

[aiken_d]

Dateline: February, 1890

New "photographs" may be useless for archival purposes

Scientiests say that the dyes used in so-called "cameras" may not provide the kind of lasting record that traditional stone tablets have provided. In fact, left in bright sunlight for 50 years or more, photographs may be completely unreadable by even the latest 1890 technology.

This will surely mean the demise of modern civilization, since future generations are very unlikely to care enough about the past to devote any energy at all into preservation and reclamation of older information. Anything that can't be read by 1900 is likely to be lost forever.

It's yet another sad commentary on the state of modern civilization, and one more reason why manufacturers of stone tablets and chisels shouldn't throw into the towel too soon.

Cheers
-b

Dupe from Scientific American 1995

[paulatz]

I'm sureI have read the same article several years ago,I cannot remember were, maybe on Scientific American or such. After a search on sciam.com I have found this dated January 1995, more than ten years ago. Are we reading the older news ever posted on slashdot?

You time travel...

[Mendy]

...back and find a computer to read it on. I mean that's what they'll do when they need an IBM 5100 isn't it? :)

What about all the horse manure in 2000?

[Fredge]

This question is akin to somebody in 1900 asking what the world would be like in 2000 when the population kept growing and everybody had horses on the street - "think of all that manure accumulation - how will we walk without stepping in crap?"

The point is - the question is irrelevant. In 100 years, assuming the continued growth of storage mediums, the average personal user will have access to terrabytes, if not more, for personal use. I imagine that the most basic of ISPs (if such an entity continues to exist separately from other existing utilities) will provide users with gigabytes of personal space online to keep store/back-up their data. The only reason to put things on physical mediums will be for short-term backups.

I think a more pressing question is "will we be able to find the needle in the haystack?" Sure - Google does a decent job of indexing the internet now but even they are not 100%. Also the fact that while they may not be 'EVIL' today, it only takes 1 CEO change for them to become what most other companies are and then it's up to the next do-gooder to start an index from scratch. Then, assuming you can find stuff, you'll have to break the 200Mb encryption key. Luckily, the local Kinkos will have a quantum computer that you can use for $7.50/hour.

keep it live

[petermgreen]

copy it from one media format to the next BEFORE the old one dissapears or keep your data on your hard drives and copy it to your new ones each time you upgrade.

Low capacity removable media like floppies and to some extent CDs is the enemy of data preservation because it makes the job of copying stuff to fresh media require far more human labour.