Bit Rot Stalks Your Digital Keepsakes

axlrosen writes "The NYTimes has an article about the problems of digital archiving. How many of your digital memories will still be around 50 years from now, considering lost disks, incompatible formats, hard drive crashes, fading CD-Rs, etc.? Unfortunately Peter Briggs' solution won't work for most of us. The only real way to make sure that your grandkids get to see your digital photos is to make real photographic prints from them. (When I bought my Mom a digital camera I installed Picasa for her, and made sure she knows to order real prints of all the pictures she wants to survive through the ages...)"

Comments

Every 2-3 years

[zeke-o]

move your stuff to the next "permanent" media

Re:Every 2-3 years

[Tet]

move your stuff to the next "permanent" media

Or rather, dispense with the concept of permanent media altogether. I realised a few years ago that the only sane way to protect my data was to have it all online all the time. I store my data on redundant arrays of disks in two geographical locations (my house and my parents' house, synced nightly via rsync). This is IMHO a far better solution than backing up to tape or CD/DVD. LVM makes the process of moving the data to bigger disks trivial. Where it falls down is for really large volumes of data. Places like CERN that generate terabytes of data per day are going to struggle in the not too distant future. Archived data will become a real problem (even more than it is now).

Re:Every 2-3 years

[sfjoe]

This doesn't protect your data from user error or malicious deletion or corruption of data ...

Yes, but photos aren't protected from fire, flood or other natural disasters. The point of the article was to make a system that was at least as good at long-term archiving as print photos.

Umm

[TheKidWho]

"REAL Photos" wear out too.

Re:Umm

[jellomizer]

While it is ture that they wear out too, But if you keep them in good contition they can easilly last for hundreds of years. The oldest Data I have seen is from the early 1980's and they are probably electronic data from a decade earlier too.

Re:Umm

[philbert26]

"REAL Photos" wear out too.

This is especially true if you print them out at home. Which makes me even madder that I fell for that "here's a cheap printer with a gazillion DPI" scam that Epson was running a few years back. Once I added the cost of photo paper and cartridges, it was more expensive than developing the pics.

Perpetual backups

[Gentoo Fan]

Short of having titanium punchcards with your data bits punched in (and even then...) you are simply going to have to keep backing up and backing up. I'd rather have my data on 2 new hard drives than a dozen decade-old ones.

Re:Perpetual backups

[elmegil]

Not only insightful, but also "duh" obvious. Guess what? All those photos your grandparents took? They're fading. They're not perpetual. You're going to have to have them digitized and reprinted if you want "prints" that last forever. Every medium degrades, some faster, some slower. Digital is not so much subject to decay as it is to obsolecense, but the same principles apply. Keep doing technology refreshes and you should be fine.

Re:Perpetual backups

[Jeremy Singer]

Heiroglyphics and cuniform is still readable after millenia, and requires only our eyes to decode. A hard drive without a compatible computer to run and decode its contents is useless.

Re:Perpetual backups

[Confused]

Oh what a wonderful idea! Get 5 cheap disks, make an raid array and store everything. Safe backup for eternity.

So what's going to happen a few years down the road:

Assuming you you kept your disks well, you'll end up with fine very fine disks of historical value which you won't be able to attach to any computer of the day. Can you read MFM and RLL disk today (assuming you still know what those were)? Do you still find a controller that fits into your computer? Is the operating system still able to read the file system (Anyone still remember Ontrac disk partition programs)? Do you still have programs able to open the files you saved?

If you decide to keep your data always online, sooner or later some accident will take care of them without you realising the fact.

If you trust storage companies, who do you trust to be still in business 30 or 40 years from now and still honoring your contract? If they go bankrupt you lost everything.

If you want to preserve some of your work to show your grand-children, you better take a backup medium that has been around for the last 30 years and still can be widely and read. This leaves us with ... with ... uh ... paper. Everything else can only be considered a fad.

This might sound a little pessimistic, but my cupboard full of old floppy disks (8 and 5 1/4 inch, hard and soft sectored, CPM and other formats), 9 track tapes, optical disks and other mementos from work done in the past don't leave much faith in long term storage for digital data.

Re:Perpetual backups

[dcsmith]

Heiroglyphics and cuniform is still readable after millenia, and requires only our eyes to decode. A hard drive without a compatible computer to run and decode its contents is useless.

The only problem with that is finding the hieroglyph and cuneiform drivers for my brain.

Re:Perpetual backups

[pthisis]

So what's going to happen a few years down the road:

Assuming you you kept your disks well, you'll end up with fine very fine disks of historical value which you won't be able to attach to any computer of the day

But I don't give a crap about the disks, I want to preserve the DOCUMENTS. Having lost a lot of photos in a house fire I can tell you paper isn't near indestructible. And...

It's not like my documents aren't the very FIRST thing I copy over to every new computer I get. I still have source code I wrote on the VIC 20 in 1983 and all my old school papers from junior high (when I started using a word processor) on; I have no way to read the tape drive it used, but I did copy it over to the C64, then an 8088 (which probably had MFM drives), then at least 5 more machines on to the current one. On top of which there are various copies sitting on my parent's machine, various CD-ROMs, and a hosting account.

There was some mild formatting degredation when I converted old Word documents to a more stable format, but I stopped using Word in 1991 and have been with a format (TeX) that places future ability to render it identically as a high-priority goal--even barring that, the content is generally easy to see in a text editor. But complex math documents I created over a decade ago render identically (plus I have the postscript output alongside the source)

Anyway, for much of my stuff the original medium has degraded beyond belief. So what? Digital is trivial to copy, and my documents travel with me. Pretty much the only time I lose something is if there's a power outage as I'm creating it, and even that tends to be pretty limited since my autosave interval isn't that big.

Re:A few things

[1000101]

"2) I don't have the time to sign up "

But you have time to read a story on /. and post a reply??

Formats

[gmuslera]

Digital content could be "refreshed", just copying it to newer, bigger, cheaper and with more far on time expiration date each time (i.e. when i bought my cd burner, made a backup of my old diskette-based info there).

But the main problem is not the "end of life" of media used for storage, is the format in which the information is. In 50 years, will be an application that opens/process that information? One of the advantage of having information in open formats is that in the worst case, you can have all the information to be able to process them. But if you stored your information using an applicaiton with its own propietary/closed format, and the company just decided to not support that format anymore, or just closed, you could have lost your information, even if the media where it is stored still retains it well.

But very differently..

[vhold]

The fact that digital data rarely goes from "Perfect" to "Ok" to "umm not so good" to "What is that?".. it tends to go from "Perfect" to "Gone/Maybe not gone but very expensive to retrieve," makes it's worth discussing the finer points of digital archival versus analog.

Re:WWLD?

[suso]

Maybe funny, but there is some truth to that. For instance, I run webhosting service that has been around since '97 and I have moved all the user data to new machines at least 4 times now as I upgrade the machine. Theoretically, someone could put their precious pictures on the server and have them "live forever".

Probably a good idea for a profitable service would be a gigantic digital safety deposit box.

Perserving Electonic Data is oposit of Paper Data

[jellomizer]

Phisical Data such as paper, stone, ... will demish the more that it is handled and there is often some loss when it is copied, but you can keep it in a safe box for hundreds if not thousands of years. Digital Data is the oposit, In order for electronic digital data to survive it needs to be moved around and each copy is the same as it was before. That is why the music indrustry hates MP3 way more then copying Tapes. With MP3 each copy is as good as the first. With tapes they can only be copy only a fiew times before the quality gets really bad. And there is only a limit on how many times the master tape can be played. But Data just as long as it is moving it is more protected.

gee, nice ads on that link to Peter Briggs ...

[contrapuntalmindset]

The link ( http://www.boingboing.net/2004/11/06/alien_v_preda tor_scr.html ) to the info on Peter Briggs has porn ads, for those to whom it matters. Couldn't you have warned us?

Re:gee, nice ads on that link to Peter Briggs ...

[dema]

In defense of boingboing (which is a great site for news), the only "porn ad" they have is for suicide girls, and while I visit boingboing daily, I have never seen any nudity in the ad. It is always a risque picture, but hardly qualifies as "pornography." The suicide girls site on the other hand... (:

Quit whining

[bigtangringo]

Posting links that require login isn't particularly new. Do you complain about them EVERY time they're posted?

Use http://www.bugmenot.com/

Thank me later.

Kodak FUD?|

[MDMurphy]

Sounds like FUD put out by Kodak, or maybe Epson, and not "news".

Photos, slides and negatives don't last forever, just one look at the slides my Dad had in his house in Hawaii will illustrate that. But moving them to a new form of media is a lot more cumbersome moving 5 CD-Rs to a single DVD.

"Printing" is a bad way to save a picture, inkjet printouts degrade faster than true photos. You'd need to output to a real photo to get the same lifespan as a photo. Oh, and if you do, keep the digital copy, it's going to be better than a scan of the photo that's been sitting on the mantel.

Are there many consumers out there with more than 120GB of family digital photos? A spare hard drive is cheap these days as an additional place to store a copy.

Want to have your photos at home as well as somewhere safe in case of fire? It would be pricy to made dupes of all your slides or photos, but a second set of CDs pretty cheap.

There might be people who saved digital photos on floppys ( like those who got the cheesy Sony floppy cam ), but that media is not opsolete yet and for $20 you can have a USB floppy drive to let you move them to a CD.

Old media meant that the cost of the dupe was pretty much the same cost of the original. This doesn't lend itself to redundant copies at multiple locations for most people. Digital lends itself to duplication, just ask any movie pirate.

There are films from the 20's that are lost forever. Thanks to DVD pirates, we have enough redundant copies of Star Wars that it will never be gone.

Re: Kodak FUD?|

[Anonymous Coward]

except it isn't FUD. It's very much a reality of government organisations, research insitutions et al that deal with information that has to be kept available for (in it's extreme) eternity for society at large as a requirement by law. Ask NARA for fucks sake!

Given, most 'information' is not important for society at large.. but that which is, how do you deal with that?

keep in mind that "oh, put it on the internet" doesnt work if the information has any kind of secrecy attached to it.. such as military documents, hospital journals, bank records, white house-to-guantanamo emails etc. Do you really want your wife to know who you're fucking or how fast you gamed away your house in that pokergame with the pals back in 88?

You're just accusing it of being FUD because you don't want to retract your head from the sand and deal with an issue.

Re: Kodak FUD?|

[Minwee]

This becomes a bigger problem when you have more than a handful of vacation photos.

NASA filled several volkswagens worth of magnetic tape with images and data from the 1976 Viking missions to Mars, but lost more than 20% of it due to tape decay. You can't just run out and buy a new hard drive to back that all up on, although they did try.

At one point during the effort to preserve old data, it was remarked that the tapes were degrading faster than they could be copied.

But what do you do with all that data once you have rescued it from disaster? You throw it on a shelf and wait until it is about to expire again while you move on to collect even more data. Eventually you will wind up as the modern day equivalent of medieval archivists who spent their entire lives trying make copies of books before they rotted away, only to start over again at the beginning as the first copies they made start to fade away.

What happens when you slip? What happens to a set of old CDs that get lost during a move, only to be found by your children when they are sorting through your attic after your death? Or what happens when the government decides that they can save some money by cutting back "wasteful" spending on data libraries?

It's all gone. Welcome to the Library of Alexandria. Please, no smoking in the stacks.

Watch out for mistakes

[bill_mcgonigle]

I store my data on redundant arrays of disks in two geographical locations (my house and my parents' house, synced nightly via rsync).

Do you run rsync with --delete? If not, how do you deal with moved files? If so, how do you deal with accidental deletion?

I grant that you've solved the decaying media problem, but I've lost more data to screwups than to bitrot.

I think it might be more accurate to say...

[Anonymous Coward]

...too mentally sane to attract a wife.

Re:Tell me about it

[geoffspear]

By that time your OS install will take about 2 TB on its own, and all of your image files will be 2 gigapixel images with 128-bit color, and your multi-terabyte drive will fill up just as fast as the small stack of floppy disks that used to hold all of the data you could imagine needing.

Re:let 3rd parties store your data

[DavidLeblond]

Until Shutterfly either goes belly-up or changes their user policy. Then you're screwed.

Who really gives a shit, anyway?

[csoto]

We take hundreds of digital pix of my son all the time. Most of them are throw-aways. Who really cares what happens to the rest? He's not going to care about the photos when he's middle-aged any more than I care about the photos sitting in his grandparent's attic. Truly important stuff will be taken care of by people who care about it. All the rest is just a challenge for future anthropologists.

This "all my shit is important" attitude is one of the most annoying aspects of the "blog phenomenon." Who really gives a shit about what crap oozes out of your head? Seriously.

Problem? What Problem?

[Titusdot Groan]

Every couple of years I buy a new computer (10MB -> 500Mb -> 10Gb -> 60Gb -> 120Gb) and copy everything over.

I make a backup of everything important once a year and take copy to my parents and the cottage. I take an incremental backup with me anytime I go visiting.

My kids will have bigger computers and any digital photos will just live on by being on their computers. And their grandkids computers and ...

Recently one of my aunts scanned in and touched all my grandmothers photo album. Now that album lives on CD and Hard-Drives of most of her 13 kids and 35 grandkids. Now nobody really cares who gets the original album.

Digital medium is SOLVING the problem of the loss of this type of heirloom data -- not introducing a problem.

Online Backups.

[Firethorn]

There already are.

If you figure that most people's data is under a gig. We're not talking about system images here, just the "my documents folder" and it's ilk. Less than a hundred megs/month with most people, including photographs, unless they really love their mpeg home movies.

With decent broadband and some system to do the backups during non-peak hours, you can easily do tens gigabytes a month. Will it cost? Yes, but it's like doing your own car repairs. Unless you have a garage equiped like the shop, they can do it quicker and easier than you. Is it worth paying $20 a month for not having to worry about backups for a couple hours a month, would you seriously consider it at $20/year?

2nd paid advert for "web backup" on google

Re:Tell me about it

[vasqzr]

If you drop and scratch a DVD, you could lose ten thousand photos.

If you drop a photo album, you'll scratch a picture or two.

For anything I want to keep, I'll stick to a 35mm camera. For ebay or computer stuff, I'll use a digital camera.

Re:Tell me about it

[Total_Wimp]

And when you replace those DVDs in 20 years with something even better, the photos will still be in 100% perfect condition. Try that with an actual print of the picture.

This guy's advice is not smart. Bascially he's saying "take your perfect copy that might die at some point and replace it with an imperfect copy that is guaranteed to deteriorate with age." Heck, I'll just laser print all my documents for backup as well. We all know there's no way they could possibly be lost then. We all know going analog is much safer than backing up and refreshing the data on new media periodically because all those prints of movies, music and documents from 75 years ago look and sound so damn good.

I'll take my chances with backing up and copying data periodically over my skills as a museum currator any day.

TW

Re:Domesday Project?

[Reziac]

Yes! That's exactly what I was trying to think of, but my brain is suffering from bit rot, and the data couldn't be recovered... good thing I had your brain to use as a backup. :)

Re:Tell me about it

[dorsey]

Depends on your priorities. For me, the risk that I might scratch the disc and lose those 10k photos is far outweighed by the hassle and expense of storing and/or transporting 10k pieces of paper.

Re:Tell me about it

[mrchaotica]

So burn multiple copies then! Or try putting the disc in a jewel case! Or maybe even do something as crazy as making multiple copies, putting them in jewel cases, and storing one in a safe deposit box!

Uh-oh, I think they heard me -- some men in white lab coats are knocking at my door. Gotta go!

Re:Tell me about it

[LEgregius]

Yeah, I have digital photos from 98 or so. I haven't lost any pictures, come to think of it. I just back them up again every so often and leave them on a hard drive at all times. That way if a drive fails, I have the backup that is not very old, if a cd fails, well, I have the hard drive still. Plus, I backup photos multiple times a year, so the last few backups would still be fine in case of losing the latest backup and a drive. Not to mention that all photos are also copied to my wife's computer.
I think I would be much more likely, given how well we have taken care of photo prints, to keep the digital ones.
Actually, I have still have descendents of shell scripts in my home directory on my mac that I wrote on my slackware 2.3 PC in 1995.

Re:Tell me about it

[Anonymous Coward]

A well perserved ARCHIVAL print,
meaning using archival inks, and acid free paper will
last +100 years. Some of the first photographic prints ever made are still in existence! Why, because they are well preserved, and archived.

Your ~100$ epson inkjet printer, or drug store photocenter do not create archival prints.

Check out light impressions.com for more info.

Re:You have got to be kidding me.

[gr0kCalvin]

Word. Saying an actual picture will be more permanent than a digital copy is ridiculous. Maybe more permanent than ONE digital copy, but that's the whole point of digital, being that you can copy it.

Re:Tell me about it

[Skuld-Chan]

In 20 years will you have a picture viewer that can look at that pristine digital picture?

Many cameras are taking pictures using camera raw - camera raw pictures can really only be read in a few programs right now - one of them is photoshop.

Take pictures made on computers 20 years ago - can you read those pictures easily right now? You'd probably have a hard time reading pictures made with graphics app that dec, quantel etc made back then.

Re:Tell me about it

[Max Threshold]

Better yet, don't put all your eggs in one basket!

You can restore (or pay somebody to restore) a badly deteriorated, hundred-year-old photographic print, and get remarkable results. Imagine trying that with any digital media.

And you say you don't want to be a museum curator, but you're choosing the option that will require exactly that. Digital image archival will require meticulously cataloging, inspecting, and duplicating your media library. If you make prints, you can stick them in a shoe box and forget about them for fifty years, until your grandkids find them in some dusty corner of your attic and marvel at them... not caring a bit if they're a little faded.

Re:Tell me about it

[feidaykin]

I tend to believe there is an upper limit to how much storage a human being can possibily use... I would imagine a few petabytes would be more or less un-fill-able. While this may sound like a "640k ought to be enough" style quote, I think that at some point home users will have enough storage capacity to store more than the entire sum of recorded human history (not sure if that would even go beyond a few petabytes) and at that point, I think we will basically be done increasing storage capacity for home users.

Re:There's still a single point of failure

[INetEngineer]

What I think a Backup should be...

First of all, the backup should be a "pull" rather than a "push". This would eliminate the problem of a hacker getting on to the primary machine and "discovering" how to hack into the backup server.

I think the "ultimate" solution would be an OS that handles auto-backup of the file system. But, since I don't know of one, I suppose a backup server with "pull" software would be 2nd in line. Then, client-installed, scheduled backup software would come 3rd.

Here's my basic requirements list for backup software in no particular order: (feel free to tear it apart, add to it, or modify it)

1. Incremental - only creating new files, updating updated files, and deleting deleted files.
2. Staged - multiple backups should be stored across time, for example once each day is 1 backup and once each week is another
3. Compressed - the technology should be configureable to settle the tradeoff between size reduction and speed
4. Archived - backups themselves should be archived for a specified amount of time
5. Across multiple machines
6. Across multiple geographic locations
7. Provide regular monitoring reports to the administrator (e-mail, mobile alerts, whatever...)
8. Validate the backup against originals
9. Provide alerts to the administrator when the machine to be backed up is not available (for backup software that "pulls")

10. *Personally, I wouldn't include having the backup program watch for corrupted file systems. This should be the job of the computer itself and any anti-virus/spyware/adware software and possibly a monitoring server.

    I wrote a console program for Windows that takes care of 1, 2, 3, 4, and 7 (scheduling is actually performed by Windows scheduler). It "pushes" the files for backup, rather than "pulling"... so, I went against my own rule there.

    I tried Microsoft Windows XP backup for a while, but had some problems with the format. I definately think file backups ought to be in STANDARD FORMATS!!!!!!

Re:Tell me about it

[Total_Wimp]

And you say you don't want to be a museum curator, but you're choosing the option that will require exactly that.

I'm not a photo geek. The pictures I take are snapshots, not art. When I print them out they end up in a frame from Target or a photo album from Wal-Mart or sometimes a shoe box on my shelf. These pictures will not last and I'm unwilling to go to the effort of to print them and store them in such a way that they do last.

I am a tech geek though. I have more than a dozen functioning and used computers in my house at them moment and uncounted, but huge amounts of hard, floppy and compact disks. I have an excellent backup scheme (including off-site) for things that are important to me.

I'm unwilling to be a museum currator, but I have no problem being a backup expert. That's who I am and I'm going with my strengths. I honestly feel my grandkids have a much better chance to see a great picture of their mom when she was growing up if I use my archival scheme. Though they're not going to be computer geeks like me, I believe the "copy to new media/format as it becomes available" method will give better results to Joe Average too, because they're buything the same Target frames and Wal-Mart albums that I am.

TW

Re:Tell me about it

[Mordaximus]

That would be the case if we'd have stopped at ASCII text files for documentation. But as storage and processing power increase, we step up to the plate to fill it. From simple text, to PDFs and images and embedded video. From simple hypertext pages, to Flash and Embedded MIDI and VRML.

Multimedia grows significantly : NTSC PVR requirements now are nothing compared to HDTV requirements, forget whatever we dream up in the next 20 years.

Games take as much storage as a couple of hours of motion video.

Microsoft Products balloon out of control.

Look at the rate of growth of the Linux kernel source in the past 10 years!

The only thing that's changed : people get rid of less to make room.

No, we'll never have enough storage.