Slashdot Mirror

Perhaps the medium is, but the related technology that makes the medium useful isn't. The drives can run thousands of dollars, and require specific technologies on the servers. On top of that you need software to run it, AND competent backup admins that can handle it.

Not that disk based solutions are significantly better, but they certainly have the ability to be significantly less complex ( which is always a good thing ).

The costs of SSDs start to look quite good when you're dealing with long-term preservation of a lot of data too. Yes, the storage cost itself isn't wonderful, but the fact that it is small and dense and solid state and able to be safely kept online (instead of having to physically move it about) greatly cuts the cost and risks of it. There's some interesting work going on in this area, and the answers being arrived at are often not at all intuitive.

That's the value of digital. Copies are perfect. Make lots.

This is similar to the answer that a lot of people interested in data preservation have come to over the years, including archivists and Kevin Kelly, one of the people behind the Long Now Foundation.

However, one thing that people might not think of from your advice is to keep things hot, to keep them moving. You can't tell when a bunch of burned DVDs start to go bad. Knowing whether your data is good requires constant integrity checks, to make sure that you can still get your data back.

Personally, the way I handle this is by running two hot servers in different locations with all of my critical data on both, with rsync updating nightly. With continuous checksumming turned on, that forces reads, so I know all the data is still safe.

But for those who aren't quite that geeky, then I think a service like Amazon's S3 is the way to go. You keep one local drive hot, and one set of data in the cloud. And you regularly verify that all the data you want is still there. Yes, this costs more money than another local disk. But S3 data sits in at least two spots in a secure data center on an infrastructure designed by people who really get reliability.

Libraries are also trying to maintain local copies of online journals. Think disaster recovery with offsite storage of backups. See http://www.lockss.org/. Please don't diss libraries. They have been trying to maintain access to human scholarship for centuries. Not that my college term papers were any great addition to the store of knowledge, but I saved them on 5 1/2 inch floppies and couldn't easily open them ten years after they were written. Not every website from the early days is backed up. I'm sure everything can't been saved in the Internet Archive http://www.archive.org/. That's part of what libraries do, they select what to save for their local users, because nobody has the resources to permanently save everything. And just when someone has the hubris to think they can, a natural disaster like the fire of Alexandria or Hurricane Katrina comes along to show us our folly.

This isn't news -- at least not to those of us who deal with data.

The typical procedure is to do a media refresh (ie, copy it) every few years, and to check for damage. There are concepts like LOCKSS (Lots of Copies Keeps Stuff Safe), so those joking about BitTorrent aren't that far off, but it's a little more structured than that.

Dan Cohen gave a talk recently on "Can Today's Scientific Data Be Preserved? The Specter of a 'Digital Dark Age'", which touched on not only the issue of media failure, but also the loss of the knowledge to extract the encoded information. (much like the 'lost languages' that we don't understand now, how do we make sure that future generations have the necessary hardware and software to get the data back out?)

What's disapointing is just how fast the media is failing. Vendors give a 'mean time to failure' estimate that's based on perfect storage, and that they have no real ways of testing (because, well, if you say it's 40 years, are we going to have to wait 40 years before using it?). Even if you're duplicating your tapes, what happens when all of the copies were put on the same potentially bad batch of tapes?

Quite likely, we're going to lose data. And some of it's going to be because we no longer have copies of the data. The rest is going to be lost because there's so much crap being saved that doesn't need to be that we can't find stuff that still has value in the future.

Digital material is the easiest to back-up and keep secure.

Libraries are using the LOCKSS Program to build libraries! With publishers, our community is working to retain libraries as long-term memory organizations in the electronic environment.

People with responsibility for scholarly assets agree: digital preservation is important. With your help, librarians and publishers are asking two fundamental questions: From this moment on, who will have custody of societies' electronic information? From this moment on, who will control and govern societies' electronic archival assets?

Join us! The community is working to insure important scholarly assets remain available in a distributed, self-repairing, robust, digital preservation system.

LOCKSS is OAIS compliant, LOCKSS migrates content forward in time, and LOCKSS continually audits and repairs the content. LOCKSS is open source software -- the system is freely available for you to examine and use.

What appears to have been established here is that electronic storage media does not last for long (compared with, as mentioned in previous comments, written text or stone engravings), irrespective of the filesystem used on that media. Therefore perhaps a single archive copy is not enough. What about a distributed system such as LOCKSS http://www.lockss.org/lockss/Home would be better for archive storage as it essentially abstracts the hardwaee and filesystems that the data is stored on.

It's called "Lots of Copies Keeps Stuff Safe" and it's even got standards and software.

This was also my conclusion, print out the pictures and stick them in a nice album using starch-based glue. Will be very quaint in 2021.

But what about the videos? :-) Flip book?

I don't really want to present my daughter with a pack of SCSI DLTs and the hardware to go. Surely something more simple... portable DVD player, maybe, complete with power supplies. If only I could be sure DVDs would last.

Probably the best option would be a link where she can find her data online and live. Idea for a new kind of business? www.everlastingdata.com, guaranteed archival for 100 years!

I threw up a comment about LOCKSS, which does something like this for libraries.

Here is a project based on peer-to-peer concepts that aims to preserve information over long periods of time without depending on specific media, readers, etc.

Seems far more realistic, after all this is what most of us do with valuable data, we copy it from hard disk to hard disk over time.