Large IDE Drives as Long-Term Archival Media?

PlatterMan asks: "The question of how to cope with backing up disk drives which are rapidly increasing in size, onto tape and other backup devices which aren't scaling in size as quickly isn't new to Slashdot. Neither is the use of single, raided, and removal disks as backup devices, this has been covered numerous times on Slashdot in e.g. here and here. One thing I haven't really seen discussed however is the feasibility of disk drives as medium to long-term archival media, say 5 to 10 years. Like many people I'm in the position of now having multiple machines with a combined data pool of about 220 Gig, and backing up these onto DDS or DLT tapes is slow and manual to do, and expensive in tape costs. So I'm looking to add a removal drive bay to my primary backup machine and pick up a bunch of large IDE drives, so that I can do regular disk to disk backups over 100 Meg Ethernet (and for my machines which are in cages, over the Net) pulling out and alternating the backup drives on a 3-way backup cycle."

"Backups are of no use without offsite archival copies so I plan to take one set of disks out of the pool, and archive them offsite on a quarterly basis.

However, I've heard horror stories about the data retention and usability off older disks which have been shelved for archival, for example disk stiction - where people try to restore data off of a 4 to 5 year old drive only to find that the disk won't spin up due to solidification of lubricants, or that they've experienced data degradation.

I'd be interested in the Slashdot crowd's opinion on using large IDE drives as an archival media. Clearly one possible problem is being able to get hold of a machine in the future with a suitable IDE interface to plug them into for restoration, but I can't see IDE disappearing within 5 years (maybe 10 though). I'm more interested in experiences and opinions on the suitability of the disks themselves for long-term archival.

• Is stiction still likely occur on newer makes of IDE drives or have manufacturers beaten the problems which caused this in the past?
• Likewise how likely is bit drop-out and general data degradation over say a 5 year and 10 year period, and what do people think would be the likely maximum feasible time that a shelved drive would be usable for?
• Any suggestions as to how would I need to store drives in order to minimize these types of problem and maximise their feasible life as archival media.

Thanks!"

Comments

Mission Critical Data..

[xchino]

Speaking from experience I can give this bit of advice for archiving critical information. Use a solid state device, don't even consider a magnetic solution, unless losing some or all of the data won't ost you your job.

Steve Gibson

[Jucius Maximus]

Please don't flame me for quoting Steve Gibson, but I think he's right on this account: "There are only two kinds of hard drives -- Those that have failed and those that will fail."

Hard drives are not non-volatile storage.

Not a horrible Idea, just not a great one.

[briancnorton]

Using magnetic media to back up magnetic media isnt the greatest idea in the world, but it can work. Hard drives fail, and when they do, you want to have the data available so that you can get to it. The IDEAL way to do this is to contract an outside company or manage for yourself a backup server which does incremental backups as often as you need and periodically burns them to a more permanant media like DVD. If you cant afford this or dont like the idea, then you can burn DVDs on your own. A good program will track files for incremental backup and 220 gigs can fit on something like 50 DVDs, with maybe 1 more per session (assuming that not all files are constantly changed) Obviously a lot depends on what you have, how much money you are spending, and what you need.

Why Tape Is Good

[Jucius Maximus]

Tape may be inconvenient but it is still a true backup medium. With hard drives, the reading and writing hardware are enclosed with the platters. So when the read head of the HDD fails, your data may be 100% intact on the platters but you can't get at it without professional help. How many other parts in the HDD could fail without harming the platters? A lot!

With tape, the failure of a tape drive doesn't separate your from your data (unless it catches on fire with the tape in it or something.) You can just get a new tape drive and you are good to go again.

Thus, tapes are very good because the storage medium and the read/write hardware are separated and not interdependent.

Re:Why Tape Is Good

[Rolo Tomasi]

Thus, tapes are very good because the storage medium and the read/write hardware are separated and not interdependent.

Bullshit. Tapes are intended as a short-term backup medium. Google for NASA magnetic tapes, and you will find a lot of interesting stories. Like e.g. this one:

Right now, ACRES is updating storage of 120,000 gigabytes of data collected since 1979, primarily from remote-sensing Landsat satellites passing over Australia. Landsat images are among the most voluminous of space-based data, making ACRES one of the largest data repositories of its kind in the world, Trezise said.

The data now are housed on optical tape, having been rescued from disintegration in the early 1990s from aging high-density magnetic tapes. That first rescue operation occurred just in the nick of time, Trezise said, since the magnetic tapes were starting to get sticky on their spools.

Re:Why Tape Is Good

[sunspot42]

Yes, tape is a "true backup medium", but it is *not* a suitable medium for longterm archival storage - at least, none of the affordable formats up till this point (like the DAT-derived DDS format) have been. There's a big difference between a backup medium (a copy that's probably replaced every day / week / month and is intended for use in the immediate future) and archival storage (a copy that's intended for use 5+ years in the future).

While the failure of a tape drive won't separate you from your data (unless the drive damages the tape at the same time it fails . . .), tapes themselves deteriorate over time. Here's an article about the problems the National Archives here in the United States have encountered with preserving copies of the Nixon tapes on DDS's audio cousin DAT. An excerpt:

"During the National Archives' routine monitoring of the tapes'
condition, the analog reel-to-reel copies have shown no signs of
deterioration whereas there is an estimated 5-10% catastrophic failure
rate among the DATs in the collection. There appears to be no pattern
to the failures. It has occurred on new tapes that were recorded six
months ago, and it has occurred on tapes that were recorded six years
ago. It has occurred on all brands of DAT purchased throughout the
previous seven years. Accordingly, the archivists routinely reduplicate
these DATS on multiple copies. As insurance, archivists also transfer
DAT copies back onto analog reel-to-reel preservation copies. Unlike
the other preservation analog copies, these copies have not been
filtered and closely "mirror" the original tapes. Therefore, in the
future when technology has progressed, the archivists can retrieve
conversations that are extremely close to the original audio recordings
and enhance these with the latest technology."

Leading audio preservationists have issued their own warning. This company deals with audio preservation, and has some interesting things to say about tape formats - analog and digital.

Of course, DDS tapes have supposedly been manufactured to a higher standard than their Audio DAT cousins, sport finer particles and stronger binders, and the format includes additional error correction and redundancy. Still, these issues with a modern tape format like Audio DAT are not an encouraging sign for those seeking to perform archival storage using DDS and it contemporaries. HP for example only claims a 10 year archival life for DDS. Contrast that with the 75-100 year lifespans Kodak and TDK are claiming for CD-R.

These longevity issues won't just be confined to older tape backup formats though, if history is any indication. It's the nature of the medium. I think Sony is currently claiming a 30-year lifespan for AIT, and HP something similar for their new format, but of course we only have a couple of years' experience with them so far, and given the incredible data density of those formats, if something should go wrong with either of them the results could be catastrophic. Unexpected deterioration has certainly happened with tape before - witness this article composer and synthesizer pioneer Wendy Carlos put on her website, as well as her own experience with her older tape masters.

Hard drives certainly aren't a great archival medium either, but I wouldn't be so quick to assert that tape is superior. At least drives have the advantage of being sealed from the outside atmosphere, and contain within them all the logic and hardware required to extract that information in the future. The only big issues I can see are, will there still be equipment to interface with them in 10 to 20 years (probably, since IDE is so widespread) and will the drives still spin up in 10 to 20 years (who knows). It's that second issue that's the real buzzkill for HD's as a longterm storage medium. Manufacturers won't even issue a decent warranty on drives anymore. What does that say about their planned longevity?

Me, I think your best bet is DVD. But if you really want to be able to read that data in the future, I'd suggest copying it to at least two different formats, perhaps AIT *and* DVD. Don't forget to check on it every few years, too. If there's any sign of deterioration, you'll hopefully be able to make another clone before the failure becomes catastrophic (perhaps to a superior format that hasn't even been invented yet). If you want something you can just throw in a hole and forget about, sorry - that media doesn't exist.

Long Term Storage

[caseydk]

The Library of Congress is attempting to answer this question as they have huge amounts of media that is on highly degrading (nitrate-based films) materials.

Their answer? A huge RAID array starting at 180TB and growing steadily over time.

Your answer? Probably figure out which of the data is fixed and which of it changes and attempt to back up accordingly. Does all 220gb change on a weekly basis? That seems unlikely...

Re:Ask who's actually doing it.

[DJPenguin]

Well, don't know about LucasFilm, but Pixar use massive tape libraries (we are talking robots with 100+ drives and tens of thousands of slots.)

Incremental backups every HOUR, tape drives spinning all the time. They are a customer of the company I work for. (Veritas)

Re:Tapes *is* the right medium for long term backu

[glesga_kiss]

I've had numerous IDE hard disk failures, yet not a single data tape failure so far.

You speak of not having tape failures, but you omit one important fact; how many times have you successfully retrieved data from tape?

IDE disks will fail from continual use, and that failure will generally be obvious, but what way do you have of knowing that you genuinely don't have any tape failures, if all you are doing is rewriting over the same tapes?

Re:A lot of folks will say....

[Jerf]

On a smaller scale (personal), this is essentially what I do.

First, only some personal data is critical, not the GBs of operating systems and programs I can redownload/recompile if necessary. Things like documents, saved games (you'd think it's unimportent until you play the first 2/3s of Fallout 2 five times and can't stomach getting far enough to see how it all turns out, because you'd have to play that 2/3s again...), email maybe, whatever, but some limited amount. 10MB can go a long way... that's a lot of programming, for instance. (Been working on a project for about half a year now and I'm just ready to break 300KB of code...)

Then, set up a live backup amounst all the disks you have on various machines. I use unison so that I can change files in the repository on any machine and have the changes propogate correctly, instead of the unidirectional updates rsync does.

Use symlinks to put everything you need into one directory, and tell Unison to follow the symlinks, not archive them directly. Then just run that every so often on the machines, and you're set.

Once more of my family gets set up with always-on connections, I intend to set up a family-level repository of backed up files with Unison, so that "off-site backups" are a weekly script run without intervention by the family, making off-site backups across the state (or country, or world) easy. This will protect the scanned pictures and other things in the family heritage easily and effectively.

Which reminds me, the first always-on connection just came online and I really ought to talk to that member about a reciprocating backup setup...

Re:GraniteDigital is what I use

[Jeremy Erwin]

paper burns at 451 degrees F (232 Celsius)
media starts to melt at 125 degrees F (52 Celsius)

A fireproof safe thats rated for paper storage only isn't going to cut it.

Re:Sod CD-R! Go With DVD recording

[ibennetch]

And for keeping tabs on what is on which disk... I've been using a freeware program called "Cathy" (I don't have any links)...Although I don't know whether it'll do DVD's, I haven't tried.

Cathy is avalible for download here. According to these sites it will handle many disk formats ("CD-ROMs, LS120, Iomega Zip and Jaz disks, or even diskettes"). The link to the home page is broken.

Re:warranty period

[fishbowl]

"Who the fuck has 220GB of personal data? "

I'm getting there, in audio data.

My own music, that I write and record, so, going down to the store to replace it isn't exactly an option.
It's also on DAT, and on CD audio, so you could say
I have a backup, but that's not really true -- the DAT is the source material, and a CD would represents one view of some of the data.

Am I going to buy a $65,000 SAN tape library machine, just because I'm getting into volume? (No.) Would I like an inexpensive solution that is less cumbersome than CDR? (Yes.)

Re:Tapes are a expensive waste of time

[JWSmythe]

Burnt CD's (like you'd use at home) have a shelf-life of about 10 years. Then the medium starts to oxidize (the metallic film, not the plastic itself), and flakes..

So, you have a 10 year backup.. It all depends on how important your information is. If it's that important, I'd put it on a RAID5 where it can be monitored. As drives fail, replace them. Continue migrating to newer arrays in the future.. Expensive, but I konw perfectly well any drive will fail. I've had several hard drives, that would fail to spin up properly after sitting for a few days.. Some of them, they only way they'd start is if I hit the side of the drive with a screwdriver..

You have to expect failure of your medium. If he wants to be very sure, use multiple backup methods.. RAID5's in multiple locations, and CD's. Someone will need to monitor all of it occasionally. Make sure the RAID's (and their associated machine) are running. Make sure the CD"s are oxodizing...

Even floppy disks die of old age. I found a few boxes with Novell Unix. They're is years old, and most of the floppies couldn't be read. They were brand new, still in the sealed boxes and envelopes. I finally found a boot disk that would work, but it would bomb out trying to install under VMWare (I was curious).

Is that data really going to be useful to you in 10 years? That's the important question. People are all paranoid of loosing Email and the like now, but in 1 year they don't care about it any more. In 2 years, it's just wasted space. In 10 years, they won't even know who or what they were talking about..