Slashdot Mirror
Large IDE Drives as Long-Term Archival Media?
"Backups are of no use without offsite archival copies so I plan to take one set of disks out of the pool, and archive them offsite on a quarterly basis.
However, I've heard horror stories about the data retention and usability off older disks which have been shelved for archival, for example disk stiction - where people try to restore data off of a 4 to 5 year old drive only to find that the disk won't spin up due to solidification of lubricants, or that they've experienced data degradation.
I'd be interested in the Slashdot crowd's opinion on using large IDE drives as an archival media. Clearly one possible problem is being able to get hold of a machine in the future with a suitable IDE interface to plug them into for restoration, but I can't see IDE disappearing within 5 years (maybe 10 though). I'm more interested in experiences and opinions on the suitability of the disks themselves for long-term archival.
- Is stiction still likely occur on newer makes of IDE drives or have manufacturers beaten the problems which caused this in the past?
- Likewise how likely is bit drop-out and general data degradation over say a 5 year and 10 year period, and what do people think would be the likely maximum feasible time that a shelved drive would be usable for?
- Any suggestions as to how would I need to store drives in order to minimize these types of problem and maximise their feasible life as archival media.
Without normal/regular use, you WILL have problems trying to read from them in 4-5 years time. Hell, the way most IDE drives are these days (note the recent reduction in warrenty time periods), you'll be lucky if the drives last 2 years even WITH regular use.
Since IDE HD manufacturers recently decreased their warranty period, I'd be *really* reluctant to trust 'em 10 years from now.
People here are saying, "Don't even think about using IDE!". Well he has no choice, does he? Tape has several drawbacks as the author mentions his comment to Slashdot. He has asked for advice on IDE. If this is not a feasible option, recomend some others (besides tape). Or ARE THERE NONE?
The Welkin: Online Music Reviews
that disks will rot, so you can't trust them.
I counter with this: tapes rot too. In fact, any tape older than one year that I've had to go back to has been worthless (read: it had deteriorated data).
Tape is a really bad medium to trust, but we keep buying it because we can't think of a better solution. Personally, I think the way to go is just to give up and admit that disk is not cheap. You need to back up your data to a live mirror system with identical storage (hourly rsync does a nice job) and then you need to arrage a service that can back up your data to remote live mirror systems. Note that in both cases I said "live mirror". You don't want a backup sitting on a cold box because you never know the quality of it until you need it.
The remote backup part is expensive, but it's the only reliable way. You seed it by tape (full backup to tape, and mail them to the vendor) and then use dedicated lines to keep a regular incremental update going.
If one of those two backup systems fail you know about it right away and you fix it. No more tapes rotting on a shelf only to be discovered when your data goes south.
I'm sorry, but 220GB easily handled by backup tape. With SDLT and AIT tape capacities exceeding 100GB per tape, two tapes can easily handle your load.
If you have the budget, get an autoloader so you can perform a full backup in one session, or two tape drives for that matter.
Personally, i am backing up 600+GB onto tape and it works well. I've had numerous IDE hard disk failures, yet not a single data tape failure so far.
Tapes are fine for backups, but I never expect to pull complete and usable data off of them after 6 months. Why? Tape degrades - it's nothing more than rust on platic. As humidity and temperature change, you can end up with a solid roll which will stick to your tape drive heads and result in whole patches of magnetic coating coming off. I worked on a project restoring data from 10+ year old reel-to-reel tape, and it was a nightmare. 1 out of 4 tapes was completely unusable.
Even worse, tape drive formats keep changing - and since tape drives are guaranteed to wear out, where are you going to get a working tape drive to restore data 5, 10, 15 years from now? I've gone through 3 tape drives in the last 8 years - thank god I got a CD burner early, that data I can still read (although it's about time to start recopying stuff from 1996.)
Basically, if you entrust your data to tape long term, you have to continuously copy that data to new tapes, and or new tape formats. Where tape has traditionally shined is as a short-term backup format, although with the drop in DVD-burner drives/media, and the high-cost of high-capacity tape drives/media, this may no longer be the case (assuming you get some peon to do the big backup on DVDs, and you get to do daily diffs - otherwise, having a bank of tape drives is cheaper on staff time.)
No flame, other than the term 'RAID 5'. Tapes aren't as dangerous as hard drives, but they can still mess up. It's not like they're garounteed beyond all odds. So a RAID 5 IDE array takes care of your data.
... it is possible to cost effectively mirror an IDE RAID system over broadband Internet and do it securely. If you are a major corporation surely your campus is large enough to simply run fiber to two corner and put mirrored backup at each location. ... buy a truckload of disks when you do the initial installation? (:
I'm currently using Dell NAS machines as archival backups.
Bonuses (as I see them):
Online 100mbit access to old data.
Cheap!
Fits in a physically small space.
Negatives:
Higher failure rate than tape. Pop fizzle, your data is gone.
Difficult to take off site.
Long-term replacement isn't really an option. (for RAID replacement)
The way we negate the negatives (double negative, is that a possitive?):
-Failure rate / Data loss is countered by RAID
-Taking it offsite
-Long term replacement of RAID drives
No sig for you. YOU GET NO SIG!
Obviously, you've never had a tape physically fail.
Maybe it's just me, but after the experiences I've had the last year with crappy tapes, I'm surprised the "tape as a backup medium" idea hasn't been seen for the farce that it is.
Backing up to IDE or SCSI? Good short term solution, but I don't think I'd trust my backup drives for more than 1 year, tops.
Burn to CD? Good long term solution, just not practical due to the file sizes involved. Burn to DVD isn't much better.
It's time for something new. Hell, maybe it will turn into the next "killer thing" and revitalize the economy.
I vote for soft bubble memory
Just once, I'd like it if someone called me "Sir".
Without adding, "You're creating a scene."
"I have $500 to spend on a backup solution for my 220GB data pool, and I was thinking of buying 4 120GB IDE drives along with an IDE RAID1 card and useing the array for backups, anyone have other ideas?"
"No way, you are insane. IDE is horribly unreliable and you will surely lose your data. You need a $6000 tape drive, if you can't afford it you are better off with no backups at all"
I love going down to the elementary school, watching all the kids jump and shout, but they dont know I'm using blanks.
Bits rot. Under the most perfectly controlled environment the damn stuff still goes bad. Be realistic, anticipate this, do everything you can to slow it down, but plan for it and make provisions when you first put your archiving strategy in place. Tapes are likely more robust the platters as there's fewer critical parts to go wrong but nothing is perfect.
Yes they're cheap but we've far less experience with these media then we do with tape and studies are showing that they dyes may not be as stable as first thought. Heck, there's even a bug out there that eats some of these. There's also the question of long-term standards in some cases like DVDs.
Nothings worse then losing one part of an archive at one site, another part at a different site, and being unable to easily reconcile the two to get a good whole set. Make sure that however you archive things, same media or different media, that partial archives can be reconciled.
Years ago there was a big scramble to recover the US Govt's 1950 Census. It had been stored on steel tape and the required Unisys readers were no longer. (Much of the data was available but the entire raw set wasn't.) Eventually a working one was built from cannibalized parts in museum and private collections but the lesson was clear: Don't depend on the readers. The same goes for the recent BBC Domesday Book debacle - nobody could read the optical disks. Any good archive scheme will call for the material to be re-read and re-transcribed regularly in order to ensure the entire recovery-chain still works: Hardware, software, OS's, etc. If recovery becomes difficult migrate the material.
All too often folks archive everything 'cause they're too lazy to determine what is actually necessary and what isn't. Combine this with the difficulty of later having someone unfamiliar try to winnow down the material and this becomes a real problem. Even worse is later trying to find the useful material among all of the dross. Establish clear policies of what can be archived and make folks justify their material. Just as importantly make sure the costs are clear up front, even to the point of charging them a rate covering several years of storage initially. Suddenly some pack-rat deciding EVERYTHING they've ever typed is potentially a goldmine isn't so funny. Lastly, run everything past Legal: Some of this they don't want hanging around any longer then necessary.
I don't read ACs: If a post isn't worth so much as a nom de plume to its author then I wont bother either.
However, if this is going to have *any* chance of working, you will need to read the drives on a regular basis. I would pop each drive in a machine and (in linux) do a "dd if=/dev/hdc of=/dev/null" to read the entire drive. I would do this monthly.
Why you ask? Because modern hard drives are sophisticated and they auto-correct errors *before* they become a problem. Hard drives will do things like correct recoverable errors and rewrite weak sectors when they encounter them. Thus if you go over every sector of the drive every once in awhile, you will use the drives auto-correction features to your advantadge (and protect against the drive fading, which would be my primrary concern, not stickage (which is easy to fix)).
Religion is a gateway psychosis. -- Dave Foley
One very important thing to consider : With certain types of tape drives, a misaligned head can render your tape media useless in another drive of exactly the same type. DLT is a good example of this. You can write and read to your heart's content on the same drive, but try to read a tape written in one drive on another and you can be sunk (professional data recovery experts with the proper tools can work around this, but it's expensive, and the whole point of this discussion was the need for "professional help" if certain parts of the hardware fail.)
-- Minds are like parachutes... they work best when open.
But is printing a whole character per bit, or even byte, efficient? I'm curious how much data a laser printer could store on a piece of paper. Is it realistic to expect individual bits printed at 300dpi to actually be retrievable? Perhaps on a good 600dpi or 1200dpi printer.
300dpi gives us almost 11KBytes per square inch. Figure 70 square inches on a letter page with 1/2" margins. That's 770KB. Print full duplex and you're looking at 1.5MB per page, or roughly a floppy disk (coincidence?) You wouldn't want to back up your MP3 collection, but for an archival method that is likely to last 100 years it's not too bad. Factor in compression and you are probably getting a 100x increase in storage density over plain text. Kind of a neat thought.
And to top it all off, I back it all up to a DDS-4 DAT autochanger. Yes, those six tapes will only hold 120gb, but the amount of important data on my disk drive is far less than 120gb (it is actually less than 20gb, including the original 44.1khz .wav recordings of all my original songs, and fits onto one tape easily).
Do you *REALLY* need a backup of your .mp3 collection?! Probably not. Do you *REALLY* need a backup of all those ISO CDROM images that you downloaded for fifty versions of Linux and a half dozen versions of FreeBSD? Probably not. But that's the sorts of things that are taking up 80gb plus on my hard drives -- i.e., utterly disposable cruft. Which is true for most personal computers.
Send mail here if you want to reach me.