Slashdot Mirror
Large IDE Drives as Long-Term Archival Media?
"Backups are of no use without offsite archival copies so I plan to take one set of disks out of the pool, and archive them offsite on a quarterly basis.
However, I've heard horror stories about the data retention and usability off older disks which have been shelved for archival, for example disk stiction - where people try to restore data off of a 4 to 5 year old drive only to find that the disk won't spin up due to solidification of lubricants, or that they've experienced data degradation.
I'd be interested in the Slashdot crowd's opinion on using large IDE drives as an archival media. Clearly one possible problem is being able to get hold of a machine in the future with a suitable IDE interface to plug them into for restoration, but I can't see IDE disappearing within 5 years (maybe 10 though). I'm more interested in experiences and opinions on the suitability of the disks themselves for long-term archival.
- Is stiction still likely occur on newer makes of IDE drives or have manufacturers beaten the problems which caused this in the past?
- Likewise how likely is bit drop-out and general data degradation over say a 5 year and 10 year period, and what do people think would be the likely maximum feasible time that a shelved drive would be usable for?
- Any suggestions as to how would I need to store drives in order to minimize these types of problem and maximise their feasible life as archival media.
Without normal/regular use, you WILL have problems trying to read from them in 4-5 years time. Hell, the way most IDE drives are these days (note the recent reduction in warrenty time periods), you'll be lucky if the drives last 2 years even WITH regular use.
the reason ide disks are so cheap these days is their components are substandard. have you ever wondered why SCSI disks are still so much more expensive? and the people who use SCSI disks _also_ spring for expensive tape backups?
this is because disk is a poor medium for backup and long term storage.
the problem youre running into here is simple. IDE allows you to have big datacenter type storage without the capital expenditure. however, what the IDE drive peddlers dont tell you (and what you fail to grasp) is that big datacenter type backups are very expensive. and cutting costs is not going to help you at all. good luck getting *anything* off an ide disk thats been sitting for ten years.
For example, DVD.
Make sense?
Good.
...
...you're walking down the hall with a 3 foot stack of drives and you trip over an ethernet cable...and all the drives take a sailing course through the air and land on the concrete floor.
I'm not a betting man, but I bet if that were a stack of DLT tape, you might still be able to read them after that hypothetical incident.
// Agent Green (Ian / IU7 / KB1JQO)
// IEEE 802.3: All 10base Are Belong To Us
Since IDE HD manufacturers recently decreased their warranty period, I'd be *really* reluctant to trust 'em 10 years from now.
I think it would be a bad idea to rely on IDE drives as one's only source of backup. Especially if you aren't planning on using any stripping or parity. The large IDE drives are, the more prone to failure they appear to be. Ask anyone thats bought a 60-100 IBM deathstar drive lately. The added wear that would occur from joustling them around as you pull them in and out of the drive bays all the time seems like it would also make the time between failures greater. What is proposed in the story might work fairly well for a home user, but I think it would fall apart in a business setting.
People here are saying, "Don't even think about using IDE!". Well he has no choice, does he? Tape has several drawbacks as the author mentions his comment to Slashdot. He has asked for advice on IDE. If this is not a feasible option, recomend some others (besides tape). Or ARE THERE NONE?
The Welkin: Online Music Reviews
Used to be in the data backup biz, you should really start with evaluating what you are actually backing up. Most people backup applications and temp files that really are not going to help much. Also, do you really need to archive all of that stuff even if you are anal? Another thing to consider is, will the media be supported and will you have the proper drivers for the disk drives handy. 220 Gigs is surely still in the land of tapes, I hate them more than most, but would not suggest the use of an IDE Hard Drive. my 2 cents
that disks will rot, so you can't trust them.
I counter with this: tapes rot too. In fact, any tape older than one year that I've had to go back to has been worthless (read: it had deteriorated data).
Tape is a really bad medium to trust, but we keep buying it because we can't think of a better solution. Personally, I think the way to go is just to give up and admit that disk is not cheap. You need to back up your data to a live mirror system with identical storage (hourly rsync does a nice job) and then you need to arrage a service that can back up your data to remote live mirror systems. Note that in both cases I said "live mirror". You don't want a backup sitting on a cold box because you never know the quality of it until you need it.
The remote backup part is expensive, but it's the only reliable way. You seed it by tape (full backup to tape, and mail them to the vendor) and then use dedicated lines to keep a regular incremental update going.
If one of those two backup systems fail you know about it right away and you fix it. No more tapes rotting on a shelf only to be discovered when your data goes south.
I'm sorry, but 220GB easily handled by backup tape. With SDLT and AIT tape capacities exceeding 100GB per tape, two tapes can easily handle your load.
If you have the budget, get an autoloader so you can perform a full backup in one session, or two tape drives for that matter.
Personally, i am backing up 600+GB onto tape and it works well. I've had numerous IDE hard disk failures, yet not a single data tape failure so far.
So, how is Pixar archiving it's film data? How about LucasFilm? I'd think from the amount of data they work with, thos guys would be the best at answering that question.
;-)
Personally, for long term storage, I'd go with redundant backups of differing media. Maybe hard drives (stored properly in anti-static bags with silica gel), as well DLT stored in a similar fashion. Increase your odds of support by future architecture.
For daily backups, hard drives are surely the way to go. Faster, cheaper, easy to replace, longer lasting media in my opinion. Anyone who says otherwise is trying to cover their job as a tape changer.
Someone, please shake me from this wide-awake nightmare.
What we have is a plethora of opinions about what's wrong with IDE. Why don't you solve your problems. 1) It's unreliable over the long run. - Have multiple disks that will be unlikely to fail at exactly the same time. 2) Remote storage - Get a friend that has some space and get him to store a disk for you. 3) Jostling hard drives might reduce life expectancy. - Perhaps try remote incremental backups over the net. Do one large copy locally then locate the disk at a remote connection and backup incrementally. ( remember the local part for the first copy - I got a note from my ISP when I did it over the net.) 4) Lubricant failing in disk. - Two things, you can either replace the disks on a cycle, perhaps every two or three years. ( prices keep going down) or keep the disks spinning on a remote computer that is UPS protected. There are many other ways to do this. What we need here is people with solutions. Not more problems. Think outside the box.
In the battle between good and evil, evil has more fun...
If the tape drive electronics fails, you can get another tape drive and still read the tape. If the IDE drive electronics fail, the data on the drive is unreachable without massive and expensive intervention.
...phil
"For a list of the ways which technology has failed to improve our quality of life, press 3."
Many people forget that remote backups require no on-site hardware or software and don't require you to spend hours upon hours configuring things.
Even better is that any flood, tornado, or fire at your house or business will not ruin your tape, dvd, cd, or hard drive backups. You simply connect to your remote backup location and restore your old data onto your new hardware. It's that simple, and it's cheap in comparison to spending $3,000 on a tape backup device that only stores 150GB of data per cartridge.
You may want to see if this remote backup company has services that fit your needs (I don't work for them, so it's not a plug). Basically, they state the following as the main appeals to remote backup:
Your data is continuously backed up as it changes, 24 hours a day, so it's always up to date. And it's stored electronically at Iron Mountain® data centers, where more than half the Fortune 500 protect their data.
No-Wait Recovery - Instantly recover your data to the point of failure, eliminating downtime and data loss from relying on a previous night's backup. And a unique web interface allows you to initate restores from any Internet browser, anywhere.
No Tapes, No Hassles, Lower Costs - Tape-less backup and recovery means no hardware or software to buy and a fully automated process requiring little employee time or resources. Lower your data protection costs while freeing IT resources for other tasks.
If you celebrate Xmas, befriend me (538
I know it sounds like a stupid question, but why are you backing up data? What are you trying to solve
Short term failure
A luser makes a mistake, or there's a glitch in last night's source code library, and all your current data is foobarred. In scant minutes, you can restore lost data from overnight backups, (or even hourly incrementals), and you are the hero. Realistically, you're just doing your job, and you'll never get thanked for it.
Complete Failure
In the event of a building fire/server room flood/earthquake/Act Of Dog, then you may need to retrieve all your companies data from as near back as possible. This backup should be off-site, and as frequent as feasibly possible
Long Term storage
This is for archiving of a project, etc, and should be off-site. Also for archiving source code in case your company goes belly-up, so that customers can still use and modify your software (in escrow).
Ask yourself which scenario you are dealing with, then the answer as to which media is the one to use may be clearer.
Tapes are fine for backups, but I never expect to pull complete and usable data off of them after 6 months. Why? Tape degrades - it's nothing more than rust on platic. As humidity and temperature change, you can end up with a solid roll which will stick to your tape drive heads and result in whole patches of magnetic coating coming off. I worked on a project restoring data from 10+ year old reel-to-reel tape, and it was a nightmare. 1 out of 4 tapes was completely unusable.
Even worse, tape drive formats keep changing - and since tape drives are guaranteed to wear out, where are you going to get a working tape drive to restore data 5, 10, 15 years from now? I've gone through 3 tape drives in the last 8 years - thank god I got a CD burner early, that data I can still read (although it's about time to start recopying stuff from 1996.)
Basically, if you entrust your data to tape long term, you have to continuously copy that data to new tapes, and or new tape formats. Where tape has traditionally shined is as a short-term backup format, although with the drop in DVD-burner drives/media, and the high-cost of high-capacity tape drives/media, this may no longer be the case (assuming you get some peon to do the big backup on DVDs, and you get to do daily diffs - otherwise, having a bank of tape drives is cheaper on staff time.)
No flame, other than the term 'RAID 5'. Tapes aren't as dangerous as hard drives, but they can still mess up. It's not like they're garounteed beyond all odds. So a RAID 5 IDE array takes care of your data.
... it is possible to cost effectively mirror an IDE RAID system over broadband Internet and do it securely. If you are a major corporation surely your campus is large enough to simply run fiber to two corner and put mirrored backup at each location. ... buy a truckload of disks when you do the initial installation? (:
I'm currently using Dell NAS machines as archival backups.
Bonuses (as I see them):
Online 100mbit access to old data.
Cheap!
Fits in a physically small space.
Negatives:
Higher failure rate than tape. Pop fizzle, your data is gone.
Difficult to take off site.
Long-term replacement isn't really an option. (for RAID replacement)
The way we negate the negatives (double negative, is that a possitive?):
-Failure rate / Data loss is countered by RAID
-Taking it offsite
-Long term replacement of RAID drives
No sig for you. YOU GET NO SIG!
Obviously, you've never had a tape physically fail.
Maybe it's just me, but after the experiences I've had the last year with crappy tapes, I'm surprised the "tape as a backup medium" idea hasn't been seen for the farce that it is.
Backing up to IDE or SCSI? Good short term solution, but I don't think I'd trust my backup drives for more than 1 year, tops.
Burn to CD? Good long term solution, just not practical due to the file sizes involved. Burn to DVD isn't much better.
It's time for something new. Hell, maybe it will turn into the next "killer thing" and revitalize the economy.
I vote for soft bubble memory
Just once, I'd like it if someone called me "Sir".
Without adding, "You're creating a scene."
"I have $500 to spend on a backup solution for my 220GB data pool, and I was thinking of buying 4 120GB IDE drives along with an IDE RAID1 card and useing the array for backups, anyone have other ideas?"
"No way, you are insane. IDE is horribly unreliable and you will surely lose your data. You need a $6000 tape drive, if you can't afford it you are better off with no backups at all"
I love going down to the elementary school, watching all the kids jump and shout, but they dont know I'm using blanks.
That is exactly the reason I created this account. I love to put up insightful/informative posts that are laced with tons of profanity. I just get a kick out of watching people get pissed off that there IS useful information, but that there is also a lot of profanity. Just goes to show you that language means nothing. It's my way of getting back at the grammar nazis and the people who feel that informative/insightful mods should only go to people who don't use profanity. It's like they have some kind of weird disease that makes them think that just because some swears a whole FUCKING lot, mans that they don't have anything useful to say. This isn't true. Some of us just swear because it has style...
Un-news
>> Not all of us home computer users with 200GB of files happen to have $5,000 in their back pockets for a backup device.
For most, the only mission critical stuff is maybe their quicken files, some documents, and whatnot. I can backup whats actually important to me on a couple of CD-Rs.
If you had 200GB of stuff that was worth backing up, you'd pay what it costs to do so.
It's just a cost/benefit thing.
Is your MP3 collection worth 5,000$?
1000$ for a redundant offsite raid-5 array?
500$ for 2 more rotated backup drives?
How about 100$ to dump it onto CD-R or DVD-R?
Or simply move the data from IDE to the new format when the situation arises. This will be low-cost because of the ever falling cost per gigabyte. A few hundred gigs will have negligable costs in 5-10 years.
Bits rot. Under the most perfectly controlled environment the damn stuff still goes bad. Be realistic, anticipate this, do everything you can to slow it down, but plan for it and make provisions when you first put your archiving strategy in place. Tapes are likely more robust the platters as there's fewer critical parts to go wrong but nothing is perfect.
Yes they're cheap but we've far less experience with these media then we do with tape and studies are showing that they dyes may not be as stable as first thought. Heck, there's even a bug out there that eats some of these. There's also the question of long-term standards in some cases like DVDs.
Nothings worse then losing one part of an archive at one site, another part at a different site, and being unable to easily reconcile the two to get a good whole set. Make sure that however you archive things, same media or different media, that partial archives can be reconciled.
Years ago there was a big scramble to recover the US Govt's 1950 Census. It had been stored on steel tape and the required Unisys readers were no longer. (Much of the data was available but the entire raw set wasn't.) Eventually a working one was built from cannibalized parts in museum and private collections but the lesson was clear: Don't depend on the readers. The same goes for the recent BBC Domesday Book debacle - nobody could read the optical disks. Any good archive scheme will call for the material to be re-read and re-transcribed regularly in order to ensure the entire recovery-chain still works: Hardware, software, OS's, etc. If recovery becomes difficult migrate the material.
All too often folks archive everything 'cause they're too lazy to determine what is actually necessary and what isn't. Combine this with the difficulty of later having someone unfamiliar try to winnow down the material and this becomes a real problem. Even worse is later trying to find the useful material among all of the dross. Establish clear policies of what can be archived and make folks justify their material. Just as importantly make sure the costs are clear up front, even to the point of charging them a rate covering several years of storage initially. Suddenly some pack-rat deciding EVERYTHING they've ever typed is potentially a goldmine isn't so funny. Lastly, run everything past Legal: Some of this they don't want hanging around any longer then necessary.
I don't read ACs: If a post isn't worth so much as a nom de plume to its author then I wont bother either.
As a general-purpose backup medium, hardcopy isn't very practical.
BUT...
Consider that printouts on archival paper can be expected to last 100+ years. Tapes flake and fade, disks stick, cd's oxidize. Nothing else even comes close to paper! (well, maybe stone tablets...) Human readability is another plus.
For certain types of info, a printout is definitely the best choice.
I think most people have missed the point here.
If the backup is of data that must be archived and stored long term then it's worth sitting down and copying it onto a pile of DVD-R's or similar. They will still be readable when a hard disk will have long since siezed up.
On the other hand if the backup is simply to guard against those "what do you mean you typed rm -rf *" moments then a copy on removeable hard disks will be more than adequate.
Personally I think a combination of the two is probably best, optical / tape media for archiving, and HD's for everyday stuff. For example, I don't need all my MP3's on tape but my wife has a copy of her thesis on CD-R.
Steve.
A latent existence
However, if this is going to have *any* chance of working, you will need to read the drives on a regular basis. I would pop each drive in a machine and (in linux) do a "dd if=/dev/hdc of=/dev/null" to read the entire drive. I would do this monthly.
Why you ask? Because modern hard drives are sophisticated and they auto-correct errors *before* they become a problem. Hard drives will do things like correct recoverable errors and rewrite weak sectors when they encounter them. Thus if you go over every sector of the drive every once in awhile, you will use the drives auto-correction features to your advantadge (and protect against the drive fading, which would be my primrary concern, not stickage (which is easy to fix)).
Religion is a gateway psychosis. -- Dave Foley
If that sounds like a few decades of experience in the storage industry, well, it is! Never bet the ranch on a single tape. Make multiple backup tapes, always verify with read after write, and always disperse the copies to geographically diverse locations. Like the Boy Scouts, Be Prepared!
Oh yes, have spare drives on hand for whatever tape format you use. It will become obsolete some day, long before entropy takes its toll on the tapes' data.
One very important thing to consider : With certain types of tape drives, a misaligned head can render your tape media useless in another drive of exactly the same type. DLT is a good example of this. You can write and read to your heart's content on the same drive, but try to read a tape written in one drive on another and you can be sunk (professional data recovery experts with the proper tools can work around this, but it's expensive, and the whole point of this discussion was the need for "professional help" if certain parts of the hardware fail.)
-- Minds are like parachutes... they work best when open.
If you want to trust your data w/ Iron Mountain, go right ahead! Never have I seen a more incompetently run corporation, especially when you consider what they do, which is "safeguard" your data. Safeguard, yeah, like putting ID labels on DLT tapes so they can't be used again without removing the polyester film tag(very hard to remove) from the tape door. Or, simply leaving your data in a hallway instead of following the instructions left for them when they make drop offs. We don't mind the whole world having access to our sensitive data, no - the hallway is a great place to leave something important unattended. These are things that we actually had to deal with, aside from the missed pickup dates, endless billing problems after service was cancelled, and the absolute living hell we had to go through to get our last batch of tapes back once we decided to drop their service.
All this because there is no competition in the world of offsite data storage.
But is printing a whole character per bit, or even byte, efficient? I'm curious how much data a laser printer could store on a piece of paper. Is it realistic to expect individual bits printed at 300dpi to actually be retrievable? Perhaps on a good 600dpi or 1200dpi printer.
300dpi gives us almost 11KBytes per square inch. Figure 70 square inches on a letter page with 1/2" margins. That's 770KB. Print full duplex and you're looking at 1.5MB per page, or roughly a floppy disk (coincidence?) You wouldn't want to back up your MP3 collection, but for an archival method that is likely to last 100 years it's not too bad. Factor in compression and you are probably getting a 100x increase in storage density over plain text. Kind of a neat thought.
Most of the storage technology R&D money is going into hard drives which is why they are so inexpensive and will continue to be. Tape storage technology R&D money is simply unable to compete.
I've predicted for a long time that tapes will become obsolete and hard drives will be used to back up hard drives. A very interesting example are firewire hard drives. A small premium on the cost of an already cheap dist drive, you have a fully plug and play high performance solution. For the cost of a tape drive and 10 tapes you can buy 10 firewire hard drives and rotate the drives instead of tapes.
The advantages are enormous.
Fast recovery time
Fast seek time
Proven reliable technology and much more reliable that tape.
Inexpensive and becoming less so
Easily networkable (just mount the drive on the network) and fully supported by any self-respecting OS
When you're done with the backup, you have some spare storage for anyone..
There may be some work to do to make some of the backup software talk to a hard drive, but there are probably so many different solutions you can use, you probably don't need to worry.
I bought 2 firewire drives a while ago and have never looked at tapes since. Love it....
Just encode your data into a pr0n video and share it on gnutella. That data will never be 'lost' !
And if you're really clever, you would take advantage of the fact that levels of greyscale are easily discernable. Leave a seperation space on all sides of each dot ( so they're more easily decoded ) to form a grid system. Yes, your storage capacity will drop by a factor of 4, but you can easily encode 8 bits ( a factor of 256 ) into the dot.
Most laserprinters can do 8-bit greyscale.
But for redundancy:
- Make two dots for each 8-bit piece of data, the 8-bits and it's complement. This is only good at error detection, although theoretically you could add error correction at a capacity cost.
- Add 256 calibration dots every few inches to make up for aging of the ink and media. We can assume that the cameras will have much higher resolution than the printer, so they can tell the difference even if the levels have faded together.
You could pack a whole lot of data on paper if you put your mind to it.
Man is the animal that laughs.
And occasionally whores for Karma.
My god, the sole voice of reason here!
Anyone that trusts these new large-capacity IDE drives for backup is an idiot. IDE wasn't reliable at the best of times traditionally, but the large-capacity ones have created a whole new level of unreliability.
Especially if the brandname is Western Digital or Maxtor.
Not to be a naysayer...but I will anyways. What happens in 30 years when a massive electromagnetic field wipes out all digital machines (possibly in conjuction with some attempt by humans to wipe out the robots taking over the world...those damn robots!)? By then 15 years of scientific publication may be more or less completely digital, and all gone, gone. Better hope we never lose access to that handy-dandy resource electricity....
only infrmatn esentil to understandn mst b tranmitd
Needless to say that google doesn't archive binaries.
where's all that Karma?
And to top it all off, I back it all up to a DDS-4 DAT autochanger. Yes, those six tapes will only hold 120gb, but the amount of important data on my disk drive is far less than 120gb (it is actually less than 20gb, including the original 44.1khz .wav recordings of all my original songs, and fits onto one tape easily).
Do you *REALLY* need a backup of your .mp3 collection?! Probably not. Do you *REALLY* need a backup of all those ISO CDROM images that you downloaded for fifty versions of Linux and a half dozen versions of FreeBSD? Probably not. But that's the sorts of things that are taking up 80gb plus on my hard drives -- i.e., utterly disposable cruft. Which is true for most personal computers.
Send mail here if you want to reach me.
So one way would be to both preserve a general specification of how to read the data, and then the data itself. So not only would you need a method of encoding the song onto paper, but you'd need to include the details of an algorithm - simple enough that people whose language may be very different from ours - can recreate it using their machines of the time. And then they can feed the data into it, and replay the music/video/whatever as we intended it to be seen.
-
It also won't help you if despite all your care in packing the drives, one suffers a head crash enroute, even tho you don't remember hitting any bumps. (I've had that happen.) Nor if the drive decides that being inactive for a few months is cause to lose all the data. (I've had that happen too. Several times, with HDs that had been perfectly reliable in regular service.)
While "another HD" is probably the only practical backup for today's BIG drives, I personally would only trust that as a backup if it's powered up and running in a stable location.
Which isn't very helpful in this discussion.. but I think I'd make that "swap time" more like 3 months. In my experience, *if* a HD is going to lose data just from sitting around, it does so starting at around 6 months of idleness. Those that don't have the problem seem to keep data more or less forever, but (other than Conner HDs, which could be counted on to have the problem) I haven't found it's something you can predict in advance of the event.
~REZ~ #43301. Who'd fake being me anyway?
Other than periodically transferring to the to ensure long term readability, does anyone have any serious suggestions?
printing out on paper isn't that dumb a suggestion as a belt-and-braces backup for long term storage.
Finally, what format are you going to store that data in? Is Word 2000 going to be around in 30 years? ASCII or XML? All important choices...
I suggest you should change the medium you are using to backup from time to time. You currently have 220GB of data to backup. I would create a RAID5-Array of IDE disks which can hold upto 300 .. 400GB of data. As soon as the ammount of data is increasing and coming close to the maximum ammount of data your can store/backup, add an extra disk to your RAID.
I also want to suggest to re-evaluate your
backup system every 3 to 5 years to see if is still fits in the backup strategie you have.
And if required changing to a new backup system.