Large IDE Drives as Long-Term Archival Media?

PlatterMan asks: "The question of how to cope with backing up disk drives which are rapidly increasing in size, onto tape and other backup devices which aren't scaling in size as quickly isn't new to Slashdot. Neither is the use of single, raided, and removal disks as backup devices, this has been covered numerous times on Slashdot in e.g. here and here. One thing I haven't really seen discussed however is the feasibility of disk drives as medium to long-term archival media, say 5 to 10 years. Like many people I'm in the position of now having multiple machines with a combined data pool of about 220 Gig, and backing up these onto DDS or DLT tapes is slow and manual to do, and expensive in tape costs. So I'm looking to add a removal drive bay to my primary backup machine and pick up a bunch of large IDE drives, so that I can do regular disk to disk backups over 100 Meg Ethernet (and for my machines which are in cages, over the Net) pulling out and alternating the backup drives on a 3-way backup cycle."

"Backups are of no use without offsite archival copies so I plan to take one set of disks out of the pool, and archive them offsite on a quarterly basis.

However, I've heard horror stories about the data retention and usability off older disks which have been shelved for archival, for example disk stiction - where people try to restore data off of a 4 to 5 year old drive only to find that the disk won't spin up due to solidification of lubricants, or that they've experienced data degradation.

I'd be interested in the Slashdot crowd's opinion on using large IDE drives as an archival media. Clearly one possible problem is being able to get hold of a machine in the future with a suitable IDE interface to plug them into for restoration, but I can't see IDE disappearing within 5 years (maybe 10 though). I'm more interested in experiences and opinions on the suitability of the disks themselves for long-term archival.

• Is stiction still likely occur on newer makes of IDE drives or have manufacturers beaten the problems which caused this in the past?
• Likewise how likely is bit drop-out and general data degradation over say a 5 year and 10 year period, and what do people think would be the likely maximum feasible time that a shelved drive would be usable for?
• Any suggestions as to how would I need to store drives in order to minimize these types of problem and maximise their feasible life as archival media.

Thanks!"

Comments

Print!

[grub]

Print out all your data in hexadecimal and store it in a large vault. If and when a data loss occurs you just need to re-type all the data back in.


yes I'm being facetious

Re:Print!

[Anonymous Coward]

Use paper tape and avoid retyping! If you are really hardcore you can punch metal tape like the US military and achieve nuclear survivability.

Re:Print!

[Anonymous Coward]

As a general-purpose backup medium, hardcopy isn't very practical.

BUT...

Consider that printouts on archival paper can be expected to last 100+ years. Tapes flake and fade, disks stick, cd's oxidize. Nothing else even comes close to paper! (well, maybe stone tablets...) Human readability is another plus.

For certain types of info, a printout is definitely the best choice.

Re:Print!

[alexburke]

Have you ever tried to grep three boxes of greenstripe?

Not a pretty sight, let me tell you...

Re:Print!

[Bonker]

While funny, this guy has hit the nail on the head. Without constant, vigilant backups, plastic and magnetic media don't mean dick in the long run.

If you're serious about keeping data for ever and ever, but also want convenience, you have to back up both ways.

1. Go ahead and keep data on that harddrive, but you're stucking buying another one to replace it, at least every year or so, just to make sure. This gives you the highest convenience for reinstating that data when (not if) it is corrupted.

2. Print it out. Print out all of it on non-acid paper with archival ink with the most expensive commercial printer that money can buy. Images, text, what have you. If you don't have a hard copy, you don't have the data for the long term. Once it's all printed out, put it in air and water-tight containers and then put it in a temperature controlled vault somewhere, preferrably underground so that it remains termperature controlled, even if power is lost for a long time.

Re:Print!

[IdleTime]

Why?
I doubt that he has over 100MB of data typed in that is worth saving.

Since I bet this is all pr0n, printing it is bad since Jenna Jameson looks bad in hexadecimal!

Re:Print!

[RollingThunder]

I've always referred to that as "vgrep". As in visual, of course.

Re:Print!

[danimrich]

Needless to say that google doesn't archive binaries.

Re:Print!

[Reziac]

Which is exactly why professional writers often make hardcopy of every day's work. You can type (or OCR) an original manuscript back in a helluva lot easier than you can re-create it.

(Which reminds me, I need to do that again myself.)

Um you've pretty much answered your own question.

[MisterFancypants]

Hard drives are a horrible archival medium.

Without normal/regular use, you WILL have problems trying to read from them in 4-5 years time. Hell, the way most IDE drives are these days (note the recent reduction in warrenty time periods), you'll be lucky if the drives last 2 years even WITH regular use.

t's the next AYB^H^H^H Soviet Russia

[Dental Plan]

Backing up to IDE hard drives.... That's a paddling

Not using SCSI like you should... That's a paddling

The right tool for the job is a tape drive, if you don't use it.... That's definitly a paddling.

Re:t's the next AYB^H^H^H Soviet Russia

[CaseyB]

Not using SCSI like you should

What does the choice of drive electronics have to do with the reliability of the drive? I think it's a given that either connection will be fast enough for backup purposes.

Re:t's the next AYB^H^H^H Soviet Russia

[Anonvmous Coward]

"Backing up to IDE hard drives.... That's a paddling
Not using SCSI like you should... That's a paddling "

Imagine a Beowulf cluster of IDE...ow.. that's a paddling.

Re:t's the next AYB^H^H^H Soviet Russia

[Guspaz]

320GB of SCSI storage is expensive, wheras you can get 320GB in a single IDE drive for a fraction of the cost. Most people agree that IDE is superior for non-speed-critical mass-storage.

Re:t's the next AYB^H^H^H Soviet Russia

[Joe Enduser]

1.Imagine a beowolf cluster of Soviet Russia jokes
2.????
3.Profit!!!!!!!

Re:t's the next AYB^H^H^H Soviet Russia

[falzer]

Starin' at my sandals... that's a paddlin'. Paddlin' the school canoe.. oh, you'd better believe that's a paddlin'.

Re:t's the next AYB^H^H^H Soviet Russia

[dillon_rinker]

1. Imagine beowulf cluster of Soviet Russia.
2. ????
3. All your Portman are belong to grits!!!!!

Re:t's the next AYB^H^H^H Soviet Russia

[CaseyB]

Dude, many manufacturers use the SAME DAMN HDA for both SCSI and IDE variants. Tell me again why SCSI is more _reliable_ if it's using an IDENTICAL disc underneath?

Re:t's the next AYB^H^H^H Soviet Russia

[belloc]

The right tool for the job is a tape drive, if you don't use it.... That's definitly a paddling.

Bzzzt.

Mis-quotin' our old buddy Jasper? You bet that's a paddlin'.

Here you go: "Talking out of turn, that's a paddling. Staring out the window, that's a paddling. Looking at my sandals, that's a paddling. Paddling the school canoe, you bet that's a paddling."

Belloc

Mission Critical Data..

[xchino]

Speaking from experience I can give this bit of advice for archiving critical information. Use a solid state device, don't even consider a magnetic solution, unless losing some or all of the data won't ost you your job.

Re:Mission Critical Data..

[Spire]

CD-ROM or DVD-ROM, I would guess -- with lots of redundancy (such as multiple copies of everything, plus devoting a large percentage of discs to PAR-style files). It's really, really cheap.

Re:Mission Critical Data..

[ackthpt]

Offsite backup copies are also a good part of any intelligently thought out recovery system. You're business burns to the ground, gets flooded or trashed by a twister? Get new hardware and your offsite backups and you should be back in business with a minimum of hassle.

Re:Mission Critical Data..

[Wolf Eyelash]

Has anyone had luck using a CD-RW disk for automated backups?
I've tried to keep one in my laptop (has a combo DVD-reader/CD-writer drive) for doing automated backups but so far I've had 3 CD-RW disks become corrupt after doing backups once a week for about a month or two. Is this a pervasive problem or just something particular to my drive/disks?

Re:Mission Critical Data..

[xchino]

But if you'll remember, the article was not about back up, but about archiving. There is no reason why lightning should reach an archived HDD, as it is not connected to anything.. for Mission Critical Servers however (constantly read and written to) I'd agree that a solid state would not be a wise choice..

Steve Gibson

[Jucius Maximus]

Please don't flame me for quoting Steve Gibson, but I think he's right on this account: "There are only two kinds of hard drives -- Those that have failed and those that will fail."

Hard drives are not non-volatile storage.

Re:Steve Gibson

[LoudMusic]

No flame, other than the term 'RAID 5'. Tapes aren't as dangerous as hard drives, but they can still mess up. It's not like they're garounteed beyond all odds. So a RAID 5 IDE array takes care of your data.

I'm currently using Dell NAS machines as archival backups.

Bonuses (as I see them):
Online 100mbit access to old data.
Cheap!
Fits in a physically small space.

Negatives:
Higher failure rate than tape. Pop fizzle, your data is gone.
Difficult to take off site.
Long-term replacement isn't really an option. (for RAID replacement)

The way we negate the negatives (double negative, is that a possitive?):
-Failure rate / Data loss is countered by RAID
-Taking it offsite ... it is possible to cost effectively mirror an IDE RAID system over broadband Internet and do it securely. If you are a major corporation surely your campus is large enough to simply run fiber to two corner and put mirrored backup at each location.
-Long term replacement of RAID drives ... buy a truckload of disks when you do the initial installation? (:

Re:Steve Gibson

[sporty]

True.. but having say, a raid system, and periodically swapping out one of the mirror drives for blank one is a quick-and-dirty solution. At least then, you aren't using your drives all the time. And if one fails, you still have two :)

No, not an ideal, but it works. Btw, drop a tape, you scratch a corner. Drop a drive, drop a few tears :)

Re:Steve Gibson

[ikeleib]

Actually, using RAID5 on tapes is not unusual. It has the same benefits that RAID5 disk arrays have. It allows for the loss of one tape, as well as increased throughput. This technique can actually be extended to any media.

Re:Steve Gibson

[micromoog]

The fatal single point of failure in a RAID array is the filesystem. No matter how many redundant disks, mirroring systems, etc. you have going, a corrupted allocation table will still junk the whole thing.

Re:Steve Gibson

[LoudMusic]

My basic methodology is to make a 40gb tape, stick it in a vault, and move the data to the archive server where it's marked read only. Then I dump it from the 'active' file server. Works pretty well, actually.

Re:Steve Gibson

[xA40D]

"There are only two kinds of hard drives -- Those that have failed and those that will fail."

True. But if you plug the drive in once a month and rsynch with your live machine, then store it safely in a nice moisture free environment, the chances the backup will die a death is low.

When the HDD in your live box dies, promote the backup, and buy a new one. If you do the odd incremental backup to tape you can resurect a dead machine in a few hours.

Hard drives are not non-volatile storage.

But truly, how much information is "non-volatile". With periodic pruning and archiving my crucial data has fitted onto a single CD for several years now.

Re:Steve Gibson

[jcoy42]

Negatives:
- You probably populated the array with the same drives from the same manufacturer, all from the same batch.

And they will probably all fail at about the same time.

I've had three hard discs die...

[Big Mark]

...on me within five years. 'Nuff said, methinks. Don't use IDE hard-discs as a backup medium. It's just... wrong.

-Mark

Use another backup medium.

[cybermace5]

For example, DVD.

Make sense?

Good.

Re:Use another backup medium.

[nogoodmonkey]

DVD is only 4.7gb.

Re:Use another backup medium.

[CaptainSuperBoy]

While cheaper than DLT, DVD-R maxes out at 4-5 GB. It would not be effective for backing up that much data.

Re:Use another backup medium.

[zapfie]

Use another backup medium. For example, DVD.

Um..

Data pool: 220gb
DVD: 4.7gb
DVDs needed: 47

That's a shitload of swapping to need to do for regular archival purposes.

Re:Use another backup medium.

[cybermace5]

If you actually wait around looking for mods to metamod...you seriously need some friends, or projects, or goals, or a life in general.

I'm surprised anyone took it seriously. It was more of a joke than anything. Still, what other archival method is out there? One that uses relatively cheap media, and doesn't die if someone waves a magnet nearby, or degausses their monitor while you walk past with the box of tapes? I think the only other option is CD-R. Now THAT would take a while...I have personally had to transfer data from a dying MO jukebox to stacks and stacks of CD-Rs.

Re:Use another backup medium.

[Mr.Intel]

My opinion: if you have the money, get a library (I use an Adic Scalar 100).

Back in the day (1999), I designed and built a backup system for a hospital that used the Scalar 1000. It had 8 (12 max) DLT 40/80 drives. We had an Ultra Sparc II with two 64bit PCI differential controllers (two drives per channel) and a 64 bit fiber gigabit card. The server was hooked into a gbic on the back of a 3Com switch stack with 100 servers hooked into it. We did about 1.5TB per night on that sucker. Bandwith going through the GBIC with all drives running was ~780Mb/s peak.

Yeah, if you are even a bit serious about your backups, a library with DLT or comparable tapes is the way to go. Drives are cheaper, break easier, last half as long and are much more fragile. Besides, tapes like offsite better than anything with lube in it.

Re:Use another backup medium.

[nogoodmonkey]

Yes, we have a DVD-RAM drive at work that is double sided. It is nice because it will back up the root filesystem (if its under 4.7gb), then we can boot right from it. Try doing that with tape! ;-)

GraniteDigital is what I use

[TheCodeFoundry]

I back up close to 300GB on a nightly basis using GraniteDigital's FIRE Vue(TM) FireWire 1394 IDE Ultra ATA Systems

I have 6 120GB Maxtor's and rotate them nightly, storing them in a fireproof safe, rated for paper storage. Granted, if a fire occurs, I'm not sure if the data storage would survive, but I think that would be the least of my worries, at that point. The Firewire works great and is very fast.

Re:GraniteDigital is what I use

[coyote-san]

At the least, toss the media into freezer-weight ziplock bags. Better yet is double-bagging it - put the media in a smaller bag, and then in a larger bag with smaller bag's opening on the 'far' side.

Paper-rated "fire safes" work by putting a media that undergoes a phase change at high temperatures, releasing steam in the process. (Think of the latent heat involved in freezing and melting ice, same theory is used to keep the interior of the safe at a reasonable temperature.)

The only problem is that paper tolerates steam fairly well. Ditto the smoke that can make its way into the safe. The paper may be damaged, but it is still readable. Computer media will be destroyed. Fortunately freezer-weight plastic is more than adequate to block the steam, leaving only small openings in the seal. Even this is modest, and the second bag is mostly to allow you to avoid smearing soot onto the media as you remove it from the bag.

Re:GraniteDigital is what I use

[Jeremy Erwin]

paper burns at 451 degrees F (232 Celsius)
media starts to melt at 125 degrees F (52 Celsius)

A fireproof safe thats rated for paper storage only isn't going to cut it.

Good idea...except...

[Agent Green]

...you're walking down the hall with a 3 foot stack of drives and you trip over an ethernet cable...and all the drives take a sailing course through the air and land on the concrete floor.

I'm not a betting man, but I bet if that were a stack of DLT tape, you might still be able to read them after that hypothetical incident.

Re:Good idea...except...

[zrodney]

I think you would be smart to invest in a push cart
to move a three foot stack of 100 gig + IDE backup
drives.

I bet you couldn't even lift 3 feet of disk drives
they are really really heavy when they are large,
and have sharp edges.

Re:Good idea...except...

[apoc.famine]

Eh, I bounce DLT tapes off the floor on a regular basis, and just about all worked fine afterwards. The trick is to say, "It's still good, it's still good." before you pick it up.

Disk "stiction"

[Anonymous Coward]

I haven't really had any problems with stuck spindles since the early 90's with the old Quantum drives they used to stick in Macs. I have a number of Seagate Barracudas that had been sitting idle since approximately 1996/7 that I just fired up last week. All of them (about 40) worked and still had their data, which actually happened to be usenet archives that I'd been saving.

I'm certain manufacturers have gotten even better with lubrication issues over the last 7 years and I don't think I'd waste too many cycles worrying about it. With the price of large capacity DLT/AIT tape these days, it sounds like backing up to cheap IDE disk is a viable option.

Cheers,

Just Another Anonymous Coward

Re:Disk "stiction"

[rworne]

And for the SCSI-lover crowd:

I have a 2GB HP Drive that has been powered down since 1998. I fired that puppy up last August, and it refused to spin at all.

I disconnected the drive and "rotated" it by spinning it (whole drive) quickly, hoping the inertia of the platters would break the "stiction".

It worked, and now the drive powers up with no probems.

Now I also have a 1992-era Seagate 5 1/4" fullheight SCSI drive that's been powered down for even longer. Last I remembered, it sounded like it was gonna die hearing it sloooowly spin up. (I think I can, I think I can...). As far as I care, it can stay spun down.

Re:Disk "stiction"

[Paul Johnson]

I remember Sun workstation "pizza boxes" with disks that suffered from stiction. The IT support people would pick the whole pizza box up a couple of inches and drop it. This unstuck the disks.

Paul.

warranty period

[Clover_Kicker]

Since IDE HD manufacturers recently decreased their warranty period, I'd be *really* reluctant to trust 'em 10 years from now.

Re:warranty period

[Drakantus]

You are missing the point. What is your backup method for backing up 220GB?

Oh, you don't backup 220GB of personal data on a regular basis?

So, you trust having no backup at all over having a backup on an unreliable medium?

Re:warranty period

[TheAwfulTruth]

That warranty is assuming continual use for 3 years. It's entirely possible that buying a new drive, and using it once a month for a monster backup, that it could easily last 10 years. The Q really is, will you still have a machine with an IDE bus that can read it in 10 years. I'd jump to serial ATA if you really want to reach that length of storage time as parallel ATA has only a couple years left on it.

Re:warranty period

[artemis67]

In 10 years, the drive will be worthless as a backup media, anyway... Heck, in 3 years it'll be worthless, because when he upgrades his RAID array in 3 years, it will be much, much bigger. In 3 years he may be trying to back up a terrabyte, and his 200 GB IDE drive will be far too small for the job.

Not that that'll be a problem, though, because he'll just pick up some 1 TB FireWire2 drives then for about what he's paying now for 200 GB.

Re:warranty period

[Clover_Kicker]

You are missing the point. What is your backup method for backing up 220GB?

Tapes are designed for backups. If you seriously need to backup 200GB, then you are looking at DLT or better, and it ain't cheap.

Oh, you don't backup 220GB of personal data on a regular basis?

Who the fuck has 220GB of personal data? Seriously, for the cost of backing up that much porn, you can just go down to the store and buy the legit DVDs. While you're at it, you can stop off at the record store and buy some albums so you can re-rip your MP3s.

Just because you have 220GBs of hard drives in your machines doesn't mean you need to back up every byte.

C:\WINDOWS>ver

Windows 98 [Version 4.10.2222]

C:\WINDOWS>du |sort |tail -1
353472k ./

C:\games\Diablo II>du |sort |tail -1
1378784k ./

C:\games\Diablo II>du save

1696k save/old/
3328k save/

Pop quiz - if I wanted to back up this machine, do I

• backup 1.5GB of Windows and Diablo binaries
• backup 3 megs of Diablo II save files (would fit on 2 fucking floppies, FFS.) because I have my Win98SE and Diablo II+LOD CDs on the shelf.

My documents (resume, web pages, GNU Cash files, email etc.) live on a server, where they are in fact backed up nightly to a second hard drive.

Every couple of months I burn a CD of the latest backup tarfiles. Cheap CDRs are a half-assed long-term archival solution, but the price is right.

Some things (Mozilla installer, service packs) are so ephemeral that they aren't worth backing up, i.e. when you need them there will probably be a new version available anyway.

What about my MP3s and pr0n? When I've got enough new stuff I burn a CD full. Every year or so it's worth re-burning the MP3s so that I've got the same genre on a given CD. When you've got Sarah McLaughlin, Mozart, Dead Kennedies, Suicidal Tendencies, Reverand Horton Heat and Johnny Cash on the same CD, there isn't a person in the world who won't make fun of you.

So, you trust having no backup at all over having a backup on an unreliable medium?

I did not recommend that no backup be performed. I said that I do not trust IDE drives for long-term archival use.

If you are determined to archive to IDE, fill your boots - it ain't my data.

Re:warranty period

[Clover_Kicker]

>My personal experience leads me to believe that
>hard drive volatility is exaggerated lately.
>Old Western Digital drives that I had in my 386
>over 10 years ago had bad sectors every so often,
>but since then, my high speed hard drives run fine.
> I ain't advocating using a hard drive as a backup
>medium just yet, but for personal use, I'm running
>3 hard drives in a parallel RAID setup without
>fear.

As someone else pointed out, RAID doesn't help if you overwrite or delete something by accident.

Re:warranty period

[fishbowl]

"Who the fuck has 220GB of personal data? "

I'm getting there, in audio data.

My own music, that I write and record, so, going down to the store to replace it isn't exactly an option.
It's also on DAT, and on CD audio, so you could say
I have a backup, but that's not really true -- the DAT is the source material, and a CD would represents one view of some of the data.

Am I going to buy a $65,000 SAN tape library machine, just because I'm getting into volume? (No.) Would I like an inexpensive solution that is less cumbersome than CDR? (Yes.)

Re:warranty period

[gvonk]

Who the fuck has 220GB of personal data? Seriously, for the cost of backing up that much porn, you can just go down to the store and buy the legit DVDs. While you're at it, you can stop off at the record store and buy some albums so you can re-rip your MP3s.

Are you kidding? 220GB (100 full-length porno DIVXs and 600 MP3 albums) would cost $300 to back up, $600-800 if you needed redundancy. Even if you got the porn for $15 DVD (cheap) and the albums for $10 (really cheap), you would be paying $7500 to buy the media.

Re:warranty period

[Kaa]

Who the fuck has 220GB of personal data?

And what's so weird about it?

A scan of a single frame of a 35mm film, on a high-end consumer film scanner will create a file... let's see:

The scanner is 4000dpi, so the resulting image is about 4000x6000 pixels. We are working in 16-bit-per-color-channel mode, so that's 6 bytes per single pixel. A bit of multiplication get you 144Mb. As a practical matter, the film frame is slightly smaller so your output TIFF file is about 120Mb in size. That is for a single 35mm film frame.

So raw scans of slightly under 2000 film frames will already hit the 220Gb figure.

Still think it's a ridiculous number?

Re:warranty period

[Clover_Kicker]

A scan of a single frame of a 35mm film, on a high-end consumer film scanner will create a file... let's see:
[SNIP]
so your output TIFF file is about 120Mb in size. That is for a single 35mm film frame.

You've described yourself as a high end user.

That means you pay high-end prices for your backup needs, or try to kludge something for cheap.

• Tapes are good backup solutions. Tapes and tape drives are are very expensive.
• IDE hard drives are kludgy backup solutions, and have serious disadvantages compared to tapes.
  • HDs tend to be non-removeable, which makes it harder to have a decent rotation, complicates off-site backup, etc.
  • individual HDs are more fragile then individual tapes
  • RAID is not a substitute for backups.
  • HDs are not designed to sit on the shelf for a few years and then be read. It very well might work, but it very well might not. We can piss about it all night, but we'll never prove anything either way. I won't risk archiving to off-line IDE HDs, you can do whatever the hell you like.

The above facts may be annoying, but wishful thinking for cheap solutions doesn't make IDE HDs a quality backup solution.

If you want to backup to IDE, knock yourself out. (I do, as stated above.) But you should go in with your eyes open, and consider all the possible pitfalls - the better to plan around them.

Some people seem to think my attitude is too negative. When you're planning a backup scheme for your precious data, you should be paranoid. Really, really, paranoid. All hardware sucks, get used to it. Tech work is nothing more then trading off the strengths and weaknesses of individual components - don't blind yourself to the weak points of your favorite technologies, or Mister Murphy might pay you a nasty visit some day.

Good luck, BTW, to everyone in this discussion. May you never lose a byte.

Re:warranty period

[Reziac]

As to whether you back up OS and programs as well as data -- how much time will it take you to reinstall, reconfigure, relocate all the drivers and patches you need, get everything to play nice together again, and behave exactly how you like it.. and that's assuming you can remember how you got it to that state in the first place. I once worked out, in real work hours, how long it would take to redo my old Win3.1 install: it totaled up at TWO WEEKS if I treated it as a fulltime job. Gods know how long it would take to reconfigure my much more complex Win32 setups. It's not the major installs that are the problem, it's all the little tweaks you've long since forgotten you did, or even how to do.

So yes, I back up everything, because my time and aggravation are worth something too. It's not such a big deal on a system where the programs are set in stone and only need one full backup and the occasional incremental update, but it can be on a system whose contents are often in flux.

Re:warranty period

[Reziac]

Hey there, long time no read!

Alas, that's pretty much true of anything that one wishes to keep a clean original for. Frex, I could ditch all my original photo scans once I've twiddled 'em down to a JPG for the web, but I regard those original scans as archival copies, to be kept pristine for future reference, and as a hedge against losing the original hardcopy.

So I have lots of "needlessly large" files, and need lots of HD space. *sigh*

Or it could just be that junk fills the space allotted. :)

Re:warranty period

[SmittyTheBold]

When you've got Sarah McLaughlin, Mozart, Dead Kennedies, Suicidal Tendencies, Reverand Horton Heat and Johnny Cash on the same CD, there isn't a person in the world who won't make fun of you.

Yeah, I see your point.

Who the fuck puts Johnny Cash last on a CD like that? It's just basic mix(tape)ology.

=)

Re:warranty period

[GooseKirk]

Who the fuck has 220GB of personal data? Seriously, for the cost of backing up that much porn, you can just go down to the store and buy the legit DVDs.

Actually, if you watch DVDs on your PC like I do, it's cheaper to just copy them to the hard drive than buy them. A 120GB drive is $165 at NewEgg. An average DVD is probably 6GB. That's 20 DVDs per $165 hard drive. Figure $20 per DVD, and suddenly the hard drive is a bargain.

You'd spend $400 for 20 DVDs.

You'd spend $200 for a hard drive plus rentals, plus you get quick, easy access to your movies, plus you get flexibility... need space for other stuff? Well, how often did you watch "Point Break" anyway? The downside is, you can't really bring your movies over to a friend's house to share unless you get an external enclosure and he's got his PC hooked up to an entertainment center too, but on the other hand, you might be better off just leaving "Point Break" as your own little secret, anyway.

If you want to back 'em up, though, then you're into the same ballpark, cost-wise. But if it's just DVDs, why bother? You can always just rent or borrow 'em again. Hopefully in the next few years there will be affordable and huge backup options.

Can't wait for 320GB drives...

Bad Idea..

[Suppafly]

I think it would be a bad idea to rely on IDE drives as one's only source of backup. Especially if you aren't planning on using any stripping or parity. The large IDE drives are, the more prone to failure they appear to be. Ask anyone thats bought a 60-100 IBM deathstar drive lately. The added wear that would occur from joustling them around as you pull them in and out of the drive bays all the time seems like it would also make the time between failures greater. What is proposed in the story might work fairly well for a home user, but I think it would fall apart in a business setting.

Re:Bad Idea..

[NeMon'ess]

By your logic every HD in a LAN gamer's computer should corrupt in a year. What about laptop drives? How much sturdier are they built? I say use em as backup material, just know that in 10 years after 1000 hours of use they have to be replaced. That won't matter since in 10 years drives should be up to a terabyte.

Re:Bad Idea..

[addaon]

I think it would be a bad idea to rely on IDE drives as one's only source of backup. Especially if you aren't planning on using any stripping or parity.

How would stripping help reliability?

Okay, okay. How would striping help reliability, then?

--

Re:Bad Idea..

[ottffssent]

The 75GXPs and to some extent the 60GXPs suffered higher-than-normal failure rates, but these were unrelated to higher areal density. The drives used glass platters and load/unload technology which was apparently not up to the task yet.

Read your MTBF ratings - hard drives are getting better and better. The Seagate drive (admittedly, SCSI) I bought almost 2 years ago claimed 1.2 million hours (over 100 years 24/7). I've seen about 500,000 hours quoted for mainstream IDE drives - about 50 years. Figure that those numbers are under optimal conditions, so divide by 10. Now you've got an "adjusted" MTBF of about 5 years.

Put the drives in a RAID5 array (or an array that allows >1 simultaneous failure if you're really paranoid) in a properly-cooled cabinet in a properly-cooled room. Spinup the drives and read them every month (cat the contents to /dev/null) to let the drives' autocorrection take effect and to check for failed drives. Let the RAID array rebuild if necessary.

When the price of the drives you are using reaches about $50, the drives are at the end of their product lifetime - buy a year's worth of spares and when they're gone copy the data to a new array as above using newer technology (such as serial ATA) and larger drives. Repeat ad nauseum.

Alternately, keep the arrays spinning and use AFS or similar software to replicate volumes across multiple servers. This will protect you from single-drive failures in each array, and multi-drive failures in one array. Keep your individual AFS servers on separate power.

A note about power: BAD POWER KILLS COMPUTERS. Don't skimp on power. Make sure your circuitry can handle the load. Install good power conditioning equipment. Install good battery backups and/or generators. Don't put all your servers on one circuit.

In short, data archiving isn't hard. It's just expensive and time-consuming. Ask yourself what your data is worth and what failures you need it to be able to withstand (one dead drive? 5 dead drives? 1 dead server? Lightning strike? Earthquake?) and build appropriately.

Alternatives...

[anarchima]

People here are saying, "Don't even think about using IDE!". Well he has no choice, does he? Tape has several drawbacks as the author mentions his comment to Slashdot. He has asked for advice on IDE. If this is not a feasible option, recomend some others (besides tape). Or ARE THERE NONE?

Re:Alternatives...

[sl3xd]

Your experience is fairly atypical. I typically budget for a replacement IDE drive every three years on the machines I admin, because that's about how often they go south in my experience. For anything I consider critical, it's going to be stored on the SCSI drive(s). Peroid. Then the already ultra-reliable SCSI drives will be backed up to tape and/or CD-R/DVD-R. But for cheap non-critical storage that is easily replaced (like games, Office, or Windows in general), IDE's awfully tempting.

Re:Here's a tip..

[Jucius Maximus]

This is proof that we need a +1, Troll moderation.

Not a horrible Idea, just not a great one.

[briancnorton]

Using magnetic media to back up magnetic media isnt the greatest idea in the world, but it can work. Hard drives fail, and when they do, you want to have the data available so that you can get to it. The IDEAL way to do this is to contract an outside company or manage for yourself a backup server which does incremental backups as often as you need and periodically burns them to a more permanant media like DVD. If you cant afford this or dont like the idea, then you can burn DVDs on your own. A good program will track files for incremental backup and 220 gigs can fit on something like 50 DVDs, with maybe 1 more per session (assuming that not all files are constantly changed) Obviously a lot depends on what you have, how much money you are spending, and what you need.

Re:Not a horrible Idea, just not a great one.

[Yohahn]

News for you

4.7 GB = about 4.3 GiB

GB being 1000 * 1000 * 1000 bytes

GiB being 1024 * 1024 * 1024 bytes

Gotta love those marketers, making us redefine our units so they can spread confusion and fool the customer into thinking they get more.

Re:Not a horrible Idea, just not a great one.

[briancnorton]

Your math is just fine, except that it would only need to be done once for the full set, and in an industrial setting this isnt a big problem. 10 burners working together could pound this out in 3 hours. Reliable backups are expensive. No way around it.

rock and chisel

[Lxy]

with all the stories I've seen about being unable to retrieve data from just 15 yrs ago (because the format is unreadable, not because the media deteriorated) I'm convinced that archiving data using a chisel and a rock is the best way to go.

Re:rock and chisel

[nsample]

I know this parent was modded up as +Funny, but it's actually +Informative. "Rock and chisel" are the best thing we have, and there's a real trend toward using it more. Take a look at Norsam's HD-Rosetta. It's an etched nickel plate designed to last for thousands of years. Vive la Rock & Chisel!

Re:rock and chisel

[Anonvmous Coward]

"with all the stories I've seen about being unable to retrieve data from just 15 yrs ago (because the format is unreadable, not because the media deteriorated) I'm convinced that archiving data using a chisel and a rock is the best way to go."

He's right, you know. Look at the info we're gathering from fossilized remains of dinosaurs! Once they found the petrified remains of a velociraptor next to a picket sign protesting the use of fossil fuels.

Re:rock and chisel

[IdahoEv]

with all the stories I've seen about being unable to retrieve data from just 15 yrs ago (because the format is unreadable, not because the media deteriorated) I'm convinced that archiving data using a chisel and a rock is the best way to go.

Oh yeah, and when Microsoft standardizes us all on MS Alphabet 2.0, then where will you be?

You can't just run a rock through an open-source translation tool, you know.

Re:rock and chisel

[Lxy]

Interesting stuff.

I'm trying to find a /. article from way back about some manuscript that was digitally stored in 1972 and was unreadable now, and had to go back to egypt or wherever and re-interpret it from the rock it was inscribed on, several thousand years ago. While it isn't practical for terabytes worth of data storage, it does pose the question of why we insist on using digital storage.

Re:rock and chisel

[pomakis]

I believe you're referring to the Domesday book, covered by Slashdot here and here.

How often do you plan on replacing the drives?

[kalidasa]

If you're rotating the drives (using one for your main drive and two for backup for a month, then rotate every month) and you keep some money in the bank to replace the one that's going to fail about every 6 months, you should be ok. Well, except if something affects the whole box (fire, etc.).

Tape really isn't that expensive.

[Wakko Warner]

What you're proposing will cost no less than a high-quality AIT drive, which, though you may need to span tapes in the most extreme of situations, will give you quite a bit of capacity. You can pick up 90GB native-capacity AIT drives now for around $500 or so on eBay. The media is affordable, too.

- A.P.

Re:Tape really isn't that expensive.

[shoemakc]

You can pick up 90GB native-capacity AIT drives now for around $500 or so on eBay.

Yes, if you're looking for an HVD SCSI device that only does 35GB uncompressed. 90GB is an entirely arbitrary number.

Real 100GB backup solutions start at 1000 or so, even used.

-Chris

Why Tape Is Good

[Jucius Maximus]

Tape may be inconvenient but it is still a true backup medium. With hard drives, the reading and writing hardware are enclosed with the platters. So when the read head of the HDD fails, your data may be 100% intact on the platters but you can't get at it without professional help. How many other parts in the HDD could fail without harming the platters? A lot!

With tape, the failure of a tape drive doesn't separate your from your data (unless it catches on fire with the tape in it or something.) You can just get a new tape drive and you are good to go again.

Thus, tapes are very good because the storage medium and the read/write hardware are separated and not interdependent.

Re:Why Tape Is Good

[BlankTim]

Obviously, you've never had a tape physically fail.

Maybe it's just me, but after the experiences I've had the last year with crappy tapes, I'm surprised the "tape as a backup medium" idea hasn't been seen for the farce that it is.

Backing up to IDE or SCSI? Good short term solution, but I don't think I'd trust my backup drives for more than 1 year, tops.

Burn to CD? Good long term solution, just not practical due to the file sizes involved. Burn to DVD isn't much better.

It's time for something new. Hell, maybe it will turn into the next "killer thing" and revitalize the economy.

I vote for soft bubble memory

Re:Why Tape Is Good

[sterno]

Two points:

1) Tapes can physically fail too
2) If your hard drive fails in a way that does not destroy the data itself, a data recovery company can recover it.

If you want to have a reliable archival backup, you should probably insure that your backup is redundant.

Re:Why Tape Is Good

[Rolo Tomasi]

Thus, tapes are very good because the storage medium and the read/write hardware are separated and not interdependent.

Bullshit. Tapes are intended as a short-term backup medium. Google for NASA magnetic tapes, and you will find a lot of interesting stories. Like e.g. this one:

Right now, ACRES is updating storage of 120,000 gigabytes of data collected since 1979, primarily from remote-sensing Landsat satellites passing over Australia. Landsat images are among the most voluminous of space-based data, making ACRES one of the largest data repositories of its kind in the world, Trezise said.

The data now are housed on optical tape, having been rescued from disintegration in the early 1990s from aging high-density magnetic tapes. That first rescue operation occurred just in the nick of time, Trezise said, since the magnetic tapes were starting to get sticky on their spools.

Re:Why Tape Is Good

[skroz]

One very important thing to consider : With certain types of tape drives, a misaligned head can render your tape media useless in another drive of exactly the same type. DLT is a good example of this. You can write and read to your heart's content on the same drive, but try to read a tape written in one drive on another and you can be sunk (professional data recovery experts with the proper tools can work around this, but it's expensive, and the whole point of this discussion was the need for "professional help" if certain parts of the hardware fail.)

Re:Why Tape Is Good

[zrodney]

"With tape, the failure of a tape drive doesn't separate your from your data (unless it catches on fire with the tape in it or something.)"

Which actually does happen from time to time. I have
had many VXA tapes trashed by the autoloader/drives
in my 15 tape backup system. That's painful because
the tapes hold more than 30 gig each.

Re:Why Tape Is Good

[njdj]

Does anybody have statistics on what fraction of mag tapes are still readable after 10 years?

In my experience, mag tapes are pretty much worthless as a backup medium even for 5-year storage, but that's based on relatively few tapes and maybe I've just been unlucky. Has anyone stored hundreds of tapes for 10 years and then tried to read some of them?

Re:Why Tape Is Good

[sunspot42]

Yes, tape is a "true backup medium", but it is *not* a suitable medium for longterm archival storage - at least, none of the affordable formats up till this point (like the DAT-derived DDS format) have been. There's a big difference between a backup medium (a copy that's probably replaced every day / week / month and is intended for use in the immediate future) and archival storage (a copy that's intended for use 5+ years in the future).

While the failure of a tape drive won't separate you from your data (unless the drive damages the tape at the same time it fails . . .), tapes themselves deteriorate over time. Here's an article about the problems the National Archives here in the United States have encountered with preserving copies of the Nixon tapes on DDS's audio cousin DAT. An excerpt:

"During the National Archives' routine monitoring of the tapes'
condition, the analog reel-to-reel copies have shown no signs of
deterioration whereas there is an estimated 5-10% catastrophic failure
rate among the DATs in the collection. There appears to be no pattern
to the failures. It has occurred on new tapes that were recorded six
months ago, and it has occurred on tapes that were recorded six years
ago. It has occurred on all brands of DAT purchased throughout the
previous seven years. Accordingly, the archivists routinely reduplicate
these DATS on multiple copies. As insurance, archivists also transfer
DAT copies back onto analog reel-to-reel preservation copies. Unlike
the other preservation analog copies, these copies have not been
filtered and closely "mirror" the original tapes. Therefore, in the
future when technology has progressed, the archivists can retrieve
conversations that are extremely close to the original audio recordings
and enhance these with the latest technology."

Leading audio preservationists have issued their own warning. This company deals with audio preservation, and has some interesting things to say about tape formats - analog and digital.

Of course, DDS tapes have supposedly been manufactured to a higher standard than their Audio DAT cousins, sport finer particles and stronger binders, and the format includes additional error correction and redundancy. Still, these issues with a modern tape format like Audio DAT are not an encouraging sign for those seeking to perform archival storage using DDS and it contemporaries. HP for example only claims a 10 year archival life for DDS. Contrast that with the 75-100 year lifespans Kodak and TDK are claiming for CD-R.

These longevity issues won't just be confined to older tape backup formats though, if history is any indication. It's the nature of the medium. I think Sony is currently claiming a 30-year lifespan for AIT, and HP something similar for their new format, but of course we only have a couple of years' experience with them so far, and given the incredible data density of those formats, if something should go wrong with either of them the results could be catastrophic. Unexpected deterioration has certainly happened with tape before - witness this article composer and synthesizer pioneer Wendy Carlos put on her website, as well as her own experience with her older tape masters.

Hard drives certainly aren't a great archival medium either, but I wouldn't be so quick to assert that tape is superior. At least drives have the advantage of being sealed from the outside atmosphere, and contain within them all the logic and hardware required to extract that information in the future. The only big issues I can see are, will there still be equipment to interface with them in 10 to 20 years (probably, since IDE is so widespread) and will the drives still spin up in 10 to 20 years (who knows). It's that second issue that's the real buzzkill for HD's as a longterm storage medium. Manufacturers won't even issue a decent warranty on drives anymore. What does that say about their planned longevity?

Me, I think your best bet is DVD. But if you really want to be able to read that data in the future, I'd suggest copying it to at least two different formats, perhaps AIT *and* DVD. Don't forget to check on it every few years, too. If there's any sign of deterioration, you'll hopefully be able to make another clone before the failure becomes catastrophic (perhaps to a superior format that hasn't even been invented yet). If you want something you can just throw in a hole and forget about, sorry - that media doesn't exist.

Re:Why Tape Is Good

[Lumpy]

Exactly. and those searching for a cheap solution must mean their data has little value?

Me? I have a 20 tape DLT jukebox here. I easily got management to buy it when I mentioned that to them. "we can go cheap, but then why backup at all if your data is that worthless."

if you have valuable data, are you going to store it in a paper box buried in the back yard or a nice vault?

Re:Why Tape Is Good

[SJS]

Obviously, you've never had a tape physically fail.

Nine-track tapes and seven-track tapes
And tapes without any tracks;
Stretchy tapes and snarley tapes
And tapes mixed up on the racks --

Take hold of the tape
And pull off the strip,
And then you'll be sure
Your tape drive will skip.

-- Uncle Colonel's Cursory Rhymes

organize your data

[jayhova]

Used to be in the data backup biz, you should really start with evaluating what you are actually backing up. Most people backup applications and temp files that really are not going to help much. Also, do you really need to archive all of that stuff even if you are anal? Another thing to consider is, will the media be supported and will you have the proper drivers for the disk drives handy. 220 Gigs is surely still in the land of tapes, I hate them more than most, but would not suggest the use of an IDE Hard Drive. my 2 cents

Long Term Storage

[caseydk]

The Library of Congress is attempting to answer this question as they have huge amounts of media that is on highly degrading (nitrate-based films) materials.

Their answer? A huge RAID array starting at 180TB and growing steadily over time.

Your answer? Probably figure out which of the data is fixed and which of it changes and attempt to back up accordingly. Does all 220gb change on a weekly basis? That seems unlikely...

Re:Long Term Storage

[Tackhead]

> The Library of Congress is attempting to answer this question as they have huge amounts of media that is on highly degrading (nitrate-based films) materials.
> Their answer? A huge RAID array starting at 180TB and growing steadily over time.

Last time I looked, one Library of Congress was only 10TB, and I bought a 100G drive for $100.

So my rig sported a cool 0.02 LoC in my rig. I felt gr8. I mean, I 0wn3d.

Now you're telling me I only have 0.00055555 Libraries of Congress? I f33l s0 l4m3.

Bastards.

Re:Long Term Storage

[caseydk]

they're concerned about some of thomas edison's first movies and the like... they were all made on nitrate-based material that decomposes (into nitroglycerine. boom!)..

they want to provide public access for all of this stuff in the long term since it's all public domain...

10TB is nonsense.

Re:Long Term Storage

[Tackhead]

> What are some examples of uses of this long-term inactive storage? I can understand like maybe a one-year archive, but you can keep that on an active raid array if need be.

For home use, pretty obvious - MP3z, DiVX movies, and TiVO shows. Re-downloading the MP3z and pr0n^H^H^H^HDiVX movies would suck because you'd have to find and re-download the missing files. Likewise, re-recording the TV shows on your PVR would be a pain because you'd have to wait for your cable company to air them again.

You'd keep the RAID array up at home, and that'd be your media box.

But you'd still need a couple of 120G drives down at the local bank in the event of a fire at home. Or better yet, at your Grandma's house 100 miles away, if you live in an earthquake-prone area.

And while I've mentioned the time cost of re-downloading, this risk really isn't about piracy - even if you own everything on your media server, you can't re-encode your CD or DVD collection if said collection has been transmogrified into a melted lump of goo or shards of polycarb.

Every time you visit Grandma, or once every six months, whichever is less frequent, you swap drives.

(Just remember to pack the drives securely when you drive to Grandma's. And drive carefully. RAID won't protect you if all the drives in the array go sailing through the window at 60 mph ;-)

Re:Long Term Storage

[Directrix1]

Also, the obvious answer to the authors question is buy 900 trillion flip-flops and hook them up to the wall outlet with a old rusty pair of bronze wires, feed the flip-flops through 900 trillion lines coming from a decoded IDE output in realtime. Weeeeh! Or you could take a top suspended in a vacuum with magnets on the edges of it. And for every bit you want to encode on it, you have an electrical field deliver an impulse of (2^bit number)*(bit-value)*(whatever unit you want) to the top in the vacuum. Later when you want to decode it, just start at the largest bit number and provide an opposing impulse and if the direction of the top changes then return a 0 otherwise return a 1. Simple, infinite data storage, fairly easy implementation, all you need is something to monitor the speed and make corrections when necessary (or observe the amount of friction and have it do periodic adjustments). You see how easy this is?

Re:Long Term Storage

[dattaway]

The Gentoo Linux distribution currently weighs in at 12GB of 6300 tarballs. Expand that out into built trees of source code, you can easily fill up a 160GB disk. Source code is great reading material for a slow day...

Re:Long Term Storage

[soulsteal]

So my rig sported a cool 0.02 LoC in my rig.

This post brought to you by the Redundant Department of Redundancy Department.

Re:Long Term Storage

[Reziac]

It also won't help you if despite all your care in packing the drives, one suffers a head crash enroute, even tho you don't remember hitting any bumps. (I've had that happen.) Nor if the drive decides that being inactive for a few months is cause to lose all the data. (I've had that happen too. Several times, with HDs that had been perfectly reliable in regular service.)

While "another HD" is probably the only practical backup for today's BIG drives, I personally would only trust that as a backup if it's powered up and running in a stable location.

Which isn't very helpful in this discussion.. but I think I'd make that "swap time" more like 3 months. In my experience, *if* a HD is going to lose data just from sitting around, it does so starting at around 6 months of idleness. Those that don't have the problem seem to keep data more or less forever, but (other than Conner HDs, which could be counted on to have the problem) I haven't found it's something you can predict in advance of the event.

Re:Long Term Storage

[Reziac]

Actually, not much of a bump at all. As someone pointed out in a previous similar discussion, a HD can achieve 75G shock just from a drop of a few *inches*.

I just replaced a client's HD that had apparently had the same thing happen when they moved, and I'd put the fear of ghod into them about handling their machines like eggs.. so I'm pretty sure nothing got dropped or bumped in any obvious fashion.

A few years ago I saw some Gateway stats on shipped whole machines, to wit: 30% were DOA due to the HD suffering damage enroute. Eeeep!!

Yeah, we see HDs tossed around like it's nothing all the time (watch how vendors handle 'em at computer swap meets!) and most HDs don't seem** to suffer anything for it, but it does happen. And ol' Murphy is just waiting for us to be toting around the HD that's got the only good copy of our critical data...

** Altho given that you don't SEE damage on modern HDs til all the bad cluster swap space gets used up, I do wonder how much this shortens their lifespan.

A lot of folks will say....

[ajs]

that disks will rot, so you can't trust them.

I counter with this: tapes rot too. In fact, any tape older than one year that I've had to go back to has been worthless (read: it had deteriorated data).

Tape is a really bad medium to trust, but we keep buying it because we can't think of a better solution. Personally, I think the way to go is just to give up and admit that disk is not cheap. You need to back up your data to a live mirror system with identical storage (hourly rsync does a nice job) and then you need to arrage a service that can back up your data to remote live mirror systems. Note that in both cases I said "live mirror". You don't want a backup sitting on a cold box because you never know the quality of it until you need it.

The remote backup part is expensive, but it's the only reliable way. You seed it by tape (full backup to tape, and mail them to the vendor) and then use dedicated lines to keep a regular incremental update going.

If one of those two backup systems fail you know about it right away and you fix it. No more tapes rotting on a shelf only to be discovered when your data goes south.

Re:A lot of folks will say....

[Chrisje]

Yes, tape will rot. As will anything that is magnetic.

DDS tape has a guaranteed data retention period of 2 years, but then you may face head alignment problems if you replace the drive. DLT and LTO have data retention periods of 5 years approx. Head alignment problems don't form a problem because of the nature of the mechanism.

This is however not the point. The point it that a harddrive is not an ARCHIVAL medium. Neither is tape. Harddrives are the work horses for on-line data and tape is meant as a BACKUP. Backup meaning a copy for safe-keeping under a very limited time (ie next week, when tuesdays tape is run again, or... well, you get the point... ).

CD's (CD-R(W)) offer a theoretical data retention span of 20-100 years depending on who you ask. So that is safer, but still not perfect.

A Service Level Agreement with a maintenance company would do the trick too, but is expensive.

But why archive? Doesn't an automated backup to a tape robot with a weekly rolling schedule combined with a RAID 1/5 solution for your single disk failures satisfy your needs? What is so damn important that you need Off-Site ARCHIVAL rather than off-site backups?

With the falling prices of both tape and disk cost per megabyte, it's affordable to keep all relevant data on the drives of the server and then do backup to tape if needed.

Just my 2$c.

Re:A lot of folks will say....

[Jerf]

On a smaller scale (personal), this is essentially what I do.

First, only some personal data is critical, not the GBs of operating systems and programs I can redownload/recompile if necessary. Things like documents, saved games (you'd think it's unimportent until you play the first 2/3s of Fallout 2 five times and can't stomach getting far enough to see how it all turns out, because you'd have to play that 2/3s again...), email maybe, whatever, but some limited amount. 10MB can go a long way... that's a lot of programming, for instance. (Been working on a project for about half a year now and I'm just ready to break 300KB of code...)

Then, set up a live backup amounst all the disks you have on various machines. I use unison so that I can change files in the repository on any machine and have the changes propogate correctly, instead of the unidirectional updates rsync does.

Use symlinks to put everything you need into one directory, and tell Unison to follow the symlinks, not archive them directly. Then just run that every so often on the machines, and you're set.

Once more of my family gets set up with always-on connections, I intend to set up a family-level repository of backed up files with Unison, so that "off-site backups" are a weekly script run without intervention by the family, making off-site backups across the state (or country, or world) easy. This will protect the scanned pictures and other things in the family heritage easily and effectively.

Which reminds me, the first always-on connection just came online and I really ought to talk to that member about a reciprocating backup setup...

Re:A lot of folks will say....

[Gary]

I dunno. You lose one of the most important, to me, features of regular backups. Versioning. If you change a config file, it gets backed up, and then you decide you made an error how do you recover in the mirroring scheme you describe? You could argue that I should be using something else for versioning, like a CVS repository, but that's too painful in a large multi-user environment.

In my experience backups are used more often for this situation than they are for disaster recovery.

Re:A lot of folks will say....

[Havokmon]

Personally, I think the way to go is just to give up and admit that disk is not cheap. You need to back up your data to a live mirror system with identical storage (hourly rsync does a nice job) and then you need to arrage a service that can back up your data to remote live mirror systems.

Note that in both cases I The remote backup part is expensive, but it's the only reliable way. You seed it by tape (full backup to tape, and mail them to the vendor) and then use dedicated lines to keep a regular incremental update going.said "live mirror".

I agree wholeheartedly. Though, I would note, that IDE is the perfect solution for your redundancy. All you need is space. It doesn't have to be the fastest, or the highest quality mirror. Buying 20 IDE drives and having half of them fail is still cheaper than high capacity SCSI. Do a RAID 50 (IIRC, two RAID 5's - mirrored) offsite, and use rsync to mirror your data over your Inet line. Or string your mirror. Have your 'backup' offsite RAID rsync off the primary offsite RAID. I'd bet the only people who would have problems with that are the ones doing heavy graphics.

Check out Rackspace for your offsite needs, I didn't think they were that expensive, at least compared to an actual archival facility. Pick your favorite encryption method to secure it. Hell of a lot cheaper than a point to point.

Those people yelling 'insecure' apparently don't have an issue with their data being driven all around town. You want banking info? Just steal the grey box out the the '80 Ford Escort. OTOH, A 'man-in-the-middle' attack requires just that. So, if possible, host at your own ISP.

Re:A lot of folks will say....

[Mattsson]

That tapes go unusable are often due to storing them in a unhealty environment.
If you take the trouble to contact the manufakturer of you specific brand of tape, they can usually advise you about what temperature, dustlevels and air moisture levels they should be stored in.
I've had no trouble recovering data from 4 - 5 year old backups that's been stored correctly.

The correct way of making sure about long term backups is to build/rent a storagearea with a controlled climate within those recommendations.
Another thing to keep in mind is magnetic fields.
It might, for an example, be a bad idea to store them on a shelf made out of metal, in a room close to the local transformationstation (if you've got one inside your building that is) or the electrical feed for you building.
You also have to make sure that the fireextinguising system you use won't damage the tapes if something should happen.

And, of course, keep at least a few of your tapes in a remote location from your regular tapes.
You never know. Even if that place is separate from you servers you might still catch a fire, break in, earthquake, crazy terrorist pilot or waterleakage at that location.

Some of this would also apply for storing your ancient backupequipment, cd's, dvd's and harddrives.
But I don't understand why some people say that it would be safer to backup to SCSI disks... :-/
I thought the biggest difference was in the electronics, not in the mechanical parts of the drive.
Hmm... Though, you *would* expect to get higher quality mechanism when buying a 10x as expensive disk. :-)

This post is getting long... I'll be quit now.

Re:A lot of folks will say....

[ednopantz]

Live mirror, etc. hourly rsync does a nice job

The only trouble is that this won't protect you from data corruption or destruction owing to malice or error. A backup that reflects the state of the original data 1 hour ago is worthless if somebody deleted your customer database two hours ago. A well synched backup merely replicates your loss in two places.
You need different frequent backups, and for that you need cheap storage.

Re:A lot of folks will say....

[avdp]

Consider saving ever version of a config file as config.YYYYMMDD (or whatever scheme you want) prior to changing it. That'll do the trick.

Of course, you'll still need backups in case someone types rm -Rf, but doing backups for versioning purpose is ridiculous.

Re:A lot of folks will say....

[ajs]

Good points.

Your distinction between backup and archival is interesting, but I've been in this business for 12+ years, and that's the first time I've heard of someone *not* expecting a backup to be archived by default (certainly, you make an exception for some short-term backups, but that's an exception, not the rule).

What you suggest a "backup" is for, is why I use netapps. Their snapshot feature provides me with not just online, but user-accessible backups that happen in a matter of seconds on whatever schedule you wish. They also require very nearly the least space possible (it's a block-by-block copy-on-write at it's core, which means that, e.g., a growing log file will not result in a backup of all of the old log data). You can then look at /mountpoint/.snapshot//your/files and see what they looked like in the past.

Add in netapp's snap-mirror and you have the first part of what I described (though I suggested rsync, since most people don't have netapps). Snap-mirror is basically a snapshot that is coppied over to a target netapp and applied like a diff. Snap-mirror allows you to keep two netapps as synchronized as you like with minimal data transfer. However, when I last used it, I did have some CPU contention issues, you may want to check with netapp about how it behaves now.

Re:A lot of folks will say....

[Artifex]

CD's (CD-R(W)) offer a theoretical data retention span of 20-100 years depending on who you ask. So that is safer, but still not perfect.

I've had CDs fail on me within 3 months - the data just disappeared. I made the CDs, verified them, used them a week later, but when I went to them 3 months later, they were unreadable.

These shelf lives are just projections based on laboratory tests at best.

Re:A lot of folks will say....

[angst_ridden_hipster]

Run "rdiff-backup". You get the best of both worlds.

http://rdiff-backup.stanford.edu

Re:A lot of folks will say....

[kcbrown]

You could argue that I should be using something else for versioning, like a CVS repository, but that's too painful in a large multi-user environment.

Too painful? What are you talking about? It's really easy to set up a cron job that goes through all of the users' directories and checks in those files that are either newer than their RCS version or which don't have an RCS version.

Set it up to run once per night and periodically back up the RCS files and you're done.

Re:A lot of folks will say....

[Lumpy]

In fact, any tape older than one year that I've had to go back to has been worthless (read: it had deteriorated data).


you are using low quality backup. I have DLT tapes here that are over 5 years old that we STILL pull data from. (although we DESTROY any and all backups thatare over 7 years old. you cant suponea what doesn't exist.)

How much is your data worth? if you say "NO WAY!" to the $3500.00 DLT drive and $150.00 each for the tapes then your data is worth less than that... and that's fine. every datacenter manager (yes your only server at your 10 person business IS your datacenter) has to make a decision as to the value of their data. we decided that our data is worth $35,000.00 for the backup server, DLT jukebox and 100 tapes and high quality software. I do a COPY backup every night of the servers. a complete backup is 3 70Gig DLT tapes the jukebox can write to two tapes ata time and no backup job spans more than one tape. (you break up the jobs between tapes) plus every night is a complete backup none of the stupid incremental crap. that way I can restore from any set. if everything is destroyed except for the set I took home last week, we are only behind by 1 week.

the best way to get a good backup solution is first , stop messing with toys for something important like your backup. if your tape backup isn't DLT then you are using a toy, plain and simple. second.. have management put a price on their data. if the company's information is only worth $199.00 plus $25.00 per tape for Travan backup. then they need to know that there is a really good chance that the backup is not going to do much in safeguarding anything.

if the CEO has to drive a $75,000.00 company car, why the hell don't they have a backup solution that is 1/3rd the price of it?

Re:A lot of folks will say....

[ajs]

"In the five years that I did phone support, my main problem was people that had been "in the business" for 12+ years and had never seen some of their misconceptions corrected"

You missed my point. It wasn't *my* misconceptions. What I said was in the 12+ years I've been doing this (not meant to impress you with my experience so much as impress upon you that I have a fairly good sample set) no one in the industry has ever asked me for a backup that wasn't for archival purposes unless something very unusual was going on (e.g. we're performing a paranoid backup before an upgrade).

Given that, I just don't see what the place of tape is in the picture. I've used DLT, 8mm, TK-50, TK-75, R2R Mag, etc. Each one of those media I've seen fail after months or years of storage, and I've never seen a tape last more than 5 years.

So, that's the bottom line for me. If someone asks me if we can back up to tape I say, sure but don't expect to be able to read that tape in a year. The response is often shocked disbelief (why would good folks like HP sell crap, they ask). All I can do is outline my experience and the experience of many of my fellow engineers. Then I give them the really bad news: how much it's going to cost to reliably back up and archive their data.

In the end most opt for tape and simply live in the "we'll get our data if we need it" dreamworld until a disaster happens....

Re:A lot of folks will say....

[ajs]

I'm drawing on my history in many firms. I've been in financial institutions (investment and banking), software companies, data services companies, and a dot com. The dot com wanted only some backups to become permaunent record.

Hope that helps.

Tapes *is* the right medium for long term backup

[MooRogue]

I'm sorry, but 220GB easily handled by backup tape. With SDLT and AIT tape capacities exceeding 100GB per tape, two tapes can easily handle your load.

If you have the budget, get an autoloader so you can perform a full backup in one session, or two tape drives for that matter.

Personally, i am backing up 600+GB onto tape and it works well. I've had numerous IDE hard disk failures, yet not a single data tape failure so far.

Agreed

[phorm]

Definately agreed. I just had to replace a 20Gb Maxtor that died... just before the end of it's warrantee period (1yr). These things have a short warrantee for a reason, mainly because they don't live very long anymore.
As drives get bigger, I think this problem will only get worse until they figure out a way to get more storage with less cram and degredation.

Ask who's actually doing it.

[f2professa]

So, how is Pixar archiving it's film data? How about LucasFilm? I'd think from the amount of data they work with, thos guys would be the best at answering that question.

Personally, for long term storage, I'd go with redundant backups of differing media. Maybe hard drives (stored properly in anti-static bags with silica gel), as well DLT stored in a similar fashion. Increase your odds of support by future architecture.

For daily backups, hard drives are surely the way to go. Faster, cheaper, easy to replace, longer lasting media in my opinion. Anyone who says otherwise is trying to cover their job as a tape changer. ;-)

Re:Ask who's actually doing it.

[DJPenguin]

Well, don't know about LucasFilm, but Pixar use massive tape libraries (we are talking robots with 100+ drives and tens of thousands of slots.)

Incremental backups every HOUR, tape drives spinning all the time. They are a customer of the company I work for. (Veritas)

Re:Ask who's actually doing it.

[f2professa]

duh - what am I saying? LucasFilm probably freezes their backups in carbonite. They should be perfectly preserved, if they survive the freezing process. ;-)

Re:Ask who's actually doing it.

[jafac]

as a former Veritas employee - I'd say that DJPenguin's probably talking about NetBackup, not VxVM.

Re:Ask who's actually doing it.

[haggar]

I see. I think "Veritas" as a company name is a bit overloaded. In Finland we got Trans Veritas oy and Det Norsek Veritas oy (oy is something like inc.). I just discovered that there is another US based Veritas, "Veritas DGC inc.". People seem to suffer of lack of fantasy, I guess. Not to diss your former employer ;o)

Re:Ask who's actually doing it.

[jafac]

Oh, go ahead and dis them. I'm bitter.
Veritas is Latin for "Trust" - so no wonder people want to use it to name their business.

To be fair, the Veritas side of the business is fairly engineering-oriented, and is actually a really good company.
But one of their major purchases was a very marketing-oriented (Windows) company. 3 guesses which one it was and the first two don't count. So you get the bad with the good, I guess.

With the dot-com bust, I'm just awaiting the inevitable divestiture with shadenfreude. . .

Re:Ask who's actually doing it.

[haggar]

Uh... somehow my Latin knowledge seems to tell me that Veritas means "truth", as it's chief meaning :o)

As for the three guesses, gee, I give up already. Blame it on my headache. Or my flu. So, which one?

Make triple backups

[anthony_dipierro]

and refresh them onto new disks every year. Problem solved, though at 30 times the cost (assuming 10% interest rate and that you can't sell the used drives).

Slashdot - the "Jackass" of tech support

[HotNeedleOfInquiry]

Here's some more questions:

Can I use my laser printer to print on Gummy Bears?

Can I dry my cat in the microwave?

Can I put rice in my car radiator?

Can I unplug all the fans in my computer so it will run quieter?

Can I run 120 VAC on the spare CAT5 pairs?

Re:Slashdot - the "Jackass" of tech support

[HotNeedleOfInquiry]

Sorry, I only had 5 minutes to come up with something. The boss is stalking around looking for slashdot goof-offs like me.

Re:IDE ?

[Gudlyf]

"because in 5-10 years from now, IDE may not even exist anymore..."

In that case, you could always just buy a new, cheap system for the purpose of reading the IDE disks, and keep that in the vault with the drives "just in case".

I'm not saying this idea with backing up to IDE is a good idea, though. Drop a tape on the floor while you're running to the tape drives for a critical restore, no biggie. Drop a drive on the floor in the same situation, you'd better hope your resume wasn't one of the files needing a restore.

Back to topic, he asked about IDE storage

[panthera]

What we have is a plethora of opinions about what's wrong with IDE. Why don't you solve your problems. 1) It's unreliable over the long run. - Have multiple disks that will be unlikely to fail at exactly the same time. 2) Remote storage - Get a friend that has some space and get him to store a disk for you. 3) Jostling hard drives might reduce life expectancy. - Perhaps try remote incremental backups over the net. Do one large copy locally then locate the disk at a remote connection and backup incrementally. ( remember the local part for the first copy - I got a note from my ISP when I did it over the net.) 4) Lubricant failing in disk. - Two things, you can either replace the disks on a cycle, perhaps every two or three years. ( prices keep going down) or keep the disks spinning on a remote computer that is UPS protected. There are many other ways to do this. What we need here is people with solutions. Not more problems. Think outside the box.

Eggs and baskets

[phil reed]

If the tape drive electronics fails, you can get another tape drive and still read the tape. If the IDE drive electronics fail, the data on the drive is unreachable without massive and expensive intervention.

Crappy backups better than nothing

[jolshefsky]

I don't know how "pro" you want to go with this, but I ran into a similar situation and resigned myeslf to the same solution. My DDS2 SCSI tape drive is getting to be too small at 4/8GB. I would like to have a tape solution, but it's too expensive for my purpose. I get drives as pulls and last-years-models so I only spent US$150, but with tapes at US$10, even 8GB is absurdly small. If I were to go with new equipment and step up to DDS-4, I'd be out about US$1000 for the drive and another US$20 for each 20-40GB tape. Total cost for a basic 3-tape rotating backup: US$1060.

On the other hand, I could spend (as I have) US$40 on a basic (a.k.a. el-cheapo) FireWire-IDE case, US$30 for 3 removeable IDE enclosures, and (eventually) about US$70 each for 3 60GB IDE drives. Total cost: US$280.

What do I sacrifice? Not much ... one of the drives might fail. At that point I'd just replace it with another US$70 capacity drive (which would probably be larger.) If I needed to restore something from backup, I'm already looking at up-to 24-hour old data, and if that drive happened to die, possibly 48-hour ... it's unlikely that all the drives would fail at once.

The advantages? I can use the US$780 I save for something else and I don't have to worry about shelling out another US$1000 every four years just to scale to "current" requirements. I don't know what the upper limit of an IDE drive is these days (i.e. what can the ATAPI bus handle) but even 200GB is pretty big for me right now.

Anyway, just a few thoughts. The basic thing is lower cost for nearly the same risk ... tapes fail too, you know. Remember, too, that this story would be very different if I had to handle 50 machines instead of 2.

Re:Crappy backups better than nothing

[geekoid]

bear in mind that if(when) you controller goes bad, it could corrupt the data on ALL your drives.
I have seen that happen. Yes tapes fail, but if you loose one tape completly due to hardware failure, you still have you others. Pus tapes are easier to maintain legacy information, if that is needed, obviously.

Re:Crappy backups better than nothing

[jridley]

I don't know what the upper limit of an IDE drive is these days (i.e. what can the ATAPI bus handle)

Currently the large disk standard is "BigDrive" by maxtor. It can handle up to 128 petabytes (1 pB = 2^50 bytes) so that's 128*1024 = 131072 terabytes.

You should be good for a few months.

The more immediate problem is that 32 bit OS's only can typically handle 2 terabytes due to 2^32 * sectors; of course that limitation can be avoided by using different filesystems.

May I recommend remote backup?

[ekrout]

Many people forget that remote backups require no on-site hardware or software and don't require you to spend hours upon hours configuring things.

Even better is that any flood, tornado, or fire at your house or business will not ruin your tape, dvd, cd, or hard drive backups. You simply connect to your remote backup location and restore your old data onto your new hardware. It's that simple, and it's cheap in comparison to spending $3,000 on a tape backup device that only stores 150GB of data per cartridge.

You may want to see if this remote backup company has services that fit your needs (I don't work for them, so it's not a plug). Basically, they state the following as the main appeals to remote backup:

Your data is continuously backed up as it changes, 24 hours a day, so it's always up to date. And it's stored electronically at Iron Mountain® data centers, where more than half the Fortune 500 protect their data.

No-Wait Recovery - Instantly recover your data to the point of failure, eliminating downtime and data loss from relying on a previous night's backup. And a unique web interface allows you to initate restores from any Internet browser, anywhere.

No Tapes, No Hassles, Lower Costs - Tape-less backup and recovery means no hardware or software to buy and a fully automated process requiring little employee time or resources. Lower your data protection costs while freeing IT resources for other tasks.

Why are you backing up?

[Cap'n Canuck]

I know it sounds like a stupid question, but why are you backing up data? What are you trying to solve

Short term failure
A luser makes a mistake, or there's a glitch in last night's source code library, and all your current data is foobarred. In scant minutes, you can restore lost data from overnight backups, (or even hourly incrementals), and you are the hero. Realistically, you're just doing your job, and you'll never get thanked for it.

Complete Failure
In the event of a building fire/server room flood/earthquake/Act Of Dog, then you may need to retrieve all your companies data from as near back as possible. This backup should be off-site, and as frequent as feasibly possible

Long Term storage
This is for archiving of a project, etc, and should be off-site. Also for archiving source code in case your company goes belly-up, so that customers can still use and modify your software (in escrow).

Ask yourself which scenario you are dealing with, then the answer as to which media is the one to use may be clearer.

Re:failures

[tomhudson]

Sure, drives will fail in continuous use, but running them for an hour every two weeks, and in this case, doing an archive (dd if=/dev/hda1 of dev=/hdb1) where the head isn't skittering back and forth a million times, isn't going to cause much wear and tear.

My suggestion: check out the usb removable drive trays. This way, you can hot swap/back up w/o powering down.

As for the comment about the two different types of hard drives - this can be said about anything. for example, there are two kinds of cars - those that have turned to junk and those that will turn to junk. Will that stop you from driving a car?

Tapes are NOT a long term archival medium.

[silentbozo]

Tapes are fine for backups, but I never expect to pull complete and usable data off of them after 6 months. Why? Tape degrades - it's nothing more than rust on platic. As humidity and temperature change, you can end up with a solid roll which will stick to your tape drive heads and result in whole patches of magnetic coating coming off. I worked on a project restoring data from 10+ year old reel-to-reel tape, and it was a nightmare. 1 out of 4 tapes was completely unusable.

Even worse, tape drive formats keep changing - and since tape drives are guaranteed to wear out, where are you going to get a working tape drive to restore data 5, 10, 15 years from now? I've gone through 3 tape drives in the last 8 years - thank god I got a CD burner early, that data I can still read (although it's about time to start recopying stuff from 1996.)

Basically, if you entrust your data to tape long term, you have to continuously copy that data to new tapes, and or new tape formats. Where tape has traditionally shined is as a short-term backup format, although with the drop in DVD-burner drives/media, and the high-cost of high-capacity tape drives/media, this may no longer be the case (assuming you get some peon to do the big backup on DVDs, and you get to do daily diffs - otherwise, having a bank of tape drives is cheaper on staff time.)

Re:Tapes are NOT a long term archival medium.

[aero6dof]

If you're worried about changing tape formats, why not archive a tape drive too. If you're really paranoid, archive an entire computer system.

You might also get extra security for using Open Source software on the system because then you have a chance of opening up the code and port the backup format to whatever you're currently using.

Re:Tapes are NOT a long term archival medium.

[Burdell]

Tapes are fine for backups, but I never expect to pull complete and usable data off of them after 6 months. Why? Tape degrades - it's nothing more than rust on platic.

So use a reliable tape format and store it properly. When stored properly, DLT has a shelf life rated in decades.

Even worse, tape drive formats keep changing - and since tape drives are guaranteed to wear out, where are you going to get a working tape drive to restore data 5, 10, 15 years from now?

So use a tape format that is backward compatible. Today's SDLT drives can still read all the old DLT formats.

Where tape has traditionally shined is as a short-term backup format, although with the drop in DVD-burner drives/media, and the high-cost of high-capacity tape drives/media, this may no longer be the case

Check the shelf life of CD-{R,RW} and DVD[+-]{R,RW}. Most of the CD/DVD media is only rated for a five year life at most. Mastered CDs and DVDs will be readable for decades, but burned CDs and DVDs won't be.

The bigger problem with really long term backups is with the data format used by the backup software. If you use a backup program that only runs under Windows, what are you going to do when you need to recover that data in 10 years, and you only have Linux (or the other way around, the point still stands)? This is where Open Source software is good, because (assuming you can still find the source) you can always decode the data stream.

Media Rotation

[devnullkac]

A couple possible risk mitigaters would be to rotate your media more often (no pun intended) and use a RAID rather than single drive as an archive of any given media.

For media rotation, you should depend on any one drive for no more than a couple years and then retire it (copying its contents to a new drive if they weren't already expired). It may be a bit expensive, but if you're looking for high speed storage, it may just beat the alternatives. Depending on the size of your organization, you can perhaps "waterfall" the retired drives into less critical roles (e.g. desktops)

Backing up to a RAID would involve using (possibly) trios of hard drives together, plugged into removable bays that connect to a RAID controller. That permits one of the drives to fail while still allowing your data to be recovered without having to think about it.

Re:Why would your disks be

[ptomblin]

You haven't priced DLT tapes recently, have you? I bought 10 USED 15Gb DLT IIIs for $100, and that was a good price. They cost $35 each new. That's over $2 per gig, compared to about a dollar per gig for IDE drives.

Re:Tapes *is* the right medium for long term backu

[viper21]

Sure, it can handle my load, but can I afford it?

Amazon - 110/220GB Sdlt Internal SCSI Tape Drive

Not all of us home computer users with 200GB of files happen to have $5,000 in their back pockets for a backup device.

Is there any way to do this much backup, say, for less than $500?

-S

Just copy it around

[photon317]

The "right" way to make your data reliable is with mirroring of various sorts. On-site backups are kinda silly except when you're using them operationally because you dont have the disk capacity to do otherwise for infrequently used data. Backing up to removable media should be exclusively for offsite storage.

So get two drives and mirror your data, and you're covered in the case of drive failures. If your worried about a whole machine going up in smoke, maybe do a nightly or hourly rsync to another machine across the room.

If your home data is important enough to need offsiting (usually a home user's "important" data amounts to what could fit on a CDROM, not 220 gigs - the rest is probably multimedia fluff that you can stand to re-encode or download in teh case of a tornado or fire), then consider rsyncing with a freind at night over your DSL or cablemodems in a mutual arrangement. Encrypt the data before syncnig it over if it's sensitive.

If you're a business with large volumes of data that need to be offsite in case of disaster, then the best practice is still tape drives of some sort, and an offsite storage service like Iron Mountain.

Yeah, and the first poster. . .

[kfg]

got modded as a troll too, even though, for once, the first post was on topic, mildly funny, actually had a *point*, and was labeled as facetious in the body of the text.

Go figure. Neither you nor he deserved to be modded as trolls. Troll is overused because it gets used as a catch all for various posts that the moderator finds offensive. It's perfectly possible to be offensive to someone without being a troll or flamebaiting. A subtlty that some of the moderators haven't grasped. In fact, I'm not sure some of them have even bothered to check the *definition* of the world troll and flamebait.

So, in short, you did not serve as an example of why we need the troll moderation.

You serve as an example of why we need an *asshole* moderation.

I hope that makes you feel better, have a nice day.

KFG

Re:Why would your disks be

[Sloppy]

And I don't believe 220 gigs of IDE space is cheaper than 220 gigs on tape.

That's how it used to be, but you're out of date. Take a look at the numbers; it's shocking and no one would have believed it ten years ago. From a $/byte perspective, tape is becoming obsolete, right about ... now. Ignoring the cost of the tape drive (which is pretty high for large-capacity tape), both are at about a dollar per Gigabyte, and disk price has been improving much faster. A year from now, tapes will cost more than the disks they back up.

About all tape has going for it over disk, are physical robustness issues (the lack of the "stiction" problem that he mentioned, the fact that dropping a tape onto the floor is less scary than dropping a disk, etc).

Same old song and dance...

[mengel]

If you want a Real Archive, you should

1. always keep at least 2 copies of any data
2. copy the data every six months to a year

Just build it into the plan.

Its the only way you really know you can read it. Then when you need to shift the data to newer media, its just part of the routine.

Plan to keep the bits forever, and rewrite and/or replace the media regularly.

That said, actual shelf-life of disk drives would be an important cost planning number. I've never seen such a number published.

There's a reason...

[giminy]

There are some important things to remember when comparing price/quality.

IDE drives have gotten cheap because everybody uses them. So companies expect Joe Blow to buy their disk drive because it has more GB per dollar. They'll use cheap motors, cheap heads, and cheap platters to lower their cost so they can get Joe to buy their disk.

Tape drives, on the other hand, stay expensive because they are not subject to the normal consumer market. So they retain higher quality parts and smaller capacity. Most people don't have to do full backups every night, and again most incremental backups aren't overly huge, so non-gargantuan tape capacities are okay for a lot of the market. The drive and tape manufacturers aren't about to go ruining reliability just to satisfy the high-capacity end of the market; they can make enough money doing what they're doing.

So use tape if you want your data. Use IDE if you want to look like you're making backups. It's your money (and hopefully only your data)...

Re:Tapes *is* the right medium for long term backu

[Drakantus]

"I have $500 to spend on a backup solution for my 220GB data pool, and I was thinking of buying 4 120GB IDE drives along with an IDE RAID1 card and useing the array for backups, anyone have other ideas?"

"No way, you are insane. IDE is horribly unreliable and you will surely lose your data. You need a $6000 tape drive, if you can't afford it you are better off with no backups at all"

IDE vs SCSI

[HiyaPower]

IDE drives have a reputation as being substandard. This is actually not the case overall. Most IDE drives are housed in machines on the office or house carpet which is one of the worst enviroments in the world for dust, dirt, grit, dog pee and heat. SCSI drives are often housed in the nice clean, cooled machine room. Further, IDE drives in personal machines go through many, many more spinup cycles than that 24/7 server disk does. Check the MTBF figures, they will also specify a number of spinups figure, since this is where the bearing wear is.

Backup tape bleeds and needs to be re-generated from time to time. It is no answer to the problem. Just ask anyone who has done the "tape salvager" routine on the 9 track. However, the same thing will happen with disks. the thermal stuff will weaken bits, etc. Personally a raid 5 array on removable ide that is periodically re-read to regenerate lost bits is the way to go.

As far as doing the same thing that everyone else does with tape & SCSI, most folks have a lemming viewpoint. They do not like to stick their heads out and put their careers on the line for something that differs with what everyone else does.

Re:Tapes *is* the right medium for long term backu

[glesga_kiss]

I've had numerous IDE hard disk failures, yet not a single data tape failure so far.

You speak of not having tape failures, but you omit one important fact; how many times have you successfully retrieved data from tape?

IDE disks will fail from continual use, and that failure will generally be obvious, but what way do you have of knowing that you genuinely don't have any tape failures, if all you are doing is rewriting over the same tapes?

Re:Here's a tip..

[Trolling4Dollars]

That is exactly the reason I created this account. I love to put up insightful/informative posts that are laced with tons of profanity. I just get a kick out of watching people get pissed off that there IS useful information, but that there is also a lot of profanity. Just goes to show you that language means nothing. It's my way of getting back at the grammar nazis and the people who feel that informative/insightful mods should only go to people who don't use profanity. It's like they have some kind of weird disease that makes them think that just because some swears a whole FUCKING lot, mans that they don't have anything useful to say. This isn't true. Some of us just swear because it has style...

My tape backup story

[bigmouth_strikes]

My company provides mission-critical systems at the core of its customers business. Needless to say, data cannot MUST NOT be lost, so therefore backing up both transactional and referential data is an essential part of the setup.

Some 10 years ago, we weren't as good with this. We usually let our customers handle their backups themselves, maybe on our recommendations. One company had a tape backup running every night, backing up the essential parts of the database. After they'd been up and running for a few years, they had a failure and lost the disk with the database. Fine, just retrieve it from the tape backup. Well, it turns out they had run the same tape all along. It had effectively been polished blank by the magnetic heads over years, and not a trace of their database could be found.

They didn't have any other recent backups, so they were pretty much screwed. I think we lost their business after that.

What ever became of C3D?

[JoeShmoe]

The mythological 100GB per disc flouroptical storage medium? Was it all a scam? Did it get killed by patents? Did they run out of money?

You remember Constellation 3D right? You first heard about them in 1999...try searching google for "Constellation 3D" or "C3D" or "Eugene Levich" (the CEO). I see various articles about partnering with this or that, but nothing past 2001. And their website has been offline so long that google doesn't even have it cached.

So...whatever became of this? Is the concept sound? Can someone else resurrect it? Or was it just a lie like the guy who pretended to have the ultimate streaming video codec but in reality ran coax through the power cord?

- JoeShmoe

.

Re:Tapes *is* the right medium for long term backu

[Anonymous Coward]

>> Not all of us home computer users with 200GB of files happen to have $5,000 in their back pockets for a backup device.

For most, the only mission critical stuff is maybe their quicken files, some documents, and whatnot. I can backup whats actually important to me on a couple of CD-Rs.

If you had 200GB of stuff that was worth backing up, you'd pay what it costs to do so.

It's just a cost/benefit thing.

Is your MP3 collection worth 5,000$?
1000$ for a redundant offsite raid-5 array?
500$ for 2 more rotated backup drives?
How about 100$ to dump it onto CD-R or DVD-R?

The Archive Economics Conundrum

[unfortunateson]

This is probably the second-oldest problem in IT (after 'how do I get a girlfriend?').

Tapes and optical are slow, and either painful manual processes, or expensive robotics. By the time you put the robotics and the drives together, the fixed disks are cheaper. It gets much worse if your goal is near-line storage -- but just archival is expensive.

Even compressing the files (which I wouldn't recommend), you're talking about dozens of DVDs for a full backup of 220GB. Incrementals make it a little easier to back up, but have costs to find each file you need to restore, and reduce your overlap in redundancy [sic].

Time * a = money
Storage space * b = money
Disaster Recovery costs * c = money
Service Agencies * d = money
Capital Investment * e = money .... * y = money
(Consultant to calculate a through y) * z = money

Personally, I'd only use swappable hard drives for quick-fix backups: rotate two or three for short-term backups, with my longer-term and archival going to some other media. Sooner or later, you'll need to upgrade those hard drives to a larger capacity, and that will be expensive compared to buying another spindle of DVDs.

Of course next year, you may need blu-ray DVDs to handle your backup.

OK... this is the third-oldest problem, after 'Should I buy now or later when the next kewl stuff comes out?'

the absolute surefire way to back something up...

[rebelcool]

i thought of this the other day. It's kind of amusing, but it would work for absolutely important data.

Get alot of archive quality, acid-free paper. Get a printer with alot of archive quality ink and print out the data in binary. Dots or slashes would work fine for the 1's and 0's.

Archive quality paper and ink lasts for hundreds of years. Should you lose the data on a magnetic or other storage medium, you could always run these papers through a scanner with some OCR and retrieve the data.

Sure, a fire or flood could damage these if you don't have them protected against that, but at least you won't have to worry about deteoriation of the medium.

Re:Tapes *is* the right medium for long term backu

[MooRogue]

True. The tape drive solution is oriented towards businesses who have the money for a backup device :)

From the poster's requirement of needing offsite backup, i was assuming that it was for a business.

For home users, you can probably afford one generation behind. A DLT 8000 (40GB/80GB Compressed) drive on eBay runs for about $500. A DLT 7000 (35/70GB) runs for $300-500, so it is possible to do tape backup on a budget. It's the usual tradeoff between time and money, so you'll need to spend more time changing tapes.

On the other hand, for home use, i only archive my data onto CD since most of the data I have does not change and does not need incremental backups..

Perfect Storage Medium

[techsoldaten]

For my clients, I always suggest the use of stone and / or clay tablets for all mission critical data archive projects, regardless of size or scope. Bablyonian and Greek models of data retention from as far back as 4,500 years ago are (in many cases) superior to the models we commonly use today, with much of the physical meadia having survived electrical storms, tornadoes, floods, fires, and wars on every scale imaginable with a data corruption rate of zero and without the benefit of a climate controlled room, dedicated security staff, or even a closet for media storage. Imagine the elegance of a 84'3/4 STROM (Stone Tablet Read Only Memory) machine hooked up to your Slackware Archive server for performing restorations, and the ST Binary Writer you have networked to your backup systems and kept physically over by the quarry... nice! The TCO for slab is far less than that of tape archives, considering you can store the media in a pile of mud and hose it down when you are ready for a restoration.

M

Re:this is idiotic.

[chef_raekwon]

err no. scsi drives are much more durable. these drives are not identical. how do you spin the same hardware twice as fast, without failure? fact is, you can't. remember, scsi drives run at 10000rpm, or 15000 rpm. not 7200.

think before you post.

Heresy

[iangoldby]

My first question would be whether you really need to back up all that 220 GB of data?

If that seems heretical, think about it for a moment. Do you have 220 GB of information, or do you just have 220 GB of data? How much of it could be regenerated? mp3s ripped from CD can be re-ripped. Someone mentioned the enormous backup requirements CGI animators run into, but in the worst case, could these rendered files be regenerated in a few days from the original script? You have to weigh the cost of waiting for it to re-render against the cost of backing the whole thing up.

And what about installed software? If you know what is installed, you only have to back up the machine-specific configuration and customisations, because you can re-install the software. It will take longer, yes, but again, you weigh up the pros and cons.

On my home system, I only back up my user data directory, /etc, and a few other miscellaneous bits. That's a tiny fraction of the data stored on the disk, but it is pretty much all of the real information. If I lose my hard disk, I reckon I could get up and running again in a day or so (which for me is fine), and I'd probably find I've cleared out a load of cruft while I'm at it.

Re:Heresy

[Decimal]

Heh, I "back up" my data on my own hard drive. The D: partition keeps everything I'd ever want and if my OS ever screws up I simply reformat C:. That's not to say I don't have have a CD with essential data on it burned.

Re:this is idiotic.

[dfung]

> It's likely that SCSI drives are identical to
> IDE drives apart from the interface.

Uh huh. Well, why don't you send me a couple of those 15K rpm IDE drives and we'll see how they compare. Oh there aren't any? OK, I'll take some 10K drives then? None of those either? You do have 7200 rpm ATA drives? Darn, there's hasn't been a 7200 rpm SCSI drive for a couple of years now...

Yes, they must be exactly the same...

SCSI drives cost more because these days most of them end up in server or enterprise level applications and are optimized for that world. So there's higher rotational speed, faster transfer and cache, higher head seek speed, and probably beefier construction.

Non-volatile: no such thing

[sterno]

well first of all, I've never seen any hard drive evaporate :). But beyond that technicality, there's no such thing as permanent storage. There are relative degrees of permanence and generally, the permanence is inversely proportional to the convenience. Storing a book on stone is pretty permanent, but it's a pain. Even still, the stone will slowly dissolve over time, and you could accidentally drop it and have it shatter.

You need to determine what is good enough for your needs. Will a hard drive last 5-10 years in storage and still retain data reliably? What about a DVD? What about paper? I have a computer running as my router, the drive in that dates back to my junior year of college making it roughly 7 years old. Still works fine.

Re:Non-volatile: no such thing

[Alien Being]

"accidentally drop it and have it shatter"

Moses: I bring to you these fifteen [crash], ten, ten ommandments.

Re:the absolute surefire way to back something up.

[jim3e8]

Yes. It's also a good way to pretend you're swamped with work. "This stack of paper in my inbox? It's actually Coldplay's latest single."

Oh yea?

[FreeLinux]

Seems to me that you should use the most modern solution out there. You want off-site storage and you want redundancy and you might like it to be distributed.

Sounds like P2P would be the ticket here. Just upload all your files onto Kazza and Gnutella and then let nature take its course, scattering them all over the internet.

Anybody see a problem with this? Seems like a "legal" use for P2P has finally shown up.

Re:the absolute surefire way to back something up.

[tomhudson]

Lets have some fun :-)

And just how many tons of paper are you going to need to reliably back up a terabyte in dots and dashes?

Assuming double the standard density (160 chars per line instead of 80, 132 lines per page instead of 66), which actually works out to quad density, you get 160x132=20120, say

1. 20k per page
2. 50 pages = 1 mb
3. 50k pages = 1 gb
4. 50m pages = 1tb

Now let's assume boxes of 5000 sheets. 10,000 boxes, at, say 20 pounds a box = 200,000 lbs, or 100 tons. Man, give me the toner franchise for this!

Safety First . . .

[Badgerman]

and second, and third.

DON'T use a hard drive for your backups. Find something, ANYTHING more solid, more reliable, and more compatible - the gods know what hell you'd go through using one of those 10 years from now - but I'll bet you things will still read your tapes and CDs.

Hard Drives having moving parts. For long term storage, moving parts==bad. They're highly vulnerable and wear themselves out, also bad.

One thing you may want to consider is a backup heirarchy. Some things get backed up so often on such and such media, etc. Maybe some things CAN get backed up on a hard drive, but it shouldn't be anything you can't live without or recover from loss quickly.

I say this not as a hardware guy, but a programmer who's seen storage go wrong all to often - then *I* get to rebuild or find things. Or, I have to think backup because NO one else has.

Five Points About Archiving

[maggard]

1. Accept that you can't just stick magnetic media on a shelf (in a vault, even climate-controlled) and expect it to last forever.

   Bits rot. Under the most perfectly controlled environment the damn stuff still goes bad. Be realistic, anticipate this, do everything you can to slow it down, but plan for it and make provisions when you first put your archiving strategy in place. Tapes are likely more robust the platters as there's fewer critical parts to go wrong but nothing is perfect.

2. Accept that CD & DVD don't have 100-year lifespans, mebbe not 10 year, and possibly far less.

   Yes they're cheap but we've far less experience with these media then we do with tape and studies are showing that they dyes may not be as stable as first thought. Heck, there's even a bug out there that eats some of these. There's also the question of long-term standards in some cases like DVDs.

3. Checksums and multiple-backups (that reinforce eachother) are a necessity.

   Nothings worse then losing one part of an archive at one site, another part at a different site, and being unable to easily reconcile the two to get a good whole set. Make sure that however you archive things, same media or different media, that partial archives can be reconciled.

4. Everything evolves - Keep updating backups.

   Years ago there was a big scramble to recover the US Govt's 1950 Census. It had been stored on steel tape and the required Unisys readers were no longer. (Much of the data was available but the entire raw set wasn't.) Eventually a working one was built from cannibalized parts in museum and private collections but the lesson was clear: Don't depend on the readers. The same goes for the recent BBC Domesday Book debacle - nobody could read the optical disks. Any good archive scheme will call for the material to be re-read and re-transcribed regularly in order to ensure the entire recovery-chain still works: Hardware, software, OS's, etc. If recovery becomes difficult migrate the material.

5. Be pragmatic about what you archive.

   All too often folks archive everything 'cause they're too lazy to determine what is actually necessary and what isn't. Combine this with the difficulty of later having someone unfamiliar try to winnow down the material and this becomes a real problem. Even worse is later trying to find the useful material among all of the dross. Establish clear policies of what can be archived and make folks justify their material. Just as importantly make sure the costs are clear up front, even to the point of charging them a rate covering several years of storage initially. Suddenly some pack-rat deciding EVERYTHING they've ever typed is potentially a goldmine isn't so funny. Lastly, run everything past Legal: Some of this they don't want hanging around any longer then necessary.

Re: Five Points About Archiving

[isaac_akira]

Now, I would guess that Beta VCRs would be a better analogy, but I think that TV stations used much of them for archival, and they could be more popular than I think.

I was suprised to learn a few months ago that Sony is *still* making new Betamax VCRs, though they said they were going to stop making by the end of this year. (Article about the end of Betamax)

As long as a format reaches critical mass, I think you will be able to find readers for as long as the media will hold up. If people are still using the media, then other people will keep making or repairing the readers for them.

My last backup...

[cide1]

The last time I did a full backup, ~5 months ago, I used over 70 cdr's. Sure it sucked, but they are cheap, the format is standard, not going anywhere soon, and I didnt have to buy any new hardware. It looks like DVD formats have stabilized a fair amount, and most data doesn't change often. If I had 220 GB, I would probably write a script that would compare time stamps on files, against a prior backup, and then you are only doing incremental backups. If your data is worth more than the reliability of DVD, your data is worth enough to fork over the 5 grand for a professional tape drive.

Here's a good idea (but crappy product)

[JoeShmoe]

Check out PowerFile for a good large size long term backup option.

Pros:

* Gold CD-Rs have a shelf life of several decades so a PowerFile full of gold CD-Rs would give you 160GB of storage that would last longer than you live. I'm not sure what the shelf life of DVD-Rs are but if it's at least ten years, then you can have close to a terabyte of storage.
* You can populate the slots as you need them. Start with 50 DVD-Rs and then add more when your data needs increase.
* The data stays "online" so you can still access it (unlike with tape) so you can free up some hard drive space by moving stuff you access infrequently
* If you use CD-RW or DVD-RW discs then you can perform an incremental backup and save some time

Cons:

* More expensive than they should be...a Sony 300 disc DVD/CD changer only costs $700 yet as soon as you connect anything to a computer it is suddenly a couple grand more
* Slow, obviously since it has to spin to the correct disc to access it. However, for sequential operations like backing up (writing one disk after the next) it should work fine.
* Software sucks...they actually have the gall to charge extra for the drivers that you access the drive over a network. I'm not sure exactly how but basically you can't change the disc over the network unless you pay extra

What I would dearly love to see is someone like Linksys make a cheapo version of this, but instead of designing it as a stupid fireware component, toss in their little linux flash computer and make it networkable from the start.

OR...maybe someone smart out there can figure out a way to make one of those home theater changers into an ATAPI device that I can wire in directly to my computer, or said linux flash computer.

Is anyone aware of anything close to these PowerFile things, only at a level that is more in line with what they actually are (just a DVD drive with a little robotics to insert and eject discs into a carosel).

- JoeShmoe

.

Re:Um you've pretty much answered your own questio

[Bobulusman]

I have a 20 mb (yes, you read that right) hard drive from 1989 that I can still read just fine. I've hooked it up once or twice over the years just for the nostalgia.

I pretty much agree

[kiwimate]

Obviously, you've never had a tape physically fail.

Maybe it's just me, but after the experiences I've had the last year with crappy tapes, I'm surprised the "tape as a backup medium" idea hasn't been seen for the farce that it is.

Here's the thing. Tape is fine as a short term backup medium. It's relatively cheap (and I do emphasize the word "relatively"), which means you can throw it out after a year and get new tapes. But anyone who expects to be able to retrieve data from tapes that are four or five years old is dreaming.

Long term solutions? As you say, it's time for something new. Tape or disks simply aren't viable in the long term, and optical, while more permanent, has not kept pace in terms of speed and capacity.

You could always ...

[Greedo]

Stegnographize your data and hide it in an amateur pr0n video.

To restore from backup, search with Kazaa.

Serial ATA? Nah, go with USB or Firewire...

[Ricdude]

...removable hard drives. Pick one or the other (or use both, for greater redundnacy). Better yet, here's a neat drive enclosure: http://www.addonics.com/products/external_hdd/comb o_hd.asp . Add the appropriate cable to the back for USB, Firewire, IDE, or PCMCIA connection. How's that for versatility?

With the USB and/or Firewire hard drives approaching or exceeding 120 GB capacity, you'd only need a handful to get a decent set of rotating backup images. And with multiple interfaces on some of the drives, you have a greater chance of being able to read the media in the future. Good Luck

Given SerialATA's emergence..

[Inoshiro]

I wouldn't back any data onto a parallel ATA device for use in the far future.

Re:Solution to your answer

[SlamMan]

"Surely your company can afford you one of these."

Are you kidding?? That's 9 large. We have enough trouble convicing them they need to backup, much less et something like that. Lots of people need lower cost solutions than that.

What's the purpose of the backup?

[lga]

I think most people have missed the point here.

If the backup is of data that must be archived and stored long term then it's worth sitting down and copying it onto a pile of DVD-R's or similar. They will still be readable when a hard disk will have long since siezed up.

On the other hand if the backup is simply to guard against those "what do you mean you typed rm -rf *" moments then a copy on removeable hard disks will be more than adequate.

Personally I think a combination of the two is probably best, optical / tape media for archiving, and HD's for everyday stuff. For example, I don't need all my MP3's on tape but my wife has a copy of her thesis on CD-R.

Steve.

Backup questions

[Deathlizard]

Is stiction still likely occur on newer makes of IDE drives or have manufacturers beaten the problems which caused this in the past?

My guess is that most manufactures have improved on their designs in order to minimize this problem, especially since companies like maxtor have been pushing their 5400 RPM Drives as a backup solution, but there is still a chance that it could happen and for an offsite solution it wouldn't be the best choice

Likewise how likely is bit drop-out and general data degradation over say a 5 year and 10 year period, and what do people think would be the likely maximum feasible time that a shelved drive would be usable for?

I had 30 IBM ps/2 50z in my basement at one time. I think 3 of them had good hard drives. 3/30 is not good odds considering that they are only 15 years old. Newer Drives have improved on these older mediums but they are also more dense, increasing the chance of degradation over time.

Any suggestions as to how would I need to store drives in order to minimize these types of problem and maximise their feasible life as archival media.

Keep them away from heat and humidity and spin up frequently. although some manufactures vary is their suggestions

Basicially, a Hard drive array is a good idea in a daily to weekly backup schedule. especially if they are networked on a rack or in a server and have RAID 5 reliability.

Anything beyond weekly should be done on tape backups. the ATL Tape Libraries are nice rack tape loaders, and are networked.

also make sure that the backup system can be racked. if they can you can put a backup rack in a different networked building from the servers and that could constitute an off site backup in many cases.

Re:the absolute surefire way to back something up.

[rworne]

As long as we are on that track, the Internet was designed to withstand nuclear attack, so its obviously the best choice: archive, encrypt and have others mirror your data.

I know, I know, how do you get these people to do it? And how much will it cost? Easy, and I can get them to do it for free.

Name the backup DIVX_The_Twin_Towers.avi and put it up on Gnutella or WinMX. Problem solved.

Some advice

[Monkelectric]

Let me first go on record and say you are a complete fool if you think this will work ... Bite the bullet and buy a 100gb native DLT drive. At my last job I backed up 2.6TB on a DLT+autoloader, I know 220 gigs *seems* like alot of data, but you're small time.

However, if this is going to have *any* chance of working, you will need to read the drives on a regular basis. I would pop each drive in a machine and (in linux) do a "dd if=/dev/hdc of=/dev/null" to read the entire drive. I would do this monthly.

Why you ask? Because modern hard drives are sophisticated and they auto-correct errors *before* they become a problem. Hard drives will do things like correct recoverable errors and rewrite weak sectors when they encounter them. Thus if you go over every sector of the drive every once in awhile, you will use the drives auto-correction features to your advantadge (and protect against the drive fading, which would be my primrary concern, not stickage (which is easy to fix)).

Re:RAID-5 not panacea

[LoudMusic]

Yeah, that's the big issue - reducing your single-points of failure. The big one is the RAID controller. But in most cases you can just replace the broken hardware and the system will recover.

Wind...Rewind....Check...Wind...Rewind.

[hughk]

In the bad old days, most of our archives would have to be on tape as disk capacity was very limited by comparison.

We used to always make two identical backups. The tapes would be automatically called up by the archive system and would be wound and checked. If a tape contained too many bad spots then it would be recovered by a copy from the remaining good tape. Note that bad spots were generally regarded as reoverable as each tape block was in a group protected by an XOR redundancy block which could recreate any single block dropped.

The winding/rewinding as the tape was checked got over any problems with tape settling and this process worked reasonably well with several media changes over about 20 years.

RAID != Backup

[Door-opening Fascist]

RAID is not a backup solution. RAID improves reliability in the face of drive failure, meaning that it will ensure that your rm -fr / will execute even if you lose a drive in the process.

Re:RAID != Backup

[j-turkey]

RAID is not a backup solution. RAID improves reliability in the face of drive failure, meaning that it will ensure that your rm -fr / will execute even if you lose a drive in the process.

You are correct in this assertion.

However, if I were to (make the mistake of) look(ing) into RAID as a backup solution, I would not use RAID 5. I would set a RAID 1+0 solution with snapshot mirroring. This way, I can lose up to half of the physical drives in the array before losing my data (rather than losing my data if I lose one drive and a parity drive). RAID 1+0 is also far faster than RAID 5...and with the snapshots, the system would not be prone to rm -rf / issues. However, this would still be vulnerable to filesystem problems.

Just my $0.02

-Turkey

2 years on IDE, 6 on SCSI

[Mu*puppy]

Served a 2 year mission for my church back in '98, so I removed my old Western Digital 2.6G HD completely and put it in a drawer while I was gone (-after- making CD backups of critical stuff along the philosophy of CYA, mind you). Came back, and all the data was just fine (in fact, that same drive is chugging away in a Linux box now). Old Micropolis 1G SCSI drive from a (failed) BBS box, hadn't been powered up since '96... until 1 month ago. Spun up just fine, and everything I've read so far has been fine, in terms of data integrity.

For current drives though, I'd say "No way." The advances in drive storage size come from pressing more and more data into smaller spaces, meaning magnetic drift in time will affect them much more adversely than even older drives. Smaller and more compact also means the internal mechanisms need to be more precise, narrower tolerances for more points of failure. Older drives were more robust in many ways. 350M SCSI Seagate, read head came off one arm, wires shorting out on the platter. Took it apart, removed the platter, and the damn drive served without flaw for 3 more years in the home server until the box was retired. Try -that- with a drive nowadays...

Rotating backups on tape (with a tape cleaning & replacement schedule), off-location backup rotation, and 'hard medium' backup (CD-R, DVD-R, -not- R/W) of crit. files on a monthly/quarterly basis, and you can be covered for just about anything...

In any case, don't do like us...

[Jugalator]

The computer department I'm part of hasn't taken backups for around a year and we have recently restarted the backup procedures.

Everyone is thinking their projects has been backed up regularly at least once a week or so.

I think we'll just keep quiet about that. :-)

disks as the middle tier

[theonetruekeebler]

In professional settings I have used disks as the middle tier of an n-tier database backup. This was a tremendous help when we needed to minimize backup-imposed down/freeze time, and turned out to be a great fast-restore asset when we were running our primary system in, ah, experimental configurations.

So maybe you could do it like this:

1. on your backup machine, create a single gigantic backup volume
2. do a full backup to it once, and again every week
3. spool the full backup to tape at your leisure
4. do incremental backups to the volume every few hours, or minutes, or whatever fits your schedule and makes you happy
5. spool the incremental backups to tape at your leisure
6. swap tapes offsite as per normal backup methodologies

Now you have superfast backups, superfast recoveries, redundancy, the best of all worlds.

But on the whole IDE drives are commodity-scale junk. If you're using them as your primary, make sure you're doing at leastRAID-5, but more sanely mirrored pairs, and most sanely springing for a DAT drive.

Re:Has DLT tape ever worked consistently?

[geekoid]

SOmething is wrong with your system, or something is happening to the tape. I've done a lot of work with DLT, and your failure rate is way out of proportion.
I would regularly, I mean several time a day, move a tape from system to system for testing purposes.

It could work

[oldstrat]

Your question made me wonder, so...
I pulled out a 386sx board and booted up an old Novell server I had on a 40MB IDE drive from 6 years ago.

No problem, and a scan of the drive indicated it was in good shape.


Of course your mileage may differ

Re:Why would your disks be

[Mysticalfruit]

Obviously you haven't purchased any DLT tapes recently...

Lets just say you go with 40GB DLT tapes...

220/40 = 5.5 DLT tapes to back up your data.

DLT tapes cost 50 bucks a piece. 6 tapes * 50 bucks = 300 bucks just for the tapes.

Oh yeah, now you've gotta buy a DLT drive as well... and if you plan on doing any real backups your not going to sit there and load 6 tapes in succession into the drive so your going to need a library of some kind. So, tack on 5000 bucks for a library... I'll make the assumption that your using a some free archival software, otherwise you'd have to tack on some big money for that as well...

So... 5300 dollar tape solution vs. 500 harddrive solution...

You choose...

Use a dedicated server, not bare drives

[Tassach]

I wouldn't use removable hard drives as a backup mechanism - they are too vulnerable to mechanical damage to be handled that much.

That being said, I think a large pile of IDE drives could be a good backup mechanism, when used as part of a dedicated backup machine. I would take a low-end PC, stick an IDE RAID card in it, and slap in 6 decent IDE drives in a RAID 0+1 configuration (striped mirror pairs). This gives you a large degree of redundancy, and you don't have to worry about interface problems because you are treating the whole system as a black box -- and I doubt ethernet is going away any time soon. A wake-on-lan ethernet card would be a good addition if you don't want to keep it energized 24x7. Stiction shouldn't be a problem since you'll be powering the machine up on a pretty regular basis.

You should be able to build a backup server like this for $1200 or less. Ideally, you'd have 2 of these beasties, one on-site and one off-site. Either swap them out manually or synch over the network, it really dosn't matter. I'd be a whole lot more comfortable moving around a single tower case than a bunch of IDE drive trays.

Try this thing

[phr2]

Niscoa CD Mug. It's a CD autoloader robot that moves CD's (or DVD's) in and out of a regular drive or burner. It's intended for bulk duplication but you can also use it for reading, etc.; and it has enough vertical motion to be able to feed around 5 drives vertically stacked in a tower case. The robot is controlled by a serial port and the protocol is documented on the vendor web site. It should be feasible to run the whole thing from a Perl script under GNU/Linux, or whatever.

It doesn't give random access to the discs, but for backups and archiving you don't really need that. There are also comparable devices from Primera and others.

That said, the DVD Changer thing looks like a pretty good product except for the stupid marketing around it. I'd rather just buy the mechanism with no software and run it myself. Do you think the mechanism itself is crappy, or just the way they sell it?

Re:Try this thing

[JoeShmoe]

I think it's crappy the way they market it as a one of those CD servers that you can get from practically anywhere. It's not a CD server. The best CD server is a hard drive of CD images and either Daemon Tools or Nero DriveImage to simulate a hardware CD-ROM drive.

It's crash-free extremely slow online data storage. Perfect for a home media server, which is I believe where the concept first appeared (said Sony DVD/CD changer).

I don't know how durable or sturdy the mechanism is, but I'd imagine it is comparible to the home theater versions and those have no real trouble. The worst case is that a disc gets jammed and the carousel loses placement, in which case you have to dial it back to 0 and then let it reinitialize itself. Probably the newer ones can do an even better job.

My problem with the DVD Changer thing is they are targetting/limiting themselves to a corporate or business environment...one that is much more likely to go out and purchase a CD server or a tape drive. Since this PowerFile product is somewhere inbetween those two products, I can't see it being a top seller at all.

However, for a home media server and especially for the home user to backup large quantities of data that doesn't ever change, or changes very rarely, it is perfect. Why someone doesn't sell a carousel kit that you add your own drive and logic to is beyond me.

- JoeShmoe

.

Re:Try this thing

[phr2]

There might be some mechanical issues in letting users swap drives in the DVDChanger device, but yeah, I agree, they should just offer a reasonable choice of configurations (fast CD, DVD-R, etc.), don't overcharge for the drives, and publish the interface. Right now they charge a huge premium for the DVD-RAM version and they don't offer DVD-R.

That said, I don't think a carousel or random access loader is that important. For backup, something like a CD duplicator (e.g. the Niscoa or Primera), which works on a stack of media in serial fashion, should be fine. The Primera mechanisms in particular seem a heck of a lot more robust than those home theater mechanisms. They are built for commercial CD duplication on a 24/7 basis.

I called the DVDchanger.com sales number and got a recording saying nobody was available and inviting me to leave a number. I didn't leave one but might call again sometime.

Re:Try this thing

[JoeShmoe]

I think the disc mechanimsm should be handled at a much lower lever than the user would ever touch. This "device" should appear as a single storage medium to the host OS, not as a collection of removable media.

The device should load a directory tree when new media is inserted, so that to the user it is a single "drive" of X capacity (varies depending on the amount on discs inside) that has the following contents. Accessing a directory would cause the device to insert the appropriate disc and send the information. There would be a lag, but no more so than the time it already takes Explorer to refresh when you pop in a new CD.

While the CD feeder is nice for backup, the economics of having a TB of storage for less than $1000 is appealing to me. I would rip every CD and DVD I own and put them all online for myself and my family to access. The originals would go into nice protective boxes in the nice cool. Every TV show I'm ever remotely interested in would be cached and available ondemand (Simpsons marathon night anyone?).

Robert X. Cringley once wrote about what he calls the 10x factor. He said that if a new technology comes out that is 10x better or 10x cheaper than existing capacity, people will stampede to it. The ZIP drive is a perfect example. Relatively inferior, notoriously fussy, bad software support, and ultimately irrelavant, people still stampeded to it because it was at least a 10x jump up from what was commonly available (floppies). When CD burners came out at a much higher price point, people stampeded to them as well.

With average hard drive capacities around 100GB and the largest commmon media around 5GB we are desperate for the next 10x leap. I think this type of device could be it...the other alternative I see is going back to laserdisc (12") size media that would have the capacity to hold probably somewhere around 100GB. One of the two needs to happen, I'm drowning in hard drives.

- JoeShmoe

.

Now I feel scared..

[stevey]

I have a pop3 mailserver which I maintain for our company. I'm planning to switch this to IMAP - having all the users mail on that one box.

THe backup plan goes like this:

• The mail itself on one 120Gb disk [Debian installed on a small 4Gb drive].
• A redundent second 120Gb disk in the same box rsync'd every 30 minutes.
• An rsync job every hour copying to a remote machine.
• The remote machine getting backed up to tape every night.

Is this such a bad idea? Should I not use the 120gb drives for the mailstore in the firstplace?

Some REAL long-term archival media

[phr2]

The next best thing to rock and chisel.

exabyte vxa-2?

[pangloss]

last time i was trying to find a (relatively) inexpensive tape backup solution, i remember that ecrix had some promising looking stuff.

anyway, it appears exabyte acquired them and since i last looked there is now a 2nd generation drive--80GB native. drives are SCSI or firewire, and are ~1000.00. not sure what the tapes cost.

anyone have any experiences to relate regarding these? how these compare to dlt? i see that ibm is selling these as an option to some of their servers now.

Re:the absolute surefire way to back something up.

[BeBoxer]

But is printing a whole character per bit, or even byte, efficient? I'm curious how much data a laser printer could store on a piece of paper. Is it realistic to expect individual bits printed at 300dpi to actually be retrievable? Perhaps on a good 600dpi or 1200dpi printer.

300dpi gives us almost 11KBytes per square inch. Figure 70 square inches on a letter page with 1/2" margins. That's 770KB. Print full duplex and you're looking at 1.5MB per page, or roughly a floppy disk (coincidence?) You wouldn't want to back up your MP3 collection, but for an archival method that is likely to last 100 years it's not too bad. Factor in compression and you are probably getting a 100x increase in storage density over plain text. Kind of a neat thought.

Not that new

[AppyPappy]

In the early 90's we spent $1500 for a 3 gig drive that we used to back up our workstations. We then backed up that drive to tape. It was infinitely faster than screwing with tapes in the night.

Right now I am backing up 53 workstations to a hard drive file using Retrospect. I then copy the file to another server and backup that server. Somewhere, I will have a copy of those backups because it exists on two machines and a tape.

Re:the absolute surefire way to back something up.

[ncc74656]

And just how many tons of paper are you going to need to reliably back up a terabyte in dots and dashes?

If you were actually going to produce some kind of machine-readable dead-tree backup, it's more likely that you'd produce a type of 2D barcode that could be scanned back in and read. Assuming an 8x10" grid at 200 dpi (the remaining area can be used for alignment and checksumming), you could get about 390K per page (single-sided...you could also double that by making it a "flippy," and you wouldn't need a notch-cutter :-) ). You're still looking at a little over 5 tons for 1 TB, but it's an improvement. 200 dpi should be well within the abilities of currently-available laser printers and scanners. If you wanted to try 300 dpi, you'd more than double your capacity and get about 879K per page (single-sided).

The cheapest, and most long lasting backup.

[teamhasnoi]

The oral tradition! Have many children, give them each 10 pages to memorize. To make things easier, you can name them Sector 237, Cylinder 13004 and such.

As disk space grows, so does your family/backup.

To see examples of how this works see: Mad Max - Thunderdome, The Bible, American Indians, The Fellowship of the Ring, Aesops Fables, and the Legend of How the Great Nog Vomited the Earth and Heavens in Ancient Times, Before the Oceans Drank Atlantis.

I have heard rumors that this is how Google archives.

Re:failures

[tomhudson]

1. Unfortunately, most hard disks made today fail in less than 5 years. See the recent flame wars over the manufacturers' reduction in the warranty period to 1 year from 3.
2. Of course the head isn't in contact with the media, but the wear on the pivot bearing is what causes a lot of drive failures. One day, you hear the drive suddenly timing out, then trying to re-position the head on track 0, wait a few seconds, and try again ... your pivot bearing is gone! The spindle bearings are still ok - they have only one, constant, force to contend with, whereas the pivot/actuator is subjected to rapidly-changing forces as it steps back and forth across the tracks.

But I think most /. readers would agree with you, most hard drives should last through 5 years, but most don't/won't/can't be depended on, certainly not for archival purposes, where it has to work.

Archival vs. Backup

[ToasterTester]

Most the people I talk to, and reading many of these replies don't understand the difference between archiving and backup. Before hard drive prices became so cheap both archive and backup were handled the same way, but now you have look at the difference.

Archiving data is usually for business reasons, usually legal like for IRS. So this is data that needs to be able to be restored at some time in the future. For IRS I believe you have to keep business records for ten years minimum. Last place I was at tapes were on a one year rotation cycle, and we made a yearly archive set never touched.

Now backup is usually for safety reasons. So the dumb ass marketing jerk who wipes out the new ad copy due to the printer tomorrow can be saved. This is usually for short term and hard drives doing snapshots are good for this.

I worked for one the major ISP's and email is all stored on NAS storage and they keep about five days worth of snapshots to do restore a mailbox if necessary. Same with web content. This stuff to too dynamic to waste time and money doing tape backup. Tape is only done on business data.

So today to save money and backup-window time you need to design storage systems dividing data in to what needs to be archived vs. backed up.

Also when designing new archive systems don't forget to plan to archive your old backup server and tape library. What good are tapes if you don't have the software and hardware that can read them to do restores.

Re:Sod CD-R! Go With DVD recording

[ibennetch]

And for keeping tabs on what is on which disk... I've been using a freeware program called "Cathy" (I don't have any links)...Although I don't know whether it'll do DVD's, I haven't tried.

Cathy is avalible for download here. According to these sites it will handle many disk formats ("CD-ROMs, LS120, Iomega Zip and Jaz disks, or even diskettes"). The link to the home page is broken.

Tapes are expensive and unreliable

[ozzee]

Most of the storage technology R&D money is going into hard drives which is why they are so inexpensive and will continue to be. Tape storage technology R&D money is simply unable to compete.

I've predicted for a long time that tapes will become obsolete and hard drives will be used to back up hard drives. A very interesting example are firewire hard drives. A small premium on the cost of an already cheap dist drive, you have a fully plug and play high performance solution. For the cost of a tape drive and 10 tapes you can buy 10 firewire hard drives and rotate the drives instead of tapes.

The advantages are enormous.

Fast recovery time

Fast seek time

Proven reliable technology and much more reliable that tape.

Inexpensive and becoming less so

Easily networkable (just mount the drive on the network) and fully supported by any self-respecting OS

When you're done with the backup, you have some spare storage for anyone..

There may be some work to do to make some of the backup software talk to a hard drive, but there are probably so many different solutions you can use, you probably don't need to worry.

I bought 2 firewire drives a while ago and have never looked at tapes since. Love it....

It would be -

[kfg]

The difference between an asshole and a troll is that an asshole is being, in some way, well, an asshole. A jerk. Being an asshole means that you have done something negative that * didn't need to be done that way.* Like swearing a lot making a point which didn't require the swearing for the point to be made. Your content was valid, even perceptive. But you were an asshole in the way you said it. You were *gratuitously* abusive.

The original poster who wanted to mod you troll was at least right in that respect in respect that some trolls should moddable UP though.

Some trolls are art and deserve to be respected and recognized as such. Right now we can only use funny for that, which isn't always accurate. A true troll has a *point.* To elicit response. Sometimes that response is baited in a way that isn't just art, it's *high* art.

Read the letters and short essays of Mark Twain. Many of them some of the finest trolls every penned.

Swift was a master of the troll. All of Gulliver's Travels is nothing but one massive, and brilliant, troll, and was treated as such at the time.

By the way, the difference between a troll and a flamebait is that the flamebait is a post specifically designed to illicit responses from assholes.

KFG

Re:the absolute surefire way to back something up.

[2nd Post!]

But each of your 20k per page can easily encode a unicode value, which means you can cram 2 bytes per spot, or only 50 tons per terabyte.

But how about a 600dpi laser printer, 8"x10"?

For good readability, we can use:

***
**
*
*
**
***

For (1,0) which gives us 3 dots per bit, or 200 bits per inch. A square inch would then give us 40,000 bits, or 5,000 bytes. A sheet of 8x10 then gives us 400,000 bytes. Or if you tweak the margins, 400k per page. So that's already 20 times your density. Increase the resolution to 1200dpi, and you can increase the data density to 1600k per page.

We can also use different encodings: Right now we use 9 bits to encode 1 bit of information (really, really, redundant). We can probably safely use the following encoding to double our data density:

***

***

*
*
*
*
*
*

So this further gives us 2 bits of information in the same 3x3 square, which increases our data density another 2fold: 800k or 3200k per page. At 1200dpi, that's 3mb per page, so that 1gb == 333 pages, and 1tb == 333k pages. 67 boxes, or 134 pounds per terabyte.

There are more variations of course. We can increase density to 4 bits per 3x3 square. With a bit of thought, we can also increase the density up to the theoretical limit of 2^9 values in a 3x3 square, but we want to include some leeway for data redundancy...

So by doubling to 4 bits per square, we require only 70 pounds per terabyte. By doubling again to 8 bits per square, That's down to 35 pounds.

That much (little) paper... is actually lighter than a terrabyte of digital storage!

Re:the absolute surefire way to back something up.

[tomhudson]

Unfortunately, the so-called "archival" papers, while "rated" for 100 years, won't last anywhere near that long without some degradation. Then, if you're going to store it that densely, you've got to make allowance for putting the data into "tracks", so you have to leave spaces between each row. Cuts your 300 dpi down to, say, 100. Add check-summing data, so that you can recover from dirt, toner falling in the cracks, etc. And now, let's make the dashes twice the size of the dots. Cuts your storage by another 50%. Now, let's put spaces between the dots and dashes - otherwise, you get one LOOOONG dash. Your 11kb per square inch is now less than 0.5kb per square inch. Oh, and don't do duplex printing, you'll have transfer of toner onto the drum from the previously-printed side. Net result == about 30kb to 50kb per page... Oh well, maybe we should try microfiche ... or bit-encode the data into fake avi files and record them on VCR tape - cheap media for sure.

Re:the absolute surefire way to back something up.

[schmink182]

To take this a little farther, a helpful reference tells us some useful information.

2000 sheets of 8-1/2 x 11, 20# laserwriter paper weighs 20 lbs.
First of all, this changes your estimate of weight from 100 tons to 250 tons.

Typical yield of paper: 125 lbs per tree
250 tons (500000 lbs) divided by 125 lbs per tree gives us 4000 trees.

440 trees per acre
This, after division, gives us 9 acres of trees destroyed for backing up 1 TB of data. Seem worth it? :)

Re:failures

[ncc74656]

My suggestion: check out the usb removable drive trays. This way, you can hot swap/back up w/o powering down.

FireWire would be better in this capacity...it's faster, it doesn't bog down your computer, and the controllers only run about $40 or so (though if all backups will be done through one computer, you can get by with just one card). USB's OK for low-to-medium-speed devices, but it's not up to handling high-speed devices such as hard drives.

Re:Sod CD-R! Go With DVD recording

[jhawkins]

I just downloaded Cathy from http://rvas.webzdarma.cz/, the developer's page. The latest version is only a few days old. Looks like a nice simple program (Windows only). It's a 53 KB exe file, no installer, no frills. I've been looking intermittently for this type of program for a while.

Re:the absolute surefire way to back something up.

[tomhudson]

Ouch! Even worse! :-)

Samsung offers three year warranty standard

[Splork]

Samsung still offers a three year warranty standard with their drives.

In SOVIET RUSSIA...

[ralmeida]

...the Beowulf cluster imagines you!

Isn't that what Google is for?

[Pedrito]

Put it on a web page, make sure Google caches it, and voila, a back-up you don't have to maintain and it'll probably last longer than a tape drive or any given hard drive. Also slightly faster than the hammer and chisel method suggested by others.

Truth be told, I've thought and thought and thought about how to back up data reliably over long term and I have yet to find a good solution. I used to think CDs were great until I moved to Mexico and had my CDs eaten by a fungus. Now I don't trust them at all.

I also encountered a mold that attacked VHS tapes and I'm sure it attacks other types of tapes, so they're of no use.

Short of storing your media in a vacuum, I have no idea how you store stuff long-term without risk of loss.

Re:Tapes are a expensive waste of time

[fishbowl]

"I could never understand why zip drives became so popular when the disks cost a small fortune"

At first, we thought we finally were seeing a replacement for the floppy; something fast enough, with a decent amount of storage. It would have been great, provided that the media had also replaced the floppy disk in terms of price. I still think a fast, cheap, 10meg floppy would have been nice. Most of the reasons I wanted that are taken care of nowadays by the fact that "everybody" has the ability to get whatever file via http, so, the pattern of handing a floppy to someone is a bit antiquated. Still, I'd never have given a $15.00 zip disk to someone with the abandon that I would a CDR. And now I can get CDR's that will fit in my shirt pocket, I can make them bootable, so the floppy is just about obsolete :-)

After a year or two, we needed to start seeing ZIP drives from all the drive vendors, and, we needed to be able to buy the media for, say, $1.00 a piece in bulk. Neither of these happened; Iomega's drives were often defective, and the price, convenience, and performance did not add up. We're still using floppies. Some people use CDRW. Most "consumers" don't "backup" their systems. When they have a crash, that's when they get their new systems. Clean slate; usually some tears shed and bitching and moaning about "important" data, but, it's really the exception when a home computer user has data on their disk that, if they lost it, they go bankrupt or go to jail.
At worst, it's inconvenient. And the ones who DO have important data tend to be aware of the risks, and tend to have a hardcopy backup.

I find the more important a document or record is, the more likely it will fit on a floppy disk and on paper in a legal-size file folder.

As for tapes, they still fill a need in the office/datacenter.

This sort of thing:
http://h18006.www1.hp.com/products/storage works/es l9595sl/description.html#qs

Re:the absolute surefire way to back something up.

[Maxwell_E]

Why not just use punch tape? Or better yet punch cards! Hey, it'd give the geezers a chance to reminisce... Umm, just don't have it punched in Florida.

Re:disks

[zogger]

--ok, this makes sense. Hmm, seems like a nice business though if there was a machine that was affordable by at least some of the more upscale white box shops where they could do this as a service, perhaps one of those boxes with the integrated gloves. You put the hard drive in there, evacuate all the air-and dust presumably then- replace the platters, then re seal it back up. Just a thought. It sounds like a decent idea if those engineering details could be solved somewhat cheaply. I honestly don't know how the various data recovery outfits do this currently, maybe they already do similar? I had a neighbor try to get a price on recovering some financial data from a failed drive and it was way expensive they were telling me, so expensive they didn't do it but I have forgotten what they said they were quoted for the attempt.

long distance backup

[HeyBob!]

Read about this years ago: Send a laser transceiver way out in space and have another in earth orbit (always able to see the far satellite). Encode your data in a continuous stream to the far end, and it sends it back, which is then sent out again. The further the distance, the more you can store in the stream. Although you have to wait until it comes around again if you want to access or change it. You can use radio instead of lasers and you might want a bunch of them in case of a failure at either end.

DVD-R

[aminorex]

I'd rather change out 50 blank plastic disks once
than endure the need to redundantly and repeatedly
spin up multiple copies of a single 200GB IDE drive.

I do hope you've optimized the *algorithm* first.
Archiving deltas, etc.

Another approach is to just keep the backup always
hot, and geographically distributed, e.g. using
WebRAID.

Long-term backups

[Pedrito]

Wow, how appropriate that this came up today. I have a directory called "Utils" that I keep a bunch of dos programs that I've hauled around from computer to computer, job to job, for quite some time. I was going through it today looking for a program and started noticing dates on some of these. I've managed to haul this directory around since 1986!!

My oldest program is from Sept. 9, 1986 - DIRERASE.EXE
I also have a DOS editor that a buddy of mine wrote and I still use called, simple "E.EXE". 3/21/1987.
I have a dos de-tar (as in the unix tar utility) program from 1988. And, anyone remember this: Spinrite.com. It allows you to adjust the interleave on your hard drive. Well, it doesn't seem to work too well under Windows 2000 :-(

Funny that I just noticed all this today.

Re:the absolute surefire way to back something up.

[Jucius Maximus]

"As long as we are on that track, the Internet was designed to withstand nuclear attack, so its obviously the best choice: archive, encrypt and have others mirror your data."

Just encode your data into a pr0n video and share it on gnutella. That data will never be 'lost' !

Re:Sod CD-R! Go With DVD recording

[Mr.Intel]

Here is his new homepage and link to the latest version of Cathy.

Enjoy.

disk duplication and offsite backup

[MikeFM]

I have a terabyte plus of data at home. My usual backup method is to have a backup server with enough disk space to mirror all my data and using frequent cron jobs to keep the data on the backup server synced with the machines it backs up. Critical files I also back up online and now and then to external media (dvd-r's kept in a safe at a different location). If you have a safe room in your building put your backup server in there. A room safe from theft, fire, flood, and tornado is an ideal location. Offsite secure locations are a must.

Any storage medium will die eventually. Count on it. I'd suggest keeping a live copy of the data that you can verify is correct using checksums on a backup server and making new backups every so often. If you don't recreate known-good copies of the data on a regular basis eventually that data will just degrade.

Re:Tapes are a expensive waste of time

[JWSmythe]

Burnt CD's (like you'd use at home) have a shelf-life of about 10 years. Then the medium starts to oxidize (the metallic film, not the plastic itself), and flakes..

So, you have a 10 year backup.. It all depends on how important your information is. If it's that important, I'd put it on a RAID5 where it can be monitored. As drives fail, replace them. Continue migrating to newer arrays in the future.. Expensive, but I konw perfectly well any drive will fail. I've had several hard drives, that would fail to spin up properly after sitting for a few days.. Some of them, they only way they'd start is if I hit the side of the drive with a screwdriver..

You have to expect failure of your medium. If he wants to be very sure, use multiple backup methods.. RAID5's in multiple locations, and CD's. Someone will need to monitor all of it occasionally. Make sure the RAID's (and their associated machine) are running. Make sure the CD"s are oxodizing...

Even floppy disks die of old age. I found a few boxes with Novell Unix. They're is years old, and most of the floppies couldn't be read. They were brand new, still in the sealed boxes and envelopes. I finally found a boot disk that would work, but it would bomb out trying to install under VMWare (I was curious).

Is that data really going to be useful to you in 10 years? That's the important question. People are all paranoid of loosing Email and the like now, but in 1 year they don't care about it any more. In 2 years, it's just wasted space. In 10 years, they won't even know who or what they were talking about..

Micropolis

[Mu*puppy]

Well, according to this, their parent company Singapore Technologies filed bankruptcy in late '97, and rather then try reorganizing under Chap. 11, they just liquidated the company..

It totally depends...

[shylock0]

IDE backups might work for short term archival; particulary if you backing up daily work and overwriting the next day. Case in point: one of the companies I've done some consluting for needed a way to back up their 10 gig accounting database each night. All they really wanted protection against was a fire; the accouting database was pretty much the cornerstone on which this company (a wholesaler) ran. The solution was simple: buy an external 20 gig firewire drive, back up entire data drive from server, place drive in fire safe.

However, for long-term archival backups, IDE might not be the best idea. Drives do tend to get corrupted, and if you're not careful about letting them spin down completely each time before you remove them, or they get a little too much shock in transit, you could lose all your data. If you're looking for 1-2 year archival, IDE should probably work, but not much longer than that.

The next option would be tape backups. If you have the money, I would spring for one -- an autoloader if you can -- mainly because it will be more reliable than IDE. Recent experience suggests that tape media (at least the old TRAVAN kind, and some older 8mm DAT) has a shelf life of about 4 years after you write it. I recently tried to recover, for a client, some monthly non-incremental backups for the period 1995-1998 (they are the subject of an IRS audit). The tapes were a mix of older tape (TRAVAN and compat. earlier standards) and DAT media, depending on which of two servers they came from (and how old). Only one (a DAT) of the 24 tapes from 1995 (12 DAT, 12 TRAVAN) gave us 100% of the backed up data. 20 gave us partial, and 3 were completely unusable. The 96 tapes were a little better (12 complete data, 1 unreadable, 11 partial). 97 saw 20 complete, 4 partial; in 1998 all the data was fine. After 1998, the company switched completely to DAT; all the tapes later than '98 worked fine. So DAT isn't the best long-term storage medium. It is also worth noting that the tapes weren't kept in climate-controlled conditions, but instead in an shielded box in a cabinet in a manager's office. The office was air-conditioned, and the temp rarely got above 74, but company is in NYC and it can get pretty humid in the summertime. From talking with some colleagues, don't expect more than 5 years from tape unless you've got it in a climate-controlled environment.

One solution, however, is to backup to tape and then restore and backup to new tape once a year. After their debacle with the IRS, the aforementioned company is going to start doing that in the future.

Another good question is, what are you backing up? If its documents, and you're looking for long-term storage, the best solution is to print the documents out on acid-free paper, put them all in a box, and archive them at a storage facility. Although with 220 gigs of data it sounds like you've got quite a bit more than just documents...

Re:It totally depends...

[Junta]

A firewire drive in a fireproof safe, shouldn't they cancel each other out?

But seriously, putting tapes, drives, or even optical media in a fireproof safe I would not classify as being safe from fire, unless you have a really kick ass fireproof safe that *also* insures the heat level doesn't get high enough to cause problems. Just keeping the temperature below the combustion point of most materials isn't enough.....

Re:the absolute surefire way to back something up.

[default luser]

And if you're really clever, you would take advantage of the fact that levels of greyscale are easily discernable. Leave a seperation space on all sides of each dot ( so they're more easily decoded ) to form a grid system. Yes, your storage capacity will drop by a factor of 4, but you can easily encode 8 bits ( a factor of 256 ) into the dot.

Most laserprinters can do 8-bit greyscale.

But for redundancy:

- Make two dots for each 8-bit piece of data, the 8-bits and it's complement. This is only good at error detection, although theoretically you could add error correction at a capacity cost.

- Add 256 calibration dots every few inches to make up for aging of the ink and media. We can assume that the cameras will have much higher resolution than the printer, so they can tell the difference even if the levels have faded together.

You could pack a whole lot of data on paper if you put your mind to it.

Use the fact that it is digital to your advantage.

[Kjella]

A copy of a copy of a copy of a copy is... the original! Sync it to as many harddrives as you need to have at least one working, and preferably online/availible so you'll know of failures (scripted CRC self-verification for instance). RAID 5 (checksums), RAID 1 (mirroring), multiple backup/offsite machines if you need to. If it's important enough I'd go for security in numbers and replace as they fail. Will it have decades of lifetime? No. But unless you let it decay to the point where all copies are lost, you'll still have your data.

Kjella

My experience with long-term IDE

[inode_buddha]

Point 1.
Make sure you select a very well-made drive, don't cut costs there. Example: I have a 20-year old Mountain HardCard that still works fine. However, I have had cheap 3-year old drives fail.

Bringing up point 2:
If you try it, make sure to use an "exercise" schedule for all the drives in your backup set. For example, once a week for each drive, plug it into a spare box and ensure that it spins up, spins down, and the read/write arm travels its full sweep. Maybe do some read/writes at various places on the platter surfaces, just to be sure.

It works for me, so I hope this helps.

tape is unreliable

[g4dget]

With tape, the failure of a tape drive doesn't separate your from your data (unless it catches on fire with the tape in it or something.) You can just get a new tape drive and you are good to go again.

Not really. If you just keep tape sitting on the shelf, magnetization from one layer will transfer to the next and the tape will become unusable over time. The oxide will also start flaking off after a while and the carrier will become brittle.

While you can be lucky and read tape after a few decades, you can't really rely on it for more than a few years.

Re:tape is unreliable

[fw3]

magnetization from one layer will transfer to the next .... oxide will also start flaking off after a while and the carrier will become brittle.

Write-thru is only a problem with quik-format / fixed head tapes. These became substantially unreliable as tape media became stronger (using PET/mylar backing) allowing the distance from one oxide layer to the next to become smaller.

DAT/dds, 8mm and other current tape media use a far thinner oxide layer allowing higher density at the price of lower gauss values. They work because the helical scan / rotating head increases the velocity sufficiently to allow use of the weaker signal.

Lower gauss values also effectively eliminates write-thru. I've read (many) decade-old 8mm tapes and in a decade of using this medea I've only seen a couple of tape failures.

Re:the absolute surefire way to back something up.

[n9hmg]

I still think somebody will make glass MO archive media, with gold as the reflective surface, but if you're going to use paper, use 2d barcodes... about 1.1K/in^^2, for around 9.5K/side.
Oh, and to be sort of on-topic for the actual story, My friends at Seagate say that modern drives should start up fine after many years proper storage. I still don't trust them (the drives, not the friends).

The Answer to All Your Problems

[jonbrewer]

StorageTek, the company that makes those nifty tape-backup robots, has a new product based on IDE RAID called "BladeStore" that should suit your needs.

My backup method.

[gid]

I use a mirror hard drive to backup my data, which I do on a weekly basis. If either drive goes bad, then hopefully I'll know about it. If the data is more was important like as in your case, then I'd either make my backup machine either redundant raid, or have two seperate machines that do the backing up. The idea is that both dataset won't go bad at the same time. :) And since both are in kind of active use, I'll know when pretty soon one goes bad.

I used to have all my data on a single unbackuped up 80 gig hd. A while back on slashdot there was an article on incremental backups via rsync. So I looked at it's feasiblity for a bit and I ended up buying two 120 gig hds, one drive resides in my linux workstation/server that I'm using now, it holds the live, in use copy of my data. And the other drive is in a dual boot xp/linux machine that sits right next to it. It's usually in xp when I want to play games, but once I week I fire the machine up to backup my data.

All in all I'm backing up around 90 gig to a 120 gig drive, plus I keep one previous copy of the data on the drive. I also use the same machine and hd to back up a server I have in maryland. I keep 3 incremental copies of this data since it's smaller. I'm counting on most of the files not to change, or else I wouldn't be able to store all of this on a single drive.

While you won't be able to back up 220 gigs of data with a single hard drive, you could easily have a mirror machine with software raid in linux that runs rsync on data.

You also wouldn't want to have the machine sittings right next to each other obviously. :) For my purposes, having both machines next to each other is pefectly acceptable risk, besides, it's MY data. If the room catches on fire, the data is toast, along with the rest of my belongings. If there's a huge electrical surge, I'm not sure what's gonna happen, both machines are plugged into a UPS. I'll just pray to God in this case.

Anyway, here's the scripts that I ended up using, I used scripts from the previously mentioned slashdot article as a guidline.

http://pimpbot.qooqle.com/gid/backup/

pimpbot.sh is the actual script that invokes rsync, then we have the exclude file, plus there's backup.sh which get executed nightly on the server that's being backuped up, it backups up the database, and other random files that's more machine specific.

live disks or optical

[g4dget]

Live, spinning RAIDs are good. Their individual drives will fail after a few years, but you just keep replacing them. As long as you are reasonably vigilant, your risk of total data loss is low.

If you want a medium you can keep on a shelf, properly stored optical disks or optical tapes are probably your best bet. While nobody knows for certain, they are much less likely to show the kinds of degradations you get with purely magnetic media. You can buy special archival CD-Rs (and, presumably, DVD-Rs) that should last for decades.

Magnetic tape is probably the worst long-term backup medium. While you can get lucky and it can last a few decades, data on it often becomes unreadable after a few years through a variety of mechanisms.

Check out this kodak link

[rebelcool]

Kodak's been working on this problem for quite some time.
Endura paper

When properly stored, this paper will suffer no degradation for 200 years. And that's even with complex color photo development. Something like storing binary data or barcodes should be readable on such paper for much longer than that.

Re:Check out this kodak link

[tomhudson]

Because thermal stability - or album keeping - is so important to the people buying professional photographs, testing and reporting on light fade only would be relevant only to consumers who store their prints in a lighted freezer.</quote>

It's nice to see material that is both honest and humorous. Too many of the "high-quality" papers aren't, and use the same "quality" testing assumptions that give us 200,000 hour MTBFs for hard disks that fail uniformly after less than 10% of that.

Mind you, this just begs the question - does the light stay on when you close the door? :-)

Re:the absolute surefire way to back something up.

[krel]

there's always the back of the paper
maybe you could write checksums on the backs

Tower of Babel here we come

[xipho]

Not to be a naysayer...but I will anyways. What happens in 30 years when a massive electromagnetic field wipes out all digital machines (possibly in conjuction with some attempt by humans to wipe out the robots taking over the world...those damn robots!)? By then 15 years of scientific publication may be more or less completely digital, and all gone, gone. Better hope we never lose access to that handy-dandy resource electricity....

My Backup Solution

[ReelOddeeo]

Get one of those USB 2.0 external drive enclosures with an open front, into which you can insert an IDE drive. (This is basically an external USB 2.0 drive using an IDE drive, but WITHOUT the drive.) Use those removable drive cages that let you remove an IDE drive through the front panel.

Cost of external USB 2.0 case: <$100.
Cost per backup media...
Drive: $80
Removable Cage: $20


Spending $100 per removable backup media might seem a lot, until you compare with high end tape. Plus you don't need a $1000 or more tape drive.

Advantages? Backup is fast. Restore is great -- just mount the drive read only, and interactively hunt for the file you want and drag it back to your drive.

Each backup consists of a bare drive with those removable rails attached to its side.

You, and the first poster. . .

[kfg]

have managed the fine art of being both funny and insightful at the same time. My congratulations to you sirs.

I have in my house, not more than a couple of feet from me right now, these things called "shelves." On these shelves are these things called "books." A few of these books were written by people such as Einstein and Eddington and printed going on 100 years ago. In fact, I have a few other books nearly twice that old. The data on all of these books is still completely "readable."

What's more, I have every expectation that they will continue to be readable by my children's children and *their* children's children. Amazing, isn't it? I can pass a magnet over these books, I can douse them with water, I can infest them with mold and mildew and the books themselves may be destroyed as integral objects but the *data* will still be recoverable. They also have no EULA attached, and although I can't copy them and sell the *copies* if they are below a certain age, I can treat the physical object as my property.

What I can't do is burn them. If my house goes up in flames so do my books. They arn't perfect, but they won't just "fade away" or go "poof" in the middle of the night for no apparent reason.

All electronic and magnetic media are inherently highly volatile. That includes tape.

However, I do have, sitting right here on my desk next to my shelves, machines that can copy these books to volatile electronic and magnetic forms and *back again* to print or nonvolatile digital form. In fact, if the book is old enough, I often find that the work of converting them to magnetic and electronic media has already been done for me.( if we all did just ONE unique book that would be millions of books commonly availble in electronic form. Think about. Just one. Maybe even a short one)

In an odd twist of fate the old fiction of the Xerox commercial comes true. The best "backup device" known to man is a monk with a copy machine.

I once got modded as flamebait ( and I honestly don't know why, troll maybe, but not flamebait) for pointing out that the best PDA I knew of, and the one I personally used, was a pocket sized spiral bound notebook I got from CVS for $.69. It's cheap, nonvolatile, easily replacable anywhere in the world, has an infinite "battery" life, is fully pen compliant, can go through a metal detector without a qualm, can transfer its data to *any* computer in any data format and I've even run it through the wash and still recovered it's data. Hell, I can throw the thing against the wall as hard as I can as often as I want and the damned thing doesn't even get *scrathed* let alone lose data. I love the thing.

Pen and paper is still the optimum solution to many problems, except perhaps having a fun new toy to play with.

On my shelves, since I once worked a few days with an archeological doctoral student in Mexico, are a few bits of stone and clay with hyrogliphics on them several hundred years old. These bits of meaningful earth have been buried, rained on, trodden on, smashed, earthquaked dozens of times, burned and god knows what else, and their data is still recoverable.

Let's see your tape drive do that. I'm serious. There's a lesson to learned here.

Here's the best way to permenantly backup long term digital data. Take a giant titanium platter 5mm thick and punch 5mm holes in it for each "on" bit. On the first "track" of the platter make sure to make a code key easily decipherable by anybody with any sort of mathmatical knowledge.

There ya go. A backup that will last longer than the pyramids stand. You could even shoot it off into space, unprotected, with reasonable assurance that it would survive the trip until "someone" picked it up.

Ok, so that's not very reasonable for the average Joe who just wants to protect their mp3 collection. What has any of this got to do with you?

Plenty actually. In the first place, don't store anything in a computer format that can be more effectively stored in nonelectronic form. Except perhaps as a backup of *that* media. Books are a prime example. If your computer goes down and you need to refer to a manual to get it back up again that manual is worthless if *it's on the computer.* ( And did I mention that books have no EULA? That's a *feature*)

That isn't to say that you shouldn't have ebooks and CD based manuals, but that those are convieniences and backups. The *books* themselves are the primary data.

Don't rely on your PDA to keep critical phone numbers and such. You WILL drop it. It WILL get stolen, or have its batteries run out, or go through a *hidden* metal detector.

*Write them down* and carry the written copy. If anything happens to your PDA you're still good to go, and restore the data to your PDA/computer when time permits, like when you're just bouncing around your Days Inn room wondering what the hell there is to do in this God forsaken town anyway.

What to do about all that data that is inherently digital, or at least more viable in digital form?

Did you ever hear the phrase " The net is the computer"? Just for this once take it seriously. I've been a strong opponent of storing data on the net. Drives are big. That's why we're having this discussion in the first place, isn't it? They hold a LOT of data. Why put your sensitive personal data, data that, perhaps, could someday cause you considerable grief if it fell into the "wrong" hands ( and this doens't *necessarily* mean the FBI. Ever get divorced? Trust me, you don't want some things to be recoverable from an outside party) on the net? It makes no sense. The net is a *party line* capable of permenantly recording everything that goes across it. Certain documents belong ONLY on paper, or your own *secure* machine, or in your lawyers safe. Period. Please, take this advice very, VERY seriously. You may well have reason to thank me for it some day no matter how doofey you think it is now. .NET? I'll tell you what you can do with your .NET buddy. My OS and *all it's apps* take up less than one percent of of my HD space. HD's are getting bigger in quantum leaps. Spreadsheet programs are getting bigger in tiny, tiny little increments that are slowly getting tinier. I'm not going to run my apps across the net. My data stays home. The only cache of my data is on *my* drive, which I can physically destroy if need be.

Do likewise, and it wouldn't hurt to encryt your whole bloody drive either. Just in case.

But all of that being said, do I *really* need a local backup of my files downloaded from the Gutenberg Project? They're already out there on the net. They're mirrored all the hell over the place. Someone else is already taking the trouble to swap drives, swap tapes and transfer the files to up to date media. It only takes a few minutes to download the whole bloody lot. If you lose your net connection, even permenantly, you know 20 other people who can grab it for you, and a half dozen locations where you can use a public computer to get the stuff. Even if all the mirrors to down at once, and *stay* down, you can always go on usenet and ask around. You'll find what you're looking for.

For this kind of noncritical data plastered widely across the net that you can, in all probability, recover faster from the net than from local backup ( remember, it takes *time* to find physical media, load it up, find the files you're looking for, and transfer them), simply *don't bother.*

It isn't worth it. Even if it takes slightly *longer* to do it this way than from local backup. The time you save in making the backups and keeping them up to date more than makes up for any trouble.

And remember, this sort of full backup is *disaster* recovery. Like if your house burns down/your entire system gets infected with a virus/your HD crashes/your dog eats your homework. In this sort of situation the small amount of time it takes to recover from the net is peanuts compared to the other things you're going waste time and money on. So don't sweat it.

If you take care of your system properly most of these things aren't even likely to occur,except the new HD that goes down after only one week. For the most part a *full* backup is mental insurance that your data can be recovered * somehow.* If it's on the net, and it's something like the Gutenberg Project, it's going to be there. Relax.

So, what about your mp3 collection? That took you years to build, and would take you weeks to rebuild, and maybe the RIAA will win out and the stuff will become unavailable. Well, for these you want to back them up to CD anyway, so you can play them in alternative devices, make copies for your friends, etc., right? Well there ya go. Don't bother putting this stuff on tape or anything. You've got it already. Make two copies, give one to a friend. That friend now has copies to play, and you've got off site backup. Have the friend do the same thing with HIS collection and you're BOTH protected. Well looky here, in certain cases and situations socialism DOES work ( much to the RIAA's distress).

The same goes for image files. It took you years to build up the pr0n collection. Some of it from pay sources that have gone *poof*, or that the FBI closed down. All of it from all over the place in any case. It would take you weeks to rebuild it, if you could do it at all. So sure, back this stuff local, but again, do it to do some form of digital disc ( CD or DVD). Find a friend with the same taste in pr0n and do the same as you did with the mp3's.

Repeat if you have video material.

Now take all of your system discs. OS stuff, Quake, Photoshop, etc., and copy them. Store the *original* offsite, maybe with a friend ( you do *have* friends don't you? I forgot to ask. A lawyer's safe or safe deposit box can substitute) or family member. Don't forget to write any applicable install codes *directly on* the copy.

Now if the worst comes to worst you can simply reinstall everything. You might even find that half of this shit you never even *bother* reinstalling. Go figure.

So now what are you left with? Surprisingly little in all likelihood. You're going to want to back up all of your system settings. Sure, it won't take you that long to reinstall the system itself, but the settings took you ages to do and you don't even remember them all. Back 'em up. Obvious data like your resume and thesis, software projects, financial data, Quake player data, etc..

Ok, what's left? For the average home user, and even the not so average, you may be surprised to discover you have now * done it all.*

Wait a minute, what happened to the traditional full backup?

Well, the thing you have to remember is that a *home* computer and *business* computer are inherently different when it comes to backup. When a business computer goes down it has to go back up NOW! And it has to go back up *exactly* as it was when it went down. This need makes all of those tapes backing up everying in full, right down to the most common thing they could just grab from the web, absolutely and completely necessary.

A *home* computer isn't like that. Your house has just burned down or your HD has crashed and you need to be sure you can recover everything, or even only *most* of it, within a reasonable time frame at minimal trouble and *expense.* Tapes, tape drives, and full backups all the time are, in and of themselves, trouble and expense. All against a future disaster that might well never come. You don't need to do it NOW. You just need to be able to rest assured that you *can* do it.

To hell with the full backup. Reinstall your OS and apps and restore just your settings. Sure, it takes a little longer, but that time goes against all that tape swapping and storing. You might actually come out ahead.

Now just download the stuff you can as time permits or as you need them. Get your stuff back from your friend if your local backups went up in smoke with your house ( you can live without Houses of the Holy for two days, honest, and even the jpeg of Natalie Portman's head Photoshopped onto Pamela Anderson's naked body) and you're back in business.

Now you're just left with incremental backups. For some people this might have to be done every day. For some they can legitimately wonder if every month is really just being paranoid.

Ok, so you really, really, REALLY just don't feel comfortable without a full backup? Maybe you're just the sort who really thinks he *can't* live without Houses of the Holy for even one extra minute he doesn't have to?

Ok, NOW the idea of mirroring drives for a backup makes sense because you aren't relying on it as your *primary* means of backup. You're only going to use it if your house burns down *tomorrow*, not in five years and in such a case it could save you some time. Hey, if it rows your boat, go for it Sparky.

I might even end up doing it that way myself.

KFG

Ten year old data

[Eric Green]

I actually have a lot of data that is now 16 years old, including the source code (6502 assembly language) for a BBS program that I wrote as a kid. The secret: Regular migration of data to newer/larger media. From 1541 floppy to Amiga via serial port and xmodem, from Amiga to Linux via serial port and uucp, and on Linux, periodic moving of the data to newer hard drives as I upgrade my systems. I also now maintain a copy of my data in CVS, so that if something gets accidentally erased or changed, I can retrieve a copy. My CVS archive, too, periodically gets moved to newer/larger/faster hard drives.

And to top it all off, I back it all up to a DDS-4 DAT autochanger. Yes, those six tapes will only hold 120gb, but the amount of important data on my disk drive is far less than 120gb (it is actually less than 20gb, including the original 44.1khz .wav recordings of all my original songs, and fits onto one tape easily).

Do you *REALLY* need a backup of your .mp3 collection?! Probably not. Do you *REALLY* need a backup of all those ISO CDROM images that you downloaded for fifty versions of Linux and a half dozen versions of FreeBSD? Probably not. But that's the sorts of things that are taking up 80gb plus on my hard drives -- i.e., utterly disposable cruft. Which is true for most personal computers.

Re:Ten year old data

[marauder404]

Do you *REALLY* need a backup of your .mp3 collection?! Probably not. Do you *REALLY* need a backup of all those ISO CDROM images that you downloaded for fifty versions of Linux and a half dozen versions of FreeBSD? Probably not. But that's the sorts of things that are taking up 80gb plus on my hard drives -- i.e., utterly disposable cruft. Which is true for most personal computers.

Absolutely true. Most of my personal data can fit on a CD or two at the most.

Except for digital photographs. With a 3 MP camera pumping out 2-2.5 MB photographs each (lossless compression; it's the closest thing to a digital negative) and hundreds of photos a month, it's very easy to get a collection of digital media that is not replaceable simply by downloading it again from another source. I'm glad I don't have a DV video camera yet -- otherwise I'd be spending all of my cash on new storage. I don't have a good way of backing up my data reliably, so I'm still scouring this story for good ideas.

Actually thats the real reason i thought of it

[rebelcool]

Awhile back I was brainstorming up some ways to preserve current mediums which only exist in digital or magnetic forms. Like audio and video. Suppose you want to preserve a specific bit of video for viewing 2000 years from now. Not only do you need a medium for storage that will last that long, but you must deal with the certainty that no equipment will exist capable of playing it.

So one way would be to both preserve a general specification of how to read the data, and then the data itself. So not only would you need a method of encoding the song onto paper, but you'd need to include the details of an algorithm - simple enough that people whose language may be very different from ours - can recreate it using their machines of the time. And then they can feed the data into it, and replay the music/video/whatever as we intended it to be seen.

Re:Actually thats the real reason i thought of it

[AvitarX]

I bet 2000 years from now they can figure out the algorythem if they know what they are looking for. If they know that it is binary information, information therory will still be the same. The ways to encrypt information will be the same. I bet they could figure out the data, if they knew it was binary data. Think about what we can do with old languages. Besides, how do you explain the algorythm? Plain text I would imagine, because I ascii would need explenation if you encoded it digitaly. Maybe someone with more knowledge can fill me in on why we need to explain this stuff, but I don't think that cracking an algorythm can be much harder then crypto cracking.

Re:this is idiotic.

[dougmc]

Darn, there's hasn't been a 7200 rpm SCSI drive for a couple of years now...

I only have to provide one example to prove you wrong, but here's a bunch for you. Some of these drives are older than two years old, but many are not.

Note that for backup use, you don't need a 15k drive. In fact, 3.6k would probably be just fine. Remember, this article is about the feasability of using hard drives as backup media -- after all, you can buy 120GB IDE drives for half as much as you could buy 120GB of DLT media (nevermind the drive itself!)

I definately prefer SCSI drives -- not because they're `so much more reliable' (they're not) and not because they're `so much faster' (they're not, not anymore) but because the interface is so much more efficient. My computer doesn't grind to a halt while accessing the SCSI drives like it does with the IDE drives.

As far as this article goes, I think that buying large IDE drives for backups is a reasonable plan, as long as you understand the limitations -- the drives (IDE or SCSI, high-end or low-end, it doesn't matter) may not spin back up after 10 or 20 years of sitting in a box, and you have have a hard time finding a computer that can even read it.

Consider if backups had been made like this 15 years ago. Do you have a computer that can read MFM hard drives? You may have a MFM controller sitting in the garage, but there's many possible problems --

It's probably ISA (maybe even 8 bit!) Does your computer even have an ISA slot anymore?

Does your OS support it?

Does the controller even work? If you were nice to it, it's in an anti-static bag, but there's a good chance you weren't.

Do you have cables for it?

Do you remember the needed jumper settings?

etc.

Yes, you could probably read from the drive -- assuming that it still works. But it would be a lot of work -- and let's hope you don't destroy/erase the drive through a mistake.

Note that this problem is not restricted to reading old hard drives -- old tapes have this sort of problem too. Finding a tape drive to read a 9 track tape is quite difficult nowadays, and while you can pay people to do it for you, they're going to charge you a lot. DAT and 8mm drives aren't so rare, but what's the block size? Was it variable or fixed? Was compression enabled? (does your drive even support it?) Was it stored in tar, bru, cpio or dump format? (if it's dump, dump from which OS?) Even worse, it may be in some proprietary backup program format -- made by a company that disappeared ten years ago!

And let's suppose you can read all your data, and you now have it on the hard drive on your modern computer. The data is 15 years old -- what's going to read it? What's a .pic file? .fli? This .tif file doesn't seem readable by anything today! .gif -- was pretty new back then, but we can still read it today.

Again, this problem isn't restricted to any one form of backup. So far, the only form of archival that has truly lasted the test of time has been data printed on paper. Microfiche and similar things have lasted a long time, but have yet to be tested for hundreds of years. But so far, with computers, even 30 years is a very long time to go back.

But for your normal backups (oops! I just deleted all my data!) using IDE hard drives instead of a tape drive isn't so bad, as long as you treat them like tapes (keep a few generations, keep offsite backups, etc.) and don't expect them to last more than a few years at most.

Re:Tapes *is* the right medium for long term backu

[cybergibbons]

Are the "*"s round the "is" to highlight the fact that the grammer is wrong? You should have said "Tapes are the right media for long term backup" or "Tape is the right medium for long term backup".

leather and ink

[superyooser]

Use carbon-based ink on leather scrolls, put them in pottery jars, and store the jars in an arid desert cave. The Dead Sea Scrolls were preserved this way for almost 2000 years.

Re:leather and ink

[angst_ridden_hipster]

Yes, but some of the scrolls were in pretty bad shape.

There's a lot of debate as to how to fit the fragments together -- different approaches lead to different interpretations of the text.

They didn't use CRC on Aramaic or Syriac in those days ... I guess they probably still don't.

Re:the absolute surefire way to back something up.

[Yarn]

Laser printers do gray scale by dithering, you lose resolution. Good idea though. Better storage medium would be black/white photographic film like microfiche.

Here's your backup script

[Ardeaem]

--Begin #! /bin/sh mv $1 /dev/null End-- Benefits: 1. No worrying about media 2. Saves space Drawbacks: 1. May be difficult to get your data back 2. No GUI (yet)

Re:the absolute surefire way to back something up.

[2nd Post!]

The point was that at 6gb a pound, a paper solution would outlast a magnetic media solution because it was optically readable and be able to survive much harder conditions.

Re:keep in mind Gibson is a known ASS

[saskboy]

Vmyths ought to steer this OP right.
Mr. Gibson does about as much good for computers, as Symantec Antivirus does for JokeADay.com.

In SOVIET RUSSIA...

[-douggy]

The funny moderates you

[sa]

don't even consider long lasting backups

[indefinite]

I wouldn't use hard drives for back up in any scenario. They are really unreliable. I'm sure we all have a few drives that *just died* lying around. That's why you need something that has the storage mechanism separate from the storage medium.

So, use tape drives for short term. Don't worry about massive data as it is likely to not be changed the next backup around. There are many apps out there that will backup only what actually changed. After the first backup, you can just leave the tape in the drive and automate it.

Then every year copy your tapes to DVD's, as at least these will last few years and put them in a place outside your house, like a safe box or something. Burning is a hassle, especially with lots of data, but once a year is not that bad.

This will only last you a little longer though. So you will need to copy those DVD's onto other media/formats.

In 10 years, chances are that it will be a challenge finding a device capable of reading your backup format/media. Even DVD's will not last that long. They will be replaced by drives 1/10th the size and 1000 the capacity. Not to mention the file systems will likely go through a big change as well.

Re:failures

[WilliamX]

USB 2.0 has a higher speed than FireWire.

whats a picture look like in binary?

[rebelcool]

100101010... etc. What does a text file look like in binary? 1010100100 etc.

There is no difference between a text file and a video file, except in how the voltage pulses are acted upon. Thus, an explanation must be given as to how to properly interpret the strings of binary digits. Suppose you compress the data, why would they be using the same compression algorithm, or even know how to detect it? If you want someone to be able to read it, wouldn't you want to make it as easy as possible for them?

I was reading an anecdote the other day about early NASA data tapes. The machines to read them have long since disappeared, so in order to get the data off they pulled some 80 year old engineers who built the things out of retirement to interpret the data.

Re:whats a picture look like in binary?

[AvitarX]

is it really that hopless to read an 80 year old format?

If so SETI has us so played it ain't even funny.

My point thouhg is that in 2000 years any language we have to communicate is goin to be quite foreign.

Unless you can draw a pictogram of an algorythim I don't think you are going to be too much help.

Re: the absolute surefire way to back something up

[atam]

A typical letter-size paper is 8 x 11 inch. So it can contain only 88 KB per page. A bit of an improvement to the 20 KB of the original scheme but still not high enough for Terabyte of data. In addition, this data pattern probably requires compatible hardware/software to read it back. Considering it is non-open source, how do you know it is still available 10 years from now?

Controller technology vs. reliability

[billstewart]

Maybe you're buying good SCSI disks and el-cheapo IDE drives, but the real difference in them is the controller cards, not the disks themselves. If you're using the disks for archive, not for active use, they're not going to be under the same load or wear&tear as the primary service disks. And you're probably going to RAID-5 them anyway.

The real problem isn't usually hardware rot - it's data format rot and interface standards rot. IDE is a bit tired, but it'll be supportable much longer than N-1 of the current DVD-R standards, so you probably want both. In my attic, I've got a probably-good 9-track backup tape of some of my early work, and some newer probably-good Sun cartridge tapes, and I might still have a Sun tape reader, if I can find a copy of SunOS 3.5 for my Sun-2 diskless workstation to bootstrap something to read it with.... Not a high priority - anything useful I did back in those days is probably on a Usenet archive somewhere.

70 CDR's???

[CaptainSuperBoy]

You are an insane person..

IDE drive been through a fire

[Latent Heat]

My dad's workshop burnt to the ground in 1999 -- I mean a real charred mess. I pulled a 400 MB (yup, a really ancient) IDE Western Digital drive out of the Gateway computer housing, connected it to power and a drive cable, and was able to back everything up to CD-ROM. The drive smells really bad, so I keep it in a sealed plastic bag and don't try to run it, but I was able to get the data off it.

Do you suppose this worked because it was an old, low-density drive?

Periodic Re-Copying is the only viable method

[billstewart]

*Everything* is a horrible archival medium. The IDE drives will support you just fine while you're starting to address the hard problems. Don't bother with 100MB-Ether if you can conveniently do GigE though - it's gotten really cheap.

• Data format rot and hardware medium support rot will kill your data much more thoroughly than equipment failure will.
• Only one of the writable-DVD formats will still be readable in 5 years - I don't know which one, because it will be a market support issue, not a hardware reliability issue. Copy protection may make that worse.
• The file system format your data is stored in won't be readable by Windows in five years, though it will probably be readble by Linux.
• Any data format that isn't basically text will be obsolete - HTML and XML will be readable, but word processor files will be almost totally useless, and at least half of your non-text-like database formats will be mostly useless. You can keep a copy of MSWord and Excel, and the original CDROM installation media, but it won't run on Windows 2007XYZ, and your WordStar 5.25" floppies aren't much use today either. Closed-source will rot faster than open-source, but both will rot eventually.
  
• That 6GB disk drive you bought three years ago isn't worth using, but the data fits nicely in a corner on that $120 120GB drive you picked up last week. The 120GB drives will be pretty boring in 5 years, whether they're IDE or SCSI, but the data will fit nicely in a corner on the old 1TB drive you'll be using for spare storage in a couple of years or the newer 8GB drive, but you may have a bit of trouble reading those tiny little 8KB blocks from the wimpy 2MB cache.

So yes, use some kind of IDE RAID system for your backups, and be sure to save some backup computers and operating systems every year or two to read the old disks.

• Depending on how much of your data is files as opposed to databases, come up with some reasonable strategy for full and incremental backups, and what stays on-line, on-site-but-off-line, and off-site, and if possible, keep a computer off-site as well.
• Keep a database of what you've got backed up, and where.
• At least annually, copy at least some of your full backups from old disks to new ones, and depending on your incremental or journaling structure, maybe more than that.
• Make ABSOLUTELY sure your financial people have their records in some format that's readable and usable, and review with them every year to make sure their backups are in a usable format. Audits of seven-year-old financial data are no fun even when you *can* find the supplies for /dev/iso-9000-clay-tablet-scriber and programs that read the old-format sound files to output carrier pigeon calls to /dev/audio.

Right you are.

[rebelcool]

Indeed, the most difficult part is how to describe it. This is a field of study though, one that interestingly, SETI has a part in. How would we communicate with extraterrestrials in a meaningful way?

Presumably, one way would be mathematics. 2+2=4 everywhere in the universe. Therefore, a system based on some fundamental math properties would be useful.

The book 'Contact' briefly dived into this, as the contacters sent diagrams for the machine, along with a primer on how to make sense of them by using various universal things to describe the concepts used in the schematics.

For example, to describe what the symbol for pi is, you might try drawing a circle, a line halfway across the circle and a couple of dots to describe "2piR", which is the universal equation for circumference.

Re:Tapes are a expensive waste of time

[patchmaster]

Burnt CD's (like you'd use at home) have a shelf-life of about 10 years. Then the medium starts to oxidize (the metallic film, not the plastic itself), and flakes..

TDK rates their CD-R lifespan at more than 70 years when stored at 30C (~86F). Kodak claims a 95% confidence that 95% of Kodak media will have a lifetime greater than 217 years if stored at 25C, 40% humidity. One would assume the cheap generics will last not quite as long as the Kodak CD-Rs, though I suspect they'll last longer than I will if kept at room temperature and out of bright light.

Re:failures

[ncc74656]

USB 2.0 has a higher speed than FireWire.

Only theoretically...in practice, my understanding is that FireWire is still faster (delivers something closer to its maximum speed) and still doesn't bog down your computer as much. (Think about it...Apple designed FireWire to work properly, while Intel designed USB to drive demand for faster and faster processors.)

PaperDisk

[cameldrv]

www.paperdisk.com claims that they can get either 660K or 1MB depending on resolution on a sheet of paper. How long a piece of paper will last when encoded with this density is unknown, but with good paper I'd bet it's a hell of a lot longer than any disk. Furthermore, even at that density, there's a huge ammount of physical redundancy in the data storage. If the paper gets to be fifty years old or so, I would imagine that the technology would be available to cheaply scan at ultra-high resolution to compensate for any degradation.

Ancient Hard Drives

[tuuw]

A little off topic but I have an old 40Mb Miniscribe hardrive which was removed from an IBM XT around 1987, been in storage ever since. A few weeks ago I got curious and decided to plug it in to a spare box I had lying around to see what would happen. Fairly easy since the original MFM controller and cables had been kept with the drive.

I was amazed to find that not only was all the data on the drive intact but the thing booted up straight into MSDOS 3.1 with no problems.
MSDOS 3.1 and Word for DOS 5.0 really scream on a PIII 800 !

Anyway, curiosity satisfied I put the drive back in storage. I figure I'll try it again in another ten years or so... if I can still find a motherboard with ISA slots

I've never had a hard drive fail that was in storage, not counting the one that rattled around on the dashboard of my car for 6 months. I still dont think I'd recommend using them for long-term archival media though

RAID-1 plus drive rotation

[Phil Karn]

A while ago I got tired of swapping DAT tapes during full backups of ever-bigger disk drives, and of having to minimize my use of the system while they ran. I also got pretty tired of repeated hard drive failures, as I had purchased a couple of those jinxed IBM hard drives made in Hungary.

So after a brief look at hardware RAID I realized that the software RAID support in Linux was all I really needed. Since this is my own machine, I didn't really need the hot-swap capability of a hardware RAID controller.

I bought two 100GB Western Digital drives and set them up in a RAID-1 configuration. A month later, I bought another drive, replaced one of the drives in the machine with it, and put the removed drive in the safe. A month after that, I bought another drive and repeated the process, this time moving the drive in the safe to an off-site location.

Every month or so I repeat the process, rotating the second drive of the array through my various offline storage locations. The real beauty of this (especially vs tape) is that I only need enough downtime to swap the drives and reboot the system; the mirror reconstruction runs in the background as I use the system normally.

The use of RAID-1 gives me complete protection against data loss in the event one of the online drives fails (though I've had no failures yet with the WD drives). If both drives are somehow ruined (e.g., by a fire within the computer), or if I accidentally delete something important, I have my first offline backup, less than a month old. If that's also ruined (e.g., my whole house burns down and the fire-rated safe fails to protect the drives it contains) I have my off-site drive, which is less than 2 months old. Obviously I could easily extend this process with more drives and more offsite storage locations.

Because the backup drives are regularly rotated into online service, bearing stiction should be less likely to occur. And if an offline drive were to fail when I bring it back into service, so what? It was about to get overwritten anyway.

Naturally, I also continually back up especially important files (e.g., email, work projects, documents, etc) to various machines over the network, as that's the easiest and most effective way to protect small amounts of data. But when it comes to periodic full backups of big disks, nowadays I just don't see any practical alternative to disk-to-disk copying. And RAID-1 is the easiest way to do that copying.

Changing backup device from time to time

[SkunkAh]

I suggest you should change the medium you are using to backup from time to time. You currently have 220GB of data to backup. I would create a RAID5-Array of IDE disks which can hold upto 300 .. 400GB of data. As soon as the ammount of data is increasing and coming close to the maximum ammount of data your can store/backup, add an extra disk to your RAID. I also want to suggest to re-evaluate your backup system every 3 to 5 years to see if is still fits in the backup strategie you have. And if required changing to a new backup system.

Storage Important Too

[CharlieO]

The storage conditions are important too - there is a tract of Landsat data that is forever lost because one of the store rooms flooded and mildew killed the tapes.

Whats even more frustrating is where you have the tapes, but not the readers and machines to understand them. I understand some of the Apollo data may never be recovered because no-one thought to archive some of the tape mechanisms and computers at the time - something that is now done semi routinely at NASA

Beam it up Scotty

[Bazzargh]

Buy yourself an array of radio telescopes, and beam all your data into space. Perfect data recovery (in 50 years time) will be assured through the use of alien technology[1].

This follows the best practice of always storing critical data offsite in case of disasters like, er, the sky falling in.

-Baz

[1] of course you'll have to escape the slave labour camp first.

Ugh, DV cameras... :(

[Andy Dodd]

Bought one two months ago.

I had purchased an 80 gig HD to replace my failing SCSI drives (Bus was flaking out.) around that time.

Guess what I just bought this week? Another 120 gig HD with an external Firewire/USB 2.0 enclosure.

Re:slow and manual?

[SuiteSisterMary]

And if you want to be REALLY fast, you can RAID stripe across tapes.

I had a Dell PowerVault 130T; 4 DLT4 drives, 28 slots for tapes, and a roboarm. There was just something indescribably nice about having four separate backups going at once, all nice and zippy.

Huh?

[fw3]

Yes tapes and drives fail. However the effective claimed life of good media is 30 years. If you really needed better, of course there's optical media (crystaline / stable writeable media's been available for a long time also).

In nearly a decade of using 8mm tape I've written a few gig nightly to 2 drives I'm approaching 100 TB of data written and nearly 100% successful restore operations. I have exactly one instance of data-loss in that time and many instances where the rotation scheme saved someone from a poorly timed deletion or other accident / error.

Total failures, 4 dead tapes, 1 drive refurbished. I don't have a controlled environment or especially perfect storage conditions, yet I've had no problem reading data that's 8-10 years old.

Just pulled the july 8, 1995 tape and read it just fine (nothing older than that happens to be onsite just now).

Sure, it's a pain when something breaks, but honestly I put more time into working out the methods to not be surprised than I've ever lost due to lost data i.e. we don't rely on any one tape ofr anything critical.

Perhaps more importantly planned backups have saved entire systems when the o/s got wedged, and if ( - it's not happened yet) a system gets rooted we can restore known-state with minimal disruption.

Ohh and btw I've seen the spindle bearings of out-of-service hard-drives freeze when left off for extended periods -- those are not odds I like at all.