Large IDE Drives as Long-Term Archival Media?

PlatterMan asks: "The question of how to cope with backing up disk drives which are rapidly increasing in size, onto tape and other backup devices which aren't scaling in size as quickly isn't new to Slashdot. Neither is the use of single, raided, and removal disks as backup devices, this has been covered numerous times on Slashdot in e.g. here and here. One thing I haven't really seen discussed however is the feasibility of disk drives as medium to long-term archival media, say 5 to 10 years. Like many people I'm in the position of now having multiple machines with a combined data pool of about 220 Gig, and backing up these onto DDS or DLT tapes is slow and manual to do, and expensive in tape costs. So I'm looking to add a removal drive bay to my primary backup machine and pick up a bunch of large IDE drives, so that I can do regular disk to disk backups over 100 Meg Ethernet (and for my machines which are in cages, over the Net) pulling out and alternating the backup drives on a 3-way backup cycle."

"Backups are of no use without offsite archival copies so I plan to take one set of disks out of the pool, and archive them offsite on a quarterly basis.

However, I've heard horror stories about the data retention and usability off older disks which have been shelved for archival, for example disk stiction - where people try to restore data off of a 4 to 5 year old drive only to find that the disk won't spin up due to solidification of lubricants, or that they've experienced data degradation.

I'd be interested in the Slashdot crowd's opinion on using large IDE drives as an archival media. Clearly one possible problem is being able to get hold of a machine in the future with a suitable IDE interface to plug them into for restoration, but I can't see IDE disappearing within 5 years (maybe 10 though). I'm more interested in experiences and opinions on the suitability of the disks themselves for long-term archival.

• Is stiction still likely occur on newer makes of IDE drives or have manufacturers beaten the problems which caused this in the past?
• Likewise how likely is bit drop-out and general data degradation over say a 5 year and 10 year period, and what do people think would be the likely maximum feasible time that a shelved drive would be usable for?
• Any suggestions as to how would I need to store drives in order to minimize these types of problem and maximise their feasible life as archival media.

Thanks!"

Comments

Print!

[grub]

Print out all your data in hexadecimal and store it in a large vault. If and when a data loss occurs you just need to re-type all the data back in.


yes I'm being facetious

Re:Print!

[Anonymous Coward]

Use paper tape and avoid retyping! If you are really hardcore you can punch metal tape like the US military and achieve nuclear survivability.

Re:Print!

[Anonymous Coward]

As a general-purpose backup medium, hardcopy isn't very practical.

BUT...

Consider that printouts on archival paper can be expected to last 100+ years. Tapes flake and fade, disks stick, cd's oxidize. Nothing else even comes close to paper! (well, maybe stone tablets...) Human readability is another plus.

For certain types of info, a printout is definitely the best choice.

Re:Print!

[alexburke]

Have you ever tried to grep three boxes of greenstripe?

Not a pretty sight, let me tell you...

Re:Print!

[Bonker]

While funny, this guy has hit the nail on the head. Without constant, vigilant backups, plastic and magnetic media don't mean dick in the long run.

If you're serious about keeping data for ever and ever, but also want convenience, you have to back up both ways.

1. Go ahead and keep data on that harddrive, but you're stucking buying another one to replace it, at least every year or so, just to make sure. This gives you the highest convenience for reinstating that data when (not if) it is corrupted.

2. Print it out. Print out all of it on non-acid paper with archival ink with the most expensive commercial printer that money can buy. Images, text, what have you. If you don't have a hard copy, you don't have the data for the long term. Once it's all printed out, put it in air and water-tight containers and then put it in a temperature controlled vault somewhere, preferrably underground so that it remains termperature controlled, even if power is lost for a long time.

Um you've pretty much answered your own question.

[MisterFancypants]

Hard drives are a horrible archival medium.

Without normal/regular use, you WILL have problems trying to read from them in 4-5 years time. Hell, the way most IDE drives are these days (note the recent reduction in warrenty time periods), you'll be lucky if the drives last 2 years even WITH regular use.

t's the next AYB^H^H^H Soviet Russia

[Dental Plan]

Backing up to IDE hard drives.... That's a paddling

Not using SCSI like you should... That's a paddling

The right tool for the job is a tape drive, if you don't use it.... That's definitly a paddling.

Mission Critical Data..

[xchino]

Speaking from experience I can give this bit of advice for archiving critical information. Use a solid state device, don't even consider a magnetic solution, unless losing some or all of the data won't ost you your job.

Re:Mission Critical Data..

[Spire]

CD-ROM or DVD-ROM, I would guess -- with lots of redundancy (such as multiple copies of everything, plus devoting a large percentage of discs to PAR-style files). It's really, really cheap.

Steve Gibson

[Jucius Maximus]

Please don't flame me for quoting Steve Gibson, but I think he's right on this account: "There are only two kinds of hard drives -- Those that have failed and those that will fail."

Hard drives are not non-volatile storage.

Re:Steve Gibson

[LoudMusic]

No flame, other than the term 'RAID 5'. Tapes aren't as dangerous as hard drives, but they can still mess up. It's not like they're garounteed beyond all odds. So a RAID 5 IDE array takes care of your data.

I'm currently using Dell NAS machines as archival backups.

Bonuses (as I see them):
Online 100mbit access to old data.
Cheap!
Fits in a physically small space.

Negatives:
Higher failure rate than tape. Pop fizzle, your data is gone.
Difficult to take off site.
Long-term replacement isn't really an option. (for RAID replacement)

The way we negate the negatives (double negative, is that a possitive?):
-Failure rate / Data loss is countered by RAID
-Taking it offsite ... it is possible to cost effectively mirror an IDE RAID system over broadband Internet and do it securely. If you are a major corporation surely your campus is large enough to simply run fiber to two corner and put mirrored backup at each location.
-Long term replacement of RAID drives ... buy a truckload of disks when you do the initial installation? (:

Re:Steve Gibson

[ikeleib]

Actually, using RAID5 on tapes is not unusual. It has the same benefits that RAID5 disk arrays have. It allows for the loss of one tape, as well as increased throughput. This technique can actually be extended to any media.

GraniteDigital is what I use

[TheCodeFoundry]

I back up close to 300GB on a nightly basis using GraniteDigital's FIRE Vue(TM) FireWire 1394 IDE Ultra ATA Systems

I have 6 120GB Maxtor's and rotate them nightly, storing them in a fireproof safe, rated for paper storage. Granted, if a fire occurs, I'm not sure if the data storage would survive, but I think that would be the least of my worries, at that point. The Firewire works great and is very fast.

Re:GraniteDigital is what I use

[coyote-san]

At the least, toss the media into freezer-weight ziplock bags. Better yet is double-bagging it - put the media in a smaller bag, and then in a larger bag with smaller bag's opening on the 'far' side.

Paper-rated "fire safes" work by putting a media that undergoes a phase change at high temperatures, releasing steam in the process. (Think of the latent heat involved in freezing and melting ice, same theory is used to keep the interior of the safe at a reasonable temperature.)

The only problem is that paper tolerates steam fairly well. Ditto the smoke that can make its way into the safe. The paper may be damaged, but it is still readable. Computer media will be destroyed. Fortunately freezer-weight plastic is more than adequate to block the steam, leaving only small openings in the seal. Even this is modest, and the second bag is mostly to allow you to avoid smearing soot onto the media as you remove it from the bag.

Re:GraniteDigital is what I use

[Jeremy Erwin]

paper burns at 451 degrees F (232 Celsius)
media starts to melt at 125 degrees F (52 Celsius)

A fireproof safe thats rated for paper storage only isn't going to cut it.

Good idea...except...

[Agent Green]

...you're walking down the hall with a 3 foot stack of drives and you trip over an ethernet cable...and all the drives take a sailing course through the air and land on the concrete floor.

I'm not a betting man, but I bet if that were a stack of DLT tape, you might still be able to read them after that hypothetical incident.

Re:Good idea...except...

[apoc.famine]

Eh, I bounce DLT tapes off the floor on a regular basis, and just about all worked fine afterwards. The trick is to say, "It's still good, it's still good." before you pick it up.

warranty period

[Clover_Kicker]

Since IDE HD manufacturers recently decreased their warranty period, I'd be *really* reluctant to trust 'em 10 years from now.

Re:warranty period

[Clover_Kicker]

You are missing the point. What is your backup method for backing up 220GB?

Tapes are designed for backups. If you seriously need to backup 200GB, then you are looking at DLT or better, and it ain't cheap.

Oh, you don't backup 220GB of personal data on a regular basis?

Who the fuck has 220GB of personal data? Seriously, for the cost of backing up that much porn, you can just go down to the store and buy the legit DVDs. While you're at it, you can stop off at the record store and buy some albums so you can re-rip your MP3s.

Just because you have 220GBs of hard drives in your machines doesn't mean you need to back up every byte.

C:\WINDOWS>ver

Windows 98 [Version 4.10.2222]

C:\WINDOWS>du |sort |tail -1
353472k ./

C:\games\Diablo II>du |sort |tail -1
1378784k ./

C:\games\Diablo II>du save

1696k save/old/
3328k save/

Pop quiz - if I wanted to back up this machine, do I

• backup 1.5GB of Windows and Diablo binaries
• backup 3 megs of Diablo II save files (would fit on 2 fucking floppies, FFS.) because I have my Win98SE and Diablo II+LOD CDs on the shelf.

My documents (resume, web pages, GNU Cash files, email etc.) live on a server, where they are in fact backed up nightly to a second hard drive.

Every couple of months I burn a CD of the latest backup tarfiles. Cheap CDRs are a half-assed long-term archival solution, but the price is right.

Some things (Mozilla installer, service packs) are so ephemeral that they aren't worth backing up, i.e. when you need them there will probably be a new version available anyway.

What about my MP3s and pr0n? When I've got enough new stuff I burn a CD full. Every year or so it's worth re-burning the MP3s so that I've got the same genre on a given CD. When you've got Sarah McLaughlin, Mozart, Dead Kennedies, Suicidal Tendencies, Reverand Horton Heat and Johnny Cash on the same CD, there isn't a person in the world who won't make fun of you.

So, you trust having no backup at all over having a backup on an unreliable medium?

I did not recommend that no backup be performed. I said that I do not trust IDE drives for long-term archival use.

If you are determined to archive to IDE, fill your boots - it ain't my data.

Re:warranty period

[fishbowl]

"Who the fuck has 220GB of personal data? "

I'm getting there, in audio data.

My own music, that I write and record, so, going down to the store to replace it isn't exactly an option.
It's also on DAT, and on CD audio, so you could say
I have a backup, but that's not really true -- the DAT is the source material, and a CD would represents one view of some of the data.

Am I going to buy a $65,000 SAN tape library machine, just because I'm getting into volume? (No.) Would I like an inexpensive solution that is less cumbersome than CDR? (Yes.)

Re:warranty period

[Kaa]

Who the fuck has 220GB of personal data?

And what's so weird about it?

A scan of a single frame of a 35mm film, on a high-end consumer film scanner will create a file... let's see:

The scanner is 4000dpi, so the resulting image is about 4000x6000 pixels. We are working in 16-bit-per-color-channel mode, so that's 6 bytes per single pixel. A bit of multiplication get you 144Mb. As a practical matter, the film frame is slightly smaller so your output TIFF file is about 120Mb in size. That is for a single 35mm film frame.

So raw scans of slightly under 2000 film frames will already hit the 220Gb figure.

Still think it's a ridiculous number?

Bad Idea..

[Suppafly]

I think it would be a bad idea to rely on IDE drives as one's only source of backup. Especially if you aren't planning on using any stripping or parity. The large IDE drives are, the more prone to failure they appear to be. Ask anyone thats bought a 60-100 IBM deathstar drive lately. The added wear that would occur from joustling them around as you pull them in and out of the drive bays all the time seems like it would also make the time between failures greater. What is proposed in the story might work fairly well for a home user, but I think it would fall apart in a business setting.

Alternatives...

[anarchima]

People here are saying, "Don't even think about using IDE!". Well he has no choice, does he? Tape has several drawbacks as the author mentions his comment to Slashdot. He has asked for advice on IDE. If this is not a feasible option, recomend some others (besides tape). Or ARE THERE NONE?

Not a horrible Idea, just not a great one.

[briancnorton]

Using magnetic media to back up magnetic media isnt the greatest idea in the world, but it can work. Hard drives fail, and when they do, you want to have the data available so that you can get to it. The IDEAL way to do this is to contract an outside company or manage for yourself a backup server which does incremental backups as often as you need and periodically burns them to a more permanant media like DVD. If you cant afford this or dont like the idea, then you can burn DVDs on your own. A good program will track files for incremental backup and 220 gigs can fit on something like 50 DVDs, with maybe 1 more per session (assuming that not all files are constantly changed) Obviously a lot depends on what you have, how much money you are spending, and what you need.

rock and chisel

[Lxy]

with all the stories I've seen about being unable to retrieve data from just 15 yrs ago (because the format is unreadable, not because the media deteriorated) I'm convinced that archiving data using a chisel and a rock is the best way to go.

Re:rock and chisel

[nsample]

I know this parent was modded up as +Funny, but it's actually +Informative. "Rock and chisel" are the best thing we have, and there's a real trend toward using it more. Take a look at Norsam's HD-Rosetta. It's an etched nickel plate designed to last for thousands of years. Vive la Rock & Chisel!

Re:rock and chisel

[Anonvmous Coward]

"with all the stories I've seen about being unable to retrieve data from just 15 yrs ago (because the format is unreadable, not because the media deteriorated) I'm convinced that archiving data using a chisel and a rock is the best way to go."

He's right, you know. Look at the info we're gathering from fossilized remains of dinosaurs! Once they found the petrified remains of a velociraptor next to a picket sign protesting the use of fossil fuels.

Tape really isn't that expensive.

[Wakko Warner]

What you're proposing will cost no less than a high-quality AIT drive, which, though you may need to span tapes in the most extreme of situations, will give you quite a bit of capacity. You can pick up 90GB native-capacity AIT drives now for around $500 or so on eBay. The media is affordable, too.

- A.P.

Why Tape Is Good

[Jucius Maximus]

Tape may be inconvenient but it is still a true backup medium. With hard drives, the reading and writing hardware are enclosed with the platters. So when the read head of the HDD fails, your data may be 100% intact on the platters but you can't get at it without professional help. How many other parts in the HDD could fail without harming the platters? A lot!

With tape, the failure of a tape drive doesn't separate your from your data (unless it catches on fire with the tape in it or something.) You can just get a new tape drive and you are good to go again.

Thus, tapes are very good because the storage medium and the read/write hardware are separated and not interdependent.

Re:Why Tape Is Good

[BlankTim]

Obviously, you've never had a tape physically fail.

Maybe it's just me, but after the experiences I've had the last year with crappy tapes, I'm surprised the "tape as a backup medium" idea hasn't been seen for the farce that it is.

Backing up to IDE or SCSI? Good short term solution, but I don't think I'd trust my backup drives for more than 1 year, tops.

Burn to CD? Good long term solution, just not practical due to the file sizes involved. Burn to DVD isn't much better.

It's time for something new. Hell, maybe it will turn into the next "killer thing" and revitalize the economy.

I vote for soft bubble memory

Re:Why Tape Is Good

[Rolo Tomasi]

Thus, tapes are very good because the storage medium and the read/write hardware are separated and not interdependent.

Bullshit. Tapes are intended as a short-term backup medium. Google for NASA magnetic tapes, and you will find a lot of interesting stories. Like e.g. this one:

Right now, ACRES is updating storage of 120,000 gigabytes of data collected since 1979, primarily from remote-sensing Landsat satellites passing over Australia. Landsat images are among the most voluminous of space-based data, making ACRES one of the largest data repositories of its kind in the world, Trezise said.

The data now are housed on optical tape, having been rescued from disintegration in the early 1990s from aging high-density magnetic tapes. That first rescue operation occurred just in the nick of time, Trezise said, since the magnetic tapes were starting to get sticky on their spools.

Re:Why Tape Is Good

[skroz]

One very important thing to consider : With certain types of tape drives, a misaligned head can render your tape media useless in another drive of exactly the same type. DLT is a good example of this. You can write and read to your heart's content on the same drive, but try to read a tape written in one drive on another and you can be sunk (professional data recovery experts with the proper tools can work around this, but it's expensive, and the whole point of this discussion was the need for "professional help" if certain parts of the hardware fail.)

Re:Why Tape Is Good

[sunspot42]

Yes, tape is a "true backup medium", but it is *not* a suitable medium for longterm archival storage - at least, none of the affordable formats up till this point (like the DAT-derived DDS format) have been. There's a big difference between a backup medium (a copy that's probably replaced every day / week / month and is intended for use in the immediate future) and archival storage (a copy that's intended for use 5+ years in the future).

While the failure of a tape drive won't separate you from your data (unless the drive damages the tape at the same time it fails . . .), tapes themselves deteriorate over time. Here's an article about the problems the National Archives here in the United States have encountered with preserving copies of the Nixon tapes on DDS's audio cousin DAT. An excerpt:

"During the National Archives' routine monitoring of the tapes'
condition, the analog reel-to-reel copies have shown no signs of
deterioration whereas there is an estimated 5-10% catastrophic failure
rate among the DATs in the collection. There appears to be no pattern
to the failures. It has occurred on new tapes that were recorded six
months ago, and it has occurred on tapes that were recorded six years
ago. It has occurred on all brands of DAT purchased throughout the
previous seven years. Accordingly, the archivists routinely reduplicate
these DATS on multiple copies. As insurance, archivists also transfer
DAT copies back onto analog reel-to-reel preservation copies. Unlike
the other preservation analog copies, these copies have not been
filtered and closely "mirror" the original tapes. Therefore, in the
future when technology has progressed, the archivists can retrieve
conversations that are extremely close to the original audio recordings
and enhance these with the latest technology."

Leading audio preservationists have issued their own warning. This company deals with audio preservation, and has some interesting things to say about tape formats - analog and digital.

Of course, DDS tapes have supposedly been manufactured to a higher standard than their Audio DAT cousins, sport finer particles and stronger binders, and the format includes additional error correction and redundancy. Still, these issues with a modern tape format like Audio DAT are not an encouraging sign for those seeking to perform archival storage using DDS and it contemporaries. HP for example only claims a 10 year archival life for DDS. Contrast that with the 75-100 year lifespans Kodak and TDK are claiming for CD-R.

These longevity issues won't just be confined to older tape backup formats though, if history is any indication. It's the nature of the medium. I think Sony is currently claiming a 30-year lifespan for AIT, and HP something similar for their new format, but of course we only have a couple of years' experience with them so far, and given the incredible data density of those formats, if something should go wrong with either of them the results could be catastrophic. Unexpected deterioration has certainly happened with tape before - witness this article composer and synthesizer pioneer Wendy Carlos put on her website, as well as her own experience with her older tape masters.

Hard drives certainly aren't a great archival medium either, but I wouldn't be so quick to assert that tape is superior. At least drives have the advantage of being sealed from the outside atmosphere, and contain within them all the logic and hardware required to extract that information in the future. The only big issues I can see are, will there still be equipment to interface with them in 10 to 20 years (probably, since IDE is so widespread) and will the drives still spin up in 10 to 20 years (who knows). It's that second issue that's the real buzzkill for HD's as a longterm storage medium. Manufacturers won't even issue a decent warranty on drives anymore. What does that say about their planned longevity?

Me, I think your best bet is DVD. But if you really want to be able to read that data in the future, I'd suggest copying it to at least two different formats, perhaps AIT *and* DVD. Don't forget to check on it every few years, too. If there's any sign of deterioration, you'll hopefully be able to make another clone before the failure becomes catastrophic (perhaps to a superior format that hasn't even been invented yet). If you want something you can just throw in a hole and forget about, sorry - that media doesn't exist.

organize your data

[jayhova]

Used to be in the data backup biz, you should really start with evaluating what you are actually backing up. Most people backup applications and temp files that really are not going to help much. Also, do you really need to archive all of that stuff even if you are anal? Another thing to consider is, will the media be supported and will you have the proper drivers for the disk drives handy. 220 Gigs is surely still in the land of tapes, I hate them more than most, but would not suggest the use of an IDE Hard Drive. my 2 cents

Long Term Storage

[caseydk]

The Library of Congress is attempting to answer this question as they have huge amounts of media that is on highly degrading (nitrate-based films) materials.

Their answer? A huge RAID array starting at 180TB and growing steadily over time.

Your answer? Probably figure out which of the data is fixed and which of it changes and attempt to back up accordingly. Does all 220gb change on a weekly basis? That seems unlikely...

Re:Long Term Storage

[Tackhead]

> The Library of Congress is attempting to answer this question as they have huge amounts of media that is on highly degrading (nitrate-based films) materials.
> Their answer? A huge RAID array starting at 180TB and growing steadily over time.

Last time I looked, one Library of Congress was only 10TB, and I bought a 100G drive for $100.

So my rig sported a cool 0.02 LoC in my rig. I felt gr8. I mean, I 0wn3d.

Now you're telling me I only have 0.00055555 Libraries of Congress? I f33l s0 l4m3.

Bastards.

Re:Long Term Storage

[Tackhead]

> What are some examples of uses of this long-term inactive storage? I can understand like maybe a one-year archive, but you can keep that on an active raid array if need be.

For home use, pretty obvious - MP3z, DiVX movies, and TiVO shows. Re-downloading the MP3z and pr0n^H^H^H^HDiVX movies would suck because you'd have to find and re-download the missing files. Likewise, re-recording the TV shows on your PVR would be a pain because you'd have to wait for your cable company to air them again.

You'd keep the RAID array up at home, and that'd be your media box.

But you'd still need a couple of 120G drives down at the local bank in the event of a fire at home. Or better yet, at your Grandma's house 100 miles away, if you live in an earthquake-prone area.

And while I've mentioned the time cost of re-downloading, this risk really isn't about piracy - even if you own everything on your media server, you can't re-encode your CD or DVD collection if said collection has been transmogrified into a melted lump of goo or shards of polycarb.

Every time you visit Grandma, or once every six months, whichever is less frequent, you swap drives.

(Just remember to pack the drives securely when you drive to Grandma's. And drive carefully. RAID won't protect you if all the drives in the array go sailing through the window at 60 mph ;-)

Re:Long Term Storage

[Directrix1]

Also, the obvious answer to the authors question is buy 900 trillion flip-flops and hook them up to the wall outlet with a old rusty pair of bronze wires, feed the flip-flops through 900 trillion lines coming from a decoded IDE output in realtime. Weeeeh! Or you could take a top suspended in a vacuum with magnets on the edges of it. And for every bit you want to encode on it, you have an electrical field deliver an impulse of (2^bit number)*(bit-value)*(whatever unit you want) to the top in the vacuum. Later when you want to decode it, just start at the largest bit number and provide an opposing impulse and if the direction of the top changes then return a 0 otherwise return a 1. Simple, infinite data storage, fairly easy implementation, all you need is something to monitor the speed and make corrections when necessary (or observe the amount of friction and have it do periodic adjustments). You see how easy this is?

Re:Long Term Storage

[soulsteal]

So my rig sported a cool 0.02 LoC in my rig.

This post brought to you by the Redundant Department of Redundancy Department.

Re:Long Term Storage

[Reziac]

It also won't help you if despite all your care in packing the drives, one suffers a head crash enroute, even tho you don't remember hitting any bumps. (I've had that happen.) Nor if the drive decides that being inactive for a few months is cause to lose all the data. (I've had that happen too. Several times, with HDs that had been perfectly reliable in regular service.)

While "another HD" is probably the only practical backup for today's BIG drives, I personally would only trust that as a backup if it's powered up and running in a stable location.

Which isn't very helpful in this discussion.. but I think I'd make that "swap time" more like 3 months. In my experience, *if* a HD is going to lose data just from sitting around, it does so starting at around 6 months of idleness. Those that don't have the problem seem to keep data more or less forever, but (other than Conner HDs, which could be counted on to have the problem) I haven't found it's something you can predict in advance of the event.

A lot of folks will say....

[ajs]

that disks will rot, so you can't trust them.

I counter with this: tapes rot too. In fact, any tape older than one year that I've had to go back to has been worthless (read: it had deteriorated data).

Tape is a really bad medium to trust, but we keep buying it because we can't think of a better solution. Personally, I think the way to go is just to give up and admit that disk is not cheap. You need to back up your data to a live mirror system with identical storage (hourly rsync does a nice job) and then you need to arrage a service that can back up your data to remote live mirror systems. Note that in both cases I said "live mirror". You don't want a backup sitting on a cold box because you never know the quality of it until you need it.

The remote backup part is expensive, but it's the only reliable way. You seed it by tape (full backup to tape, and mail them to the vendor) and then use dedicated lines to keep a regular incremental update going.

If one of those two backup systems fail you know about it right away and you fix it. No more tapes rotting on a shelf only to be discovered when your data goes south.

Re:A lot of folks will say....

[Chrisje]

Yes, tape will rot. As will anything that is magnetic.

DDS tape has a guaranteed data retention period of 2 years, but then you may face head alignment problems if you replace the drive. DLT and LTO have data retention periods of 5 years approx. Head alignment problems don't form a problem because of the nature of the mechanism.

This is however not the point. The point it that a harddrive is not an ARCHIVAL medium. Neither is tape. Harddrives are the work horses for on-line data and tape is meant as a BACKUP. Backup meaning a copy for safe-keeping under a very limited time (ie next week, when tuesdays tape is run again, or... well, you get the point... ).

CD's (CD-R(W)) offer a theoretical data retention span of 20-100 years depending on who you ask. So that is safer, but still not perfect.

A Service Level Agreement with a maintenance company would do the trick too, but is expensive.

But why archive? Doesn't an automated backup to a tape robot with a weekly rolling schedule combined with a RAID 1/5 solution for your single disk failures satisfy your needs? What is so damn important that you need Off-Site ARCHIVAL rather than off-site backups?

With the falling prices of both tape and disk cost per megabyte, it's affordable to keep all relevant data on the drives of the server and then do backup to tape if needed.

Just my 2$c.

Re:A lot of folks will say....

[Jerf]

On a smaller scale (personal), this is essentially what I do.

First, only some personal data is critical, not the GBs of operating systems and programs I can redownload/recompile if necessary. Things like documents, saved games (you'd think it's unimportent until you play the first 2/3s of Fallout 2 five times and can't stomach getting far enough to see how it all turns out, because you'd have to play that 2/3s again...), email maybe, whatever, but some limited amount. 10MB can go a long way... that's a lot of programming, for instance. (Been working on a project for about half a year now and I'm just ready to break 300KB of code...)

Then, set up a live backup amounst all the disks you have on various machines. I use unison so that I can change files in the repository on any machine and have the changes propogate correctly, instead of the unidirectional updates rsync does.

Use symlinks to put everything you need into one directory, and tell Unison to follow the symlinks, not archive them directly. Then just run that every so often on the machines, and you're set.

Once more of my family gets set up with always-on connections, I intend to set up a family-level repository of backed up files with Unison, so that "off-site backups" are a weekly script run without intervention by the family, making off-site backups across the state (or country, or world) easy. This will protect the scanned pictures and other things in the family heritage easily and effectively.

Which reminds me, the first always-on connection just came online and I really ought to talk to that member about a reciprocating backup setup...

Re:A lot of folks will say....

[Havokmon]

Personally, I think the way to go is just to give up and admit that disk is not cheap. You need to back up your data to a live mirror system with identical storage (hourly rsync does a nice job) and then you need to arrage a service that can back up your data to remote live mirror systems.

Note that in both cases I The remote backup part is expensive, but it's the only reliable way. You seed it by tape (full backup to tape, and mail them to the vendor) and then use dedicated lines to keep a regular incremental update going.said "live mirror".

I agree wholeheartedly. Though, I would note, that IDE is the perfect solution for your redundancy. All you need is space. It doesn't have to be the fastest, or the highest quality mirror. Buying 20 IDE drives and having half of them fail is still cheaper than high capacity SCSI. Do a RAID 50 (IIRC, two RAID 5's - mirrored) offsite, and use rsync to mirror your data over your Inet line. Or string your mirror. Have your 'backup' offsite RAID rsync off the primary offsite RAID. I'd bet the only people who would have problems with that are the ones doing heavy graphics.

Check out Rackspace for your offsite needs, I didn't think they were that expensive, at least compared to an actual archival facility. Pick your favorite encryption method to secure it. Hell of a lot cheaper than a point to point.

Those people yelling 'insecure' apparently don't have an issue with their data being driven all around town. You want banking info? Just steal the grey box out the the '80 Ford Escort. OTOH, A 'man-in-the-middle' attack requires just that. So, if possible, host at your own ISP.

Tapes *is* the right medium for long term backup

[MooRogue]

I'm sorry, but 220GB easily handled by backup tape. With SDLT and AIT tape capacities exceeding 100GB per tape, two tapes can easily handle your load.

If you have the budget, get an autoloader so you can perform a full backup in one session, or two tape drives for that matter.

Personally, i am backing up 600+GB onto tape and it works well. I've had numerous IDE hard disk failures, yet not a single data tape failure so far.

Ask who's actually doing it.

[f2professa]

So, how is Pixar archiving it's film data? How about LucasFilm? I'd think from the amount of data they work with, thos guys would be the best at answering that question.

Personally, for long term storage, I'd go with redundant backups of differing media. Maybe hard drives (stored properly in anti-static bags with silica gel), as well DLT stored in a similar fashion. Increase your odds of support by future architecture.

For daily backups, hard drives are surely the way to go. Faster, cheaper, easy to replace, longer lasting media in my opinion. Anyone who says otherwise is trying to cover their job as a tape changer. ;-)

Re:Ask who's actually doing it.

[DJPenguin]

Well, don't know about LucasFilm, but Pixar use massive tape libraries (we are talking robots with 100+ drives and tens of thousands of slots.)

Incremental backups every HOUR, tape drives spinning all the time. They are a customer of the company I work for. (Veritas)

Slashdot - the "Jackass" of tech support

[HotNeedleOfInquiry]

Here's some more questions:

Can I use my laser printer to print on Gummy Bears?

Can I dry my cat in the microwave?

Can I put rice in my car radiator?

Can I unplug all the fans in my computer so it will run quieter?

Can I run 120 VAC on the spare CAT5 pairs?

Re:Slashdot - the "Jackass" of tech support

[HotNeedleOfInquiry]

Sorry, I only had 5 minutes to come up with something. The boss is stalking around looking for slashdot goof-offs like me.

Re:IDE ?

[Gudlyf]

"because in 5-10 years from now, IDE may not even exist anymore..."

In that case, you could always just buy a new, cheap system for the purpose of reading the IDE disks, and keep that in the vault with the drives "just in case".

I'm not saying this idea with backing up to IDE is a good idea, though. Drop a tape on the floor while you're running to the tape drives for a critical restore, no biggie. Drop a drive on the floor in the same situation, you'd better hope your resume wasn't one of the files needing a restore.

Eggs and baskets

[phil reed]

If the tape drive electronics fails, you can get another tape drive and still read the tape. If the IDE drive electronics fail, the data on the drive is unreachable without massive and expensive intervention.

Crappy backups better than nothing

[jolshefsky]

I don't know how "pro" you want to go with this, but I ran into a similar situation and resigned myeslf to the same solution. My DDS2 SCSI tape drive is getting to be too small at 4/8GB. I would like to have a tape solution, but it's too expensive for my purpose. I get drives as pulls and last-years-models so I only spent US$150, but with tapes at US$10, even 8GB is absurdly small. If I were to go with new equipment and step up to DDS-4, I'd be out about US$1000 for the drive and another US$20 for each 20-40GB tape. Total cost for a basic 3-tape rotating backup: US$1060.

On the other hand, I could spend (as I have) US$40 on a basic (a.k.a. el-cheapo) FireWire-IDE case, US$30 for 3 removeable IDE enclosures, and (eventually) about US$70 each for 3 60GB IDE drives. Total cost: US$280.

What do I sacrifice? Not much ... one of the drives might fail. At that point I'd just replace it with another US$70 capacity drive (which would probably be larger.) If I needed to restore something from backup, I'm already looking at up-to 24-hour old data, and if that drive happened to die, possibly 48-hour ... it's unlikely that all the drives would fail at once.

The advantages? I can use the US$780 I save for something else and I don't have to worry about shelling out another US$1000 every four years just to scale to "current" requirements. I don't know what the upper limit of an IDE drive is these days (i.e. what can the ATAPI bus handle) but even 200GB is pretty big for me right now.

Anyway, just a few thoughts. The basic thing is lower cost for nearly the same risk ... tapes fail too, you know. Remember, too, that this story would be very different if I had to handle 50 machines instead of 2.

May I recommend remote backup?

[ekrout]

Many people forget that remote backups require no on-site hardware or software and don't require you to spend hours upon hours configuring things.

Even better is that any flood, tornado, or fire at your house or business will not ruin your tape, dvd, cd, or hard drive backups. You simply connect to your remote backup location and restore your old data onto your new hardware. It's that simple, and it's cheap in comparison to spending $3,000 on a tape backup device that only stores 150GB of data per cartridge.

You may want to see if this remote backup company has services that fit your needs (I don't work for them, so it's not a plug). Basically, they state the following as the main appeals to remote backup:

Your data is continuously backed up as it changes, 24 hours a day, so it's always up to date. And it's stored electronically at Iron Mountain® data centers, where more than half the Fortune 500 protect their data.

No-Wait Recovery - Instantly recover your data to the point of failure, eliminating downtime and data loss from relying on a previous night's backup. And a unique web interface allows you to initate restores from any Internet browser, anywhere.

No Tapes, No Hassles, Lower Costs - Tape-less backup and recovery means no hardware or software to buy and a fully automated process requiring little employee time or resources. Lower your data protection costs while freeing IT resources for other tasks.

Why are you backing up?

[Cap'n Canuck]

I know it sounds like a stupid question, but why are you backing up data? What are you trying to solve

Short term failure
A luser makes a mistake, or there's a glitch in last night's source code library, and all your current data is foobarred. In scant minutes, you can restore lost data from overnight backups, (or even hourly incrementals), and you are the hero. Realistically, you're just doing your job, and you'll never get thanked for it.

Complete Failure
In the event of a building fire/server room flood/earthquake/Act Of Dog, then you may need to retrieve all your companies data from as near back as possible. This backup should be off-site, and as frequent as feasibly possible

Long Term storage
This is for archiving of a project, etc, and should be off-site. Also for archiving source code in case your company goes belly-up, so that customers can still use and modify your software (in escrow).

Ask yourself which scenario you are dealing with, then the answer as to which media is the one to use may be clearer.

Tapes are NOT a long term archival medium.

[silentbozo]

Tapes are fine for backups, but I never expect to pull complete and usable data off of them after 6 months. Why? Tape degrades - it's nothing more than rust on platic. As humidity and temperature change, you can end up with a solid roll which will stick to your tape drive heads and result in whole patches of magnetic coating coming off. I worked on a project restoring data from 10+ year old reel-to-reel tape, and it was a nightmare. 1 out of 4 tapes was completely unusable.

Even worse, tape drive formats keep changing - and since tape drives are guaranteed to wear out, where are you going to get a working tape drive to restore data 5, 10, 15 years from now? I've gone through 3 tape drives in the last 8 years - thank god I got a CD burner early, that data I can still read (although it's about time to start recopying stuff from 1996.)

Basically, if you entrust your data to tape long term, you have to continuously copy that data to new tapes, and or new tape formats. Where tape has traditionally shined is as a short-term backup format, although with the drop in DVD-burner drives/media, and the high-cost of high-capacity tape drives/media, this may no longer be the case (assuming you get some peon to do the big backup on DVDs, and you get to do daily diffs - otherwise, having a bank of tape drives is cheaper on staff time.)

Re:Tapes are NOT a long term archival medium.

[Burdell]

Tapes are fine for backups, but I never expect to pull complete and usable data off of them after 6 months. Why? Tape degrades - it's nothing more than rust on platic.

So use a reliable tape format and store it properly. When stored properly, DLT has a shelf life rated in decades.

Even worse, tape drive formats keep changing - and since tape drives are guaranteed to wear out, where are you going to get a working tape drive to restore data 5, 10, 15 years from now?

So use a tape format that is backward compatible. Today's SDLT drives can still read all the old DLT formats.

Where tape has traditionally shined is as a short-term backup format, although with the drop in DVD-burner drives/media, and the high-cost of high-capacity tape drives/media, this may no longer be the case

Check the shelf life of CD-{R,RW} and DVD[+-]{R,RW}. Most of the CD/DVD media is only rated for a five year life at most. Mastered CDs and DVDs will be readable for decades, but burned CDs and DVDs won't be.

The bigger problem with really long term backups is with the data format used by the backup software. If you use a backup program that only runs under Windows, what are you going to do when you need to recover that data in 10 years, and you only have Linux (or the other way around, the point still stands)? This is where Open Source software is good, because (assuming you can still find the source) you can always decode the data stream.

Just copy it around

[photon317]

The "right" way to make your data reliable is with mirroring of various sorts. On-site backups are kinda silly except when you're using them operationally because you dont have the disk capacity to do otherwise for infrequently used data. Backing up to removable media should be exclusively for offsite storage.

So get two drives and mirror your data, and you're covered in the case of drive failures. If your worried about a whole machine going up in smoke, maybe do a nightly or hourly rsync to another machine across the room.

If your home data is important enough to need offsiting (usually a home user's "important" data amounts to what could fit on a CDROM, not 220 gigs - the rest is probably multimedia fluff that you can stand to re-encode or download in teh case of a tornado or fire), then consider rsyncing with a freind at night over your DSL or cablemodems in a mutual arrangement. Encrypt the data before syncnig it over if it's sensitive.

If you're a business with large volumes of data that need to be offsite in case of disaster, then the best practice is still tape drives of some sort, and an offsite storage service like Iron Mountain.

Re:Why would your disks be

[Sloppy]

And I don't believe 220 gigs of IDE space is cheaper than 220 gigs on tape.

That's how it used to be, but you're out of date. Take a look at the numbers; it's shocking and no one would have believed it ten years ago. From a $/byte perspective, tape is becoming obsolete, right about ... now. Ignoring the cost of the tape drive (which is pretty high for large-capacity tape), both are at about a dollar per Gigabyte, and disk price has been improving much faster. A year from now, tapes will cost more than the disks they back up.

About all tape has going for it over disk, are physical robustness issues (the lack of the "stiction" problem that he mentioned, the fact that dropping a tape onto the floor is less scary than dropping a disk, etc).

Re:Tapes *is* the right medium for long term backu

[Drakantus]

"I have $500 to spend on a backup solution for my 220GB data pool, and I was thinking of buying 4 120GB IDE drives along with an IDE RAID1 card and useing the array for backups, anyone have other ideas?"

"No way, you are insane. IDE is horribly unreliable and you will surely lose your data. You need a $6000 tape drive, if you can't afford it you are better off with no backups at all"

Re:Tapes *is* the right medium for long term backu

[glesga_kiss]

I've had numerous IDE hard disk failures, yet not a single data tape failure so far.

You speak of not having tape failures, but you omit one important fact; how many times have you successfully retrieved data from tape?

IDE disks will fail from continual use, and that failure will generally be obvious, but what way do you have of knowing that you genuinely don't have any tape failures, if all you are doing is rewriting over the same tapes?

the absolute surefire way to back something up...

[rebelcool]

i thought of this the other day. It's kind of amusing, but it would work for absolutely important data.

Get alot of archive quality, acid-free paper. Get a printer with alot of archive quality ink and print out the data in binary. Dots or slashes would work fine for the 1's and 0's.

Archive quality paper and ink lasts for hundreds of years. Should you lose the data on a magnetic or other storage medium, you could always run these papers through a scanner with some OCR and retrieve the data.

Sure, a fire or flood could damage these if you don't have them protected against that, but at least you won't have to worry about deteoriation of the medium.

Perfect Storage Medium

[techsoldaten]

For my clients, I always suggest the use of stone and / or clay tablets for all mission critical data archive projects, regardless of size or scope. Bablyonian and Greek models of data retention from as far back as 4,500 years ago are (in many cases) superior to the models we commonly use today, with much of the physical meadia having survived electrical storms, tornadoes, floods, fires, and wars on every scale imaginable with a data corruption rate of zero and without the benefit of a climate controlled room, dedicated security staff, or even a closet for media storage. Imagine the elegance of a 84'3/4 STROM (Stone Tablet Read Only Memory) machine hooked up to your Slackware Archive server for performing restorations, and the ST Binary Writer you have networked to your backup systems and kept physically over by the quarry... nice! The TCO for slab is far less than that of tape archives, considering you can store the media in a pile of mud and hose it down when you are ready for a restoration.

M

Oh yea?

[FreeLinux]

Seems to me that you should use the most modern solution out there. You want off-site storage and you want redundancy and you might like it to be distributed.

Sounds like P2P would be the ticket here. Just upload all your files onto Kazza and Gnutella and then let nature take its course, scattering them all over the internet.

Anybody see a problem with this? Seems like a "legal" use for P2P has finally shown up.

Re:the absolute surefire way to back something up.

[tomhudson]

Lets have some fun :-)

And just how many tons of paper are you going to need to reliably back up a terabyte in dots and dashes?

Assuming double the standard density (160 chars per line instead of 80, 132 lines per page instead of 66), which actually works out to quad density, you get 160x132=20120, say

1. 20k per page
2. 50 pages = 1 mb
3. 50k pages = 1 gb
4. 50m pages = 1tb

Now let's assume boxes of 5000 sheets. 10,000 boxes, at, say 20 pounds a box = 200,000 lbs, or 100 tons. Man, give me the toner franchise for this!

Five Points About Archiving

[maggard]

1. Accept that you can't just stick magnetic media on a shelf (in a vault, even climate-controlled) and expect it to last forever.

   Bits rot. Under the most perfectly controlled environment the damn stuff still goes bad. Be realistic, anticipate this, do everything you can to slow it down, but plan for it and make provisions when you first put your archiving strategy in place. Tapes are likely more robust the platters as there's fewer critical parts to go wrong but nothing is perfect.

2. Accept that CD & DVD don't have 100-year lifespans, mebbe not 10 year, and possibly far less.

   Yes they're cheap but we've far less experience with these media then we do with tape and studies are showing that they dyes may not be as stable as first thought. Heck, there's even a bug out there that eats some of these. There's also the question of long-term standards in some cases like DVDs.

3. Checksums and multiple-backups (that reinforce eachother) are a necessity.

   Nothings worse then losing one part of an archive at one site, another part at a different site, and being unable to easily reconcile the two to get a good whole set. Make sure that however you archive things, same media or different media, that partial archives can be reconciled.

4. Everything evolves - Keep updating backups.

   Years ago there was a big scramble to recover the US Govt's 1950 Census. It had been stored on steel tape and the required Unisys readers were no longer. (Much of the data was available but the entire raw set wasn't.) Eventually a working one was built from cannibalized parts in museum and private collections but the lesson was clear: Don't depend on the readers. The same goes for the recent BBC Domesday Book debacle - nobody could read the optical disks. Any good archive scheme will call for the material to be re-read and re-transcribed regularly in order to ensure the entire recovery-chain still works: Hardware, software, OS's, etc. If recovery becomes difficult migrate the material.

5. Be pragmatic about what you archive.

   All too often folks archive everything 'cause they're too lazy to determine what is actually necessary and what isn't. Combine this with the difficulty of later having someone unfamiliar try to winnow down the material and this becomes a real problem. Even worse is later trying to find the useful material among all of the dross. Establish clear policies of what can be archived and make folks justify their material. Just as importantly make sure the costs are clear up front, even to the point of charging them a rate covering several years of storage initially. Suddenly some pack-rat deciding EVERYTHING they've ever typed is potentially a goldmine isn't so funny. Lastly, run everything past Legal: Some of this they don't want hanging around any longer then necessary.

Re:Um you've pretty much answered your own questio

[Bobulusman]

I have a 20 mb (yes, you read that right) hard drive from 1989 that I can still read just fine. I've hooked it up once or twice over the years just for the nostalgia.

Re:Non-volatile: no such thing

[Alien Being]

"accidentally drop it and have it shatter"

Moses: I bring to you these fifteen [crash], ten, ten ommandments.

You could always ...

[Greedo]

Stegnographize your data and hide it in an amateur pr0n video.

To restore from backup, search with Kazaa.

Re:the absolute surefire way to back something up.

[rworne]

As long as we are on that track, the Internet was designed to withstand nuclear attack, so its obviously the best choice: archive, encrypt and have others mirror your data.

I know, I know, how do you get these people to do it? And how much will it cost? Easy, and I can get them to do it for free.

Name the backup DIVX_The_Twin_Towers.avi and put it up on Gnutella or WinMX. Problem solved.

Some advice

[Monkelectric]

Let me first go on record and say you are a complete fool if you think this will work ... Bite the bullet and buy a 100gb native DLT drive. At my last job I backed up 2.6TB on a DLT+autoloader, I know 220 gigs *seems* like alot of data, but you're small time.

However, if this is going to have *any* chance of working, you will need to read the drives on a regular basis. I would pop each drive in a machine and (in linux) do a "dd if=/dev/hdc of=/dev/null" to read the entire drive. I would do this monthly.

Why you ask? Because modern hard drives are sophisticated and they auto-correct errors *before* they become a problem. Hard drives will do things like correct recoverable errors and rewrite weak sectors when they encounter them. Thus if you go over every sector of the drive every once in awhile, you will use the drives auto-correction features to your advantadge (and protect against the drive fading, which would be my primrary concern, not stickage (which is easy to fix)).

Re:Has DLT tape ever worked consistently?

[geekoid]

SOmething is wrong with your system, or something is happening to the tape. I've done a lot of work with DLT, and your failure rate is way out of proportion.
I would regularly, I mean several time a day, move a tape from system to system for testing purposes.

Re:Why would your disks be

[Mysticalfruit]

Obviously you haven't purchased any DLT tapes recently...

Lets just say you go with 40GB DLT tapes...

220/40 = 5.5 DLT tapes to back up your data.

DLT tapes cost 50 bucks a piece. 6 tapes * 50 bucks = 300 bucks just for the tapes.

Oh yeah, now you've gotta buy a DLT drive as well... and if you plan on doing any real backups your not going to sit there and load 6 tapes in succession into the drive so your going to need a library of some kind. So, tack on 5000 bucks for a library... I'll make the assumption that your using a some free archival software, otherwise you'd have to tack on some big money for that as well...

So... 5300 dollar tape solution vs. 500 harddrive solution...

You choose...

Some REAL long-term archival media

[phr2]

The next best thing to rock and chisel.

Re:the absolute surefire way to back something up.

[BeBoxer]

But is printing a whole character per bit, or even byte, efficient? I'm curious how much data a laser printer could store on a piece of paper. Is it realistic to expect individual bits printed at 300dpi to actually be retrievable? Perhaps on a good 600dpi or 1200dpi printer.

300dpi gives us almost 11KBytes per square inch. Figure 70 square inches on a letter page with 1/2" margins. That's 770KB. Print full duplex and you're looking at 1.5MB per page, or roughly a floppy disk (coincidence?) You wouldn't want to back up your MP3 collection, but for an archival method that is likely to last 100 years it's not too bad. Factor in compression and you are probably getting a 100x increase in storage density over plain text. Kind of a neat thought.

Not that new

[AppyPappy]

In the early 90's we spent $1500 for a 3 gig drive that we used to back up our workstations. We then backed up that drive to tape. It was infinitely faster than screwing with tapes in the night.

Right now I am backing up 53 workstations to a hard drive file using Retrospect. I then copy the file to another server and backup that server. Somewhere, I will have a copy of those backups because it exists on two machines and a tape.

Re:the absolute surefire way to back something up.

[ncc74656]

And just how many tons of paper are you going to need to reliably back up a terabyte in dots and dashes?

If you were actually going to produce some kind of machine-readable dead-tree backup, it's more likely that you'd produce a type of 2D barcode that could be scanned back in and read. Assuming an 8x10" grid at 200 dpi (the remaining area can be used for alignment and checksumming), you could get about 390K per page (single-sided...you could also double that by making it a "flippy," and you wouldn't need a notch-cutter :-) ). You're still looking at a little over 5 tons for 1 TB, but it's an improvement. 200 dpi should be well within the abilities of currently-available laser printers and scanners. If you wanted to try 300 dpi, you'd more than double your capacity and get about 879K per page (single-sided).

The cheapest, and most long lasting backup.

[teamhasnoi]

The oral tradition! Have many children, give them each 10 pages to memorize. To make things easier, you can name them Sector 237, Cylinder 13004 and such.

As disk space grows, so does your family/backup.

To see examples of how this works see: Mad Max - Thunderdome, The Bible, American Indians, The Fellowship of the Ring, Aesops Fables, and the Legend of How the Great Nog Vomited the Earth and Heavens in Ancient Times, Before the Oceans Drank Atlantis.

I have heard rumors that this is how Google archives.

Re:Sod CD-R! Go With DVD recording

[ibennetch]

And for keeping tabs on what is on which disk... I've been using a freeware program called "Cathy" (I don't have any links)...Although I don't know whether it'll do DVD's, I haven't tried.

Cathy is avalible for download here. According to these sites it will handle many disk formats ("CD-ROMs, LS120, Iomega Zip and Jaz disks, or even diskettes"). The link to the home page is broken.

Re:the absolute surefire way to back something up.

[2nd Post!]

But each of your 20k per page can easily encode a unicode value, which means you can cram 2 bytes per spot, or only 50 tons per terabyte.

But how about a 600dpi laser printer, 8"x10"?

For good readability, we can use:

***
**
*
*
**
***

For (1,0) which gives us 3 dots per bit, or 200 bits per inch. A square inch would then give us 40,000 bits, or 5,000 bytes. A sheet of 8x10 then gives us 400,000 bytes. Or if you tweak the margins, 400k per page. So that's already 20 times your density. Increase the resolution to 1200dpi, and you can increase the data density to 1600k per page.

We can also use different encodings: Right now we use 9 bits to encode 1 bit of information (really, really, redundant). We can probably safely use the following encoding to double our data density:

***

***

*
*
*
*
*
*

So this further gives us 2 bits of information in the same 3x3 square, which increases our data density another 2fold: 800k or 3200k per page. At 1200dpi, that's 3mb per page, so that 1gb == 333 pages, and 1tb == 333k pages. 67 boxes, or 134 pounds per terabyte.

There are more variations of course. We can increase density to 4 bits per 3x3 square. With a bit of thought, we can also increase the density up to the theoretical limit of 2^9 values in a 3x3 square, but we want to include some leeway for data redundancy...

So by doubling to 4 bits per square, we require only 70 pounds per terabyte. By doubling again to 8 bits per square, That's down to 35 pounds.

That much (little) paper... is actually lighter than a terrabyte of digital storage!

Re:the absolute surefire way to back something up.

[tomhudson]

Unfortunately, the so-called "archival" papers, while "rated" for 100 years, won't last anywhere near that long without some degradation. Then, if you're going to store it that densely, you've got to make allowance for putting the data into "tracks", so you have to leave spaces between each row. Cuts your 300 dpi down to, say, 100. Add check-summing data, so that you can recover from dirt, toner falling in the cracks, etc. And now, let's make the dashes twice the size of the dots. Cuts your storage by another 50%. Now, let's put spaces between the dots and dashes - otherwise, you get one LOOOONG dash. Your 11kb per square inch is now less than 0.5kb per square inch. Oh, and don't do duplex printing, you'll have transfer of toner onto the drum from the previously-printed side. Net result == about 30kb to 50kb per page... Oh well, maybe we should try microfiche ... or bit-encode the data into fake avi files and record them on VCR tape - cheap media for sure.

Re:the absolute surefire way to back something up.

[schmink182]

To take this a little farther, a helpful reference tells us some useful information.

2000 sheets of 8-1/2 x 11, 20# laserwriter paper weighs 20 lbs.
First of all, this changes your estimate of weight from 100 tons to 250 tons.

Typical yield of paper: 125 lbs per tree
250 tons (500000 lbs) divided by 125 lbs per tree gives us 4000 trees.

440 trees per acre
This, after division, gives us 9 acres of trees destroyed for backing up 1 TB of data. Seem worth it? :)

Re:the absolute surefire way to back something up.

[Jucius Maximus]

"As long as we are on that track, the Internet was designed to withstand nuclear attack, so its obviously the best choice: archive, encrypt and have others mirror your data."

Just encode your data into a pr0n video and share it on gnutella. That data will never be 'lost' !

Re:Tapes are a expensive waste of time

[JWSmythe]

Burnt CD's (like you'd use at home) have a shelf-life of about 10 years. Then the medium starts to oxidize (the metallic film, not the plastic itself), and flakes..

So, you have a 10 year backup.. It all depends on how important your information is. If it's that important, I'd put it on a RAID5 where it can be monitored. As drives fail, replace them. Continue migrating to newer arrays in the future.. Expensive, but I konw perfectly well any drive will fail. I've had several hard drives, that would fail to spin up properly after sitting for a few days.. Some of them, they only way they'd start is if I hit the side of the drive with a screwdriver..

You have to expect failure of your medium. If he wants to be very sure, use multiple backup methods.. RAID5's in multiple locations, and CD's. Someone will need to monitor all of it occasionally. Make sure the RAID's (and their associated machine) are running. Make sure the CD"s are oxodizing...

Even floppy disks die of old age. I found a few boxes with Novell Unix. They're is years old, and most of the floppies couldn't be read. They were brand new, still in the sealed boxes and envelopes. I finally found a boot disk that would work, but it would bomb out trying to install under VMWare (I was curious).

Is that data really going to be useful to you in 10 years? That's the important question. People are all paranoid of loosing Email and the like now, but in 1 year they don't care about it any more. In 2 years, it's just wasted space. In 10 years, they won't even know who or what they were talking about..

Re:the absolute surefire way to back something up.

[default luser]

And if you're really clever, you would take advantage of the fact that levels of greyscale are easily discernable. Leave a seperation space on all sides of each dot ( so they're more easily decoded ) to form a grid system. Yes, your storage capacity will drop by a factor of 4, but you can easily encode 8 bits ( a factor of 256 ) into the dot.

Most laserprinters can do 8-bit greyscale.

But for redundancy:

- Make two dots for each 8-bit piece of data, the 8-bits and it's complement. This is only good at error detection, although theoretically you could add error correction at a capacity cost.

- Add 256 calibration dots every few inches to make up for aging of the ink and media. We can assume that the cameras will have much higher resolution than the printer, so they can tell the difference even if the levels have faded together.

You could pack a whole lot of data on paper if you put your mind to it.

My experience with long-term IDE

[inode_buddha]

Point 1.
Make sure you select a very well-made drive, don't cut costs there. Example: I have a 20-year old Mountain HardCard that still works fine. However, I have had cheap 3-year old drives fail.

Bringing up point 2:
If you try it, make sure to use an "exercise" schedule for all the drives in your backup set. For example, once a week for each drive, plug it into a spare box and ensure that it spins up, spins down, and the read/write arm travels its full sweep. Maybe do some read/writes at various places on the platter surfaces, just to be sure.

It works for me, so I hope this helps.

Tower of Babel here we come

[xipho]

Not to be a naysayer...but I will anyways. What happens in 30 years when a massive electromagnetic field wipes out all digital machines (possibly in conjuction with some attempt by humans to wipe out the robots taking over the world...those damn robots!)? By then 15 years of scientific publication may be more or less completely digital, and all gone, gone. Better hope we never lose access to that handy-dandy resource electricity....

Ten year old data

[Eric Green]

I actually have a lot of data that is now 16 years old, including the source code (6502 assembly language) for a BBS program that I wrote as a kid. The secret: Regular migration of data to newer/larger media. From 1541 floppy to Amiga via serial port and xmodem, from Amiga to Linux via serial port and uucp, and on Linux, periodic moving of the data to newer hard drives as I upgrade my systems. I also now maintain a copy of my data in CVS, so that if something gets accidentally erased or changed, I can retrieve a copy. My CVS archive, too, periodically gets moved to newer/larger/faster hard drives.

And to top it all off, I back it all up to a DDS-4 DAT autochanger. Yes, those six tapes will only hold 120gb, but the amount of important data on my disk drive is far less than 120gb (it is actually less than 20gb, including the original 44.1khz .wav recordings of all my original songs, and fits onto one tape easily).

Do you *REALLY* need a backup of your .mp3 collection?! Probably not. Do you *REALLY* need a backup of all those ISO CDROM images that you downloaded for fifty versions of Linux and a half dozen versions of FreeBSD? Probably not. But that's the sorts of things that are taking up 80gb plus on my hard drives -- i.e., utterly disposable cruft. Which is true for most personal computers.

Actually thats the real reason i thought of it

[rebelcool]

Awhile back I was brainstorming up some ways to preserve current mediums which only exist in digital or magnetic forms. Like audio and video. Suppose you want to preserve a specific bit of video for viewing 2000 years from now. Not only do you need a medium for storage that will last that long, but you must deal with the certainty that no equipment will exist capable of playing it.

So one way would be to both preserve a general specification of how to read the data, and then the data itself. So not only would you need a method of encoding the song onto paper, but you'd need to include the details of an algorithm - simple enough that people whose language may be very different from ours - can recreate it using their machines of the time. And then they can feed the data into it, and replay the music/video/whatever as we intended it to be seen.

Re:the absolute surefire way to back something up.

[Yarn]

Laser printers do gray scale by dithering, you lose resolution. Good idea though. Better storage medium would be black/white photographic film like microfiche.

Here's your backup script

[Ardeaem]

--Begin #! /bin/sh mv $1 /dev/null End-- Benefits: 1. No worrying about media 2. Saves space Drawbacks: 1. May be difficult to get your data back 2. No GUI (yet)

PaperDisk

[cameldrv]

www.paperdisk.com claims that they can get either 660K or 1MB depending on resolution on a sheet of paper. How long a piece of paper will last when encoded with this density is unknown, but with good paper I'd bet it's a hell of a lot longer than any disk. Furthermore, even at that density, there's a huge ammount of physical redundancy in the data storage. If the paper gets to be fifty years old or so, I would imagine that the technology would be available to cheaply scan at ultra-high resolution to compensate for any degradation.

RAID-1 plus drive rotation

[Phil Karn]

A while ago I got tired of swapping DAT tapes during full backups of ever-bigger disk drives, and of having to minimize my use of the system while they ran. I also got pretty tired of repeated hard drive failures, as I had purchased a couple of those jinxed IBM hard drives made in Hungary.

So after a brief look at hardware RAID I realized that the software RAID support in Linux was all I really needed. Since this is my own machine, I didn't really need the hot-swap capability of a hardware RAID controller.

I bought two 100GB Western Digital drives and set them up in a RAID-1 configuration. A month later, I bought another drive, replaced one of the drives in the machine with it, and put the removed drive in the safe. A month after that, I bought another drive and repeated the process, this time moving the drive in the safe to an off-site location.

Every month or so I repeat the process, rotating the second drive of the array through my various offline storage locations. The real beauty of this (especially vs tape) is that I only need enough downtime to swap the drives and reboot the system; the mirror reconstruction runs in the background as I use the system normally.

The use of RAID-1 gives me complete protection against data loss in the event one of the online drives fails (though I've had no failures yet with the WD drives). If both drives are somehow ruined (e.g., by a fire within the computer), or if I accidentally delete something important, I have my first offline backup, less than a month old. If that's also ruined (e.g., my whole house burns down and the fire-rated safe fails to protect the drives it contains) I have my off-site drive, which is less than 2 months old. Obviously I could easily extend this process with more drives and more offsite storage locations.

Because the backup drives are regularly rotated into online service, bearing stiction should be less likely to occur. And if an offline drive were to fail when I bring it back into service, so what? It was about to get overwritten anyway.

Naturally, I also continually back up especially important files (e.g., email, work projects, documents, etc) to various machines over the network, as that's the easiest and most effective way to protect small amounts of data. But when it comes to periodic full backups of big disks, nowadays I just don't see any practical alternative to disk-to-disk copying. And RAID-1 is the easiest way to do that copying.