Ask Slashdot: Best Offline Storage Method For Large Archives?

An anonymous reader writes "I have a collection of large projects (Indesign files with associated images), which are typically 40GB to 60GB each. In this current climate, what is the 'best' method of archiving these? Spinny magnets? Solid state drives? USB? Tape? Blu-ray? All have pros and cons and price considerations. If I remove the price issue (my data is important to me), does this change the choice?"

Comments

Rotational media

[BWJones]

For this project, we have multiple multi-terabyte (5-18 terabyte) datasets that need backup. We have online and offline strategies and the offline strategy is simply multiple, redundant copies on hard drives stored in static proof containers onsite and off site.

Hard drives are *very* cheap all things considered, are easy to store, take up very little physical space and if things go badly, restoring from them is faster than just about any other method. For datasets in the GB range, its a no-brainer to go with hard disks.

Re:Rotational media

[Mad Merlin]

I concur on this point, online storage really makes the most sense. Cheap, high performance (for sequential read/write) and easily expandable. You can get a single machine with dozens of SATA drives in it (including the drives) for way under 5 figures. When drives fail, they're simple to replace, and every couple years, migrate the whole thing to newer (faster, bigger) drives. Mirror your data unless you don't care about it. RAID 1/10 for really small datasets (2-4 drives), RAID 6 for moderate size datasets (5-10 drives) and RAID 60 for anything bigger.

A very important note to keep in mind... stay away from hardware RAID! When your controller dies, so does all your data, unless you have an identical spare controller card (buy it up front, they won't exist in a couple years). The same goes for fake RAID (ie, software RAID driven by the BIOS), but s/controller card/motherboard/g;. Pure software RAID (ie, using mdadm) is a safe bet.

Re:Rotational media

[lucm]

> and a _good_ storage controller such as a xyratex.

I would rather run Windows Home Server on a RAID-0 of IBM DeathStars installed in a HP Pavilion than deal with a Xyratex.

Re:Rotational media

[Keruo]

> Online storage makes no sense.
In storage, online means the data is connected and instantly available(harddrive etc) vs offline(dvd,tape etc)

Re:Rotational media

[vegiVamp]

Fully concur, and let me sum it up: The best type of long-term storage is "redundant".

Large removable disk on the low end, tape highend

[mlts]

BD-R disks are an idea, and relatively inexpensive, but your best bang per buck would be large removable disks in the 2-3 TB range. The reason I state "disks" plural is for obvious reasons.

I would also use a program like WinRAR with a recovery record, or one of the PAR utilities used for USENET to store your files in. This way, you can tell if there was file corruption, and have a good chance of recovering from it.

For serious stuff where money is less of an issue, I'd consider a LTO-5 tape drive and multiple tapes. Tapes tend to last longer than HDDs because they have very few moving parts.

Don't forget to see about copying your archives to new media every couple years. It isn't uncommon to be able to pop a 10+ year old tape or HDD in and pull off the contents... but it isn't uncommon either to find the HDD clicking, or the tape full of hard errors.

Bare Drives via Hot Pluggable Trayless SATA

[bynick]

Screw tape... you pay $2,000 USD for the drive, $50+ per tape for a couple of hundred gigs. Go with bare drive external: Install a trayless SATA bay for 3.5" hard drives... this will run you $12. Buy some bare SATA drives.. these run $50 for 1TB and are available up to 3TB. I buy bare drive hard cases for about $3 each. My Intel ICH10R on-board RAID controller supports hot-swap -- so in effect it's a big 3.5" floppy.. that's right. If your tape drive breaks, you're out another two grand. This is far less expensive, faster, higher density, and random access. In addition, you can boot from it. Want RAID0? Install two trayless SATA bays for a total of $24 and back up in pairs.

We really need.....

[pcjunky]

Eternally Yours, The case for the development of a reliable repository for the preservation of personal digital objects.

http://explorer.cyberstreet.com/CET4970H-Peterson-Thesis.pdf

Tape/climate control

[triffid_98]

You can't argue with Tape. It's been proven to last since the 1960's if kept in a climate controlled space (dry/cool). Just make sure to keep a spare tape drive handy (just ask NASA), because spare parts for 40 year old tape drives are surprisingly difficult to locate.

Optical isn't even close, assuming you're talking burned discs. Taiyo Yuden claims a 70 year shelf life, but they have only been around for what, 8 years tops?

Hard drives are an option if you've built a redundant array, but even with that you're still going to be out of luck if you burn up your raid controller.

Torvalds quote

[bmo]

"Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it" - Linus Torvalds

--
BMO

Dedup or Tape

[lucm]

If price is not an issue, a great solution is to go with a data-deduplication device (such as EMC DataDomain or IBM Protectier). If you were to host one unit in your basement and the other in coloc environment far from your home, you could setup replication and have a very reliable archive. Coloc of a 1U device can be quite cheap, I have one of them for which I pay less than 100$ a month.

If you have a smaller budget, then the best cost-benefit is still found on tape, and it can even work in case of network disruption. Like Andrew Tannenbaum said: "Never underestimate the bandwidth of a station wagon full of tapes". A single LTO-5 tape is very cheap (50-60$) and can store 1.5TB (can easily double that with dedup).

There are other interesting technologies out there, such as MAID, which you can use as a VTL with a good backup software to maintain a reliable archive, however cheap disks are cheap and in a MAID configuration they might not last as long as typical disks because of the on/off behavior.

Re:BitTorrent hash check

[QRDeNameland]

Yeah, that's a great solution if all you want to do is detect corruption, but note the GP's point about havng "a good chance of recovering from it". The only way to recover with BitTorrent is to have another copy available to replace any bad blocks. PAR2, on the other hand, is able to recover any random missing X% of data from a dataset as long as X% of PAR2 data was generated.

Real solution

[jcoy42]

There are only 2 real solutions if you want real long term storage. The first is you become Linus and just dump it on a server and let the rest of the world back it up, and the second is you make your data a religious text somehow. Because those guys with translate it for centuries to come, even if it means sitting 50 dudes in a room for 3 years with nothing but a feather, ink, and parchment.

come to think of it, same thing.