How Do You Backup 20TB of Data?

Sean0michael writes "Recently I had a friend lose their entire electronic collection of music and movies by erasing a RAID array on their home server. He had 20TB of data on his rack at home that had survived a dozen hard drive failures over the years. But he didn't have a good way to backup that much data, so he never took one. Now he wishes he had.

Asking around among our tech-savvy friends though, no one has a good answer to the question, 'how would you backup 20TB of data?'. It's not like you could just plug in an external drive, and using any cloud service would be terribly expensive. Blu-Ray discs can hold a lot of data, but that's a lot of time (and money) spent burning discs that you likely will never need. Tape drives are another possibility, but are they right for this kind of problem? I don' t know. There might be something else out there, but I still have no feasible solution.

So I ask fellow slashdotters: for a home user, how do you backup 20TB of Data?" Even Amazon Glacier is pretty pricey for that much data.

Comments

reduce the amount

[JeffSh]

At home, I didn't feel like paying for 2 large arrays to store my data, so if I rip any media, I always rip it to DIVX. 800 MB for a DVD or even bluray rip is a great economy, saves me money on primary storage and also enables me to back it up. I accept the loss of quality as I can always reference the original media if I want.

Another option in the future may be subscription services which have HD content, thus eliminating my need to roll my own. We'll see what happens there.

Re:reduce the amount

[ncc74656]

RAID-5 uses up 1 disk worth for striping, so net space in an 8-drive array is 7-drives worth (about 27TB using 4TB drives). The problem with RAID-5 is that you are 2 disks away from failure and rebuilds often kill the disks.

RAID-6 uses 2 disks worth for striping, so net space in an 8-drive array is 6-drives worth (about 23TB using 4TB drives). Is able to survive a double-disk failure before data loss. Still has some of the same issues as RAID-5.

I use Greyhole for media and document storage. It handles disks of unequal size (currently running one 3TB and two 1.5TB drives), and you can choose the level of redundancy you need. In my case, movies, TV shows, etc. get a single copy (one file exists on one drive), while documents and photos get two copies (one file exists on two drives). If a drive goes bad, you only lose the files on that drive...and only for the files for which you selected no redundancy. With redundancy, extra file copies are recreated on the remaining drives from the surviving copies; this process is most likely less stressful on the disk set than a RAID rebuild.

My movies, TV shows, and music are backed up to BD-R, stored in a binder at work. They hold ~20GB each, as I'm using dvdisaster to guard against media errors. When a 2TB drive failed, I brought the backup (currently about 190 discs) home and restored the files that had gone missing. Backup and restore are managed by scripts, with information about what files are on what discs held in a MySQL database that gets periodically backed up off-site as well. The initial backup took several months (on and off) to finish, and the last time I needed to restore, it took about a week, but now I just burn a disc when I have about enough new data to fill one. Burning and verifying takes a few hours, but it's something you can start and walk away.

Build another server

[EmagGeek]

If you want to back up 20TB of data, you have to pay for it.

Build another server and rsync hourly.

Amazon Glacier

[uiucgrad]

I use Glacier and its great. 20 TB is about $200 a month which to me does not seem like all that much money for backing up that much data. The biggest problem from a home users perspective is getting all of that data to Amazon. Hopefully he lives somewhere where fiber is available to his house.

Re: Crashplan

[Anonymous Coward]

Crashplan offers unlimited storage, yes, but they limit it indirectly by slowing down uploads.

I recently paid for a crashplan account to back up ~6TB of media, and at the speeds I'm seeing the initial backup is going to take more than a year. I have 100Mbit/s fiber at my home and can max it easily with other services.

So for 20TB, it's going to take many years to back up. I don't think that's a practical backup solution. There's a decent chance you're going to lose your data before the initial backup completes. And if crashplan goes under, you have to start all over again with the next "unlimited except for rate" provider, and have no backup in the meantime.

Re:Hmmm...

[ewhac]

I had only hard of LTO tapes quite recently, and I did a very tiny bit of poking around. The latest generation is LTO-6, whose tapes can hold 2.5TB each (uncompressed). The tapes themselves are quite modestly priced -- an LTO-4 tape cartridge (800GB uncompressed) costs about $30 each.

The drives, however, are not cheap. New drives appear to start at around $1200. Used drives are all over the place -- I've seen some on eBay with an opening bid as low as $350. Also, all LTO drives appear to have either an LVD SCSI or a SAS interface, which means you'll also need a controller card. There appears to be no such thing as a SATA LTO drive.

Plus you get to re-live all the joys of selecting tape vendors, and placing bets on whose tapes are going to last for 20 years.