Slashdot Mirror
How Do You Backup 20TB of Data?
Sean0michael writes "Recently I had a friend lose their entire electronic collection of music and movies by erasing a RAID array on their home server. He had 20TB of data on his rack at home that had survived a dozen hard drive failures over the years. But he didn't have a good way to backup that much data, so he never took one. Now he wishes he had.
Asking around among our tech-savvy friends though, no one has a good answer to the question, 'how would you backup 20TB of data?'. It's not like you could just plug in an external drive, and using any cloud service would be terribly expensive. Blu-Ray discs can hold a lot of data, but that's a lot of time (and money) spent burning discs that you likely will never need. Tape drives are another possibility, but are they right for this kind of problem? I don' t know. There might be something else out there, but I still have no feasible solution.
So I ask fellow slashdotters: for a home user, how do you backup 20TB of Data?" Even Amazon Glacier is pretty pricey for that much data.
Asking around among our tech-savvy friends though, no one has a good answer to the question, 'how would you backup 20TB of data?'. It's not like you could just plug in an external drive, and using any cloud service would be terribly expensive. Blu-Ray discs can hold a lot of data, but that's a lot of time (and money) spent burning discs that you likely will never need. Tape drives are another possibility, but are they right for this kind of problem? I don' t know. There might be something else out there, but I still have no feasible solution.
So I ask fellow slashdotters: for a home user, how do you backup 20TB of Data?" Even Amazon Glacier is pretty pricey for that much data.
I would say use floppies, but I'm kind of old and out of touch now.
Crashplan has unlimited storage. I use their home plan; it's unlimited for up to 10 machines. I think I am backing up about 6TB there now.
I have a 16 TB media collection at home that I just back up on more hard drives.
External hard drives in USB cases + Robocopy works great for me.
I don't respond to AC's.
"My friend (read I) lost 20TB of pirated content! What should my friend have done different?"
How about, ask yourself, how much of that content were you intending to ever consume again. Yeah, you can most likely delete 95% of it, that's 1TB of content that you might use again.
Hoarders! *lol*
He could have always bought a sufficiently large tape-library from ebay - but I guess the data wasn't worth that much.
That's always the first pair of questions to ask: how much is it worth and how much would it cost to recreate?
If the answer is somewhere between "I don't know" and "Well, it's not that much", then he just should stop hoarding that much stuff.
He could have built a filer with ZFS and sent daily snapshots to a 2nd filer - but that wouldn't have helped him if the house burnt down...
Windows 2000 - from the guys who brought us edlin
Figure out the theory of everything.
Then you can always recompute your data from scratch.
If Pandora's box is destined to be opened, *I* want to be the one to open it.
I do the exact same thing with high res pictures. I immediately will take the full resolution raw image and convert it down to a 320px gif. Or maybe a 10% quality jpeg. You get great economy that way too. Who wants to keep a 30+MB image around when you can have almost the same thing in 10kB instead!
You could always just call up the NSA and ask them to restore the data. Odds are good they have a copy of it...
Whenever you buy storage, you should buy the necessary backup capacity at the same time. You should never buy storage without buying backup capacity. Budget for it right from the start. If you can't afford the backup, you can't afford the storage. This may mean getting half as much storage as you'd like, but that's just the way it has to be. You probably wouldn't buy a car without an engine. It wouldn't do its job. So don't buy storage without backup. If you do, you have a storage system that can't do its job.
I agree, I've been using Crashplan for three years and the unlimited space it's really great BUT... ...I'm not sure about the bandwidth they provide: how long it will take to upload 20 TB?
Anyway, I don't see what's the problem in using external drives for backup. Here in my lab I've realized that the best way to backup X Terabytes is to have another storage with X Terabytes...
It's not like you could just plug in an external drive, and using any cloud service would be terribly expensive. Blu-Ray discs can hold a lot of data, but that's a lot of time (and money) spent burning discs that you likely will never need. Tape drives are another possibility, but are they right for this kind of problem? I don' t know. There might be something else out there, but I still have no feasible solution.
Lets start from the top: You *can* plug in an external drive, it's called a complete hardware duplicate of your array (or perhaps for space/cost consideration, a single disk based copy held offline and synced regularly). Not hard and not terribly expensive (i would go with this solution personally). Cloud? Yep the bandwidth and storage even on something like Amazon Glacier would be prohibitive to all but the most financially independent geeks. Bluray doesnt hold enough (even at 50gb/disc you need 400 of them, groan). So, tapes? You bet your ass tapes are designed to do exactly this task, why do you think they are still in use? You can get individual tapes at 1/1.5TB, but for a one man operation they are probably going to cost you more than the first solution (offline spinning disks) and they are a pain to manage properly.
Now what is this doing on ask slashdot? A pencil, some scratch paper, and 15 minutes between amazon.com and newegg.com would tell you the prices of every solution. Oh, right, they need a chance to tee up some targeted ads for Carbonite, Mozy, Crashplan, etc.
These "unlimited" claims always turn out to be lies. When will we learn?
My friend paid for an "unlimited" account from JustCloud for backup. He stored 1.8 TB on it and then they "fair use"'d his ass and canceled his account. They didn't even give him a refund for the rest of the money he prepaid.
Connect a raspberry pi and configure it as a backup server and let it copy all to /dev/null... ...
Then put aside the money you would have invested in a "better" solution, put it in a safe bank (under your mattress)
and wait until you need to restore something..
Most probably you'll enjoy the money more
Just assuming that your friend had a fully legal collection, I would think that all he needs to do is ask the media companies for a new copy. Because the media industry tells us that we do not buy music, we buy licenses, right?? So even if we lose the bits-and-bytes which are easy to replace, then we still hold a license and the media companies should facilitate that your friend can exercise his licensed rights..
[/sarcasm]
To Terminate, or not to Terminate, that's the question - SCSIROB
If IBM punch cards were used, 1 GB equals approximately 47 cubic yards (assuming 80 bytes per 187x86x0.18mm per card) and about 70,000 lbs (at 2.42 g per card), so one standard railroad boxcar (limited by both cubic capacity and weight) could hold about 3 GB. 20 TB would need over 6000 boxcars of punch cards; at 60 feet per boxcar, that's a freight train about 70 miles long.
I wasn't looking for it to go anywhere really other than pointing out the absurdity of saying that taking a bluray rip down to a 800MB divx rip results in just an acceptable loss of quality.
I'm by no means a audio/videophile snob, but you either have a blind and/or deaf if you can't see a MAJOR quality deficiency with a 800 bluray rip. What's the point of having a bluray movie if the first thing you normally do is make it look like crap?
To /.ers saying that 1TB+ tapes would be a good idea to do this backup, please:
Add some references and price of such hardware and media that would suit best home usage.
The dataset isn't that huge. Tape can write at speed at least as fast as disk - LTO-5 writes at up to 280MB/sec - far faster than you can read the source at which isn't likely to be fast disk. The seek for a single-file restore will be slower than disk but after the initial seek, the read will be as fast as from a typical archive disk (no, you're not archiving 20TB to SSD, nor are you storing the source data on SSD either)
However, the change rate for this application is likely to be low. That makes it very feasible to do random testing from the new backups where a minute to do the tape mount/seek is not a problem. You won't be writing more than a single tape in any single run (LTO-5 is ~1.5 TB of uncompressed data).
For $2K, you'll have the LTO-5 drive. Add $500 for 20 tapes and you can back up the entire set (once) plus a bunch of incrementals. I haven't done the math with LTO-6 which is faster and holds more data. If you want multiple generations, tape is a lot cheaper per TB than disk. The initial drive cost hurts but after that, the price is good at $15/TB or so.
Now this joke has really come full circle.
The problem with RAID-5 is that you are 2 disks away from failure and rebuilds often kill the disks.
No. The problem with RAID-5 is that during a rebuild, there is a reasonably possible chance you could have a UBE, and lose one bit, making perfect recovery of the array impossible. Only a stupid controller would consider a UBE to be a failed drive and trash the entire array. On RAID-6, you still have the same possibility of a UBE, but the chances that two separate drives would experience one on the same exact block during a rebuild are so astronomically slim as to be irrelevant.