How Do You Backup 20TB of Data?

Sean0michael writes "Recently I had a friend lose their entire electronic collection of music and movies by erasing a RAID array on their home server. He had 20TB of data on his rack at home that had survived a dozen hard drive failures over the years. But he didn't have a good way to backup that much data, so he never took one. Now he wishes he had.

Asking around among our tech-savvy friends though, no one has a good answer to the question, 'how would you backup 20TB of data?'. It's not like you could just plug in an external drive, and using any cloud service would be terribly expensive. Blu-Ray discs can hold a lot of data, but that's a lot of time (and money) spent burning discs that you likely will never need. Tape drives are another possibility, but are they right for this kind of problem? I don' t know. There might be something else out there, but I still have no feasible solution.

So I ask fellow slashdotters: for a home user, how do you backup 20TB of Data?" Even Amazon Glacier is pretty pricey for that much data.

Comments

If you want to hoard bits...

[polymeris]

> It's not like you could just plug in an external drive [...]
Why not? Maybe not one, but 10 or 20 of them.

Re: Don't bother.

[binarylarry]

Exactly, do you really need to hoard all that content?

Do something about your hoarding problem

[Anonymous Coward]

"My friend (read I) lost 20TB of pirated content! What should my friend have done different?"

How about, ask yourself, how much of that content were you intending to ever consume again. Yeah, you can most likely delete 95% of it, that's 1TB of content that you might use again.

Hoarders! *lol*

Re:Do something about your hoarding problem

[ustolemyname]

Not all of us have access to the time machine required to know *which* 1TB that is.

Are you willing to share yours?

Re:Do something about your hoarding problem

[LordLimecat]

This problem isnt unique; most people have trouble curating their data. That doesnt change the fact that the problem is mostly self-created, and the best solution isnt to find another place to stuff the 20TB. Its to take the time to cull it down to a reasonable size and then back it up.

Don't hoard

[rainer_d]

Were those 20T of original movies and music or just stuff he downloaded via bittorent?

He could have always bought a sufficiently large tape-library from ebay - but I guess the data wasn't worth that much.
That's always the first pair of questions to ask: how much is it worth and how much would it cost to recreate?
If the answer is somewhere between "I don't know" and "Well, it's not that much", then he just should stop hoarding that much stuff.

He could have built a filer with ZFS and sent daily snapshots to a 2nd filer - but that wouldn't have helped him if the house burnt down...

Plan for backup before you buy

[Ktistec Machine]

Whenever you buy storage, you should buy the necessary backup capacity at the same time. You should never buy storage without buying backup capacity. Budget for it right from the start. If you can't afford the backup, you can't afford the storage. This may mean getting half as much storage as you'd like, but that's just the way it has to be. You probably wouldn't buy a car without an engine. It wouldn't do its job. So don't buy storage without backup. If you do, you have a storage system that can't do its job.

Re:reduce the amount

[edxwelch]

> I always rip it to DIVX. 800 MB for a DVD or even bluray rip is a great economy
I do that as well, but I found out to my horror that all my DVD's had become unreadable over time. So, probably good idea to test your backups from time to time

Hilarious

[jeffmeden]

It's not like you could just plug in an external drive, and using any cloud service would be terribly expensive. Blu-Ray discs can hold a lot of data, but that's a lot of time (and money) spent burning discs that you likely will never need. Tape drives are another possibility, but are they right for this kind of problem? I don' t know. There might be something else out there, but I still have no feasible solution.

Lets start from the top: You *can* plug in an external drive, it's called a complete hardware duplicate of your array (or perhaps for space/cost consideration, a single disk based copy held offline and synced regularly). Not hard and not terribly expensive (i would go with this solution personally). Cloud? Yep the bandwidth and storage even on something like Amazon Glacier would be prohibitive to all but the most financially independent geeks. Bluray doesnt hold enough (even at 50gb/disc you need 400 of them, groan). So, tapes? You bet your ass tapes are designed to do exactly this task, why do you think they are still in use? You can get individual tapes at 1/1.5TB, but for a one man operation they are probably going to cost you more than the first solution (offline spinning disks) and they are a pain to manage properly.

Now what is this doing on ask slashdot? A pencil, some scratch paper, and 15 minutes between amazon.com and newegg.com would tell you the prices of every solution. Oh, right, they need a chance to tee up some targeted ads for Carbonite, Mozy, Crashplan, etc.

Re: Hmmm...

[MightyYar]

Well, it's local storage to a computer somewhere. If that computer can run Backblaze, then super.

Re:Hmmm...

[sdo1]

Which, if it takes about 1 minute to load each one, it will take you a mere 243 years to do the backup.

Re:Crashplan

[Anonymous Coward]

I would agree with CrashPlan, although I have never ever ever seen it upload faster than 3Mb/s. Uploads are slower than hell, but it does indeed work.

Re:reduce the amount

[cdrudge]

I wasn't looking for it to go anywhere really other than pointing out the absurdity of saying that taking a bluray rip down to a 800MB divx rip results in just an acceptable loss of quality.

I'm by no means a audio/videophile snob, but you either have a blind and/or deaf if you can't see a MAJOR quality deficiency with a 800 bluray rip. What's the point of having a bluray movie if the first thing you normally do is make it look like crap?

Please post Tape backup ref

[advid.net]

To /.ers saying that 1TB+ tapes would be a good idea to do this backup, please:

Add some references and price of such hardware and media that would suit best home usage.

Re:Please post Tape backup ref

[Anonymous Coward]

Stop ! Stop ! Stop ! Stop .... Seriously STOP !

There are a few comments here that are really trying to help, and most are wasting time. This is a real problem, it's a real problem we all have - even if we don't think too hard about it. I bet that lots of you have a usb stick with a 100% backup of some critical files... perhaps even multiple copies ( 200%, 400% coverage )

To solve this problem - you have to understand what your requirements are ( surprising eh - but nobody is talking about it ). It's clear that most of the comment have no idea about managing large data volumes, and they clearly don't understand backup solutions.

Here are some questions - the set of answers to these questions decide which backup solution is optimal for THIS use case ...

What 'backup' problem are we trying to solve ?
- A single spare copy ? Multiple copies ?
- Is all the data in the backup 'equal' ?
- How often does the data change - or is it mostly static ?
- How much data loss from the primary source are you willing to accept ? ( In otherwords how closely does the backup have to match the live data set )
- If you lose the primary source ; how quickly do you need the data set back ? ( Can you backfill the dataset while continuing to work on the live system, or are you hard down until a full restore has completed ).
- How long does a backup need to live ? ( If you do a fresh full backup every week, and you write it to a biodegradable wafer, and it dissolves in 6 months ; but you have copies for every intervening week - do you care that the 6 month old version is nolonger readable ) ... OR do you need to restore from a year ago, 5 years ago, 10 years ago (think financial records))

Figure out the answers to those questions - and you might be a ways towards deciding what solution you actually need.
There are commercial software systems (and opensource ones like amanda/zmanda) that understand the data management lifecycle - and are setup to support solutions like 'intermittent full backups ; with daily/hourly incremental snapshots'. . but then there are the filesystems that support snapshotting. Look at the features that the full service solutions provide - they might prompt the conversation to help you understand what you actually require.

So - if you have 15TB of mostly static data (movies etc. ) - then you can archive that to long term media.
If you have 5TB of changing data ( music library, playlists, new mp3s, new movies ) - then you can snapshot that using the filesystem hourly , and back it up to the cloud weekly (with incrementals) - and then dump it to long term archival every 6 months..

Where the (15TB, 5TB, 6 months) values in the above statement are completely made up - and are not the ones you should use unless I guessed right.

None of the 'use tape, use cdr, use punched paper, use lasers to etch it on the moon' solutions will work if you don't first understand the problem you are trying to solve.

This isn't hard - but if you don't know what you want, then every solution is the wrong one and too expensive

Re:reduce the amount

[Artraze]

Agreed.

Regardless of whether or not 20TB is hording / excessive / inefficient, what it almost certainly is is replaceable. Let's face it, you aren't CERN, most of you data is probably media that you can reacquire with relative ease. It's not being stored because it's irreplaceable it's being stored because it's convenient. A RAID isn't too bad, but add in managing backups and where has that convenience gone? If it costs $10+/month to backup your ripped/downloaded movies, why not just sign up for Netflix?

Just make a list of all the replaceable data (e.g. videos you have the original disc for) you have and then buy an external hard disk / Blurays to back up the rest. If you lose your RAID, well, it'll be annoying to rebuild, but you built it once... (Besides, I doubt you could restore 20TB over residential internet less time!)

Re:reduce the amount

[jedidiah]

If you've already got one large array then you are already by definition half way there. If you then decide not to go the rest of the way then you are at the same time being both extravagant and a cheap bastard. It's a wonderfully stupid paradox.

If you've got one then you should get the 2nd one or not bother with the first one to begin with.

Re:Hmmm...

[Anne Thwacks]

LTO6 tapes hold 6.25 TB, so you probably only need 4. Hardly fills a station wagon - the basket of a push-bike would do fine.

Of course, you might want to make more than one copy of your data. https://www.amazon.co.uk/dp/B006K1FSKA