Ask Slashdot: What's The Best Way To Backup Large Amounts Of Personal Data? (foxdeploy.com)

An anonymous Slashdot reader has "approximately two terabytes of photos, currently sitting on two 4-terabyte 'Intel Rapid Storage' RAID 1 disks." But now they're considering three alternatives after moving to a new PC: a) Keep these exactly as they are... The current configuration is OK, but it's a pain if a RAID re-sync is needed as it takes a long time to check four terabytes.

b) Move to "Storage Spaces". I've not used Storage Spaces before, but reports seem to show it's good... It's a Good Thing that the disks are 100% identical and removable and readable separately. Downside? Unknown territory.

c) Break the RAID, and set up the second disk as a file-copied backup... [This] would lose a (small) amount of resilience, but wouldn't suffer from the RAID-sync issues, ideally a Mac-like "TimeMachine" backup would handle file histories.

Any recommendations?
This is also a good time to share your experiences with Storage Spaces, so leave your answers in the comments. What's the best way to backup large amounts of personal data?

Comments

Commit it to memory!

[danomac]

Memorize it! Just don't take any head injuries or you won't remember anything.

More seriously, back up to hard drives is the only viable option. Then make sure you have more than one backup drive and store one at some other site. Relative maybe?

Cloud options with that kind of storage would take forever to upload. And I've heard of people having stuff randomly go missing on their cloud service, not the entire contents, but a file here and there. I'm not so sure that's a good option.

For storing on-site you can get a fire rated media safe, but they can be quite a bit more expensive than a regular safe.

Re:Commit it to memory!

[danomac]

In addition, I forgot the 3-2-1 backup principle. 3 copies of data, on at least 2 different types of media, and 1 copy off-site.

Re: Commit it to memory!

[Anonymous Coward]

FreeNAS with RAIDZ2 and daily snapshots turned on, synced to an offshore duplicate located in a remote site if your internet connection can handle it or located at your neighbours house if not.

Re: Commit it to memory!

[techno_dan]

I found FreeNAS to be the best and cheapest solution. Was not that hard to set up, and like you said, snapshots are great.

Re: Commit it to memory!

[UnknownSoldier]

I'll 4th that as well. FreeNAS + RAIDZ FTW !

Also, make you sure you:

* run memtest on your RAM
* run the manufactures' provided sector diagnostic utility over the hard drive BEFORE inserting into production.

Re:Commit it to memory!

[BoRegardless]

There are "UL fire rated" safes for paper documents, but those will destroy hard drives and those are pretty pricey.

Data storage safes have to be more expensive yet.

Re:Commit it to memory!

[KozmoStevnNaut]

+1 for the 3-2-1 principle.

I've got the local copy on my PC, which is manually synced to a NAS and automatically backed up to Google Drive. Probably not 100% ideal, but at ~$10/month for 1TB, I'm quite satisfied.

Re: Commit it to memory!

[RockDoctor]

duplicate located in a remote site if your internet connection can handle it or located at your neighbours house if not.

Not if your neighbour is in the same building as you (what we call here "terraced" or "semi-detached" housing), no. I used to keep my offsite backup at my friends house, about 15 minutes walk down the road. Or maybe a work colleague who lives some distance away.

Otherwise, the 3-2-1 principle works fine.

Re:Commit it to memory!

[yevp]

Hey! I wrote a blog post about that once -> https://www.backblaze.com/blog...

Come the fuck on

[Anonymous Coward]

2 Terabytes is nothing.

Here's how you do this:

10 You buy an external hard disk that is 4 Terabytes or larger, and USB 3.0.
20 Copy the fucking files to that thing.

You're done. Now you have two copies: one on whatever bad idea you have as your main drive, and the other on a physically separate drive.

Not good enough? GOTO 10

Re:Come the fuck on

[spire3661]

You forgot checksumming and verification after transfer.....You have something on the other drive after the transfer, you wont know what until you verify it.

Re:Come the fuck on

[ShanghaiBill]

2 Terabytes is nothing.

Indeed. You can buy a 1TB USB thumb drive on Amazon. So all this data will fit on two of them. If all your data will fit in your pocket, with room left for both your cellphone and wallet, then it is not "large amounts of data".

Re:Come the fuck on

[Anonymous Coward]

Are you serious? What a sloppy response. Relies on a manual process (therefore prone to human error, forgetting, inconsistent method of copying, etc). Doesn't scale if the RAID array grows larger than the biggest USB3 disk you can buy.

Re:Come the fuck on

[AmiMoJo]

Bad idea, because it requires on-going effort. Most people will forget, or get lazy.

For most people encrypted online backup is the best option. I use Spideroak (I took up the unlimited space special offer, about £100/year), but there are others. It's automatic, happens constantly in background. I've got over 4TB on Spideroak, only took a few months to upload. Obviously you need a reasonable upload speed and no/high data caps.

Re:Come the fuck on

[Blrfl]

And when your house goes up in a fire, all of your external drives go with it.

Re:Come the fuck on

[joe_frisch]

Agreed! The minor changes I would make (and do for my own few TB of files).

Have a script the runs the backup. I use rsync on linux.
Make two copies, one that mirrors, one that just adds files.
Use two backup disks, always have one at a remote location (your work) so you don't lose data in a house fire.

If it is a single command (my is "backup") then its easy to remember to do every week.

I actually have 3 backups. One at home. Two at different work sites that I cycle through. I do my backups from a linux machine that doesn't provide write access to my main windows machines. That makes me a little more resistant to hacks (since they would have to hack two different OSs.

Re:Come the fuck on

[Chelloveck]

What he said. And for ongoing backup, keep the disk at a buddy's house and rsync your files to them periodically. And reciprocate. Keep their backup disk at your place and let them rsync to you. Done. You're safe and you've made the world a better place.

Although I imagine that our "anonymous Slashdot reader" who asked this question wouldn't know rsync if it bit them on the ass, being the marketing person for Storage Spaces and all. Come on, the only purpose of such a fucking obvious question is to get some front-page name recognition for the product. Nice timing, too, slipping it onto the feed Sunday night, ready for everyone's Monday morning Slashdot-and-coffee ritual. Kudos.

Re:Come the fuck on

[CAOgdin]

Despite your vulgarity, there are another options: I always keep one of my three backup drives in the trunk of my car (the second is standby, the first is connected, and I rotate them regularly). The car is almost always with me, and the first thing I'd do in case of fire is to get the car out of the garage.

If that's not adequate, you can rent a cheap safety deposit box at your local bank. I did that for years, especially when the data included a lot of client data on them (I'm now retired).

Re: Come the fuck on

[Blrfl]

Uh, 'scuse me there, Sparky... I didn't set the title for this thread. Please see the complaint department operated by Mr. Anonymous Cowherd above.

Re: Come the fuck on

[Blrfl]

And I'm well aware of the other options, thanks.

Re:Come the fuck on

[phoophy]

And then buy another external drive, 4TB or larger, and do it again. Preferably a different brand of drive inside, preferably a an enterprise storage drive. And keep it off-site.

Re:Come the fuck on

[mlts]

As others have said, 4TB isn't that much. The key is to have a 3-2-1 plan for the data -- 3+ copies, 2 on different media, one offsite:

First, I'd recommend purchasing a NAS appliance. Synology and QNAP offerings are inexpensive and even though one can build their own system with FreeNAS or something else, a small NAS appliance takes up relatively little in wattage, which is nice for the electric bill. I also like the fact that you have the ability to encrypt data, and segment it into shares. Some NAS models even allow for snapshots. They are not too expensive -- an ARM based dual-drive NAS is about $150 + drives.

For four terabytes, I would recommend a Synology DS216+ ii (the reason for the long name is that the DS216+ had components which were discontinued, so the mark 2 edition is current. This NAS model is x86 based and can use btrfs to detect bit rot on the RAID volumes) Then, drop in two WD Reds (6 or 8 TB), and you have RAID 1.

Second, buy an external USB drive to plug to the NAS. RAID and snapshots are nice, but this provides a true backup mechanism.

Third, get an offsite backup mechanism. QNAP and Synology have software that can back up to a number of providers, and back stuff up encrypted. There are many offsite backup providers out there.

Fourth, consider a manual offsite mechanism, even if it is another external hard drive that you plug in, dump the contents of the NAS to, remove, and put offsite somewhere. This way, if you lose your NAS and Net connection, you still have some means to access your data.

Re:Come the fuck on

[Anonymous Coward]

Unnecessary for the most part. A direct HDD -> HDD file transfer will have plenty of error checking on the part of the operating system and drivers/firmware during the copy. If you don't trust a simply copy procedure then you're gonna be stressing every single time you use a computer, it's unnecessary worry.

Now, a backup using tape, or a backup using some container format that isn't as transparent as basic files on a filesystem I can understand some extra testing is warranted.

Re: Come the fuck on

[Oligonicella]

Then why would you presume someone else wouldn't be and use them? We are, after all, speaking of people wanting to have backups of their data for later.

Re:Come the fuck on

[demonlapin]

Bank safety deposit box FTW: I've got one that holds plenty of stuff, it costs $40/year. Could easily put ten hard drives in there without even limiting the space.

Re:Come the fuck on

[demonlapin]

I use Syno + offsite. Have a DS412+ (old now, I know) but that's what was available when I bought it. I populated it with two WD Reds and two Seagates, but both Seagates have failed (one in warranty, one not) - so it's WD from here on out unless I find some strong evidence like Backblaze saying there's a better choice.

Re:Come the fuck on

[Wolfrider]

--Best answer I've seen so far, except for the ZFS responses.

--My question is, how has Synology gotten btrfs to be "stable" when it's still considered to be "experimental" on a regular Linux distro? I've seen reports of people losing all their data on btrfs and still consider it to be at *least* 3-5 years behind ZFS.

https://wiki.archlinux.org/ind...

https://lwn.net/Articles/67681...

Re:Come the fuck on

[mlts]

From what I have read, btrfs is stable if used in a RAID 1 configuration. However, it seems that RAID 5... is a completely different story altogether. I'm sure it has gotten better, but as of now, I've not read anything about catastrophic data loss on anything but striped arrays.

Given that it appears to be stable, btrfs does have a few advantages, mainly being able to handle bit rot, as well as snapshots (which are definitely not backups, but they are another tool to help prevent data loss.)

If btrfs is a concern, one can run without it. The less expensive ARM based units don't have it as an option, but they do offer RAID (assuming a more than single drive model.)

All and all, NAS appliances have their use. Especially with offloading stuff that would take up resources on an active machine or VM. For example, having the appliance handle basic Git repositories is nice, as well as DNS caching.

Re:Come the fuck on

[The_Revelation]

Yeah, to be fair, Slashdot used to cover slightly more in-depth articles than "how to synchronize your puny data between more than one device" - I'm not sure how this article got through the /. firewall unless the admins just dug up a post they forgot to publish during the 90's. Ask us about PB/ZB+ backup and storage and things will start getting interesting. 2TB? They probably sell consumer flash disks with more available space.

Re:Come the fuck on

[hcs_$reboot]

the only purpose of such a fucking obvious question is to get some front-page name recognition for the product

Indeed the "Storage spaces" link he gives goes to one very specific product. Disgusting.

Re:Come the fuck on

[Z00L00K]

A cron job will fix that.

Re:Come the fuck on

[KozmoStevnNaut]

Well, that depends on whether you want to restore everything right now, or if you can pick and choose to only restore the most critical files first, and let the rest run in the background for a couple of weeks.

For me personally, my Google Drive backup is the last resort if both my PC and my NAS go tits-up at the same time, in a lightning strike or natural disaster of some sort. In that case, I don't need all of my backed up data restored instantly, I can start with the most important personal documents and leave the bulk data for later.

Of course, I'm only backing up personal documents and my music collection, if you're backing up major programming projects or something, of course you'll want the whole thing restored as quickly as possible. In that case, an off-site backup somewhere closer with a fatter pipe or a manual weekly disk exchange is probably best. Online backups take a lot less effort, though.

Re:Come the fuck on

[KozmoStevnNaut]

Keeping a disk in the trunk of your car and rotating them out regularly is a good idea, but it depends on the climate where you live. If it's generally nice and temperate, no problem. But if you live in an area that gets either very hot or very cold, that'll kill a hard drive right quick, especially if you don't watch out for condensation.

I appreciate the "my car is almost always with me" reasoning, though :-)

Safety deposit box is a good idea, too. Somewhat more effort involved, but of course that trade-off depends on how important your data is.

Re:Come the fuck on

[KozmoStevnNaut]

And it's not like data backup is the only use for a deposit box. Keep all your important papers and other things in there too, everything that cannot be kept digitally. The bank is a hell of a lot less likely to burn down or float away than your house is.

It may seem like common sense to a lot of us, but I'm sure someone out there is going to go "oh yeah, good idea!"

Re:Come the fuck on

[KozmoStevnNaut]

AFAIK, Btrfs can do the same thing, ie. spanning data over multiple disks, while keeping each of them individually readiable. Unless I'm mistaken?

Re:Come the fuck on

[KozmoStevnNaut]

Probably one of the best options for a NAS is the HP Microserver. It's a full PC in a small form factor with 4 disk slots, it'll run any OS you'd like. Surprisingly inexpensive and lean on power usage, too.

One of those with FreeNAS or Rockstor, and you're ready to go.

Re:Come the fuck on

[KozmoStevnNaut]

It's HGST for me, unless I see some extremely compelling evidence of higher reliability from another manufacturer. I've had Maxtor, Seagate, WD, Samsung and host of other brands die on me, but never a HGST drive.

Of course, I've probably jinxed it now.

Re:Come the fuck on

[thegarbz]

Bad idea, because it requires on-going effort

Only if you set it up poorly. I'm not an expert programmer. Actually I'm a pretty damn terrible programmer who programs only with a minimum of 3 Google windows open explaining what the hell I'm supposed to type in, yet it took me about 2 hours (you guys could probably do it in 5 min) to setup my Linux machine so that whenever I plug my HDD in it automatically runs a backup script and emails the results to me.

This is also a feature of many home NAS systems include the really cheap and terrible ones. Plug in an external drive via USB, push the button on the front of the box and bam, duplicate.

If this is too much effort for you to protect your data then the only conclusion is you don't have data worth protecting.

They key problem with online backups is not so much the backup (unless you actually have large meaningful data like when I get home and have 60GB of photos to make a copy of) but the recovery process. You took a month to make the initial backup, would you be so patient when you don't have your data, and it's no longer a "background" operation?

Re:Come the fuck on

[pakar]

Bitflips on HDD's are more common than you think, especially when the data is not read (to allow the disk firmware to detect the degradation and rewrite the data).

Only way to keep safe from this is to use a checksumming filesystem + having more than 1 copy of the data.

Possible setups with a BTRFS filesystem: (probably possible with ZFS or any of the other filesystems with checksumming support)

Option 1: One HDD with two partitions. BTRFS setup as raid1 between the partitions and files are copied here. for history create a snapshot before copying the next round of data.
Option 1 protects against bit-flips and general corruption in the filesystem, but you are required to "scrub" the filesystem every time you do a backup..

Option 2: Two HDD's. BTRFS setup as raid1. The rest is the same as option 1.
Option 2 protects against one backup-device having a hardware failure.

Option 3: Two HDD's. BTRFS setup as raid1. The rest the same as option 1. After each backup you take one of the disks and store it offsite and add a fresh disk to the raid1. Swapping between 3 disks (one local, one in transit and one stored remotly) could be an idea too.
Option 3 protects against hw failure of the backup and at the same time protects against data-loss due to a breakin or a fire.

Re:Come the fuck on

[donaldm]

You forgot checksumming and verification after transfer.....You have something on the other drive after the transfer, you wont know what until you verify it.

It is usually a good idea to have a decent file-system on your backup disk. FAT and ExFAT are just not good enough and have you ever read the the disclaimer on NTFS?

Personally, I would rather use ext4 which is actually used for professional databases and data that you don't want to see corrupted. Of course, you are quite right in stating that you should always check your backup data by doing spot recoveries or checks using checksums such as md5.

I have actually seen so called professional backups that were written by, well "hacks" and while all the database infrastructure was backed up correctly the most important thing the database was not. Needless to say when the hard disk failed (yes they did not use RAID) it was very easy to get back the infrastructure although that could easily be got back from first principles but the most important part, the database was missing.

I always think the most important question to ask when people want to backup their data is "How much do you think your data is worth?"

Re:Come the fuck on

[sjames]

It depends on the features used. Regular old btrfs or the mirrored modes work quite well. DO NOT touch the RAID 5 mode with a ten foot pole or there will be tears. That is where the "I lost my data" stories come from.

As for being ahead or behind, ZFS has some advanced features that btrfs is lacking. The reverse is also true. ZFS treats snapshots as special while in btrfs, a snapshot is just a COW clone that happens to be marked read only. Btrfs is a lot more flexible about expanding the filesystem, especially in mirror mode.

Re:Come the fuck on

[ImdatS]

To add on this, if you are paranoid like I am:

- Install a RAID-System, where you have the live data (NET = 2TB)
- Backup this RAID-System to TWO external drives (alternating); each external HDD having 2TB
- When you work with your photos, don't wait for the hourly backup, kick it off every 30 minutes or so
- Every week, replace one of the external HDDs with a another one moving the replaced HDD to somewhere OFFSITE (with a Date+Time-Label of the last backup on it)

This gives you speed-of-availability, easy backup and an offsite solution.

I mange (nearly like this) about 13TB of stuff + with a similar setup another 1TB of stuff. It works quite reliably (yes, it is some work)

And: I also have a few spare HDDs in case my RAID fails. It is not so bad if a RAID fails and you have to rebuild it - 2TB is rebuilt very quickly - compare that to my last rebuild of 8TB ---

Re:Come the fuck on

[ImdatS]

Replying to oneself is quite stupid, but...

- You only need three external HDDs - you would replace the external one with an offsite one

But still, keep two spare HDDs available in any case

For external HDDs I use an IceBox where I can easily slot in 3.5"-HDDs without have to have an enclosure..

Re:Come the fuck on

[Anonymous Coward]

My method is the following: Every couple of years I buy a hard drive that is twice as large as the previous one. I migrate all my shit to that and leave the old one lying around somewhere.

Re:Come the fuck on

[Thor Ablestar]

Nevertheless there was a bank search in GB some 3-4 years ago. Bank cells have been opened in search of drugs and illegal profits. And you have been able to receive the wedding rings of your grandgrandgrandgrandgrandgrandgrandparents in condition that you have a proof of purchase ONLY.

Re:Come the fuck on

[houghi]

If you run Linux, why are you typing it in every week? Use crontab or if your PC is not on all the time, make it part of you shutdown and/or startup process.
If it is a portable, let it verify if you are connected to you homework or not before it launched.
A nice program tro use for incremential backups is http://savannah.nongnu.org/pro...

Re:Come the fuck on

[Joce640k]

How much time does it take to restore all your data from the internet? Downloading 4 TB of data would theoretically take about 20 days on my internet connection. For me that isn't acceptable, I want to have things restored by the next day at the latest.

Jeez, that's quite a porn addiction you've got there.

Re:Come the fuck on

[omnichad]

Whole manufacturers are not a safe bet, but product lines are. Just look at the stats for Backblaze's quarterly report. Sure, your environment might be more/less harsh, but manufacturing defects are a bigger problem.

Based on their report, there are definitely some great HGST. And some really bad WD Red. My Toshiba drives in my home RAID are going strong, but looks like they are having closer to 4% failure rate overall (though that is more recent, they had much lower stats when I bought).

Re:Come the fuck on

[omnichad]

RAID5 has been deprecated for small RAIDs ever since 1TB drives were around. Regardless of the filesystem. Minimum of RAID6 or RAID10 for large drives, because there's too many bits in one failed drive to rebuild without another drive going.

Re:Come the fuck on

[fnj]

Stupid goddam braindead suggestion. Manual syncing, to a crappy slow undercooled USB hard drive, yet - yech.

Assuming the OP's question is a serious one, and assuming the OP's commitment is serious (both of which are questionable assumptions), the answer is to set up a server or NAS with 7 drives in a RAID-Z3 pool; 1TB drives would be plenty, as you would end up with 4TB of triply-redundant storage. FreeNAS, FreeBSD, or Ubuntu 16.04, take your pick, would serve fine. All three of those support ZFS natively.

With a single RAID-Z3 pool, you can lose ANY 1, 2, or 3 drives AT THE SAME TIME without losing any of your data. And with ZFS all your data is AUTOMATICALLY checksummed, and every read operation checks the checksum, and any errors in any read are AUTOMATICALLY corrected before that read returns. You don't have the problem of non-ZFS RAID0, where the system has no possible way of knowing which of your two mirrors is right if for any reason their data differs, and in fact no way of routinely even detecting such an error, let alone notifying the user. And you don't have the problem of the write-hole vulnerability of non-ZFS RAID5.

With ZFS you don't have to take the system off-line to resilver ("rebuild") if you lose a drive and have to replace it. The pool remains perfectly usable during the resilver. In fact for anything less than extremely heavy use, you can barely tell anything is "wrong" with the performance during the resilver.

Obviously this is only a start. A damn good start, but you still need to incorporate off-site backups one way or another.

Re:Come the fuck on

[KozmoStevnNaut]

I'm currently on a Samsung 840 EVO SSD as my system drive, with a 2TB WD Green, a 3TB HGST Deskstar NAS, and a 2TB Seagate Barracuda 7200.14 for storage.

The SSD has around 1 year of actual power-on time, the storage disks each have around 1.5-2 years of power-on time. No failures reported by SMART so far.

You could say I'm sort hedging my bets by not sticking with a single vendor. The SSD will probably be the first disk to go, due to the firmware issues on the 840 EVO drives. It's been good so me so far, though.

Re:Come the fuck on

[RockDoctor]

The bank is a hell of a lot less likely to burn down

Probably true, what with workplace regulations on fire protection etc.

or float away than your house is.

Ah, now that's a deal more dubious. Particularly if you mean "get flooded" as a subset of "float away".

Banks tend to be in towns ; towns are frequently built alongside or along both sides of rivers ; rivers tend to flood. You need to check your bank's location and look at historical records (pre-historical if you've only got a millennium or two of written records) of flooding, then be utterly ruthless. (The same strategy is good for seeking living accommodation too.)

Re:Come the fuck on

[KozmoStevnNaut]

Oh, I know. That's why I bought an apartment in a building that's significantly above sea level.

Well, relatively speaking. This is Denmark, afterall, one of the flattest countries in the world.

Re:Come the fuck on

[partofthepuzzle]

I checked out the spideroak web site. The only way to find out anything in depth is via "Find Out More" or "Take the Tour", which forces you to *download and install the software*! WTF? Lost me right there.

Re:Come the fuck on

[AmiMoJo]

Scroll down on the tour page, they have screenshots.

Having said that, trying it is always going to be the only way to do a proper evaluation. I use a VM for such testing. I tried pretty much all the available online backup services. Spideroak's client is slightly above average, their privacy and security is good, and the price was right.

Re:Come the fuck on

[RockDoctor]

Ground-floor living is for the wheel chair-bound.

Re:Come the fuck on

[Wolfrider]

> Btrfs is a lot more flexible about expanding the filesystem, especially in mirror mode.

--Yah, most of the confusion about expanding ZFS filesystems OTF stems from RAIDZ. You *can* expand RAIDZ, but if you want your I/O to stay "sane" you need to duplicate the configuration to another vdev and add it to the pool. E.G. you have a RAIDZ of 5x1TB disks, you kinda need to create another vdev of 5x1TB disks and add that to the pool, or else it won't be balanced right.

--Expanding a *mirrored* ZFS pool however, is dead easy. You can start with a single disk, add a mirror to it, wait for resilver, and then add another set of (same-size/brand) disks to make it zRAID10. Then add another similar-hardware set of mirrored disks whenever you need to expand. ZFS also makes it easy to replace disks in-place AND will "see" the extra space every time you complete a mirror "column" (2 disks in the same set) if you set the pool properties right (autoexpand=on, autoreplace=on). You can also do triple mirroring, which I would definitely recommend if you have 8TB+ disks.

--So who cares if you lose 1/2 your disk space with RAID10, I would argue that with anything 1TB+ you *need* that real-time mirror - and disk prices have come down, you can get a 2TB NAS drive for under $90 these days. RAIDZ rebuild times (at least on Linux) are reported to potentially be extremely long since they haven't worked on the "speed" part of ZFS yet; rebuilding a RAID10 column/mirror is orders of magnitude faster since it doesn't have to replay every transaction or do a bunch of calculations.

--Took me a while to do the research on Linux+ZFS, but if you implement it correctly you can get a *lot* of benefits. ;-)

Re:Come the fuck on

[sjames]

The more versatile part is you can add disks one at a time to btrfs when mirroring and have it make reasonable use of it. If a disk fails and there's enough room left over you can even rebalance on an odd number of drives and be fully redundant until you can get another disk in. All of that together means you can upgrade to new bigger disks without going offline (naturally, performance takes a hit during the upgrade).

At the same time, I also have ZFS in production and no intention to migrate.

Re:Come the fuck on

[AmiMoJo]

Nice system, but you still need to remember to rotate disks off-site for security.

Re:Come the fuck on

[cthulhu11]

Your recovery time is thus limited by their available bandwidth. Anything I care about I back up to CrashPlan. Easy to manage and very offsite. rsync gives you a point in time. Which may wipe a version of the file you want. If you're going to use rsync, at least use rsnapshot.

Re:Come the fuck on

[thegarbz]

Nice system, but you still need to remember to rotate disks off-site for security.

Indeed. I have a terrible memory so as I replied in another thread, you need to make something important a routine. You don't remember routines, you just do them.

Tuesday's I play squash so I need to take an extra bag to work. Just the usual stuff, squash racket, shoes, towel, 8TB HDD, shirt, and pants. Nothing out of the ordinary there at all :)

RAID is not backup

[bad_fx]

Say with with me: "RAID is not backup!"

Re:RAID IS NOT BACKUP

[networkzombie]

I forgot to mention that the 2 other disks should be offline when not backing up. One of them preferably offsite. If they are online, they are not backup, they are vulnerable copies.

Re:RAID is not backup

[Frobnicator]

Say with with me: "RAID is not backup!"

Indeed. There is also a difference between backup and archive.

RAID = This is running live, and I need a duplicate that is instantly available so I can keep running in case one drive fails. The trick is that if there is an operation that destroys data (e.g. ransomware infection that encrypts your stuff) then you lose all disks. This is why RAID is not backup.

Backup = Just in case the machine dies, or I accidentally delete a bunch of stuff, or a virus hits, I can restore from the backup. This generally follows the 3-2-1 rule: At least three copies, at least two media, at least one off site. Businesses often use D2D2T systems for this.

Archive = I will probably never look at this again, but I absolutely need to keep a copy around for historic or business reasons. Think about services like Iron Mountain or Amazon Glacier. Tape archives that are quite cheap and almost certainly never reviewed again. This is along the lines of "show me the obituaries from a newspaper published 7 May 1957", or similar.

For the original story, it seems like he is looking for an archival solution rather than a backup solution.

Re:RAID is not backup

[Anonymous Coward]

Of course RAID isn't a backup technology. It's a way of providing fault tolerance across large filesystems. It does this by alerting administrators to failed drives, and allowing them to be swapped in & out while the filesystem stays online. At that task, it works reasonably well, although it does need to be supported by a robust alerting & "hands+feet" strategy. That's why it's still in widespread use in enterprise environments. They have the $$ and manpower to make it work.

Conversely, maintaining a good backup of your data (vs keeping it online) is a different beast. For that you have a whole bunch of other technologies like incremental copy, snapshotting, and clever combinations of the two, that store the resulting backups on everything from another RAID array, to tape systems, USB3 portable drives, remote filesystems, cloud solutions, etc etc.

What the OP seems to be asking is "what backup strategy should I consider to back up 2TB of personal data using SOHO technologies?" Personally, I wouldn't even consider doing it locally, as it's prone to human error and keeps all the data in the same location (thus failing to protect against the two most likely causes of data loss in a home environment: you forgot to run the backup, or your house got flooded/burnt/ransacked). I'd consider a cloud-based solution (rsync.net or something similar) as it solves both those issues, albeit at a higher ongoing (capex) cost rather than just a straight capital cost for a USB3 portable drive. It's hard to say an ongoing cost would be acceptable in this case, as the OP didn't mention whether $$ was a factor.

Re:RAID is not backup

[Solandri]

RAID is not a backup, unless it has a snapshot feature. 2 TB is a pittance. I've got 7 TB on a 4x4 TB ZFS raid-z volume (12 TB usable), and my setup is not really that big. It's set up with FreeNAS, which supports ZFS (redundancy like RAID, plus file integrity checks, plus snapshots). The snapshots allow you to roll back to previous version of files, thus covering the weakness of RAID which makes it not-a-backup.

If most of the 2 TB is photos, you've got two cheap cloud storage options as a second backup (in case your house burns down taking your NAS with it). Google Photos is free and allows unlimited storage of photos up to 2048x2048 resolution (and 1080p videos up to 15 min, might have been reduced to 5 min). Amazon Prime comes with unlimited cloud storage of photos of any resolution.

Re:RAID is not backup

[Jamu]

If I can potentially restore data from it, it's backup!

Re:RAID is not backup

[Gr8Apes]

RAID is not a backup, unless it has a snapshot feature.

Even then, it's not a backup in the true sense. A controller failure that sporadically writes to disk or multiple disk failures (both which I've seen occur) can crash any RAID system. The only true backup is a copy that is separate from the system running it.

Re: RAID IS NOT BACKUP

[sexconker]

If you're using some proprietary automatically-expanding RAID, "XRAID", or some crazy RAID 3 or RAID 7 or RAID 5+2 or whatever horseshit then yeah.
If you're using RAID 1, 5, 6, or 10, you'll be able to recover data on many different controllers.

Re:RAID is not backup

[gweihir]

A snapshot feature is not a backup either. It can serve as a basis of one though. A backup must be an _independent_ copy.

Re:RAID is not backup

[tobiasly]

Say with with me

Ah, a redundancy pun in a RAID discussion. Very clever.

Re:RAID is not backup

[hjf]

Tape is cheap. Tape drives on the other hand...

Re:RAID is not backup

[dgatwood]

The problem with cloud-based solutions is that the cost for backing up several terabytes of data is typically several orders of magnitude higher than building your own RAID array, and the performance of Internet-based backup absolutely sucks beyond measure unless you're the sort of person whose data needs are measured in tens of megabytes.

• To back up 2 TB over a typical cable modem (say 3 megabit upload speed) will take you 61 days. Over typical DSL (300 kilobits per second), it will take almost two years.
• If you lose your original copy, getting the data back will be almost as painful. On a fairly fast cable modem (30 mbps), assuming the cloud-based backup server can completely saturate your downlink (which is by no means guaranteed), it will take you 6 days of continuous downloading to restore the backup. Over 3 megabit DSL, again, that number goes up to 60 days.

The ideal solution, if you can pull it off, would be to build a small concrete bunker in your yard, run power out to it, put a UPS and power conditioner in there to protect against bad power, put a RAID array in there, wire it with Ethernet to your house underground, put a watertight door on the thing, add a power cutoff that shuts down power if water does get inside (e.g. a GFI breaker and an unused extension cord whose output end is lower than your equipment), and hope for the best.

But more realistically, I would tend to suggest an IOSafe fireproof RAID array loaded up with five 6 TB drives (or maybe even 8 TB drives). Put it in a closet somewhere, and hope for the best. If you want to increase your protection a bit, you could also get two RAID expansion cabinets, store them at work, and periodically bring one home, clone your main RAID array to it, and bring it back

Re: RAID IS NOT BACKUP

[NotAPK]

Would you go further and claim that RAID1 disks can always be read as independent disks after they have been removed from the array?

Serious question.

Re:RAID is not backup

[ryanmc1]

Crashplan offers this for free using their software. It is easy to setup.

Re:RAID is not backup

[ryanmc1]

You start out by saying that online backup is too expensive, then you go on to propose an outrageously expensive solution, and settle on a slightly more expensive option. In addition, storing your backup in the same location as your main copy is not smart, even if it is in a bunker or fire proof safe.

Online backup is cheap. Most start at ~$60 a year for unlimited backup. I have been using them for several years, and if you follow the 3-2-1 strategy as explained above, the online backup is the last resort for restoring. Usually when you get to it you only need a few files that your other backup is missing.

Re:RAID is not backup

[swillden]

The problem with cloud-based solutions is that the cost for backing up several terabytes of data is typically several orders of magnitude higher than building your own RAID array

Nonsense. One order of magnitude more, at most. On-line storage costs are on the order of $100 per TB per year. There's no way you can build and maintain your own solution for $1 per TB per year, which would be two orders of magnitude less. "Several" orders of magnitude would be at least four, putting you in the range of a $0.01 per TB per year. Even $10 per TB per year would be tough to reach, if you want any redundancy, and if you value your time at all -- and while you're amortizing the cost of your up-front hardware investment over several years in order to get close to that level, on-line storage costs will continue dropping, so at the end of those years the savings would be even smaller than they appear now.

Plus, backup storage which is located on-premises is inherently inferior to off-site storage, because a whole range of disasters that take out your primary storage whack your backup, too. Fireproof safes are a partial solution, but not a complete one... and not a cheap one.

No,the best approach is to use a cheap, unreliable, local backup, not bothering with bunkers or safes or even much redundancy, plus use an online service. The local copy is your normal recovery source, the online service is your final fallback.

Personally, I just replicate my data to a couple of local machines (the machines are there anyway, so throwing a little more storage in them doesn't cost much) and keep another copy on Google Drive, which is $120 per TB per year, but I managed to get 1 TB free (in perpetuity) as part of some promotion, and I currently have just under 2 TB of data that I care about (mostly photos), so my net cost is about $60 per TB per year for the online component, plus another $25 per year for an extra 4 TB drive that cost $100 and I expect to get four years out of (will probably go longer, but could die sooner).

Upload time sucks, but only for the initial upload, which I did two years ago. After that, incremental additions are pretty negligible. A full restore from the remote copy would take a long time, but I can easily get individual files on an as-needed basis. Actually, I find I use the remote copy quite frequently to grab particular photos or files on various devices, so it provides some functional value as well as disaster protection.

Re:RAID is not backup

[gweihir]

True, but it does not protect at all against the most common environmental issues like a PSU going berserk, lightening strike, theft, vandalism, etc. Calling it a "backup" is naive. But hey, it is your data, I could not care much less.

Re:RAID is not backup

[gweihir]

Indeed.

Re:RAID is not backup

[gweihir]

You seem to have no clue what a snapshot is. Maybe read a definition sometime before shooting off your mouth?

Re:RAID is not backup

[dgatwood]

Online backup is cheap. Most start at ~$60 a year for unlimited backup.

I'm having a hard time believing that $5 per month is even possible for anything approaching truly unlimited storage. Just storing 2 TB on Amazon glacier storage would cost three times that much. I assume they count on most of their users treating unlimited as tens of gigabytes. If everybody were storing 2 TB, I'd expect those numbers to go way, way up.

But even if you assume that $5 is your total cost from the cloud provider, that still isn't your total cost. After all, time has value, plus your internet connection costs money. Backing up 2 TB over a typical home Internet connection can take anywhere from many days up to years, which means if your storage needs are that large, you're going to want a faster Internet connection or you'll lose your mind. Tack on another $30 a month for that.

In addition, storing your backup in the same location as your main copy is not smart, even if it is in a bunker or fire proof safe.

Hence my suggestion of periodically cloning your RAID and keeping the clone at work.

Re:RAID is not backup

[dgatwood]

Nonsense. One order of magnitude more, at most. On-line storage costs are on the order of $100 per TB per year.

I was going based on my experience with AWS, which is about $30 per TB per month for spinning storage, or $360 per TB per year. An 8 TB hard drive should typically last you about five years, and costs about $250, or about $6.25 per terabyte per year. That isn't quite two orders of magnitude, but it is pretty close. Of course if you're willing to wait several hours to start getting your data back, you can use glacier storage, and that's cheaper, but there are tradeoffs. :-)

Upload time sucks, but only for the initial upload, which I did two years ago. After that, incremental additions are pretty negligible.

Must be nice. I backed up over 12 GB Sunday night, and that was only one week worth of incremental backups for my personal laptop. Over my DSL connection (soon to be retired), that would have taken two days. It would take several hours even over my new cable modem service. It took five minutes to back up locally. That time difference makes the difference between me being willing to back up regularly and never backing up.

Obviously, YMMV, but I would imagine that somebody with multiple terabytes of personal data is probably either a photographer or videographer, and therefore has the same sorts of nightmare backups that I do. But I'm just guessing here. For all I know, it could be a porn collection. :-)

Re: RAID IS NOT BACKUP

[sexconker]

I wouldn't claim always, no. A lot of controllers will recognize that it's a foreign disk and let you get at the data if you import the foreign config, but I'm sure there are some controllers that fuck up even basic RAID 1.

Re:RAID is not backup

[swillden]

Must be nice. I backed up over 12 GB Sunday night, and that was only one week worth of incremental backups for my personal laptop. Over my DSL connection (soon to be retired), that would have taken two days.

That is negligible. I mean, it's not like you have to sit there and watch it. Just start it in the background... maybe limit it to use only a percentage of your bandwidth to avoid making everything else suck, and let it run.

That time difference makes the difference between me being willing to back up regularly and never backing up.

And as a result, if your house burns down you'll lose everything.

Obviously, YMMV, but I would imagine that somebody with multiple terabytes of personal data is probably either a photographer or videographer, and therefore has the same sorts of nightmare backups that I do.

Photographer. Amateur, though, so I don't generate gigabytes every week. Only some weeks.

Gotta say it

[93 Escort Wagon]

RAID shouldn't be considered a backup.

It's old school and sub-optimal, but I still handle our personal backups the simple way - a big backup disk at home, and another big backup disk I keep in a locked drawer at my office and bring home every few months. Our computers are Macs, so we use Time Machine to do the backups. Our media "server" is backed up to those same disks, but with an rsync script.

I keep thinking I should probably start paying for an encrypted cloud-based backup solution... I haven't done anything about it though. (I know you can host your own, but running that sort of server at my office would be frowned upon).

Re:Gotta say it

[gweihir]

RAID is neither "old school" or "sub-optimal". It does serve a different purpose than backup: Raid is for reducing downtime, no other purpose. (RAID0 is not really RAID, as the "R" does not apply.)

Re:Gotta say it

[93 Escort Wagon]

RAID is neither "old school" or "sub-optimal".

That was a different paragraph where I explained about how I currently back things up... and which definitely has a "sneaker net" aspect to it.

RAID is not backup...

[FlyHelicopters]

Backblaze is what I use, if your backup isn't off-site, then it isn't really backup...

You can also burn to DVDs or BR or use external hard drives and move them offsite, but that takes time and effort...

Two drives with copies of the same files sitting side by side is not backup.

Re:RAID is not backup...

[Jason Levine]

I'll second BackBlaze - but with the caveat of expecting your initial upload to take a long time depending on your Internet speeds. I have a 15/1 connection so the ~1TB that I wanted to back up took me about 8 months. (I couldn't use my full 1Mbps upstream bandwidth for backup traffic.) Now that this is done, however, it's pretty much automatic. New data gets written and the backup occurs. They even have an app you can use so you can access your data no matter where you are.

If you need to restore from backup, BackBlaze will ship you a thumb drive or external hard drive for a fee. The fee is refunded if you send the drive back (thus ensuring that people don't abuse this service) and it beats having to download TBs of data.

Besides BackBlaze, I back up everything on to two external hard drives. This way, if one drive blows, the other drive keeps the data safe. As another person posted, follow the 3-2-1 rule. 3 copies of the data (for me, 2 external HDs and 1 on BackBlaze), 2 different mediums (e.g. external HDD and cloud), and 1 copy offsite (e.g. BackBlaze or another cloud provider).

Re:RAID is not backup...

[Sarpent]

Backblaze has worked well for me. I have 2.6 TB of data that took me just over two weeks to upload to their servers (FIOS). I was pretty well backed up onsite, but was still vulnerable to theft and fire since my backups weren't offsite.

Re:RAID is not backup...

[l810c]

I absolutely love iDrive. I have used it for years.

I TB $52/year. Everything backed up. Unlimited computers/devices. I have 9 computers and 3 phones all backed up automatically.

Even though I use source control, I occasionally mess something up. iDrive keeps 10 versions of my source files. Saved my ass several times over the years.
I'm wondering what happens when we do pass 1 TB as the the site shows only a 10TB plan for $374/year. I would like a 2TB plan(You might be able to call sales and negotiate)

Honestly though, if my only option was the 10TB plan, I would do it. It would be worth it with all of the features.

I love it.

http://www.pcmag.com/article2/...

Re: RAID is not backup...

[FlyHelicopters]

The best solution is to have multiple copies on external local drives.

If the drives are physically in the same place as the computer, it isn't a backup.

Fire, theft, etc.

Re:RAID is not backup...

[im_thatoneguy]

I have over 10TB on Backblaze for $5/mo. Works great and recovery is easy.

I would add though that if you want more control and more flexibility I've started using Backblaze's B2 API and SyncBack, Cloud Berry or whatever software backup solution you prefer. That costs about $5/month per TB but has the advantage of control over hash checks and retention.

Re:RAID is not backup...

[FlyHelicopters]

I absolutely love iDrive. I have used it for years.

The features look nice, the size just isn't enough...

I have 26TB backed up with BackBlaze, that level of storage would be expensive with most other services...

I also use CrashPlan for critical files, but they throttle after awhile making backing up that much data impossible. They claim they don't, but they are lying about that.

Carbonate also throttles, but at least they are honest about it. BackBlaze doesn't and will run 10 threads to backup faster.

Re:RAID is not backup...

[chill]

BDXL ranges between 100 Gb and 128 Gb per disc, if you want to go that way. For durability, use M-Disc for 1,000 year storage.

RAID IS NOT BACKUP

[networkzombie]

1) RAID IS NOT BACKUP unless you have another read only set.
2) STORAGE SPACES IS NOT BACKUP unless you have another read only set, and please, it is JBOD with some added features.
3) You are exchanging RAID sync issues with backup sync issues.

I would setup hardware RAID, but that is not related to what you need... Backup to two other disks. Upgrade disk size and technology as needed. A 4TB disk is like $140

Print them all....

[julian67]

Print them all and put them in labelled shoeboxes.

Is 100% availability a requirement?

[whoever57]

If you can live the possiblity that you can't access your data (porn stash?) for a day or two, then you don't need RAID.

Break up the RAID and use the freed-up hard drives for a backup. Buy some extra drives and rotate the backup drives so that there is always at least full one copy off-site, preferably encrypted.

ZFS & LTO.

[0100010001010011]

ZFS for on disk storage. I've had a theseus RAIDZ2 pool for almost 6 years now that's moved multiple computers, OSs and drives and hasn't lost (that I can find) any of my Pictures.

Tape for archival. LTO-4 drives are fairly in expensive (compared to losing everything).

Re:ZFS & LTO.

[Thor Ablestar]

LTO4 is my personal choice. Later generations of LTO are more expensive per byte. Both LTO-5 cartridges and drives are more than twice expensive than LTO-4. And LTO-4 are at least faster than my 2-TB HDDs.

You would need much more than 3 tapes. At least 2 generations of backup would need 6 tapes. If you partition your data to portions of not more than 800 GB each and declare some of them frozen, you may economize on tapes. But it's still slightly cheaper and much more reliable than HDDs.

Be warned that the drive would preferably have a 1-digit indicator. Also, I prefer Fibre Channel and Qlogic cards.

Cloud Based Backup

[friedmud]

Better to get it offsite. One fire/flood/etc. and your data is toast. Not too mention that RAID IS NOT BACKUP (RINB).

I'm a "serious amateur" photographer (about 1TB of photos currently) and I've been using CrashPlan for the last two years and I'm happy with it. They allow you to create a local encryption key that even they don't know so it seems pretty secure. The first upload can take a while (depending on your internet service) but everything is quick after that point.

In addition to that I also use TimeMachine on my Mac so I have a local backup of everything.

Re:Cloud Based Backup

[fustakrakich]

Better to get it offsite.

Better to get it off-planet. You never know..

Re:Cloud Based Backup

[mukinrestak]

I ended up going with backblaze too, and they're fantastic for the price, but they do have some major flaws.

The first flaw is the inability to specify folders to INCLUDE instead of exclude. It's very tedious depending on how your drives are organized to exclude everything that doesn't need backup.

The second is that some of their exclusions are mandatory.

The third is no Linux support.

The fourth is no ability to prioritize which files need backed up first. If I download a movie, then do my taxes, then do some work, It'd be nice to be able to specify that the folder with movies in it is less important than my job stuff and my taxes and should be backed up after those.

Re:Cloud Based Backup

[ben_kelley]

+1 to this. You can also back up to a "friend" (where "friend" = someone else with CrashPlan who is prepared to let you store some encrypted files, possibly in exchange for you storing some of their files). If said "friend" lives close, sneakernet around a copy of your current CrashPlan backup, and just import it to jump-start the remote copy.

Re:Cloud Based Backup

[friedmud]

"The first flaw is the inability to specify folders to INCLUDE instead of exclude."

That was my problem with Backblaze. In my case I ONLY wanted to backup my photos (I backup everything else in other ways).

Crashplan lets you choose exactly which directories to backup.

Re:Cloud Based Backup

[friedmud]

Reading comprehension: "In addition to that I also use TimeMachine on my Mac so I have a local backup of everything."

Not too mention... your backup provider getting shut down does not instantly delete all of your local data. It would be a freak incident for them to get shut down the day you need a backup.

And finally: Crashplan is nothing like Megaupload....

Re:Cloud Based Backup

[friedmud]

Please point to a reputable source that substantiates your claim. Just putting it in ALL CAPS does not make it true.

How would it not be reliable? Over the last two years I've spot checked multiple times I can could recover any file I wanted.

How is it not secure? I have a locally generated key that they are not given... even if they are breached there is no way to decrypt the data.

Like I said: I do use local backup as well for peace of mind (and ease of recovery) but I haven't seen any reason not to put faith in Crashplan.

Re:Cloud Based Backup

[swillden]

Say this with me folks: CLOUD STORAGE IS NOT RELIABLE NOR IS IT SECURE IN ANY WAY!!!!!

Bullshit.

Good providers are at least as reliable as your local drives. They could fail, but so could your local backups... and when your house burns down, the odds that your backup service provider dies at the same time is miniscule (barring some planet-scale catastrophe, in which case you probably won't care anyway).

As for security, encrypt if you're worried about it. Personally, there's nothing in my backup data that's particularly sensitive, so I don't bother. Most of the backup services automatically encrypt everything anyway.

Re:Cloud Based Backup

[Grishnakh]

Wouldn't it make more sense for them to launch some radio relays into space and then use smaller, cheaper ground based dishes to communicate with those relays? This would also eliminate the problem with the Sun being in the way (position one relay far away from the Earth so that you can beam the signal around the Sun).

Not an advert - but Backblaze

[forgottenusername]

https://www.backblaze.com/clou...

$5/month unlimited data size (writes).

You can sync files back over or they will actually ship you a HD with your data; if you return the drive you get a refund of the drive cost but you're also free to keep it.

The cost for individual file reads is reasonable too.

No muss no fuss

Re:Not an advert - but Backblaze

[hcs_$reboot]

Backblaze:

Linux, BSD, Unix and other *nix systems:
These operating systems are not supported and Backblaze can not be installed on them

Re:Not an advert - but Backblaze

[KozmoStevnNaut]

There is no Backblaze application for *nix, but you can use their B2 storage from any OS: https://www.backblaze.com/b2/c...

Still, I wish they would hurry up and port the proper app to *nix.

Re:Not an advert - but Backblaze

[thegarbz]

Sure that will only take close to a year to complete my backup. This fetish for pushing terabytes of data over the internet for someone else to look after is stranger than most sexual fetishes out there.

Re:Not an advert - but Backblaze

[barc0001]

So how do you do offsite backups for your personal data? If you don't use one of these type of services I imagine it'll be prone to forgetfulness as it'll have to be a manual process like taking a drive to your safety deposit box, etc.

The point is these online backup services have a place and it's as a "the house is gone/on fire/burglarized" last resort, not as an end all be all.

Re:Not an advert - but Backblaze

[thegarbz]

I have a work place, stuff taking a detour and paying someone to hold onto stuff for me. 2 HDDs, one sitting in my desk drawer at work. Each Monday night I plug my backup in, and on Tuesday I take it to work and bring the other one back. I plug it in and an automatic script backs up my NAS and emails me the results.

You don't forget routine. If you're forgetful, make something a routine. The easiest way to do that is tie it to an existing routine like something you do on the specific day of the week, or set a reminder in your outlook calendar. That's why I pick Tuesday. I'm incredibly forgetful so I tie my backup together with preparing my sports back Monday night. Tuesday is Squash day, so I have to take an extra bag to work, which makes it easy to run down the list: towel, racket, pants, shirt, shoes, socks, 8TB HDD. I mean you don't really consider your data important if you're forgetful enough not to back it up right?

The point of these online backup services is that they serve a purpose for a select subset of use cases. Small data sets are perfect for this. That precludes it's usefulness if you're into photography or worse recording videos. They are also great right up until you have an actual failure and your background backup operation suddenly becomes an excruciatingly slow foreground operation.

Also while I have quite a bit of faith in Backblaze and I'm quite a fan of their work, the same is not universal around Cloud storage providers, the ones who a notorious at offering you a service, oh wait we'll change the price, oh wait you now have a limit, or now there's a bandwidth limit, oh we got acquired by someone you have 5 days to suck your data off before we delete it all for you. Once bitten....

RAID is NOT backup!

[gweihir]

RAID is fine to reduce downtime, but completely unsuitable as a replacement for backup.

The RAID does not have the following things which you critically need from backup (the following list is not complete):
- resilience against operator error (accidentally delete/overwrite files, e.g.)
- geographic redundancy, usually not even safe against the box killing the disks, lightening, fire, theft, etc.
- too few copies: Usually 3 (!) independent backup copies used in rotation are considered the minimum. RAID1 gives you one and it is not independent.

My recommendation is to get at least 3 external USB disks, and establish a backup with them, because currently you have none.

Steps:
- Select a backup interval. This represents the maximum time-interval for which you think losing new data is acceptable
- At the end of each interval, do the following:
      1. Fetch oldest backup disk from off-site location
      2. Put backup copy on it, making it the newest backup. Make sure to do a file-by-file comparison.
      3. Move disk to off-site location

For somewhat reduced reliability keep the oldest copy at home and do the following:
      1. Make backup, overwriting oldest copy. Make sure to do a file-by-file comparison.
      2. Move new backup to off-site location and fetch oldest from off-site location.

An "off-site location" can be anything from a garden-shack to a storage locker at work to an arrangement with a neighbor or a friend you see regularly.

If you think this it too much effort, then your data must not be worth much. This is pretty much the agreed minimum experienced sysadmins want. Of course, there are always those that never lost any important data and they almost universally think this is way too much effort. Many of them learn in time when whatever they do results in that loss.

Re:RAID is NOT backup!

[gweihir]

Bullshit. There is very little possibility for data to get corrupted unnoticed on an external USB disk. In fact, one of the very few is in-memory and then the data is just as corrupted on RAID.

Re:RAID is NOT backup!

[gweihir]

That is not RAID. That is (mis-)using RIAD as a sort-of snapshot backup scheme. This idea can work in principle but has several problems:

1. raid-sync is often painfully slow and can take days for large disks
2. you always need to sync full disks
2. you need to know how to make the extracted disk a 1-disk RAID array for recovery (may be anything from trivial to very difficult)
3. you do not get verify and most data-loss in backups is because people did not verify
4. you need to handle naked disks safely
5. you can only do backup on disks that fit the RAID
6. you cannot do compressed backups
7. if you mess up, there is a real risk of killing all data on the raid
8. you can get an inconsistent filesystems state that way. Not really a problem with a good filesystem.

I am sure I have missed some things here. I tried this for a few weeks and found it to basically be the worst possible option. Still better than no backup.

Re:RAID is NOT backup!

[radarskiy]

"What's wrong with using RAID to..."

It's not wrong, it's *irrelevant*. RAID adds nothing to the backup process.

Re:RAID is NOT backup!

[Thor Ablestar]

There is a LOT of possibility for data to get corrupted while transferred to USB disk. While SATA and USB are covered with some error correction codes, the internal memory of SATA to USB chip isn't. And EVERY USB rack I have ever tried to backup my data has an overheat problem. It does not matter when you copy some DVD movies but when you copy 2TB then the probability of failure nears 100%, with destruction of file system and need to fsck all the backup. You need to add some heat sink to the chip, or else.

Re:RAID is NOT backup!

[omnichad]

2. you need to know how to make the extracted disk a 1-disk RAID array for recovery (may be anything from trivial to very difficult)

If it's a proper RAID1, just clone the disk and set up a 2-disk RAID again.

Re:RAID is NOT backup!

[gweihir]

Your rally have no clue what checksumming goes into USB and modern HDDs. The most likely candidate for corruption on external disks (and I never said it was impossible) affects internal disks as well.

Re:RAID is NOT backup!

[gweihir]

And where did you miss that a backup needs to be followed by a compare? After that works, it really does not matter what your obviously "trash" quality SATA-to-USB converter messes up. I never had this problem despite using a _lot_ of USB disks. Maybe you should not buy the very cheapest ones?

Re:RAID is NOT backup!

[gweihir]

And if it is the trash you can buy, you may have a problem from the disk being marked as "dirty". I take it you have never actually done this?

Re:RAID is NOT backup!

[gweihir]

Well, there are those that get this and those that will suffer a catastrophic data loss sooner or later. Enough of the second group around. Usually they are much less proud of their flawed little ideas after it has happened to them.

Re:RAID is NOT backup!

[omnichad]

I use software RAID (mdadm), and I had a transient issue on a RAID5 that marked 2 drives (of 3) as dirty and the whole thing quit working. It wasn't fun, but it was not particularly hard either. Just adding ---assume-clean when re-adding the disk worked just fine.

Re:RAID is NOT backup!

[gweihir]

"mdadm" is very well behaved for this scenario and in general. For mdadm this is indeed relatively easy to do.

I had the misfortune to work with some commercial hardware-RAID cards where I am not even sure it can be done without major effort. Personally, I have stopped using any kind of hardware-RAID, because monitoring, management, reliability and performance typically all suck badly.

Re:RAID is NOT backup!

[omnichad]

I guess I never have used hardware RAID personally, but I've always assumed that any info on the drives would be stored in NVRAM of the controller, not on the drive itself. And that a RAID1 should be 2 100% mirrors as if they were standalone drives.

Re:RAID is NOT backup!

[gweihir]

That is what I used to assume.

What I do

[jonwil]

I have a Western Digital external HDD and every so often (whenever I have some free time but trying to do it as frequently as possible) I connect it up to my PC and run a program (SyncBackFree) that does a copy of my 3 main drives. When I am not using it to back up, the external disk lives in my desk drawer.

If I had the money I would buy a second external HDD and keep one offsite swapping the 2 periodically just to deal with issues like a house fire that could destroy both my main PC and my external HDD but even without that, I have a reasonable solution that will help protect against viruses, hardware failure and many potential disasters (e.g. authorities saying "there is a bushfire/flood/massive storm/whatever comming, you need to pack a suitcase and evacuate to a safer place")

Cloud vendor - Backblaze

[jt-socal]

I use https://www.backblaze.com/ and like it. I have about 3.5T backed up with them. You need a fairly speedy upload internet connection to back up that kind of data, but it works great for me. Also, they will not back up NAS, just hard drives on your computer, so keep that in mind.

Here is what I do...

[mad_dog3283]

I have three 2TB disks; we'll call them 1, 2, and 3. Disks 1 and 2 are RAID 1; disk 3 is stored in a fireproof safe. Every so often, I break the array and move either disk 1 or 2 (we'll use disk 2 in this example) to the safe, and rebuild the array with disks 1 and 3, using disk 1 as the source. This is only a means of protection if you have software RAID, since in a hardware RAID the third "floating" disk will only be readable by the controller that synced it.

Re:Here is what I do...

[jeff4747]

Virtually all "fireproof" safes are rated for paper not catching fire. A hard disk will lose all its data far below that temperature.

I highly recommend moving the disk in the safe to somewhere offsite.

Re:Here is what I do...

[omnichad]

While you can get a data-rated safe that can keep drives at a cool enough temperature for longer, you also aren't guaranteed that the building will be a total loss. A fire two rooms away may get hot, but not hot enough to kill a drive in a standard fireproof safe.

2 TB

[wxjones]

is a lot of porn!

Re:2 TB

[hcs_$reboot]

Why would you need to store porn (let alone backup) when you get a lot of it online?

Re:Depends on how long term

[maliqua]

Undetermined amount of time until a large corporation gets bored of offering a service:
Cloud storage

NAS

[CanadianMacFan]

For an onsite option I bought an NAS that has room for two drives which I mirrored. (I would have liked a larger one that I could have gone with RAID-5 or RAID-6 but money prevented that.) I have my computers use Time Machine to back up to the NAS. I also use my NAS for BitTorrent Sync and other services (mail, DNS, proxy).

Ideally you would want an offsite option too in case something happened to your house.

unRAID

[deusued]

on mobile, or I might provide a little more info.

Been using unRAID for years; RAID resiliency with any set of disks. Neat technology that is extensible.

Use a service

[romango]

Get BackBlaze or Carbonite. It's ~$50.00 per year. 1PC. No size limit. I had my house broken into and the thieves took my computer and all the backups. Also a fire could wipe out all your backups, if stored in the same place.

Make a restore plan first

[houghi]

I have the following:
1) 1 SDD that I work on and another that is mirrored every day. If one disk fails, I have another. This is my working disk.
2) Incremential backup of data that changes often, like emails or some directories I work in. Mostly use if I delete a file by accident. Just copy it back and be done. This goes to a NAS.
3) Data that does not changes often, like movies, images and music is stored on a NAS.
4) Second NAS to backup the data of the first NAS.
5) Essential data (less than 10MB) is put on my website on a personal directory. This is data that I might need in case of the house burning down.

So when something goes wrong (unless the house burns down, but the I have other problems and my music is not one of them.) I have a way to restore it.

The most important thing however is not to backup, but the knowledge on how to restore it. You need to test that out from time to time. I have people seen who did backups to /dev/null to test it and forgot to remove that parameter.

What you can do if you REALLY need to have things off site, like photos and other things that you can't replace is just buy a dedicated HD that you put this data on and keep it in a drawer at your office. Once a month or so you take it home and add the new data.
And if that disk is full, buy a new one or a bigger one. If data is really THAT important, the price of the HD is well worth it.

But again, test the restore.

Re: 500 GB Blu-Ray

[brando56894]

HDDs don't scratch easily ;-)

Re:Come the .... on

[Anonymous Coward]

rsync {options} --checksum

I like rsync because it's versatile and robust, you can pretty much use it on any device.

Re:500 GB Blu-Ray

[TheRealMindChild]

A 500GB bluray, you say? Where can I buy such a thing?

What is the format of the original pictures?

[Streetlight]

Presumably the OP has his pics on the SDs used in the cameras that took the shots. That is, unless the SDs were reformatted and used again. Many folks I know just buy new SD cards and retain the old ones as first level storage. Keep them in a fireproof safe or safe deposit box at a bank. After that follow the best practices for backing up including cloud storage.

Re:What is the format of the original pictures?

[gweihir]

Unless you have (very expensive or very small) SLC SD cards, that is pretty worthless. With a bit of bad luck, data may get corrupted within a year.

Re:What is the format of the original pictures?

[thegarbz]

I've not had my camera too long but for the cost of the SD cards I'd be able to afford a high end tape backup system by now.

Re:What is the format of the original pictures?

[omnichad]

Did you miss something?

After that follow the best practices for backing up

SD cards are cheap. If you're a photographer, stick to 2GB or 4GB cards and use 2 or more per shoot. Keep them forever, because you included the cost of them (next to nothing) in your fee. Sure, the longevity isn't there, but it's a cheap no-effort backup. And you could hold hundreds in a small off-site box.

Re:What is the format of the original pictures?

[gweihir]

Media without reasonable data-life are not backup. They are just trash sitting in a box somewhere. So it is not a "cheap, no-effort backup", it is an overpriced no-backup waste of space.

Re:What is the format of the original pictures?

[omnichad]

Self-discharge of brand new flash memory that only has one write cycle is easily going to be 5-10 years. That's a better track record than some optical media. And all you have to do is plug it in to a computer for a while once every 5 years to refresh.

At under $5 for 4GB, it doesn't hurt to always use SD cards that have zero write cycles anyway on a paying gig. In that case, the SD card is already trash - at least it can be put to use as a backup.

Re:What is the format of the original pictures?

[gweihir]

Your number sucks badly. I have had flash media fail in as little as 6 months form discharge and that was after 2-3 overwrites.

Incidentally, your claim reminds me a lot of what many people defending DVDs as backup claim. It is straight in the region of wishful thinking. Sure, it may work, but it also may leave you without that critically needed backup without warning and it may do so with pretty high probability.

Re:Depends on how long term

[vux984]

The odds backblaze or crashplan or carbonite spontaneously decide to shutdown without notice the same day my house burns down is pretty low, after all.

DVDs

[inode_buddha]

How about backing up to DVDs?. They hold lots of data. Had an article here the other day about optical drives...

Re:DVDs

[Antique Geekmeister]

A Terabyte of disk equals approximately 250 4.7 GB DVD drives. The material is unlikely to pack efficiently, so the roughly 20% of spare space on the DVD's would be unwise to try to optimize much further. That's a very awkward backup system to maintain.

Re:DVDs

[Wolfrider]

--It's not very popular these days, but you could buy a Blu-Ray M-disc burner and a pack of 16x25GB M-Disc Blu-Rays for archival storage(think "stone media") for under $180, and burn ~375GB (uncompressed) onto 16 Blu-Ray DVDs...

--Actually I just checked and it looks like you can now buy 100GB M-disc blu-rays, albeit for a higher cost (and it may not be the same reliable stone-based media with scratch-resistant coating, according to 1 review I read.)

--About a year ago, I did a serious appraisal of all the data I *really* had to put under a "NEVERLOSE" label; and across all my PCs and laptops, it was under 25GB. Most of that was my CD music rips collection. You don't have to keep EVERYthing if you prioritize a little. Most of the other stuff is VMs and copies of things that are available on $something-else in the house.

https://en.wikipedia.org/wiki/...

http://www.mdisc.com/

Tape backup -

[Harlequin80]

Go onto ebay and buy an 2nd hand LTO3 or LTO4 tape drive for $150 - $300. Plug it in, write your files to the tape. For 2tb you would need 3 LTO3 tapes (assuming compression 800gb each). Take said tapes and drive them to another house.

Decide what timeframe of loss is acceptable. ie 4 weeks, 4 months, 12 months. That is you maximum backup cycle time. Every X period of time take a new set of tapes to your offsite backup location. Buy tapes equal to at least 3 full cycles, that way on your third backup trip you take the oldest set home and re-use them.

This is the process I use every 3 months and I have an HP Ultrium 960 sitting on top of my NAS. I also use your normal google drive type backup, but it is my second stage, rather than first stage backup. I'm not quite at the same size as you, 1.1tb, so it's 2 tapes not 3. I bought a box of 50 new lto3 tapes for $100.

Re:Tape backup -

[Thor Ablestar]

1. Don't assume anything about the LTO compression.
2. Drives with one digit indicator (Read: IBM and such) are preferable due to the full set of local tests and other useful procedures. Also IBM has much more documentation on it's drives.
3. LTO4 drive would work with LTO3 cartridges. They are cheap but the postage (I live in Russia) make LTO4 a preferable choice.

Re:Tape backup -

[Harlequin80]

LTO almost always achieves a 2:1 compression using SLDC unless the data is pre-compressed or particularly random resulting in no compression. For the purposes of photos or video a 2:1 ration would be a safe assumption to use.

LTOs will read their own generation and the tapes of 3 previous generations. They will also write to their own and the previous generation of tapes. It is this strict adherence to a standard which makes them such great backup options.

I don't understand what you mean about drives with 1 digit indicator? Given HP and IBM are both primary contributors to the LTO project and have access to the same materials I don't see much in the way of difference.

Re:Tape backup -

[NotAPK]

Look, I've never used LTO so I could be talking out my ass, but I must disagree with this:

"For the purposes of photos or video a 2:1 ration would be a safe assumption to use."

Compressed video uses some of of the most advanced domain-specific file compression routines available, without going to the fractal-based stuff that NASA uses. There is no way that LTO's compression (or any other general compression routine) is going to put a dent in a H.265 compressed video file.

JPEG may show a 15-20% improvement, since the new JPEG encoding that DropBox developed showed a 22% improvement so there is some wriggle room in JPEG.

However, from what I've read I am very confident that LTO compression does wonders for general office files: databases, spreadsheets, accounts, documents, server logs, and so on.

BTW, I found the posts about LTO to be very interesting and I think I may invest in a second hand tape drive and start using it. One way to get more mileage out of the unit will be to prepare backups for family and friends. Currently I maintain two file servers at remote locations (home and office) and allow friends and family to backup to them via DropBox. If I had an LTO drive I could simply take snapshots of the DropBox backup folder and send the tapes off site. The security of the backups are low-risk, since the data is already encrypted by the DropBox client before it even leaves the user's PC. I'm not really happy with DropBox but for now it fills a need. Maybe in the future I will switch to something more powerful like OwnCloud.

Re:Tape backup -

[Harlequin80]

If the videos have been processed to h.264 or h.265 then yes you will see almost no improvement at all. In fact your files will most likely be larger.

However that wasn't the premise I was basing it on. I was basing it on uncompressed mpegs or avi's taken straight off the recording device. That is certainly what I use for my home videos and it wouldn't make sense to process them into something like 264 as it makes them a huge pain to work with in video editing software. I also made an assumption, I know mother of all fuckups, that if he has 2tb of photos they are most likely in RAW. Otherwise that is a whole lot of photos as the OP doesn't mention video at all.....

Re:Tape backup -

[Thor Ablestar]

The compression is already built in in the recording device. Tested in both RAW and PRORES of BMPCC camera as well as processed MP4 video. It's not a case for BMCC camera RAW but the LTO compression is not specifically suited for it.

Raid is not backup.

[Guspaz]

I'm ultimately repeating what has been said over and over again here, but perhaps this will add a tiny bit more emphasis:

RAID IS NOT BACKUP.

Get BackBlaze for off-site backups. Or CrashPlan. Whatever. Just get something that is off-site that isn't going to lose your data when your RAID dies because of a controller failure, or a fire, or a flood, or an earthquake, or because a virus or hacker nuked your disks.

Why is RAID 1 not a backup?

[jmcbain]

The OP says he is using RAID 1 (mirroring). Why is RAID 1 not a backup?

Re:Why is RAID 1 not a backup?

[lakeland]

Because fire, theft or even a careless delete statement would take it down.

Normally people want to protect their data from more than just hardware failure

Re:Why is RAID 1 not a backup?

[aliquis]

Trojan / virus encrypt your files.

And then what? .. But I don't know if one will necessarily be safe with regular backups too, what about if it infect and then start encrypting files later? Sure you don't need to boot and run anything from that file-system but ..

Re:Why is RAID 1 not a backup?

[silas_moeckel]

Lets see, your desktop PC gets a bit out of whack and suddenly the whole file system in borked. The OP says he is using Intel fake raid on windows. Other common things, your windows box gets a virus and starts encrypting everything sure snapshots (if the malware can not touch them) will save you from this. In general bitrot is a thing modern drives have CRC's etc but not realy suficient, things like ZFS, BTRFS, Gluster etc etc can do bitrot detection far about a single sector. Things like snapraid can do it for arbitrary sets of drives.

In any event you would want at least one copy thats offline preferably with physical distance.

Re:Why is RAID 1 not a backup?

[jon3k]

Because RAID 1 replicates everything. Including file system corruption, viruses and accidental deletions.

Re:Why is RAID 1 not a backup?

[Lawrence_Bird]

Because any corruption to files is instantly mirrored on the "backup" drives in the array.

"Back up," not "backup."

[Anonymous Coward]

Idiot

Bluray

[grumbel5969]

2TB is not a lot of data and can be stored on about 40-80 Bluray discs depending on if you use 25GB discs or 50GB discs. It's not exactly the fasted way to do it, but for long term archival it works quite well and requires no extra effort once the discs are created. On Linux dirsplit is a useful tool for chunking the data into BluRay sized portions. xoriso can be used for burning.

In addition to that get some USB HDDs for regular day to day backups. Rotate them to an offsite location for extra protection.

Re:Bluray

[ben_kelley]

Do Bluray discs keep long term? Back when the amount of data I had to back up was an order of magnitude smaller I used to back up to DVDs. Very time consuming, and I don't think they keep for more than a few years.

Re:Bluray

[grumbel5969]

Lifetime of Blurays is said to be rather long, in the 50-100 years range. How trustworthy that data is, is hard to tell. The LTH variety of Bluray is supposed to be worse than the HTL version, so it might be worth to go for HTL as they cost more or less the same. There are also M-Disc BD-R for archival that claims to last around 1000 years, but they are quite a bit more pricey.

Re:Bluray

[Wolfrider]

--To prevent confusion, it looks like the correct name for the package is " xorriso " (FYI)

xorriso - command line ISO-9660 and Rock Ridge manipulation tool

Re:Add Remote Backup

[Mr D from 63]

Is there simple a plug & play solution for installing a NAS/server (encrypted) at a family member's home, and using that as an offsite backup? That way two family members could keep each other offsite backup, but not be able to access the files. For a while I was hoping BTSync would serve this purpose, but as I recall it never quite had the right functionality.

It seems that someone would have created such an animal.

USB to sata dongle plus 2TB SSD

[Ritz_Just_Ritz]

I'm assuming the person thinks the data is important enough to spend a bit of $$$ to make sure it doesn't go .

I'd get one of those USB 3.0 to SATA dongles, connect a 2TB SSD to it and copy the data onto the SSD. Then I'd do a quick checksum to make sure the source and destination copies were the same. Then I'd put that SSD on a shelf somewhere other than where I keep the computer where the data is stored.

Today, that SSD+dongle would likely run you about $700. It's about $500 if you break the backup into a pair of 1TB devices.

A year from now the cost will likely be half that (or less).

Anyhow, that's what I do today, though I only back up about 1TB of pics and important docs. Total cost for me is about $250.

Best,

Re:USB to sata dongle plus 2TB SSD

[Wolfrider]

--Umm, you do realize that SSDs are:

a) WAY expensive for backing things up to, and

b) An un-powered SSD drive will eventually degrade and LOSE ITS DATA in a fairly short amount of time (for Backup purposes)? This gets worse with Triple-level-and-up (TLC) Cell structures, BTW. They basically need an electric refresh to keep the cell structure from flipping to another position.

--Depending on the temperature/humidity it's stored in, SSD degradation could be detected in as low as several months or - if you're lucky - possibly as much as a couple of years. But if you don't fire it up every so often and run a data-consistency check, how would you know if your files are succumbing to bit-rot?

--There are many, many more options for backups that don't cost *nearly* as much as SSDs - that's not really what they're intended for. I can see buying an SSD if you want faster startup times on your PC, are into gaming, or you do a lot of virtualization suspending/resuming (R/W multiple gigabytes) every day. SSD's are designed to be faster than spinning disks, NOT necessarily long-lasting without power.

--For now, it looks like the best thing to do is keep your data online, have multiple rotating backups, store some stuff off-site, and copy data from old-drive to new-drive before it breaks. (I would even say real-time Mirroring or RAIDing is getting to be essential for any disk over 1-2TB.) But if you're storing your main backups on SSD media, you're over-spending *and* may be risking data loss if you don't power up the drive every so often.

--JMHO, but I would look into something like M-DISC for reasonable amounts of long-term archival storage. 4.7GB DVD M-Discs were made to the highest standard; 51% sure about the 25GB Blu-Ray M-Discs, not sure about the 100GB BD-R multi-layer discs. (Cloud backup is OK I guess as long as you don't mind 3-letter-agency snooping and you don't have a slow Internet with data caps, but encryption is definitely recommended before uploading.)

Refs:
http://www.makeuseof.com/tag/h...

http://www.anandtech.com/show/...

https://www.maketecheasier.com...

My recommend: Two drives and software purchase.

[Bomarc]

I'm sure that everyone has an opinion on this.... so here is my recommendation: Get two more drives (I would say 6 TB each). Rotate between them at a rate that you fell comfortable 'loosing your changed data'. That might mean every night, every week or every month. This way your primary data will be available all the time in case of drive failure. You will ave two backup's at any point in time; one of which is "current".

In addition: I recommend that you download install and run "Hard Drive Sentinel"; I've been using it for years, and it is able to aid in the prediction of failure of multiple dives that I've had; resulting since I've used it with no data loss. This software can see into the drives of the RAID array; and to help you to know if you might expect a failure of one of the drives in your RAID array-- or if one of your back-up drives starts to experience problems.

Re:RAID 1

[gweihir]

And on the minus-side, it is not a backup at all.

Here's what I do

[zerofoo]

I have a 16TB array with about 3TB of data stored on it (Synology NAS device).

I backup to 3 external USB drives rotating them once a month. One is attached to the NAS device. One is stored in my fire-proof safe. One more is stored in a safe at work.

I also backup my data to Amazon's Glacier service just in case of a regional disaster (think a Katrina style event).

The only thing that is manually required in my setup is the external drive rotation and occasionally checking the backup disks to make sure the files are readable. Everything else happens automatically nightly.

Re:Here's what I do

[krray]

I love Synology hardware -- don't trust their updates. One of their recent updates wiped my "root" account on the device(s). They can muck with admin, but should never touch root, but I digress...

RAID-6 is the way to go IMHO. Synology's built-in backup mechanisms didn't give me a warm fuzzy feeling -- it all goes into a database. When shit hits the fan I want quick quick quick access to my (or your) files.

I chose the two Synology approach. Each with 8 drives, RAID-6 (dual hard drive failure). One onsite. One off. This is for the house...

Here's the trick: rsync over ssh to a local hard-linked differential backup. I have 5T of live data currently with hundreds of full backup sets -- this is only using 8T on the remote backup array today (so 3T of new or different files over a few hundred days). The Mac's backup to the array to sparsebundles which is then rsync'd to hard-linked differential backup sets. Only the changed bundle files are transferred, but each set is a complete bundle.

On the remote it looks like this: 1/all_my_files and then 2/all_my_files ... 199/all_my_files. Only when the oldest backup of the file is deleted is the data really released. Deleting 1/some_file will release the inode, but the data/file may still exist in 2/some_file, etc...

I have bash scripts / cron doing it all. Assuming backup set "1" is in place the rsync for "2" may look like:
rsync -avx --human-readable --progress --stats --delete --ignore-errors --force --link-dest=1 -e 'ssh -p ${PORT} ' root@${IP}:/volume1/${REMOTE_DIR}/ 2

News for nerds, and stuff that matters?

[Anonymous Coward]

This is worthy of a Slashdot headline?
More worrying is the fact that it was posted by an editor (maybe?).

Next iteration of Slashdot (which will probably happen in 6 months when it's sold off again), may I recommend you also change the name or at least the tag line to something more along the lines of "Microsoft/Windows 10 Help desk, how may I help you?".

Maybe use Clippy or Microsoft's search puppy in the logo and Reddit for the comments.

Re:Add Remote Backup

[Mr D from 63]

Crashplan has this. I'm not a shill. I'm a customer though.

I checked it out, it might be a solution. Thanks. From reading their info, I can't tell if the remote backup is stored as encrypted data.

Re:Make it less than 100 GiB

[Harlequin80]

It's video that pushes it up. I have lots of video of my kids growing up and I don't want to lose that.

BTSync to friend/family/neighbor if only

[justcauseisjustthat]

It would be great if BTSync (now Resilio Sync) had an encrypt end folder feature or something like that, imagine backing up to another persons place for only the cost of direct connect storage or NAS (on their system, assuming you don't exceed data caps).

Re:Add Remote Backup

[Blrfl]

You're not even shilling, because they don't charge you for backups not stored on their network.

(Also a Crashplan customer at home and work. They're awesome.)

That problem has already been solved

[luis_a_espinal]

You forgot checksumming and verification after transfer.....You have something on the other drive after the transfer, you wont know what until you verify it.

By the tits of Baal, rsync or xcopy /v or robocopy in combination with fciv.

Second verse, same as the first

[davmoo]

For the 45th time in this thread, RAID is not backup. And all of you who are saying yes it is will change your minds the first time your array blows up and you have no other backup. Let's say you're running RAID 1 or 5. A drive dies. You stick in a new drive. You now better be praying and sacrificing animals in the hopes that you don't have another drive die before the array is rebuilt, which could take 12 hours or more if you're using 2TB or larger drives. If you value your files, then you have something in addition to your RAID array.

If you seriously value your files, you also have a fully automatic offsite backup, and one that retains older versions. I use CrashPlan. $5.99 a month for unlimited backup of one machine. As of the time I'm typing this, I have approximately 860,000 files amounting to 2.6TB backed up (semi-pro photographer). Yes, that first backup took about 3 months, but you gotta start sometime.

Option C

[GameboyRMH]

Easy, option C. RAID is not a backup.

Back up, not backup.

[Anonymous Coward]

Backup is the noun. It's the end result of backing up. Just like "stand up". Stand up, please. Two words is a verb.

Ask Slashdot? Ask Hillary Instead!

[Provocateur]

N/T

Air Gap a Copy to Prevent Ransom-ware attacks

[Proudrooster]

My backup strategy is simple:

1. 8TB RAID 1 NAS for everyday use
2. Periodic backups of the NAS using rsync to a backup disk I keep offsite
3. Encrypted backups to Google Drive (slow, but FREE)

The problem with NAS is that if you ever get hit by a disaster or ransomware attack, you lose it all. You need a backup of your NAS data offsite and offline.

I am currently running the Western Digital Mirror Gen 2 which let's you plug in a USB 3.0 device, then use SSH to access the device, then use rsync to update your backup. Always, unmap the NAS from any systems before hand to prevent a ransom-ware attack.

Example:
rsync -aHAX source_dir dest_dir

The trick to keeping data safe if making lots and lots of copies. This is what cloud providers rely on.

After that, you could use near line storage like archival quality Blu-Rays 50GB or get an LTO3 tape drive and write 800GB (almost 1 TB).

I've always wondered about USB3.0 to UltraSCSI converters. Do these work?

That could be a good solution.

did U ever try google photo with unllimited size ?

[BouffeMoiLaChatte]

I use googlephoto as my "offline& unlimited" storage, i dont care about google looking at my birthday pictures and google compression is not an issue.

for personnal office documents i use others cloud services synched with multiple computers i own.

My backup method is called "321" rule of thumb.

3 backups at least
2 different Media
1 Media offsite (and offline is better)

onedrive

[ljw1004]

2tb isn't that big. It's apparently still in the range of normal family use. If you spend $100/yr to get MS Office for your family, you also get 5tb of onedrive space.

(irritation: Limited to 1tb per account, so you'd need to share it...)

Just use Storage Spaces + Crashplan

[slacklinejoe]

Sure, Storage Spaces is just fancy JBOD, but it works really well, is supported and isn't tied to hardware for migrating down the line. That said, you need to back that stuff up if it's of any importance. CrashPlan is highly recommended for good reason. There are others, but you are best off with something that can handle versioning to a local disk as well as getting that stuff sync'ed offsite.

FreeNAS or any similar project

[guruevi]

FreeNAS would do what you want it to do. You can then back up to another disk or set of disks or somewhere online. As far as online goes, I'd choose a good VPS provider with sufficient disk space or rsync.net for sending ZFS streams. You could also go with a 'cheap' backup provider like BackBlaze or CrashPlan, both have their ups and downs but for that amount of space, a file-based backup (which is what rsync is) is typically insufficient especially if you're planning on growing.

File-based backups on some of my systems can take a month or so to complete over gigabit, I'd shudder to think the time it takes for 10Mbps or lower speed Internet connections. With rsync.net you can send them a physical set of disks, have them import it and ZFS sends literally takes overnight to "backup" a 100TB+ system.

"Storage Spaces" is one of those techs that MS will (has) abandon(ed), it's too buggy and slow to be usable and is/was part of MS Home Server. It's basically a very high overhead software-based RAID.

only here: the three step program

[burni2]

Preparation:
1.) have a physical local central server - that has encryption capabilities (A)

2.) have a physical local backup server that only exports just the bare discs as iscsi devices (B)

3.) have a root server sitting on the internet with ssh-tunneled iscsi exported discs which are filled by the central server. (C)

- keep your encryption keys private and existent

Programs:
rsync

Execution
Step1) sync A -> B / completly (fast)
Step2) sync A -> C / completly (slow-er)

Step3) keep A -> B in sync automatic rsync
Step4) keep A -> C in sync autoamtic rsync (during
sleep hours)

Loop Step3 & Step4 for your entire life.

Yes, its not perfect, but the implementation is pretty easy and ..

good enough because:
The root server sitting on the internet is essentially "dumb" the central server will handle all the encryption so the root server will only receive "garbage" data.

Your rootserver can plausably deny any knowledge about the content!

If somebody manages to break into the root-server all one gains is a root-server - which is bad enough.

But nobody can extort the information stored without knowing the keys.

However the encrypted data can be erased altered -> detection measure HMAC. .. moving without knowing ..
Moving all the data to a new root server is a short and easy task. Without any need for decryption and possibly leakage of uncrypted information or keys to the internet

Attach the "raw" iscsi devices from your old server to your new server.

Let it copy the whole disk content.

And you have a 1-2-1 a safe copy of your data.

Storage Spaces

[javaguy]

Plenty people have pointed out the flaws with this approach, around 3/2/1 and "raid is not a backup". I have hard drives in multiple locations with incremental backups - mirrors are vulnerable to ransomware / cryptoware. Critical information is also in CrashPlan, which is versioned.

The question also asks about experience with storage spaces. Around a year ago I purchased two HGST 4TB drives, formatted them ReFS with all integrity checking / fixing options turned on, and put them into a storage spaces mirror on W10. Since then they've worked fine - just like a single regular hard drive. Performance is fine with my older i7-2600K processor and 16GB RAM, but I don't do anything particularly disk intensive, though I do edit up to 1000 RAW photos at a time and haven't seen a problem. The windows event log seems buggy (maybe because I started on the W10 beta) and I can't actually get any information out of the event viewer, so I have no idea if it's detecting and / or fixing issues in the background, but I can access any of my files all the time so it's likely working fine.

The drive does get very, very fragmented according to Defraggler. The Windows tool defragments somewhat, though it still leaves many fragments. Defraggler can try to defragment it, but it's exceptionally slow. I don't know whether or not it's required.

GAH! "Hi! I'm Earth, have we met?"

[X86BSD]

Seriously, if you are not running BSD with ZFS by *now* you are hopelessly ignorant and/or just a glutton for PAIN. These stories just make me wonder who these people are? I come from the times when all we had was UFS and dump/restore, tar, and let me tell you it was PAIN! Managing data on UFS, backups, expanding partitions, was fun#$ing pain on a level you cannot understand. Now? Now we have ZFS, we have stopped pooping in our water. Best solution is get all that data copied to a ZFS stripped mirror pool pronto. Create another small box with another striped pool and zfs send/receive from one to the other. Create snapshots on the main box every 5 minutes. They cost nothing. And there are a ton of scripts out there to automate this and manage creation/deletion of snapshots. Storing data on anything but ZFS today should be punishable by death.

Re:Amazon S3

[110010001000]

Or Amazon Glacier. Much cheaper.

do you really need it?

[Anonymous Coward]

were those phots all unique and brilliantly composed, were they shot at the 4k/holographic/quantum leap mode of your ipad? shrink it, re-encode or scrap it; I don't miss most of my digital junk, its more stress worrying about it then when I look at it and ask myself if its really important. Even old photos that are basically redundant, one with hat, without hat, scarf on head....all in the same minute. But if you must horde then get a cheap disk and leave it there, I can't imagine one would need all of it at their fingertips, plug it in at reunions or what not.

Re:Depends on how long term

[vux984]

how long does it take claim your insurance, replace your hardware, then retrieve your backup. Realistically there could be weeks or months involved.

Unless I were actually caught in the fire and in intensive care for weeks months, I'd have a laptop within hours.

And if I WERE caught in the fire, and in a coma for months, and crashplan went out of business at the same time...

a) risk management is about managing risk. you can't eliminate it.
b) losing my photos would not be the biggest issue in my life.
c) if I did deem the photos so valuable that this was a risk i wasn't prepared for, I could have 2 cloud providers for just a bit more $$... or 3....
c) risk management is about evaluating risk.

Do it right.

[Mysticalfruit]

To echo what I've seen quickly, RAID is not backups. If something (like cryptolocker) suddenly encrypts all your files... you're still completely utterly FUCKED.
If your domicile burns to the group, you're fucked. You need something that'll keep your data consistent, and keep it safe from theft/fire/destruction.

I'm going to ignore any cost considerations... whatever.

Build two identical boxes. The first box is going to sit in your place of residence, the second is going to sit elsewhere (like your parents house, or a buddies)
Each box is running your OS dejour that naively supports ZFS, a pair of 4TB drives, setup as a ZFS mirror volume. Use whatever software you use to copy data to the local box, rsync, crashplan, whatever, once that task is completed, you'll drop a snapshot of all the pertinent file systems you care about. You now, remotely drop a snapshot on our offsite box and then kick off an rsync that catch any changes.

1. If something has drastically gone wrong with your data locally, you can always revert (in this case promote) your latest snapshot (or whatever snapshot you like best). Plus you can thumb through all your snapshots and see what you like...

2. Even if something hasn't gone wrong, but you liked an old version of something, you have that too... though you'll have to decide on a strategy for how many snapshots you're willing to keep. zfs snapshots are now scm but sometimes you realize you don't have something under control and then you wish you did.

3. You've got a full duplicate of this elsewhere in case fire/flood/theft/etc.

4. Monthly do scrubs of your file system just to make sure everything is healthy. Want to expand your volume? Add two more disks and you can tack on another mirror. Yeah, it won't be balanced but you'll suddenly have more space.

Better Question: How do you organize your data?

[hackus]

You can backup anything, my problem is how do you organize it so that you can use it, find it and back it up.

I finally came up with:

1) LINUX iSCSI serrver; Organizes the physical storage devices with external USB sticks or eSATA.
2) Create a SAMBA server to centralize access to my stuff.
3) Diversify that access with OpenVPN on my Android Phone, Tablet and Windows PC, LINUX Desktop. (ThinkPAD).

It was a cool project and now I know where everything is, and how to secure it.

I recommend the same route because once you organize the info you can use lots of different software to back stuff up.

I back my stuff up with AMANDA, and it works really good.

Re:Better Question: How do you organize your data?

[X86BSD]

That doesn't organize your data. That just dumps in one pile. How do you *find* anything is the problem. The best app I have found to do this is called NeoFinder. It allows you to organize all your files in one application, tag them with metadata, create thumbnails of common formats and the best part of it is it allows you to store the applications library data file ON the server. It's the only app that allows you to do this, everyone else forces you to store it client side. Bundle this with a ZFS server and Samba and you are golden! It runs on OS X and Windows. It's not open source but I don't care because every other OS "solution" to managing/organizing/tagging files is utter fucking trash at best and a tire fire at worst.

Re:Storage Spaces

[scdeimos]

All modern Windows defrag tools are varying amounts of slow as they're using the Defrag API built over the top of NTFS.

Try MyDefrag (previously known as JkDefrag). It comes with a bunch of profiles for placement/ordering of folders and files on a running disk, plus you can write your own using its scripting language.

Keep your RAID, build a server

[Trogre]

Keep your RAID-1. It will protect against single disk failure and improve read performance.

Build a server. Just a little one. An old desktop PC will suffice.
Put in a cheap SSD (64GB will do), and a 3 or 4 TB HDD.
Install your OS of choice to the SSD (Debian, BSD, anything low maintenance).
Write or procure a script on the server to rsync the contents of your desktop PC to the large HDD, with the --backup switch, rotating monthly. This way you get overwritten or deleted content put in another folder (usually the day of the month), so you get one month to recover accidentally deleted or overwritten files.
Set this script such that it starts ten minutes after the server boots up, waits another ten minutes, then powers down.
Set the server to Wake on Lan.
Have your desktop PC (or other server if you have one) send a WoL magic packet to the server once a day.
Turn off the server.
If you ever need to retrieve anything, send a WoL packet to the server, wait for bootup, log in and kill the backup script (which should still be sleeping if you do it within 10 minutes).

Voila, cheap robust offline backup.

Re:Keep your RAID, build a server

[oh_my_080980980]

This. It's amazing how remarkably simple a backup server is but for some reason people feel like they need something special.

Backups

[multi io]

https://www.jwz.org/doc/backups.html

That is a lot of porn

[BradMajors]

You don't have 2TB of important personal data.

Step 1 is to separate your important personal data from the rest of the data.

Re:500 GB Blu-Ray

[Thor Ablestar]

If EMP kills your drive then you just buy a new one (It's also true for the tape drive) - in condition that a post-apocalyptic world would need your data. If EMP kills your HDD then you are out of luck.

But I'd still prefer not to store my data on anything optical.

I use this

[stolidobserver]

I just use two 3TB usb 3.0 drives and run a software called Allway Sync. I've used it for years now and haven't had any problems. It analyzes your folders and makes it's own folder to store data on changes. You are then presented with a list of deletions/additions/changes that you can check through and verify that you have indeed performed those actions. It will sync in any direction you want. You can also schedule automatic backups (I wouldn't, for obvious reasons). I carry a 256gb usb stick with me everywhere that contains files that are truly important to me. I have found this to be the easiest method for daily use.

Re:Some questions and options

[Thor Ablestar]

First you need to ask yourself some questions:
1. what are you trying to protect against? Hard Drive Failure?, Multiple hard drive failures? Fire? Theft? Disk/file corruption? Destruction of your whole home/work? Everything?

Golden words. As our ex-president Medvedev once said, they would be cast in granite.

I'd add one more threat: Party Van.

Syncthing

[jon3k]

Find a friend, buy him a 2TB HDD, install and configure syncthing. Offer to do the same for him. Seriously, it's very good.

All good info

[tbarnett67]

Lots of great info here. Unless you have copies of your data off-site, its not backed up. If your home burns down with your expensive NAS with whatever RAID config, your data is gone forever. I RAID & burn to BluRay at home, and I also use BitTorrent Sync to backup my data onto friends/relatives computers in other states. You can create read-only folders so your buddies won't delete your stuff. And if you have pictures of your wife's jigglies, zip them up in a password-protected archive so your buddies won't share them with their buddies. :) TB

Upload to Anonymous ftp

[rsilvergun]

and let someone else do it?

Re:Come the .... on

[Z00L00K]

Or if you run Microsoft you use the command "xcopy /s/e/v a:*.* b:" (Tweak as needed)

Can We Drop the "Online" Vapor?

[rally2xs]

C'mon, online backup? Really? The poster said "terabytes." Cable companies in this area say "hundreds of kilobits per second" as an upload speed. That'd be 10's of kilobytes per second. How long? Get optimistic at, say, 800 kbps -> 80 - 100 kBps and you have a really long time. Lessee, 2 X 10^12 bytes / 1 X 10^5 kB/s = 2 X 10^7 seconds = 20 million seconds to upload 2 terabytes. 20 X 10^6 seconds / 3.6 X 10^3 seconds / hour = about 5.5 X 10^3 hours, or 5,500 hours. 5,500 hours / 24 hours / day = 229 days. I aborted Carbonite some years ago when I had only a couple hundred gigabytes,it was _NOT_ uploading every single file on my disk, and looked like it was going to exceed 3 weeks to do it.

Re:Can We Drop the "Online" Vapor?

[rally2xs]

Yeah, units _are_ important, and 2 X 10^12 B / 10^5 B/sec (10^5 B/sec = 100kBs) = 2 X 10^7 sec. An extra "k" crept into the units label in my original writing, but the proper divisor is 10^5, or 100,000 B/s, resulting in 2 X 10^7 seconds, and that makes for a very long time to do upload 2 Tb from these parts of the world, and this is Virginia, not Zimbabwe.

Re:ecc for home backup?

[NotAPK]

"thats pretty extreme on wallet."

Not really. Though you will probably pay more for a motherboard that supports it.

Worth it. Look up the stats on memory errors. With the volume of data commonly being handled (say 10s TB per week, if backing up a 2TB working set every day) the chance of a memory error becomes realistic.

Backup server +archive to cloud

[kosmosik]

First of all build yourself a decent local file server. Simple PC with two 4TB drives will do. Then load it with FreeBSD and create simple mirrored ZFS volume on these two drives (also have volume for the OS itself). Then share that volume with whatever protocol you prefer (SMB would be OK). Then move all your stuff to this fileserver. Now you have all your data on a self healing volume - this is very important because with few TB data you WILL notice that the data gets degraded over time - you will randomly loose some bits (so also individual photos) here and there.

Now when you have all that data in one place (file server) configure it to archive (backup) the data online to some cloud service like Amazon, Blackblaze or similar.

Crashplan between friends

[Troed]

You probably have a friend with a similar question. The solution is that you buy a backup disk the size you need, they buy a backup disk the size they need, you both install it at the others place and then you run Crashplan between yourselves. That gives you off site version controlled backups.

Done.

It is encrypted

[Immerial]

Crashplan's remote backups are encrypted with their key. If you want to be super paranoid/safe, you can also go with the plan that allows for your own personal key instead of theirs... but you better not lose it or you're really fucked!

Re:It is encrypted

[Mr D from 63]

Great info. Thanks

The best 20th century techniques ever!

[dscottj]

Sneakernetting a hard drive back and forth to the office? Really? That crap was an old PITA in 1999.

Do yourself a favor, sir. Get a consumer-level NAS that supports backup to Amazon's cloud. Buy their unlimted plan for $5/mo. Set up the client the way you like it. Boom. Done.

Re:Depends on how long term

[omnichad]

What about shutting down your account without notice? I'm pretty sure both sets of terms allow for that, and there's no guarantee that there won't be a mistaken or malicious reason for your account to get shut down at the worst possible time.

Re:Depends on how long term

[omnichad]

Are you kidding? You don't wait on the insurance reimbursement. You get out there and buy the equipment with 1-Day shipping. At least if we're talking about vitally important personal data.

RAID is not backup

[rocqua]

RAID is not backup. RAID is hardware resiliency, but it does nothing against either accidental / cryptolocker deletion. Nor does it help in case of fire.

I heard great things about backblaze. I don't use it, but only because it doesn't work well with my NAS setup.

Storage Spaces

[Not-a-Neg]

I dumped Linux on my file server that was using LVM for storage pools due to problems with Samba disconnecting in the middle of video streams and switched to Windows 10 using Storage Spaces instead. Windows 10 has been rock-solid with the only nagging issue being the automatic reboots from updates. My setup uses a full tower case with dual 4x3 5.25" SATA hot-swap modules from iStarUSA connected to a HighPoint RocketRAID (non-RAID card.) I have a total of 8 WD RED drives, currently 6 4TB drives and 2 6TB drives divided evenly into two Storage Spaces of 18TB each. One of the Spaces serves as the primary "Archive" and the 2nd set serves as the "Backup". I then have SyncBack Free running as a scheduled task in the wee hours of the morning to sync the two spaces, basically duplicating whatever is on the Archive to the Backup. This is not without risks, I run the risk of massive downtime in the event that any part of the computer fails, and also if something goes screwy with the Archive the automated sync will just duplicate the screw-up to the Backup. The data is not valuable to me though, just backups of media I own physical copies of (I do not torrent anything.) My long term plan is to build one smaller system with 4 drives to serve as a secondary Backup that will be synced manually. The cost of the drives is the main thing holding me back at this point. At present I upgrade 2 drives a year with larger capacity drives. I originally started with 2TB WD RED drives and have cycled through 3TB and 4TB drives. Not a single drive has failed on me. The 2 6TB drives were this years edition and I expect to swap in 2 more next year, maybe even step up to 8TB if they become affordable. Constantly upgrading storage drives is how I avoid catastrophic failure of drives, in fact after more than 20 years I have not had a single drive fail on me while in use. Even with SSD drives I have cycled them out within 3 years of original installation.

A New Individual Backup Paradigm Is Needed

[tabdelgawad]

In my view, people are too rigid with 'RAID Is Not Backup' and '3-2-1' schemes. I think these ideas are fine in an enterprise setting, but in an individual setting, people have different needs that depend on what they're guarding against. And the likelihood of what they're guarding against will vary depending on individual settings. For example:
1) For most people, the most likely risk is hardware failure. The first line of defense has to be a scheme that can survive the failure of a hard drive. In that sense, hardware RAID is ok, though something software-based (Storage Spaces, DrivePool, the various RAID-like schemes) is much better.
2) The risk of flood/fire will vary depending on whether you live in a flood-zone basement or a city high-rise. If you live in the latter, there's no point wasting sleep on guarding against an unlikely risk.
3) In 30 years of computer use, I don't think I've ever deleted something accidentally that was beyond recovery. Again, this may not apply to you if you're a command line jockey, but if you live in GUI world, this is not a risk you need to worry about.
4) It's hard to corrupt data in software. Programs can get borked, and OS installs/updates can be messed up, but data usually remains accessible unless there's a hardware failure.
5) How likely is a ransomware attack? What if you're careful about security? Is it worth it guarding against this risk?
6) You have to weigh the recovery effort/cost vs the protection effort/cost. Rather than spending time and money making sure you implement and maintain a 3-2-1 scheme, you might consider living with some small risks of data loss or time-consuming recovery.

$320 from Newegg

[bagofbeans]

ASRock C2550D4I Mini ITX Server Motherboard

Intel Avoton C2550 Quad-Core Processor

Has 12x SATA, takes ECC memory

Re:$320 from Newegg

[NotAPK]

Thanks, looks like a great choice for a file server, especially if running ZFS.

DLT or LTO (i.e. tape).

[aussersterne]

Some people swear by optical media for archival and backup, but I've had trouble restoring data with different optical devices and media just 3-5 years after write, so I don't trust them.

Tape, on the other hand, is venerable and proven—so long as you stick to what the big boys use.

At the top end, DLT and LTO are both still very expensive, but as they age out, they end up on eBay relatively inexpensively. The mechanisms are very robust, repairs and replacements are readily available, media is in channels, compatibility is very good.

You can pick up a used-but-verified LTO-4 drive for $200 on eBay. SAS controller, $20-$40. Media ~$20/ea for 800GB/1600GB per cartridge. So you can get rolling at less than $300 for a complete backup and go from there.

If you want to run cheapskate, DLT-VS1 ("DLT-V4") drives often come up on eBay tested and working for $80-$100 for SATA, eliminating the need for a host adapter of any kind. The VS-160 tapes (160GB/320GB on a DLT-V4 drive) can pop up in boxes of 10 for $100-$120. So if you're patient, you can get rolling there for under $200 if you get lucky, though you'll wait around a long time and switch a lot of tapes to get your full backup done.

Just avoid helical scan tapes at all costs (AIT, DDS/DAT). The reliability is crap and the media quality is crap. Wine linear tape (DLT, LTO) is what you want if you're going to run data onto tape for backups. This opinion comes from two decades of experience.

An illustration

[MSG]

https://www.penny-arcade.com/n...

Lots of people are repeating the refrain "RAID is not a backup." To that, I want to add an illustration of what that means. If there's only one place your data is located, you don't have backups.

If your data only exists on a "backup" drive, then it isn't backed up. You need to have two, and a single RAID volume doesn't provide that. No matter how many disks are in it, a RAID volume is just one "place." The same goes for Storage Spaces. If your disks are mirrored, then corruption or accidental deletions will remove the data from both.

Of the three proposals, only the last one would actually give you a backup.

Personally, I want as much distance from my data and its backup as is reasonably possible, so my recommendation would be for cloud backups or, if you don't like that idea or the price, then a small NAS for backups. A WD My Cloud 4TB (which will be 2TB in RAID1 mode, which I recommend) runs $180.

For some real peace of mind,

[tehlinux]

I suggest RAID0. No need to wait for syncing and no need to worry about recovering any of your data when it inevitably crashes.

Storage Spaces + Crashplan

[SummerKitty]

I have been using storage spaces on a Windows 2012 R2 server for 3+ years now and it's been fine. I have 6 x 3TB drives in a parity storage space using REFS (not NTFS) and run a SMART monitoring ("WindowsSmart") program that emails me every day. I have no idea why this functionality wouldn't be built into the server. Twice it's reported questionable health (one time was CRC errors due to a bad cable, the other time, the drive was accumulating errors); I replaced the drive and the array rebuilt quickly. This is software, not hardware raid, of course, and it does sometimes pause when copying lots of new data.

The most important data (documents, family photos and videos) are backed up to CrashPlan. I'm currently pushing 5TB there. I have it set to throttle uploads to 5mb/sec (I have TimeWarner-now-Charter's '200mb down/20mb up plan without caps) and, while I can't say how long it took to initially synchronize, it keeps up nicely now. I back up raw VHS family video captures, which are 14GB/hour so the originals are always available.

The data that's not backed up onto Crashplan is backed up onto a local QNAP 8-disk NAS that I was able to pick up for a great price. I stick old drives into it and use RSYNC to manually duplicate the remaining data across the QNAP's varied 1.5 and 2TB drives. If one of them croaks (and it happened once), I just replace it and hope the Windows 2012 server doesn't croak until the data is copied again.

But all the stuff I wouldn't want to lose in a fire is in Crashplan. It's very inexpensive in the US (something like $60/year for unlimited backup storage) and kind of a no-brainer for anyone with the upload bandwidth and no cap. And yes, I have recovered some accidentally deleted files, although I can't swear they're all available.

The only thing I'd do differently is use NAS drives in the server instead of cheaper desktop drives. The prices are much closer now than they were back in 2013. I noticed when the one drive was failing that copies to the server would pause for 30 seconds at a time, but the array never dropped offline. Presumably NAS drives wouldn't exhibit this problem.

Multiple, independent spindles

[jonfrei]

I use a file replication program to create a second copy on an external disk whenever a file update in the designated directories happens. Then it creates another copy to another external disk once daily. Periodically (I have a weekly reminder in my calendar, but sometimes I do it more often) I bring home yet another external drive from work, replicate to it, and take it back to work with me. On top of that, I have TimeMachine running. I also periodically (every time I update it) dump my photo archive to a 512 GB SSD & a 512 GB thumb drive -- I expect to need to go bigger sometime next year. These stay in my go-bag, so they are always with me. Worst case, a single disk failure shouldn't cost me any data and two failures should cost me at most a day. If the house burns down, I should not lose more than a week's worth of non-photo data. RAID has burned me too many times in the past -- I consider it nothing more than a single spindle in my backup scheme...

Got a buddy nearby?

[phorm]

Now how you do this really depends on your location and/or ISP, but some ISP's do not count in-network traffic against your bandwidth, so if you have a buddy on the same ISP you could set him/her up with a little box and a few hard drives, running something like "ez-ipupdate" to keep track of either his IP or yours. Have a sync job that runs regularly.

Worried about your somebody seeing your tax info in the backups (or ya'know, pictures of your wang, etc)? Run something like "duplicity" which encrypts the backup.

Now if your ISP does count in-network bandwidth, or just otherwise sucks, and your buddy lives farther away but still within visual range? Do what some friends of mine did and setup a point-to-point wifi. It's actually pretty cool what bandwidth you can get with a pringles can, metal colander, and some cheap wireless gear. If your buddy happens to be a neighbour, then you could also trying stringing some cat5e/cat6 between your houses.

WTF?

[AF_Cheddar_Head]

He is backing up photos, why are you assuming a 2TB working set every day?

A weekly full -->2TB with daily incrementals --> maybe 20MBs doesn't equal 10s of TB per week.

I agree on springing for the extra cost of ECC memory (motherboard + ECC RAM) on a FreeNAS box but let's be realistic here on the volume of backup necessary.

Re:WTF?

[NotAPK]

"With the volume of data commonly being handled"

I was speaking generally about workloads that most enthusiasts may encounter these days. I can go out for the weekend and shoot a few GB of photos and a few hundred GB of video very easily. Which you're right, isn't anywhere close to 10TB a week.

But when I wrote that post as well as exaggerating the data load I was also imagining that the user might make complete snapshots once a week: i.e. a full copy of everything. We both know that's a stupid way to do it, but it happens.

Heck: if I buy a new 10TB HDD and want to migrate everything over to it, then it will take a week to copy and I'll be facing the reality of potentially hidden and uncorrectable memory errors occurring during the copy simply based on the volume involved.

Anyway, we both agree: ECC for the win.

Re: Come the .... on

[mythosaz]

robocopy source destination /mir /r:0 /w:0 /xj

xj is the default for most modes, but more likely better safe than sorry.

Don't get your destination and source backwards with /mir :)

USB Drives

[ebvwfbw]

Do not, I repeat, DO NOT trust online backup. Look at the terms of service and such. Unless you don't care about your stuff, don't use them. I know several people that have lost their machines that way. All they can say is - "so sorry." I advised a woman tech writer to not do that. She was in tears as her laptop was away for over 1 month being repaired. Carbonite has a 30 day retention. Then - DELETE! The laptop came with a new disk drive. I salvaged a lot of her stuff off of an old deleted volume that she meant to have destroyed. Lead a horse to water... Besides, when you upload it someplace, who has access? Do you trust them? Who knows what they're doing with it.

Buy USB mechanical drives, back up to them. Do yourself a favor and check them out from time to time and have multiple copies, and better yet, off site. Sure, you can crypt them. Use a simple password that you'll remember because if you put it in your phone, you may not have your phone anymore when you need it. Sometimes when things go bad, they really go bad. You may have what's on your back and little more. I know people that have lost their whole house. I nearly did.
Every month I run a fsck on them just to make sure. They are normally kept in a commercial grade safe. Not a cheapo sams club safe this sucker is over 700 Lbs.

I shoot events as a sideline and have done since

[aussersterne]

the late '90s in digital.

I have a library of about 180k photos. You retain originals in case someone goes back to a contact sheet and wants a reprint or an enlargement a decade later or something. At a typical event I will shoot between 100 and 1,000 images. Sometimes, depending on conditions, I will shoot RAW.

My current gear is 24mp SLR and generated files are on the order of 12-15MB each for JPG images. I can easily lay down 12GB a shoot or 50GB in a week.

I keep an online 12TB RAID-1 library and then have 3 backup sets on LTO, rotated, with one set always offsite.

I know a person that does video editing and production as a sideline for corporate clients, mostly working on online ad videos and 30-second spots. They keep archives as well, because it's not uncommon for a client to come back several times over a period of several years to want minor tweaks to something that's already run (for versioning or feature changes, slightly different voice track, color edits, text overlay edits, etc.). They have even larger data needs.

Point being: even many individuals and small businesses *do* have legitimate, productive needs, and your condescending view is just a tad narrow.

Some tools

[strombrg]

For speed, it's hard to beat an rsync wrapper like Backup.rsync: http://stromberg.dnsalias.org/... . It doesn't deduplicate super-well, and it doesn't compress on disk, but like I said, it's fast. It's quite good at removing old data that you don't care about anymore.

For frugality, consider something like backshift: http://stromberg.dnsalias.org/... . It deduplicates quite well, and compresses everything (including most metadata) with xz. It also makes it easy to expire old data, unlike a lot of backup software you'll see.

Full disclosure: I wrote both of them.

Here's a table comparing some common backup tools: http://stromberg.dnsalias.org/...

ZFS

[hoggoth]

You need to protect against (1) the hard drive dying, (2) the whole house burning down, (3) bit-rot making some of your photos unreadable over time, (4) ransom-ware encrypting or deleting your photos.

Here's what I did:

I set up a small cheap PC 'backup server' with ZFS on Linux and two mirrored 4TB drives. Dropbox keeps my photos synced directly to this backup server and a simple script copies over any new photos to a folder. Another simple script makes ZFS snapshots of any new files or changes every 15 minutes. On ZFS this takes almost no space or time, only the changed disk blocks use any space. I set up a second identical backup server and put it in my office. If my office didn't allow that I could have put it at my brother's house. My server does a ZFS send periodically to keep the servers in sync. The server does a weekly 'zfs scrub' to check the file checksums.

The mirror makes sure a dead hard drive doesn't lose my files. The snapshots make sure if someone accidentally deletes files or a ransom-ware gets them I can get them back. The scrub protects against bit-rot. The zfs send to the second server protects against the house burning down.

Alternatively if you don't want to mess with Linux and ZFS, just use Crashplan's free option and have it store a second copy of your backups on a second computer at your office or friend's house.