Slashdot Mirror
Ask Slashdot: What's The Best Way To Backup Large Amounts Of Personal Data? (foxdeploy.com)
An anonymous Slashdot reader has "approximately two terabytes of photos, currently sitting on two 4-terabyte 'Intel Rapid Storage' RAID 1 disks." But now they're considering three alternatives after moving to a new PC:
a) Keep these exactly as they are... The current configuration is OK, but it's a pain if a RAID re-sync is needed as it takes a long time to check four terabytes.
b) Move to "Storage Spaces". I've not used Storage Spaces before, but reports seem to show it's good... It's a Good Thing that the disks are 100% identical and removable and readable separately. Downside? Unknown territory.
c) Break the RAID, and set up the second disk as a file-copied backup... [This] would lose a (small) amount of resilience, but wouldn't suffer from the RAID-sync issues, ideally a Mac-like "TimeMachine" backup would handle file histories.
Any recommendations?
This is also a good time to share your experiences with Storage Spaces, so leave your answers in the comments. What's the best way to backup large amounts of personal data?
b) Move to "Storage Spaces". I've not used Storage Spaces before, but reports seem to show it's good... It's a Good Thing that the disks are 100% identical and removable and readable separately. Downside? Unknown territory.
c) Break the RAID, and set up the second disk as a file-copied backup... [This] would lose a (small) amount of resilience, but wouldn't suffer from the RAID-sync issues, ideally a Mac-like "TimeMachine" backup would handle file histories.
Any recommendations?
This is also a good time to share your experiences with Storage Spaces, so leave your answers in the comments. What's the best way to backup large amounts of personal data?
Memorize it! Just don't take any head injuries or you won't remember anything.
More seriously, back up to hard drives is the only viable option. Then make sure you have more than one backup drive and store one at some other site. Relative maybe?
Cloud options with that kind of storage would take forever to upload. And I've heard of people having stuff randomly go missing on their cloud service, not the entire contents, but a file here and there. I'm not so sure that's a good option.
For storing on-site you can get a fire rated media safe, but they can be quite a bit more expensive than a regular safe.
Microfilm
Upload it all to Freenet and let other people store it for you.
2 Terabytes is nothing.
Here's how you do this:
10 You buy an external hard disk that is 4 Terabytes or larger, and USB 3.0.
20 Copy the fucking files to that thing.
You're done. Now you have two copies: one on whatever bad idea you have as your main drive, and the other on a physically separate drive.
Not good enough? GOTO 10
Say with with me: "RAID is not backup!"
RAID shouldn't be considered a backup.
It's old school and sub-optimal, but I still handle our personal backups the simple way - a big backup disk at home, and another big backup disk I keep in a locked drawer at my office and bring home every few months. Our computers are Macs, so we use Time Machine to do the backups. Our media "server" is backed up to those same disks, but with an rsync script.
I keep thinking I should probably start paying for an encrypted cloud-based backup solution... I haven't done anything about it though. (I know you can host your own, but running that sort of server at my office would be frowned upon).
#DeleteChrome
Keep it local either on raid or normal disk. Use something like Arq to backup to cloud provider of choice. You want remote backup.
Backblaze is what I use, if your backup isn't off-site, then it isn't really backup...
You can also burn to DVDs or BR or use external hard drives and move them offsite, but that takes time and effort...
Two drives with copies of the same files sitting side by side is not backup.
1) RAID IS NOT BACKUP unless you have another read only set.
2) STORAGE SPACES IS NOT BACKUP unless you have another read only set, and please, it is JBOD with some added features.
3) You are exchanging RAID sync issues with backup sync issues.
I would setup hardware RAID, but that is not related to what you need... Backup to two other disks. Upgrade disk size and technology as needed. A 4TB disk is like $140
Print them all and put them in labelled shoeboxes.
If you can live the possiblity that you can't access your data (porn stash?) for a day or two, then you don't need RAID.
Break up the RAID and use the freed-up hard drives for a backup. Buy some extra drives and rotate the backup drives so that there is always at least full one copy off-site, preferably encrypted.
The real "Libtards" are the Libertarians!
ZFS for on disk storage. I've had a theseus RAIDZ2 pool for almost 6 years now that's moved multiple computers, OSs and drives and hasn't lost (that I can find) any of my Pictures.
Tape for archival. LTO-4 drives are fairly in expensive (compared to losing everything).
Backup to 500 GB Blu-Ray.
An external HDD is not safe from EMP.
Better to get it offsite. One fire/flood/etc. and your data is toast. Not too mention that RAID IS NOT BACKUP (RINB).
I'm a "serious amateur" photographer (about 1TB of photos currently) and I've been using CrashPlan for the last two years and I'm happy with it. They allow you to create a local encryption key that even they don't know so it seems pretty secure. The first upload can take a while (depending on your internet service) but everything is quick after that point.
In addition to that I also use TimeMachine on my Mac so I have a local backup of everything.
Put a server in your bathroom and let Russian gov't know about it.
Amazon Glacier + FastGlacier.
https://www.backblaze.com/clou...
$5/month unlimited data size (writes).
You can sync files back over or they will actually ship you a HD with your data; if you return the drive you get a refund of the drive cost but you're also free to keep it.
The cost for individual file reads is reasonable too.
No muss no fuss
It's expensive, but worth it. s3cmd (http://s3tools.org/s3cmd) is great at incremental backups. You can use Reduced Redundancy Storage (RRS) to save money. It still has 99.99% durability which is better than you can achieve locally even with RAID 1.
but i wouldn't bother for only 4tb.
buy 3 4tb drives, place at least one offsite and don't forget to spin them up every half year or so - and recopy a few years down.
RAID is fine to reduce downtime, but completely unsuitable as a replacement for backup.
The RAID does not have the following things which you critically need from backup (the following list is not complete):
- resilience against operator error (accidentally delete/overwrite files, e.g.)
- geographic redundancy, usually not even safe against the box killing the disks, lightening, fire, theft, etc.
- too few copies: Usually 3 (!) independent backup copies used in rotation are considered the minimum. RAID1 gives you one and it is not independent.
My recommendation is to get at least 3 external USB disks, and establish a backup with them, because currently you have none.
Steps:
- Select a backup interval. This represents the maximum time-interval for which you think losing new data is acceptable
- At the end of each interval, do the following:
1. Fetch oldest backup disk from off-site location
2. Put backup copy on it, making it the newest backup. Make sure to do a file-by-file comparison.
3. Move disk to off-site location
For somewhat reduced reliability keep the oldest copy at home and do the following:
1. Make backup, overwriting oldest copy. Make sure to do a file-by-file comparison.
2. Move new backup to off-site location and fetch oldest from off-site location.
An "off-site location" can be anything from a garden-shack to a storage locker at work to an arrangement with a neighbor or a friend you see regularly.
If you think this it too much effort, then your data must not be worth much. This is pretty much the agreed minimum experienced sysadmins want. Of course, there are always those that never lost any important data and they almost universally think this is way too much effort. Many of them learn in time when whatever they do results in that loss.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I have a Western Digital external HDD and every so often (whenever I have some free time but trying to do it as frequently as possible) I connect it up to my PC and run a program (SyncBackFree) that does a copy of my 3 main drives. When I am not using it to back up, the external disk lives in my desk drawer.
If I had the money I would buy a second external HDD and keep one offsite swapping the 2 periodically just to deal with issues like a house fire that could destroy both my main PC and my external HDD but even without that, I have a reasonable solution that will help protect against viruses, hardware failure and many potential disasters (e.g. authorities saying "there is a bushfire/flood/massive storm/whatever comming, you need to pack a suitcase and evacuate to a safer place")
My company backups data to solid state optical tape disks. Last year we backupped over 1 million gigglebytes. This year we'll be backupping over 2 million.
See how dumb it looks when you try to use "backup" as a verb?
I use https://www.backblaze.com/ and like it. I have about 3.5T backed up with them. You need a fairly speedy upload internet connection to back up that kind of data, but it works great for me. Also, they will not back up NAS, just hard drives on your computer, so keep that in mind.
I have three 2TB disks; we'll call them 1, 2, and 3. Disks 1 and 2 are RAID 1; disk 3 is stored in a fireproof safe. Every so often, I break the array and move either disk 1 or 2 (we'll use disk 2 in this example) to the safe, and rebuild the array with disks 1 and 3, using disk 1 as the source. This is only a means of protection if you have software RAID, since in a hardware RAID the third "floating" disk will only be readable by the controller that synced it.
Reprise the theme song and roll the credits!
is a lot of porn!
My SIG is a P226
Thousands of years:
Papyrus and cave paintings seem to hold up.
Hundreds of years
file-> Print
Dozens of years
Magnetic tape
A couple years
A USB drive
For an onsite option I bought an NAS that has room for two drives which I mirrored. (I would have liked a larger one that I could have gone with RAID-5 or RAID-6 but money prevented that.) I have my computers use Time Machine to back up to the NAS. I also use my NAS for BitTorrent Sync and other services (mail, DNS, proxy).
Ideally you would want an offsite option too in case something happened to your house.
on mobile, or I might provide a little more info.
Been using unRAID for years; RAID resiliency with any set of disks. Neat technology that is extensible.
Lots and lots. If some go bad, you only lose FOEVA a very small number. Raid1 you lose EVERTING! You can find lots and lots of shareware (FREE!) floppy file managers on almost any decent bulletin board!
Get BackBlaze or Carbonite. It's ~$50.00 per year. 1PC. No size limit. I had my house broken into and the thieves took my computer and all the backups. Also a fire could wipe out all your backups, if stored in the same place.
I have the following:
1) 1 SDD that I work on and another that is mirrored every day. If one disk fails, I have another. This is my working disk.
2) Incremential backup of data that changes often, like emails or some directories I work in. Mostly use if I delete a file by accident. Just copy it back and be done. This goes to a NAS.
3) Data that does not changes often, like movies, images and music is stored on a NAS.
4) Second NAS to backup the data of the first NAS.
5) Essential data (less than 10MB) is put on my website on a personal directory. This is data that I might need in case of the house burning down.
So when something goes wrong (unless the house burns down, but the I have other problems and my music is not one of them.) I have a way to restore it.
The most important thing however is not to backup, but the knowledge on how to restore it. You need to test that out from time to time. I have people seen who did backups to /dev/null to test it and forgot to remove that parameter.
What you can do if you REALLY need to have things off site, like photos and other things that you can't replace is just buy a dedicated HD that you put this data on and keep it in a drawer at your office. Once a month or so you take it home and add the new data.
And if that disk is full, buy a new one or a bigger one. If data is really THAT important, the price of the HD is well worth it.
But again, test the restore.
Don't fight for your country, if your country does not fight for you.
rsync {options} --checksum
I like rsync because it's versatile and robust, you can pretty much use it on any device.
The easiest backup is RAID-1. You literally don't have to do anything once you set it up.
Presumably the OP has his pics on the SDs used in the cameras that took the shots. That is, unless the SDs were reformatted and used again. Many folks I know just buy new SD cards and retain the old ones as first level storage. Keep them in a fireproof safe or safe deposit box at a bank. After that follow the best practices for backing up including cloud storage.
In a time of universal deceit, telling the truth is a revolutionary act. George Orwell
It's by Microsoft and the description on the page you linked is just buzzwords.
Make a proper NAS with your favourite GNU/Linux distribution or with something specialized like FreeNAS, don't use RAID but do use ZFS (more mature) or btrfs (more linux-friendly)
How about backing up to DVDs?. They hold lots of data. Had an article here the other day about optical drives...
C|N>K
1. You get a bunch of HDDs.
2. You label one HDD "pictures", the other "movies", the 3rd "music", the 4th "games", the 5th "books/texts", the 6th "forgive me for i have sinned tier porn".
3. You think up a nice way to organize all that shit into folders so you can clearly see and find what you want.
4. You put the HDD into one of those plastic containers for HDDs which keep it safe from moisture and shit.
5. Now you can come back and fap to your old hardcore gay porn collection 20 years into the future without any issues.
Go onto ebay and buy an 2nd hand LTO3 or LTO4 tape drive for $150 - $300. Plug it in, write your files to the tape. For 2tb you would need 3 LTO3 tapes (assuming compression 800gb each). Take said tapes and drive them to another house.
Decide what timeframe of loss is acceptable. ie 4 weeks, 4 months, 12 months. That is you maximum backup cycle time. Every X period of time take a new set of tapes to your offsite backup location. Buy tapes equal to at least 3 full cycles, that way on your third backup trip you take the oldest set home and re-use them.
This is the process I use every 3 months and I have an HP Ultrium 960 sitting on top of my NAS. I also use your normal google drive type backup, but it is my second stage, rather than first stage backup. I'm not quite at the same size as you, 1.1tb, so it's 2 tapes not 3. I bought a box of 50 new lto3 tapes for $100.
I'm ultimately repeating what has been said over and over again here, but perhaps this will add a tiny bit more emphasis:
RAID IS NOT BACKUP.
Get BackBlaze for off-site backups. Or CrashPlan. Whatever. Just get something that is off-site that isn't going to lose your data when your RAID dies because of a controller failure, or a fire, or a flood, or an earthquake, or because a virus or hacker nuked your disks.
The OP says he is using RAID 1 (mirroring). Why is RAID 1 not a backup?
Seriously my personal data never goes above 1GiB, at that size it's trivial to back up, even cheap usb sticks are safe enough when you can use a handful for redundancy, also less fragile than HDDs... Will you ever really look at every one of your photos again? I doubt it, is it really that personal, is it important that all of it be reliably backed up? I have plenty more than 1GiB of data, but it's under 1 GiB that I actually care about, maybe you need to stop hoarding data :P
Idiot
2TB is not a lot of data and can be stored on about 40-80 Bluray discs depending on if you use 25GB discs or 50GB discs. It's not exactly the fasted way to do it, but for long term archival it works quite well and requires no extra effort once the discs are created. On Linux dirsplit is a useful tool for chunking the data into BluRay sized portions. xoriso can be used for burning.
In addition to that get some USB HDDs for regular day to day backups. Rotate them to an offsite location for extra protection.
I'm assuming the person thinks the data is important enough to spend a bit of $$$ to make sure it doesn't go .
I'd get one of those USB 3.0 to SATA dongles, connect a 2TB SSD to it and copy the data onto the SSD. Then I'd do a quick checksum to make sure the source and destination copies were the same. Then I'd put that SSD on a shelf somewhere other than where I keep the computer where the data is stored.
Today, that SSD+dongle would likely run you about $700. It's about $500 if you break the backup into a pair of 1TB devices.
A year from now the cost will likely be half that (or less).
Anyhow, that's what I do today, though I only back up about 1TB of pics and important docs. Total cost for me is about $250.
Best,
I'm sure that everyone has an opinion on this.... so here is my recommendation: Get two more drives (I would say 6 TB each). Rotate between them at a rate that you fell comfortable 'loosing your changed data'. That might mean every night, every week or every month. This way your primary data will be available all the time in case of drive failure. You will ave two backup's at any point in time; one of which is "current".
In addition: I recommend that you download install and run "Hard Drive Sentinel"; I've been using it for years, and it is able to aid in the prediction of failure of multiple dives that I've had; resulting since I've used it with no data loss. This software can see into the drives of the RAID array; and to help you to know if you might expect a failure of one of the drives in your RAID array-- or if one of your back-up drives starts to experience problems.
I have a 16TB array with about 3TB of data stored on it (Synology NAS device).
I backup to 3 external USB drives rotating them once a month. One is attached to the NAS device. One is stored in my fire-proof safe. One more is stored in a safe at work.
I also backup my data to Amazon's Glacier service just in case of a regional disaster (think a Katrina style event).
The only thing that is manually required in my setup is the external drive rotation and occasionally checking the backup disks to make sure the files are readable. Everything else happens automatically nightly.
I would like something like this:
I have a "storage box" I keep in my home, something like a NAS - from time to time, I might add/remove/replace a hard drive - this is the standard fix/break/repair type items. I can copy files to this.
I have a duplicate box at my siblings house, my sibling has one to for her stuff. The two boxes sync with something like RSYNC, maybe we have 3 boxes - other brothers or sisters.
I would like to have little files prioritized (done within a few hours) of addition to the storage box ... that can take a few days, I don't need it right away. I'm willing to take that risk.
If I add something huge - ie: a 4gig DVD
I would like my stuff encrypted at my sisters house, somethings I want to keep private - yes my sister is an angle, but she does not need to know everything, she feels the same way about me looking at her stuff.
It should not require a subscription service, it's my sister - she has internet also.
Perhaps from time to time, the two systems might say: "You have too much to transfer - please send hard drive (X) to your sisters house and have her send you one of hers - maybe this is fedex, ups, or maybe I have lunch or dinner with my family.
This is worthy of a Slashdot headline?
More worrying is the fact that it was posted by an editor (maybe?).
Next iteration of Slashdot (which will probably happen in 6 months when it's sold off again), may I recommend you also change the name or at least the tag line to something more along the lines of "Microsoft/Windows 10 Help desk, how may I help you?".
Maybe use Clippy or Microsoft's search puppy in the logo and Reddit for the comments.
It would be great if BTSync (now Resilio Sync) had an encrypt end folder feature or something like that, imagine backing up to another persons place for only the cost of direct connect storage or NAS (on their system, assuming you don't exceed data caps).
You forgot checksumming and verification after transfer.....You have something on the other drive after the transfer, you wont know what until you verify it.
By the tits of Baal, rsync or xcopy /v or robocopy in combination with fciv.
For the 45th time in this thread, RAID is not backup. And all of you who are saying yes it is will change your minds the first time your array blows up and you have no other backup. Let's say you're running RAID 1 or 5. A drive dies. You stick in a new drive. You now better be praying and sacrificing animals in the hopes that you don't have another drive die before the array is rebuilt, which could take 12 hours or more if you're using 2TB or larger drives. If you value your files, then you have something in addition to your RAID array.
If you seriously value your files, you also have a fully automatic offsite backup, and one that retains older versions. I use CrashPlan. $5.99 a month for unlimited backup of one machine. As of the time I'm typing this, I have approximately 860,000 files amounting to 2.6TB backed up (semi-pro photographer). Yes, that first backup took about 3 months, but you gotta start sometime.
I want a new quote. One that won't spill. One that don't cost too much. Or come in a pill.
It's closed source, so you have no idea what it is actually doing. Nor, can anyone fix any of the several bugs it has. I suggest using something that isn't so embarrassing that the authors can't release their bad source code.
LOL!
Easy, option C. RAID is not a backup.
"When information is power, privacy is freedom" - Jah-Wren Ryel
Backup is the noun. It's the end result of backing up. Just like "stand up". Stand up, please. Two words is a verb.
N/T
WARNING: Smartphones have side effects--most of them undocumented.
And then run Acronis. Which offer the smallest backups at the fastest speed compared to other solutions, especially the free ones.
If it's truly a backup that you want use AWS Glacier. It's relatively cheap and extremely reliable. Just don't treat it like Dropbox or iCloud. It's for backing up and archiving. That's it. You won't pay much until you need to get it back, but even that is pretty affordable if you end up actually needing it.
Best of all, you can script it via the CLI.
Do not worry! Go0gle has already backed it all up for you... Forever...
My backup strategy is simple:
1. 8TB RAID 1 NAS for everyday use
2. Periodic backups of the NAS using rsync to a backup disk I keep offsite
3. Encrypted backups to Google Drive (slow, but FREE)
The problem with NAS is that if you ever get hit by a disaster or ransomware attack, you lose it all. You need a backup of your NAS data offsite and offline.
I am currently running the Western Digital Mirror Gen 2 which let's you plug in a USB 3.0 device, then use SSH to access the device, then use rsync to update your backup. Always, unmap the NAS from any systems before hand to prevent a ransom-ware attack.
Example:
rsync -aHAX source_dir dest_dir
The trick to keeping data safe if making lots and lots of copies. This is what cloud providers rely on.
After that, you could use near line storage like archival quality Blu-Rays 50GB or get an LTO3 tape drive and write 800GB (almost 1 TB).
I've always wondered about USB3.0 to UltraSCSI converters. Do these work?
That could be a good solution.
I use googlephoto as my "offline& unlimited" storage, i dont care about google looking at my birthday pictures and google compression is not an issue.
for personnal office documents i use others cloud services synched with multiple computers i own.
My backup method is called "321" rule of thumb.
3 backups at least
2 different Media
1 Media offsite (and offline is better)
2tb isn't that big. It's apparently still in the range of normal family use. If you spend $100/yr to get MS Office for your family, you also get 5tb of onedrive space.
(irritation: Limited to 1tb per account, so you'd need to share it...)
Sure, Storage Spaces is just fancy JBOD, but it works really well, is supported and isn't tied to hardware for migrating down the line. That said, you need to back that stuff up if it's of any importance. CrashPlan is highly recommended for good reason. There are others, but you are best off with something that can handle versioning to a local disk as well as getting that stuff sync'ed offsite.
FreeNAS would do what you want it to do. You can then back up to another disk or set of disks or somewhere online. As far as online goes, I'd choose a good VPS provider with sufficient disk space or rsync.net for sending ZFS streams. You could also go with a 'cheap' backup provider like BackBlaze or CrashPlan, both have their ups and downs but for that amount of space, a file-based backup (which is what rsync is) is typically insufficient especially if you're planning on growing.
File-based backups on some of my systems can take a month or so to complete over gigabit, I'd shudder to think the time it takes for 10Mbps or lower speed Internet connections. With rsync.net you can send them a physical set of disks, have them import it and ZFS sends literally takes overnight to "backup" a 100TB+ system.
"Storage Spaces" is one of those techs that MS will (has) abandon(ed), it's too buggy and slow to be usable and is/was part of MS Home Server. It's basically a very high overhead software-based RAID.
Custom electronics and digital signage for your business: www.evcircuits.com
Preparation:
1.) have a physical local central server - that has encryption capabilities (A)
2.) have a physical local backup server that only exports just the bare discs as iscsi devices (B)
3.) have a root server sitting on the internet with ssh-tunneled iscsi exported discs which are filled by the central server. (C)
- keep your encryption keys private and existent
Programs:
rsync
Execution
Step1) sync A -> B / completly (fast)
Step2) sync A -> C / completly (slow-er)
Step3) keep A -> B in sync automatic rsync
Step4) keep A -> C in sync autoamtic rsync (during
sleep hours)
Loop Step3 & Step4 for your entire life.
Yes, its not perfect, but the implementation is pretty easy and ..
good enough because:
The root server sitting on the internet is essentially "dumb" the central server will handle all the encryption so the root server will only receive "garbage" data.
Your rootserver can plausably deny any knowledge about the content!
If somebody manages to break into the root-server all one gains is a root-server - which is bad enough.
But nobody can extort the information stored without knowing the keys.
However the encrypted data can be erased altered -> detection measure HMAC. .. moving without knowing ..
Moving all the data to a new root server is a short and easy task. Without any need for decryption and possibly leakage of uncrypted information or keys to the internet
Attach the "raw" iscsi devices from your old server to your new server.
Let it copy the whole disk content.
And you have a 1-2-1 a safe copy of your data.
Plenty people have pointed out the flaws with this approach, around 3/2/1 and "raid is not a backup". I have hard drives in multiple locations with incremental backups - mirrors are vulnerable to ransomware / cryptoware. Critical information is also in CrashPlan, which is versioned.
The question also asks about experience with storage spaces. Around a year ago I purchased two HGST 4TB drives, formatted them ReFS with all integrity checking / fixing options turned on, and put them into a storage spaces mirror on W10. Since then they've worked fine - just like a single regular hard drive. Performance is fine with my older i7-2600K processor and 16GB RAM, but I don't do anything particularly disk intensive, though I do edit up to 1000 RAW photos at a time and haven't seen a problem. The windows event log seems buggy (maybe because I started on the W10 beta) and I can't actually get any information out of the event viewer, so I have no idea if it's detecting and / or fixing issues in the background, but I can access any of my files all the time so it's likely working fine.
The drive does get very, very fragmented according to Defraggler. The Windows tool defragments somewhat, though it still leaves many fragments. Defraggler can try to defragment it, but it's exceptionally slow. I don't know whether or not it's required.
Seriously, if you are not running BSD with ZFS by *now* you are hopelessly ignorant and/or just a glutton for PAIN. These stories just make me wonder who these people are? I come from the times when all we had was UFS and dump/restore, tar, and let me tell you it was PAIN! Managing data on UFS, backups, expanding partitions, was fun#$ing pain on a level you cannot understand. Now? Now we have ZFS, we have stopped pooping in our water. Best solution is get all that data copied to a ZFS stripped mirror pool pronto. Create another small box with another striped pool and zfs send/receive from one to the other. Create snapshots on the main box every 5 minutes. They cost nothing. And there are a ton of scripts out there to automate this and manage creation/deletion of snapshots. Storing data on anything but ZFS today should be punishable by death.
First you need to ask yourself some questions:
1. what are you trying to protect against? Hard Drive Failure?, Multiple hard drive failures? Fire? Theft? Disk/file corruption? Destruction of your whole home/work? Everything?
2. what's your budget?
3. how many copies of data do you want and where?
4. If you're looking at a cloud backup service then what's your bandwidth? How much of your internet usage are you happy to allocate to backups? how much is your data change rate? (i.e no sense using a cloud backup provider if you change your data faster than you can upload it)
Some options:
1. Cloud Backup service (e.g. Backblaze, many here)
2. Cloud Storage provider (e.g. Dropbox, Amazon Glacier)
3. Your own solution (e.g. FreeNAS, external usb drive, eSATA (external SATA) drive, home server, unison, xcopy etc...).
If you do use your own server solution then I'd recommending having a look at ZFS filesystem (e.g. zfsonlinux)
were those phots all unique and brilliantly composed, were they shot at the 4k/holographic/quantum leap mode of your ipad? shrink it, re-encode or scrap it; I don't miss most of my digital junk, its more stress worrying about it then when I look at it and ask myself if its really important. Even old photos that are basically redundant, one with hat, without hat, scarf on head....all in the same minute. But if you must horde then get a cheap disk and leave it there, I can't imagine one would need all of it at their fingertips, plug it in at reunions or what not.
To echo what I've seen quickly, RAID is not backups. If something (like cryptolocker) suddenly encrypts all your files... you're still completely utterly FUCKED.
If your domicile burns to the group, you're fucked. You need something that'll keep your data consistent, and keep it safe from theft/fire/destruction.
I'm going to ignore any cost considerations... whatever.
Build two identical boxes. The first box is going to sit in your place of residence, the second is going to sit elsewhere (like your parents house, or a buddies)
Each box is running your OS dejour that naively supports ZFS, a pair of 4TB drives, setup as a ZFS mirror volume. Use whatever software you use to copy data to the local box, rsync, crashplan, whatever, once that task is completed, you'll drop a snapshot of all the pertinent file systems you care about. You now, remotely drop a snapshot on our offsite box and then kick off an rsync that catch any changes.
1. If something has drastically gone wrong with your data locally, you can always revert (in this case promote) your latest snapshot (or whatever snapshot you like best). Plus you can thumb through all your snapshots and see what you like...
2. Even if something hasn't gone wrong, but you liked an old version of something, you have that too... though you'll have to decide on a strategy for how many snapshots you're willing to keep. zfs snapshots are now scm but sometimes you realize you don't have something under control and then you wish you did.
3. You've got a full duplicate of this elsewhere in case fire/flood/theft/etc.
4. Monthly do scrubs of your file system just to make sure everything is healthy. Want to expand your volume? Add two more disks and you can tack on another mirror. Yeah, it won't be balanced but you'll suddenly have more space.
Yes Francis, the world has gone crazy.
You can backup anything, my problem is how do you organize it so that you can use it, find it and back it up.
I finally came up with:
1) LINUX iSCSI serrver; Organizes the physical storage devices with external USB sticks or eSATA.
2) Create a SAMBA server to centralize access to my stuff.
3) Diversify that access with OpenVPN on my Android Phone, Tablet and Windows PC, LINUX Desktop. (ThinkPAD).
It was a cool project and now I know where everything is, and how to secure it.
I recommend the same route because once you organize the info you can use lots of different software to back stuff up.
I back my stuff up with AMANDA, and it works really good.
Got Geometrodynamics? Awe, too hard to figure out? Too bad.
Keep your RAID-1. It will protect against single disk failure and improve read performance.
Build a server. Just a little one. An old desktop PC will suffice.
Put in a cheap SSD (64GB will do), and a 3 or 4 TB HDD.
Install your OS of choice to the SSD (Debian, BSD, anything low maintenance).
Write or procure a script on the server to rsync the contents of your desktop PC to the large HDD, with the --backup switch, rotating monthly. This way you get overwritten or deleted content put in another folder (usually the day of the month), so you get one month to recover accidentally deleted or overwritten files.
Set this script such that it starts ten minutes after the server boots up, waits another ten minutes, then powers down.
Set the server to Wake on Lan.
Have your desktop PC (or other server if you have one) send a WoL magic packet to the server once a day.
Turn off the server.
If you ever need to retrieve anything, send a WoL packet to the server, wait for bootup, log in and kill the backup script (which should still be sleeping if you do it within 10 minutes).
Voila, cheap robust offline backup.
"Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
https://www.jwz.org/doc/backups.html
Hammer + chisel + slab or rock
Can't get a better backup medium than that.
You don't have 2TB of important personal data.
Step 1 is to separate your important personal data from the rest of the data.
I just use two 3TB usb 3.0 drives and run a software called Allway Sync. I've used it for years now and haven't had any problems. It analyzes your folders and makes it's own folder to store data on changes. You are then presented with a list of deletions/additions/changes that you can check through and verify that you have indeed performed those actions. It will sync in any direction you want. You can also schedule automatic backups (I wouldn't, for obvious reasons). I carry a 256gb usb stick with me everywhere that contains files that are truly important to me. I have found this to be the easiest method for daily use.
It is free and automatically resizes to 16 mp or 1080p. And tags everything using ML.
4 tb? I bet they forgot how to count that low.
Find a friend, buy him a 2TB HDD, install and configure syncthing. Offer to do the same for him. Seriously, it's very good.
Lots of great info here. Unless you have copies of your data off-site, its not backed up. If your home burns down with your expensive NAS with whatever RAID config, your data is gone forever. I RAID & burn to BluRay at home, and I also use BitTorrent Sync to backup my data onto friends/relatives computers in other states. You can create read-only folders so your buddies won't delete your stuff. And if you have pictures of your wife's jigglies, zip them up in a password-protected archive so your buddies won't share them with their buddies. :)
TB
and let someone else do it?
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
thats pretty extreme on wallet.
just copy to usb disks or a cheapo nas thats not routed to internet. filo stack of usbs would be better but more work. run a check after making a backup to make sure you can read it... these are backups were talking and not archival and automating to nas could leave you syncing a deletion.
even if it was archival there are fairly limited options to storing terabytes on a budget and hd's are the most practical even then. cheapo raid usb/nas thingys have their own set of pitfalls like destroying the whole array in one event too, better just use the money for separate unconnected disks and once you go through the stack check what you are making backups of again and that your script copies and checks the right stuff.
Or if you run Microsoft you use the command "xcopy /s/e/v a:*.* b:" (Tweak as needed)
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
Robocopy
C'mon, online backup? Really? The poster said "terabytes." Cable companies in this area say "hundreds of kilobits per second" as an upload speed. That'd be 10's of kilobytes per second. How long? Get optimistic at, say, 800 kbps -> 80 - 100 kBps and you have a really long time. Lessee, 2 X 10^12 bytes / 1 X 10^5 kB/s = 2 X 10^7 seconds = 20 million seconds to upload 2 terabytes. 20 X 10^6 seconds / 3.6 X 10^3 seconds / hour = about 5.5 X 10^3 hours, or 5,500 hours. 5,500 hours / 24 hours / day = 229 days. I aborted Carbonite some years ago when I had only a couple hundred gigabytes,it was _NOT_ uploading every single file on my disk, and looked like it was going to exceed 3 weeks to do it.
I use Cloudberry to backup to Amazon Glacier, it had great logging of the backups it runs and has been very reliable for me. One thing: data restores are not free. Cheap, but restoring the whole 2tb of photos could be many dollars, or more. I should figure that out for myself!
I use syncthing tool, mainly for two reasons - it is quite reliable, and it supports file versioning. So, if something screws your files on one end, the other syncing machines are most likely to have good (and easy to recover) data.
It is kind of a CPU hog, though.
First of all build yourself a decent local file server. Simple PC with two 4TB drives will do. Then load it with FreeBSD and create simple mirrored ZFS volume on these two drives (also have volume for the OS itself). Then share that volume with whatever protocol you prefer (SMB would be OK). Then move all your stuff to this fileserver. Now you have all your data on a self healing volume - this is very important because with few TB data you WILL notice that the data gets degraded over time - you will randomly loose some bits (so also individual photos) here and there.
Now when you have all that data in one place (file server) configure it to archive (backup) the data online to some cloud service like Amazon, Blackblaze or similar.
You can setup pools with different raids, which is nice.
I use Storage spaces for all my raids...internal & external.
You might want to consider REFS too, to be super secure since they're your pics.
I've got a file server that has a useful storage capacity of 15TB. It consists of 8 WD Red 3TB drives in RAID6 with a hotspare available. These are attached to an LSI RAID controller with two mini SAS interfaces. The controller is there to allow not pursuing a motherboard with at least 8 SATA3 ports and a compatible socket. Also there's always one more HDD and RAID controller around in case of a failure. But as others have said RAID is not backup. So there are also 3x3TB HDDs used for weekly backup of everything. These drives are kept offsite except once a week during the actual backup. I was considering a tape backup but LTO drives supporting newer types of tapes are way too expensive. Someone mentioned Backblaze, which is definitely a cool option but I have no way of uploading 9TB of data to them.
Though, for only 2TB of data this here is an overkill in my opinion. I would recommend something like a single drive that is big enough and syncing the data on regular basis.
You probably have a friend with a similar question. The solution is that you buy a backup disk the size you need, they buy a backup disk the size they need, you both install it at the others place and then you run Crashplan between yourselves. That gives you off site version controlled backups.
Done.
it's in my head
ZFS.
RAID sucks. Set up ZFS and use live snapshotting to get the "time machine" file history feature.
Scrubbing and re-silvering run in the background and do not take your offline.
(then add some kind of off-site backup and you're golden!)
Crashplan's remote backups are encrypted with their key. If you want to be super paranoid/safe, you can also go with the plan that allows for your own personal key instead of theirs... but you better not lose it or you're really fucked!
If the rsync checksum fails after copying terabytes of data, how do you know which drive is corrupted? You don't, all you know is they're different.
I have about 18TB of data on a raid array. I back it up using bare drives, a drive dock, and unix scripts which break up the backups using the alphabet.
If we account 10MB per picture that makes [at least] 200,000 (i.e. two hundred thousand) photos! Like 55 years of pictures at ten pictures a day, 24/7... IIF a single individual and with that quantity chances are most of them can be trashed since hardly to be viewed again... I'd say time to tidy up the place. But, well, my opinion only.
Sneakernetting a hard drive back and forth to the office? Really? That crap was an old PITA in 1999.
Do yourself a favor, sir. Get a consumer-level NAS that supports backup to Amazon's cloud. Buy their unlimted plan for $5/mo. Set up the client the way you like it. Boom. Done.
AMCGLTD.COM. Where cats, science fictio
I heard great things about backblaze. I don't use it, but only because it doesn't work well with my NAS setup.
I dumped Linux on my file server that was using LVM for storage pools due to problems with Samba disconnecting in the middle of video streams and switched to Windows 10 using Storage Spaces instead. Windows 10 has been rock-solid with the only nagging issue being the automatic reboots from updates. My setup uses a full tower case with dual 4x3 5.25" SATA hot-swap modules from iStarUSA connected to a HighPoint RocketRAID (non-RAID card.) I have a total of 8 WD RED drives, currently 6 4TB drives and 2 6TB drives divided evenly into two Storage Spaces of 18TB each. One of the Spaces serves as the primary "Archive" and the 2nd set serves as the "Backup". I then have SyncBack Free running as a scheduled task in the wee hours of the morning to sync the two spaces, basically duplicating whatever is on the Archive to the Backup. This is not without risks, I run the risk of massive downtime in the event that any part of the computer fails, and also if something goes screwy with the Archive the automated sync will just duplicate the screw-up to the Backup. The data is not valuable to me though, just backups of media I own physical copies of (I do not torrent anything.) My long term plan is to build one smaller system with 4 drives to serve as a secondary Backup that will be synced manually. The cost of the drives is the main thing holding me back at this point. At present I upgrade 2 drives a year with larger capacity drives. I originally started with 2TB WD RED drives and have cycled through 3TB and 4TB drives. Not a single drive has failed on me. The 2 6TB drives were this years edition and I expect to swap in 2 more next year, maybe even step up to 8TB if they become affordable. Constantly upgrading storage drives is how I avoid catastrophic failure of drives, in fact after more than 20 years I have not had a single drive fail on me while in use. Even with SSD drives I have cycled them out within 3 years of original installation.
-==- Buy a Mac and leave me alone!
In my view, people are too rigid with 'RAID Is Not Backup' and '3-2-1' schemes. I think these ideas are fine in an enterprise setting, but in an individual setting, people have different needs that depend on what they're guarding against. And the likelihood of what they're guarding against will vary depending on individual settings. For example:
1) For most people, the most likely risk is hardware failure. The first line of defense has to be a scheme that can survive the failure of a hard drive. In that sense, hardware RAID is ok, though something software-based (Storage Spaces, DrivePool, the various RAID-like schemes) is much better.
2) The risk of flood/fire will vary depending on whether you live in a flood-zone basement or a city high-rise. If you live in the latter, there's no point wasting sleep on guarding against an unlikely risk.
3) In 30 years of computer use, I don't think I've ever deleted something accidentally that was beyond recovery. Again, this may not apply to you if you're a command line jockey, but if you live in GUI world, this is not a risk you need to worry about.
4) It's hard to corrupt data in software. Programs can get borked, and OS installs/updates can be messed up, but data usually remains accessible unless there's a hardware failure.
5) How likely is a ransomware attack? What if you're careful about security? Is it worth it guarding against this risk?
6) You have to weigh the recovery effort/cost vs the protection effort/cost. Rather than spending time and money making sure you implement and maintain a 3-2-1 scheme, you might consider living with some small risks of data loss or time-consuming recovery.
Imposing Libertarian views on everyone online since 1992.
ASRock C2550D4I Mini ITX Server Motherboard
Intel Avoton C2550 Quad-Core Processor
Has 12x SATA, takes ECC memory
Some people swear by optical media for archival and backup, but I've had trouble restoring data with different optical devices and media just 3-5 years after write, so I don't trust them.
Tape, on the other hand, is venerable and proven—so long as you stick to what the big boys use.
At the top end, DLT and LTO are both still very expensive, but as they age out, they end up on eBay relatively inexpensively. The mechanisms are very robust, repairs and replacements are readily available, media is in channels, compatibility is very good.
You can pick up a used-but-verified LTO-4 drive for $200 on eBay. SAS controller, $20-$40. Media ~$20/ea for 800GB/1600GB per cartridge. So you can get rolling at less than $300 for a complete backup and go from there.
If you want to run cheapskate, DLT-VS1 ("DLT-V4") drives often come up on eBay tested and working for $80-$100 for SATA, eliminating the need for a host adapter of any kind. The VS-160 tapes (160GB/320GB on a DLT-V4 drive) can pop up in boxes of 10 for $100-$120. So if you're patient, you can get rolling there for under $200 if you get lucky, though you'll wait around a long time and switch a lot of tapes to get your full backup done.
Just avoid helical scan tapes at all costs (AIT, DDS/DAT). The reliability is crap and the media quality is crap. Wine linear tape (DLT, LTO) is what you want if you're going to run data onto tape for backups. This opinion comes from two decades of experience.
STOP . AMERICA . NOW
RAID is an availability tool, not a backup tool.
A backup regime protects from:
- user and sysadmin error
- software bugs
- security compromise
- fire and theft
Backups are also good if you end up painted into a corner and need to migrate the format in which the data is stored with delete/restore.
RAID makes the former two worse, not better. If you can't afford 2x the storage, I think you can't do backups, not with the current SATA command set. If you can afford 2x but no more, then do rsync backups, no RAID (though I understand this is flippant because you don't want filesystems bound to disk sizes). If you can afford more than 2x storage, then convert each backup replica into a RAID that allows losing two disks: either a three-way mirror or a reed-solomon scheme, and do rsync between the two RAIDs.
Am at 28TB of data so far, and counting. More than 200K pictures and counting. Getting worse by the years with bigger sensors.
I have multiple copies of my stuff.
For the top of the crop:
1- What's new, what's to be processed, and my best pictures (in raw and lossless post-processed) on my main laptop, which has a 750GB 2nd drive just for that (yeah, removed the DVD burner - no use for it anyways)
2- Post-processed pictures in JPEG low compression are on my web site for purchase and for safekeeping.
3- My computer (Mac) has a Time Machine running towards a 4TB external drive. So far, it's able to cope with 1.5 years worth of backlog by being careful.
For everything: ... slooow) in RAID-5, but with 3TB drives. That drive has lost 1 failed drive so far.
3- All my pictures are sorted with my picture management software, in a single folder, on my server. The server has a FW800 16-drive bay in RAID-5, full of 2TB (these were the usual back when I started using it). So far, I replaced about 8 failed drives. They are now on a roll, and consistently die every 6 months.
4- I have a full server backup, with a USB 2 (yeah
For the worst case scenario:
5- My friend keeps a copy of my important pictures in RAW and in postprocessed lossless compression at his home. I update this every few months.
My next steps will be to replace my old 2010 server with a recent machine, and the drive bays with USB 3 / Thunderbolt / ... boxes, with 7x archival 8GB and 1x 2TB SSD for caching so the drives can spin down.
So far, my rig HAS broken down twice for every single piece. So, take any number, and they broke at least twice for irrecoverable failures, each number individually. I even had a case where 2 numbers broke down at the same time, making me lose weeks of time to rebuild stuff. -- These things happen. Whether software or hardware failure, or hacking (never happened to me so far), these all will break your flow. Just make sure your backupS are automated, works well, efficient, and up to date. Please don't wait for it to break before checking if your backup works :) At least check it once per 1-2 months, in case.
Good luck! And you are part of the solution! Most people don't care about backups!
I have used Microsoft Storage Spaces with 2 x 4TB disks for many years and there are some advantages. I've replaced my motherboard, CPU, and Windows Operating System and the Storage Spaces RAID just works. Its plug and play and that part is fantastic. You don't need to worry about how to move the RAID from old hardware to new hardware.
The downside is the access read speeds. It takes several seconds for the spinup to occur and it causes frequent lag. The other major downside is gaming... forget about playing games installed on your Storage Spaces RAID. Most games just crash to desktop or boot to black screens. This includes pretty much all the modern Ubisoft and Deep Silver games. None of these games run on Storage Spaces RAID.
Oh look. An article about backing up files. I have files. I should back files up. What should I use.
Storage Spaces? What is that? "Click".
And you even provided a handy web link. So thoughtful
Schmuck.
https://www.penny-arcade.com/n...
Lots of people are repeating the refrain "RAID is not a backup." To that, I want to add an illustration of what that means. If there's only one place your data is located, you don't have backups.
If your data only exists on a "backup" drive, then it isn't backed up. You need to have two, and a single RAID volume doesn't provide that. No matter how many disks are in it, a RAID volume is just one "place." The same goes for Storage Spaces. If your disks are mirrored, then corruption or accidental deletions will remove the data from both.
Of the three proposals, only the last one would actually give you a backup.
Personally, I want as much distance from my data and its backup as is reasonably possible, so my recommendation would be for cloud backups or, if you don't like that idea or the price, then a small NAS for backups. A WD My Cloud 4TB (which will be 2TB in RAID1 mode, which I recommend) runs $180.
This. Xcopy is a poor relation compared to robocopy. Xcopy is OK for small shell scripts but useless for big backup sets.
I suggest RAID0. No need to wait for syncing and no need to worry about recovering any of your data when it inevitably crashes.
Most linux users don't know this, but the man pages were named after Chuck Norris. Chuck Norris fsck'ing hates noobs!
I use a file replication program to create a second copy on an external disk whenever a file update in the designated directories happens. Then it creates another copy to another external disk once daily. Periodically (I have a weekly reminder in my calendar, but sometimes I do it more often) I bring home yet another external drive from work, replicate to it, and take it back to work with me. On top of that, I have TimeMachine running. I also periodically (every time I update it) dump my photo archive to a 512 GB SSD & a 512 GB thumb drive -- I expect to need to go bigger sometime next year. These stay in my go-bag, so they are always with me. Worst case, a single disk failure shouldn't cost me any data and two failures should cost me at most a day. If the house burns down, I should not lose more than a week's worth of non-photo data. RAID has burned me too many times in the past -- I consider it nothing more than a single spindle in my backup scheme...
I have been using storage spaces on a Windows 2012 R2 server for 3+ years now and it's been fine. I have 6 x 3TB drives in a parity storage space using REFS (not NTFS) and run a SMART monitoring ("WindowsSmart") program that emails me every day. I have no idea why this functionality wouldn't be built into the server. Twice it's reported questionable health (one time was CRC errors due to a bad cable, the other time, the drive was accumulating errors); I replaced the drive and the array rebuilt quickly. This is software, not hardware raid, of course, and it does sometimes pause when copying lots of new data.
The most important data (documents, family photos and videos) are backed up to CrashPlan. I'm currently pushing 5TB there. I have it set to throttle uploads to 5mb/sec (I have TimeWarner-now-Charter's '200mb down/20mb up plan without caps) and, while I can't say how long it took to initially synchronize, it keeps up nicely now. I back up raw VHS family video captures, which are 14GB/hour so the originals are always available.
The data that's not backed up onto Crashplan is backed up onto a local QNAP 8-disk NAS that I was able to pick up for a great price. I stick old drives into it and use RSYNC to manually duplicate the remaining data across the QNAP's varied 1.5 and 2TB drives. If one of them croaks (and it happened once), I just replace it and hope the Windows 2012 server doesn't croak until the data is copied again.
But all the stuff I wouldn't want to lose in a fire is in Crashplan. It's very inexpensive in the US (something like $60/year for unlimited backup storage) and kind of a no-brainer for anyone with the upload bandwidth and no cap. And yes, I have recovered some accidentally deleted files, although I can't swear they're all available.
The only thing I'd do differently is use NAS drives in the server instead of cheaper desktop drives. The prices are much closer now than they were back in 2013. I noticed when the one drive was failing that copies to the server would pause for 30 seconds at a time, but the array never dropped offline. Presumably NAS drives wouldn't exhibit this problem.
I use a file replication program to create a second copy on an external disk whenever a file update in the designated directories happens. Then it creates another copy to another external disk once daily. Periodically (I have a weekly reminder in my calendar, but sometimes I do it more often) I bring home yet another external drive from work, replicate to it, and take it back to work with me. On top of that, I have TimeMachine running. I also periodically (every time I update it) dump my photo archive to a 512 GB SSD & a 512 GB thumb drive -- I expect to need to go bigger sometime next year. These stay in my go-bag, so they are always with me. Worst case, a single disk failure shouldn't cost me any data and two failures should cost me at most a day. If the house burns down, I should not lose more than a week's worth of non-photo data. RAID has burned me too many times in the past -- I consider it nothing more than a single spindle in my backup scheme...
Now how you do this really depends on your location and/or ISP, but some ISP's do not count in-network traffic against your bandwidth, so if you have a buddy on the same ISP you could set him/her up with a little box and a few hard drives, running something like "ez-ipupdate" to keep track of either his IP or yours. Have a sync job that runs regularly.
Worried about your somebody seeing your tax info in the backups (or ya'know, pictures of your wang, etc)? Run something like "duplicity" which encrypts the backup.
Now if your ISP does count in-network bandwidth, or just otherwise sucks, and your buddy lives farther away but still within visual range? Do what some friends of mine did and setup a point-to-point wifi. It's actually pretty cool what bandwidth you can get with a pringles can, metal colander, and some cheap wireless gear. If your buddy happens to be a neighbour, then you could also trying stringing some cat5e/cat6 between your houses.
The problem with having another backup is when you get bit rot on your data. Remember those old Amiga disks that are unplayable today? Or that old Winrar archive which you can not open anymore because of corrupt data? Or old VHS casettes that are fuzzy? Data rots given enough time. Say you get corrupt data on a winrar file without noticing it, and you back it up to another disk. Now all your copies are corrupt and you have no correct file. The same when you copy to three disks or off site. You never know when a file gets corrupt, and there is a risk you spread the corruption to every copy.
So how to solve this problem? You need to checksum every file every month or so. And as soon you are going to backup your file, you need to verify the file data integrity by comparing checksums. So you need to store all checksums in a file and regularly compare every file to the checksum file. And when you modify a file, you need to update the checksum. Do this on three different hard disks if you value your data. You see that this quickly gets out of hands when you have many files, and try to keep only the most recent file. This can be a real pain, and if you screw it up juggling lot of files you are toast.
So, are there any automatic software doing this for you? The software should automatically compare checksums as soon as you touch a tiny amount of data block on the disk. And if there is a corruption, the software should repair the data and inform you. And you could just tell it to examine every data block on the entire disk and it should compare the checksums whenever you tell it to do so, say every week or so. This software exists, and you dont have to do anything. All this happens in the background without you having to do manual steps or so. It is all automatic. And it is called "ZFS", which is a filesystem and it is much safer than hardware raid, according to computer scientists. Research shows it is the safest storage solution out there. And it is free open software.
https://en.wikipedia.org/wiki/ZFS#Data_integrity
The drawback? It does not exist on Windows. You need to use Solaris, Linux or FreeBSD or Mac OS X. Or you use FreeNAS which is a storage appliance and it is very easy to setup. Read more on FreeNAS.
He is backing up photos, why are you assuming a 2TB working set every day?
A weekly full -->2TB with daily incrementals --> maybe 20MBs doesn't equal 10s of TB per week.
I agree on springing for the extra cost of ECC memory (motherboard + ECC RAM) on a FreeNAS box but let's be realistic here on the volume of backup necessary.
robocopy source destination /mir /r:0 /w:0 /xj
xj is the default for most modes, but more likely better safe than sorry.
Don't get your destination and source backwards with /mir :)
Do not, I repeat, DO NOT trust online backup. Look at the terms of service and such. Unless you don't care about your stuff, don't use them. I know several people that have lost their machines that way. All they can say is - "so sorry." I advised a woman tech writer to not do that. She was in tears as her laptop was away for over 1 month being repaired. Carbonite has a 30 day retention. Then - DELETE! The laptop came with a new disk drive. I salvaged a lot of her stuff off of an old deleted volume that she meant to have destroyed. Lead a horse to water... Besides, when you upload it someplace, who has access? Do you trust them? Who knows what they're doing with it.
Buy USB mechanical drives, back up to them. Do yourself a favor and check them out from time to time and have multiple copies, and better yet, off site. Sure, you can crypt them. Use a simple password that you'll remember because if you put it in your phone, you may not have your phone anymore when you need it. Sometimes when things go bad, they really go bad. You may have what's on your back and little more. I know people that have lost their whole house. I nearly did.
Every month I run a fsck on them just to make sure. They are normally kept in a commercial grade safe. Not a cheapo sams club safe this sucker is over 700 Lbs.
I would recommend you upload to Google photos, they will be searchable and de-duplicated in case of multiple copies
the late '90s in digital.
I have a library of about 180k photos. You retain originals in case someone goes back to a contact sheet and wants a reprint or an enlargement a decade later or something. At a typical event I will shoot between 100 and 1,000 images. Sometimes, depending on conditions, I will shoot RAW.
My current gear is 24mp SLR and generated files are on the order of 12-15MB each for JPG images. I can easily lay down 12GB a shoot or 50GB in a week.
I keep an online 12TB RAID-1 library and then have 3 backup sets on LTO, rotated, with one set always offsite.
I know a person that does video editing and production as a sideline for corporate clients, mostly working on online ad videos and 30-second spots. They keep archives as well, because it's not uncommon for a client to come back several times over a period of several years to want minor tweaks to something that's already run (for versioning or feature changes, slightly different voice track, color edits, text overlay edits, etc.). They have even larger data needs.
Point being: even many individuals and small businesses *do* have legitimate, productive needs, and your condescending view is just a tad narrow.
STOP . AMERICA . NOW
First I'll point out that this is NOT a backup. If you want a backup/archive either upload it to a server or put it on media and get it offsite.
However for the use case you are currently using your RAID configuration, which I'm guessing is just I want to have all of my stuff somewhere I can easily access, but (probably) won't get lost due to hardware failure, get a drobo.
For speed, it's hard to beat an rsync wrapper like Backup.rsync: http://stromberg.dnsalias.org/... . It doesn't deduplicate super-well, and it doesn't compress on disk, but like I said, it's fast. It's quite good at removing old data that you don't care about anymore.
For frugality, consider something like backshift: http://stromberg.dnsalias.org/... . It deduplicates quite well, and compresses everything (including most metadata) with xz. It also makes it easy to expire old data, unlike a lot of backup software you'll see.
Full disclosure: I wrote both of them.
Here's a table comparing some common backup tools: http://stromberg.dnsalias.org/...
You need to protect against (1) the hard drive dying, (2) the whole house burning down, (3) bit-rot making some of your photos unreadable over time, (4) ransom-ware encrypting or deleting your photos.
Here's what I did:
I set up a small cheap PC 'backup server' with ZFS on Linux and two mirrored 4TB drives. Dropbox keeps my photos synced directly to this backup server and a simple script copies over any new photos to a folder. Another simple script makes ZFS snapshots of any new files or changes every 15 minutes. On ZFS this takes almost no space or time, only the changed disk blocks use any space. I set up a second identical backup server and put it in my office. If my office didn't allow that I could have put it at my brother's house. My server does a ZFS send periodically to keep the servers in sync. The server does a weekly 'zfs scrub' to check the file checksums.
The mirror makes sure a dead hard drive doesn't lose my files. The snapshots make sure if someone accidentally deletes files or a ransom-ware gets them I can get them back. The scrub protects against bit-rot. The zfs send to the second server protects against the house burning down.
Alternatively if you don't want to mess with Linux and ZFS, just use Crashplan's free option and have it store a second copy of your backups on a second computer at your office or friend's house.
- For the complete works of Shakespeare: cat