Slashdot Mirror
Ask Slashdot: What's The Best Way To Backup Large Amounts Of Personal Data? (foxdeploy.com)
An anonymous Slashdot reader has "approximately two terabytes of photos, currently sitting on two 4-terabyte 'Intel Rapid Storage' RAID 1 disks." But now they're considering three alternatives after moving to a new PC:
a) Keep these exactly as they are... The current configuration is OK, but it's a pain if a RAID re-sync is needed as it takes a long time to check four terabytes.
b) Move to "Storage Spaces". I've not used Storage Spaces before, but reports seem to show it's good... It's a Good Thing that the disks are 100% identical and removable and readable separately. Downside? Unknown territory.
c) Break the RAID, and set up the second disk as a file-copied backup... [This] would lose a (small) amount of resilience, but wouldn't suffer from the RAID-sync issues, ideally a Mac-like "TimeMachine" backup would handle file histories.
Any recommendations?
This is also a good time to share your experiences with Storage Spaces, so leave your answers in the comments. What's the best way to backup large amounts of personal data?
b) Move to "Storage Spaces". I've not used Storage Spaces before, but reports seem to show it's good... It's a Good Thing that the disks are 100% identical and removable and readable separately. Downside? Unknown territory.
c) Break the RAID, and set up the second disk as a file-copied backup... [This] would lose a (small) amount of resilience, but wouldn't suffer from the RAID-sync issues, ideally a Mac-like "TimeMachine" backup would handle file histories.
Any recommendations?
This is also a good time to share your experiences with Storage Spaces, so leave your answers in the comments. What's the best way to backup large amounts of personal data?
Memorize it! Just don't take any head injuries or you won't remember anything.
More seriously, back up to hard drives is the only viable option. Then make sure you have more than one backup drive and store one at some other site. Relative maybe?
Cloud options with that kind of storage would take forever to upload. And I've heard of people having stuff randomly go missing on their cloud service, not the entire contents, but a file here and there. I'm not so sure that's a good option.
For storing on-site you can get a fire rated media safe, but they can be quite a bit more expensive than a regular safe.
2 Terabytes is nothing.
Here's how you do this:
10 You buy an external hard disk that is 4 Terabytes or larger, and USB 3.0.
20 Copy the fucking files to that thing.
You're done. Now you have two copies: one on whatever bad idea you have as your main drive, and the other on a physically separate drive.
Not good enough? GOTO 10
Say with with me: "RAID is not backup!"
Backblaze is what I use, if your backup isn't off-site, then it isn't really backup...
You can also burn to DVDs or BR or use external hard drives and move them offsite, but that takes time and effort...
Two drives with copies of the same files sitting side by side is not backup.
1) RAID IS NOT BACKUP unless you have another read only set.
2) STORAGE SPACES IS NOT BACKUP unless you have another read only set, and please, it is JBOD with some added features.
3) You are exchanging RAID sync issues with backup sync issues.
I would setup hardware RAID, but that is not related to what you need... Backup to two other disks. Upgrade disk size and technology as needed. A 4TB disk is like $140
Better to get it offsite. One fire/flood/etc. and your data is toast. Not too mention that RAID IS NOT BACKUP (RINB).
I'm a "serious amateur" photographer (about 1TB of photos currently) and I've been using CrashPlan for the last two years and I'm happy with it. They allow you to create a local encryption key that even they don't know so it seems pretty secure. The first upload can take a while (depending on your internet service) but everything is quick after that point.
In addition to that I also use TimeMachine on my Mac so I have a local backup of everything.
https://www.backblaze.com/clou...
$5/month unlimited data size (writes).
You can sync files back over or they will actually ship you a HD with your data; if you return the drive you get a refund of the drive cost but you're also free to keep it.
The cost for individual file reads is reasonable too.
No muss no fuss
RAID is fine to reduce downtime, but completely unsuitable as a replacement for backup.
The RAID does not have the following things which you critically need from backup (the following list is not complete):
- resilience against operator error (accidentally delete/overwrite files, e.g.)
- geographic redundancy, usually not even safe against the box killing the disks, lightening, fire, theft, etc.
- too few copies: Usually 3 (!) independent backup copies used in rotation are considered the minimum. RAID1 gives you one and it is not independent.
My recommendation is to get at least 3 external USB disks, and establish a backup with them, because currently you have none.
Steps:
- Select a backup interval. This represents the maximum time-interval for which you think losing new data is acceptable
- At the end of each interval, do the following:
1. Fetch oldest backup disk from off-site location
2. Put backup copy on it, making it the newest backup. Make sure to do a file-by-file comparison.
3. Move disk to off-site location
For somewhat reduced reliability keep the oldest copy at home and do the following:
1. Make backup, overwriting oldest copy. Make sure to do a file-by-file comparison.
2. Move new backup to off-site location and fetch oldest from off-site location.
An "off-site location" can be anything from a garden-shack to a storage locker at work to an arrangement with a neighbor or a friend you see regularly.
If you think this it too much effort, then your data must not be worth much. This is pretty much the agreed minimum experienced sysadmins want. Of course, there are always those that never lost any important data and they almost universally think this is way too much effort. Many of them learn in time when whatever they do results in that loss.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Get BackBlaze or Carbonite. It's ~$50.00 per year. 1PC. No size limit. I had my house broken into and the thieves took my computer and all the backups. Also a fire could wipe out all your backups, if stored in the same place.
I have the following:
1) 1 SDD that I work on and another that is mirrored every day. If one disk fails, I have another. This is my working disk.
2) Incremential backup of data that changes often, like emails or some directories I work in. Mostly use if I delete a file by accident. Just copy it back and be done. This goes to a NAS.
3) Data that does not changes often, like movies, images and music is stored on a NAS.
4) Second NAS to backup the data of the first NAS.
5) Essential data (less than 10MB) is put on my website on a personal directory. This is data that I might need in case of the house burning down.
So when something goes wrong (unless the house burns down, but the I have other problems and my music is not one of them.) I have a way to restore it.
The most important thing however is not to backup, but the knowledge on how to restore it. You need to test that out from time to time. I have people seen who did backups to /dev/null to test it and forgot to remove that parameter.
What you can do if you REALLY need to have things off site, like photos and other things that you can't replace is just buy a dedicated HD that you put this data on and keep it in a drawer at your office. Once a month or so you take it home and add the new data.
And if that disk is full, buy a new one or a bigger one. If data is really THAT important, the price of the HD is well worth it.
But again, test the restore.
Don't fight for your country, if your country does not fight for you.
Presumably the OP has his pics on the SDs used in the cameras that took the shots. That is, unless the SDs were reformatted and used again. Many folks I know just buy new SD cards and retain the old ones as first level storage. Keep them in a fireproof safe or safe deposit box at a bank. After that follow the best practices for backing up including cloud storage.
In a time of universal deceit, telling the truth is a revolutionary act. George Orwell
The odds backblaze or crashplan or carbonite spontaneously decide to shutdown without notice the same day my house burns down is pretty low, after all.
How about backing up to DVDs?. They hold lots of data. Had an article here the other day about optical drives...
C|N>K
Go onto ebay and buy an 2nd hand LTO3 or LTO4 tape drive for $150 - $300. Plug it in, write your files to the tape. For 2tb you would need 3 LTO3 tapes (assuming compression 800gb each). Take said tapes and drive them to another house.
Decide what timeframe of loss is acceptable. ie 4 weeks, 4 months, 12 months. That is you maximum backup cycle time. Every X period of time take a new set of tapes to your offsite backup location. Buy tapes equal to at least 3 full cycles, that way on your third backup trip you take the oldest set home and re-use them.
This is the process I use every 3 months and I have an HP Ultrium 960 sitting on top of my NAS. I also use your normal google drive type backup, but it is my second stage, rather than first stage backup. I'm not quite at the same size as you, 1.1tb, so it's 2 tapes not 3. I bought a box of 50 new lto3 tapes for $100.
And on the minus-side, it is not a backup at all.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Because fire, theft or even a careless delete statement would take it down.
Normally people want to protect their data from more than just hardware failure
You forgot checksumming and verification after transfer.....You have something on the other drive after the transfer, you wont know what until you verify it.
By the tits of Baal, rsync or xcopy /v or robocopy in combination with fciv.
For the 45th time in this thread, RAID is not backup. And all of you who are saying yes it is will change your minds the first time your array blows up and you have no other backup. Let's say you're running RAID 1 or 5. A drive dies. You stick in a new drive. You now better be praying and sacrificing animals in the hopes that you don't have another drive die before the array is rebuilt, which could take 12 hours or more if you're using 2TB or larger drives. If you value your files, then you have something in addition to your RAID array.
If you seriously value your files, you also have a fully automatic offsite backup, and one that retains older versions. I use CrashPlan. $5.99 a month for unlimited backup of one machine. As of the time I'm typing this, I have approximately 860,000 files amounting to 2.6TB backed up (semi-pro photographer). Yes, that first backup took about 3 months, but you gotta start sometime.
I want a new quote. One that won't spill. One that don't cost too much. Or come in a pill.
That doesn't organize your data. That just dumps in one pile. How do you *find* anything is the problem. The best app I have found to do this is called NeoFinder. It allows you to organize all your files in one application, tag them with metadata, create thumbnails of common formats and the best part of it is it allows you to store the applications library data file ON the server. It's the only app that allows you to do this, everyone else forces you to store it client side. Bundle this with a ZFS server and Samba and you are golden! It runs on OS X and Windows. It's not open source but I don't care because every other OS "solution" to managing/organizing/tagging files is utter fucking trash at best and a tire fire at worst.
C'mon, online backup? Really? The poster said "terabytes." Cable companies in this area say "hundreds of kilobits per second" as an upload speed. That'd be 10's of kilobytes per second. How long? Get optimistic at, say, 800 kbps -> 80 - 100 kBps and you have a really long time. Lessee, 2 X 10^12 bytes / 1 X 10^5 kB/s = 2 X 10^7 seconds = 20 million seconds to upload 2 terabytes. 20 X 10^6 seconds / 3.6 X 10^3 seconds / hour = about 5.5 X 10^3 hours, or 5,500 hours. 5,500 hours / 24 hours / day = 229 days. I aborted Carbonite some years ago when I had only a couple hundred gigabytes,it was _NOT_ uploading every single file on my disk, and looked like it was going to exceed 3 weeks to do it.
--Umm, you do realize that SSDs are:
a) WAY expensive for backing things up to, and
b) An un-powered SSD drive will eventually degrade and LOSE ITS DATA in a fairly short amount of time (for Backup purposes)? This gets worse with Triple-level-and-up (TLC) Cell structures, BTW. They basically need an electric refresh to keep the cell structure from flipping to another position.
--Depending on the temperature/humidity it's stored in, SSD degradation could be detected in as low as several months or - if you're lucky - possibly as much as a couple of years. But if you don't fire it up every so often and run a data-consistency check, how would you know if your files are succumbing to bit-rot?
--There are many, many more options for backups that don't cost *nearly* as much as SSDs - that's not really what they're intended for. I can see buying an SSD if you want faster startup times on your PC, are into gaming, or you do a lot of virtualization suspending/resuming (R/W multiple gigabytes) every day. SSD's are designed to be faster than spinning disks, NOT necessarily long-lasting without power.
--For now, it looks like the best thing to do is keep your data online, have multiple rotating backups, store some stuff off-site, and copy data from old-drive to new-drive before it breaks. (I would even say real-time Mirroring or RAIDing is getting to be essential for any disk over 1-2TB.) But if you're storing your main backups on SSD media, you're over-spending *and* may be risking data loss if you don't power up the drive every so often.
--JMHO, but I would look into something like M-DISC for reasonable amounts of long-term archival storage. 4.7GB DVD M-Discs were made to the highest standard; 51% sure about the 25GB Blu-Ray M-Discs, not sure about the 100GB BD-R multi-layer discs. (Cloud backup is OK I guess as long as you don't mind 3-letter-agency snooping and you don't have a slow Internet with data caps, but encryption is definitely recommended before uploading.)
Refs:
http://www.makeuseof.com/tag/h...
http://www.anandtech.com/show/...
https://www.maketecheasier.com...
.
== WolfriderV6 == I'm willing to admit that *I just might* be wrong... Are you??
He is backing up photos, why are you assuming a 2TB working set every day?
A weekly full -->2TB with daily incrementals --> maybe 20MBs doesn't equal 10s of TB per week.
I agree on springing for the extra cost of ECC memory (motherboard + ECC RAM) on a FreeNAS box but let's be realistic here on the volume of backup necessary.