Slashdot Mirror
Ask Slashdot: How Do You Manage Your Personal Data?
New submitter multimediavt writes "Ok, here's my problem. I have a lot of personal data! (And, no, it's not pr0n, warez, or anything the MPAA or RIAA would be concerned about.) I am realizing that I need to keep at least one spare drive the same size as my largest drive around in case of failure, or the need to reformat a drive due to corrupt file system issues. In my particular case I have a few external drives ranging in size from 200 GB to 2 TB (none with any more than 15 available), and the 2 TB drive is giving me fits at the moment so I need to move the data off and reformat the drive to see if it's just a file system issue or a component issue. I don't have 1.6 TB of free space anywhere and came to the above realization that an empty spare drive the size of my largest drive was needed. If I had a RAID I would have the same needs should a drive fail for some reason and the file system needed rebuilding. I am hitting a wall, and I am guessing that I am not the only one reaching this conclusion. This is my personal data and it is starting to become unbelievably unruly to deal with as far as data integrity and security are concerned. This problem is only going to get worse, and I'm sorry 'The Cloud' is not an acceptable nor practical solution. Tape for an individual as a backup mechanism is economically not feasible. Blu-ray Disc only holds 50 GB at best case and takes forever to backup any large amount of data, along with a great deal of human intervention in the process. So, as an individual with a large data collection and not a large budget, what do you see as options for now (other than keeping a spare blank drive around), and what do you see down the road that might help us deal with issues like this?"
I think you already have the answer
It's great that you know how fast his connection is and exactly what data restrictions his ISP imposes. I'm actually rather impressed you can be 100% sure his computer is connected to the internet at all. All I know is that if I had that much data, the time it would take to upload would probably be longer than the time it takes for the HDD to wear down and implode.
Seriously. How much crap do you really need to keep around?
So your disks are full and possibly broken. You don't want to have more disks, you don't want tape or optical medias, and a storage provider (aka The Cloud) is not an option... Then you have three solutions "down the road":
1) Delete stuff
2) Invent a new compression algorithm that will allow you to reuse the same disks forever without losing data
3) Rely on magic*
*might overlap with solution #2
lucm, indeed.
If there's some personal data you're missing at some point, just ask Google or the NSA ... But seriously, I've never made backups and not even bothered to copy over stuff from old PCs to newer ones when I upgraded (I keep old hard drives in a closet just in case there's something old I'm missing, but I never really do). The only personal stuff I keep safe is images on my iPhone (backed up on the PC) and email (safe-ish on the server at work). If I needed more space, I'd go with Wuala due to its relative safety (redundant storage, client side encryption) - but it's only free for 2GB or so nowdays. But ask yourself: do you really need all that data? I don't think so.
"I love my job, but I hate talking to people like you" (Freddie Mercury)
Obviously you still have an issue of tracking things down on the rare occasions when you actually need some of your family photos. But you can rest assured that they're in there somewhere and weren't purged last time you needed a few GB for more webserver logs.
Maybe the first step is to de-dup the existing data. You'll still have some manual intervention to check possible duplicates, but it's a first step towards tackling the bigger problem.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
I think it's time to admit that you're a hoarder. What exactly -is- your personal data that's so precious? I run a server just to keep my skill set up and run my side business, but I've only managed to accumulate around 600GB of data, only about 35GB of it is 'mine', the rest is client backups.
So first admit that you're a hoarder, then decide if you wan to address that issue or indulge it. If you choose to indulge it, you're going to want to build a small home server. Something with a low-end 64-bit CPU (i3?), a gigabit LAN port, and lots SATA ports and 3.5" drive bays. Buy a bunch of high-quality (WD RE4?) matching drives that fit your data needs times two (you're RAIDing space away). Once you have that, install Linux on it, build a software RAID-1 or 0+1 array (don't do RAID-5 unless you can handle days of rebuild time), and format it with something accessible (read: in the kernel, like EXT4). Create a share on the array with Samba and happily access it from all your machines (don't bother with Netatalk or NFS; CIFS is great on all platforms). As your data needs grow, you can add drives in pairs or replace drives with larger ones and grow the volume. If you need backup, you'll want another array, preferably on another low-end box (an enclosure on your desktop?) but it can be built on a RAID-0 or JBOD to save money.
"Sometimes, I think Trent just needs a cup of hot chocolate and a blankie." -Tori Amos on Nine Inch Nails
It's that time again, is it?
http://ask.slashdot.org/comments.pl?sid=2452630&cid=37557630
Either.. ..or..
A: Buy that HDD. Yes, they're a bit more expensive right now
B: Wait a few months, prices will come down again, buy that HDD then. Yes, you may lose your data in the mean time.
Now stop asking or I'm going to pull over.
In typical "I need IT advice, but I have preconceived notions about how things should work and am not willing to budge on that" fashion, the asker has discounted some reasonable options without specifying the reasons that won't work for him, and failed to provide some super useful info like how large his data actually is, how often it changes, how much existing data changes, how much new data there is, and how quickly it grows.
So it could be that the reasons for his concern are unmerited, and GP merely points out that if his concern is privacy, there's ways to use the cloud safely. In typical Slashdot fashion, you rebuke the potential shortcomings of the advice without knowing whether those shortcomings actually apply to the asker.
Backup should be provided in depth, several prongs provides the best redundancy and the least single points of failure. Cloud storage is an excellent option for one of the prongs given certain factors. If most of the data rarely changes (pretty typical for very large data sets), incremental bandwidth usage past the initial storage is usually not much more than the data growth rate. As observed, it can be done in a way that respects privacy and safety.
Cloud storage has two main advantages over local backup solutions. You won't run out of disk space, and it's off-site (so a house fire won't take out your data set). Any on-site solution automatically fails that level of redundancy. Storage on S3 is ridiculously inexpensive any more.
I have about 6 TB of data that I need to keep backed up. I have about 12 years of digital photography and video originals, including stuff like wedding and honeymoon photos, as well as the birth and first years of my children's lives. When people suffer house fires, one of the most common and greatest laments are the things that can't be replaced - usually photographs.
My solution is four tiers. I have a local RAID0 in my Mac Pro. I have Time Machine backups of that (this is hands-down the best consumer on-site backup solution on the market). I rsync those files to a local RAID10 NAS device (Synology are a bit pricy, but they are completely worth it, really excellent built-in software with a lot of features you might find surprisingly useful, and you can purchase expansion bays to extend capacity as you're running low). Then finally I back up to Amazon S3 in encrypted form with JungleDisk (I no longer recommend this software, I own a copy of it from before it was bought by RackSpace, the quality has gone down since RackSpace bought it and "improved" it, plus I gather you now have to pay a monthly subscription, AND pay for your own storage - crap).
The only way my data is in jeopardy is if my house burns down (takes out 3 local redundancy & backup solutions) on the same day that Amazon has critical failure. And it's all 100% automated, Raid0 happens at time of write, TimeMachine alerts me if there's problems creating a backup and gives me local history, my NAS warns me by email & SMS if it so much as writes too slowly (my rsync cron script emails me if it can't reach the NAS for some reason), and JungleDisk does a nightly sync with S3, and sends me weekly reports so I can be sure that it's doing its job. I have quick local access, and slow offsite access if everything else fails (I'd probably go bum my work's huge pipe to do the initial restore if I had to rely on that).
"Storage on S3 is ridiculously inexpensive any more.
I have about 6 TB of data that I need to keep backed up."
So you mean that 6000/month*0.125$=750$/month is cheap?
Or did I miss something?
I gave up with the idea of an useful sig...
My solution is four tiers. I have a local RAID0 in my Mac Pro. ...
You do understand what RAID0 is, right? RAID0 is strictly for performance and offers zero data redundancy or failure protection. In fact, since you need both disks to function to read your data - you're essentially halving the MTBF of using one disk. Perhaps you meant RAID1? (a mirrored set)
Did you actually use it for recovery?
Both my rotated TimeMachines were corrupt. They never complained during backups, but failed miserably while trying to recover my Pictures HDD.
Only some of the backup files were corrupt, but when you try to recover a complete disk with TM, it's all or nothing, and the process stops after the first error, leaving you in the dust.
I had to write a parsing script with ruby, "cp -avX", ditto and chmod in order to get my system back.
It wasn't so hard, but it sure was stressful with one disk down, two corrupt disks and no other backup to get my pictures back.
BTW, TimeMachine doesn't backup every file in your system, and is too stupid to realize that it should not begin from scratch after recovery : it needs twice the storage after that, because it thinks every file is new.
My drives weren't big enough, so I had to wipe the backups and lose the local history.
Fuck it. I began using Carbon Copy Cloner since then, and never looked back.
It's free as in donationware, it works, it gives you a bootable backup that you can actually test and rotate properly, it can easily be automated, it archives the files that you've deleted between backups, and uses much less space than TimeMachine.
I hear SuperDuper is just as good.
TimeMachine is some crappy software with nice looking interface that gives you a false sense of security.