Slashdot Mirror
Ask Slashdot: How Do You Manage Your Personal Data?
New submitter multimediavt writes "Ok, here's my problem. I have a lot of personal data! (And, no, it's not pr0n, warez, or anything the MPAA or RIAA would be concerned about.) I am realizing that I need to keep at least one spare drive the same size as my largest drive around in case of failure, or the need to reformat a drive due to corrupt file system issues. In my particular case I have a few external drives ranging in size from 200 GB to 2 TB (none with any more than 15 available), and the 2 TB drive is giving me fits at the moment so I need to move the data off and reformat the drive to see if it's just a file system issue or a component issue. I don't have 1.6 TB of free space anywhere and came to the above realization that an empty spare drive the size of my largest drive was needed. If I had a RAID I would have the same needs should a drive fail for some reason and the file system needed rebuilding. I am hitting a wall, and I am guessing that I am not the only one reaching this conclusion. This is my personal data and it is starting to become unbelievably unruly to deal with as far as data integrity and security are concerned. This problem is only going to get worse, and I'm sorry 'The Cloud' is not an acceptable nor practical solution. Tape for an individual as a backup mechanism is economically not feasible. Blu-ray Disc only holds 50 GB at best case and takes forever to backup any large amount of data, along with a great deal of human intervention in the process. So, as an individual with a large data collection and not a large budget, what do you see as options for now (other than keeping a spare blank drive around), and what do you see down the road that might help us deal with issues like this?"
1. Buy hard drive from brand A
2. Buy hard drive from brand B
3. put in seperate esata enclosures
4. backup to both drives.
"I'm sorry 'The Cloud' is not an acceptable nor practical solution." Not sure what brand tin-foil hat you're wearing, but there are cloud backup solutions that encrypt your data *before* it leaves the machine. I use CrashPlan (I can't speak for others) and I've verified the encryption myself by capturing the traffic leaving my machine, even when CrashPlan was backing up to other machines on my own private network. Even the data it writes to locally-attached hard drives is encrypted. So there's at least one company who gets it right.
One way to save a bit of cash is to buy a USB eSATA drive dock (single or double) with some bare eSATA drives. This cuts the enclosure out, and allows you to buy bare drives, which are often cheaper than enclosed drives.
You could also consider Drobo or one of the Wiebetech multi-drive RAID containers. But encryption + cloud isn't all bad.
"large data collection and not a large budget"
This is your problem right there. You can't enter into a a situation like this without planning a budget for the inevitable failures. I suggest purchasing a new larger drive (3TB are common now) and migrating the data from the problematic drive. Then migrate the data from several older smaller drives. This will reduce the component count (points of failure), save you power (cost in the long run) and keep you ahead of failures. You should plan on doing this periodically to maintain the integrity of the data.
Can You Say Linux? I Knew That You Could.
Agreed. I've been gradually rotating larger backup drives in and smaller backup drives out over the last 10 years or so. Right now I have about 2 TB's of unique data in my archive which is kept on the host machine if it is regularly accessed or duplicated on another external hard drive. Everything (I care about) has two copies at all times. As my archive grows, I'm going to have to upgrade my archive device's capacity, but that's a given, no matter what you do, if you want it stored locally, you'll have to add capacity somewhere obviously. DVD-R's and BluRay discs aren't a viable option in my opinion, because I've got a ton of old self-burned discs that I recently had to toss because they were rendered useless from laser rot, even though they were in sealed containers in a cool, dry place.
The cloud is, to me, not a backup solution. I see it as a way to globally access my data and I use it as such. No sensitive data of mine will go to the cloud because the likelihood of needing access to it without warning is completely nil, so in my case, it's limited to media that I want constant access to. Now, the cloud definitely has the potential to serve as a backup solution, don't get me wrong, but there's just too much uncertainty involved in the cloud these days, especially as concerns the government nuking sites from orbit without warning, whether justified or not.
However, I agree with some others that are telling you to do some house-cleaning. I recently went through my backups and found 300 GB's worth of crap that I hadn't accessed or used dating back to the early 2000's that I was saving for some stupid reason. Disc Images for ancient games that don't even run well on modern systems (or require a lot of fucking hassle to get running well), music that I haven't listened to in half a decade, old-ass videos that I'd downloaded from the internet back before there was such a thing as youtube, etc. Not to say that everyone's data is as silly as mine was, but it just added up over the years...
This doesn't help much for those of us with crappy internet - I've only got about 300K (bits) upload speed, and at that speed backing up 1TB would take around a year.
FWIW, my strategy is to keep truly important stuff on a raid enclosure (and backup to other disks periodically), and to just live with the fact that there's really nothing irreplaceable about the rest.
https://www.googleapis.com/urlshortener/v1/url?shortUrl=http://goo.gl/rIh07 { "kind": "urlshortener#url", "id": "http://goo.gl/rIh07", "longUrl": "http://www.backblaze.com/partner/af3012", "status": "OK" } Trying to sell cloud solutions on Slashdot? You must be new here.
Link contains a referral ID, so Shikaku is earning from this, but not willing to say so.
Eventually, it ends up at http://www.backblaze.com/
You nearly said it, but I'll just come out and ask: what is this vast quantity of personal data? If it isn't downloaded movies or rips of DVDs the OP owns... Maybe he's a film maker or prolific but unpublished musician.
We need more info because the only option, other than deleting stuff and throwing money at the problem, is some clever solution based on the specific data in question.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
Right. Other than buying new disks, there is no good solution.
The asker seems to be looking for some kind of "join all my small disks together" solution. And yes, he can do this. RAID-0 or LVM. But... don't do it! If even one of those disks fails all the data is effectively gone. The solution is cheap to implement but totally worthless. Sorry, your 250Gb SATA disk now belongs in a museum.
RAID-5/6 is, IMO, also a bad idea; there are too many instances where the controller has failed or multiple disks have failed.
The asker explicitly excludes cloud solutions. It's depressing that people have recommended various cloud solutions nonetheless. Apart from not being answers to the question, these solutions are totally awful for large quantities of data. Amazon S3 may be nearly free if you want to store a few gigabytes, but if you want to store a few terabytes you are going to pay through the nose, and all the other service providers are the same. 2Tb would cost $234 per month just for storage, transfer cost not included. For the price of two weeks of S3 storage you can buy a 2Tb external disk. For the price of upload, download and a month's storage, you can buy four or five such disks and have as much redundancy as any normal person could ever need.
You're an immobile computer, remember?
RAID sucks as a backup because if you accidentally delete a file off your RAID storage, it gets deleted from all the drives in the RAID. Your file is not safe as it would be on a backup.
RAID is for redundancy. So you don't have any downtime if a HDD fails. Without RAID, a HDD failure would mean downtime until you can get a new drive and restore from a backup. With RAID, your array and your business keeps chugging along as if there were no failure, and you can replace your failed HDD at your leisure.
Rebuilding a RAID array with a failed drive has been simple and automatic in my experience. Pop out the dead drive, plug in the new one, and it'll start rebuilding automatically. Your data is still accessible during the rebuild, although access times and transfer speeds may be degraded. Depending on the amount of data, a rebuild can take anywhere from a few hours to a few days. A second failure while rebuilding means all your data is gone. So you want to keep backups of everything on your RAID array.
If you just want to glom a bunch of old drives together to use as a backup drive, you want a multi-bay JBOD/RAID enclosure like this or this. Be forewarned that if you plug these in over eSATA, you need an eSATA port with port multiplication. No laptop eSATA port I've found does, so you'll need to rely on USB or built-in hardware RAID/JBOD to use these with a laptop.
If you want something which will sit on your network acting as a file server, you want a NAS like this or this or this. You can read NAS comparisons at Small Net Builder. But keep in mind what I said above - even if you get a NAS, you will still need to make backups of it.
Unless, of course, you get the safe that's rated for computer media.
This is an area where you really need to RTFTS (tech specs) as it will tell you EXACTLY what kind of fire, temperature and duration it will protect a specific kind of media for.
Reading code is like reading the dictionary - you have to read half of it before you can go back and understand it.