Slashdot Mirror
Ask Slashdot: How Do You Manage Your Personal Data?
New submitter multimediavt writes "Ok, here's my problem. I have a lot of personal data! (And, no, it's not pr0n, warez, or anything the MPAA or RIAA would be concerned about.) I am realizing that I need to keep at least one spare drive the same size as my largest drive around in case of failure, or the need to reformat a drive due to corrupt file system issues. In my particular case I have a few external drives ranging in size from 200 GB to 2 TB (none with any more than 15 available), and the 2 TB drive is giving me fits at the moment so I need to move the data off and reformat the drive to see if it's just a file system issue or a component issue. I don't have 1.6 TB of free space anywhere and came to the above realization that an empty spare drive the size of my largest drive was needed. If I had a RAID I would have the same needs should a drive fail for some reason and the file system needed rebuilding. I am hitting a wall, and I am guessing that I am not the only one reaching this conclusion. This is my personal data and it is starting to become unbelievably unruly to deal with as far as data integrity and security are concerned. This problem is only going to get worse, and I'm sorry 'The Cloud' is not an acceptable nor practical solution. Tape for an individual as a backup mechanism is economically not feasible. Blu-ray Disc only holds 50 GB at best case and takes forever to backup any large amount of data, along with a great deal of human intervention in the process. So, as an individual with a large data collection and not a large budget, what do you see as options for now (other than keeping a spare blank drive around), and what do you see down the road that might help us deal with issues like this?"
I think you already have the answer
1. Buy hard drive from brand A
2. Buy hard drive from brand B
3. put in seperate esata enclosures
4. backup to both drives.
I run a RAID5 array on a spare box for backups, totaling 8TB before file system and RAID takes out its chunk. It's only turned on during backups, and is a fairly cheap solution for lots of storage if you look for sales on drives.
Sounds like you have the digital diogenes syndrom. What about you tidy up your room.. i mean, data ?
"I'm sorry 'The Cloud' is not an acceptable nor practical solution." Not sure what brand tin-foil hat you're wearing, but there are cloud backup solutions that encrypt your data *before* it leaves the machine. I use CrashPlan (I can't speak for others) and I've verified the encryption myself by capturing the traffic leaving my machine, even when CrashPlan was backing up to other machines on my own private network. Even the data it writes to locally-attached hard drives is encrypted. So there's at least one company who gets it right.
One way to save a bit of cash is to buy a USB eSATA drive dock (single or double) with some bare eSATA drives. This cuts the enclosure out, and allows you to buy bare drives, which are often cheaper than enclosed drives.
You could also consider Drobo or one of the Wiebetech multi-drive RAID containers. But encryption + cloud isn't all bad.
"large data collection and not a large budget"
This is your problem right there. You can't enter into a a situation like this without planning a budget for the inevitable failures. I suggest purchasing a new larger drive (3TB are common now) and migrating the data from the problematic drive. Then migrate the data from several older smaller drives. This will reduce the component count (points of failure), save you power (cost in the long run) and keep you ahead of failures. You should plan on doing this periodically to maintain the integrity of the data.
Can You Say Linux? I Knew That You Could.
I have a solution I call the "Buddy NAS". Go out and get two cheap computers. It could be a PC or a mini-NAS or a low-end server. Anything that will hold multiple hard drives. You jam both full of hard disks and use them as a backup/NAS server. One PC is kept at your place, the other at your friend's house.
Both computers have an account for you and an account for your friend (it helps if your friend is nerdy and "gets" backup solutions). Both of you now have a backup solution in your own home and a remote backup server at a friend's place. Two copies of your data, one remote. Basically it's like having local and cloud storage for you and your friend and it'll cost less than a grand if you shop around. If neither of you have static IPs you can use dyndns.org to connect to the remote boxes. Bandwidth shouldn't be an issue if you use rsync to backup changed files nightly.
I have invested in USB backup drives of about the same total capacity as my primary storage drives. Yes, that's a lot of hard drive space for backups, but it's really the only practical solution that I have found. Just think of it as the cost of not losing all that data to the inevitable drive failure.
An external eSATA drive dock and a stack of 2TB drives might be a somewhat better way to go about it, at least backups and restores would be faster than the USB drives.
I can offer a couple of suggestions... What I did was buy a used Dell Poweredge 2950 on eBay for about $500 bucks shipped and I added 4 x 1tb SATA drives to it and I run a raid 5 setup with 3tb of usable space across the four 1tb drives. This solution cost me less then $1000 and I have a nice playground to experiment with VMWare ESXi.
I know that's not exactly budget conscience but it works great for me.
If I were on a tight budget I would just buy a 2tb USB drive from Newegg or somewhere similar. It looks like you can buy a name brand for about $130 bucks.
If you have a little bit more money to spend you could always buy a couple of 2tb internal SATA drives and run RAID-1 mirroring on them. You could put these into an old computer and make a little NAS linux server...
If you're saying you have no money to spend then maybe you need to consider cleaning up your data. Often times all those "personal files" that you think you need to keep... Really aren't required. Just my 2 cents but this problem is very solvable.
Seriously. How much crap do you really need to keep around?
Cheapo used market PC, invest in some large drives and a couple of drive docks, install FreeNAS.
Take a weekend to organize your data however it makes sense (by year, subject, file type, whatever), and store it on a particular drive. Rinse wash and repeat. Depending on how important the data is, store in a fireproof safe onsite, or offsite. When (read: if) you need the data again, dock the drive and retrieve.
Personally, I'm about to liberate myself from years of data. I'm tired of all these bloody drives and the annual, "I really want to look at _______ again". It's amazing how much of that crap has zero personal value anymore. (This isn't a comment on your data, but mine.)
#SickNotWeak
One, you can't have enough backup images as something always seems to go wrong. Should include at least one offline unplugged "safe" unit. I know, it's a hassle to keep them up to date.
Two, the longer you wait the less all this backup space costs, so don't buy too much too soon.
So your disks are full and possibly broken. You don't want to have more disks, you don't want tape or optical medias, and a storage provider (aka The Cloud) is not an option... Then you have three solutions "down the road":
1) Delete stuff
2) Invent a new compression algorithm that will allow you to reuse the same disks forever without losing data
3) Rely on magic*
*might overlap with solution #2
lucm, indeed.
It depends on the data, but many formats compress really well when using WinRAR. Many of my files, for example, that reach nearly 10:1 compression. Unless we are in the same profession, I wouldn't set your expectations that high, but I imagine on average you could get your data usage down to 40%. If I'm right, maybe you could winRAR several folders from the failing drive to the smaller drives, and not necessarily need to get more space available.
That said, I really do think the suggestion of buying another drive is spot on. I saw a 2 terrabyte drive for $120 at Best Buy yesterday.
"I like to lick butts!" by MobileTatsu-NJG (#32700246) (Score:5, Informative)
As several people have said you already answered the question yourself. Spare HDD + Blueray.
You can achieve what you want by also changing the way you think about your data.
How much of your personal data is live? As in, how much of it do you access constantly, and need immediate access to?
Here's what I do, I have discrete HDD set up for each data type (not needed...but I had spare ~500gb drives so it's how I did it) There are broken down to Music, Projects, Video, and Photos. Each of them is synced monthly to a 2TB external drive that is spun up only to do a differential backup.
Data that I haven't accessed in 6 months (mostly phots and old closed projects) is moved to Archival grade DVD and removed from the Archival HDD.
So irreplaceable things (3 decades of photos, years of work) are stored and can be accessed within a few moments, less important but commonly accessed stuff (music and instructional videos, or documents I use every day) are live and backed up on the Archive.
*A)bort, R)etry, I)nfluence with large hammer.*
You must give up some of it, or transfer it to some other, long-lived medium.
Otherwhise, I suggest you face reality and invest accordingly
Windows 2000 - from the guys who brought us edlin
Drobo -> mostly reliable local backup
BackBlaze -> mostly reliable offsite backup
You might want to substitute a ZFS-based FreeNAS for the Drobo, if you're so inclined. It's less automatic, but seems just as reliable.
All of my personal data is in my home directory and easily backed up to non-volatile media (which I do a few times a year, but not as often as I should.)
All of the project data is on SourceForge or company project servers, so there are duplicate copies of that.
I hardly think of my music or movies as "personal" data nor as irreplaceable. Were I still playing video games, I don't think I'd bother backing up game data, either.
When people talk about needing entire drives for their personal data backups, I have to wonder: WTF have you got ON there?
I do not fail; I succeed at finding out what does not work.
If there's some personal data you're missing at some point, just ask Google or the NSA ... But seriously, I've never made backups and not even bothered to copy over stuff from old PCs to newer ones when I upgraded (I keep old hard drives in a closet just in case there's something old I'm missing, but I never really do). The only personal stuff I keep safe is images on my iPhone (backed up on the PC) and email (safe-ish on the server at work). If I needed more space, I'd go with Wuala due to its relative safety (redundant storage, client side encryption) - but it's only free for 2GB or so nowdays. But ask yourself: do you really need all that data? I don't think so.
"I love my job, but I hate talking to people like you" (Freddie Mercury)
Obviously you still have an issue of tracking things down on the rare occasions when you actually need some of your family photos. But you can rest assured that they're in there somewhere and weren't purged last time you needed a few GB for more webserver logs.
Maybe the first step is to de-dup the existing data. You'll still have some manual intervention to check possible duplicates, but it's a first step towards tackling the bigger problem.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
I have a large amount of personal data as well (also, no pr0n), though I realized some place along the line that it wasn't that important not to lose it. When I die most of will probably be tossed anyhow. Who is going to want to sort through it all.
With that said, I still don't want to get rid of all my data, so I have a drobo with 5-2TB drives, and I also have a linux box with raid set up, that backs up the drobo. But really, think hard about how much effort and expense you want to put towards keeping data. There is probably a whole lot of it that can just be tossed because you will never look at it again. It will probably save you a bunch of time and money to go through all the data and get rid of the stuff you'll never actually use. This actually works well for garages too.
My fileserver uses RAID and makes a separate (encrypted) backup to an external USB hard drive (fortunately, my data hasn't grown faster than hard drive sizes so I can fit it all on a single 2TB drive, to ensure file integrity, periodically I have rsync verify file checksums,)
As a secondary backup, I use a 1TB notebook drive locked in a USB enabled fire safe:
http://www.amazon.com/SentrySafe-QA0121-Fire-Safe-Waterproof-Storage/dp/B00166187Q/ref=cm_cr_pr_product_top
I used metal straps to tie it down and lock it to my computer desk in the hope that if someone comes in to steal the computer, they'll just grab it and run without prying off the data safe. The safe is only rated for 30 minutes @ 1500 degrees so it's not a perfect solution, but better than nothing.
For my really important data (old tax returns, scanned in records and receipts, etc) I back them up to the cloud. For photos, I keep the full-size image locally (some TIFFs, mostly JPG's), but keep a lower res lower quality image in the cloud. All of this is less than 20GB.
Most of my big data is DVD's that I've ripped and I'll count on insurance to replace them if they are lost - I don't even back them up to the drive in the fire safe.
I had a similar problem. I had let a friend borrow an external for a backup and it came back write protected! He was using windows, I mac and linux, and could not figure out why/ how this happened. I was at capacity on that drive as well as two others and needed space bad (needed to back up a laptop before installing new os). A few other friends came to my rescue and allowed me to borrow some of their drives until i could find a more permanent solution. The suggestion to build a raid box is a good one.It will allow you to build up your storage capacity over time. My advice is to use multiple methods: back up locally AND send your most essential data to the cloud via an encrypted service (this will protect against a fire, theft, etc. Plus its handy to have access to your files from any computer). I like wuala personally, but spideroak is also very nice solution as well. There are lots of good, secure solutions now that are relatively affordable
I think it's time to admit that you're a hoarder. What exactly -is- your personal data that's so precious? I run a server just to keep my skill set up and run my side business, but I've only managed to accumulate around 600GB of data, only about 35GB of it is 'mine', the rest is client backups.
So first admit that you're a hoarder, then decide if you wan to address that issue or indulge it. If you choose to indulge it, you're going to want to build a small home server. Something with a low-end 64-bit CPU (i3?), a gigabit LAN port, and lots SATA ports and 3.5" drive bays. Buy a bunch of high-quality (WD RE4?) matching drives that fit your data needs times two (you're RAIDing space away). Once you have that, install Linux on it, build a software RAID-1 or 0+1 array (don't do RAID-5 unless you can handle days of rebuild time), and format it with something accessible (read: in the kernel, like EXT4). Create a share on the array with Samba and happily access it from all your machines (don't bother with Netatalk or NFS; CIFS is great on all platforms). As your data needs grow, you can add drives in pairs or replace drives with larger ones and grow the volume. If you need backup, you'll want another array, preferably on another low-end box (an enclosure on your desktop?) but it can be built on a RAID-0 or JBOD to save money.
"Sometimes, I think Trent just needs a cup of hot chocolate and a blankie." -Tori Amos on Nine Inch Nails
It's that time again, is it?
http://ask.slashdot.org/comments.pl?sid=2452630&cid=37557630
Either.. ..or..
A: Buy that HDD. Yes, they're a bit more expensive right now
B: Wait a few months, prices will come down again, buy that HDD then. Yes, you may lose your data in the mean time.
Now stop asking or I'm going to pull over.
Drives die, sometimes without warning (and old statistic by IBM says 50% of the time there is no warning). You could just throw everything away, as you are going to lose it anyways, sooner or later. Or you could find the resources needed to make sure you have everything on at the very least two drives (one of which should not be connected or running). There is nothing that can replace reasonable backup.
As a side note: Common sysadmin wisdom is to have 3 independent backups in addition to the original.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
You might be on the next spinoff of Hoarders programs, a digital hoarders show.
In this show, redundancy, old versions, and files that haven't been opened in 5+ years are brought into question, for which you will be embarrased to defend... You will attempt to justify why you still have linksys drivers for a wrt54g you don't even have anymore. And no, the DVD ISO of the Alvin and the Chipmunks movie, that you never burned or watched, is not worth saving.... Neither are about 85% of the digital pictures you took (you know, the ones that were the 'bad shot' that you took before finally getting the good one).
Take a day or two, go through it chunk by chunk, and purge! PURGE!
Compression is your friend. Also, weeding out things you don't use. Hording is problem in the digital age as well.
A friend and I each have FreeNas servers with multi TB raid-z.
Some of our data we keep mirrored between them.
The servers are physically brought together occasionally for a full sync, but most of the time rsync -n is done over the internet to see what needs to be updated and the data transferred on a removable hard drive.
1984 was not supposed to be an instruction manual.
I just put everything on Facebook -- hey, where's everybody going?
Floppies
More floppies
More floppies
(I can sell you some cheap!)
Chaos maximizes locally around me.
I was given a p4 optiplex desktop which I am using for my fileserver. I have a pair of 1.5tb disks and a pair of 2tb disks in mirrors running freeNAS on a usb thumbdrive. The optiplex is great because is quiet, fast enough and the case conveniently fits 4 drives(and free). I have a cheap 4 sata card for the drives and I splurged on a decent intel gigabit nic after couple other cheap ones with no luck (asus and dlink I believe). If you do the thumbdrive thing, make sure and buy two and dd the contents over, one failed on me and now I have to manually manage my raid via cli. Of course this does not protect me from myself, but I haven't lost too much due to my own stupidity. I used to have a larger fileserver with all sorts of little drives, but I replaced them with a couple big ones and I'm much happier for it. Lots of people swear by drobos. One person I know plugs a decent usb drive into his hacked router. Rsnapshot is good for automatic incremental backups.
As others have noted, if you can't afford AT LEAST another drive, serious problem right off the bat. One wonders what the data is worth given this.
I'll move on, assume the data is worth AT LEAST another drive or two (we're talking a couple hundred bucks at most, come on):
1. drbd: raid to a low cost, remote machine with similar sized drive. Dead drive is now recoverable.
2. amanda or similar backup to drive on remote machine. No, not tape, just virtual on disk. Now have a backup history as well in case one needs that file that one deleted 6 months ago.
Yes, cost is a couple hundred and some older machines, but really, what is your data worth in the first place?
Anything is possible given time and money.
Get older version LTO drive, get identical harddisks with spares and do full drive image backups with incremental backups in-between.
Mirroring disks (Raid 0+1) is good too.
All that other stuff - DVD/Blue Ray, cloud is chickenshit. Drives die or act up.
Run Linux on one drive and do image backups from there.
Zerofill the empty space,
dd if=/dev/zero | split --verbose -b 2000m -d - ZERO
compress the image:
dd if=$DEVICE | gzip -v | split --verbose -b 2000m -d - NAME
and write the chunks to tape.
People have different needs. Some needs are imposed by either employers or the wonderful US Govt. for mandatory data retention. Others are your life's design work that you want to retain until you die. Other data you want to pass to your kids. If you can't afford to lose it keep multiple backups on multiple media in multiple locations. Books & pamphlets have been written on this. Transfer the data to new media once a year or two or three & keep all working drives.
No single storage device local or remote is immune from disaster. The Alexandria Library succumbed and took with it countless early human treasures. Wars have done in archives all over the world. Lightning, outages and power surges can defeat the best protections even when electronic equipment is turned off, but still plugged in (laptops are better when left unplugged, which is actually a great asset).
Backup is one thing; recovery is another and it can be GUT WRENCHING. The recovery process needs as much thought as backup.
A Clue or Two: A business partner had his MBPro backed up to 2 external HDs. Not great, but OK. Said MBPro crashed on the Lion upgrade. No way to know whether it was hardware or software and the MBPro should have at that point been off limits for use until carefully checked out. He happens to live in an area subject to lightning and outages which can affect anyone (even with a UPS). However, he reinstalled the Snow Leopard and plugged the first BU HD in an attempt to reload the data; HD became corrupted. Should have stopped, but then the 2nd HD was corrupted. Moral of the story; Recover data from a backup to an external HD running on another computer than the one that got mucked up.
The cost of 3-4 external 2-3 Terabyte hard drives and a couple cases or RAID box is dirt cheap compared to the value of the hours you put in on your computer each year as are Blue Ray drives & disks.
Caution: Someone on this list mentioned putting drives and disks in a "fireproof safe" or "fireproof file cabinet"; wrong! The UL approved boxes are designed only to protect "paper" for a given amount of time in a typical fire by releasing steam (212 deg. F = goodbye DVD/BR disks). Once the fireproof agent uses up its water...Farenheight 456 takes care of all contents...permanently. This is why multiple locations are needed.
Except the latest batch of Synology machines (like my recent DS1512+) seem to have a hardware flaw. You reboot them and they die. And then just TRY getting Synology tech support over the weekend........NOT HAPPENING.
Raid and an NAS is a great thing. My Infrant (Netgear) NV has been solid for 5 some years now. Synology seems like a nice product, but if the thing is going to freeze up and die on a reboot and I can't get tech support for it over the weekend........fuck that.
It's either on the beat or off the beat, it's that easy.
I moderate therefore I rule!
--
SparkleShare looks and works like Dropbox, but is actually just a fancy automated self-hosted GIT repo, (which you can interact with using GIT commands on a remote repo if that is what you want to do).
The wiki explains how to encrypt things (and the encfs recipe doc'd on the wiki also works with Dropbox, etc.)
I think the project has matured really well, but still isn't really well-known, and doesn't even get mentioned much on the slashdots, although that's where I heard about it.
www.sparkleshare.org
https://github.com/hbons/SparkleShare/wiki
You can't be ahead of the curve, if you're stuck in a loop.
I'm assuming to start with that you have backups of everything in some fashion, with which you could put it all back together if your biggest drive suddenly failed spectacularly.
In that case, how important is uptime to you? Since this is personal data, I'm guessing that you could live without live access to it for a few days. And given that, I think your best bet is not to keep a spare just sitting around, but to only buy one when you need it. Hard disk prices keep going down, and the price for the same drive six months from now is almost guaranteed to be lower than it is now, barring things like the Thailand floods. The other big advantage of this method is that when you upgrade to bigger drives, you don't have to immediately upgrade your spare as well -- and if you manage to go a whole upgrade cycle without needing the spare, then you've saved yourself the purchase of an entire drive.
At the moment, because your drive is possibly failing, then yes you need to get that spare. But if the current drive is actually failing, it won't be a spare so much as a replacement, and then you're back to the same situation.
I was in a very similar situation about 3 months ago, but with 80+ gigs of data. I had pictures that don't compress well, personal documents all sorts that needed to save. I used to just back them up to an external drive. Then i would hear stories about something happening to peoples computes and their external drive that was left connected. I never wanted to go to the "cloud" and loose my data, besides I had a very slow DSL connection so uploading would be a pain. I went researched the cloud solutions went round and round then held my nose and purchased a subscription to Mozy. Yes it took about 4 weeks to get my data upload. Of course i was using the internet connection for other things at the time also. At the time I figured what is 4 weeks when i have gone for so long with very few other options. After everything was done I now have Mozy running in the background and every few house it backs up. I love it I feel more comfortable about my data being safe. I don't have to worry as much about it. I still use my external drives for things like an extra copy of my Quicken data but that may change as I change my habits as Quicken data is also backed up by Mozy. For me I worry, what could happen to a computer, drive, even raid corruption, fire, flood, tornado, etc. I am paying for it and i know some want something free or less cost, but if i want to spend less i guess i could use the free version of Mozy and just do a few docs and no pictures. However family pictures are just as important as some of my personal documents. I know some may prefer one of the other backup services but right now I can say I am impressed with Mozy, even on my slow connection.
I do much the same, with a few minor tweaks:
- Rather than NFS, I use a tarpipe: Source box uses xinetd. It's a little faster, and uses less processor time at the backup box. Which is important, because...
- The backups box also compresses the data it gets (pigz or bpzip2, depending where the data is from) and encrypts it (gpg - have to use assymetric encryption so the backup box doesn't need to store the key) in case of theft. It takes rather a lot of processing power to handle this, but a Core 2 Quad does the job nicely.
As you're running it over the internet you'd want to replace the xinetd-and-tar with an ssh connection (You can make ssh run tar directly in place of a shell) in order to avoid sending plaintext over the internet. Depending on the data, you might want to move the compression to the source device as well so reduce the pain inflicted upon your usage caps.
SDLT tape drive and some tapes. If your "personal data" is not worth the $800.00 to buy a good used SDLT drive and a few tapes, then it's not worth backing up.
Just do not dink around with theoretical "backup solutions" that are not proven. and no, hard drives are not "proven" for reliable and long term backup. I have DLT tapes from 16 years ago that I know for a fact I can still read.
If your data is important, You dont screw around with consumer hard drives that are known to have a low MTBF.
Do not look at laser with remaining good eye.
My personal and family data (not including ripped DVDs etc) are about 1 TB. Mostly photographs and video with my DSLR so the files tend to get large...but I also have a ton of documents, app installs, and all sorts of misc data. I must admit I'd be curious as to what fills multiple external HDs for personal data but to each their own.
Good organization outweighs medium in my case. 2xExternal 2 TB HDs - primary and secondary...and then a third stored off site at my parents that I update about 3 times a year, so if the worst happens I'm 6 mons out of date, but its usually about 4. And thats if both my primary and secondary go down. Thats a cost of about $300 total and a little a bit of effort.
"A little bit of effort" is defined by how you organize. Backing up manually means I don't rely on software or a service, but it requires some forethought. For me I break it up by data type and usually year...sometimes I go one more by how that data was acquired (photos I add who took the picture). This is important because I put anything new into a diff folder so I know whats new and whats not. It took me a couple of years to get to the structure I have but I sometimes add small tweaks. The effort or time now is fairly miniscule.
What I'm trying to get at is this : if you're prepared to put a small amount of time in every now and again, with an initial overhead, you can do this very easily and cheaply.
Example:I move my archivable personal data off my HD as gzipped tarballs, and regularly backup my home and root directories separately as gzipped tarballs (that way I have a history of backups too, just in case). On ther other hand, I rsync my music collection separately because despite its size, I use all of it regularly, but I don't want it weighing down every single backup of my home. So you may have to come up with your own solutions for your own specific situation, but in general: consolidate and optimize!
stripe the drives, export the whole pile as a share and copy everything to the server.
Christ, this is 2012, storage is a solved problem.
I want to delete my account but Slashdot doesn't allow it.
I have a local Microsoft Homeserver for onsite backups. I also have an external drive encrypted with TruCrypt (opensource) where I spin off every 90 days my loved ones (files) just in case the house burns down. I take this external drive to work and lock it in my file cabinet. It is encrypted to make it difficult for anyone to steal the data but not the drive. I have too much data (like most people) between pictures, music and other personal data to play with uploading to the internet or writing to any optical disk. Obviously my risk is that both drive copies die at the same time.
Megaupload
Whether YOUR data is of concern to the CopyrightStaatsPolezei doesn't matter. The moment Hollywood sees some site as infringing, their allies in "our" government and elsewhere will bring it down. You may note that the many innocent users of Megaupload, including, if some reports are correct, some government agencies, STILL don't have their data back and may never get it back.
What you don't own, you don't control.
is the nerdy way to go, I did and it kicks ass specially when the ex-gf deleted my stuff
When I started my own business, I got myself a safety deposit box. Every time I go to the bank to cash my cheques, I use an external 3.5" HD mount, backup critical files using a script, take a hard drive into the bank and remove last month's copy. It has the added benefit of being fire and earthquake proof (two things I won't have if I stored them in my home.
I only have about 200GB of data I consider "critical" though, but the same thing would scale up. At ~$40/year for a small safety deposit box it's one of the cheapest and most robust mechanisms for keeping your critical data safe.
I have a second internal drive that is automatically synced with the main hard drive that contains all the important data. (No RAID, just plain old backup.) Moreover, all important data is backed up from one machine to another and to an online server.
For both of it I'm using Crashplan with a long-term contract. It's quite affordable and the software works really fine both for online and local backups. I don't understand why the "cloud" is not a solution apart from the obvious fact that you should avoid companies that use the word "cloud" too often. I've been using Jungledisk and Crashplan without complaint so far. Then again, I've never been struck by disaster so I don't know how easy it is to get the data back.
If you're storing a lot of illegal content like pirated movies then you'd probably be better off with choosing a European backup provider, but in any case you need to have encrypted online backup if you really care about the integrity of your data. No local backup can safe you from a flood or a fire.
If the data isn't worth a few $120 2TB hard drives then it's not worth keeping, is it? 3TB hard drives are around $200-$225 also.
You ruled out: hard drives, tape, discs and cloud storage. What exactly do you expect us to say here? There isn't some other magical form of storing data we've been hiding from you.
Of my four internal drives, only one has non-recoverable data on it (business and personal). The others are things like games and applications which can always be reinstalled. So a second drive is the same size as the first, and a nightly backup combination of windows backup and a few scripts copies that data from the first to the second drive. I like having recent backups fully accessible.
On the long-term side, every few weeks, or months, I copy from the second drive to a drive in the closet -- which really should be off-site, but I don't really worry about those types of things in my life.
So between the two, I can really only lose a single day of work. For anything particularly impressive, I just drag the file to the second drive when I'm done that work, to make me feel better.
But that's it. At this point, the closet has about ten drives covering the last ten years. The second drive has a good two years on it, fully compressed of course. And it takes me about two years to fill a backup drive. Of course, my business files are mostly text. If I were playing with more images than I do, I'd go through drives about ten times as quickly, which would mean one backup drive per quarter. Which is still only $300 per year these days, so that's fine.
First, let's look at your problem: You are gathering too much data. Either the data is 100% needed and irreplaceable, or it isn't. If it isn't, your first step is to treat your data just like you would physical junk that accumulates in your house.
Create Three folders.
1. Critical Keep
2. Unsure
3. Toss
Go through your data and MOVE it to one of those three folders. If it isn't critically important data that you would be upset that you lost and can't be recreated (wedding videos, etc) It goes in the Critical Keep folder. If you aren't sure about it right now, but you can't declare it for folder 1, put it in 2. Anything else "old install files, backup data from a windows 98 machine, etc" That stuff can be deleted. Be harsh with yourself. Think of it like moving from house to house, if you haven't opened that box by your third move, just toss it in folder 3.
Repeat the process until you either have everything in your Critical Keep folder, or your delete folder.
Now, hopefully you have reduced the size of the data you are using to something marginally manageable. I'm a data hoarder, and I've managed to keep the rate of growth of my data to lag behind the general rate of growth of HDD capacity. Now for the fun stuff:
Two things you want to avoid.
1. Loss due to a dying disk
2. Loss due to a destroyed home (fire, theft, etc)
Here was my budget solution that resulted in a fire and forget backup system that is suitable for a home user and is about as minimal as you can get for cost.
3 Disk Drives.
A primary drive to run the operating system and hold installed programs and two LARGE data drives in a RAID1 configuration.
Static data files (Video, pictures, etc) get stored on the RAID1
A scheduled process (once per month for me) backs up the OS drive to a virtual HD file on the RAID1. The files on the RAID1 are then backed up to a cloud storage service (Carbonite in my case).
So, what is the result of this?
My operating environment is backed up monthly. The only thing I lose here is configuration changes or programs installed since the last backup (less than 30 days for me)
The RAID1 ensures that my personal/static data is protected from a single disk failure, and helps a bit with read performance for the static (and large) files.
Should a cataclysmic failure occur and my entire computer is lost to something like a fire, remember that I've been sending what is on the RAID0 out to the cloud (carbonite), so when I can rebuild a computer I can just download the (very large) offsite backup from the cloud to my new machine.
The downsides I have right now:
1. I maintain the windows backup as a VHD file because it allows me to ensure that the backup data is 'packaged'. I don't know the exact details about windows backup, but given that Carbonite sometimes excludes system files I didn't want to risk an important hidden/system file being missed in the backup. In addition I didn't like how it could only backup to the root folder of a drive. The downside is that the resulting 100GB file is a pain to backup, which is why I restrict the backup histerisis to 30days (previously I had it backup every 3 days) This keeps it from continually uploading the VHD file to carbonite.
2. The HDDs for the raid1 lose half their total capacity in that configuration. I used it because it let me only have to use 2 drives and the performance boost. If you can afford 3 drives, go for a RAID5.
3. Most Motherboards support RAID natively now. However, I understand that you can run into issues with hardware RAID if you have to switch to a different hardware solution. I haven't tested this, but it could potentially be an issue if you use a RAID5 from hardware and your motherboard fails and you can't replace it with an exact model. The good news here though, is if you have been backing up to the cloud, typically it's done on a per file basis, and thus you don't have to worry about this. Just download your stuff ba
Out of modpoints but really liked a post? 1BDkF6TtmmeZ3yqXbz9yhdYVqRYnwFoXDj
split(1)
And a lot of Gmail accounts. About 300, I think.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
unRAID (cheap, expandable, protected data) coupled with inexpensive offsite disk backup (eg your parents house, providing you don't live there already!).
Delete your porn
The rest of your personal data will fit on a floppy.
We're up to 2 NAS units now, with 7TB[*] of disk space between them, all backed up on schedule. The USB backup drives are rotated every few weeks with another set kept in a secure place in the garage.
[*] One NAS unit doubles up as media server, so it's got a load of movies & music in addition to user files in its 6TB. The other one is our web server and email server with only 1TB of disk space.
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
There is important data, and then there is IMPORTANT data.
Movies, games, music, documents are important data; that is to say I care about having it easily accessible, and at my fingertips, but I wouldn't lose more than a night's rest were it to disappear. Code is IMPORTANT data; that is to say that if it were to disappear, heads would roll.
As such, and owing to the evil overlord rule of not having one of anything important, the important stuff is in plain sight, and exists in several variant copies; it only requires a mind of uncommon origin to understand where it is, and what it's worth. Finally, I keep the general ideas behind the important stuff memorized, in case those copies disappear. To this end, I apply the general rule behind the fictional philosopher's stone-> anyone who is meant to use my stuff would understand it, and anyone who is not will not; thus, it's only the intelligence of the seeker than limits them. It has been successful in keeping the bungling burglar, the thuggish criminal, and the demanding tyrant at bay.
This all comes down to another evil overlord rule: the IMPORTANT stuff will not be clearly labeled as such.
I am John Hurt.
The words "Fireproof Safe" is by the definition of Underwriter Labs. It merely refers to being fire resistive for a given amount of hours in a typical fire "for paper".
Forget DVDs and CDs and any hard drives surviving a fire in one of these "fireproof" devices. They are designed to release steam to keep the temperature at 212 def F until the fireproofing material exhausts all its water at which time the temperature goes up and, well...you can imagine what.
My solution to this problem is painfully simple: about 5 years ago I bought 5 drives 500GB each. I have put a server (made from old parts, like pentium IV and so on) in the basement (where nobody hears it, and it can be as noisy as hell). I installed debian on it and configured cron to call rsnapshot three times per day for doing automatic backups of all PCs in my family. I never touched this machine since then.
With one exception: 3 years ago I started to run out of space, so I bought 2 HDDs 2 TB each, reconfigured raid6, which was extremely easy because for raid I am using mdadm, which supports such operations online. Also I had few more spare drives during the years, so I kept adding them to the array, and currently there are 9 HDDs in this PC. It is very noisy, but nobody cares about that.
It runs flawlessy, untouched for years, and nobody cares about it, except for when somebody in my family accidentally loses or deletes a file. Then suddenly backup comes very handy.
Rsnapshot is especially good, because it keeps hardlinked copies of data from last week, 2 weeks ago, last month, and much more, depending on how you configure /etc/rsnapshot.conf. Currently I have backups dating back about 2 years, with granularity of 1 month. And it only occupies the space on HDD to reflect the changes between data, thanks to hardlinks.
So my raid6 array has total size about 4TB and still 500GB free. And I feel this will last at least a year or two. In case of problems I can start deleting copies that are more than 1 year old. While most recent snapshot uses about 2 TB or such.
Rsnapshot also can backup windows machines, so you don't need to worry about compatibility. Though I don't have windows machines and I don't test that in practice ;)
#
#\ @ ? Colonize Mars
#
best way to slim down = calculate folder size then sort by size. go from biggest to smallest. osx has it built in and if you have win7 FS-Inspect does the job.
I organize all my data into three categories, (a) stuff I need backup including old versions, (b) stuff I want backup of latest version, and (c) stuff I don't mind losing.
Two external backup drives, one is kept in another location, the other at home. Every once in a while I switch the two, so there is always an offsite copy that is at most a few months old.
Simple and stupid setup for making copies; each external drive is formatted with an encrypted filesystem. Made some scripts for copying with "rsync" for files which go into categories (a) and (b). Using some additional scripts for recursive "copying" of old versions with hard linking (on an ext filesystem) to preserve history for files in category (a).
Keeping an off-site backup is important to protect against some risks such as fire, and it also helps prevent losing data in case some malicious software or accidental "rm -rf" or the like would wipe your backup. Keeping old versions of category (a) data helps protect against a scenario where master data is lost, and an rsync operation removes the data on the backup.
If you want to be really paranoid, you could make the backup from a remote computer so that if the master you are backing up is compromised, it would not be able to wipe the backup (at least not "type a" data). Personally I found it was not worth the hassle, given the redundancy added by the offsite backup.
If this is data which is important to you, keep in mind a RAID really gives you very little protection. It only offers some protection against hardware failure, but any virus or accidental wiping will kill your data just as well as it had been a single disk. If the data is important, you need at least one external backup.
You really, really want offsite backup in addition to whatever you do at home/office. If "the cloud" works for you, fine, but for many the bandwidth issues are a big problem
One option is to have, in addition to whatever drives you run live, enough extra drives to back up your data twice. Keep one set at home, and another in a safe deposit box. Depending on your risk tolerance, you can use packaged drives, or carefully swap bare drives in eSata enclosures or the like. Backup as often as you can on the set at home. Then every few weeks/months whatever meets your needs, do a swap; put the up-to-date backups in the safe deposit box; take the other set home and use it for new fresh backups. With that as a base, you can usually use cloud-based solutions to make sure there are daily or immediate offsite backups of truly critical data that's changing often.
Trust me: disasters do happen. You can lose a lot if you haven't prepared.
Buy more harddrives. Seriously, just ask yourself how much pain it would be if would loose the data. It's easily worth a couple of hundred dollars/euros.
As for what strategies, incremental vs 1:1 backups etc. I have recently written two blog entries for a filmschool where I regularly teach where they have the same problem - huge amounts of data, little cash, loose organisation. Have a read if you like (the newest blog post covers incremental backups with hardlinks under windows and osx which has some nice properties but is not always the best solution) http://www.danielbachler.de/node/121
I have 2 10TB RAID 6 servers running Debian that sync every night. There is also a emergency backup on Windows that is just 4 2.5 tb drives chained together.
only one everything
news for you, pal, "the game' has nothing whatsoever to do with electronic technology. It's been going on for ages. Just as a for-instance, the Roman Catholic church together with monarchs would mindfuck their subjects with B.S. about religion and patriotism, to take the worker's wealth and have power and dominion. Old, old scam, same old shit different century.
might be needing an expensive pci-express card and cable to go with that, cost goes to +$2,000 to get started eh?
Repeat:
Buy a new 2TB drive, copy your data from your smallest drives to that one, then sell them at eBay.
until you have all your data on new 2TB drives
At the end you should have money for an extra drive.
If you like (and use Linux) you can start a RAID5 with one drive (+1 missing), copy your data onto it, then add another drive, do a reshape, migrate your data...
And at the end add the last drive to regain redundancy..
I would suggest cleaning up unneeded data. I know it is not the kind of answer you are looking for but I am sure there are things that you do not really need, maybe duplicates or older version of sets of files. By making a throughout clean-up, you can probably gain many GB, you then might have enough space to swap your files around to be able to reformat your fleaky drive. Yon then have the time to think over to find a more durable solution for when this happens again.
"Storage on S3 is ridiculously inexpensive any more.
I have about 6 TB of data that I need to keep backed up."
So you mean that 6000/month*0.125$=750$/month is cheap?
Or did I miss something?
I gave up with the idea of an useful sig...
Basically here is what I do:
Incremental ghost of the OS to a separate drive 3 times a week with a full on Sunday, on the alternate days it does a file based backup of require data to same backup hard drive of everything that has changed. No deletion of filebased backup but Ghost gets overwritten weekly
Monday I bring home a portable hard drive using truecrypt from work and hook it up. Over night it does a mirror of the backup drive which gets brought back to work on Tuesday.
I am only an individual so I really do not have to worry about having a data farm or anything else like that. I have had decent success with cloud storage. I went with SpiderOak and that haven't been bad. It is fairly cheap (about a $1 per 10gb), has clients that will run in Mac, Linux, Windows, Android, iOS, and your data can be accessed anywhere with an internet connection by logging into your account on their site. They claim you data is encrypted before it leaves your PC to their servers and remains encrypted. SpiderOak also claims that they do not have access to your encryption keys so they can never look at your data. What I use is not the best service in the world, they are comparable to Dropbox but doesn't have the features that Dropbox does, but SpiderOak gets the job done. I just needed something that I could keep a backup of my important stuff on and if I need to reformat/reinstall for whatever reason I can get my important data back. However, I stumbled upon the Cyphertite project the other day and I am considering giving that a try.
The question really is how much do you value your data? A little? A lot? My solution is a dual solution (albeit still waiting for the 2nd part to arrive). Online I have a subscription to CrashPlan (although there are other various services available which will do a similar job). You can get the software which will backup your computer (or selected folders) to another computer with the software installed over the internet (e.g. your parents if there is enough free space). If you pay a subscription you can back up your files encrypted to CrashPlans servers (and I think you can even put in your own encryption key), albeit it can take a few days to do this. You can even get family packs for multiple computers.
The 2nd part for which I am waiting is a networked attached storage - I am getting a Synology product, although again there are other companies making these. The model I am getting will have 2 spare bays for hard disks of your choosing, and then you can run a backup on your computer to these which will keep the discs up to date. You can also use this as a file server, as well as a media server, bittorrent client etc. (see the synology website if you are really interested). You can stuff a couple of 2TB drives in there and even implement some sort of RAID.
So you can then have an onsite and an offsite backup with a NAS and crashplan. The 3rd part of the solution probably is to trim down what you store as I can vouch I have a lot of crap that really doesn't need to be saved. Then do regular backups of the really important bits (for me this is not my itunes folder) to DVD-R.
Overall it comes down to how much is your data worth and how much are you willing to spend?
This is beginning to worry me as well. I have several machines on a network and a 500GB external drive. A lot of my files are my own work including a lot of video which I would really not like to lose. So far I have made 1 copy of everything on DVD, but I am not happy with that, especially as I now have found that DVD-R seems to have a shelf life, as discs that I could read now seem to have errors, although this could be down to drive issues.
I would love to get off site storage using an on-line system, but it would swamp my broadband just to get the current files saved, and I think BT would look askance at me maxing out my upload bandwidth for hours on end to cope with the backlog.
I recently put a 200GB drive out of a broken laptop into a USB case and I am putting some files on that, but given its history I cannot really trust it. I do wonder if some of the old tricks of using a VHS tape might be worth looking at again, at least they are cheap and I think that if proper storage conditions are available they would have decent life.
I once thought that files that came from the net didn't need to be backed up, but I am now finding that some files are no longer there, dead web sites or "new" versions which don't actually have the same content. So this is going to make things worse as every file I really like or use is going to have to be held locally.
For me this is just a matter of entertainment or hobby interest, so I cannot justify (or afford) a high cost tech fix for the problem, but perhaps we have found a niche market for a low cost / true archive quality storage device for home use>
2 > profit !
so all we need is step 1 > invent device.
nec sorte nec fato
I have a 6x2TB array and a 4x320gb array in my server, the former with HD204UI drives on an Intel SASUC8I controller, and the latter with various 320GB notebook drives I've collected, in a 6-bay 5.25" rack on the motherboard's controller. There's also a 120GB SSD in that rack to run ESXi and most of the guest OSes.
We keep all of our crap on the 4x320 array, which is backed up to the other array.
For data to last it has to live. Data lives by continually replicating itself. ... more replication.
For quite a few years now hard drives have been the only economical and practical backup solution for mere mortals (enterprise is moving there too). The best way to work around drive failures is
Right now my setup involves continual backup to a RAID 1 drive with Time Machine (use whatever works on Windows/Linux) which is then rsynced to a remote server I setup at somebody else's house as burglar backup (after an initial local rsync on the source machine and sneaker netting the hard drive). This gives my data 4 drives to live in and I'm not sure I'm happy with that.
And now there's 200 comments where the people are proud of their kilowatt server arrays which are powered 24 hours a day for their PHOTO ALBUMS? Are you people shitting me? I mean, you're putting me on, right? You don't really use up 10,000 kWH per year storing your family photos, do you?
Hey, I've just invented the electric elevator-button-pusher. I save a TON of finger wear and tear.
Sometimes, humanity makes me sad.
RAID sucks as a backup because if you accidentally delete a file off your RAID storage, it gets deleted from all the drives in the RAID. Your file is not safe as it would be on a backup.
RAID is for redundancy. So you don't have any downtime if a HDD fails. Without RAID, a HDD failure would mean downtime until you can get a new drive and restore from a backup. With RAID, your array and your business keeps chugging along as if there were no failure, and you can replace your failed HDD at your leisure.
Rebuilding a RAID array with a failed drive has been simple and automatic in my experience. Pop out the dead drive, plug in the new one, and it'll start rebuilding automatically. Your data is still accessible during the rebuild, although access times and transfer speeds may be degraded. Depending on the amount of data, a rebuild can take anywhere from a few hours to a few days. A second failure while rebuilding means all your data is gone. So you want to keep backups of everything on your RAID array.
If you just want to glom a bunch of old drives together to use as a backup drive, you want a multi-bay JBOD/RAID enclosure like this or this. Be forewarned that if you plug these in over eSATA, you need an eSATA port with port multiplication. No laptop eSATA port I've found does, so you'll need to rely on USB or built-in hardware RAID/JBOD to use these with a laptop.
If you want something which will sit on your network acting as a file server, you want a NAS like this or this or this. You can read NAS comparisons at Small Net Builder. But keep in mind what I said above - even if you get a NAS, you will still need to make backups of it.
A 2TB drive is something like $120.
Is protecting your data worth more to you than $120?
If yes, buy another drive. If not, erase a bunch of junk you've accumulated.
I use a Sans Digital 5 bay external SATA raid array for all the pics from 2 Canon digital cameras. http://www.newegg.com/Product/Product.aspx?Item=N82E16816111172 It's not backup. It's storage. I've had it for over a year, and it's pretty damn solid. It is NOT cheap, and you'll want a reasonably hefty UPS to handle it and the computer it's on. ( On the other hand, it is not Expensive either, for what you get. ) You MUST go the sans digital website and, pretty much only, buy the drives they recommend. If you do research and get RAID ready drives, you can call them and discuss with them your drive of choice, and then buy, at your own risk of course. I went this route. Bought the whole thing, external enclosure & SATA card, Drives, thru newegg. It does need a slot for the card. It slows your boot down. It is fast. It's RAID 5 w/ a hot spare as configured( i think). For Disaster recovery, you will still need something else. Overall, I love the thing. For backup, you might be able to get away with one of those top-loading sata drive holders. Move stuff off to two cheapy drives and store someplace offsite. Good luck.
I recently had a similar issue and now use: Local: 3x 3TB Hitachi high end drives RAID 5 in external enclosure. Remote: Data that is really critical gets synced using rsnapshot (incrementally) to 'my own cloud', a dedicated server running Debian that I pay about $60 per month for (Fasthosts). It also runs a web/email/WebDAV server which is handy to access the incremental backups by the web. This way I get redundancy with fast access and decent security (all over ssh). Running your own cloud is pretty trivial these days, well worth the effort.
HDDs are kinda cheap.
Have the manufacturers recovered from last year's Thailand flood yet?
Not trying to troll here. I'm serious. Consider that you might be simply storing too much stuff.
What are you asking? The ONLY answer is buy an extra
drive(s) that match the total amount of stuff you don't want
to lose, copy it all over.
Problem solved.
Anything less than this answer is less. More is buying double
the amount of stuff you don't want to lose and keep a rotation
of your backups.
Further, break down each group of stuff you collect each month
or quarterly and burn to a DVD.
Don't forget you have to store these either offsite or in a fire-proof
storage, or best... both. Otherwise, it's all an exercise in futility.
Did you really just ask this question in Slashdot? Not sure what's
worse, plain obvious questions that have been either asked so
many times or is answered with a simple google search or obvious
promotion of advertisers stuff.
I used to be proud (and snarky of course) when I said that I read /. /. is failing. Nothing NEW is covered here any more. It's
Now, I don't mention it... it's irrelevant to the newest generation and
the current generation I'm sure has noticed that even as an news
aggregator
always old by the time it hits the front page sometimes painfully so.
It's also obvious what stories get picked up and which ones get looked
over. And then, to break up the monotony...
"How should I back up my data that I don't want to lose?"
effin brilliant
-AI
For me, it is far better to grasp the Universe as it really is than to persist in delusion
My setup:
Currently, they setup's been running for 2 years, and I've only ever used the online backup service for testing my backups, and I've never had to hot-swap a drive. I've got almost 2 TB of data in use, and I've finished ripping my DVD collection, so growth is slow now. My TV can play all my video files straight from the QNAP, and our phones can access it to download songs/audiobooks/videos through our local Wi-Fi.
Just because you're paranoid doesn't mean there isn't an invisible demon about to eat your face
A single 3TB external USB3 drive should address your needs nicely.
Just unplug it when not actually using it (to protect against power surges, lightning, etc) so it can be considered a backup[1].
I acknowledge the unplugging/replugging thing can be a pain if you need it frequently. What I'd really like to see is a hard switch of some kind that can physically interrupt both the power and USB lines to a device. It wouldn't need to cut the mains power, just the 12V/5V after the transformer, but would still need to give enough separation to prevent arcing in the event of a lightning strike.
[1] Recall that an online backup is no backup at all.
"Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
I am making my way through the comments, and want to clarify a little. I am not talking about backup. I am asking about disaster recovery or just plain drive maintenance tasks that should be done annually. The drives are my backup. Yes, good corporate data storage practice is to have spare drives around. I am talking about home. How many have 2 TB drives sitting empty on a shelf at home, just in case? I don't know anyone, personally, and I know hundreds of geek admin types of all ages and experience levels, myself included. We usually buy storage upgrades as needed and seldom have current technology, large drives just laying around because we're using them! Other than that, great stuff so far. Thanks all.
Here's my scheme: /data/ (not the OS) from all my systems. I usually buy the 'sweet spot' of most GB/$ even if it's more than I need right now. Powered off except when used.
- USB 3.0 External drive, as big as I need to back up all my
- Once a week I rsync it all over with a single script. Use cwrsync for Windows.
- If it runs out of room, then it's time to buy a bigger one and swap out the smallest drive in one of the current systems with the old backup drive. I wipe the old small drive and put it in a pile, just in case, but they rarely get used ever again.
- A small amount of really critical stuff I save off-site (rsync again) nightly with an automated script.
I haven't lost any significant data in 20 years doing it like this (though it wasn't as convenient or fast 20 years ago), even though I have had systems crater - it's easy to restore from a drive.
Backup to optical is so annoying I end up not doing it, and tape is clumsy and expensive (and for me has a very poor record of actually being able to restore anything) but if it's just a matter of turning on the external drive and running a script I can do it religiously.
Every 15 minutes my important data is Rsynced to my colleagues house and his over to mine. We then each have a USB hard disk which Rsyncs all of the local data. Works pretty well and ensures no data is lost of we have a fire/flood etc. All secured over SSH.
Duplicity backup software: $5/mo (donated to EFF)
FDCServers Atom + 2TB HDD: $45/mo
Comcast Internet with 20 Mbps upload: $130/mo
Running into your 250 GB transfer cap in just 24 hours: priceless.
Daniel
Unless you are on a very limited budget, don't reuse old drives for primary backup.
Drives are cheap. Buy a new 1.5TB drive as your primary backup.
If you really want to be careful, use your old drives for secondary backup
I probably have 30 drives on my shelf after years of backups.
If you buy used at least. I bought a 16 tape LTO3 library and 20 LTO 3 tapes for around $500 used on ebay about a year ago. It might be even cheaper now. And if you don't mind changing tapes every hour or so, LTO3 stand alone drive can be had for $200. Also, if you're only going to deal with only 1TB worth of data for a while, LTO2 is more than enough, and a used LTO2 autoloader can be had for under $200. Hard drives are never a proper backup solution. The data can be lost(without paying a few thousands for recovery at least) any time you plug in the hard drive. The tape solution is just so much more stable as a data storage platform, I'd say you look into getting used LTO2 autoloader at least. They really shouldn't cost more than a couple of hard drives.
Seriously? 10 tapes x $20/tape = $200/TB, almost twice as much as a 1 TB drive would cost you. And "tape lasts forever"? Hell no!
I use a 3TB (4TB -1) OWC Mercury RAID 5 array that is backed up constantly via Apple Time Machine.
Time Machine rocks. By the time it starts bucketing data, it will be at the 3-year mark. I keep my eye on the disk health, and am prepared to swap out as necessary.
I keep an external drive rsynced to my main drive, so I have an immediate backup, if necessary. I have used it, on occasion (sucks when I do -my primary drive is an SSD).
That's my personal data. I have a Mini that uses a Drobo to store my Web site stuff.
I don't keep any data from my "day job" on my personal systems. My system at work is backed up very well indeed.
BTW: I use git for my personal source control, and Perforce for my day job source control.
"For every complex problem there is an answer that is clear, simple, and wrong."
-H. L. Mencken
Your "problem" is not well defined. A problem is the difference between the way things are and the way you ant them to be. In your case, you have a large amount of digitally stored data and you are afraid that it might be lost. What do you need to do so you don't have that fear? (This is not a "management" problem, but I will get to that later.)
The only practical solution is to keep duplicates somewhere "safe", with safe being defined as someplace where you think you will be able to recover it in an emergency. Maybe the emergency is just a local hard drive failure, but it doesn't hurt to think of the effects of floods or Tunami. RAID is good, but I have customers who spent a lot of money trying to recover data stored on RAID arrays because they didn't have another backup. No matter what, you are going to lose some data in an emergency. Figure out how much you can afford to lose. This establishes the timing of your backups. 1 day? Ok, differentially back up your data 3 times per day on different backup media and you will not likely have to ever recreate more than a day's worth of data. Then arrange for the media to be copied or stored offsite so local disasters don't affect it.
I predict that loss of personal data is going to be a big problem a few years from now. Even today, how many people who taped their kid's birth in 8mm film can actually find a way to view it now? What about those files, ideas and manuscripts that you saved on that Z80 running CP/M and had the MFM drive? Oh, and those old CD's that are now rotted away? Management means deciding how long you want this info to be available. Do you want your great-grandkids to see how it was to live in the USA before Communism replaced our Constitution? Better plan for it.
Management might also include retrieval. I know at least one "backup solution" that issued a version of software that couldn't restore the data. I have over 3000 books. I don't need them every day, but I do occasionally like to go back and research or review some of the stuff I've read. Digital storage should allow us to have a tremendous amount of data at our fingertips if we can only put our fingers on the right set of facts... Semantic search is not quite good enough but it's improving. In the meantime, you might do worse than to use the data-cache model for retrieving your knowledge base.
You can't get around the budget problem. You can purchase reliable solutions or you increase your risk.
Family scrapbook solution: We created a scrapbook and made 6 copies which we sent to all the siblings in my generation of kids. If my brother's house burns down we can re-create his scrapbook for him. Use the same idea for digital safety and you might have a high probability of recovering it.
"The mind works quicker than you think!"
If you're going to do RAID, seriously consider RAID6 over RAID5. Yes, the extra disk costs money and a port, but the bathtub failure probability curve suggests that after infant mortality and during the 20-30 hour RAID5 rebuild after a 1-drive failure, you have a significantly non-zero probability of a second drive failure (especially considering you'd be running it at 100% load for those 20-30 hours). My other solution is "rm -rf /*".
I do consider tape the best answer to backups. No, it isn't flashy like cloud storage, or the latest Internet rendition of it (be it a glorified file share, rsync, etc.)
Modern tape drives are a lot more reliable than the old 8mms and QIC cartridges. I still have DLT tapes from '98 which are still readable.
It all depends on the size of data. For the amount the OP has, if there is money, I'd consider an external LTO tape drive. They are around $1400 on NewEgg for a LTO-3 external, plus one will need a couple C notes for a SAS card. However, LTO-3 tapes are $15-$20, so $200 would back up 3-4 TB of compressed data, and with uncompressed, definitely more. Once the drive is bought, having the ability to save off the data completely for $200 or so is pretty economical. To boot, once the read/write tap gets flicked to read-only, nothing software-wise is going to be able to corrupt the tapes, and you can always pay for LTO-3 WORM tapes if you want true tamper-resistance.
Tape will always have a place in IT, even if it is just for a place to save data cheaply long term for pleasing the auditors.
The same thing happens to people who can't throw away a candy wrapper.
The physical problem has already been solved. Buy x number of x TB hard drives and put them in some type of configuration where they are all accessible at once. The problem with (desktop) hard drives is that statistically for every xTB you read an unrecoverable read error will appear. I manage over 100TB right now at my job using enterprise grade systems and almost monthly we have the system reporting a URE. The solution for that is checksumming. ZFS works great as does BtrFS and it keeps your data intact or at least reports what is bad about it.
"File system errors" are simply unacceptable, you either use a very shitty file system or as said, there is a bad block somewhere on the hard drive causing you issues.
Custom electronics and digital signage for your business: www.evcircuits.com
Keep all your important files in a version control system. Personally, I use Perforce (it's free for 2 users or less). That gives you: multi-revision history and checkin comments, an easy way to pull a subset of files to any computer in your house, and peace of mind that you don't need to worry about kids deleting anything important as it's all stored on the server with history. Also easy to see what has changed on any computer and check those files in. And there's a big win for data integrity checks: Perforce stores the checksum of all files (and revisions) and can easily check that every file still matches the checksum in the central database. If you have any disk corruption, you'll know about it when you run 'p4 verify -q //...'. You can store files of several gigabytes each with no problem.
On top of this, I use rsync to copy the server data onto backup drives. I'm also looking at storing backups online, but haven't taken that step yet.
I've been using this system for years and I couldn't imagine being without it. It's so easy to find and retrieve exactly what I want - my resume 5 revisions ago, my tax return, photos from 2003. Even without that, the data integrity checks give a lot of peace of mind.
I keep important stuff (spreadsheets, letters, vector graphics, coding projects) in a subversion repo on my "home-server" which I keep synced with my laptop and desktop. That way, I always have backups of this stuff and I can update/commit from remote locations.
Now for the stuff I don't want in the svn because it's just too big and I don't really need a history (photos, videos), that just resides on the server, cifs-mount.
On some sundays (maybe about 4 times a year) I hook up a external sata drive, put an encrypted filesystem on there and do manual backups (not always the same stuff and not all of it, because I dont want to afford that many big drives). That drive then gets transported to alternating familiy members places for safekeeping.
SuperMicro X7SLA-H board (E150 when I bought it more than a year ago)
A couple of WD Greens(E100 each at the moment)
A low power silent PSU (E100) (WARNING: this one has no ground and therefore no decent surge protection. Always combine with an external surge protector)
Some RAM that fits (E50) (FreeNas advises 1 GB/ TB of harddisk, but will function perfectly under low loads with much much less)
A case (free if you have one idling in the attic)
FreeNAS (free).
Total: E600 for 3x 1,5 TB (3Tb under raid 5), expandable quite a bit with PCI-E Sata cards (E50 for 4 devices. Raid controllers are overrated for home use. Soft raid gives les trouble if the hardware dies)
Well, I might have a way, but it only works on a semi spherical planet in a vacuum.
I'm currently building my own multi-solution. Beige box solution is the way to go.
My game rig/media server/work station is a fast AMD Phenom with 1 150GB 10K rpm OS disk, and then four 2TB disks in RAID 6 as "storage".
Eventually, I'll separate the boxes into one Amusement machine and another for media player/storage. I have CAT6A cables in the walls here.
The storage disks are backed up over the network to my 4 disk Netgear NAS with "RAID X" (~6). I'm thinking 3 times a week.
PC and NAS are on UPS, so I hope to get a 2nd NAS placed at my brother's house (in turn for some storage space at my place) for redundancy with monthly backups.
Defining Statistics and Social Research
Don't use RAID 5.
RAID5 was a great solution when disks were very small and very expensive. Now they are relatively huge and cheap, so RAID5 is almost never a good idea.
I'd keep the music. One of my favorite hobbies is running music through Audacity to fiddle with the tempo and/or pitch. There's a ton of songs that I only sorta like as is, but they become favorites when I blast off a couple of custom adjustments.
But give or take a couple of years, we were seeing a potential explosive breakthrough in storage tech, it may not even be worth nit picking files if you think you'll want them later.
My first Journal Entry ever, in 8 years! http://slashdot.org/journal/365947/aphelion-scifi-fantasy-horror-poetry-webzine
Yes, there is an advantage of hard disks, because all it takes is just the cost of a drive. However, if there is valuable data on the drive and it conks, it is far harder to recover than a tape. A tape, you can hand to a recovery company (Kroll for example) who has equipment to get something off, be it reading raw patterns, decompressing them, and getting something back. HDDs, it isn't just the media, it is the drive heads, the platters, the magnetic media, the controller, and numerous other variables, which complicate recovery immensely.
This is purely anecdotal, but in the years of working in IT, I have encountered one hard error from a LTO tape drive among thousands of tapes. I have plenty of dead hard disks, so from experience, I have a far higher chance of recovering from a tape than from a HDD sitting on the shelf. If I use a sane backup process where there are multiple copies (even a variant of grandfather/father/son), the chance of me permanently losing data is low.
Now, other media such as 8mm and 4mm have been different, where the format wasn't designed for backups, as opposed to an audio/video format for consumers.
Plus, it is easier to manage 50 tapes than 50 hard disks. Buy a couple sheets of labels for the tapes, and that job is done. Yes, one can use HDDs, but that isn't there intended purpose.
As for long term storage, hard disks are not designed for this task. There is a reason why the warranty on all but enterprise HDDs tends to be a year, while LTO tapes have a lifetime warranty. Even though this doesn't mean one can get data back if it fails, it shows how much faith the maker of the media has in their products.
If the submitter doesn't want to just trash his disks, it might work to reorganize them. For me at least office files tend to be small - so make one of those small drives office/text data only. Then he can get a big new drive to churn all the music&videos on. If he still has more small drives to use, the third could be backup copies of software installers.
My first Journal Entry ever, in 8 years! http://slashdot.org/journal/365947/aphelion-scifi-fantasy-horror-poetry-webzine
Raid5 still gives n-1 storage for N drives, even with the larger drives available, it is still a pretty efficient way to get fault tolerant storage. Mirroring (Raid 1) is also cheap given the size of the drives available vs their cost, but is about the least efficient storage, n/2 storage for N platters. If you are going to be backing up a high volume, fault tolerance makes sense, as does a soft de-dupe if you can set it up using something like Linux on an older PC chassis.
I think the problem you need to solve first is how to keep your personal data at a reasonable size. I can't see what TBs could be. I'm sure a large portion of it not that important and might get lost without affecting your life.
The way I do it is try to keep my *most important* data at a reasonable size (a few 10 GBs), compress all that can be compressed, and put the absolutely most important part of it on Dropbox.
I use to have your problem. Then I went out and bought a Synology 211 NAS. I love it. It has a fantastic interface (DSM 4.0). It is set up for Raid 1 at the moment with mirroring. I have had integrity issues with my drives where I had to rebuild one of the drives. Other than that, I love the peace of mind I now have. Plus it is so compact... check it out.
Huh? [devShell.org]
1) git-annex (if on Unix) /dev/foo #this will erase all data on disk. If it reports even one single bad block, toss the disk out.
2) badblocks -swo foo.bb
I am a big fan of zfs for its data integrity features. What I did is below
Home Server (you can configure a minimal hardware file server with components to fit your needs, this is just what I used) Quad core Athlon X4
16gb memory
LSI HBA SAS
9x2tb hdd
The hard drives are configured using raid-z2 with one spare. I am using zfs on linux now, but freebsd works great, and their forums are very helpful for people new to bsd. This server is overpowered for my uses, but it also is a media server, web server, samba server, etc. ZFS has snapshots as well, which work great.
Off site (at a friends house)
Core2 Duo
8GB memory
onboard sata
4x2tb hdd using zfs / raidz2
This system is a low power system sitting in his closet using his internet connection. I only sync the stuff it would be difficult for me to replace. The initial filesystem was created at my house, directly off my server to RSYNC only has to send deltas.
I looked at online backup solutions, but my upload speed is so slow (512K up) that it would take forever to get the initial upload into the vapor. Further, the low end, low cost providers I would look at are more likely to fold, leaving me without a backup.
I've had a good experience with MooseFS (http://www.moosefs.org/) using my macbook pro and a couple of old BSD boxes. There are other distributed filesystems that will give you peace of mind in terms of storage resiliency - http://en.wikipedia.org/wiki/List_of_file_systems#Distributed_file_systems
I have a Norco rack mount server with an Areca RAID card. I currently have 6 2TB drives in RAID 6. They are Hitachi "green" desktop drives. The controller died a few months ago, and I RMAd it. Once I got it back, all was well. I didn't lose ANY data. I think people saying RAID card failures and drive failures is way overrated. From what I see, people are saying parity is unsafe. I just don't get it. My future plans are a Supermicro case, 2 Xeons, tons of RAM, a bunch of LSI HBAs, 24 drives (haven't decided on the size yet, the shortage is being played well by the corporations), and two a pool of 2 striped raidz2 arrays (like RAID 60). I'm not much of a fan of backups. I am a home user and my data is not worth the cost of a second server with another 24 drives. A tolerance of 4 drive failures will be plenty peace of mind for me. Cloud storage is BS unless you have some unlimited gigabit fiber ISP. Even then, certain data can become contraband and be deleted.
Buy 2 servers, preferably used storage array servers. Start a raid 5 or 6 array on one. (This server is the storage server.) This is your main storage drive. Store ALL data on it. It's helpful to have it support multiple access methods (SMB, NFS, iSCSI, etc). You could go full OS like Debian or something like OpenNas or OpenFiler (BSD based).
On the second box, add as much storage as is accesable on the first. This is the backup server. Run a cron job to regularly r-sync the data off the Storage server over to the backup server.
In this configuration, you have some redundancy in the RAID and a true backup in the second server. You also have the ability (hopefully) to drop in drives as you need so you can expand as you go. And if the hardware it's self breaks, you can simply replace it and keep going.
I do security
I use a variety of uses to backup my files. First off, it's only about 140GB compressed but I limit it to Bly "important" files and even that could be trimmed down.
My setup:
Linux box with 1 TB, encrypted HD for my data. 1 TB external HD, encrypted for weekly backups using rsync. Dropbox in encrypted container for a small selected amount of not overly personal data. I'll never drop personal data in Dropbox because access to my encryption key is available by the company. Small backup goes to Wuala. Slightly more personal data than Dropbox but nowhere near the bulk. I also have an issue of upload speed (650 Kb) and 60G cap. Every 6 months I backup (update) photos to DVDs and but in lock box and also keep set at work locked away. Finally, every year I travel to the parents place for vacation and make copy of external HD to keep at there place.
It's a bit of work to keep on top of but it becomes routine.
How many ppl actually restore these backups? Too many times ive seen backups go bad because they were never checked. I randomly do a full restore 2-3/year.
Okay, I'm sold. :)
What would be a better alternative be then? At home, I've got a Linux RAID-5 configuration running on my file/media server/MythBox. I built it several years ago with four 500 GB drives. Its performance has always been more than enough to handle standard-def MPEG2 recording from the TV encoder. Soon, though, I will need to upgrade it with larger drives and that would be a good time to switch from RAID-5.
At work, I'm about to build a new more powerful workstation/server. To say that my budget is constrained is putting it lightly. Currently, it has two 1 TB drives in a RAID-1 config. We had an external backup/snapshot drive, but it has since died. Our write speed requirements are not at all extreme, so current drive speeds should be fine.
I guess RAID-6 seems like a good alternative in these cases, supposing the array is built with a minimum of four 1 TB drives? I don't think I need the performance of RAID-10. At least, at home, I'd rather have the extra space. I'm also not considering SSD drives in these two machines as I can't justify the cost.
So much to consider...
Elrond, Duke of URL
"This is the most fun I've had without being drenched in the blood of my enemies!"-Sam&Max
I've been using RAID10 on my squeezeserver/mythTV box, personally, for about five years now. But I got a case of server-grade hard drives for free (nearly all my equipment is dumpster-diving booty) and honestly I built around what I had on hand. Outrageous performance but it's a noisy, power-hungry machine compared to most home servers, even with temperature controlled fans and so forth.
I'm about due for a home server rebuild too, and I was also thinking about RAID6 this time around. Unless I find another case of hard drives somebody's throwing out, I guess.
If you think backup is slow, wait until you try a restore or RAID rebuild.
Copy/sync to a duplicate file system on a replacement drive in an external enclosure.
When the internal drive fails, swap in the pre-loaded replacement, and refill the external enclosure.
This also allows for growth.
--
The Truth of Large Numbers - almost all numbers are larger than you can imagine.
Publish your data to book. As a process it will force you to really consider what is valuable. Your photos and coorespondanes make the cut. It will push you to put more effort into your writing diaries or lettters. Your really sensitive information can be locked in a safe. Most of that stuff is is useless. If you had two minutes to evacuate you should be able to get what really matters. And I don't think it will be your tax returns!
I could have used many other Christian or muslim sects as example. point is the NWO was set up centuries ago, too late to worry, pal. The "technology" for that can be done by hand or by computer.