Slashdot Mirror
Ask Slashdot: Best On-Site Backup Plan?
An anonymous reader writes "I know most people use backup services in the cloud now, off-site, but does anyone have good ideas on how to best protect data without it leaving the site? I'm a photographer and, I shoot 32GB to 64GB in a couple of hours. I've accumulated about 8TB of images over the past decade and just can't imagine paying to host them somewhere off-site. I don't make enough money as it is. Currently I just redundantly back them up to hard drives in different rooms of my house, but that's a total crapshoot — if there's a fire, I'd be out of luck. Does anyone keep a hard disk or NAS inside a fireproof safe? In a bunker in the cellar? In the detached garage? It's so much data that even doing routine backups bogs the system down for days. I'd love suggestions, especially from gamers or videographers who have TBs of data they need to back up, on what options there are with a limited budget to maximize protection."
There are offsite options besides the cloud. I shuffle hard drives between work and home. If you work from home, you could do the same at a friend's house or something.
http://iosafe.com
Check out my sci-fi/humor trilogy at PatriotsBooks.
Find a friend with same issue and figure out a plan to place two cheap NAS servers one at each house and back up to the NAS at your friends house encrypt if you wish and problem solved.
Works like a charm
megaupload
in the short run and in the long run. Also, storing locally does nothing to protect you from flood, fire, theft, etc... Backblaze is $5/mo, unlimited storage. I'm sure there are others with similar/better deals. What's a NAS inside a fireproof safe going to cost?
"I'd rather be a lightning rod than a seismometer." -Ken Kesey
if you earn revenue from it, pay for backups
if it has sentimental value then think about paying for backups
hard drives go bad all the time so if you're going to back up to hard disk and its important buy a few external ones and keep them in different locations
I'm a professional photographer and I keep backups of my network drives in a safe deposit box at my bank. I have a couple terabyte drives and then once a month I update the drives at the bank. This may not be very sophisticated but it's simple and it works for me.
Just a note about fireproof safes. They are fireproof in the sense that they will protect paper from getting hot enough to catch fire and burn. However, they will still get hot enough in a fire to destroy any electronics or plastic items stored inside.
First, going strictly by your requirements, I would suggest either a fireproof safe or fireproof drive enclosure. I don't have experience with the enclosures, but the safe itself should be able to handle your normal everyday fire and protect your data.
However, I'd suggest that you don't store your safe at your location at all. Surely you have a friend or someone you know that would let you borrow a few square feet of their basement for the safe. This would create a physcial barrier that would enhance your securiy if not always convenient. I'd also recommend a second copy somewhere else if this data is that important to you.
Remember that as with (almost) anything else, there is a cost-benefit tradeoff. I'm not convinced that a "cloud" based solution is your best bet anyway. But a simple, low tech solution seems to be what you need anyhow.
See my journal for slashdot ID's by year. Mine created in 2005. http://slashdot.org/journal/289875/slashdot-ids-by-year
Backing up to portable hard drives is fine. Better if you have at least two copies of the data, on different drives of course
Keeping your backups on site is great if you are concerned about user error, but offers nothing for disaster recovery. Put them in a safe deposit box. Store them at a friend's house. Put them in a Zip-lock and bury them in the woods. Do something, but get them out of the same building where your primary storage is if you really want to have a useful backup for disaster recovery.
This post comes with a double-your-money-back guarantee!
Any offense taken to this post is at your sole discretion.
Have you thought of getting a safety deposit box at a bank? Usually they're in a fire resistant box inside a fire resistant room.
Store backup copies of disks in there, and swap them out, similar to tape backup strategies.
Crashplan is wickedly cheap, and for Unlimited Storage its worth the buy
You could put some backup drives in a safe deposit box. With as much as you're storing, it may be beneficial to store just the bare drives.
A friend and I just keep external HD's for each other. Every now and again we go to the other's house and swap them. They are small... and fit easily on a shelf in an unused closet....
If you don't trust your friend enough you can even encrypt the HD....
Cheap, effective, works.
Any pro photographer will tell you that 95% of what you shoot is crap. Prune it mercilessly.
Give me Classic Slashdot or give me death!
Get a couple external 3TB drives. They are under $150 each and keep them in a safe deposit box. I store 2 at my local bank and switch them out from time to time. Depending on your bank and balance you can get them for free.
Safe deposit boxes aren't expensive and they're not a bad offsite location to store copies of you data on external hard drives. I don't really like using hard disks for long-term archiving but it's one of the lowest cost practical solutions. A tape drive would be something to look into but they're not cheap.
Why not get a firesafe?
Some of them are rated for higher temperatures than house fires usually attain, and the response time of your fire department should give you an idea of how long they need to hold out for.
If you get one that has a decent lock you can keep your gun and your pot in there without the kids playing with them.
Do not meddle in the affairs of geeks for they are subtle and quick to anger
I have two USB hard drives that I use for backing up. They have the same data on them. I keep one at home and the other in a safety deposit box at the bank. I backup about every week or two, but once a month I will do a backup, then take that drive to the bank and swap it with the other, bring it home and back up to that drive.
I don't know if you'll ever find online solutions that will store 8TB of data for free/cheap, so I suggest just loading up on the harddrives and putting them in a lockbox. This could be at a friends house, work, a hidden panel in your car, the bank, etc. There are tons of places you could put it, but if you find a cheap online solution that'd obviously be excellent.
I have redundant drives at home, but what about a fire, flood, or theft? Two options:
(1) Put the data on an encrypted hard drive and bring it to work. This is what I do. It is safe in my desk and even if someone at work broke into my desk, they wouldn't get past the TrueCrypt.
(2) Same as number 1 but use a SSD and put it in the trunk of your car if you don't work in an office. (SSD is less sensitive to vibration)
-d
"Here Lies Philip J. Fry, named for his uncle, to carry on his spirit"
If the workplace burns-down, I have the home copy. And if the home burns-down, I have the work copy. The most-important files (resume, government clearance) and small-sized text files (ebooks) I have a triple-backup through Google Drive. 5 gigabytes free of charge.
My AC stalker: " I personally agree with your posts most of the time, but that won't keep me from modding you troll"
This looks like it might work for you and I've been investigating it lately. I can't recommend it, given I haven't used it, but it's an option if you have the bandwidth to upload 8TB of files.
http://www.backblaze.com/
Backup your data and rotate drives off-site to a friend or family members house. Storage is cheap, and eSATA works like a goddamn champion.
Due to large changes in amount of data, delta type backups won't be nearly as effective, so just use something like ViceVersa or Robocopy to duplicate your new data to one of your off-site drives, and then store it off-site for free. No need for complex RAID or costly hardware because your goal is to prevent against loss of data due to catastrophic failure or disaster at your residence.
eSATA or USB3.0 (though I haven't tried USB3 yet so I can't say).
http://www.crashplan.com/
Unlimited backup for $5/mo to the cloud. FREE backup to other computers using their software which is cross platform on Windows, Mac, and Linux. I'd purchase an external HD(s), backup to it then get a friend to put it at their house. You can adopt the backup on their computer and then backup to their computer (FREE) and to your external HD(s) with their software automatically from your own computer.
Or you can just sync it to the cloud, but 8TB might take a while to get everything up there.
FunOne
Why bother with other storage technologies that degrade overtime. You already have access to cheap (but slow... in this application that doesn't really matter) solid state storage that doesn't degrade. If your objective is to simply archive the content, buy as many SDXC cards as you need to archive your content, and when you go on shoots charge them for the cost of an extra sdxc card to archive the content on and throw it into your safe.
Go to your bank, and for around $100 a year you can keep your drives in their vault. You should be able to fit 4 2GB external drives in the smallest-size box (but bring them with you to make sure).
In addition to your "hard drives in different rooms" strategy, consider keeping a copy offsite in a bank safe deposit box.
Lorem ipsum dolor sit amet, consectetuer adipiscing elit.
Repeat after me:
Raid is not a backup
And RAID 0 is never used for reliability as it has no redundancy - the more disks, the higher chance of failure. You must have meant RAID 1.
You're a temporary arrangement of matter sliding towards oblivion in a cold, uncaring universe
"does anyone have good ideas on how to best protect data without it leaving the site?"
If it's not leaving the site, you aren't protecting it. At the very least, get a few portable drives and rotate backups to a relative's house.
Buy enough hard drives to make two backups of your data. Keep one copy of your data onsite, ready for a quick restore if necessary . Keep the other copy offsite, in case there is a fire, flood or burglary at your main location(i.e., you really DO want one copy of your data to leave your site). You don't need to pay for a fancy offsite backup service -- just keep them at a friend's house and be sure to rotate your back drives regularly. This is what I do with my 3TB of data and it works very well. Use backup software that automatically copies only the changed data. Otherwise, you will not be able to do a full backup of 8TB every day. On Windows, you can do this cheaply with Robocopy. On Linux or Mac, rsync does the same thing (or use Time Machine on a Mac). You can schedule the backup script to run automatically every night.
If you need to keep all 8TBs spinning all the time, you'll probably want to look at getting hardware RAID devices. If you only need to keep the most recent data instantly available, RAID might be unnecessary.
Drop a box at a friend or relatives home, preferably one with a big pipe. Setup two storage systems, I would look at ZFS on FreeBSD (FreeNAS perhaps). And then setup async replication rules. You write to one, it will eventually catch the other one up, constantly. If yours fails, go get the other one, put them on a local highspeed network and clone again. You can increase reliability with more georedundancy. More boxes, more places. Since you are not replicating the entire dataset daily, only the change state, it wont bog things down badly.
Any company that I've ever worked for that had money to spend did tape backups and stored them in a vault offsite. Tapes get verified as they're written, and don't have parts that fail like hard drives do. They have a 30-year shelf life, and you'll always be able to find a way to read them in the future. Go to ebay, buy a used LTO3 or LTO4 drive, (400GB and 800GB uncompressed, respectively). Tapes are about $25/ea for LTO3. Then put a backup somewhere safe.
I see this all the time with photographers. Bottom line: your photographs are not all that valuable. Some are, yes. Most are not. Pare them down. Delete the bad ones, the failures, the misfocussed, the bad exposures. The greatest photographers the world has ever known are only known for a few dozen photos at best. Do you really need an 8 TB photographic archive? Who's going to ever look at them all? Save the best. Delete the rest.
Not to sound like a shill (I'm a fanboy, which, while different, will sound somewhat similar in practice), Crashplan has a free option available where you and a friend can both run it and can use it to back up to each other. If you have a photographer friend (ideally in a place far enough away that you won't be hit by the same natural disaster), this can be a pretty good option. It'll likely take awhile to do the backups, however, and you'll also need to have adequate hard drives on hand to store not only your own work, but also your friend's, which may get in the way of going cheap.
That said, for $140 (the price of a hard drive or two) you can get a 4-year subscription for their cloud hosting with an unlimited backup size. The company I work at uses their business-level product, and I recently started using Crashplan+ at home for my own computers. While it does take awhile to back up, it's painless to do so. At least so far, I prefer it quite a bit over Carbonite, which is what I was previously using at home.
Give your mom a box of backups and ask her to hold on to it, it is "stuff you made"
She'll never get rid of it.
and if the house catches fire, it will be the first thing she grabs when she runs out.
I like microcars
Set up a RAID 6 array at a friends or relatives house. Do an initial dump of all of your data to it before you bring it over to them. Offer to pay their Internet bill in exchange. Set up a VPN and run rsync between your place and theirs.
That has got to be about the cheapest and simplest off site back up you can possible have. You can even write off the cost of their Internet as a necessary business expense if you can get a receipt (since you are a photographer for a living and not a hobby).
I suspect that what Hatta said is completely true. I doubt that 100% of what you have is really worth keeping forever. I take photos when traveling and some are mistakes or just didn't turn out that well. One of things I like is that if I take enough photos, some will turn out to be really good. Note that I said "some" not "all".
If you have a friend who doesn't mind you could make a backup to the biggest hard drives you can afford and have your friend store them. I'm an IT guy and on a previous job I actually was authorized by my company to keep a set of backup tapes for our development servers at my house so we didn't have to pay an offsite storage company to store them.
I have a question - are you really going to argue that your photos are so valuable that you can't let them go into the cloud for fear that others might get copies? I'm in no position to judge whether that is true or not and even if you say it is, the rest of us may disagree. I know that some professional photographers have some really warped ideas about their own work and think that the wedding video they just got hired to shoot belongs to them and the paying customer is some kind of leech they'll deign to give one copy to under duress. If you got oodles of old wedding photos, for example, that you were hired 10 years ago to take, I doubt that it's really critical that you keep copies. I participate on a video forum and we see people all the time who want to watermark stupid crap like videos of their kids playing tee ball* because they are so delusional that they think the whole world wants to see their kid. I get that these photos may be just incredibly important to you, but do you really think that others are just waiting to steal them if they go on the cloud? Cloud backup would be your most cost effective solution.
* For the non-North Americans, "tee ball" is version of baseball that small children play in organized leagues. They hit the ball off a tee, hence the name.
I have to throw in my non-anonymous endorsement of Crashplan. You can pay their reasonable unlimited rate or work out an agreement with a friend, family member, or other person in a similar situation to backup to their instance while they backup to yours. At the same time, you can have it direct a backup to an attached USB drive or NAS... everything except their cloud storage is free. And cloud storage is a whopping $5/month for unlimited.
You have a ton of existing data, and that would take a lot of time to upload - but fortunately Crashplan allows you to seed your backup by sending in a hard drive.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
Not your next door neighbor (particularly if you are in an apartment). You need a friend who is far enough away that the same disaster that hits your home will probably not hit your friend's home.
You can ignore this advice in cases of asteroid impact, zombie apocalypse, super volcano or robot uprising.
If you're going to put things in a fire rated container, there are a few things to consider. Those containers are not "fire proof" by any means. Get one whose rating is reasonably high as they will buy you some time.
Most house fires are either a basic 'room and contents' or a much more involved fire where whole floors are exposed (and largely consumed) by flame.
When you put your fire rated container somewhere, consider that fire burns upwards, and the thermal difference from floor to ceiling is around 400 degrees F on average. Before you put the container in the basement corner, remember that firefighters use water to put out fires. Lots of water. 150-200GPM per handline and 1000-2000GPM for the big pipes on the ladder trucks. Most of the damage in a house fire is from water. You'll get us much as 6-12 inches of flooding per floor (until the firefighters cut holes in the floor to drain it so the floors don't collapse.
Also should the roof or ceiling collapse, the best places to have things are near the corners of the load bearing walls.
This is my long way of saying store your fire rated container on a solid hardwood (not particle board) or metal shelf, about knee height on a low floor near the corner by load bearing walls. This way in the event the whole house is a write off, you still have a reasonable chance of saving some of your data and personal effects.
"Omnis tuus capsa sunt inesse nos"
I use a backup service called Crashplan. They have clients for Linux/Mac/Windows and support backups either to your local network (free), "friends" machines in a p2p type configuration (free) or to their servers (paid). Everything's encrypted locally and the client app is pretty decent IMHO. Best of all is that the paid plans are pretty reasonable - I have the unlimited plan for something like $100 a year, and it really is unlimited (well, they claim it is and I have no reason to doubt them). I currently have about 3TB up there so I don't see why you'd have an issue.
The way I have it configured is that all the machines on my network backup to both my local fileserver and to their cloud. The local backup has a higher priority so any changes get pushed over the lan immediately and then batched up and sent offsite over the slower link. Speedwise I can't saturate my uplink when uploading to them but I get a pretty consistent 1-2MB/s, so figure maybe 100GB a day? I think my initial seed took a couple of weeks. I've done a couple of small test restores and download speeds were similar (although in all but complete disasters I'd be restoring from my local fileserver which is obviously far faster).
Disclaimer - Not related to the company in any way, just a very happy customer.
---- Den ene knappen er powerknapp, den andre er Bender voice knapp "Bite My Shiny Metal Ass"
I use MyCleanPC along with a custom hosts file to redirect my documents to my online backup service. I'm mostly backing up my GameMaker projects.
I use backblaze http://www.backblaze.com/ for off-site backups. $50/year for unlimited storage is more than reasonable. I currently have about 2.5TB backed up there.
`fortune -o`
Look for a used LTO3/LTO4 tape drive, then bulk-buy tapes.
Write each set of content to two tapes, ideally of different brands, and store in different places if you're really concerned.
I've been backing up to LTO3 tapes for ~3 years now, i've got 50+ tapes, mostly in my safety deposit box at the bank (cost $75/year)
LTO4 based on eBay prices right now would be an initial expenditure of ~$1k for the drive, and $25-30 per 800GB of storage.
The cloud options aren't really feasible for me, as the upload time & bandwidth cost is horrendous.
ICQ# : 30269588
"I used to be an idealist, but I got mugged by reality."
But we're talking terabytes here: most cloud storage providers will not cater for such users (or charge exorbitant fees). Moreover, he's not interested in cloud backups. There are many, many good reasons to resist dumping your stuff onto some cloud service.
Nothing sucks like a Vax, nothing blows like a PowerMac G4
My RAID goes to 11, man....
Never answer an anonymous letter. - Yogi Berra
What's a good way of ensuring data integrity (and possibly repairing any corruption) that might happen? Is two copies and a checksum enough to be able to reasonably repair a (not too) corrupt file?
Nothing sucks like a Vax, nothing blows like a PowerMac G4
Gluster can't be told where to store something. So even in a gigantic cluster you still run into the problem that 2 duplicates of a file can be located on two storage nodes that are physically near to eachother. Let's say right where your fire begins ..
Since the vast majority if your files aren't changing frequently, use daily an incremental backup. There's options out there which will let you run it at the end of the working day and turn off your machine once its done.
Can you throw some cabling into the garage? Else WiFi would probably do. Stick a NAS in there, configure the software, make a once-monthly entry into your calendar to check the backups are viable and you can forget about it. Well, until winter maybe.
Run a second backup say over the weekend and swap drives at a family/friend house. Safety deposit box if necessary.
After each shoot, be realistic about what photographs you keep. I'd wager a smallish fraction of what you're keeping now. Remember there is some overhead cost in retaining those files, nevermind in hard drives but in your time sorting through it.
Be selective. Back the good ones up - to a USB key.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
I started this long, convincing post about why cloud backups are so much better in terms of durability and availability. Then I looked at the cost.
Holy smoke! According to Amazon's handy cost calculator, your 8 TB of data would cost $915.86 per month to store in Amazon's cloud. I would argue that kind of cost may be acceptable to back up your entire livelihood, but that really depends on your cash flow, doesn't it?
My new recommendation is to burn your pictures to DVDs or blue-ray discs and bury them in your back yard. :-) Oh wait, I mean keep the discs in a large-size safe deposit box at your neighborhood bank.
[Sir Garlon] is the marvellest knight that is now living, for he destroyeth many good knights, for he goeth invisible.
As a photographer myself (Though only around 4TB of photos at current) here's my setup:
Onsite backup: Drobo to Drobo.
Offsite backup: Backblaze. I pay $4/mo (2 years prepaid). This is a secondary backup. It still has everything, but I rely on the local backup to retrieve something should my primary storage fail, and the offsite is for when things burn to the ground or someone steals my stuff or lightning takes everything out. Really it's cheap, it's just that initial backup which takes an eternity and might get you in trouble with your ISP. Fortunately I was able to use the local university 1gbit connection to reduce my initial backup time to just 18 days (straight).
Zing!
I am betting that you can recover that from your friends.
A lot of people are telling you that you're wrong, that you need "offsite backup", etc, etc. But that's not what you asked. You've got an external Hard drive already it sounds like. Most banks will rent a safe deposit box for fairly cheap. This will give you that "Fireproof" that you need. You're either going to need a computer with USB 3.0 or external hard drive(s) with ethernet capability. My suggestion would be something like a DROBO, spend the extra money to get two sets of hard drives (say a grand total of 8 3TB hard drives for the Drobo). Then do the following:
1. Mark 4 of the hard drives as "Week 1 (Drive 1,2,3,4)" or "Week A (Drive 1, 2, 3, 4)"
2. Mark the other 4 as "Week 2 (Drive 1, 2, 3, 4)" or "Week B (Drive 1, 2, 3, 4)"
3. make sure your computer has a gigabit network card in it (The drobo has a gigabit card) and get a consumer grade gigabit switch/hub.
4. Whatever week you get all this set up is Week 1, on that Thursday or Friday start a backup of the data to the Drobo, there is a Command Line command that will compare the current folder to the drobo and only copy changes (it's been a while since I've used it, I want to say robocopy, but do some googling to find it and all the variables)
5. That Saturday (assuming the backup is done) drop all 4 disks off at the bank in the safe deposit box, load the other 4 drives into the Drobo.
6. The following Thursday or Friday run the copy command again. That Saturday go pick up the drives from the bank and drop off the other drives.
7. Repeat weekly.
Depending on how often you add more files to the system and how secure you want to be/how much of a hassle it is to get to the bank, this could become an every 2 week or every month thing. Conversely it could become a every 3 day thing. It depends on how much data you are willing to lose in the event of a total catastrophic loss (tornado, fire, earthquake, etc) vs. the cost of going to the bank, cost of hard drives, etc. The initial costs will likely be heavy, and there will be a yearly charge from the bank for the safe deposit box, as well as the cost of gas for a trip to the bank every week and other costs I might not have figured into the equation, so you really do need to do a full cost/benefit analysis and decide if this, a cheap colo + Drobo, or renting space on an existing DR company's server would be the best use of your time/money, but there is an honest answer of how to do a self-ran DR set up.
Hope this gives you some things to think about.
I'm a photographer too (albeit semi-pro) and don't have that much data, about 2TB at last count. I use CrashPlan, they offer both cloud ($6/m unlimited data), local and shared (backing up to a family/friends computer). You can seed you initial backup to the cloud with physical disks sent to CrashPlan.
Upload your photos. All of them. To flickr, facebook, whatever. The good ones will survive. The great ones will be shared. The ones you're ashamed of will go viral and you'll never get rid of them again, no matter how much you want them to.
Visit http://ringbreak.dnd.utwente.nl/~mrjb/growingbettersoftware to download your free copy of the book
There is no "safe" on-site storage option. My brother had an expensive expensive fire-proof gun safe at his house - it weighed over 500 lbs.... After his house literally burnt to the ground (rural area, no one home to alert the first department, by the time the neighbor down the road saw the fire and called the fire department, all they could do was put out the fire in the surrounding brush (and cars)), he couldn't even identify any remaining pieces of the safe or the guns that were in it in the debris.
If you want your data safe, store it off site, preferably in another part of the country so it's not subject to the same local disasters as your house. Mailing snapshots of your data to an out of state relative is probably best.
I'm also a photographer, don't make a lot of money on it, and I also backup to hard disk (because nothing else is big enough. Blu-ray? It is to laugh.)
My strong (really strong) recommendation: Keep at least one backup set offsite. This is a really really strong recommendation. As a photographer, your images are absolutely the most valuable commodity you own. If your equipment is stolen, burnt or dropped off a bridge, it would really suck, but you could replace it. You can't replace your photos.
This protects your original content from catastrophes up to geological level. If your house burns down or washes off a cliff, you'll lose your most recent photos, but the bulk of your work will still exist.
To protect against geo-scale catastrophies, keep a set at a remote relative's house, swapping in a newer backup each time you visit. I have relatives at 200 and 600 miles away, not really far enough for widespread catastrophes, but better than nothing. I bet you could think of some prospects.
In summary, I don't depend on fire proof safes or detached garages. For protection from natural disaster, nothing beats your data not being there when it happens.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
big ass-magnets
Fireboxes are designed to keep paper from bursting into flame. Sure, your hard drive is mostly metal, but that PCB board on the back could melt, warp, lose solder, etc.
I guess the part I don't understand is how any of the solutions involving buying another hard drive (or three) and securing them are somehow cheaper than a tool like www.crashplan.com that is $50 a year for unlimited storage. Sure, the upload of 8TB will take a while.. so what? a single T1 (1.5Mbs) line can upload a few TB a month.
Then its all offsite. Worst case, just download the crashplan sofware, and have a buddy install it too, and use the free person to person backup feature (only pay to store on their servers) to have an offsite backup. Have him bring his PC over for a night, to sync up the initial copy.. then you will just send changes over the internet.
What are we going to do tonight Brain?
You don't necessarily have to go "cloud" to go cloud.
The company I work for has rack space in several colo centers. Some customers just want space for their own servers, which we're happy to rent them. You could buy your own RAID storage server, and buddy up to a local IT company for a bit of rack space. Saves a lot of trouble over swapping & carrying drives around. You wouldn't need high-reliability stuff like redundant power supplies or blade processing -- it's just storage.
Heck, you could go super cheap, and get one of those rack-mounted lock boxes and just lock your drives in it, inside a locked cage, inside the data center. Climate controlled, fire safe storage. Bonus to that over the bank safe deposit is 24-hour access.
I know it's off-site, which isn't what you were looking for, but just throwing that out there. That's what I'd do... I'm not impressed with virtualized cloud stuff. And I work with it!
That I'm right, and you don't like it, doesn't mean I'm a troll.
I use backblaze -- $47.50/year for a two year term and unlimited storage. For the mathematically challenged among us that's $3.96/month. Skip a couple cups of coffee a month and sleep better in more ways than one. As a bonus they show you how to build one of their 135TB storage pods here http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/ .
Dropbox Minimum teams plan (individual plans max at 500GB):
Licenses Space Yearly price
Base package 5 1000 GB $795
The preceding comment is my own, and in no way construes an opinon of the Emperor of Mankind.
These questions pop up a couple times a year, and the ultimate answer for cost effectiveness and minimal complexity is always the same.
Buy storage in groups of two or three (I'll explain the different options shortly) drives at a time. So let's say your 8 TB can be covered by three 3TB USB HDDs. So you buy a set of three 3 TB HDDs and that's your master copy that you keep at home and work from. Now duplicate that exact same setup and that's your backup copy. This gives you protection against human error (accidental deletion, overwriting, etc), hardware failure and filesystem corruption. Now buy a third set of drives. This is your offsite backup which could live in a safe deposit box, a friends house, etc. This gives you protection against events such as fire, flood, power surge, theft, etc which cause both the primary and onsite backup sets to be lost. The onsite and offsite backups should never be at your house at the same time. Periodically (you can detemine the best schedule for your situation) you should take the onsite backup and move that offsite, and conversely bring the offsite backup onsite. The schedule at which you do this would determine your RPO (amount of data you would lose if your house burned down).
The initial sync will be painful, but afterward you can use just about any sync software (there are many freebies, as well as many commercial) to keep the two backup sets in sync with the primary storage. Just let it run overnight while you're not using the system.
If you get to a point where you need to add a 4th drive, buy three identical drives (one for each storage set). Periodically you'll want to refresh the drives to stay current with technology and minimize the risk of hardware failure.
As mentioned, this approach does have the drawback of a slow initial backup, but EVERY solution is going to have that and HDD to HDD backup is likely going to be the fastest and least suck method of doing it. This is also just about the cheapest solution, and doesn't rely on any special hardware or software.
At the densities of today's HDs bitrot is not a question of IF but WHEN.
Don't get a 4 drive drobo, they do NOT have scrubbing of data, but the 5 bay and up ones do. Without scrubbing once a bit flips your picture is ruined.
Whatever solution you go with, either the file system or the hardware needs to have data scrubbing built in.
Cwm, fjord-bank glyphs vext quiz
there are better options than that.
for starters, 100GB archival gold Bluray XL discs
digistor makes them and guarantees them for life.
If you buy a 25pk spindle, they include a free drive.
They're using their grammar skills there.
Crashplan also has a seeding service, where you put your data on a physical hard drive and mail it to them.
http://support.crashplan.com/doku.php/feature/seed_service
Additionally, you can do the same thing with a friend.
http://support.crashplan.com/doku.php/feature/multi_destination_backup
i.e. Take an external hard drive, plug it into your computer, run an initial backup.
Then give it to a friend. You can push incremental backups to them over the internet.
All of your files are fully encrypted, so they can't see them.
I haven't tried either of these features myself (I only use the online service), but I like having the option...
I am another user of CrashPlan and have about 3.2TB of data backed up with them. Just make sure you don't connect to their Atlanta data center or you will be in for extremely slow upload speeds. If your upload speed is slow there is a way to force it to connect to a different data center though it may take several tries. If CrashPlan is connected to 209.208.x.x then you're connected to their horrible Atlanta data center and I recommend using the GUID trick.
Int heir client, double click on the house on the upper right. Then enter the command "guid new". It will start the entire backup over. You can always change your guid back by typing "guide guidnumber" where guidnumber is a long hex code of another guid.
They also offer the option of backing up to another user for free with their client.
-Aaron
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
I hate the cloud, and the reason is that someone can get into it period!
If you have questions about this then i suggest you keep thinking about it for a few more times until it makes sense!
I'm here for the experience, not the Hyperbole.
Even if you don't want it in the cloud, CrashPlan has a good PC to PC software (Free) that's very friendly and allows version control. Either way, you are a fool if you don't have an offsite. Most thefts or things worthy of an insurance claim would destroy or make unavailable your backups. Find a family member or friend with an acceptable internet connection, trade them external hard drives and allow reciprocal backups over the wire. Reality though, maintaining multiple TB drives is more expensive per year than paying a cloud service.
Crashplan may be unlimited, but upload bandwidth sure as hell isn't. With a 300GB/month cap on most ISP's 8TB will take 27 months to back up. You can seed the offsite by mailing a HDD to them (or 4 in this case), but you still need to upload all the new stuff on a regular basis which could easily saturate your connection or make you hit your limit every other month.
Yeah, he'd need a business-grade Comcast connection or FIOS or some other uncapped connection. Regular Comcast won't cut it, depending on how many 64GB shoots he does in a month. We were kind of given a limited amount of information.
Though it's advantages are greatly diminished when run locally, Crashplan is still a useful local backup solution. One of the nice things it does is monitor the data integrity of the backup. Unison is another free solution which does this. Unison actually caught some corruption in my family photos collection, so I'll always sing it's praises :)
I've only recovered one system with Crashplan, and that was my mother-in-law's Windows laptop that was backed up onto my basement FreeBSD server.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
Crashplan also has a service to allow you to seed your backup with a hard drive(s) It costs additional money, but helps keep that initial 8TB from taking years.. (Seeded Backup and Restore to Your Door)
I use them and although I don't have 8TB with them I do have over 150 GB and have been able to restore what I needed. I also provide my local resources as a backup target to other family members using Crashplan.
PS, I'm glad my ISP doesn't have a 300 GB cap.
Another option might be to use Crashplan with a neighbor that is within line of site and setup a private wireless network (ubiquiti Nanostation or similar product) and run crashplan between each other's houses. They get to backup their data to you, and you get to backup your data to them (however you will probably need to pay for the system since you have the lion share of the data). You could look at NAS devices that can also work with Crashplan directly. I use UnRaid with the Crashplan plugin.
Timing:
With the 64 GB every few days data feed, a 10 Mbps upload Internet connection should be able to upload that data in about 15 hours.
The more expensive profession/enterprise option: LTO 5 tapes, however you are talking sever thousand dollars just for the drive, and you may need to spend more to get software to run the drive, but for capacity and portability they work well. Just make sure you test restoring from them once in a while.
Good luck.
Safe. Deposit. Box.
...just make a copy of the file in the same directory... anything else "on site" can't really be much better than this
:)
having said that, i was stupid enough to by a netgear rnd4000 nas with a usb drive plugged into the backup port on the front panel with backup schedules at 12am every day
the backup usb drive sits on top of the nas, and the nas sits beside my workstation
if i were smart i would get another usb backup drive and rotate them with one kept offsite at work or something, but nas covers hard disk failure, the usb backup covers acidental file deletion (losing only a day's work at most)... if there was a fire that would be bad, but so many problems in the world are only resolved after a major catastrophe forces the hand of those responsible, so who am i to buck the trend
Crashplan is likely to be a reasonable option, including the free version, particularly if you're working from an office that is separate from your home, or if you have a friend, relative, or given the size of your daily data increase perhaps just a neighbor who's willing to accept a machine for you. Notes on that at the end.
What you'll want to do is set up a backup machine with plenty of disk space. This will likely need to be a single monster volume and you probably want to ensure that it's expandable; I can't help with the details of that but others here certainly can. What are the good expandable RAID options?
The backup machine will receive the initial backup sitting next to the source machine, but will then be moved offsite. Given the volume of data you described, creating a "seed" backup and just moving that would likely be more hassle than it's worth.
All machines involved should have Gigabit Ethernet, and I strongly recommend investing in a Gigabit network switch or better, an ABGN router with Gigabit ports. If you can find one, I like the WNDR3700v2 (that v2 is REALLY IMPORTANT and hard to find these days) running OpenWRT, but there are plenty of other options. You're still going to need an Internet connection for the machines to identify each other (I believe this uses Crashplan's servers even if you're not backing up to them).
Once you have all that set up, you're just going to install Crashplan on both machines, run your backup, and move the backup machine to its new home. Crashplan assigns unique IDs to machines and uses those for coordinating the backups between them, so the fact that the backup server has moved won't cause a problem.
Finally for the note about "just a neighbor" - if you're creating tens or hundreds of gigs of new data per month, just hosting your offsite backup may be a problem for many people (think bandwidth caps), and transmitting it may be just as big a problem for you. If you have or can find a reasonably close neighbor that you'll trust with an encrypted copy of your data and who's willing to host your machine, I'm going to strongly suggest that you set up the machine with a good wireless connector and have it in their house but on your wireless. Ideally this person would not be in the same building you're in (assuming condo/apartment). Wireless-N at 2.4GHz is likely to be your best bet for range, though it's more prone to interference than 5.5GHz.
fencepost
just a little off
Or just use BD-R (*non*-LTH) discs. Millenniata discs are basically BD-R media burned to look like a DVD to the player. They're great when DVD-compatibility is important, but kind of a waste if you can make use of 25-gig BD-R capacity (BD-R is cheaper per-disc than Millenniata).
Just insist on *original*-type inorganic BD-R, and stay awy from "LTH" BD-R. The same way that Millenniata is like BD R media in DVD form, LTH is DVD media in Blu-Ray form.
I would suggest storing the data on a RAID drive set for redundancy first of all. See:
http://en.wikipedia.org/wiki/RAID
Then I would suggest having a Wi-Fi connection to another RAID redundancy drive set in a building separate from your house. This could be done with Ethernet cable or such instead but I like the Wi-Fi because it leaves no wires to pickup lightning blasts. We get a lot of lightning. Wi-Fi is essentially immune to the lightning in the real world test lab here on our mountain. Ethernet cabling not so even with tons of surge suppression. Of course, implement necessary encryption and security.
Then setup a remote backup program that copies changed files from the primary RAID drive set to the remote RAID drive set every night.
Then if you really care enough to do the very best, setup a third drive that gets backed up to weekly. And if you're paranoid, enough, then another for monthly backups. Then yearly. Then century...
All, of course, surrounded by high tensile electrified fencing and 300 Ninja guard pigs.
We installed a wireless link to my mother-in-law's place. We haven't done so yet, but the intention is to have some form of fileserver for backup there. We should expect around 4GB an hour transfer rate.
https://plus.google.com/photos/103933303525261507105/albums/5692174876583065521
Photos of wireless link
I always find it surprising that whenever the subject of backup comes up here, there're always only a one or two posts that suggest tapes only to be ignored. In the present day, tapes are still the best archival backup method readily available. Sure, the entry cost is rather high, but I think it would be worth it in your case since your data pool is only getting larger. Keeping backup copies on HDDs are convenient, but I would keep a minimum of 4 copies, at least a pair in 2 different sites since HDDs always have a chance of failure when powered. Cloud is only an option if you have no issues with bandwidth. Even without bandwidth issue, I still don't feel completely comfortable subjecting my data to risks entirely out of my control. Here's what I do personally. All my archival data is backed up on LTO Worm tapes, and I have most of those data on RAID-z2 for availability. Every online data is backed up incrementally daily on a separate array, and I do weekly full backup on tapes on monthly rotation along with month full backup on yearly rotation. Everything that needs permanent storage goes to WORM. While I won't claim my backup strategy is the best for everyone, but I go to bed each night satisfied with the level of safety with my data personally.
Take your current 8 TB of backed up data (which is on redundant drives you said), store them in your mom's basement...
If he is on /. odds are that is where the live drives are already sitting... right next to his stacks of comic books, gaming mags, and empty pizza boxes. Down the hall in the closet is not exactly offsite if you ask me.
All you need is a waterproof box, a shovel and some accurate GPS co-ordinates. It's waterproof so you should be safe from a flood. It's outside so you should be safe from an house fire and it's underground, so you'll be safe from any break-in's!
For those of us who don't really know much about tapes - can you point us to some drives and media that don't break the bank?
Why, no, I haven't meta-moderated lately. Thanks for asking!
Right now, your best option is tape backup, buried in a weather-proof box. Fire won't harm it under the ground - only moisture. And nosy neighbours who reckon you've buried cash.
A couple of 'burial mounds', with tapes rotated out every year, should be about right.
As technology improves, the ideal backup solution would have the weatherproof box contain SSDs sitting in cheap USB3-SATA caddies. USB cables would come out of a weatherproofed junction and rise to the surface through a snorkel. There, a small solar panel would power the SATA caddies as the USB3 equivalent of a Zigbee (wireless USB) chip. Hey, presto - onsite but still "remote", online backup! Or the data links become fiber-optic, you could probably string a long USB and Thunderbolt directly to the snorkel.
I have tried CrashPlan a couple of times on Windows and Linux, and had to give up. It would either fail to connect, or make very slow progress. It's not my broadband, since Mozy (Windows and Mac only) is fine. I also found that on Linux it would really hammer the system when backing up (4GB dual-core system) so it was barely usable.
Possibly CrashPlan's cloud service is the problem, but I'm not very impressed with the software.
For Linux and Mac backups, it's worth using something like rsnapshot, which is rsync-based and works very well to back up over 1 TB of data. It doesn't do block-level incremental backup, and it makes complete copies of files (rsync plus hard links) but it works incredibly well without writing shell scripts. It can do hourly, daily, weekly, monthly and yearly backups, and has automatic retention i.e. purges the oldest backups in a predictable way (say after 6 months or 5 years).
rsnapshot works well for Linux and (I believe) for Mac, as long as you don't need fully bootable backups, and it should work really well for photos as 99% of them won't change after being created.
rsnapshot is very similar in concept to Time Machine on Mac, but without the nice GUI (in fact, without any GUI). Your files end up in a big file tree and can be restored with any file-copy tool.
http://rsnapshot.org/
64 GIGABYTES A DAY? Unless you are doing NASA imagery that will be analyzed for the next years, you are doing it wrong. Seriously.
online is just too slow for such big data.
but when he gets gigabytes in a few hours, its too slow to upload and to expensive to mail harddrives every day.
If you storing it uncompressed, you are most likely wasting space. Even very slight compression will be invisible in practice* and can cut down the file sizes dramatically. You can easily get down to 25-50% with no loss in visual quality. If you don't believe me, do try it.
(*) - note that what matters to people looking at the photos is practice, not theory. What matters is also that any manipulation of them is 3-4 faster. I am not talking about blocky overcompression like we had in the nineties (and still see today in crappily compressed video streams). I'm a talking about compression that you cannot see, like high-rate lossy audio compression that is indistinguishable to the originals in double-blind tests by audio experts.
Make an encrypted archive and upload them on some newsgroup server.
It's probably the most reliable and cheapest solution there is. Retention is also really good.
Why not, you make backups on tape and store them in a safe deposit box or fireproof safe. It's a great way to store data. If it's in a good environment it'll last 30 years.
Set up a NAS with 9 3TB SATA drives. rsync your data to it.
The NAS should be a RAID 01 or a RAID15 or similar. The idea would be to have two sets of three drives. e.g. sdc sdd sde make up md3 and sdf sdg sdh make up md4. Make a RAID-1 of md3 and md4 (call it md5). Format your md5 with ext4 and do your rsync. When it's time to move your data offsite, break fail md4 out of md5 and remove it. Dissassemble md4 and pull the drives. Replace the drives and rebuild md4. Put md4 back into md5 and wait a long time for it to resync. Take your md4 drives and bring them offsite (to a friend's house or a bank safe deposit box).
For security, make a LUKS volume on top of md5 and mount that. You could do all of the above with zfsonlinux in a couple steps if you want to do that (though it has some beta-ish edges at the moment).
For really budget get external eSATA adapters and an eSATA card with multiple connectors. The freestanding units where you just slide in a drive are nice. But, that's really ghetto and you'll be happier with a hotswappable setup, if you can afford it at this point (or go ghetto now and upgrade later with no changes to the drives).
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Assuming that you're talking about these: http://www.digistor.com/Blu-ray-Recordable-Media/100GB-BD-R-Printable
That's $1175 (at $47/disk) to back up 2.5TB of data. Or, about $4000 and a whole lot of free time to back up the photos he already has, plus $20 per shoot.
It's a cool option, but not exactly an inexpensive one.
--
--Hey thanks for the tip man, that's not a bad deal! I may have to check them out.
http://www.digistor.com/DIGISTOR-Mulitimedia-Blu-ray-Archive-Kit
.
== WolfriderV6 == I'm willing to admit that *I just might* be wrong... Are you??
I am a photographer myself, this is what I do: I store all my photos (both RAW+JPEG) in a Subversion repository (I guess Git could be used as well, however, I started doing this way back, before Git existed). I have a workstation on which I do post-processing of my photos. The photos I work with are in a Subversion Working Copy. I "commit" the photos to the Subversion Repository which runs on a small server with some external USB 3 harddrives on my local network. I also have a two spare external hard drives that I periodically copy the Subversion Repository onto. One of these drives is always stored at my parents' place. My parents live a few hundred miles away. Every time I visit them, I bring the other extra hard drive with me and switch it for the one at my parents. This way, I always have an off-site backup at my parents'.
My workflow is this:
This gives me several advantages:
I notice the following disadvantages:
thomasdamgaard.dk.
The OP mentions limited budget and huge volumes of data to back up locally. Working within these constraints, a solution comes into focus:
FreeNAS + rsnapshot. Both are free ($0) and accomplish essentially what Time Machine does for Macs, but to/from almost any hardware. Bear in mind that any solution offering any semblance of security for 8TB of data won't be cheap.
Probably the biggest investment will be the NAS box itself, and of that, the HDDs will most likely cost the most. 8TB of RAIDed storage will easily cost >$500. The other hardware need not be much, it just needs enough SATA ports and power to run the HDDs, plus a GigE NIC.
rsnapshot keeps very intelligent backups, only recording diffs between backups using hardlinks. It's not too difficult to set up, and it's totally automated. The net result is a bunch of, well, snapshots, going back a few hours, days, months, even years if you like (and have storage for).
There's just no way to offer a fireproof solution for this much data without investing serious cash. Definitely store the NAS as physically separate as possible from the original data. The NAS only needs power and a single Cat6 cord, so it's conceivable to place it in a detached building or something. Keep in mind, heat and humidity will become an issue in extremes.