Slashdot Mirror
Easy, Reliable Distributed Storage and Backup?
RichiH writes "Most of you are the free IT staff of friends and family, just as I am. One of my largest headaches is backing up their data. What I am looking for allows for off-site storage on multiple server machines running Linux, has Linux & Windows clients that Just Work and require zero everyday effort (although a large-ish effort to set them up is just fine), allows for granular access control, is versioned and will, ideally, allow me to grab data automagically (think photo pool for your family where your mother, sister, etc., share each other's photos). This is something I've been trying to find for years, but I've never seen anything even closely resembling what I want. With the Wall Street Journal handing out its Technology Innovation Award to Cleversafe recently, I was once again reminded of this particular itch which needs scratching. Before I deploy it, I want to ask the Slashdot community for its opinion on that piece of software, and on potential alternatives. How do you solve this problem?"
I can tell you how I solve it in a business context, but whether or not it could be scaled down to personal I'm not sure.
The problem: 2 sites each with 70-100GB of data needs offsite backup with similar criteria to your own. Bandwidth available to these sites is 2-4Mbps. The only OS involved is Linux, though I'm sure Windows could be shoehorned in somehow. A third site which has a tape streamer and someone to take tapes offsite is available. Data protection legislation means that storing it with a hosted service is illegal unless I encrypt it myself before sending it offsite - I'm only aware of one tool which claims to be able to do this and still send data as a binary delta (it uses the rsync library) and that tool is still not particularly common in Linux distributions and not very widely used. I'm nervous of trusting my backups to a tool that isn't on heavy use, particularly if strong encryption is being employed.
The Solution: A server in the third site and some judicious scripting with rsync allows it to mirror the data in the other two sites. The first sync is fairly painful, of course, but provided you don't have too much data regularly changing subsequent syncs aren't too bad. The server is backed up to tape which provides versioning capability so if someone only realises that they lost a file a week after the fact it can still be restored,
Initial effort to set up was pretty great but now it's done it JFW and requires no brain power whatsoever to run on a daily basis. I can make the data available over the VPN (of course the access speed will be dog slow) more-or-less immediately and I can make it available at LAN speed by copying it to a hard disk and courier it to the remote office in under 48 hours. A full restore of 100GB across a 2Mbps connection will take at least 4-5 days.
Ars technica did a nice review of Dropbox, titled, "How Dropbox ended my search for seamless sync on Linux" (but it works on OSX 7 Windows too) http://arstechnica.com/news.ars/post/20080914-how-dropbox-ended-my-search-for-seamless-sync-on-linux.html
http://www.bacula.org/
Runs pretty tight (low bandwidth), supports channel encryption and datastore encryption, can even create Bare Metal Recovery disks. I have a server room with LTO3 tape drives that I use to backup my clients' incremental data changes nightly, including Linux, Mac and Windows clients and servers. I have VPN's out to each client, so don't use the built-in channel encryption, but I maintain a keypair for each client.
Backup only, but I /could/ present a maintained volume as a share over the VPN. Bacula supports disk and tape volumes as backup stores. I've personally had no need to do that to date.
We're not talking terabytes here - my ISP would pwn me if that was going on, but I do circa 20G of data changes every night from clients. Some of them are laptops that are not always on or connected. Most are friends and family PC's, so it backs up when it can. I have to do almost no maintenance apart from changing a tape occasionally. The backup client is tiny and unobtrusive, even when running. On Windows it uses VSS, so it is reliable.
I have had a number of panic phone calls (esp from my kids at Uni) who have lost a thesis or the like and are utterly amazed when, after a few clicks over the phone they look at their webmail and yesterday's version is in their inbox. That's what it's all about! I am the god of lost data! Which, of course, works for me.
If you try to roll backup and distributed file-storage into the same application, you're not going to get anything useful. Aunt Sally is going to want every single file including her OS and her tax returns backed up, in case her hard drive dies, but only wants the photos -- and only some of the photos, actually -- to be visible to Grandma Suzie. If Suzie can see every file on Sally's computer, and the entire history of each file, she's not going to be able to browse the photos in a way that's at all intuitive.
And worse yet, if Sally wants to send out links to her photos to fifteen of her friends by e-mail, she needs some sort of interface to mark parts of her backup as world-readable but the rest (like her passwords and e-mail) not. If the network backup program even lets you do this, it won't give Sally a UI that she'll be able to figure out.
You can certainly get network backup services: Mozy was mentioned in an earlier comment.
If you rethink your requirements in terms of your goals, you'll probably find that both rolled into one isn't what you want, and not just because a product doesn't exist at the moment that does that — a product that does that can't possibly have a good UI. If they shouldn't notice or care about how backups are being made, how are they going to figure out how to share photos with each other?