Slashdot Mirror
Easy, Reliable Distributed Storage and Backup?
RichiH writes "Most of you are the free IT staff of friends and family, just as I am. One of my largest headaches is backing up their data. What I am looking for allows for off-site storage on multiple server machines running Linux, has Linux & Windows clients that Just Work and require zero everyday effort (although a large-ish effort to set them up is just fine), allows for granular access control, is versioned and will, ideally, allow me to grab data automagically (think photo pool for your family where your mother, sister, etc., share each other's photos). This is something I've been trying to find for years, but I've never seen anything even closely resembling what I want. With the Wall Street Journal handing out its Technology Innovation Award to Cleversafe recently, I was once again reminded of this particular itch which needs scratching. Before I deploy it, I want to ask the Slashdot community for its opinion on that piece of software, and on potential alternatives. How do you solve this problem?"
No Linux client, AFAIK (though I do run it on my MBP). It's become rather impractical for me as a photographer though, as sometimes I'll shoot enough photos that my internet connection would be completely maxed out for days on end trying to sync up the new data - and I have a decent-for-cable 1Mbps upload rate.
rsync to Amazon S3 might be an option, if only for cross-platform capabilities. No versioning though, but outside of Apple's Time Machine (obviously useless for Windows and Linux), you're not going to get that without some major headache. Any remote system is going to be horribly slow for the first sync with any typical internet connection, and quite possibly problematically slow for photographers, media horaders, and in general people with big hard drives.
How are sites slashdotted when nobody reads TFAs?
Dropbox is absolutely fantastic as a sync tool (and also has some degree of versioning), but there's no practical way as of yet to make it into a full-system backup. When 'watch folders' show up, it'll get a lot closer, but like any web-based system, it becomes impractically slow for anyone dealing with lot of data. Even digital snapshots add up quickly with the resolution of the point-and-shoot cameras, never mind if there's an actual photographer shooting RAW.
How are sites slashdotted when nobody reads TFAs?
Being easy to use (as in, not more than 3-5 mouse clicks, total) is one of my main concerns. Git definitely fails in this regard.
If you try to roll backup and distributed file-storage into the same application, you're not going to get anything useful. Aunt Sally is going to want every single file including her OS and her tax returns backed up, in case her hard drive dies, but only wants the photos -- and only some of the photos, actually -- to be visible to Grandma Suzie. If Suzie can see every file on Sally's computer, and the entire history of each file, she's not going to be able to browse the photos in a way that's at all intuitive.
And worse yet, if Sally wants to send out links to her photos to fifteen of her friends by e-mail, she needs some sort of interface to mark parts of her backup as world-readable but the rest (like her passwords and e-mail) not. If the network backup program even lets you do this, it won't give Sally a UI that she'll be able to figure out.
You can certainly get network backup services: Mozy was mentioned in an earlier comment.
If you rethink your requirements in terms of your goals, you'll probably find that both rolled into one isn't what you want, and not just because a product doesn't exist at the moment that does that — a product that does that can't possibly have a good UI. If they shouldn't notice or care about how backups are being made, how are they going to figure out how to share photos with each other?
I want distributed backups with several, for lack of a better word, working copies checked out on different machines.
Aha, now I figured out why we're all misunderstanding you. Those aren't backups. "Backups" to my ears means that you copy the entire contents of your disk or your Documents folder nightly onto tape or some other archival medium, so that in case of hardware failure you have something to restore from. Potentially you also keep prior versions around. The tapes are stored in a corner somewhere because they're never actually accessed except in an emergency, and they're destroyed after a few months.
What you want isn't backups, since it doesn't make sense for different people to share backups any more than it makes sense for different people to share a single networked hard disk or networked home directory. You just want a distributed file storage system, with automatic syncing / commits.
For storing permissions and the such, are you using a .tar container? My biggest stumbling block with my backup scheme is storing ACLs and permissions.
I've got a few ideas about doing it, but they're all kludgy or force me to walk away from my rsync scripts which are really fairly mature at this point. Furthermore, I need to get deltas downstream and packing everything in to one file pretty much defeats that purpose at the several gig level unless I'm running an rsync server to calculate the diffs. These kinds of things become problematic due to the infrastructure I'm working with.
I'm really starting to lean towards running everything over iSCSI, but then I've got to get the VPN thing going which could require some re-subnetting at either end of the tunnel. Needless to say, I'd prefer to avoid that or any other solution that requires messing with stuff that Works Right Now.
Have you dealt with these issues at all, or at least know what won't work? I'd appreciate any insights before I use a brute force method.
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
I think that the issue is faced by far more people than is readily apparent... it's the need for a VERY easy to use tool to share Our Stuff with Our Family. If my Mom and sisters were able to share all their photos with each other by carrying a USB drive around when they see each other... the most important thing they have on their computers would be backed up... the need for social file sharing is huge... we just don't have the tools to do it well yet. Something that does auto-discovery of stuff, remembers previous decisions, and just goes to work making copies in the right directions is what we need.
I watched their CTO's Google Talks presentation and it was really interesting. I got all excited, joined their beta only to realize that they - IMO - misused the technology they had and designed a rather mediocre product. Wuala wants to be a backup tool, a sharing tool, a social networking medium as well as few other things. In other words it lacks focus and wants to do everything - an approach that rarely works.
3.243F6A8885A308D313
I would code it in Perl, but I do _not_ want to expose my mom to a CLI interface.
why not make a CGI and use the web as a GUI?! :-)
Even if your parents are non techy, they've probably had some experience browsing the web, so a web GUI shouldnt be too foreign to them.
couple that with something like CGI::Application and you've got a great framework in which to build and expand what you/your parents want/need.
-- $_='ab-bc ratvarre';tr"'a-z'"'n-za-m'";print