Slashdot Mirror
Ask Slashdot: Distributed Online Storage For Families?
StonyCreekBare writes "What options are available for distributed storage for families? My two brothers, my daughter and her husband, and his mother all have homes in various parts of the country. We use various cloud storage providers to keep our shared data. This has numerous limitations and we are starting to think maybe we can do it better ourselves. We all have decent Internet connections, are all somewhat tech savvy, and think that by leveraging the Internet we can maybe provide for our needs better and at lower cost by buying some hardware and doing it ourselves. How would you go about implementing such a family-oriented, distributed cloud platform? What hardware? What applications, beyond simply the preservation and sharing of family data, (grandkids' photos, home videos, and more) would be good to leverage such a platform? Security Cameras? HTPC? VoIP? Home Automation? Primary requirements are Cheap, Secure, Reliable."
Amazon S3 with Expandrive
FIRST, you decide on what functionality you want.
THEN you look at how to achieve that functionality within your budget.
I'd use rsync as the cheapest means of replicating data between multiple sites. But once you start adding additional functionality requirements that might change.
has anyone thought of a simple VPN + NAS solution?
Off the top of my head, there's two obvious solutions I can suggest. Bittorrent Sync or git-annex. The former is easymode but limited in scope. The latter comes with a webui for some simple things, but also gives you a *lot* of power from the command line. I've never used either on Windows, but if that matters, Bittorrent Sync is probably the more stable of the two right now for that platform (but improved Windows support is the theme this month for git-annex's crowdfunded development, so it should be improving).
Obligatory FUCK BETA, because seriously... fuck it.
The next version (beta version released) of the system that runs the Synology NAS will offer synchronizing from one NAS to another. And the available products from Synology are very reliable. I have been using their products for a few years now and is a very satisfied customer.
You can read about this new feature her.
- "Every demand is a prison, and wisdom is only free when it asks nothing." Sir Betrand Russell
I think I would have each family member that wants to be part of this shared drive do something like:
- each of you buy the biggest HDD available
- setup a ssh tunnel in the form of a circle between each family member, where Alice connects to Bob, Bob connects to Charlie, and Charlie connects to Alice.
- each family member then rsync's to the next family member over, where they would do a full rsync of the shared disk, but do an rsync --delete on directories that belong to themselves, so if they delete / move files around, it makes the needed corrections on other family members shared disks without wasting space.
If you are running Windows, you can setup a scheduled task to at a time in the middle of the night to launch cygwin, open the ssh tunnel, and rsync away. If it is linux, setup a crontab. Initial coordination would need to be done to get everything right, but then it would be very automatable.
I do not suggest trying to setup a distributed filesystem across the internet. There are many pitfalls. Whereas this solution, your only concern is, 1) is ssh up? 2) did rsync run? 3) is the disk full?
Been looking for the same thing for a while. Finally settled on File Transporter. http://www.filetransporter.com... Now owned by Drobo.
Clients for every platform. Server distributions for every platform. Mobile clients too. Runs on HTTPS.
I've set up something similar for my family - love it. I've also set up something simliar for our enterprise. No complaints about the regular feature set. Just some of the enterprise level things could do with a little more work.
Our family uses simple NAS boxes (Dlink DNS-323, etc.). We put Debian on them and all the boxes use rsync over ssh in the middle of the night to synchronize the data. Pretty much every "family site" has one. They are also useful for private local storage, shared folders, etc. Everyone knows that any file they put in the "backup" folder will be looked after, everything else is just local. Been working okay for 2 years now. Note - this is not RAID, just distributed backups. Way cheaper than commercial offerings.
Rsync is a one-way synchronization. Check out Unison; it readily performs a bidirectional merge. You might have to do a little compiling, but hey, isn't that what the Family Geek is for?
I've been using Unison to sync a pair of Synology boxes that act as my cloud. (One in my office, one at home, each with a RAID-1 array.) I've also gotten it running on a pair of DLink DNS-323 boxes (yes, also RAID-1'ed). The Synology has cloud software; might be a good choice if you want to invest in a cheap small light unobtrusive (Linux) NFS/cloud/music server/etc box.
This is exactly what this product has been designed for.
Joey Hess does a great job.
http://git-annex.branchable.com/design/assistant/
The Red Matrix. Install your own hub or use a public one. https://libertypod.com
Online Storage for Families would be great.
A place where you can store the kids in the cloud while you go on holiday. Or something where you can permanently dump the in-laws without hating to store them at home where they take up valuable space. Something that puts them in deep hibernation would be nice, so that the food cost don't run rampant.
You can buy one or more Synology NAS and sync your files with all devices. Even access them on the go, from mobile/tablet. http://www.synology.com/en-glo... The cost is quite small, you reuse what HDDs you have sitting around, and you only need to do parameterization.
bittorrent sync seems to answer your problem. distributed, secure, free, etc.
everybody shares a folder, everybody's got a copy of it
VoIP, Security Cameras, HTPC, Home Automation?
I don't see these as services that should be moved outside your home. VoIP is going to be dependent on a third-party provider to start with. Security cameras are going to need to record to a in-home location due to data amounts, and putting home automation control outside your home sounds like a security risk.
In the end this still sounds like a case of wanting to share files, pictures, and video really, so you'd want the storage to be off-site. You could have your own server put up at a shared data center if you want to own the hardware, but you could get a VPS account and then tweak it as you want instead. If you do the colo solution you do have the bonus of shipping the hardware around the country for the initial (looong) backup before it's installed.
Everyone's fairly tech savvy, right?
1.) Figure out a folder structure that makes sure that everyone's data will be put somewhere and won't accidentally be overwritten by someone else's.
2.) Install BitTorrent Sync on something with a hard drive to hold it. Windows box with a USB hard drive? There's a client. OSX? Client. Ubuntu box? Client. DIY FreeNAS with a RAID-1 in a small case? There's a client. Synology or QNAP box? There's a client, albeit with a little command shell necessary. Hell, those $199 Western Digital Personal Cloud drives can run it.
3.) Create those folders on everyone else's machine, e-mailing around the BT Sync folder keys.
4.) Wait for replication of everyone else's data to your drive, and vice versa - everyone will help everyone else get a copy of the data they don't have.
5.) Profit.
Literally every question answered:
How would you go about implementing such a family-oriented, distributed cloud platform?
See above.
What hardware?
Whatever hardware you have lying around, as long as it has the storage capacity you're looking for, and can permanently stay on. A few suggestions are above, but I'm a bit of a FreeNAS guy myself, especially since you can build a half-decent one with a 2TB RAID-1 for about $400 these days. The WD Cloud Drives are about the cheapest and self-contained route to go, so they may be worth considering if you need more than 3 or 4 of them.
What applications, beyond simply the preservation and sharing of family data, (grandkids' photos, home videos, and more) would be good to leverage such a platform? Security Cameras? HTPC? VoIP? Home Automation?
Well this is the rather perplexing part, because on the one hand you're asking for decentralized storage, and then you ask why you'd use it (VoIP + decentralized storage?!? wtf??). If you need decentralized storage, one should safely be able to assume that that there's already a reason. Having said that, photos would be my first use case, with disaster recovery being the second - Acronis True Image supports backup to FTP/SMB locations, so as long as you can back up to one of them that way, the rest will distribute.
Primary requirements are Cheap, Secure, Reliable."
Cheap? BT Sync is free; you'd need storage regardless. There's 10,001 topics on Slashdot where "the most reliable form of storage" comes up. "How much do you want to spend" is inherently the question, and "Cheap" indicates "not much"...it also doesn't answer exactly how much storage you'll need. Are you undertaking a massive photo album archiving project, or capturing the last 20 years of home videos? a 2TB drive just might cut it, or not. Are you backing up everyone's laptops? 6TB, MAYBE, and single-drive solutions won't cover it anymore...but are you prepared to start forking over $600 a box, along with a weekend of your time (at least) to the cause? Are you doing a roll-your-own Netflix where everyone will add their own CD/DVD rips to the units and then let Plex Media Server work its magic?
Okay, so I lied...one of the underlying questions have been answered: how to get files to the geographically disparate places in the easiest way possible. BT Sync, at the low, low cost of 'free', resolves this. The questions regarding hardware, and how much storage you will need, and what protocols it will need to support, are wholly dependent on how much data will, in total, have to sit on each device. Answer that question, along with the follow-up of "how safe do you really, REALLY need to be?"and then you can start figuring out numbers to go along with it.
I know it is not a fancy or automated solution. But is should provide a cheap solution to most of the functions that the poster wants.
I've used rsync to push and to pull so it is bi-directional.
The main difference between rsync and Unison is what happens when file X is altered at the local site AND at the remote site between a single sync interval.
With rsync, one of the altered files will be over-written by the alterations to that file at the other site.
Whether this is a problem or not depends upon your specific situation.
I've been reading Slashdot since 2000, so going on 14 years now. But I'll be stopping next week in support of the boycott, and maybe after that, if the interface catastrophe called "Beta" goes live.
See you on Usenet at comp.misc where old school commenting is happening: no mods, no karma, no whitespace, and no advertising. Just a lot of old geeks with killfiles and a keyboard.
Uck fay Eta bay!
If this were Usenet, I'd killfile the lot of you.
Even more interesting is how all posts criticizing slashdot beta are being modded down. So many mod points. None in my possession. Wow.
Shitting on the community is time and time again proven to have the worst outcome... why don't people learn?
`echo $[0x853204FA81]|tr 0-9 ionbsdeaml`@gmail.com
Skydrive is currently the primary cloud service we are using. That is what we're considering an alternative from...
I agree that first, you should decide what your goal is. Shared storage? Sync? (Sync is not the same as shared storage because you have multiple copies of the data.) Maybe a shared storage area and multiple private areas?
I did not recommend Amazon in part because OP said they wanted to do it themselves. AWS and S3 are not "do it yourself". They rely on 3rd-party servers.
I'd say that SSH can be the solution, all by itself. The simplest case is shared storage: just set up a server on a network connection, and allow SSH access. Then, on your end-user devices, install a file manager that can use SSH (SFTP). Voila! Your own "cloud storage", on your home server, accessible from anywhere on the internet. (Use good passwords, or use SSH certificates.)
You can also set up separate areas on a disk and give people access only to a certain area.
I use file managers that treat SFTP connections like just any other hard disk. It's simple, it's transparent.
Using SSH and a simple GUI file manager, I can access my server at home from my smartphone from anywhere in the world. And on my personal home machine, I have reverse SSH set up in a shell script. If I'm leaving home for a while, I just run the script to set up reverse SSH from the server and leave it running. Then I have access to both remotely. I can access either one from anywhere.
No "streaming" servers are necessary. No fancy sharing services. No 3rd parties at all, in fact. It's just a remotely-accessible hard disk.
I haven't found it necessary to do anything else. By default I have access to the server's big archive disk from anywhere. Turn on the Reverse SSH and I have that AND my desktop machine. Just as though they were local hard disks. Except for the transmission delays, of course.
AND, I can do it via wifi or wired internet (somebody else's machine), as long as they have an SFTP-capable file mgr. program, OR I can do it through my cell phone if I have remote data enabled on the phone. (If you're on Android, try ES File Manager or, better in my opinion, Total Commander. TC is also available for other OSes.)