Slashdot Mirror
Ask Slashdot: Distributed Online Storage For Families?
StonyCreekBare writes "What options are available for distributed storage for families? My two brothers, my daughter and her husband, and his mother all have homes in various parts of the country. We use various cloud storage providers to keep our shared data. This has numerous limitations and we are starting to think maybe we can do it better ourselves. We all have decent Internet connections, are all somewhat tech savvy, and think that by leveraging the Internet we can maybe provide for our needs better and at lower cost by buying some hardware and doing it ourselves. How would you go about implementing such a family-oriented, distributed cloud platform? What hardware? What applications, beyond simply the preservation and sharing of family data, (grandkids' photos, home videos, and more) would be good to leverage such a platform? Security Cameras? HTPC? VoIP? Home Automation? Primary requirements are Cheap, Secure, Reliable."
Amazon S3 with Expandrive
What company directs 25% of its users to a partially-working, not-ready-for-production website? Please realize that Beta will not have the features that we want, because it goes against Dice's plans for Slashdot. To their advertisers, Dice presents Slashdot as a "Social Media for B2B Technology" platform. B2B - that's the reason Beta looks like a generic wordpress-based news site. A large precentage of the current userbase might be in IT, but /. is most certainly not a B2B site.
Nevertheless, Dice is desperate to make money off of Slashdot, since it has not lived up to their financial expectations, a fact that they have revealed in a press release detailing their performance in 2013:
Slashdot Media was acquired to provide content and services that are important to technology professionals in their everyday work lives and to leverage that reach into the global technology community benefiting user engagement on the Dice.com site. The expected benefits have started to be realized at Dice.com. However, advertising revenue has declined over the past year and there is no improvement expected in the future financial performance of Slashdot Media's underlying advertising business. Therefore, $7.2 million of intangible assets and $6.3 million of goodwill related to Slashdot Media were reduced to zero.
Beta is not a cosmetic change. It is a new design that deliberately ruins the one thing that makes /. what it is today -- the commenting system. There is nothing wrong with Slashdot, from the users' perspective, that demands breaking its foundations. As others have commented, this is an attempt to monetize /. at any any cost, and its users be damned. Dice views its users, the ones who create the site, as a passive audience. As such, it is interchangeable with its intended B2B crowd. We, the current users of Slashdot, are an obstacle in Dice's way.
That is why they ignore the detailed feedback they have received in the months since they first revealed Beta. That is also why they now disregard our grievances. Their claims of hearing us are a deliberate snow job. It is only pretense, since at the same time they openly admit that Classic will be cancelled soon:
"Most importantly, we want you to know that Classic Slashdot isn't going away until we're confident that the new site is ready.
Don't hold your breath waiting for Dice to fix Beta. Their vision of Slashdot is a crippled shadow of the site as it is today. Don't let them pull the wool over your eyes. Dice doesn't need us, and it wants us out.
Slashdice delenda est!
You might want to check out LOCKSS (Lots of Copies Keep Stuff Safe). http://www.lockss.org/
Tahoe-LAFS
Please post this to new articles if it hasn't been posted yet. (Copy-paste the html from here so links don't get mangled!)
On February 5, 2014, Slashdot announced through a javascript popup that they are starting to "move in to" the new Slashdot Beta design. Slashdot Beta is a trend-following attempt to give Slashdot a fresh look, an approach that has led to less space for text and an abandonment of the traditional Slashdot look. Much worse than that, Slashdot Beta fundamentally breaks the classic Slashdot discussion and moderation system.
If you haven't seen Slashdot Beta already, open this in a new tab. After seeing that, click here to return to classic Slashdot.
We should boycott stories and only discuss the abomination that is Slashdot Beta until Dice abandons the project.
We should boycott slashdot entirely during the week of Feb 10 to Feb 17 as part of the wider slashcott
Moderators - only spend mod points on comments that discuss Beta
Commentors - only discuss Beta
http://slashdot.org/recent - Vote up the Fuck Beta stories
Keep this up for a few days and we may finally get the PHBs attention.
-----=====##### LINKS #####=====-----
Discussion of Beta: http://slashdot.org/firehose.pl?op=view&id=56395415
Discussion of where to go if Beta goes live: http://slashdot.org/firehose.pl?op=view&type=submission&id=3321441
Alternative Slashdot: http://altslashdot.org (thanks Okian Warrior (537106))
Many viable technological solutions are rendered useless by PHBs that do not understand what the users actually want. Fuck beta.
/. goes rogue.
Primary requirements are Cheap, Secure, Reliable.
Cheap? Free.
Secure? That's up to you but normally, yes.
Reliable? In my experience, yes.
Is it just me, or have the "editors" been exceptionally prolific these past two days? The rate they're working, the stories might still be new and relevant by the time they're posted!
Although they try to drown out the criticism, they cannot shut out the death knell of Slashdot.
It is rude to randomly redirect visitors to beta.slashdot.
Even more so because beta sucks.
Providing a hard to find opt-out, adding /?nobeta=1 to the url, just upgrades the aggravation level from "rude" to "insulting and infuriating".
The only acceptable option is, as always, opt-in.
I guess you need reminding. a lot.
FIRST, you decide on what functionality you want.
THEN you look at how to achieve that functionality within your budget.
I'd use rsync as the cheapest means of replicating data between multiple sites. But once you start adding additional functionality requirements that might change.
I keep reading that 25% of /. users are directed to the beta site ... how come I have achieved 100% redirection to beta over the past week with NO classic? hmmmm...
FOIA request made to the NSA
It's cheap since you don't have to buy hw, sw or bandwidth. It's secure for the same reason that access to your data might be time-consuming and unreliable.
has anyone thought of a simple VPN + NAS solution?
I'm using copy.com . It's good, reliable and the price is fine.
Off the top of my head, there's two obvious solutions I can suggest. Bittorrent Sync or git-annex. The former is easymode but limited in scope. The latter comes with a webui for some simple things, but also gives you a *lot* of power from the command line. I've never used either on Windows, but if that matters, Bittorrent Sync is probably the more stable of the two right now for that platform (but improved Windows support is the theme this month for git-annex's crowdfunded development, so it should be improving).
Obligatory FUCK BETA, because seriously... fuck it.
On behalf of Dice Holdings, Inc., I'd like to formally welcome the new audience to Slashdice. To the decades old /. community, fuck you.
I'm working on an open source project to bt-sync, its not ready yet though, but bt-sync is. I'd use that for sure.
The next version (beta version released) of the system that runs the Synology NAS will offer synchronizing from one NAS to another. And the available products from Synology are very reliable. I have been using their products for a few years now and is a very satisfied customer.
You can read about this new feature her.
- "Every demand is a prison, and wisdom is only free when it asks nothing." Sir Betrand Russell
They're a bit expensive and I wish the documentation was better, but I've had some luck with the Synology products. They've got a lot of plug-in software modules, including an Asterisk PBM for VoIP, Cloud Station for folder synchronization, etc.
Make sure that you look at the specifications, if you're wanting encrypted tunnels or encrypted data on the drives, ensure that you buy one with the AES encryption set in hardware.
One more thing: I have had very poor luck with the Seagate drives I originally bought and put in a DS 412+. I replaced them with Western Digital Red NAS drives, and they work much better.
HA!
I have set up multiple clients to compress then encrypt their database backups and/or logs and then push them from their office site to a drive on a distantly located employees machine at home. Several directors have a CD that contains the decryption keys and instructions for reconstituting the data center from the backups and logs. Data is pushed during off hours. A similar setup can be utilized by your family. Keep two encrypted copies in two locations geographically separated and decryption keys/software on media in a distant location that you can control.. You can then easily recover lost data.. If you can afford raid clusters, to guard against drive failures, all the better.
I think I would have each family member that wants to be part of this shared drive do something like:
- each of you buy the biggest HDD available
- setup a ssh tunnel in the form of a circle between each family member, where Alice connects to Bob, Bob connects to Charlie, and Charlie connects to Alice.
- each family member then rsync's to the next family member over, where they would do a full rsync of the shared disk, but do an rsync --delete on directories that belong to themselves, so if they delete / move files around, it makes the needed corrections on other family members shared disks without wasting space.
If you are running Windows, you can setup a scheduled task to at a time in the middle of the night to launch cygwin, open the ssh tunnel, and rsync away. If it is linux, setup a crontab. Initial coordination would need to be done to get everything right, but then it would be very automatable.
I do not suggest trying to setup a distributed filesystem across the internet. There are many pitfalls. Whereas this solution, your only concern is, 1) is ssh up? 2) did rsync run? 3) is the disk full?
I just post stuff to the internet and let them take care of the storage. If I need anything I just submit a FOI request.
When you're talking about consumer storage for families, you need three things: reliable, easy to use, and web-based.
That said, it seems to me what you're looking for is SLASHDOT BETA. It has been designed with ease of use in mind, in fact, a complete idiot could poke around at its big images all day and have a great time, and all the whitespace means you'll never get lost, even if you have a major stroke while using it. It's reliable, because unlike the old Slashdot its daily traffic is almost zero, so you'll never have to worry about slowdowns. And it's web-based, because unlike the old Slashdot it uses lots of HTML5 and JavaScript, so that's how you know it's the web, real modern-like.
On the other hand, if you'd prefer to BOYCOTT SLASHDOT, that starts on Monday, February 10. Make sure you logout on Sunday, so that even if you want to check in to see whether Dice has got the message, Slashdot gets NO TRAFFIC from registered users.
And for all you quislings posting about "whining ACs," my real name is Neil McAllister and I am one of the first 5,000 people to register on Slashdot, which means I have been here for almost 20 years. I am fully in support of the anti-beta effort and I am willing to burn all of my karma to bring you this message.
Breakfast served all day!
Been looking for the same thing for a while. Finally settled on File Transporter. http://www.filetransporter.com... Now owned by Drobo.
Clients for every platform. Server distributions for every platform. Mobile clients too. Runs on HTTPS.
I've set up something similar for my family - love it. I've also set up something simliar for our enterprise. No complaints about the regular feature set. Just some of the enterprise level things could do with a little more work.
BitTorrent Sync http://getsync.com/
Your description seems to be the perfect use case for BTSync
There is a reason why "News for Nerds, stuff that matters" no longer appears in the header:
Slashdot Media’s brands include Slashdot and SourceForge. These technology sites provide access to tools, software and forums for enterprise IT professionals working in all industries and companies from the world’s largest to small and medium-sized firms. Slashdot and SourceForge harness the power of social that no other tech site can compete with.
Slashdot Media provides its partners with proven integrated media strategies to effectively influence technology buyers. With over 15 years experience working with the largest and most engaged professional technology communities, Slashdot Media’s expert staff continues to contribute to the success of its partners branding, demand generation, and social media marketing programs.
I, for one, abhor our new corporate overlords.
Our family uses simple NAS boxes (Dlink DNS-323, etc.). We put Debian on them and all the boxes use rsync over ssh in the middle of the night to synchronize the data. Pretty much every "family site" has one. They are also useful for private local storage, shared folders, etc. Everyone knows that any file they put in the "backup" folder will be looked after, everything else is just local. Been working okay for 2 years now. Note - this is not RAID, just distributed backups. Way cheaper than commercial offerings.
Rsync is a one-way synchronization. Check out Unison; it readily performs a bidirectional merge. You might have to do a little compiling, but hey, isn't that what the Family Geek is for?
I've been using Unison to sync a pair of Synology boxes that act as my cloud. (One in my office, one at home, each with a RAID-1 array.) I've also gotten it running on a pair of DLink DNS-323 boxes (yes, also RAID-1'ed). The Synology has cloud software; might be a good choice if you want to invest in a cheap small light unobtrusive (Linux) NFS/cloud/music server/etc box.
This is exactly what this product has been designed for.
Joey Hess does a great job.
http://git-annex.branchable.com/design/assistant/
The Red Matrix. Install your own hub or use a public one. https://libertypod.com
Online Storage for Families would be great.
A place where you can store the kids in the cloud while you go on holiday. Or something where you can permanently dump the in-laws without hating to store them at home where they take up valuable space. Something that puts them in deep hibernation would be nice, so that the food cost don't run rampant.
You can buy one or more Synology NAS and sync your files with all devices. Even access them on the go, from mobile/tablet. http://www.synology.com/en-glo... The cost is quite small, you reuse what HDDs you have sitting around, and you only need to do parameterization.
bittorrent sync seems to answer your problem. distributed, secure, free, etc.
everybody shares a folder, everybody's got a copy of it
VoIP, Security Cameras, HTPC, Home Automation?
I don't see these as services that should be moved outside your home. VoIP is going to be dependent on a third-party provider to start with. Security cameras are going to need to record to a in-home location due to data amounts, and putting home automation control outside your home sounds like a security risk.
In the end this still sounds like a case of wanting to share files, pictures, and video really, so you'd want the storage to be off-site. You could have your own server put up at a shared data center if you want to own the hardware, but you could get a VPS account and then tweak it as you want instead. If you do the colo solution you do have the bonus of shipping the hardware around the country for the initial (looong) backup before it's installed.
Will this work?
http://www.bittorrent.com/sync/get-started
Indeed, the interesting thing about using OwnCloud on Amazon is FUCK BETA!
I'm already using it to share family photos back and forth with my sister, brother, and octogenarian grandparents.
Works flawlessly and transparently, has no space limits, and requires no user training beyond setting up a folder.
I'm also using it at an art gallery I volunteer for to sync files between machines at the office and several board members, which allows both collaboration and provides us an automatic file backup.
BitTorrent Sync is an easy to use free file-syncing program.
http://www.bittorrent.com/sync/get-started
Everyone's fairly tech savvy, right?
1.) Figure out a folder structure that makes sure that everyone's data will be put somewhere and won't accidentally be overwritten by someone else's.
2.) Install BitTorrent Sync on something with a hard drive to hold it. Windows box with a USB hard drive? There's a client. OSX? Client. Ubuntu box? Client. DIY FreeNAS with a RAID-1 in a small case? There's a client. Synology or QNAP box? There's a client, albeit with a little command shell necessary. Hell, those $199 Western Digital Personal Cloud drives can run it.
3.) Create those folders on everyone else's machine, e-mailing around the BT Sync folder keys.
4.) Wait for replication of everyone else's data to your drive, and vice versa - everyone will help everyone else get a copy of the data they don't have.
5.) Profit.
Literally every question answered:
How would you go about implementing such a family-oriented, distributed cloud platform?
See above.
What hardware?
Whatever hardware you have lying around, as long as it has the storage capacity you're looking for, and can permanently stay on. A few suggestions are above, but I'm a bit of a FreeNAS guy myself, especially since you can build a half-decent one with a 2TB RAID-1 for about $400 these days. The WD Cloud Drives are about the cheapest and self-contained route to go, so they may be worth considering if you need more than 3 or 4 of them.
What applications, beyond simply the preservation and sharing of family data, (grandkids' photos, home videos, and more) would be good to leverage such a platform? Security Cameras? HTPC? VoIP? Home Automation?
Well this is the rather perplexing part, because on the one hand you're asking for decentralized storage, and then you ask why you'd use it (VoIP + decentralized storage?!? wtf??). If you need decentralized storage, one should safely be able to assume that that there's already a reason. Having said that, photos would be my first use case, with disaster recovery being the second - Acronis True Image supports backup to FTP/SMB locations, so as long as you can back up to one of them that way, the rest will distribute.
Primary requirements are Cheap, Secure, Reliable."
Cheap? BT Sync is free; you'd need storage regardless. There's 10,001 topics on Slashdot where "the most reliable form of storage" comes up. "How much do you want to spend" is inherently the question, and "Cheap" indicates "not much"...it also doesn't answer exactly how much storage you'll need. Are you undertaking a massive photo album archiving project, or capturing the last 20 years of home videos? a 2TB drive just might cut it, or not. Are you backing up everyone's laptops? 6TB, MAYBE, and single-drive solutions won't cover it anymore...but are you prepared to start forking over $600 a box, along with a weekend of your time (at least) to the cause? Are you doing a roll-your-own Netflix where everyone will add their own CD/DVD rips to the units and then let Plex Media Server work its magic?
Okay, so I lied...one of the underlying questions have been answered: how to get files to the geographically disparate places in the easiest way possible. BT Sync, at the low, low cost of 'free', resolves this. The questions regarding hardware, and how much storage you will need, and what protocols it will need to support, are wholly dependent on how much data will, in total, have to sit on each device. Answer that question, along with the follow-up of "how safe do you really, REALLY need to be?"and then you can start figuring out numbers to go along with it.
Also, on Sunday, block it at the the firewall for the week. That way you won't accidentally send any traffic their way. The rule in iptables or ufw can be taken back out after it is over.
WD has a really cool NAS device, if your audience has a bit of tech knowledge it should be easy to get one or even 2 of the devices (backup is great) send them to 2 of your family in geographically diverse locations and presto, private cloud storage for you and the family.
Cnet has a review posted here
http://www.cnet.com/network-st...
errr....umm...*whooosh* *whoosh* Is this thing on ?
Seriously. Dropbox is dead simple, which is good for non-savvy family. It just works, handling all the dirty work for you. No tunnels, rsync, no crap like that. The weakest point of Dropbox in my opinion is that any person can delete a file. But Dropbox keeps backups, and you can keep backups yourself of course. It costs money, but in the grand scheme when you consider what you'd have to buy to do it yourself, along with the time it will take, then Dropbox doesn't look all that expensive to me. I pay $200/year for 200 GB. I run a VM at my office, a VM in Azure, a server at home, and two laptops (wife and myself) along with a couple shared folders with family, and it's seamless. I keep mirrored backups on one of my VMs and my server at home, so nothing is going to get lost no matter what computer blows up, or if Dropbox blows up.
I know it is not a fancy or automated solution. But is should provide a cheap solution to most of the functions that the poster wants.
there's plenty of ways to get server power without going to amazon. I'm gradually becoming an AWS fan from using it at work, but still not for home use and managing your own server wouldn't be centrally located for the family nor worth the trouble. I and some fellow devs use Site5.com. Set up your own server starting at $25 a month. Basically, you get a spot in a data center to do as you please. Simpler than using AWS.
Crashplan is what you want. The free version allows you to cross-backup to the other boxes for free.
I'm currently doing this with my brother-in-law; he lives in Oregon and I'm in New Hampshire.
Buy a single high-capacity box and install it at one location and use that if you want an easily combined large storage unit. We used an HP Proliant N40L with 4 x3TB drives; the replacement N54L is even nicer.
I've used rsync to push and to pull so it is bi-directional.
The main difference between rsync and Unison is what happens when file X is altered at the local site AND at the remote site between a single sync interval.
With rsync, one of the altered files will be over-written by the alterations to that file at the other site.
Whether this is a problem or not depends upon your specific situation.
This may not be the cheapest solution, but it's what I'm planning to do...
You can install the client on your desktop, mobile, linux, and freebsd devices, too (you'll want a supported NAS, so something linux-based or FreeNAS or such, unfortunately nothing MIPS-based.
I'm primarily planning on just dropping something (probably a Drobo 5N) off at my mom's (she & my step-dad aren't tech savvy), and setting everything up for them, including backups of their PCs to the device, sharing of photos & videos, etc. And then setting up replication of critical, irreplaceable data between their place and mine. On my side, I already have a Synology DS1813+ that I need to set up, first.
The Synology stuff also has the capability to deal with IP Cameras for monitoring (though I suspect that if you just set up a share to dump images to, you won't need a license for it). For streaming, I believe both Drobo & Synology (as well as others) have media servers, so app-enabled TVs, BluRay players, and game consoles should be able to stream appropriate movies and music as needed.
I'm thinking the NAS, my laptop (which I'll also start backups to the NAS, likely NOT shared), my VPS, and my mobile devices (phone, tablet) will all sync, as will my wife's laptop & devices.
Yes, this is probably heavily overdone, but it also avoids putting private data on systems that I don't control, and avoids the commercial cloud providers at the same time. And, not surprisingly, I play a sysadmin for my day job, so this ties in nicely with that, too.
"The urge to save humanity is almost always a false front for the urge to rule." --H.L. Mencken
I've tried the rsync and Unison solutions. If you have all Windows boxes (2 PCs and 2 laptops for my family), the free Microsoft SyncToy is very user friendly to keep a Documents or a Pictures directory synchronized across multiple boxes.
I've been reading Slashdot since 2000, so going on 14 years now. But I'll be stopping next week in support of the boycott, and maybe after that, if the interface catastrophe called "Beta" goes live.
See you on Usenet at comp.misc where old school commenting is happening: no mods, no karma, no whitespace, and no advertising. Just a lot of old geeks with killfiles and a keyboard.
Uck fay Eta bay!
If this were Usenet, I'd killfile the lot of you.
I hope people don't mix up the beta you are mentioning and the beta on which I'm writing this now.
Is there something wrong with pogoplug? No one seems to have mentioned it yet. They are cheap, you add your own storage and backup to your personal cloud is automatic from android, iOS, macOS and windows (don't know about Linux). Each family household can host their own pogo plug and you can set up automatic sync between them. I've got one at my house and one at my mother's house. My family photos are synced both places and as soon as I take a pic on my phone it uploads to both sites within minutes, automatically.
The new Mac Pro, with a few Thunderbolt drives & Mavericks Server
Just setup your own stuff at home and vpn there. Its yours, its secure and as cheap as you like.
---- Booth was a patriot ----
Why restricting OwnCloud to an Amazon hosting?
Any shared hosting, preferably via an association that you can become part of (and control, and check its costs), will run OwnCloud perfectly well!
Here in Europe I'm running OwnCloud on All2All in Belgium; I'm pretty sure there are many such services in the US
(all2all.org)
Herve S.
you won't store your data on the NSA's hard drive, but your own
First, throw a few large disks in a spare PC at each location, install FreeNAS on a USB stick, create a ZFS filesystem. Now you can replicate snapshots between units. Rsync is there if you want it. Owncloud has a plugin you can install.
I prefer rogues to imbeciles because they sometimes take a rest.
You can see it here: http://smartos.org/
This OS allows for ZFS+DTrace+Zones+KVM
In the regards to the original poster, I would use this since the ZFS file system would protect against silent data corruption. And (Zones/KVM) would allow for Virtual Machines to be used as part of the cloud.
I'm actually thing of using this for my dad's photo collection.
I would say use Copy. It is like Dropbox but created by Barracuda. You get 15 GB for signing up, or 20 GB by using a referral like this https://copy.com?r=EOFh2o
Copy also has fair sharing, so the space is divided by the number of people sharing the folder. So if 2 people are sharing a 2GB folder they are each only charged with 1GB storage (unlike Dropbox which would charge each user with 2GB).
Their family plan allows you to have user accounts, and with a device in every location, you have a lot of duplicates if something does go down somewhere.
AeroFS is like Dropbox without the middleman; they control only credentials, but your content never touches their servers. It's free for up to three users, and $10/month per additional user.
BitTorrent Sync allows you to sync (2-ways) or backup (1-way) folders to PCs and devices (Windows, MacOS, Linux, iOS, Android) over the Internet. I'm using it to do backups for my family...
It doesn't encrypt by itself, but each OS has that option natively, and you can sync an already-encrypted (at the source) folder.
Setup is very easy: source adds the folder in BT Sync, generates a 1- or 2-way key, sends that over to the destination via email, sms, ...; destination creates a folder in the OS, than adds that folder to their BT sync and supplies the key.
You can do that for any number of folders.
The Cloud - because you don't care if your apps and data are up in the air.
missed the part about the hardware...
I'd go with a full-on PC, because it's only marginally more expensive than a NAS, and it doesn't suffer from a NAS's limitations and bugs, though it does require a bit more setup, especially because given the price of Windows licenses, you should probably go Linux.
You can find Atom motherboards with 4xSATA for $70. Add an enclosure, PSU, RAM, you're at $150 (HP sometimes have good deals on their ProLiant MicroServer). Then you need disks: add up all your data, multiply it by 2 for starters, and keep in ming you'll quickly be at x4.. a single 4TB disk leaves you with a lot more room for further expansions than 2x2TB, especially since performance is not a concern (so, no RAID).
Someone should buy an extra USB enclosure and fill it with disks for backups, to be done weekly and then disconnected and hidden away.
The Cloud - because you don't care if your apps and data are up in the air.
I like http://seafile.com/ a lot and it is also open source.
Having a setup with a number of Windows PCs (some family members cannot live without it due to DOCX files from school/work) as well as a number of Linux PCs and servers in three locations in two countries, I have set up a number of scripts to handle backups between the locations.
The PCs (i.e. laptop/desktop computers) have an icon for backing up to a remote server. This is done on demand via rsync started from scripts (bash or BAT files) to one of the three servers. The servers replicate internally to each other via cron jobs that start rsync (via a script) every hour. I could have set up automatic backups via cron on the PCs, but have chosen to do it on demand to save bandwidth.
This is the main backup of stored items such as multimedia.
For own-produced files like essays, stories, presentations, etc. I have set up a time-machine-like system using perl and the rsync --link-dest option to create a new directory structure if there are any changes since the previous version. The --link-dest option creates hard links to unchanged files, which means that it is essentially an incremental backup and yet preserves the structure of the source system for each new version. This setup has proven itself useful inasmuch that we can go back to previous versions of our files fairly easily, although that functionality has not yet been wrapped up in nice scripts or fancy graphics like Apple's product.
With a little tweaking, this setup can support encrypted storage in each location, making it possible to store private data without giving other family members access to the stored information.
Oh, and one last bit of information: The storage in each place is a NAS box with RAID.
I agree that first, you should decide what your goal is. Shared storage? Sync? (Sync is not the same as shared storage because you have multiple copies of the data.) Maybe a shared storage area and multiple private areas?
I did not recommend Amazon in part because OP said they wanted to do it themselves. AWS and S3 are not "do it yourself". They rely on 3rd-party servers.
I'd say that SSH can be the solution, all by itself. The simplest case is shared storage: just set up a server on a network connection, and allow SSH access. Then, on your end-user devices, install a file manager that can use SSH (SFTP). Voila! Your own "cloud storage", on your home server, accessible from anywhere on the internet. (Use good passwords, or use SSH certificates.)
You can also set up separate areas on a disk and give people access only to a certain area.
I use file managers that treat SFTP connections like just any other hard disk. It's simple, it's transparent.
Using SSH and a simple GUI file manager, I can access my server at home from my smartphone from anywhere in the world. And on my personal home machine, I have reverse SSH set up in a shell script. If I'm leaving home for a while, I just run the script to set up reverse SSH from the server and leave it running. Then I have access to both remotely. I can access either one from anywhere.
No "streaming" servers are necessary. No fancy sharing services. No 3rd parties at all, in fact. It's just a remotely-accessible hard disk.
I haven't found it necessary to do anything else. By default I have access to the server's big archive disk from anywhere. Turn on the Reverse SSH and I have that AND my desktop machine. Just as though they were local hard disks. Except for the transmission delays, of course.
AND, I can do it via wifi or wired internet (somebody else's machine), as long as they have an SFTP-capable file mgr. program, OR I can do it through my cell phone if I have remote data enabled on the phone. (If you're on Android, try ES File Manager or, better in my opinion, Total Commander. TC is also available for other OSes.)
Google
gmail, then drive, then drive synch'ed to the desktop:-) Not sure if it scales to what you need, but it works very well for me.
There's this program called bitsync: http://getsync.com/ which you each have a client on your computers. Share hashes with each other, and you get a distributed, synced in real time copy on each client running the software. It's free, secure, and no servers required. You each have a copy locally, and any modifications are replicated to each client.
"Freedom in the USA is not the ability to do what you want. It is the ability to stop others from doing what THEY want"
You now have the following capabilities:
I've used this setup for a year now w/o a problem...
P.S. Owncloud is unstable for large files.
P.P.S. Bittorrent sync works great...if you want to trust a closed-source application with your private data.
They mentioned family photos. There are two services that are virtually free at the moment, which makes it hard to beat with a private cloud.
Yes, Google+ photos have a 15GB cap on full-resolution photos in the free tier, but no cap on "web-resolution" photos. It's simple to upload from Picasa from Win/Mac/Linux, and of course happens automatically on most Android devices. Yeah, it won't be archival quality, but good enough to record and share the "so this happened" moments.
For all of the huge archives of digital negatives and source video content, it's nice to be able to have an offsite backup. If you don't have terabytes of storage on a friend's system, Amazon Glacier is probably the most cost-effective way to insure you have at least one place to turn to to retrieve your files. The cost structure is complex, but basically boils down to 1 cent / GB / month, and maybe a retrieval fee between 0 to 5 cents per GB depending upon how quickly you try to retrieve it all. Not bad for an insurance policy for a couple dozen GB of photos, though for 100s of GBs of videos you may want to think twice.
It took me a while to find a good straightforward Amazon Glacier upload utility, but the Java-based SAGU ( http://simpleglacieruploader.b... ) does the trick nicely. I sort my photos by month, so every year I make a big tgz and upload that big file (optionally encrypt with gnupg or something if you want, though I personally am more paranoid about not being able to get to my data than the Feds or someone doing something with my kids' baby pics). Glacier is based on a robotic tape library, so it is cheap to upload, but expensive to pull data, even the list of what you have stored. So save all of the index data for every file you upload to one or more other cloud or email systems (just not on the computer you're backing up from), so you can retrieve those archives in the future as your last resort.
Check tresorit tresorit.com, dead simple as dropbox, files are encrypted and key is not known by the operator nor NSA, supports groups and other interesting stuff related rights and sharing, backend is some cloud storage.
Plug computers do nicely for family storage. I have use pogoplug, and it's ok (not stellar). You hook a large US B hard drive to it for virtually all the storage you want. For instance, you can use an external enclosure that can hold four 3TB drives for a total of 12TB. The pogoplug has 4 USB ports on it, so you could potentially hook up 48TB. Pogoplugs can also sync with each other no matter where they are. So you and your family members can back up each other's files automatically. It's a pretty good way for everyone in the family to have offsite backups.
My big beef with pogoplugs is that they are slow. But what they do, they do well. And the customer support is really good if you have problems.
There's also tonidoplug, which I haven't tried but it's gotten good reviews.
Unison is only an option in a homogeneous environment, because both ends must run the same version, and you will have the devil's own time actually getting the same current version running on multiple platforms — especially if any of them are Windows. That doesn't make it useless, but it does make it a PITA. This, frankly, is a horrible design. It should be capabilities-based rather than version-based.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Based on Gluster
As the name implies, CloudFS is a filesystem for the cloud. What does that mean? First, it means that it’s a filesystem, with the behaviors that people – and programs – expect of filesystems, and not some completely different set of behaviors characteristic of a database or blob store or something else. Here are some examples:
You access data in a filesystem by mounting it and issuing a familiar set of open/close/read/write calls so that every language and library and program under the sun that can use a filesystem can use this one.
Files are arranged into directories which can be nested arbitrarily, without requiring the user to establish and follow some separate convention on top of a single-level hierarchy.
The data model is of a byte stream – not blocks, not records or rows – in which reads and writes can be done at any offset for any length.
Performance and consistency for small writes in large files are not reduced to near zero by doing read/modify/write on whole files.
Files and directories have owners, permissions, and other information associated with them besides their contents.
Owncloud isn't really distributed. I would recommend aerofs and/or btsync.
BTSYNC. nuff said.
the easiest cheapest way is just to get large usb drives, hook them to a computer with an operating system you already understand and mirror the drives. Since you all live in different areas the chance of all the drives failing simultaneously is pretty slim.
Local drive for one family member but appears local for all...1tb of storage.
Remotely backed up to other space monkey users.
Think Dropbox + crashplan love child.
An app leveraging MaidSafe.net would be perfect.
It's not __quite__ there yet (needs a bit more time to get fully stable on Windows), but Git Annex is designed for this job and if you use direct mode it works wonderfully well. It automatically moves binaries around between repositories and because it's Git based you can get any file that was stored in the repository, at any time.
Plus, the more you distribute the data, the more backups you have. I'd hate to loose gigabytes of family photos to a single hard drive failing. There are a few companies, like pogoplug, producing consumer friendly storage devices that are inexpensive ($20) and hide all the complex software pieces behind a nice web interface.
Also, I plan on making a slashdot-inspired site. The website will be called "Pipedot" and be reachable at pipedot.org or pipedot.com. The motto will be "News for nerds, without the corporate slant." Get it? A pipe character looks like a slash without the slant!
I'm committed to providing a free and independent resource without the influence of a parent corporate overlord. The site will be decidedly non-profit and have zero advertisements, zero adobe flash, zero google analytics, etc...
As a special bonus, the site can now be reached at fuckbeta.org! Stay tuned for more updates...
Get a network server in your home with a BD-ROM backup and 1 giga/sec Ethernet and Wi-Fi (secured). Start with two folders, Important and Trival, all else under these two. Set Important to backup (incrementally) daily while you sleep. Doing a full backup once a month and store off-site (like a safety deposit box). You in control.
Try a Synology NAS.
It does everything you are asking about. EVERYTHING.
"A learning experience is one of those things that says, 'You know that thing you just did? Don't do that.'" - DNA
The canned PR statement from Dice.com appeared at the top of this thread before the article on Distributed Storage for Families. It does not include a means for reply and it contains all of the corporate newsspeak that you expect from a group of people who have already made up their minds and will change the interface in spite of us. I have used Slashdot for under a year, as a refugee from social media and blogging, and I will drop Slashdot if the beta interface is adoped and I urge everyone to do so and to continue to bad mouth Dice and social media and blogging generally.
So, why is the Beta so bad? If you have been a user of blogs all your compute tile, or if all you know is social media, you will not be able to understand this. Blogging, the social media model, Facebook, Google (Google Groups, Google+) thwart the kind of discussions we are able to have on Slashdot and Reddit and few other Internet venues. There is good reason for this. it is so the business intelligence types can have a simple means to generate the Big Data information gathering job marketers want. A vibrant public dialogue possible on the Internet is being sacrificed for profit. Dice is expected to be no different in that they are probably trying to find professional and career information about the users of Slashdot. The Beta represents just the sort of change that this priority would represent. it is the same "Simplify" imparitive the Mark Zuckerburg defined for Facebook and is why Facebook and Goggle suck and why if the Beta is adoped here, it will sucj, and Dice is this big corporation located in New Yoirk City that does not care. It is listening to the marketers and the propagandists in corporate and politiclal America who are out to undermine democratic institutions by first using blogs and social media as ways to disrupt public discussion and debate. Capitalists are not usiversally pro-democracy, and the trends in social media reveak that. Resist the migration to Beta and leave Slashdot if it is adoped, or create an alternative.
Fuck Beta!
Use Lima and own your own data.
Check out these guys: https://www.noobaa.com/welcome
Might be the perfect fit for sharing with a family.
I have to agree with the suggestion for Bittorrent Sync, it's super easy, will mirror between sites, and runs in a really small process.
It is still in beta but Symform (http(colon)//www(dot)symform(dot)com/peer-to-peer-backup/)has a solution that might meet most of your requirements. It is a shredded, encrypted, distributed P2P backup system. I'm not clear from the documentation how well collaboration via the tool works, but they mention using it to sync devices so it should be possible. They even have a "donate space" option to expand your GB in the cloud.
Coincidentally, I just wrote about this today. There are many hosting companies who offer unlimited storage plus unlimited domain hosting/email/etc. Costs are probably less than one hundred bucks with no hardware expenses. Everyone would use FTP, they'd have their own folder with a password, etc. If they are all fairly tech savvy then FTP is a breeze. Have one or two people control the account using the CPanel and make regular backups. Your material is as secure as the hosting company plus you could use your own encryption, if necessary.
I use bit-torrent Sync to keep our families files in 2 or more places. You can set up a round robin, or simply 1 to 1 with each given family member using different directories. I use it in conjunction with snapshots on my filer.
http://gluster.org/community/d...