Slashdot Mirror
Ask Slashdot: Building a Personal FOSS Cloud?
An anonymous reader writes "Cloud-based personal data management is pretty cool... if you don't mind entrusting the entirety of your personal data to a gigantic corporation. Apart from the risks of their doing unseemly things with your data, also the security of your data is entirely in their unreliable hands. So, is it possible to build my own personal data repository, where for example, I can store my contacts and calendars to sync to multiple devices? This could be hosted on any third party hosting service assuming also that all of my data was encrypted at the data level. So even if the host wanted to look at my data, all they'd see is 1s and 0s. What are the options for the tinfoil hat wearing FOSS folks that want to participate in the cloud age?"
So even if the host wanted to look at my data, all they'd see is 1s and 0s.
That was the dumbest thing I read all day.
http://owncloud.org/
- Calendar
- Contacts
- dropbox like storage
EA David Gardner -"... but the consumers have proven that actually what they want is fun."
At what point does this involve a cloud? Renting a server(providing ftp, for example) is easy, and doesn't require anything from the "cloud age".
Also, building a server or buying one secondhand is cheap, if you want to DIY.
while(1) attack(People.Sandy);
You can write "The Cloud" on it with a Sharpie if you absolutely must.
You could check out ownCloud. According to the website it supports encryption. Remember, though, that even if data is stored in encrypted form on disk, the hosting service could recover your data by monitoring your requests to the service. If that is a concern then you'll have to host it on a machine in your basement.
https://github.com/wurp/Friendly-Backup
It works now, with some bugs. The first targeted usecase is distributed backup.
However, it can store arbitrary read-only content-addressed data as well as signed labels that point point to a particular piece of CBA data to emulate mutable data.
I have a whole slew of plans beyond backup for it, but backup seemed like the thing everyone needs and would most like to have for free on a federated data store.
OMFG, the cloud. I got to have or do the cloud. Magic Ponies in the cloud!!!!
Seriously, wtf do you really need the cloud for? Is it going to magically sync all your different data together so you can access it all the time?
No, seriously, do you think it's going to sync all your data so you can use it and access it anywhere?
No, it's not. Sure, you can access you data anywhere, but duder, we've been doing that for a couple of decades now, way to join the late train.
Unfortunately, the various corporations don't want to agree to standards, so having docs/apps/whatever working with everything isn't in the "rape as much money as we can" business plan. so nothing is going to change.
Now let's look at the Megaupload thingy. That was cloud storage, file lockers. It's not around now, is it? That is what happens to clouds, the winds blow them away. The wind? Oh ya, in this case, that's the good old USA Government, working for their Pimps, the Music/Movie Industry. You think that can't happen to any "cloud" servers? Think again. OMG, Terrorist used that server, Child porn was on that server, boom! You're data, which has nothing to do with those 2 things, is gone also. Hope you make a backup. Oh, wait, the cloud was magically supposed to back it up for you?
Cloud has been around for awhile, but we called it what it was, the internet.
Be seeing you...
What's in the cloud that is better?
slashdot ate my last comment, so just check out the link
coding is life
the safest storage is your own high speed server quality RAID 7 write-only drive
...omphaloskepsis often...
You want the above? That's easy. Access to email from anywhere, access to my contacts and my calendar, how about access to all my files? Yep got that. Though it doesn't have a fancy name like "cloud". If I were into marketing I'd call it a cloud, but right now I'll stick to calling it an "internet facing linux machine"
Yeah it's not as exciting, but it does everything the so called cloud has done and it has done it for many years before this mythical cloud has existed. My phone sees the same set of files and emails as my home desktop PC, and there's a web interface to access all the above too.
Seriously just google "Linux Groupware" and maybe "Linux Web Fileserver" and you'll have everything that the cloud has.
http://sourceforge.net/projects/funambol/
That and a server you can ssh into.
It was surprising to see the first ten or twenty posts criticizing the poster's question. Some respondents were sick of "the cloud." Others thought the question was lame.
This is a good question. While the world is moving toward "cloud"-based storage and applications, we trust unknown third parties with our information. Would it be too much to ask that we have control of our personal data while making use of the benefits that cloud has to offer? I don't think it's too much to ask, but I don't see a solution that solves the whole problem.
To the original poster - this is a work in progress, and you will see incremental steps to address your needs. At the same time, you will see that application providers continue to build solutions to trap you in their environments. Keep trying, and find a solution that works well enough while we get this right.
So do you propose a free cloud solution for starving kids in Africa? The food thingy might be a bit difficult to accomplish but if they can store their music safely at least the on line world is doing their bit to change the world.
Try the free open-source SparkleShare software and roll your your own cloud 100%. That would trump any cloud provider option if this is your concern, since all the disks and PCs are under your ownership and control.
SparkleShare is essentially a DropBox clone in terms of a GUI, which extends to recovering older versions with a right-click. It looks like DropBox, and it works like DropBox too. But it is just a scripted GIT environment. In fact if you already have a GIT Repo hosted on a server (or service) somewhere, SparkleShare is easily configured to wrk with it. Here's how you start from scratch, assuming you already have PGP keys shared with the server:
At the server, create a new, empty GIT repository:
git init --bare NEWREPOSITORY.git
At the workstation:
Normally, you might use something like the following commands to work with GIT. (these are not necessary if you use SparkleShare)
git clone ssh://user@example.com:port/home/user/NEWREPOSITORY.git
cd NEWREPOSITORY.git
git clone ssh://user@example.com:port/home/user/NEWREPOSITORY.git
The SparkleShare config:
Add Hosted Project...
Address:
ssh://user@example.com:port
Remote Path: /home/user/NEWREPOSITORY.git
This document explains how to add a layer of encryption, (which also works to secure services like DropBox btw: https://github.com/hbons/SparkleShare/wiki/Encrypting-your-files-before-transfer
You can't be ahead of the curve, if you're stuck in a loop.
I did misread this. When I think cloud computing, I am coming for a CS point of view, which is that cloud computing is the terms used to describe the efforts to make scalability of software as a service ubiquitous. Basically, the cloud is not a bunch of servers, it is the infrastructure that provides scalable services to an application layer like the web. Amazon pretty much built the best cloud and others are following their lead. So, I have been looking at OpenStack
If anyone actually thinks this question is in any way relevant, please let me know if there are other resources.
pull it out fo your phone and plop it into another device to import? If you're gonna pull all this retarded effort into the "cloud" why not just set up VNC and log into your computer at home and grab the contacts? You know something thats been available for over a decade.
by TheSpoom (715771) Uncaring Linux user here. I have nothing to add to this but please continue. *munches popcorn*
You save their music by providing food and shelter. You know, in some places people still listen to live music.
Easiest way of making sure your data is secure and still available everywhere is to store it on a encrypted filesystem image (gpg + loopback image) and copy it to many public cloud storage providers to ensure its redundant. Just make sure they are not all backed by the same infrastructure.
*snort* 27 posts so far and no one seems to really have addressed the poster's real question. (Instead, all I've read is basic suggestions like a file share, VNC/SSH, or OpenStack; all of which seem to ignore the main point: "is it possible to build my own personal data repository, where for example, I can store my contacts and calendars to sync to multiple devices?")
I've been looking for something like this for a while now, actually. From my research, I think the best way to solve this problem is to set up your own 'groupware' server on a hosted VM somewhere. You can custom-configure the VM to make sure that it stores your server-side data in an encrypted filesystem within the VM itself. [To make it that much harder for anyone from your hosting company to spy on you, naturally... ;-) ]
Then, you can use the open-source sync clients from the "Funambol" project to synchronize the contacts and calendar data on the phone with the data on the groupware server. The issue I've had is that I *also* want a non-shitty *Web* interface for calendar management... and so far, *that* has been hard to find. (I can't bring my personal smartphone into work, so I need something to be able to manage my calendar over the Internet and sync those appointments back to my phone).
So what server to use? Well, I set up an eGroupWare server a few years ago (before all this shit was called "cloud" everything :-P) and it seemed to have most of the features I wanted as far as calendar management goes. [I even locked everything down, moving the back-end database to an encrypted filesystem that wasn't auto-mounted...] Unfortunately, the default web interface kinda sucked. And the good Funambol 'web' client is only available on their own 3rd-party calendar hosting servers, which I wouldn't use because I wouldn't get to control my own data. (Again, the project only ships with a crappy text-based one out of the box :-P) So I stopped using that solution. Consequently, I never actually got all the way to the point of trying out the PalmOS(which I was using at the time)/Android/iOS Funambol clients to see how well they worked to synchronize contacts and calendar data.
Recently, I've been looking at SOGo, another open-source groupware server which apparently has a fancy Ajax-based web UI... and should also work with the Funambol open-source sync clients for all the major mobile OS devices. I haven't set it up yet, though.
Incidentally, I'd be *very* interested to hear from anyone else who's attempted to set up similar solutions about your problems and successes. Has anyone else actually tried this?
http://cloudi.org for BSD license open source project to avoid virtualization but receive fault-tolerance and scalability (along with efficiency). Includes integration with C/C++, Java, Python, Ruby, and Erlang along with various databases (PostgreSQL, MySQL, memcached, couchdb, tokyotyrant) and ZeroMQ.
What if the FOSS crowd could create a customized sort of BitTorrent thing, where you encrypt your data and then put it out as a seed. Others just automatically download it, then seed it as well, until a distributed tracking system sees enough full copies running around that it tells other clients there's no need to pull it anymore.
The trackers share information about your data, which is encrypted and named some gibberish that makes no sense but won't be likely to be duplicated. So long as a minimum number of copies are out there, it doesn't tell any more to download and seed. If total good copies go down, then it triggers reseeding again until there's enough copies.
Then you just need to know your filename(s) and encryption key(s) to get your data back. Probably not as convenient as plain old cloud storage, but could work for archival store.
Plus, if you needed to transfer files anywhere in the world, all the recipient would need is your encryption key and filename.
It runs linux, you can ssh into and install or compile whatever you want, comes in upto 4 gigs and i think they just got a dual drive one. Use the built in internet access to the twonky server or install some free cloud software.
Don't complain about syntax, grammar, or spelling. There is no.hell like input on android.
My Personal Cloud.
Shai Schticks:"You don't make peace with friends, you make peace with enemies"
Grab an old box, stick some hard drives in it with some sort of RAID, encrypt the partitions and use rsync or similar for backing things up. Want extra redundancy? Use a USB drive or buy a cheap old tape drive off ebay.
Forward SSH to it and you have "Cloud Storage". This really isn't a new concept.
Reading the parent post feels like wiping your ass with silk.
Re OP: If you really want a FOSS personal cloud, you might be interested in knowing that Facebook documents its whole platform.
Also, unless cloud got dumbed down to the point where hosting anything on a single server is "being in the cloud", you might find it easier to build a fucking toaster.
The problem with setting up a cloud, is that to get a reliable configuration you can't use less than two banks of 3 machines
1 Load Balancer/reverse-proxy or front-end, this can NOT be a VPS
1 Web Server (you can add more as necessary, these can be VPS (mySQL, memcache, etc can all be their own VPS instances)
1 Data store, absolutely NOT VPS.
Commercial/Openstack includes a 4th machine that simply operates as a boot image store/management for configuring the VPS machines.
Then you need to duplicate this bank somewhere else, like another physical location on the other side of the planet.
This is where things start to break down, because to keep the two data store machines in sync, you'll burn a fortune in bandwidth. If you're just doing things like contact storage, cloud storage is so extremely overkill.
But for web servers, you don't have any redundancy if all the hardware is in the same data center, even if you have redundant machines. You're looking at maybe 12 machines minimum to have zero possibility of fail. Then you need to ad additional edge nodes (basically LB/Reverse proxies) for where you want the data available without latency. So Europe, Australia, etc.
By the time you price this out, you've probably have a bill of materials around 100,000$ without even considering bandwidth and power to build a completely secure, disaster-proof (short of a EMP burst knocking out everything on the planet) cloud configuration. If you're just putting contacts in the cloud, this is overkill times 2000. 1200$/mo ... yeah I think this is not for you.
If you just want a way to keep your contacts with you everywhere there is internet access, pay for separate physical machine, of the lowest possible configuration (Atom, etc) and that will run you maybe 100$/mo and then make sure that it's in a country that is not run or owned by US companies, so this eliminates all the big companies like Peer1, Equinix, Telehouse, Cogent, Level3, etc, who won't let you put a machine in their center for less than 1200$/mo anyway.
When you introduce a sentence with 'apart from' you don't put 'also' in the second clause. It's implied already.
Fucking porch monkeys.
Remember back in the day when a network - any network (you LAN, corporate WAN, internet) - was represented by drawing a picture of a cloud? (This is still the case).
That is "the cloud." Access a server by first traversing that cloud in you network diagram... that server is now "in the cloud."
Services such as Amazon EC2, software such as Hadoop - these are all various entities that are accessible via the cloud.
Christ that is it.
http://code.google.com/p/s3fs/ to mount Amazon S3 storage as a directory on your computer. (FUSE based)
Then use LUKS for encryption: http://forum.xda-developers.com/wiki/Guide:Setup_Encrypted_Folder_with_LUKS
Then if even Amazon's redundancy isn't good enough for you, why not run RAID?
Maybe those fuckers should refrain from sticking their dicks in places where they shoudn't be.
What is the cost of a roll-your-own cloud solution? Most discussions about the cloud miss out on the most important element, which is the cost. People use Google because it is essentially free, and gives you very decent reliability. I know you can make your own home server super reliable, but in aggregate, if 1 million people were running their own servers, compared to 1 million on google, I would bet that the 1 million on Google's cloud would do better on uptime in aggregate. The cost of trying to get to Google levels of reliability is quite steep, and fairly prohibitive for all but the few hardcore geeks who are comfortable managing their own servers, and even then, only if they pretend their time is worthless.
Run your own fail safe data repository. Companies have been doing this for ages and it isn't that hard nor expensive to implement it at a smaller scale for your own needs. No cloud needed;-)
Just use rsync, and something similar to mysqldump and mysql replication along with 2-4 linux nodes ideally hosted on different network/providers. You can host the nodes in VMs connected to regular consumer grade DSL or cable modem connections. You could make peering agreements with friends and relatives, I host your node you host my node.
Optionally, throw in some DynDNS or alike, or better, run your own dyndns and you are pretty much done. if you do not want to run your own dns, you can also have the nodes publish their IPs on some free website hosting site. Machines can also find each other IP by exchanging emails through a third party provider like gmail.
Everything I write is lies, read between the lines.
Check out http://github.com/ohler55/orefs. It is still under development but provides encrypted remote storage if you are comfortable with a command line.
Yea, those poverty-stricken, starving kids in Africa should keep their aircraft carriers, long-range bomber aircraft and unmanned drone fleets in their own fucking country!
most of us nerds have been doing cloud computing with our own *NIX on x86 boxes for years, running home servers with lamp + SSH.
then there is this pogo plug thingy which does the same thing but for newbs who don't want to do the setup, and for cheap.
I think he was talking about their population/resource imbalance.
"When information is power, privacy is freedom" - Jah-Wren Ryel
Doesn't this question get asked here like every other week now?
---- Booth was a patriot ----
Thanks, I'll add that to the list of definitions of "cloud" that I have heard from computer scientists. "Cloud computing" is an undefined term; at this point, people use it to mean whatever they want. Scalable infrastructure, computation as a utility, storing files on a server, whatever, it's all cloud at this point.
Palm trees and 8
I use B-Folders and a PogoPlug with a usb hardrive attached. B-folders stores my sensitive data and contacts encrypted on my mobile device, and the PogoPlug doesn't require a static IP.
C'mon troll, can't you get a little more creative than just copy-pasting text with a phrase substituted? See here for a diatribe about hotdogs saving African children.
I must admit that the "hotdog mine" (and the subsequent commenters) was pretty funny.
If you actually want to "build a cloud", look at oVirt frontended with Foreman for a web interface. Create generic enough puppet modules that users can configure their servers by adding a handful of parameters and simply pointing and clicking to add classes. Back it with NFS, iSCSI, or Gluster storage. Build a cloud using all open source software. But I think you meant something sillier, so you probably want to ignore me.
Using the wikipedia definition:
"Cloud computing is the delivery of computing and storage capacity [1] as a service [2] to a community of end-recipients.".
The whole point of a cloud is to abstract a massive underlying infrastructure to deliver some type of computing service (PaaS, IaaS, SaaS, etc ad naseum) to a large group of users and to be able to scale that infrastructure seamlessly. A "personal cloud" is an oxymoron.
Granted it's not FOSS but it free as in beer and has ios and android app support. Not to mention you can run it on an inexpensive Tonido plug.
As a lawyer representing RIAA, I would like to announce our recent patent on live music. If you wish to sing a song you must either pay a licensing fee of $1,000 per tune, or cease and desist on penalty of being incarcerated in one of the private prisons we own and operate for profit on behalf of the government, thanks to the greased palms of some greedy Senators and Representatives. Our spies are everywhere, so just make sure that when you hum a tune or sing a song you have paid your licensing fee for that song, or else it must be atonal, random noise that does not infringe on RIAA's patent.
In a word ... Citadel. (Disclaimer: I am a developer on this project, and yes, I'm flogging it here.) Contacts, calendars, notes, documents, email, etc etc. One single installation without a zillion dependencies.
Tired of FB/Google censorship? Visit UNCENSORED!
I was thinking along the lines of going after public enemy #1, "Drought", by using things like Renewable Resouce Powered Water Purification devices.
So I wonder, "Why can't the U.S.Government send a billion dollars of aid to Texas in the form of Wind Turbines, and Solar Farms to power Desalination Plants?"
Maybe the troll could submit a comment about giving Soylent Green to the children?
Quit using buzzwords you don't understand!
Guys... guys... If you look closely, you can see that the text is auto-generated with "FOSS cloud" being the recurring variable in the whole post
I can image someone has a function in a script somewhere that says TrollThis($topic) or something...
Just don't react to it.
If you're more of the DIY type, like myself, I'd suggest building your own from scratch. Remus is an excellent choice for a high-availability environment. Admittedly, it's still a relatively young project, but as of Xen 4.2 (currently the unstable branch), it's been largely stable and easy to work with. You can even use DRBD as the storage backend (currently it's using a modified DRBD with a new "protocol D" synchronization method, but prot D is going to be integrated into the main DRBD branch as of DRBD 9, hopefully later this year).
Basically, you set up a normal Xen virtualization environment, but you mirror the configuration across two (or more) Xen nodes. You have two storage nodes sharing out virtual block devices (AoE is good if you'll have all the nodes on a single switch, iSCSI is good if you'll need to route over an IP network), one Xen node connects to one or more storage nodes, then DRBD running on each Xen node joins the block devices into a DRBD volume (I like to share out the whole disk/array of each node as a single block device, then create an LVM2 logical volume on each block device, and join the LVs together with DRBD). Xen then uses the DRBD volumes as virtual disks for the VMs. Once that's set up, you just configure remus to start when the VM starts, and it will checkpoint the current machine state between the two Xen nodes at a rate you specify. If one node goes down, the other picks up running the VM, without dropping a packet.
I've currently converted all our internal systems at work over to running on this cluster, and it works great. Highly recommend you take a look.
Now I am intregued, a chat bot that can synthesise a paragraph! I wish the source were sited.
Is Askemos for you?
Don't worry. I'm sure it is located at a site somewhere.
(The word you're looking for is "cited".)
:(
I wish Linus would take a few weeks off to write a distributed backup system, but he just uses public FTP servers...
Of course, there're several projects that use git as a backend, like http://www.kickstarter.com/projects/joeyh/git-annex-assistant-like-dropbox-but-with-your-own (already funded; he's also a Debian Developer).
Since git isn't a backup system, using it as one isn't as efficient as it could be, but it is powerful. Joey's project is an exciting potential Dropbox replacement. He knows what he's doing.
Obnam is also exciting: http://liw.fi/obnam/
Anyway, sorry, dude, I have had enough of Java VMs sucking up memory whether they use it or not, and taking a LONG time to start. One or two of those and you can't use the machine for much else. I wish people would leave Java for enterprise uses, if it's even good for that.
Of course you can do whatever you want. I'm just giving some feedback, because if you want users, I know there are many people who feel the same way I do.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
This is what we are all waiting for, and it's already been funded! Just a matter of time until Joey finishes it: http://www.kickstarter.com/projects/joeyh/git-annex-assistant-like-dropbox-but-with-your-own
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
So do you propose a free cloud solution for starving kids in Africa?
Greetings from a least developed country;
Your words are so true! The absurdity of it all! After all, nobody ever used technology to improve their standard of living.
P.S. In case the sarcasm has escaped you so far: Fuck You.
Hugs,
The rest of the world.
Crumb's Corollary: Never bring a knife to a bun fight.
Cloud is a nice term for "magic".
Its been a long time coming but we haven't been able to translate our expectations into ideas that could be realized until recently.
We think more and more in terms of what we are familiar with; Contacts Items, Calendar Items, Mail Items, File Items, Photo Items, Video Items - the shared common language hasn't existed until recently. Even their common formatting has been lacking.
Now we're sitting down to negotiate terms; of ownership, of privacy, of localization, of security, of accessibility (HA), of recoverability (Backup)
It won't be "baked" for a few more generations of terms, but its far easier to define what a person means by a Cloud these days.
The amusing thing are the "branded" Clouds, like - Apple, Microsoft, Google, Dropbox, Amazon waging for consumer attention.
At a guess most people and companies want to think of the cloud as a personal avatar in the Internet that represents everything about a person or corporation that is more standardized and accessible than the real thing. It has all of the above and can inhabit any device we physically own in the real world possessing it like the spirit or essence of the owner.
Personal or Corporate websites were/are like this, they're calling cards, just more interactive - but lacked in the features they provided their owners, rarely did we upload our contacts or personal data files.
Personal news aggregation sites with their posting accounts were the next stage.
Social befriending sites the next stage.
And now we're looking for a combination that includes all of these and our files, apps and legacy media uploaded to the cloud.
Near or Far, which is safer?
Local disaster or Global catastrophe, which is more likely.
Feels like standing on the edge of a diving board, watching all the other divers on diving boards, to see if they strike high or low; which end off the pool shall it be?