Ask Slashdot: Linux Mountable Storage Pool For All the Cloud Systems?

An anonymous reader writes "Many cloud systems are available on the market like: dropbox, google, sugar sync, or your local internet provider, that offer some free gigabytes of storage. Is there anything out there which can combine the storage into one usable folder (preferably linux mountable) and encrypt the data stored in the cloud? The basic idea would be to create one file per cloud used as a block device. Then combine all of them using a software raid (redundancy etc) with cryptFS on top. Have you heard of anything which can do that or what can be used to build upon?"

Comments

There are several options here

[Omnifarious]

The first, and most interesting, is Tahoe LAFS. It does come with a FUSE driver, so it can be mounted like a regular filesystem. It is cloud-based and redundant to a degree you choose yourself. All copies stored are encrypted, so the only person who can read them is you. I'm not sure though if fetching from more nodes than you strictly need to reconstruct your original file actually buys you anything with that system, but I think it does.

You could also use something like a mountable version of Google Drive and then layer fuse-encfs on top of it. That's not quite as secure as encrypting at the block layer. The overall shape of your directory hierarchy is available, even if the individual file names and their contents are obscured. That should probably be good enough for most purposes.

Re:There are several options here

[Omnifarious]

Tahoe sort of achieves this in an odd way. Directories contain hashes of the file they reference instead of an inode number. This means that a Tahoe node often doesn't even know who a file really belongs to, even though it knows its length.

The main issue with block storage is this...

Suppose you modify a data section of a file in a btrfs filesystem mounted on some kind of weird encrypted block device. There will be a whole tree of blocks that get modified, all the way up to the root node. All of these blocks have to be written before the root block is, and for a small file there will be several more blocks that need updating than there are data blocks on the file.

These two issues create a big synchronization problem and a lot of extra traffic.

In contrast, a good distributed filesystem protocol that's aware of individual files can send a single message that contains some kind of identifier for the file, and the new data it should contain. This message will often be smaller than a single filesystem block, and it will also usually be compressed before it gets on the wire. Much more efficient and while there are synchronization issues between updates to individual files, within a file there aren't any.

Don't trust the cloud

[Anonymous Coward]

My residential internet connection via Comcast is fast enough today that I can pull files off of my server at home, "cloud" style.

I have two 2TB drives in RAID1, encrypted with whatever magic `cryptsetup' performs, with port 22 of my firewall forwarded to the server. SSH only accepts logins from me. I consider my data to be more secure and easier to access (it's literally seconds away from availability on any real operating system anywhere with internet access. Windows need not apply) than anything I could get from ZOMG TEH CLOUD. Only disadvantage is speed. I'm not gonna be shunting gigabyte plus files around like this.

Added bonus: easy to add users, easy to throw up a web interface, can do whatever you want with it, since you own the hardware (!!)

Pfft, cloud. I remember when it was called 'the internet'.

Now get the fuck off my lawn.

Why do this ?

[Alain Williams]

He has not said why he wants to do this, ie what problem he is trying to solve. Depending on the question the answer may be different. Does he want a cloud because:

* data must be available from many places - ie over the Internet ?

* data is to be safe from one place (ie home/office machine) blowing up and losing everything ?

* fast access is needed from many places at once ?

Please first answer these questions so that we may provide you with what you need rather than random solutions that may not be what you need.

Re:Why do you want to combine them?

[WalrusSlayer]

Uh, methinks you haven't really used tool chains designed to maximize the value of RAW files. The camera's built-in processor does way the hell more stuff than just compress raw pixels into JPEG. White balance is a huge one, along with level curves, sharpening, and a bunch of other stuff. Much of it either one-way or very hard to unwind. And as others have pointed out, most RAW *is* compressed, just lossless.

So yeah, you can fix white-balance in a JPEG, but it's way simpler and more accurate to set the white balance if the pixels haven't already been misbalanced in the first place. Ditto for exposure. Most tools that deal with processed JPEG's don't even have an exposure adjustment---quite often the same tool that does both file types will have an exposure slide if it's RAW but not if it's JPEG. Sure, you can futz with brightness, contrast, levels, gamma, etc to correct an under-exposed shot. But sliding over to +2/3 for a slight underexposure is one click and you're done.

As a guy who has deep-drilled many a software engineering discipline in his 25 year career, and shot tens of thousands of frames as an amateur enthusiast, you can pull me out of the "photographers who don't understand the tools" pool thank you very much.

I have gone back and forth between JPEG and RAW over the years. There have been periods where, with two small children, I simply didn't have time to invest in RAW processing. And I was pleased the neutrality of the DSLR's processing anyway. Other times I knew I was shooting in challenging conditions, and set the camera to RAW+JPEG as a safety net. I've rescued many a shot that way. Recently I've been putting mileage on Lightroom and can extract an immense improvement out of the RAW's that would take me 4x the time to do if they were JPEG, and probably not end up with the same result. I now have more time to invest and the payoff is real and significant.