Organizing Data Across a Heterogeneous Net?

angst_ridden_hipster asks: "Like many people, I have a bunch of machines I use regularly. These include Linux machines, BSD machines, a Mac OS X machine, and a Windows machine. These machines are on a number of networks. All have internet connectivity. Some of them are always powered on. A few of them are not. Obviously, I have a bunch of accounts. And, it goes without saying, I have a bunch of data. What are the best approaches to sharing data? I want to be able to securely access my home data while at work, and from one machine to another, etc. Opening ssh terminals is the approach I have traditionally used, but I'm beginning to wonder if some mirroring software (e.g., Unison) might be in order. It'd provide the function of backups, as well as guaranteeing availability. Would it be wiser to tunnel nfs over ssh? Or is there some better option? Assuming I actually start mirroring data across multiple machines, I'll need to organize it in a portable taxonomy. This is almost easy, since I use cygwin on the Windows machines, so I can assume a standard Unix-ish directory structure. But this gets more complicated when there are scripts or other code involved. What about application/platform-specific data? How do other people organize their data, anyway? Are there any useful standards? I'm hoping people will describe their approaches, and why they think they're (not) the best."

Comments

AFS?

[alsta]

IBM has released Transarc's AFS as OpenAFS (http://www.openafs.org). Don't know if that is what you're looking for, but it is pretty nice. It's also portable, so it runs on various unices as well as Windows. Most can be found as binaries if you don't want to roll your own.

AFS is an NFS style implementation though, so you would have to save your files onto a special mount.

It's called a server

[FozzTexx]

What you need is something known as a "server." A server is where you can store all your files, and in some cases, account information.

With the right kind of server, it can do AppleShare, NFS, and SMB, allowing all your other machines to mount the shares and make them appear as local drives. This keeps all your data in one place, allowing for easy backups, and also makes it easy to get at the same files from any computer.

My personal preference is a Linux computer with several cheap IDE drives each on their own IDE controller (no slave drives). The drives are configured as software RAID 5 and ext3. Regular backups are setup through cron to a tape drive. Samba handles file sharing, printing, roaming profile, and PDC duties for Windoze. Netatalk 1.6cvs handles file sharing duties for pre-OSX systems. NFS is used for file sharing to *nix systems. The only thing I'm missing is a NetInfo daemon for Linux so it can act as a complete configuration server for NeXTSTEP, OPENSTEP, and MacOS X systems.

Seperate home and work!

[bluGill]

first of all, seperate your home life and work life. Then seperate the data. I understand that once in a while you need data from one place at the other, but avoid those situations.

At work: that is IS's problem. Store all work data on the work machines, and make IS do the backups. Use SSH, or other VPN when you want to work from home. Compile (or whatever) at work as much as possible. If you have data that you need on the road, get a laptop or PDA for work, and synchronize that when you are at work.

At home: set up a linux box (a 386 is enough, though you might want more) with a big disk, a UPS, and a network card. Put it in a closet or on a shelf. Install SAMBA, and Netatalk. with NFS built in (though there is better than NFS if you look, nfs is there) Use one loging for all machines.

Laptops are a problem, because you often want to use them where you can't get to the network. The first solution to that problem is 820.11. Use it at home, and look for open access on the road. With good VPN (ssh+nfs) you can get to your network server from many places. I manually synchronize only the files I need, but my laptop is rarely used outside of 802.11 areas, if you travel often, then you might need more. (CODA? AFS? )

don't use NFS

[Kunta Kinte]

Unless you want to share your data with lots of 'friends' you just haven't met yet.

NFS is used very often to mount home directories. But what is stopping someone from unplugging the workstation, plugging in a linux laptop with the IP of the legitimate workstation and mount the share, "su - user", and voila, you now have all the user's files.

That's just the simplest way. The problem is that most NFS implementations don't have *any* authentication except for IP authentication. So so other DNS attacks would work as well.

I am surpised that the most widely used network file system implementation for linux and most posix OSes has no real authentication. There *has* been authentication built in the protocol since version 3, but last time I checked, it was not supported on the linux. I was told by one guy working on the project that the problem was that there's no crypto in the kernel.

I used secure NFS on Solaris 8 for a while but I constantly lost the mounts. That but be fixed now, I don't know.

Use AFS, CVS, rsync, intermezzo, or something. But I would stay away from NFS.

Can there be only one?

[rwa2]

Well, here's my approach...

First, I try to adhere loosely to the FHS for ideas on overall organization. Even though it's mostly intended for POSIX systems, following their philosophy will really help you separate your data from your platform-dependent program files and libraries.

Most of my important stuff goes on the Linux server in /home (on an IDE software RAID1). However, I try to limit files in here to stuff that's absolutely essential to keep the size down. I occasionally mirror this offsite to my friends' servers with rsync (with the private stuff pgp encrypted). I try to make browser caches, etc. symlinks to dirs in /tmp . Try to keep only the stuff you created yourself in here.

I keep media and downloads on a plain partition under /home/ftp/pub (which is also symlinked from the http document root). That way, all my computers can easily get access to music and installers and junk.

Samba helps win32 boxes access the /home and /tmp directories.

NFS exports /home to the other UNIXen, as well as /usr for the other machines with the same CPU arch. It should be acceptable to export /usr/share to other UNIXen with different architectures.

I'd like to set up CODA, since it seems to support more different kinds clients than Intermezzo. These support disconnected operation and are good for laptops. For the meantime, I just use rsync to mirror home dirs onto my laptop, though (and just keep track of stuff that I change on the road manually :/ )

No thoughts on how to combine everything into a distributedFS so you could have parts of, say, a music archive living over several machines. There are several projects for Linux-only (PVFS) or Win32-only (more advanced network-neighborhoods). I'd say your best bet for convenience is just to make sure everything is visible from your one server and reexport it from there (invest in a switch so it doesn't deadlock your network). Until better DFSes exist, though, I think you'll get better performance and less confusion from running everything from one beefed-up server with a RAID (or two if you want failover).

Here's how I stay organized:

[oni]

I keep all the porn in a seperate directory. That seems to work pretty well.