Distributed Filesystems for Linux?

zoneball asks: "What would you use for a distributed file system for Linux? I have several GNU/Linix machines running at home, and wanted to be able to see more or less the same file tree (especially all the ~user directories) regardless of which machine I'm connected to, and where the traversal into the distributed file system space is largely transparent for the end-user. Are there any URLs or documents that compare the features, bugs, road map, stability of these and other distributed filesystems? Which offers the best stability and protection from future obsolescence?"

Zoneball looked at 3 distributed filesystems, here are his thoughts:

" Open AFS was the solution I chose because I have the experience with it from college. For performance, AFS was built with an intelligent client-side cache, but did not support network disconnects nicely. But there are other alternatives out there.

Coda appears to be a research fork from an earlier version of AFS. Coda supports disconnected operations. But, the consensus on the Usenet (when I looked into filesystems a while ago) was that Coda was still too 'experimental.'

Intermezzo looks like it was started with the lessons learned from Coda, but (again from Usenet) people have said that it is still too unstable and it crashes their servers. The last 'news' on their site is dated almost a year ago, so I don't even know if it's being developed or not"

So if you were to recommend a distributed filesystem for Linux machines, would you choose one of the three filesystems listed here, or something else entirely?

Comments

NFS

[mao che minh]

I know that this is going to be the most common answer, but just go with NFS. It's not the most secure option around, but obviously the simplest to implement and the best documented.

NFS Linux FAQ
Howto #1
Howto #2

If you find yourself needing help, try asking people at Just Linux forums, or trying the NFS mailing list.

Self Certifying File System

[nescafe]

I would use SFS, the Self Certifying File System. Assuming all the systems you are using are supported, it offers global, secure access to anything you care to export.

Well it depends...

[Tsugumi]

For my money, nfs in a LAN, afs over a WAN, it really depends on the size of the network your trying to play with.

Since openafs forked from the old transarc/IBM codebase, it looks as if it has a real future. It's used by a load of educational and research institutions (notably CERN), as well as Wall Street firms.

Background on DFS

[El Pollo Loco]

Check here for a good background on DFS. It also has a quick table comparison of the popular programs, and a walkthrough to set up Intermezzo.

PVFS

[Kraken137]

We use PVFS at work to give us a high-performance network filesystem for use with our clusters.

http://parlweb.parl.clemson.edu/pvfs/

openmosix

[joeldg]

I run an openmosix cluster with the openmosix filesystem here at work. Three computers.. no problems...
If you want to take a look..
http://lucifer.intercosmos.net/index.php
linkage and I am going to be placing some tutorials up. -joeldg

Ye olde Samba

[Anonymous Coward]

Samba works fine. I personally have approximately 5 samba mounts in my filesystem totally transparent for anybody who was to walk up and use my computer.

No need to unnecessarily complicate things here, samba is simple to set up and functions great.

Re:permissions?

[phorm]

That's what NIS is for. You can schedule regular downloads of group/passwd files, which are updated in a NIS database stored on a master server, and passed down to "slave" servers.

Intermezzo does appear to be a current project

[Dr.Zap]

While there is no new news posted on the site, ther are current tarballs on the ftp server, as recent as 5.9.03. (but that file appears to be a redux, last update to code seems to be 3.13.03)

The sourceforge page for the project (http://sourceforge.net/projects/intermezzo) shows status as production/stable but the info there looks stale too.

Re:permissions?

[Dysan2k]

To be honest, big time, but a lot of people forget the other side of life with NFS, and that's NIS/NIS+. The yp-tools include pretty good NIS support, but not sure of NIS+. Would use niether in a production environment personally, but a common Auth system which is easy to manage would solve that issue.

Could also look into LDAP (VERY complex, no good starting point that I've been able to find) and Kerbreos auth methods as well.

Should give you a central point for uids/usernames. But NFS does not have transparent mounting that I'm aware of so that you could mount, say the /home directory of 5 computers onto / on a central system and it display all the mounts simultaneously. For example:

<ECODE>
CPU1 contains: /home/foo
/home/baz

CPU2 contains: /home/tic
/home/tac

CPU3 contains: /home/toe

on CPU4, you'd do the following:
mount CPU1:/home /home
mount CPU2:/home /home
mount CPU3:/home /home

And you'd end up with on CPU4:
/home/tic
/tac
/toe
/foo
/baz
</ECODE>

If there is a way to do this, please lemme know. I've heard people talk about it in the past, but haven't seen anything come of it yet.

NFS is not a DFS

[purplebear]

Just so you all know. NFS is a network accessible FS. A DFS can also be network accessible from clients, but it physically resides on multiple systems.

Re:Mirroring file system

[dlakelan]

Whoa, you definitely need Unison.

Unison will synchronize any two file trees in The Right Way (TM).

Get the gtk version for interactive conflict resolution.

Tutorial

[TheFlu]

I just went through this process a few weeks ago and I must say I'm really glad I went through the trouble of setting it up...it's very cool. I actually wrote a tutorial about how to accomplish this by using NIS and NFS. I hope you find it helpful.

The only trouble you might run into with the setup I used is some file-locking issues with programs wanting to share the same preference files.

Re:AFS vs NFS

[pHDNgell]

I'm disturbed at the number of people who are recommending NFS as a distributed filesystem solution. While it might be easy to get going initially, I've had more long-term problems with my NFS server and client interactions than my AFS. To get my NFS clients to behave anything like AFS clients, I had to build and install an automounter that could use NIS config.

You only have to wait for the first day you want to reboot a fileserver without breaking every system on your network or waiting for startup dependencies, etc... One day, I moved all of the volumes off of an active fileserver (i.e. volumes being written) and shut the thing down and moved it to another machine room, brought it back up, and moved the volumes back. The reads and writes continued uninterrupted, no clients had to be restarted, no hung filesystems anywhere, etc...

OpenAFS all the way

[fsmunoz]

I had more or less the same basic requirements and I opted for AFS.

My needs were a little more demanding (had to be implemented in GNU/Linux, Solaris, AIX, HP-UX and as an extra Windows 2000) and grocking AFS can be difficult at first but it was the best choice by far. Stable across all the Unices, very secure (this was another requirement) and integrates perfectly with our Kerberos Domain and LDAP accounting info. It provides a unique namespace that can span multiple servers transparently, does replication, automatic backups and read-only copies, client-side cache with callbacks, has a backup (to tape) system that can be used stand-alone or integrated with existing backup structures (Amanda, Legato, TSM) AND was the basis for the DCE filesystem, DFS (as a side note I find it interesting - and sad - that most things people try to emulate this days are present in DCE , and Windows 2000 got many of the "new features" from a technology initially made for Unix :DFS, DCOM, Directory Services, SSO, DCE-RPC, etc.)

AFS is amazing and much more robust than any distributed filesystem I know of; it has shortcomings when servers time out, but apart from that it's really an excellent solution; an example I generally use to give an idea of some of the good features of AFS is a relocation of a home directory to another server. The user doesn't even notice that his home directory was moved to another server *even if he was using it and was writing stuff to disk*; at most all writing calls to his home dir have a small delay (a couple of seconds) even if his/her home dir was 5 Gb worth.

Kerberos integration is an added bonus, if you can you can use this as an excuse to kerberize your systems and form a Kerberos Domain. If you don't want to just stick with the standard AFS KA server.

In my setup I have Windows users accessing their home dirs in AFS using the Kerberos tickets they have from the Windows login and the fact that a cross-realm trust was made between the Unix DOmain and the AD; the can edit all the files they are entitled to with that ticket, and the system is so secure that Transarc used to put the source code in it's public AFS share and added the customers that bought the source to the ACL of the directory that contained it.

With all this features it would be hard not to vivedly recommend OpenAFS as the best solution for a unified, distributed filesystem. Bandwidth utilization is, in my experience, at least half of what NFS uses, which is an added bonus.

cheers,

fsmunoz

Re:OpenAFS all the way

[MilliAtAcme]

I second this "all the way" thought. I've been running OpenAFS for almost 2 years now on Debian GNU/Linux (many Thanks to Sam Hartman, the maintainer) and have never been disappointed. It's been pretty darn solid and, most importantly, has never lost any of my data through various upgrade cycles. It's a bit of a change in thinking, however, for those coming from an NFS background.

There were three big wins for me...

(1) Global file namespace managed server-side and accessible from anywhere... LAN, WAN, whatever. All clients see files in the same location.

Unlike NFS, where you have to "mount" volumes within the file system on each client, the AFS file system is globally the same, living under "/afs", so every client accesses the same information via the same file system path. A notion of "cells" makes this possible... information under a single administrative authority lives in a "cell", e.g., "/afs/athena.mit.edu" is the top-most "mount point" for a well-known cell at MIT. Volumes, in AFS parlence, also aren't tied to any particular server or even location in the name space as far as the clients know. A client doesn't have to know explicitly in it's configuration which server a given bit of information lives on, and that data can be moved around behind the scenes as necessary (increase the volume space, increase the redundancy, taken offline, etc...) All volume mounts are handled server-side. The clients only have to know about the cell database server, and that can be determined via AFSDB records in DNS. (I.e., your AFS "cell" name matches up with your domain name, e.g., /afs/athena.mit.edu matches up with "athena.mit.edu" in DNS.) So almost all management aspects are handled server-side.

(2) Client side implementations.

All my Linux and Windows machines can access the same AFS file space. An OS X client is available too, but I've not needed that to date, but might someday. I thus have all home directory information, as well as a lot of binaries, living in the AFS file space, in one place. And behind the scenes, that info is on multiple AFS servers that have RAID-5 disk arrays and weekly tape backups going on.

(3) The file system "snapshot" feature, for backups.

You can take a snapshot of volume(s) at a particular point in time and roll them onto tape without needing to take them offline. You don't have to worry about inconsistencies in the files. Folks can continue to update files but the backup snapshot doesn't change. Very much the same as the snapshot feature on Netapps. These snapshots, called backup volumes, can even be mounted in the file space so folks can get access to the old view of the volume, e.g., accidentally deleted a critical file and need it back.

And security via Kerberos is nice, especially if you already have an infrastructure. But it's not too hard to setup a single KDC to get started. In the Debian distribution docs for OpenAFS, there's a setup and configuration transcript that makes things relatively easy and clears up a lot of questions.

In summary, OpenAFS is a very good solution here.

A potted review of several distributed filesystems

[elronxenu]

Why not stick with NFS for the time being?

I went through the "is coda right for me?" phase, and also "is intermezzo right for me?" and also spent tens of hours researching distributed filesystems and cluster filesystems online ... my conclusion is that the area is still immature, I will let the pot simmer for a few more years (hopefully not many), and use NFS in the meantime.

My situation: desire for scalable and fault-tolerant distributed filesystem for home use with minimal maintenance or balancing effort. Emphasis on scalable, I want to be able to grow the filesystem essentially without limit. I also don't want to spend much time moving data between partitions. And last but not least, the bigger the filesystem grows, the less able I will be to back it up properly. I want redundancy so that if a disk dies the data is mirrored onto another disk, or if a server dies then the clients can continue to access the filesystem through another server.

All that seems to be quite a tall order. I checked out coda, afs, PVCS, sgi's xfs, frangipani, petal, nfs, intermezzo, berkeley's xfs, jfs, Sistina's gfs and some project Microsoft is doing to build a serverless filesystem based on a no-trust paradigm (that's quite unusual for Microsoft!).

Berkeley's xFS (now.cs.berkeley.edu) sounded the most promising but it appears to be a defunct project, as their website has been dead ever since I learned of it, and I expect the team never took it beyond the "research" stage into "let's GPL this and transform it into a robust production environment". Frangipani sounds interesting also, and maybe a little more alive than xFS.

On the other hand coda, afs and intermezzo are all in active development. afs IMHO suffered from kerberitis, i.e. once you start using kerberos it invades everything and it has lots of problems (which I read about on the openAFS list every day). AFS doesn't support live replication (replication is done in a batch sense) either.

CODA doesn't scale and doesn't have expected filesystem functionality: for 80 gigs of server space I would require 3.2 gigs of virtual memory, and there's a limit to the size of a CODA directory (256k) which isn't seen in ordinary filesystems. There's also the full-file-download "feature". CODA is good for serving small filesystems to frequently disconnected clients but it is not good for serving the gigabyte AVIs which I want to share with my family.

Intermezzo is a lot more lightweight than CODA and will scale a lot better, but it's still a mirroring system rather than a network filesystem. I might use that to mirror my remote server where I just want to keep the data replicated and have write access on both the server and the client, but it's again not a solution for my situation.

The best thing about intermezzo is that it sits on top of a regular filesystem, so if you lose intermezzo the data is still safe in the underlying filesystem. CODA creates its own filesystem within files on a regular filesystem, and if you lose CODA then the data is trapped.

Frangipani is based on sharing data blocks, so like NFS it should be suitable for distributing files of arbitrary size. I need to look at it in a lot more detail; this is probably the right way to build a cluster filesystem for the long haul. For the short term, Intermezzo is probably the right way for a lot of people: it copies files from place to place on top of existing filesystems.

What I did in the end:

• new server (Celeron 1.3 GHz, 512 meg RAM)
• 2 x 80 gig IDE disks
• Each IDE drive has 2 partitions (one small, one huge)
• Each partition is RAID-1 mirrored with its partner on the other disk
• The huge RAID partition is defined to Linux LVM (logical volume manager)
• Logical volumes are created within that for root, /home, etc...
• All logical volumes are of type ext3 for recoverability.

The way it works is tha

Re:NFS is not even close to secure

[tzanger]

I use a very simple script to help keep NFS secure:

IPTABLES=/usr/sbin/iptables
RPCINFO=/usr/sbin/rpc info
GREP=/usr/bin/grep
AWK=/usr/bin/awk

$IPT ABLES -F nfs
$IPTABLES -N nfs &> /dev/null
$RPCINFO -p localhost | $AWK '/portmap|mount|nfs|lock|stat/ \
{ print "iptables -A nfs -p " $3 " --dport " $4 " -j DROP" }' | \
/bin/bash

$IPTABLES -L INPUT -vn | $GREP -q 'nfs all -- !ipsec0+'
if [ $? -ne 0 ]; then
$IPTABLES -I INPUT 1 -i eth0 -j nfs
fi

Basically it only allows incoming NFS-related connections over ipsec, dropping anything that is not. NFS port allocation is dynamic by default and I know you can force ports, but this seemed far easier to scale.

One thing I have noticed (and perhaps it's common knowledge to NFS experts) is that in order to get locking to work at all, my NFS clients had to be running statd and lockd. Without 'em everything worked but locking would fail every time.

Watch for NFSv4 in the future!

[Sri Ramkrishna]

Watch for the new version of NFSv4. There are already a sample implementation in the linux 2.5 tree. NFSv4 will address most of the problems that NFSv3 and others have. Including plugin security models, namespace, and revamped ACL handling.

It's also WAN friendly, letting several operations be done at the same time with a single directive. (COMPOUND directive) It also allows you to migrate one filesystem to another with no stale filehandles. Basically, it's trying to be an AFS killer.

For more information, take a look at
http://www.nfsv4.org/

Lots of good info including the IETF spec. It's a interesting read.

The spec is not quite complete. Currently, I believe there are discussions with how NFSv4 will work with IPsec.

Cheers,
sri