Distributed Filesystem for Disconnected Operation?

juraj asks: "I'm trying to achieve the following setup: I have two offices connected via a relatively slow ADSL line, and I want a shared fileserver between the offices. I have VPN using IPSec ready, so security is less of a concern, but simply mounting a filesystem (via Samba or NFS) from one office to another is not a solution because of the speed. Also, the ADSL line is sometimes not only slow, but also disconnected. I've tried the CODA distributed filesystem to achieve replication, so that both offices have local copies of their files. The problem is, that the CODA filesystem is just a research project: it is unstable, with the venus daemon constantly falling, and sometimes when recovering from the disconnected state, one side does not recognize the changes and they are simply not propagated. Have you had any good experiences with CODA? Which versions do you use? What kind of setup did you have? How is it configured? I've also heard about OpenAFS, but similar to CODA, I've learned it is unusable in a real environment. Is there any real solution to my problem? Are there any decent solid free distributed file systems for Linux or the BSDs?"

Comments

This question

[Molina the Bofh]

is very good, and I was thinking of something like that for my mailservers. That way, I'd have 2 different machines in 2 locations, and [maildir] boxes in both. When message arrive in one, any one, it is copied to the other. When erased, same thing.

Both servers running at the same MX. So users could choose the server 1 or server 2 according to the location. And witch among them, in case one network goes down.

Although it sounds simple, I don't know any simple solution to that. Rsync won't work, as there wouldn't not be a master server. Both would have the same preference, so no server depends on the other. That's the goal.

Re:This question

[DDumitru]

Please excuse the ad here (mod down if you like).

I developed a replicated filesystem that we use with our commercial email service. The filesystem is layered under UML (User Mode Linux) and cross-replicates files between two servers, on in California, and one in Pennsylvania.

I too looked at Coda and Inter-mezzo, but was not very satisfied with their stability and/or their ability to recover from outages.

The replication that we use relies on the update nature of MailDir with Courier Imap.

Our solution uses UML to post a transaction journel to the underlying host OS layer. Application level code then cross-posts filesystem updates using HTTP transactions with curl and Apache/cgi. Transactions are delayed about 2 seconds to coalesce multiple updates into a single network event. In general, we get about 5mbit of update thruput coast to coast and it is very rare that either system is more than a couple of seconds out of sync.

I am sorry that I cannot give you the code. While the code is Linux bases, we don't actually sell (distribute) it, so we keep it in-house for our own use. Perhaps my description will give you some ideas.

The email offering is described at:

http://easyco.com/mail/index.htm

Re:This question

[Tomun]

Try offlineimap, it'll sync imapimap or imapmaildir

Re:This question

[Just Some Guy]

Your transport almost sounds like you implemented a lightweight imitation of Subversion but without the versioning capabilities.

Also, what am I missing that makes it significantly better than a triggered rsync session, i.e. one that runs either periodically or whenever a threshold of changes is exceeded?

Re:This question

[reaper20]

Also, a neat feature that offline imap has is that it can copy from 2 remote IMAP servers without the need for an intermediate PC. It's awesome.

Re:This question

[DDumitru]

Perhaps a bit more detail of the engine will help.

The actual file-system driver in UML is patched to produce an "event log" for all file updates. Because this is UML, this log can run in user-space to the host OS.

The log itself includes events such as "close file after writing", rename file, create directory, set priviledges, etc.

The log itself is sent to a pipe. The pipe is dequeued by a daemon that builds transaction "blocks". A transaction block can contain one or more transactions. When a transaction block is closed is dependent on both time, block size, and whether a block is currently "in-flight" to the far end. If the engine is not currently moving data, we close a block after 2 seconds or 500K. If the engine is busy moving data to the far end, the timeout is enlarged to 60 seconds or 500K.

The actual transfer engine is a POST operation over HTTP. The sending end is a shell script that calls curl. The actual transfers are tagged with transaction numbers so that failed transactions can be retried. The receive end is a CGI application that calls a C program to post the actual updates to the file-system. The curl/cgi engine also has the ability to compress the data with compress/gzip/bzip2 (we use gzip as a compromise compression level for our speed of lines).

We could have used rysnc for the transfer itself (and actually started to engineer this early on). In the end, the transaction latency of rsync made it impractical and we needed a different level of functionality.

intermezzo!

[Anonymous Coward]

http://www.inter-mezzo.org/

you are looking for intermezzo

http://www.inter-mezzo.org/

the same guy from coda is the leader. remember

afs -> coda -> intermezzo

Re:intermezzo!

[Elwood P Dowd]

And from their web page, there are still caveats. One of their components is advertised as "needing more work before it can be used in production."

My company uses redundant leased lines to home (different breeds and providers) to ensure that every building can access network resources at all times. Manual fail over. We're not a huge company, but we manage most of this in-house. We'd *love* to know if there's a better answer, even if it cost a lot of money.

Well. There's always a better answer on the other side of a long and expensive implementation process.

Re:intermezzo!

[Bronster]

http://www.inter-mezzo.org/

you are looking for intermezzo

Hmm.. let's just look at the mailing list again... maybe just a snip from a recent(ish) post (Mar 22 - there are 8 posts since this one, half of them spam):

| Don't post to a list without reading it also.

I read this list, what little of it I get.

| And don't complain about the state of open source
| software, if you are not ready to test it's betas.

I am certainly ready to test the betas, just that the last time I tested
Intermezzo or Lustre -- Lustre I couldn't even get patched, and
Intermezzo I couldn't get a single share working without an immediate
kernel oops -- and a fatal one, at least in that the filesystem never
worked for me.

This is not just a user-friendliness issue. I have played with
experimental software -- even had an entire machine on reiser4 for a
month or two. And that was _much_ easier.

However, I promise to test Intermezzo again when I get more time.

-----------------------

Basically Intermezzo is not finished, and the developers don't actually have the resources to bring it to production level. I'd certainly not trust it with my data, and I've been watching it for years. I've never had the time to contribute unfortunately, and have always run out of time to test it in environments where I have enough boxen to actually do a realistic test (i.e. at work).

Re:intermezzo!

[mattmcl]

Normally, I stay away from Microsoft as much as possible, but you did say that you'd be willing to spend some money... At a previous job, we were all Win2k/XP and Active Directory...we setup Microsoft DFS that kept a replicated copy of the network file share on a secondary domain controller at a remote location. The primary reason was to have an off-site copy of everything in the event of a disaster of some sort, but, if the primary domain controller/file server went down, users would not even notice - changes would just be saved to the remote server and replicated back when the primary came back online. If the network went down, it would sync up seamlessly when it came back online. It really did work surprisingly well. If you're already a Windows shop, you can't go wrong.

Re:intermezzo!

[yamla]

Would this allow someone to make changes on the master and allow someone else to make changes on the currently disconnected secondary, and automatically sync up when everything was connected up once again? From your description, it sounds like this feature was not supported and so this solution would not really meet the requirements outlined.

Re:intermezzo!

[Elwood P Dowd]

Supposedly there are a variety of reasons that DFS won't work for us. I mean, I think we already use it for hardware fault tolerance, but it won't work for us for network faults. Dunno why. I'm not the admin.

Re:intermezzo!

[mattmcl]

Yeah - that's a definite problem. I'm not sure there's a silver bullet here. The only type of system I know of that would handle this is a CVS-type system where changes can be merged.
I could be wrong, but I think if you've got two versions of a non-text document being concurrently modified in separate locations with the net down, you're going to have issues no matter what file system you're using.

Re:intermezzo!

[mattmcl]

Well, I suppose every situation is different. All I can say is that we used it through both hardware and network failures, and it always worked.

Re:intermezzo!

[bruthasj]

Unfortunately, it crashes just as much as coda.

Re:intermezzo!

[aminorex]

You never heard of Gummo or Zeppo?

Unison

[JabberWokky]

What you're looking for is something like unison. Since I don't know what you're serving off of those servers or how often you update files, I can't tell you if it will work for you. But it is robust, and with the -batch flag, it can be automated. It is quite CPU and disk intensive, that's why I say "something like". It's made more for daily or hourly syncs.

--
Evan

Re:Unison

[Anonymous Coward]

Remember folks, unison has a 2GB file size limit.

Re:Unison

[Richard_at_work]

Have a look at the Unison article on BSDstudio (link) for an indepth introduction to using Unison in this manner.

Novell ifolder

[Why Should I]

Haven't actually looked into this to any great defree, but is Novel's iFolder an option ?

It's opensourced even and available on Novel Forge.

Re:Novell ifolder

[Earlybird]

While it looks interesting, the project is labeled as pre-alpha -- not ready for production use.

Perforce (or any other Version Control system)

[shadowxtc]

I find this to be the ideal solution for keeping filesystems synchronized across slow links.

From my experience, Perforce has the best use of bandwidth and also the most intelligence when it comes to rearranging directory structures and resolving conflicts.

Unfortunately it's only free for up to two users - so it may be useless for your needs.

Re:Perforce (or any other Version Control system)

[hashinclude]

Mod Parent Up!

I have used P4 (perforce) to keep a lot of files in sync between two locations. Fortunately, I had only two locations, so the 2-user 2-client limit never was exceeded.

In case you want more clients/users, you can try for any of the following:

1. CVS (http://www.cvshome.org/)
2. GNU Arch (http://www.gnu.org/software/gnu-arch/)
3. SubVersion (http://subversion.tigris.org/)

All these are excellent source control tools, and operate over ordinary TCP/IP (don't need a special setup).

Avoid tools like Visual SourceSafe because they require a network-mapped drive to work.

http://better-scm.berlios.de/comparison/comparis on .html gives a comparitive list of version control systems out there.

Re:Perforce (or any other Version Control system)

[sudog]

Actually if you're doing Open Source development you can have unlimited users and clients. FreeBSD uses it internally for example.

OpenAFS unusable in a "real" environment?

[LoneRanger]

Bullshit. You haven't looked at it hard enough then. I used to work at a university that had 26,000+ users using an AFS filestore for their homedirs and for distributed apps across several miles of campus.

I'm sure this thing has more than surpassed terabyte size by now. It was always fast and always reliable, except when the one of server's SCSI cards would melt and start spewing errors.

AFS is better than most people give it credit for. I'll admit, it isn't easy to set up, but all the features that you get for that initial work are well worth it.

Re:OpenAFS unusable in a "real" environment?

[passthecrackpipe]

Yeah - OpenAFS is *still* really the only way to go for multi plafrom, disconnected, distributed filesystems. It positively *rocks* - the only downside from my perspective is the unwieldy kerberos management environment, but i am pretty sure that has more to do with my own lazyness and ignorance (wrt learning proper kerberos instead of simply rattling off the HOWTO) as opposed to a fundemental flaw in the system.

Re:OpenAFS unusable in a "real" environment?

[Umrick]

OpenAFS is a great solution to a problem, just not this one. It doesn't work in a detached state. On the other hand, the caching is quite aggressive, and if it's an option, you could set up two cells that trust each other and access files that way.

I'll be happier once the stable versions have two things though... >2gb file support, and support for 2.6 series kernels. Disconnected operation would be nice as well.

All of those are proposed projects, but not currently in the developement version (at least not in the changelogs).

Re:OpenAFS unusable in a "real" environment?

[wik]

AFS dies horribly if your clients lose sight of the volume location or file servers. As long as the machines are well-connected, it works great.

As far as Kerberos goes, I'd suggest the new ORA nutshell book "Kerberos: The Definitive Guide". While it doesn't go into AFS much, it explains how the thing really works and how to configure MIT and Heimdal Krb5.

- Happy AFS/krb5 site administrator

Re:OpenAFS unusable in a "real" environment?

[danpritts]

There are patches for the AFS client to support disconnected operation. They haven't made it into the core distribution yet (lack of developer resources) but if this is a must-have for you you might want to check it out. This mailing list post is a good place to start.

AFS doesn't work at my uni

[Anonymous Coward]

Monash University is using AFS on its Linux desktops. Whenever the connection to the file server goes down, everyone's sessions hang, which is clearly unacceptable.

It's quite possible that it has been incorrectly set up, but in this situation AFS hasn't delivered what it promised.

Re:AFS doesn't work at my uni

[Anonymous Coward]

Then it is incorrectly setup. That or they want it that way for some reason.

Re:AFS doesn't work at my uni

[auzy]

umm, I go to monash uni too and before they were using NFS, I haven't really tried out the AFS drives yet except over ra-clay and stuff, but from the short time I used them on the network, they are far better. I remember all the NFS probs they had (I must have lost at least 4 or 5 assignments on them and lost at least 20% in marks). AFS has disconnected operation so should be much better.. Have you tried it out this year much.. It might also be hanging because they are trying to make different fetches off the network though all the time that can't be cached.. To me its seemed to be alot better then all of those NFS stale file entries from last year, that I used to get all the time.. So far I've had no probs with AFS at monash. But then again, I'm 3rd year comp sci now so I haven't been using it as much as I used to. Then again, doesn't surprise me if its not set up right.. Monash administration is awful. they wont even support setting up a jabber server :( Network also seemed faster at monash (probably a side effect of the disconnected operation by AFS)..

Re:AFS doesn't work at my uni

[auzy]

oops, sorry, forgot to preview that so the breaks are missing.. :(

Re:AFS doesn't work at my uni

[runderwo]

AFS has disconnected operation so should be much better..

No, it doesn't. If a volume goes offline, you will be unable to access any files within it, even if they are stored in the local cache. This is precisely what Coda attempted to improve upon, as it is essentially a genetically modified AFS.

AFS works fine

[frenchkiss]

It is used in a number of university campuses across the US (as a bunch of disjoint namespaces under /afs) and works fairly reliably. I wouldn't say it's perfect but works well for day-to-day usage.

Among the most notable of the universities using afs are CMU, UNC, RPI, MIT. Furthermore, there are a number of government namespaces as well.

Try it out!

Re:AFS works fine

[raju1kabir]

Lots in other countries too:

% ls /afs
afs.hursley.ibm.com eos.ncsu.edu mcc.ac.gb sfc.keio.ac.jp afs1.scri.fsu.edu es.net md.chalmers.se si.umich.edu alw.nih.gov ethz.ch me.cmu.edu sipb.mit.edu andrew.cmu.edu federation.atd.net meteo.uni-koeln.de slac.stanford.edu anl.gov fh-heilbronn.de mpa-garching.mpg.de sleeper.nsa.hp.com asu.edu fl.mcs.anl.gov msc.cornell.edu spc.uchicago.edu athena.mit.edu fnal.gov msrc.pnl.gov sph.umich.edu bnl.gov geo.uni-koeln.de msu.edu spv.uniroma1.it bp.ncsu.edu glue.umd.edu nada.kth.se stacken.kth.se bu.edu gr.osf.org ncat.edu stat.lsa.umich.edu caspur.it graphics.cornell.edu ncsa.uiuc.edu cats.ucsc.edu hep.net nd.edu cc.keio.ac.jp hephy.at nersc.gov theory.cornell.edu ce.cmu.edu hrzone.th-darmstadt.de net.mit.edu transarc.com cern.ch iastate.edu northstar.dartmouth.edu transarc.ibm.com ciesin.org ibm.uk nrel.gov tu-chemnitz.de cipool.uni-stuttgart.de ifh.de nrlfs1.nrl.navy.mil ualberta.ca citi.umich.edu openafs.org umich.edu club.cc.cmu.edu ihf.uni-stuttgart.de others.chalmers.se umr.edu cmf.nrl.navy.mil ike.uni-stuttgart.de phy.bnl.gov uni-bonn.de cmu.edu in2p3.fr pi.infn.it uni-freiburg.de cs.cmu.edu infn.it pitt.edu uni-hohenheim.de cs.pitt.edu ipp-garching.mpg.de postech.ac.kr uni-mannheim.de cs.unc.edu ipp-hgw.mpg.de pppl.gov unity.ncsu.edu cs.utah.edu ir.stanford.edu psc.edu urz.uni-heidelberg.de cs.washington.edu isi.edu psu.edu urz.uni-magdeburg.de cs.wisc.edu isk.kth.se research.ec.org utah.edu csv.ica.uni-stuttgart.de isl.ntt.jp rhic vn.uniroma3.it dapnia.saclay.cea.fr jpl.nasa.gov rhrk.uni-kl.de w3.org desy.de lcp.nrl.navy.mil ri.osf.org wam.umd.edu dis.uniroma1.it le.infn.it rl.ac.uk wu-wien.ac.at dkrz.de lngs.infn.it rose-hulman.edu lrz-muenchen.de rpi.edu zdv.uni-mainz.de ece.cmu.edu lsa.umich.edu rrz.uni-koeln.de zdvpool.uni-tuebingen.de ece.ucdavis.edu rus.uni-stuttgart.de engin.umich.edu mathematik-cip.uni-stuttgart.de rwcp.or.jp engr.wisc.edu mathematik.uni-stuttgart.de rz.uni-jena.de

A similar question: edit/compile/run over ADSL

[Welshalian]

My problem is similar to the original poster's, with one small limitation: I have two (mostly) identical Linux machines at both ends of an ADSL link with VPN, etc. All I need to do is edit/compile/run a CLI application (no fancy graphics required). The app must compile/run on a machine at the 'office' end, but I'd like to edit on the machine at the 'home' end. I tried emacs on the 'home' machine with remote editing, as well as remotely running vi/emacs on the 'office' machine, but neither method has the responsiveness I crave, especially because I tend to hit Ctrl-S to save every few keystrokes. I also tried rsyncing after editing the filesystem on the 'home' machine, but then the process gets a bit cumbersome. Any tricks I might have missed?

Re:A similar question: edit/compile/run over ADSL

[noselasd]

Use version control ?
Edit at your local site, have a (subversion/cvs)server at the office.

Re:A similar question: edit/compile/run over ADSL

[Artichoke]

I mount via nfs over a VPN over 1Mb ADSL (rsize=8192,wsize=8192,intr,rw,async,noatime,noaut o,user) and after the Vim session is restored, don't have a problem.
An rsync based script (FWIW in Python) to xfer disparate directories and files works around the cumbersomeness problem.
As for the 'use version control' responses: I don't want to store intermediate versions of persistent files and don't want to store intermediate/temporary files at all (but don't want to recreate them from scratch every couple of day when I swap from home to/from office).

BTW: Nick, how's the AFS investigation going? {8{)}

Unison

[hak1du]

Don't bother with any of the kernel-mode disconnected file systems. For those kinds of situations, the Unison file synchronizer is a good choice: it performs bidirectional synchronization and uses an efficient protocol that only needs to send differences and some checksums across the wire. It also detects conflicts and (optionally) lets you resolve them automatically. It works on UNIX/Linux, Windows, and MacOS.

You can use rsync

[tcheud]

it looks like a simple shell script with rsync will do it: http://lists.samba.org/archive/rsync/2001-October/ 000430.html

Re:You can use rsync

[Molina the Bofh]

It's not that simple. Files should be created and deleted on both systems, not created just in one.

cvs

[psycho]

Why not just use CVS or, even better, subversion?
Just a first thought...

Re:cvs

[doctormetal]

Why not just use CVS or, even better, subversion?

You should use CVSup for this.
It has already proven its useability for syncing and updating FreeBSD systems

I think you _could_ use rsync

[tcheud]

it looks like a simple shell script with rsync will do it...
http://lists.samba.org/archive/rsync/2001-O ctober/ 000430.html

use AFS

[stonebeat.org]

use the real AFS from IBM. work very nicely.

Re:use AFS

[hirschma]

How can I buy their product? Who sells it? What does it cost?

I'd love to get started with a supported AFS... is this something that only corps can buy?

Jonathan

Re:use AFS

[Earlybird]

• How can I buy their product? Who sells it?

IBM AFS. Note that OpenAFS is a true fork of IBM's own code, and currently maintained by IBM and the community. Afaik, IBM AFS is no longer in active development. You don't need to buy anything except support.

• What does it cost?

IBM AFS client licenses have historically been "very expensive" -- that's about all I know. If you need to ask, you probably can't afford it. :)

Re:use AFS

[Anonymous Coward]

$15K per license i believe. :)

MOD PARENT UP

[Molina the Bofh]

Thanks for the tip. It seems to do what I need.

Re:tramp

[Chaostrophy]

Use the tramp package, it automates grabbing files via ssh (through multiple hosts), so you edit localy. Very hand, I really liked it.

Re:use AFS=OpenAFS

[oli_freyr]

I'm no expert, but I became curious about the difference between IBM AFS and OpenAFS and it seems that they are the same.

This means I will probably check it out for my next fileserver project... ;)

Oracle iFS?

[isj]

Have you considered Oracle iFS? Since it is based on an Oracle DB so it should be possible to do a two-way replication. Or possibly make the synchronization policy yourself by database triggers.

Foldershare

[niai]

Foldershare is a Win32 "Document Management & Real-time File Mirroring Solution".

I read that "the development team hopes to start work on Mac OS X and Linux clients within the next six months" (Jan 27th 2004).

Might want to check out SFS (www.fs.net).

[Anonymous Coward]

It's cryptographically secure and similar to NFS, but possibly too slow for your purposes.