Journaling Filesystems and Network Mirroring?

CustomDesigned asks: "We are looking at the feasibility of mirroring all changes to selected filesystems to a hot backup over an internet or WAN link. This would provide a degree of protection for a business in the wake of a disaster like Sep 11. It seems that journaling filesystems have much of the work already done. All that is needed is a hook to copy log data into a message queueing system for delivery to the hot backup, and then running fsck for the unmounted file system to apply each log update. (This is more complex for ext3 where the log data is kept internally within the filesystem.) One problem is that JFS (and I presume ext3) only journal filesystem metadata. Has anyone seen a fully journaling filesystem? Is there any other work on remote hot backups for Linux? The toolset for any such capability should include a way to measure bandwidth required for a given filesystem without actually doing it. This would allow intelligent administrative decisions to balance bandwidth costs against traditional removable media backups."

Comments

I found something about this.

[Captain Pedantic]

I was looking on the internet, and I found this cool link which seems to cover what you are looking for.

Two other options...

[TRoLLaXoR]

Ignoring how feasible either options are...

XFS, formerly of SGI's IRIX OS, is a journaling file system for use with Linux (and soon *BSD).

Another option is to run BeOS, with its BFS file system, though as we all know, Be was bought our by Palm and the BeOS is in limbo, future development a dream of only the most rabid Beophile.

DRBD

[jackyb]

Take a look at Philipp Reisner's DRBD . It does more or less what you're after, although it doesn't do it using journalling filesystems.

You might best describe it as a sort of network RAID 1

Here's some useful information...

[Adrian Voinea]

The most important article:
Realtime data mirroring under Linux

And some other resources...
The Advanced Maryland Automatic Network Disk Archiver (AMANDA)
Creating Filesystem Backups with 'rsync'
Linux Backup

Wrong question

[coyote-san]

You're asking the wrong question

If you need to maintain a consistent filesystem image across multiple sites, you don't need a journaling filesystem. You need a distributed filesystem.

There's a huge difference. Journaling filesystems write a bit more information to the local disk before they report 'success' to the caller. Distributed filesystems write data to systems at multiple sites before they report 'success.'

This means that one of your key issues is network performance. Locally distribtued filesystems (over a 100 Mbps LAN won't show much of a performance hit, but you're going to notice it if you are writing to a distributed FS with nodes in multiple cities. For a lot of applications the latency is not an issue (e.g., if you're selling commodities and need a consistent time sequence everywhere.), but in other applications the latency will be unacceptable.

Another key issue is how you handle network partitioning. You need to be able to continue functioning when the network is down, or individual nodes are down, but that means you need to handle resyncing the systems.

The good news is that this is possible, but the bad news is that there's not a lot of good free software yet. CODA is probably your best bet, but I've heard some reports that it has some serious shortcomings. I think some of those problems are because the authors misunderstood what CODA was designed to do, but not all.

If you're willing to lose a day's worth of data you would be better off making nightly backups and fedexing them to remote locations. But be sure this is acceptable - there are many applications where any data loss is unacceptable.