Slashdot Mirror
Journaling Filesystems and Network Mirroring?
CustomDesigned asks: "We are looking at the feasibility of mirroring all changes to selected filesystems to a hot backup over an internet or WAN link.
This would provide a degree of protection for a business in the wake
of a disaster like Sep 11. It seems that journaling filesystems have
much of the work already done. All that is needed is a hook to copy
log data into a message queueing system for delivery to the hot
backup, and then running fsck for the unmounted file system to apply
each log update. (This is more complex for ext3 where the log data
is kept internally within the filesystem.) One problem is that JFS
(and I presume ext3) only journal filesystem metadata. Has anyone
seen a fully journaling filesystem? Is there any other work on remote
hot backups for Linux? The toolset for any such capability should
include a way to measure bandwidth required for a given filesystem
without actually doing it. This would allow intelligent
administrative decisions to balance bandwidth costs against
traditional removable media backups."
I was looking on the internet, and I found this cool link which seems to cover what you are looking for.
None are more hopelessly enslaved than those who falsely believe they are free. Johann Wolfgang von Goethe.
Ignoring how feasible either options are...
XFS, formerly of SGI's IRIX OS, is a journaling file system for use with Linux (and soon *BSD).
Another option is to run BeOS, with its BFS file system, though as we all know, Be was bought our by Palm and the BeOS is in limbo, future development a dream of only the most rabid Beophile.
Take a look at Philipp Reisner's DRBD . It does more or less what you're after, although it doesn't do it using journalling filesystems.
You might best describe it as a sort of network RAID 1
According to this document, ext3 supports three different journalling modes, including one where all data is written to the journal, rather than just the metadata.
InterMezzo shold be able to do this.
include $sig;
1;
In the world of stock prices not in danger of being delisted, Some investors, like the ones who invest in my company, may not like the idea that you're mirroring your critical data with a relatively new, and "unheard of" filesystem. Be very careful. Remember, in this economy it's not about what you believe in; its what your investors want you to believe in. The political metalayer of the OSI model has become the most important. Then again, your data may not consist of customers financial data in bulk quantities. YMMV.
If politics and such are important in your company, Look into "real" technologies such as EMC (Sym or Clariion), Hitachi, or IBM storage, even if you have no real intention of buying. Make a brief outline as to why these systems would not be cost-effective, or fit your needs, just so your investors will know you put some thought into it.
I'm not trying to troll, i'm just offering information you may want to consider. Again, YMMV
Just my unsolicited $0.02
People interested in backup approaches might be interested in plan9's upcoming backup system Venti :
From: Sean Quinlan To: 9fans Mailing list
For those of you interested in the direction we are heading
with respect to plan 9's file system, you might want to
checkout our paper on Venti that will appear in the
USENIX fast conference.
http://www.cs.bell-labs.com/~seanq/pub.html#venti
Venti is a block level storage server that replaces the optical
juke box for a plan 9 file system. Some of the benefits include:
coalescing of duplicate blocks
compression
no block fragmentation
Also, we have switched from optical to magnetic disks as the storage
technology. I know many of you already use magnetic disks to
"fake" a worm, but for those of us using a optical juke box,
the performance improvement is rather substantial!!
seanq
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
http://www.miralink.com
This is a device which emulates a standard SCSI hard drive. Plug it into your host adapter, and any data written to it is automatically mirrored to a remote location over the Internet.
The most important article:
Realtime data mirroring under Linux
And some other resources...
The Advanced Maryland Automatic Network Disk Archiver (AMANDA)
Creating Filesystem Backups with 'rsync'
Linux Backup
Veritas Volume Replicator (VVR) is a nice product. Works great with VxFS and VxVM (Veritas File System and Veritas Volume Manager).
Of course you have to pay for them. But you get what you pay for.
--NBVB
p.s. (VxFS rocks. Saved my bacon more times than I can count!)
I know this will mirror the entire lot, not just the changes, but why not use RAID and the Network Block Device?
http://blog.grcm.net/
You're asking the wrong question
If you need to maintain a consistent filesystem image across multiple sites, you don't need a journaling filesystem. You need a distributed filesystem.
There's a huge difference. Journaling filesystems write a bit more information to the local disk before they report 'success' to the caller. Distributed filesystems write data to systems at multiple sites before they report 'success.'
This means that one of your key issues is network performance. Locally distribtued filesystems (over a 100 Mbps LAN won't show much of a performance hit, but you're going to notice it if you are writing to a distributed FS with nodes in multiple cities. For a lot of applications the latency is not an issue (e.g., if you're selling commodities and need a consistent time sequence everywhere.), but in other applications the latency will be unacceptable.
Another key issue is how you handle network partitioning. You need to be able to continue functioning when the network is down, or individual nodes are down, but that means you need to handle resyncing the systems.
The good news is that this is possible, but the bad news is that there's not a lot of good free software yet. CODA is probably your best bet, but I've heard some reports that it has some serious shortcomings. I think some of those problems are because the authors misunderstood what CODA was designed to do, but not all.
If you're willing to lose a day's worth of data you would be better off making nightly backups and fedexing them to remote locations. But be sure this is acceptable - there are many applications where any data loss is unacceptable.
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
Why aren't you looking into things like rdist/rsync, if you're just looking to mirror data? Either is a decent tool for the job, as long as you can live with the possibility of a tiny amount of data loss if you lose a machine between synchs.
- A.P.
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"