Slashdot Mirror
Journaling Filesystems and Network Mirroring?
CustomDesigned asks: "We are looking at the feasibility of mirroring all changes to selected filesystems to a hot backup over an internet or WAN link.
This would provide a degree of protection for a business in the wake
of a disaster like Sep 11. It seems that journaling filesystems have
much of the work already done. All that is needed is a hook to copy
log data into a message queueing system for delivery to the hot
backup, and then running fsck for the unmounted file system to apply
each log update. (This is more complex for ext3 where the log data
is kept internally within the filesystem.) One problem is that JFS
(and I presume ext3) only journal filesystem metadata. Has anyone
seen a fully journaling filesystem? Is there any other work on remote
hot backups for Linux? The toolset for any such capability should
include a way to measure bandwidth required for a given filesystem
without actually doing it. This would allow intelligent
administrative decisions to balance bandwidth costs against
traditional removable media backups."
Take a look at Philipp Reisner's DRBD . It does more or less what you're after, although it doesn't do it using journalling filesystems.
You might best describe it as a sort of network RAID 1
According to this document, ext3 supports three different journalling modes, including one where all data is written to the journal, rather than just the metadata.
InterMezzo shold be able to do this.
include $sig;
1;
In the world of stock prices not in danger of being delisted, Some investors, like the ones who invest in my company, may not like the idea that you're mirroring your critical data with a relatively new, and "unheard of" filesystem. Be very careful. Remember, in this economy it's not about what you believe in; its what your investors want you to believe in. The political metalayer of the OSI model has become the most important. Then again, your data may not consist of customers financial data in bulk quantities. YMMV.
If politics and such are important in your company, Look into "real" technologies such as EMC (Sym or Clariion), Hitachi, or IBM storage, even if you have no real intention of buying. Make a brief outline as to why these systems would not be cost-effective, or fit your needs, just so your investors will know you put some thought into it.
I'm not trying to troll, i'm just offering information you may want to consider. Again, YMMV
Just my unsolicited $0.02
While the current story could very quickly be resolved by a Google search, at least it's an interesting topic. I really got pissed when Cliff posted the one a couple days ago, something like, "Ask Slashdot: Don't you just HATE it when people don't QUOTE properly in E-MAIL?" When I saw that, I thought they should change the tag line to "News for Nerds, Stuff that Matters, and Geeks' Petty Annoyances."'
include $sig;
1;
The most important article:
Realtime data mirroring under Linux
And some other resources...
The Advanced Maryland Automatic Network Disk Archiver (AMANDA)
Creating Filesystem Backups with 'rsync'
Linux Backup
Veritas Volume Replicator (VVR) is a nice product. Works great with VxFS and VxVM (Veritas File System and Veritas Volume Manager).
Of course you have to pay for them. But you get what you pay for.
--NBVB
p.s. (VxFS rocks. Saved my bacon more times than I can count!)
You're asking the wrong question
If you need to maintain a consistent filesystem image across multiple sites, you don't need a journaling filesystem. You need a distributed filesystem.
There's a huge difference. Journaling filesystems write a bit more information to the local disk before they report 'success' to the caller. Distributed filesystems write data to systems at multiple sites before they report 'success.'
This means that one of your key issues is network performance. Locally distribtued filesystems (over a 100 Mbps LAN won't show much of a performance hit, but you're going to notice it if you are writing to a distributed FS with nodes in multiple cities. For a lot of applications the latency is not an issue (e.g., if you're selling commodities and need a consistent time sequence everywhere.), but in other applications the latency will be unacceptable.
Another key issue is how you handle network partitioning. You need to be able to continue functioning when the network is down, or individual nodes are down, but that means you need to handle resyncing the systems.
The good news is that this is possible, but the bad news is that there's not a lot of good free software yet. CODA is probably your best bet, but I've heard some reports that it has some serious shortcomings. I think some of those problems are because the authors misunderstood what CODA was designed to do, but not all.
If you're willing to lose a day's worth of data you would be better off making nightly backups and fedexing them to remote locations. But be sure this is acceptable - there are many applications where any data loss is unacceptable.
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
Why aren't you looking into things like rdist/rsync, if you're just looking to mirror data? Either is a decent tool for the job, as long as you can live with the possibility of a tiny amount of data loss if you lose a machine between synchs.
- A.P.
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"