Slashdot Mirror
NetBSD's Real-Time Network Backup
jschauma writes "One of NetBSD's developers, der Mouse, was interviewed by DaemonNews about his real-time network backup system (originally presented at BSDCan 2005), where changes to your local filesystem are automatically propagated to a backup server. In his interview der Mouse tells about his idea, how it works, and of course, how cool it is."
But hasn't Sun been doing this with Solaris for at least 3 years?
Any fool can criticise, condemn, and complain, and most fools do. - Benjamin Franklin
So we could have backup servers all over the world keeping track of disk write commands...
This is indeed very neat, but isn't it sorta how transactional databases have been working?
I also don't see how this solution is effectively any better than RAID... If anything, a backup server is more expensive than a second hard drive for a RAID system (though it may pay off eventually). I'd think the backup server would need to be maintained as well... and if your backup ever fails, it seems like it would require a lot to set up another.
There also seem to be a lot of limitations as far as network security, filesystems, encrypted files, etc go. Furthermore, I don't see how the bandwidth hit is worth it (though I guess that depends on where your priorities are).
Admittedly, I'm no expert on this topic... so am I totally missing something?
Capitalism: When it uses the carrot, it's called democracy. When it uses the stick, it's called fascism.
Here is the idea behind the setup I am currently using: Easy Automated Snapshot-Style Backups with Linux and Rsync.
Volume shadow storage is exactly this kind of incremental, real-time backup process. How does this differ technically from that? (Other than the fact that you can now dynamically back up your morning toast, which is useful if a slice goes up in flames...)
This idea is really cool, but implementing it by putting hooks into each device driver seems overly complicated. It also doesn't sound like they're any sort of priority setting for this or any type of data filtering.
Personally I'd like to see something like the MS filesystem in development that allows SQL calls to be run against it (not sure if there's any other filesystems that are similar). Query every 5 minutes for changed data that fits the backup parameters (within the system dir, the user's home dir, certain filetypes) and then transfer the data as the network isn't being used.
That would achieve the same thing, but more flexibly and without affecting normal use.
You have enemies? Good. That means you've stood up for something, sometime in your life. --Winston Churchill
Sounds like it's essentially a DoubleTake daemon for BSD, cool, I wonder how well it scales? Say if you wanted to fully mesh 10 or more servers or something. Sounds like it might come in handy for keeping the content in web farms in synch as well....
Obviously, RAID servers don't help you in the case of accidental deletion. And they certainly don't help if your whole computer gets blow up.
Still, you'd want to be careful with this, it would suck to back up all the temp files generated by random processes.
autopr0n is like, down and stuff.
This concept has been in play for years as a commercial product for Disaster Recovery, Veritas Volume Replicator (VVR).
Those crazy Germans.
quidquid latine dictum sit altum videtur.
How does this compare with Linux Network Block Device? Sounds very similar.
There are pretty mature commercial tools for this stuff, as well - Veritas' VVR replication comes to mind.
Shouldn't this technically be called a point in time recovery solution? When I think of a backup solution, I expect to be able to retrieve arbitrary files from an arbitrary point in time. Also, rather than mucking with the kernel, wouldn't it have been simpler to use the geom system?
Isn't this guy reinventing the wheel? Why not just run a RAID 1 setup using iSCSI? Wouldn't that accomplish the same thing a lot easier?
...how do you get ALL the data on the backup server to start with? Pushing the writes off to the backup server in real-time is identical to what the HP VA7410 SAN I work with does internally in RAID 1+0 except that this happens over the network. But how are the disks in the backup server ever going to get all the original filesystem data if that data already exists AFTER you build your backup server? Even if you have a log of writes, you can't reconstruct the data. You'll only be able to reconstruct recent changes.
-"...bad old ideas look confusingly fresh when they are packaged as technology" - Jaron Lanier (Digital Maoism on Edge.o
How is this any different from DRBD (http://www.drbd.org./
/dev/nbX). Every write is sent to the local 'lower level block device' and to the node with the device in 'secondary' state. The secondary device simply writes the data to its lower level block device. Reads are always carried out locally.
From the website:
DRBD is a block device which is designed to build high availability clusters. This is done by mirroring a whole block device via (a dedicated) network. You could see it as a network raid-1.
Each device (DRBD provides more than one of these devices) has a state, which can be 'primary' or 'secondary'. On the node with the primary device the application is supposed to run and to access the device (/dev/drbdX; used to be
If the primary node fails, heartbeat is switching the secondary device into primary state and starts the application there. (If you are using it with a non-journaling FS this involves running fsck)
If the failed node comes up again, it is a new secondary node and has to synchronise its content to the primary. This, of course, will happen whithout interruption of service in the background.
And, of course, we only will resynchronize those parts of the device that actually have been changed. DRBD has always done intelligent resynchronization when possible. Starting with the DBRD-0.7 series, you can define an "active set" of a certain size. This makes it possible to have a total resync time of 1--3 min, regardless of device size (currently up to 4TB), even after a hard crash of an active node.
In every case I've actually needed backups to date, I find that, if I did them instantly instead of nightly, I'd end up losing data. The most common need for a backup for me comes when I've made a mistake with the main data, and I need to go back to what I had, say, yesterday.
This isn't to say that instant backups wouldn't be nice for failover architectures, though. I just don't deal with systems that large, yet.
How does this protect against an rm -rf against the filesystem... I guess it would trash the backup on the other side.
Every time the server is started, it sends a command to all the clients causing a full sync of all changes that occured while the server was offline. The same thing happens when a client is restarted, it sends a full sync to the backup server, any blocks that do not match the client checksum are re-sent.
Thus the first time you ran this thing it would copy the whole disk image to the backup server. After that subsequent writes would be the only output.
This is basically RAID over the network. Personally, I can't see a lot of use for it... Just put the second drive in the machine, and use software RAID, rather than putting the second drive in a network server. Less network slowdown and congestion that way, not to mention CPU-time wasted packetizing, encrypting, etc.
As always, RAID (and now this) is not a backup solution.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
Larry Robertson came up with this concept in the mid 80s as I recall, implemented it for VMS way back then as a remote shadowing system. He told me about it in one of the Anaheim DECUS meetings back then, published it (his company being called Bear Software back then). While the idea was not patented, the idea of moving updates wide area and doing local journalling so that the "shadow" needed only to keep up with average write I/O rates, rather than peak write rates, was AFAIK new back then and he deserves credit for thinking of it and implementing it. If anyone should try to patent it, he could be contacted also to show prior invention and public description. (Another outfit that came out shortly later with something similar had, as I have been told, a copy of the Bear program in house. That suggests my belief is correct that Robertson came up with the idea first, and that duplicate invention did not occur there.)
It's almost a troll to even mention it, since there are so many things pioneered by Multics....
Oh no! the rootkit got replicated to the backup server!
It seems to me that if you use a journalling filesystem that journals everything, not just meta-data, you can just send the journal logs off to your backup device. Presuming your backup device starts with the same baseline data (i.e. a full level-zero dump) Then you would have the ability to restore your files, or entire filesystem, to the state it was at any point in time just by playing back the journal logs. Presumably a "smart" replay algorithm could be implemented that would use some sort of regular snapshots and "colescing" (sp?) of the logs to speed up restores (reducing the time spent doing the journal replay).
Such an approach would not require hooks into each device driver either, it would be entire at the filesystem level.
It needs to do a full rescan on reboot?
UGH!
That kills it for me.
I just wanted to point out that there are several FUSE-based filesystem implementations that do the same thing (functionally, not implementation-wise) and they do not require hooks in the device drivers -- they don't even care what the filesystem is for the original or the backup.
And, yes, RAID is a very good solution if you've got the money and are smart enough to recognize when a disk fails...
This article makes me laugh...get a life and don't troll Slashdot, kthx.
Nerdspeak.net
So what you have is not only a real-time backup, but also the ability to unwind any possibly damage after a break-in or other event. If your backup server is only running a journalling client, it can be made extremely secure, and also provide an excellent auditing tool.
It would also be possible to delay the streams and have arbitrarily old filesystems available. Or to use a local journal as a buffer to smooth out the IO load as it is piped off site. It could also be used to augment a non-journalling filesystem, for crash recovery purposes, assuming your filesystem provides at least some consistency guarantees. In fact Netapp does something similar by logging FS operations to NVRAM, while the filesystem only writes consistentcy points periodically.
Although I haven't read about the NetBSD work, I am sceptical that they could get the error handling to work correctly at that level. With the DragonFly journalling, there is support for transactional consistency, as well as recovering from interrupted network connections. While it is not complete, much of it is in place and functional.
See the mountctl manual page for attaching a journal to a mount, and the jscan manual page for processing the journal.
Wouldn't this also replicate deletes across to the offsite machine in near-real-time? So if one were to accidentally delete a file, or a $HOME directory, or a complete filesystem, then there would be no way to recover from this from the "backup" machines, because their files would have gotten nuked too?
Hey, Windows users, there is no such thing as "forward" slash, there is only slash and backslash.
NetApp has been doing somthing similar for a very long time. A lot of people use the Sun boxes on the frontend to boot or attach to the storage appliance and let it do the backups. It saves space and saves the server from having to do it.
oh, wait, what?
trustedworlds.net - gaming, security, and the gunk that lives in between
You are missing a few things like fire, flood, lightning etc.
with the Coda filesystem. Or am I missing something?
LedgerSMB: Open source Accounting/ERP
Heh, when I was reading this the fortune that appeared at the bottom of the page - right underneath your comment - was:
Passionate hatred can give meaning and purpose to an empty life. -- Eric Hoffer
Cute coincidence.
--Ryv
rsync -avz ~/ user@remote:homebackup
in crontab?
I don't want to read
Actually, he's been around as Der Mouse since I was in college (circa 1985). I ran the xterm replacement he wrote back then. Long, long before Theo had his hissy fit and forked off OBSD. Of course, a trivial Google would have shown that, but hey, an AC would want to miss out on an ad homen flame...
Ever considered using RAID1 in a backup system? I've not tried it, but it isn't difficult to see how it can be implemented.
Think of it in a similar way as a tape-based backup system...
1) implement RAID1,
2) have many spare disk drives (they're cheap now),
when you want a snapshot backup :
3) 'fail' one of the drives,
4) remove it,
5) install spare drive and add it to the RAID1 (it'll rebuild automatically),
6) take 'failed' drive off site/lock it in safe/whatever you would do with a tape
I'm not entirely sure how one would restore, but it should be fairly easy.
Max.
If you aren't looking for network functionality, there's a filesystem called ext3cow that lets you roll back to older versions of the contents of the filesystem.
Any program relying on (nontrivial) preemptive multithreading will be buggy.
You built this yourself? How do you handle differential compression through a web browser? How do you compare file signatures? Handle permissions?
- Essentially the same as Linux' md plus AoE (ATA over Ethernet), which is also built into the kernel, but more modular
- Essentially the same as Linux' md plus Linux' nbd (network block device), which is also built into the kernel, but more modular
- Built into the kernel when it could be a daemon that is notified of changes by the kernel instead (don't know if NetBSD's kernel does this, others do)
- It's not a backup solution but a RAID: It only protects you from disk failures, not from brain failures, something that backup solutions can do
I work at a research institute where we run a cluster of Linux workstations that boot from a server and keep all their data on a file server. Both servers are Linux boxes that use drbd to keep their configuration synchronized. The file server is a set of two front-end machines (one active, one waiting) that additionally use md+nbd to create a network RAID-1 over many back-end nodes. Each back-end node has a RAID-1 of local disks. This means that each disk in each back-end is redundant and each back-end is redundant, so that all data is stored four times instantly upon each write (yes, we do have a separate backup solution), distributed over two separate floors in different parts of the building. And this was implemented by one single totally underpaid student (granted, he's good...)!
So tell me again what's new about TFA.
Yay for you. Now go back to masturbating over pictures of tux and stop posting trolls with links that are 5 years old.
Now this is what bugs me. Why do you always assume that it's Linux users trolling you? As you have demonstrated, BSD zealots are so eminently trollable that literally anyone can have a go. Even an MCSE could probably generate tens of angry replies with a cut & pasted troll.
It's quite likely that you are in fact trying to insult a script. Have some self respect for fuck's sake.
until you can write a program in Lua to give me a BJ, speak for yourself
Dealing with the death of an operating system close to you can be one of the most traumatic experiences of your life, and you're bound to go through a range of emotions. While you may be able to work through those feelings on your own, it's often helpful to talk to a friend, a family member, or a counselor. You might also seek out a support group for people who are grieving.
... when Windows 9x died I felt nothing but an overwhelming feeling of relief, and a certain sense of vindictiveness.
Funny
The higher the technology, the sharper that two-edged sword.
Java applets can do darn near anything.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
Do you have a link to any code? Or references on the net? What you describe is a significant improvement over rsync/rdiff-backup for mobile users, and I'd like to know more.
I haven't yet decided if I'll opensource the system or not. I'm leaning towards opensourcing the client portion (the Java applet) but licensing the server software out. Or maybe I'll go with the new GPL if it protects my rights on software that is used as a service rather than distributed.
I've done a few test runs which was enough to let me know I had to look into getting a multi-TB server farm before I could open it to the public. I've been trying to get investors for that (it costs around $250/mo per TB of server space) but looks like Google is going to kill my business plan.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.