Slashdot Mirror
NetBSD - Live Network Backup
dvl writes "It is possible but inconvenient to manually clone a hard disk
drive remotely, using dd and netcat. der Mouse, a Montreal-based NetBSD
developer, has developed tools
that allow for automated, remote partition-level cloning to occur
automatically on an opportunistic basis. A high-level description of the system has been posted at KernelTrap. This facility can be used to
maintain complete duplicates of remote client laptop drives to a server
system. This network mirroring facility will be presented at BSDCAN 2005 in Ottawa, ON on May 13-15."
This would be an extremely sensitive server system. With everyones harddrive image just waiting to be blasted to a blank harddrive, the potential for misdeeds is staggering. Even in an offical capacity, I really feel uneasy if my boss was able to take a copy of my harddrive image and see what I've been working on. Admittely, yes it should all be work but here we are allowed a certain amount of freedom with our laptops and I wouldn't want to have that data at my bosses fingertips.
On the flipside, this would be a boon to company network admins especially with employees at remote sites who have a hard crash.
Another reason to build a high speed backbone. Getting my 80GB harddrive image from Seattle, while I'm in Norfolk would be a lot of downtime.
-Teiresias
...when you get that idiot (and EVERY company has at least 1 of these guys) who calls you up asking if it's OK to defrag their hard-drive after downloading a virus or installing spyware. Then, when you tell them "NO", they just tell you that they did it anyways.
Now we can just hit a button and restore everything, a few thousand miles away.
The only thing left is to write code to block stupid people from reproducing.
IGB: More fun than eating oatmeal!
Assuming you can get around bandwidth monitoring, how long before this becomes incorporated into hacking tools. Add this to a little spyware and a zombie network and things get very interesting for poorly secured networks & computers.
I've been using der Mouse to copy files for years. First I user der Mouse to click on the file, then I use der Mouse to drag it to a new location!
Maybe setup is inconvenient. Remote backups using dd and ssh (our method) was a bit of a bear to initially setup, but thanks to shell scripting and cron and key agents, it hasn't given us any problems. I've seen a few guides with pretty straightforward and mostly universal instructions for this type of thing. That being said, I do hope this software will at least get people to start looking seriously at this type of backup since it lets you store a copy off-site.
This is a block level operation, whereas rsync is file level. With this system you can restore the disk image including partitions. Restoring from rsync would require you to create the partition, format the partition and the restore the files. Also, if you need the MBR...
As the article says, this is drive imaging whereas rsync is file copying.
I, too, immediately thought of German when I saw "der Mouse" (although in German it would be "die Maus", since Maus is feminine). Since they're located in Montreal, however, it seems unlikely that they'd be inclined to use German, and would be more likely to go for a French reference. So I ask, where does the "der" come from?
Ben Hocking
Need a professional organizer?
What's the fastest way to get a server running again after a disk crash? With rsync, if I backup /home and /etc, I still have to install and configure the OS and other software. That could take a significant amount of time (possibly days). Not to mention the time spent answering the phone (is the server down? when will it be back up?)
But if I have a drive image, I could just put it on a spare server and be back up and running almost immediately. That would require an identical spare server though.
What do the big enterprises who can't afford downtime do to handle this?
If you want something for OSX
I'd suggest either
CCC (Carbon Copy Cloner)
ASR (Apple System Restore)
Rsync
Radmind
Have fun on version tracker....
While this is cool, as I thought when I saw it on KernelTrap, disk mirroring is useful in situations where the hardware is less reliable than the transaction. If you have e.g., an application-level way to back out of a write (an "undo" feature), then disk mirroring is your huckleberry.
/.Recycler, whatever). Obviously the underlying routine would have to do its own garhage collection, deleting trash files by some FIFO or largest-older-first algorithm.
Most (all) of my quick restore needs result from users deleting or overwriting files - the hardware is more reliable than the transaction. I do have on-disk backups of the most important stuff, but sometimes they surprise me.
I'd like a system library that would modify the rename(2), truncate(2), unlink(2), and write(2) calls to move the deleted stuff to some private directory (/.Trash,
Just a thought.
sigs, as if you care.
Novell Zenworks has had this capability for sometime in production environments. It also integrates with their management tools so it is easy to use on an entire network. To say this technology is newly discovered is a far cry from the truth. They also use Linux on the back end of the client to move the data to the server.
It is nice though to have something like this in the open source world though. Competition is good.
Maybe I should patent this. Ah well, I figure if I mention it now it should prevent someone else from doing so...
:)
I was thinking - I know how Ghost supports multicasting and such. I was thinking about how to take that to the next level. Something like Ghost meets BitTorrent.
Wouldn't it be great to be able to image a drive, use multicast to get the data to as many machines as possible, but then use BitTorrent to get pieces to any machines that weren't able to listen to the multicast (ie it's on another subnet or something) and to pick up any pieces that were missed in the broadcast, or get the rest of the disk image if that particular machine joined in the session a little late and missed the first part?
I think that would really rock if someone wanted to image hundreds of machines quickly and reliably.
I'm thinking it'd be pretty cool to have that server set up, and find a way to cram the client onto a floppy or some sort of custom Knoppix. Find server, choose image, and now you're part of both the multicast AND the torrent. That should take care of error checking too, I guess.
Anybody care to take thus further and/or shoot down the idea?
You can accomplish anything you set your mind to. The impossible just takes a little longer.
Just make sure the backup server is properly configured (or very nearly so) I guess.
Our nightly rsync backups have saved us many times from user mistakes (oops, I deleted this 3 months ago and I need it now), but we haven't had a chance to test our backup server in the event of losing one of our main servers. We figure we could have it up and running in a couple hours or less, since it's configured very closely to our other servers, be we won't know until we need it.
In most cases, file backups are better. Imaging a drive that is currently mounted writable and actively updated can produce a corrupt image on the backup. This is worse that what can happen when a machine is powered off and restarted. Because the sectors are read from the partition over a span of time, things can be extremely inconsistent. Drive imaging is safest only when the partition being copied is unmounted.
The way I make backups is to run duplicate servers. Then I let rsync keep the data files in sync on the backups. If the primary machine has any problems, the secondary can take over. There are other things that need to be done for this, like separate IP addresses for administrative access, and the network services being provided (so that the service addresses can be moved between machines as needed while the administrator can still SSH in to each one individually).
now we need to go OSS in diesel cars
not the same?
http://www.feyrer.de/g4u/
Use rsync and hardlinked snapshots. There are lots of examples out there. I rolled my own a while back, but if you want something relatively nicely polished and based on that idea, check out dirvish (I didn't find that until after I already had my system set up).
I really like having several months worth of nightly snapshots, all conveniently accessible just like any other filesystem, and just taking up slightly more than the space of the changed files.
Why on earth are people always so insistent on doing raw-level dupes of disks?
...decades!. It works great, and its trivial to pump over ssh:
/dev/rdsk/c0t0d0s0 | (cd /newdisk && ufsrestore f -)
/dev/rdsk/c0t0d0s0 | ssh user@machine 'cd /newdisk && ufsrestore 0f -' .. it even supports incremental dumps (see: "dump level"), which is the main reason to use it over tar (tar can to incremental with find . -newer X | tar -cf filename -T -, but it won't handle deletes).
First of all, it means backing up a 40GB with 2 GB of data may actually take 40GB of bandwidth.
Second of all, it means the disk geometries have to be compatible.
Then, I have to wonder if there will be any wackiness with things like journals if you're only restoring a data drive and the kernel versions are different...
I have been using ufsdump / ufsrestore on UNIX for
# ssh user@machine ufsdump 0f -
or
# ufsdump 0f -
So -- WHY are you people so keen on bit-level dumps? Forensics? That doesn't seem to be what the folks above are commenting on.
Is it just that open source UNIX derivative and clones don't have dump/restore utilities?
Do daemons dream of electric sleep()?
From the article, it sounds like they are using a custom kernel module to intercept all output to the drive. This would keep things from getting corrupted, yes?
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
Image backups certainly have their place for people who can understand their limitations. However, a good, automatic, versioning file backup is almost certainly a higher priority for most computer users. And under some circumstances, they might also want to go with RAID for home computers.
rsync is not scalable to large numbers of files. We set up a backuppc machine awhile ago, tried to rsync the entire backup set over to another machine... It was a miserable failure. Even if we didn't check for hardlinks, (which we have to, backuppc uses tons of hardlinks,) the rsync process completely saturated a gig of RAM before it even started syncing.
Now, rsync would have been fine if we'd unmounted the filesystem and done it on the raw partition. But there's a couple of problems with that:
It's not live. Not a big deal for us, since it's a backup machine to begin with, but still...
rsync doesn't do that. A couple of people have submitted patches to allow a flag for rsync to copy block devices as if they were files. They were tiny patches, but they were rejected out of a fear of users doing stupid things with them. I guess the usual Rsync Way is to duplicate the filesystem, so that devices are copied with mknod, not dd.
Don't thank God, thank a doctor!
RTFA: It responds to heavy load by making a log (journal?) of the blocks that need backing up, and then does them when the load is lesser. If you do it on swap, then you're insane and deserve whatever you get :)
:)
This is a good idea, even if its niche is small, but I'm interested in how it handles the encryption. If it doesn't allow key re-generation on the fly, HMACs, certificates (or at least PSKs) and other things we expect from modern (SSH, IPSec/IKE, etc) systems then it's not going to be very useful. And unless I missed something it's going to be difficult to tunnel through a system that does do these things.
Personally I use SSH to tunnel everything possible, especially from Windows where IPSec is a joke, and the thought of sending all of my disk writes over a security system that is any less secure is a worry. Just imagine the problems if a man in the middle (or just a sniffer) catches plaintext: they know what you're doing, they know the contents of what you're doing, and highly likely they know what to do to exploit what you're doing. It's a very good thing that system entropy under nix is stored in the kernel, not on disk
Sam ty sig.