Slashdot Mirror
Volume Shadow Copy For Linux?
An anonymous reader writes "I was asked to manage a number of Linux servers at work. I would like to use volume snapshots to improve my backup scripts and keep recent copies of data around for quick restore. I normally manage Windows servers and on those I would just use Microsoft's Volume Shadow Copy for this. I tried Linux LVM snapshots, but most of the servers I manage run regular partitions with ext3 file systems, so LVM snapshots will not work. I found some versioning file systems out there like ext3cow and Tux3. Those look interesting, but I need something I can use on my existing ext3 file systems. I also found the R1Soft Hot Copy command-line utility, but it does not yet support my older 2.4 Linux servers. What are you using to make snapshots on Linux?"
LVM snapshots only work fine if you are using LVM. I think the OP uses "regular partitions" to mean no volume manager in use.
I will admit that I have not tried it on Linux, but zfs is the best of the next gen filesystems. It does cryptographically assured reads and writes (remember that transitory undetected disk malfunctions occur at a rate of ~1/TB of data), it can snapshot changes, it fricken slices bread. If it had a gender, I would probably marry it (well, I guess I can date it for a while and see how things work out). http://zfs-fuse.net/
If people can connect to one another even the smallest of voices will grow loud.
--Serial Experiments Lain
RSnapshot uses a clever blend of rsync + hard links to do what you want... you can store many incremental backups in just a little more space than a full backup. you can run rsnapshot on a backup server with lots of disk space, and all you need to expose on your target machines is SSH.
you'd create "backup" users on all the target hosts, generate a PKI key pair, and put the private key on your backup server. put the public key in the "backup" account on each target machine so the backup server can securely login without a password. then you just set up rsnapshot to log into your targets and it will use rsync-over-ssh to pull the data.
http://rsnapshot.org/
Oh shit! I forgot to click "Post Anonymously"...
If you have backups, then moving to LVM is obviously the way to go if you desire snapshots. The others options are short-term hackery, LVM was designed from the ground up to do such things. And Ext3 has nothing to do with the price of butter.
To clarify, let me rephrase your question for the other way around
"I was asked to manage a number of *Windows* servers at work. I would like to use volume snapshots to improve my backup scripts and keep recent copies of data around for quick restore. I tried Windows Shadow Copy, but most of the servers I manage run MBR partitions with FAT file systems, so Shadow Copy will not work. I found some versioning file systems out there... Those look interesting, but I need something I can use on my existing FAT file systems. I also found --random freeware--, but it does not yet support my older Windows NT 3.5 servers. What are you using to make snapshots on Windows?"
Except, in that case, it makes more sense because the filesystem is the determining factor, not the volume management. If you have LVM, it doesn't matter what the underlying filesystem is, really. Stop faffing about - if you have a server, with backups, that you need snapshots on, take the hit and wipe the drives to a config that supports that... while you're there upgrade that damn kernel already. If nothing else, it will test that the backups you're making are actually worth the effort. It's like complaining that 95 on FAT16 doesn't support Shadow Copy. If you absolutely *can't* take those servers down, or am unable to restore your backups to another machine for testing such changes (whether because of compatibility, software licensing and/or bad backups), you have bigger problems than some random desire for a feature you don't actually *need* at the moment.
If they are indeed regular partitions, he can't use LVM snapshots. However, the best solution is to convert from partitions to LVM volumes. It's a little effort to do so, but switching is worth it. Second best is to wait until btrfs is more usable. As a ZFS user, I can say that filesystem-level snapshot are much nicer than LVM snapshots in lots of ways.
Another possibility is to abuse hardlinks. You can create a copy of a directory with cp -al, and then overwrite (not modify) files on the original, you'd have copy-on-write copy. If you make your backups with rsync, you can configure rsync to never write to existing files and always overwrite, then use cp -al each weak or day, to store "incremental" backups for weeks and maybe more. I personally found this solution nice, but then I installed Solaris on the backup machine and used ZFS snapshots which do the same safer, simpler and more efficiently. If the backups are stored on a separate machine, switching it to Solaris is an option.
Another thing that can be done is to keep the LVM snapshots on a separate machine, and leave the current partitions as they are. It can be done with rsync, or a drdb device can be used to sync with the server (it can be created without reformatting the partitions, but you still need to make some changes like shrinking and/or moving the data, which might destroy the data if you don't know what you're doing).
Only works if the partitions don't change while you are copying them. The big advantage of using LVM for this is that you can create snapshots on a live system, without resorting to remounting the partition read-only (and all the problems that will cause).
But really, those are his only options. If you insist on using plain ext3 and won't add a layer of between the FS to allow for this, then you have no choice but to freeze the partition while doing a volume-level back up.
There are Enterprise Linux distributions that are both supported and still run 2.4, though not for much longer. Not everyone runs Ubuntu.
"I use a Mac because I'm just better than you are."
People expect a snapshot to be immediately usable and reliable, however in practice a state of device, even if synchronized with filesystem through its transaction is not a state of all data -- some data may be in buffers, prepared to be written, and rebooting into a restored filesystem may require some cleanup of such state. In particular, SQL databases are completely unsuitable for this kind of backup (this is why they have their own backup and transaction log handling procedures), and database-like applications such as mail servers, may require reindexing.
However for purposes other than those applications, file-level backup is entirely adequate, so utilities like rdiff-backup end up providing more functionality than complicated snapshot-handling procedures -- incremental backups for subtrees, readable trees in backup media, etc.
It also should be noted that backups should not be used as a replacement of package management -- on Linux anything installed through a package manager can be uninstalled through it.
Contrary to the popular belief, there indeed is no God.
You currently have ext3 fs that are NOT on LVM. In the future, choose LVM.
The choice isn't that simple. LVM comes with its own complications, including a tendency to get volume offsets "wrong" so the file system data doesn't align nicely to RAID stripes. This is not good for performance.
Also, LVM has only recently acquired barrier support, and the combination of no barriers + write cache can be quite dangerous if power is lost. Even battery backed cache won't save you if you use a journalling file system (and everybody does these days) because request ordering isn't guaranteed.
I haven't touched Solaris since it had a 2 in front of its version number, but I must admit that I suffer from ZFS envy.
Finally! A year of moderation! Ready for 2019?
I'm troubled why people still run 2.4 server. I remember the time when I was reluctant to upgrade to 2.6, and I used prefer the older 2.4, which felt more comfortable than 2.6, regardless of how tempting the new changes sounded. But now I don't see any reason I would run this anywhere, even my router runs 2.6. Especially on newer hardware, 2.4 is really really too old.
I know there are people who probably still run Linux 2.2, but that are probably systems that are running some task well enough to require any changes, and leaving them as they are is the best. Servers are usually not like that. They need security updates, upgrades to catch up with the times, and many other changes required by the circumstances (for example, adding snapshot abilities, for which some person asked recently on Slashdot). Most production servers are not systems that you just leave running, so upgrades to the kernel are also expected and highly recommended. Not to mention that most recent distributions require 2.6.
Just upgrade your kernel using a manual build of the 2.6 kernel.
Also install static versions of the modutils ( insmod, modprobe, etc )
Use an external ( a machine with decent software ) for this so your compile doesn't break.
I have done so in the past and it works fine. ( and plan an update for those machines, anything with 2.4 is way to old... )
After that you can just use R1Soft hot copy,
http://www.r1soft.com/tools/linux-hot-copy/
This program is free ( as in beer ) to download and works with every block device.
You can even write to a block device if you really need to.
Their commercial offerings are pretty good as good. ( and DO work with the 2.4 kernel )
We use it here at work.
I heard btrfs supports something like this as well.
Any way, good luck!
Indeed. I've done so, and documented it:
http://www.tolaris.com/2010/05/06/moving-an-existing-backuppc-partition-to-lvm/
People smarter than me documented a way to move the data, live, even when the partition is nearly full. See the comments there.
Yes, it's very old. They also talk about using cvs for version control, and mention that that world has moved on to svn, and the world has moved on a couple of times since then even. We also use Nagios rather than more ancient monitoring software. But still the central ideas are sound, even with many details changed. And practical, too.
These ideas actually apply very much to cloud infrastructure. It's really all about the cloud-- considering a machine not as just "a machine", but instead thinking of it as having a base image with certain functionality bolted on top of it. Thinking of a machine not just as a machine, but as a replaceable/exchangeable component in a larger collective system. That essentially is cloud computing. The thing a lot of people don't consider is that even a smaller cluster of machines should/could be configuration managed, maintained, and viewed this way.
+1.
u r doing it rong.
If you need to keep around such old software, it needs to be running inside Xen/VirtualBox and/or become NFS-booted so that it's insulated from the hardware. That way, you're not forced to keep around old hardware to run your old software. If you insulate with Xen/VBox at the block level you can use LVM2 on the host system to do snapshots but are still constrained by the legacy filesystem. If you NFS-boot, you can use future filesystem-level snapshottable Linux filesystems to do snapshots, or you could buy a NetApp and use proprietary software to do the snapshots, or Solaris (if it sticks around), or any of a variety of things. You can argue about which level of storage insulation is best in the long run: the filesystem level has certain advantages at least with ZFS and probably with whatever future Linux snapshotting filesystem comes along becuase of variable block size: small files get blocks in 512-byte increments, and any file larger than 128kB gets a multiple-of-128kB sized block. Since blocks are the unit of compression and deduplication, you want the largest size block possible, but not too large or you suffer from the read-modify-write RAID5-like tax. A larger block will compress better and takes only one entry in the deduplication hash table. If you insulate at the block layer then all blocks will have to be the same size and a relatively small size, which makes compression and dedup work not nearly as well because they can't give big blocks to big files. however for old software you may want to change as little as possible, and especially with sketchy linux 2.4 NFS clients maybe it's not safe to run certain apps over an NFS root, or maybe your ancient distribution doesn't support NFS booting well.
One way or another, though, you need to find a way to keep the apps running while minimizing the blocks of ancient code on which you're still dependent, and this should be your overriding concern. You need to structure your plans to encapsulate the code subject to bit-rot rather than flailing around on /. for some freshmeat app-of-the-day that claims to solve * with FUSE and some stupid Perl script. This is the difference between a serious professional douchebag who surveys the industry and can smell the difference between high and low quality, and a pathetic flailing do-my-homework-for-me everything-inclding-windows-has-stengths-and-weaknesses-right-tool-for-the-job doomed blinking medicated idiot douchebag. Do not be that douchebag.
LVM snapshots also just aren't all that good.
They require you to pre-allocate space, so you have to guess how much copy-on-write difference will accumulate between the original and snapshot over the lifetime of the snapshot.
If the snapshot runs out of space, it *should* cleanly disable its self. Pity about the file system mounted from it that has no idea its backing block device just vanished. It gets messy, fast, when an LVM snapshot runs out of storage.
LVM snapshots are really inefficient, because they track all block changes, not just user-level file data and metadata. This massively bloats the snapshot, reducing how long it lives until it runs out of backing store and disables its self.
LVM snapshots don't share backing store. If you have three of them, snapshot t+3 has to store all the data snapshot t+2 and snapshot t+1 do, and so on. The differences between the real fs (t) and snapshot t+1 land up being stored three times in three separately allocated backing stores. You waste a HUGE amount of space this way, and it's hard to reliably predict how much you need so your snapshots often vanish out from under you are you're trying to use them.
LVM is useful, but for someone used to the Volume Snapshot Service (VSS) on Windows servers, to ZFS, or to any of the "enterprise-y" file systems often seen deployed with big SANs, it's just going to make them cry.
If you use --link-dest=DIR, rsync will hard link to any files that are identical, so you can have snapshots that are accessible as an entire tree but that consume little more space that a snapshot delta.
Shadow copies are not about server reliability, they're about stupid users inadvertently overwriting or deleting wrong files, and allowing them to fix their mistakes themselves, without needing to access the backup system or bothering the system administrators.
Also, versioning filesystems make a copy-on-write, so there's a backup of *every* version of a file, and not only the versions that are there when the backup (or shadowcopy) runs. It's been only this week that we've been looking fruitlessly through the backups for some vanished files. We can only assume that the files were erroneously deleted on the same day they were created, before the backups had a chance of picking them up.
What a depressingly stupid machine.