Volume Shadow Copy For Linux?

An anonymous reader writes "I was asked to manage a number of Linux servers at work. I would like to use volume snapshots to improve my backup scripts and keep recent copies of data around for quick restore. I normally manage Windows servers and on those I would just use Microsoft's Volume Shadow Copy for this. I tried Linux LVM snapshots, but most of the servers I manage run regular partitions with ext3 file systems, so LVM snapshots will not work. I found some versioning file systems out there like ext3cow and Tux3. Those look interesting, but I need something I can use on my existing ext3 file systems. I also found the R1Soft Hot Copy command-line utility, but it does not yet support my older 2.4 Linux servers. What are you using to make snapshots on Linux?"

Comments

nilfs

[pinkishpunk]

Nilfs will have those things, but you`ll have to wait till its production ready.

Re:nilfs

[Chainsaw]

Nerds I'd Like to... You know what - nevermind.

Re:nilfs

[HogGeek]

Nerds I'd like to file system?

I don't get it... /sarcasm

Re:nilfs

[Anonymous Coward]

...fsck, of course.

You're confused

[suso]

I tried Linux LVM snapshots, but most of the servers I manage run regular partitions with ext3 file systems, so LVM snapshots will not work.

Last time I checked they worked fine. I think you mean something else by "snapshots".

Re:You're confused

[pbowen]

LVM snapshots only work fine if you are using LVM. I think the OP uses "regular partitions" to mean no volume manager in use.

Re:You're confused

[verbatim_verbose]

I think you might have misread what he said.

ext3 file systems on regular partitions wouldn't be able to use LVM snapshots, because, well, they're not running on LVM volumes.

Re:You're confused

[suso]

Nevermind, I was the one that was confused. You currently have ext3 fs that are NOT on LVM. In the future, choose LVM.

Re:You're confused

[Anonymous Coward]

LVM snapshots may work fine for LVM volumes, but not for straight ext3 partitions. converting to LVM requires repartitioning.

Re:You're confused

[yargnad]

Thanks for elaborating, we will try your suggestions and report back.

Re:You're confused

[nschubach]

Isn't that like complaining that your FAT32 partitions in Windows are not supported by Shadow Volume Copy then?

I think there's a bit of double standard here in the question. The OP is stating that they want to use a feature on an older server (2.4 kernel?) that isn't available unless you update the server, reformat, or what have you. The same applies to a Windows environment.

I think the question is mis-guided. They should be asking for a SVC like feature for kernel 2.4 and ext3 systems.

Re:You're confused

[impaledsunset]

If they are indeed regular partitions, he can't use LVM snapshots. However, the best solution is to convert from partitions to LVM volumes. It's a little effort to do so, but switching is worth it. Second best is to wait until btrfs is more usable. As a ZFS user, I can say that filesystem-level snapshot are much nicer than LVM snapshots in lots of ways.

Another possibility is to abuse hardlinks. You can create a copy of a directory with cp -al, and then overwrite (not modify) files on the original, you'd have copy-on-write copy. If you make your backups with rsync, you can configure rsync to never write to existing files and always overwrite, then use cp -al each weak or day, to store "incremental" backups for weeks and maybe more. I personally found this solution nice, but then I installed Solaris on the backup machine and used ZFS snapshots which do the same safer, simpler and more efficiently. If the backups are stored on a separate machine, switching it to Solaris is an option.

Another thing that can be done is to keep the LVM snapshots on a separate machine, and leave the current partitions as they are. It can be done with rsync, or a drdb device can be used to sync with the server (it can be created without reformatting the partitions, but you still need to make some changes like shrinking and/or moving the data, which might destroy the data if you don't know what you're doing).

Re:You're confused

[nschubach]

Replace: "Volume Shadow Copy" and "VSC" respectively in my post above. Sorry.

Re:You're confused

[amorsen]

You currently have ext3 fs that are NOT on LVM. In the future, choose LVM.

The choice isn't that simple. LVM comes with its own complications, including a tendency to get volume offsets "wrong" so the file system data doesn't align nicely to RAID stripes. This is not good for performance.

Also, LVM has only recently acquired barrier support, and the combination of no barriers + write cache can be quite dangerous if power is lost. Even battery backed cache won't save you if you use a journalling file system (and everybody does these days) because request ordering isn't guaranteed.

I haven't touched Solaris since it had a 2 in front of its version number, but I must admit that I suffer from ZFS envy.

Re:You're confused

[nschubach]

The double standard being that the Linux servers wouldn't need updated where the Windows servers would. There's an update that has to happen to support the feature. Linux is not immune to this (though it would likely do the update without a total rebuild opposed to Windows.)

Re:You're confused

[Jason Earl]

With judicious use of --link-dest rsync can do this for you without having to use cp at all.

Re:You're confused

[verbatim_verbose]

Um... I'm not really following anything you're saying.

Kernel 2.4 supports LVM, he just isn't using it.

Re:You're confused

[nschubach]

I tried Linux LVM snapshots, but most of the servers I manage run regular partitions with ext3 file systems, so LVM snapshots will not work.

They have, presumably, tried and failed. It could support it, but it would have to be installed, updated or something to get it to work. There's something they're not doing on an old server that needs to be changed to support the feature they want. I was pointing out how this is not exclusive to the Linux servers.

Re:You're confused

[nashv]

He isn't complaining. You seem to be responding to his mentioning that "he knows how to do this on Windows" , by interpreting it as "Why is Linux so broken that it can't do a simple thing like that?" This isn't a Linux versus Windows thing. This is a Windows user, migrating to Linux and wants to know how to accomplish something. Constructive answers are more useful in such cases than getting defensive by alleging hypocrisy and double standards.

Re:You're confused

[nschubach]

You are getting hung up on the choice of words.

Re:You're confused

[Crackez]

Can you elaborate on how LVM gets volume offsets "wrong" and it's impact on RAID performance? I've never heard of that before. Not saying you're wrong, it's just news to me.

Also, IIRC Solaris still has a 2 in front of it. Ie, Solaris 10 is really Solaris 2.10, and is simultaneously SunOS 5.10. They have retarded naming conventions. Oh well, Oracle can only fsck it up further...

ZFS does rock btw, we've got about 80-90TB (guesstimate) spinning with ZFS right now. Our thumper has 32.5GB alone, and another HP box running x86 Solaris has 45.3TB. There's others, but not as big as those two. ZFS really does make it a breeze to manage that much storage.

Re:You're confused

[evilviper]

It can be done with rsync, or a drdb device can be used to sync with the server (it can be created without reformatting the partitions, but you still need to make some changes like shrinking and/or moving the data, which might destroy the data if you don't know what you're doing).

DRBD does not require resizing partitions. The meta-data can be kept on a small partition on an entirely different disk if you so choose. At 128MB each, with minimal data being written, you could well plug-in a small USB Flash drive, and put the DRBD metadata for numerous partitions on it, without touching any of the hard drives.

Then, on the secondary DRBD system, you just "connect" the relevant partitions when a snapshot is desired, all changes get synced up to date, and then disconnect whenever you like. It's better than a snapshot, since it's on a physically separate system. If you had a large enough hard drive array (2TB SATA drives are cheap), you could even have a single server being the "DRBD secondary" server for dozens of other servers, receiving "snapshots" periodically, and perhaps, automatically...

Re:You're confused

[Anonymous Coward]

This has been fixed in recent userland lvm tools. From lvm.conf: "By default, if a PV is placed directly upon an md device, LVM2 will align its data blocks with the md device's stripe-width. 1 enables; 0 disables." The default setting is "data_alignment_detection = 1", it's on by default and works correctly.

Re:You're confused

[Anpheus]

And they only work fine if you have no qualm with seriously degrading performance over time. Windows VSS provided snapshots, called "shadow copies" or whatever, more closely resemble ZFS snapshots than say, dumb SAN or LVM snapshots where the snapshots reside in a dedicated "snapshot area". NTFS, ZFS, have filesystem level snapshots and so the FS is able to put old and stale data relatively close together, and defrags can intelligently move stale data out of the way, making a contiguous area of disk a contiguous area of current data.

If you don't believe me, run Windows with hundreds of shadow copies on a disk, say, running PostgreSQL. Write a quick script to take a snapshot every minute and keep running PostgreSQL. Defrag after one hour, and then re-run the benchmark. Defrag, run benchmark.

Then run your favorite Linux distro with 100+ configured LVM snapshots, again, once a minute during a PostgreSQL benchmark of your choice. Defrag or do whatever you think will aid the benchmark except destroying the snapshots, and re-run the benchmark.

You will then sit patiently and wait for BTRFS or something else to save the day.

Re:You're confused

[Jeffrey Baker]

And ... why would you do this? Take a snapshot, back it up somewhere, and then delete it.

Re:You're confused

[Anonymous Coward]

"Can you elaborate on how LVM gets volume offsets "wrong" and it's impact on RAID performance?"

I'll do on his regard.

Google: LVM, md misalignment, perfomance.

Done.

Re:You're confused

[sumdumass]

SVC or CVS?

Actually, he is asking for a CVS like feature. He's basically saying, I can do this with that, how do I do it here without rebuilding the server or taking it off line. In effect, he is asking for what he doesn't know about to make the thing work like he wants.

He's basically stating that some things could work but won't work because he either has too old of a system, or the wrong file system, or they aren't using the volume manager and changes in that regard are too much. So know he wants to know what's left that could achieve this goal.

Re:You're confused

[ckaminski]

You are seriously missing out on Solaris, then. ZFS is the bomb, zones are so much better than anything Linux has to day (including OpenVZ).

Plus it supports all the applications Linux does. Whoohoo!

Re:You're confused

[Mr. Slippery]

Check out rsnapshot, which uses rsync and hard links.

Re:You're confused

[drsmithy]

And ... why would you do this?

To speed up and simplify restores.

The vast majority of "need a backup" situations are end users accidentally deleting things. This means a simple self-service restore capability that keeps, say, 1-2 weeks worth of hourly snapshots, will cover 80%-90% of restorations in a very short period of time with nearly no administrative overhead.

You can't do this with LVM (well, you can try, but performance is going to be dismal and it's going to be a nightmare to configure and manage).

Re:You're confused

[drsmithy]

The choice isn't that simple. LVM comes with its own complications, including a tendency to get volume offsets "wrong" so the file system data doesn't align nicely to RAID stripes. This is not good for performance.

This is only going to happen if your partition is misaligned, in which case whether or not you're using LVM is irrelevant.

The solution is to either a) use whole-disk PVs (preferable), or b) create properly aligned partitions.

Also, LVM has only recently acquired barrier support, and the combination of no barriers + write cache can be quite dangerous if power is lost. Even battery backed cache won't save you if you use a journalling file system (and everybody does these days) because request ordering isn't guaranteed.

Again, nothing to do with LVM, such a failure will affect any system.

Re:You're confused

[TangoMargarine]

Not even a LMGTFY link? Your sarcasm is wholly inadequate :-)

Re:You're confused

[Cramer]

The primary reason for keeping a snapshot online is to avoid the delay in having to go back to tape for a single file some moron deleted.

Otherwise, yes, older snapshots should eventually be deleted, just like tape cycling.

Re:You're confused

[DavidRawling]

OK I know that Windows just does NTFS nowadays, unless you manually create a FAT volume - but going from FAT to NTFS is a simple conversion:

convert q: /fs:ntfs

That's all there is to it.

Re:You're confused

[Anonymous Coward]

This is nothing more then a half hearted attempt to Toll about how bad Linux is and how wonderful Windows is.
I am m surprised you all took the bait.. If the guys a Admin he can research like the rest of us IT guys do.

Re:You're confused

[Craig Ringer]

LVM snapshots also just aren't all that good.

They require you to pre-allocate space, so you have to guess how much copy-on-write difference will accumulate between the original and snapshot over the lifetime of the snapshot.

If the snapshot runs out of space, it *should* cleanly disable its self. Pity about the file system mounted from it that has no idea its backing block device just vanished. It gets messy, fast, when an LVM snapshot runs out of storage.

LVM snapshots are really inefficient, because they track all block changes, not just user-level file data and metadata. This massively bloats the snapshot, reducing how long it lives until it runs out of backing store and disables its self.

LVM snapshots don't share backing store. If you have three of them, snapshot t+3 has to store all the data snapshot t+2 and snapshot t+1 do, and so on. The differences between the real fs (t) and snapshot t+1 land up being stored three times in three separately allocated backing stores. You waste a HUGE amount of space this way, and it's hard to reliably predict how much you need so your snapshots often vanish out from under you are you're trying to use them.

LVM is useful, but for someone used to the Volume Snapshot Service (VSS) on Windows servers, to ZFS, or to any of the "enterprise-y" file systems often seen deployed with big SANs, it's just going to make them cry.

Re:You're confused

[Anonymous Coward]

Go lookup dynamic vs basic volumes and quit being a prick.

It's not that simple.

Re:You're confused

[BrokenHalo]

Given that the OP apparently has no interest in changing his filesystems, it seems to me that the obvious solution is a neat little rsync script run as a cron job. Why complicate life?

Re:You're confused

[amorsen]

Again, nothing to do with LVM, such a failure will affect any system.

Of course not. That's what barriers are all about -- making journalled file systems actually work. LVM silently discarding barriers is nasty. Fortunately it has been fixed; I haven't actually tried it out though.

Re:You're confused

[amorsen]

Plus it supports all the applications Linux does. Whoohoo!

Maybe so, and I do remember my days of trying to keep various Free Software running on a variety of Unix systems with a certain fondness. However, I have no desire to relive the experience.

Re:You're confused

[Der PC]

If your'e thinking about the time when compiling a (pretty much any at all) GNU app on HPUX or AIX too like three weeks of code modifications and Makefile manipulations, then long gone are those days :)

OpenSolaris is generally not any harder than your (randomly selected) Linux distribution.

And yes, ZFS and Zones are... THE thing.

Period.

Re:You're confused

[Fr33thot]

The OP sounds like a coded attempt to slander one OS because he's using the wrong tool on an install that will not support it while claiming it can be done on the other OS (assuming that you use the right tool on an install that will support it.)

Isn't that really the double standard you are referring to?

Re:You're confused

[The_Wilschon]

need updated

Are you from Central Ohio? My wife is a linguist at OSU, and the "needs <verb>ed" construction is a frequent subject of conversation (among linguists here), since it is really only native to a very small area.

Re:You're confused

[Anonymous Coward]

Dynamic volumes are just a shitty MS attempt at logical volumes, you bald cunt.

Re:You're confused

[Bungie]

Back in the day one of my employers ran a FileMaker server for all of their databases. The FileMaker server would crash fairly often and when it was reloaded it would perform a consistency check on each database. The consitency check would process every record entry in each database to see if it was closed properly. They had multiple databases with millions of records, so it could take days before the checks would complete and the databases are brought back online.

The solution was to have multiple snapshots. If the server crashed, we'd restore the databases from the most recent snapshot. We'd lose some data (whatever was added between the snapshot and the crash), but it was worth it to avoid the lengthy downtime from the consistency checks.

Re:You're confused

[allo]

its a nice fast backup. but deleting old "volumes" is a pain in the ass. it takes hours on an ext3 filesystem. deleting duplicity-volumes is much nicer.

Re:You're confused

[Anonymous Coward]

What he's describing is more like having to convert all of your drives to Dynamic Disks before you can use Volume Shadow Copy (which you don't really have to do VSS can work on GPT or dynamic disks with NTFS partitions). On Windows you can also non-destructively convert FAT32 to NTFS partitions and GTP to Dynamic disks. Even if he updates the kernel, he still can't use snapshots on his ext3 filesystem until he converts the disk to LVM. I don't know if LVM can do non-destructive conversion or if the tools are easy to use.

Re:You're confused

[nschubach]

I am. I grew up south of Sandusky, live in Columbus now.

To be honest, I don't see any problem with it, but I claim no expertise in English. I guess the "proper" method would be "need(s) to be"? We just shortened it. ;)

Re:You're confused

[Nutria]

The solution was to have multiple snapshots.

Argh!!!!!!

The solution is to use a s/w-h/w combo that doesn't crash on a regular basis.

Re:You're confused

[Crackez]

If everyone just googled everything, there would be no ask slashdot.

Re:You're confused

[TangoMargarine]

You might note that I commented on the format of his response, not on the content. He already referenced Google before I came along. Ergo, while I do not comment on the validity of your statement, it would be more on-topic if posted as a reply to the grandparent's post.

Re:You're confused

[The_Wilschon]

Oh there is certainly nothing wrong with it. It is a dialect feature that exists. It is interesting principally because it is so extremely localized (unlike, for instance, my own native "might should"), and more specifically localized to the area surrounding the university at which said interest occurs.

You should see my wife's students, though, when she writes "The car needs washed." on the board, and then asks those who feel that such a sentence is natural to raise their hands. Half the class does almost every quarter, and then each half looks at the other half like they've got their heads on backwards. People who don't have the construction think it sounds really really odd, and people who do have it think that everyone else is crazy for thinking it odd.

no offense, but what a windows mentality

[Anonymous Coward]

Linux servers don't get backed up, they only get migrated to new hardware every five years. (BSD is the same, except for the second part.)

Re:no offense, but what a windows mentality

[mikemsd]

Sweet, I'm going to install Linux on all my systems. I didn't know that Linux could prevent natural and man made disasters as well as being a stable operating system. We've been wasting all this money on backup for all these years.

Re:no offense, but what a windows mentality

[dajekels]

Linux servers don't get backed up, they only get migrated to new hardware every five years. (BSD is the same, except for the second part.)

this guy probibly runs all his windows servers via xwindows manager. nice >.

Re:no offense, but what a windows mentality

[Anonymous Coward]

the linux server is probably sitting next to all the contracts your clients signed, which are actually more important for you

Re:no offense, but what a windows mentality

[Unoti]

Sweet, I'm going to install Linux on all my systems. I didn't know that Linux could prevent natural and man made disasters as well as being a stable operating system. We've been wasting all this money on backup for all these years.

There's a mix of humor and catty vitriol here all around, but here is something that addresses the serious point made in Grandparent's statement about it being a "Windows" way of thinking.

Take a look at Infrastructures.org which describes a whole way of thinking about server reliability and configuration. Where I work we essentially use this approach. The fundamental concepts around this approach concentrate more on system configuration, ability to pick a random server and drop it out the window and have another one just like it online in moments. It's less about backups, and far more about a more comprehensive disaster recovery/prevention type of thing. The types of approaches described there are probably more easily implemented using Unix/Linux, but is probably also possible with Windows boxes.

Re:no offense, but what a windows mentality

[alexborges]

"We've been wasting all this money on backup for all these years."

Yes, by not using AMANDA or another FOSS backup solution, youve thrown money down the drain.

Re:no offense, but what a windows mentality

[Anomalyst]

Yes, by not using AMANDA

If only one could install it on Ubuntu 10.4 LTS, oh well maybe it will be fixed next quarter.

Re:no offense, but what a windows mentality

[stefanlasiewski]

infrastructures.org looks interesting, but then I see they mention things like 'NetSaint' which was renamed to be Nagios about 7 years ago, and references to "LISA '98".

Some of this information looks old. Am I right? These days, shouldn't we be thinking more about virtualization and cloud infrastructure?

That said, they do touch upon many good ideas. It seems that many mid-sized shops do follow some similar ideas.

Re:no offense, but what a windows mentality

[Unoti]

Yes, it's very old. They also talk about using cvs for version control, and mention that that world has moved on to svn, and the world has moved on a couple of times since then even. We also use Nagios rather than more ancient monitoring software. But still the central ideas are sound, even with many details changed. And practical, too.

These ideas actually apply very much to cloud infrastructure. It's really all about the cloud-- considering a machine not as just "a machine", but instead thinking of it as having a base image with certain functionality bolted on top of it. Thinking of a machine not just as a machine, but as a replaceable/exchangeable component in a larger collective system. That essentially is cloud computing. The thing a lot of people don't consider is that even a smaller cluster of machines should/could be configuration managed, maintained, and viewed this way.

Re:no offense, but what a windows mentality

[vegiVamp]

Shadow copies are not about server reliability, they're about stupid users inadvertently overwriting or deleting wrong files, and allowing them to fix their mistakes themselves, without needing to access the backup system or bothering the system administrators.

Also, versioning filesystems make a copy-on-write, so there's a backup of *every* version of a file, and not only the versions that are there when the backup (or shadowcopy) runs. It's been only this week that we've been looking fruitlessly through the backups for some vanished files. We can only assume that the files were erroneously deleted on the same day they were created, before the backups had a chance of picking them up.

Re:no offense, but what a windows mentality

[drinkypoo]

These ideas actually apply very much to cloud infrastructure. It's really all about the cloud-- considering a machine not as just "a machine", but instead thinking of it as having a base image with certain functionality bolted on top of it.

Cloud computing is server-agnostic. Whether you use a virtual machine running on each node to provide it, or have a cluster of machines with the same architecture, or just the same software installed on all nodes, the basic principle of cloud computing is that ANY node can do a job, so work is simply allocated to ANY node. This notion of using a cluster and ONLY a cluster to do work is what is new. It is, in fact, the logical extension of what you are talking about. Data is stored on a redundant SAN (potentially simply a clustering filesystem) and thus is accessible to ALL nodes.

Or, IOW, cloud computing is not applicable to small environments where tasks are dedicated to machines. And a derivative of the approach you mention is an inherent benefit to cloud computing. It's not something you have to go out and do, because by the time your hardware looks like a cloud and your software is designed to run on the cloud, you've already achieved it.

Suck it up

[mewsenews]

You will have to migrate your servers with plain ext3 to LVM-based ext3. Short term pain for long term gain.

Re:Suck it up

[Anonymous Coward]

Agreed.

Fix it.

Re:Suck it up

[Omnifarious]

That would be my advice too. Migrate.

Eventually you're going to want to migrate those filesystems to btrfs as well, and that has really nice built-in snapshotting capability. But until then (a few years from now) move to LVM. It will save you so much hassle and be much more tested and stable than some hack like ext3cow

Re:Suck it up

[Anonymous Coward]

And this is why I like Linux so much. The oh-so friendly community and the nice robust all-in one technologies. :P

Re:Suck it up

[alexborges]

You cant do this in windows either, not with partitions. Thats why their solution is called shadow VOLUME. Cause it need VOLUMES to work.

OUr solution is called LVM Snapshots cause it needs LVM VOLUMES to work.

Now is that so hard to understand?

Re:Suck it up

[Anonymous Coward]

Ditto. LVM provides exactly what is needed. It isn't that hard to move the native partitions in to logical volumes. Where it is (because you have a truly ancient release of some distro or other) just backup with tar or something.

And stop building machines without LVM. At most only /boot should be native. Everything else in LVM. Always.

Re:Suck it up

[dhanson865]

You cant do this in windows either, not with partitions. Thats why their solution is called shadow VOLUME. Cause it need VOLUMES to work.

Our solution is called LVM Snapshots cause it needs LVM VOLUMES to work.

Now is that so hard to understand?

Well it's obvious you have never used Volume Shadow Copy because in the windows world there is no practical difference between a partition and a volume. No I'm not joking, no I'm not being a Troll.

Find a Windows server with a single drive (basic disk) and a raid Array (Dynamic Disk)
Right Click on My Computer and choose Manage
Click on disk management
right click on a unallocated portion of a "basic disk" to "create a new partition"
right click on a unallocated portion of a "dynamic disk" to "create a new volume"
Give them both drive letters
Go to the properties for each volume/partition and marvel at how you can turn Volume Shadow Copies on for a Volume and a Partition.

OMG, Microsoft doesn't care if it's a volume or a partition. Oh well, at least it works like the end user would want it to in this case.

Yes Windows uses the terminology differently than hard core linux users do but unless you understand how Windows labels things saying something like "You cant do this in windows either, not with partitions." makes you sound like you don't know what you are talking about to anyone that has seen the process it actually takes to manage disks on a windows server.

Re:Suck it up

[alexborges]

The thing is that, yes, I dont know how does windows shows it working, i do know it does it with the concept of volumes (a layer atop the partition).

Those may be in the filesystem or under it, but they are certaintly not partitions in any reasonable sense of the word.

Discs are discs, partitions are partitions (an entry in the partition table, which is not a figment of anyone's imagination), volumes are volumes and filesystems are filesystems regardless of you being in a Unix-like OS or a windows thingie.

And yes there is a practical difference in windows between partitions and volumes. Go to your windows server, open a console and fdisk your disc or san or whatever. You will see (a) partition(s) which are NOT volumes.

Re:Suck it up

[Anpheus]

Windows does it in precisely the way uncoordinated open source projects will never be able to do it. They told the NTFS team, a new team in charge of the volume snapshot service, and the team in charge of the logical disk management to work together, create and perform regression tests against each others code on every check-in and patch, and likely set up team liaisons whose sole purpose was to ensure interoperability.

If you told me a third party open source organization, without having full control of the developers and direction of both the ext filesystems and the LVM system, was going to write a service that performed the same function as the volume snapshot service, I would laugh at you. I would laugh and laugh. Open source, because of its nature, tends to attract developers who want to do something, and they want it to be the best at that something. At the same time, they don't want to tie themselves down to stable APIs because, well, it can be limiting and slow development. I totally understand why. So telling me that some third party is going to extend LVM with one API, EXT with another API, and then write a service to coordinate the two is mind-boggling. Those people would have to constantly commit code to match changes in either of the two rapidly changing projects, they'd have to fully understand the inner workings of both ext and LVM, and then they'd have to make it all work without corrupting anyone's data and ruining their reputation beyond repair.

On the other hand, you have projects like ZFS or BTRFS, which are just as monolithic, but more ambitious, and powered by the same developers I mentioned above. They want their solution to be the best. It takes a long time though because they essentially have to start from scratch and incorporate all the things that appear to be within arm's reach. But the people who start projects like BTRFS realize that it's a fool's errand to try and create interoperability between massive, disparately managed open source projects. GNOME and KDE only survive because they threw everything else out and decided to simply come with their own full suite of stuff. X is its own long story.

I don't want to diss open source, like I said, it creates magnificent pieces of software, and the developers really, truly tend to care about their projects. (Even if they can be a little defensive, sometimes.) But without a dictator forcing cooperation between different teams, you often see open source reinventing the wheel. Sure, LVM and EXT3 could theoretically work together to provide sane, fast, performant snapshots. But I'd like to meet the person who thinks they can pull that project off.

Re:Suck it up

[evilviper]

Short term pain for long term gain.

Actually, I'd call LVM the long-term pain. For all it's benefits, I don't want my servers to become unresponsive when large files are being written. Doesn't happen on regular partitions, only with LVM.

Re:Suck it up

[udippel]

Spot on, absolutely. If I had, I gave you 6 out of 5 mod points.
People might be offended, but this is the precise description of what is going on; and where the trouble (or chances?) brew.

Oh, by the way, I'd reduce your 6 mod points to 4; for the 'dictator' as a need. We have one, and as far as I can make out,there is no such thing on OpenBSD as what OP asks for.

Re:Suck it up

[ckaminski]

For what it's worth, as a guy who spends more time creating TBs of content (photos) than managing computers, I'm not moving to btrfs until RedHat is shipping it as the default filesystem for RHEL for at least a year.

I build one NAS every 4 or 5 years, and I just built an OpenSolaris EON-based ZFS NAS (nfs, smb, iscsi).

Re:Suck it up

[Anonymous Coward]

In the open source world the third team going for the best solution doesn't have to start from scratch. They have that option if that is the best technical solution but they are also free to take advantage of all the existing open source software that contains parts of the functionality they need. By contrast, if they were working in the closed source world they would have to negotiate a license (an expensive process in itself) or in the worst case either buy the company making the other solution or rewrite it from scratch.

Re:Suck it up

[Omnifarious]

I've been testing it on light duty, non-critical filesystems for a little while now. I intend to be an early adopter, but a very cautious one. It's worked fairly well so far.

I use reiserfs for most of my filesystems currently. I've only ever had one problem with it. I had lots of null characters appended to files that were actively being appended to at the time a computer crashed. The stupid jokes and badmouthing whenever it's mentioned really irk me because they don't match my experience and because I think that kind of attitude is part of why it's so poorly supported in RH.

I consider ext[234] to be horribly broken for any number of reasons. Reirserfs is not broken in any of those ways, so I've been looking for the technological successor to resierfs because it's no longer being maintained in any significant way.

As you can see, I have a lot of reasons to go for a risky fs like btrfs. And with its b-tree based structure it has a lot of the features that I think made reiserfs so much better than the ext series and is a more than worthy successor.

I really hate it when people tell me that if I want real storage I should stuff my data in a database. Filesystems exist to organize your data. If they're so bad at it that you need to use a database instead the filesystem is broken and should be fixed.

Re:Suck it up

[Cramer]

This is just a matter of "words". In windows a "volume" is any mounted filesystem. The underlying storage system is pretty much irrelevant.

Re:Suck it up

[Anonymous Coward]

Oh, you poor deluded thing.

Windows does it in precisely the way uncoordinated open source projects will never be able to do it. They told the NTFS team, a new team in charge of the volume snapshot service, and the team in charge of the logical disk management to work together, create and perform regression tests against each others code on every check-in and patch, and likely set up team liaisons whose sole purpose was to ensure interoperability.

I'm sure they did have interfaces and test suites and stuff set up, but I'm wary about any Microsoft claims of functionality. Usually, Microsoft stuff works for a few fuzzily-bounded cases, but do something interesting and it fails. And if it fails somewhere, then good luck finding someone who will own up to it and take responsibility to make sure it works.

What I love about open source is that its mechanics are completely open and accessible.

Valid point about the dictator thing, though. That's why only a few projects do exceptionally good work, such as the OpenBSD/OpenSSH project.

Re:Suck it up

[naturaverl]

The parent is joking, right? Why the hell was it marked +5 insightful?

Sure, LVM and EXT3 could theoretically work together to provide sane, fast, performant snapshots. But I'd like to meet the person who thinks they can pull that project off.

Well I hate to be the first to break this news to you, but... There never was a need for the LVM team to coordinate with the ext3 team, because the filesystem is abstracted away and irrelevant to LVM. As far as LVM is concerned, it's just a logical mapping of blocks of data to a physical volume. The data could be a real filesystem, or not... It could be raw data written to the volume without a filesystem. LVM wouldn't care. And you could still use LVM to make a sane, fast, performant snapshot of it either way.

Re:Suck it up

[allo]

ReiserFS is good in scrambling your data if the pc crashs. Had an ReiserFS failure in an overheated notebook. all ext3 partitions without error, the reiserfs completly fucked up.

Re:Suck it up

[Omnifarious]

Well, I had a PC that repeatedly crashed from overheating in the late 90s, and I only had the problem I just mentioned. No other data was scrambled.

And in recent years I've never had an issue like that with reiserfs.

Anyway, it's water under the bridge mostly. I'm just clinging to it for another year or two before I think btrfs is ready to be used seriously.

Re:Suck it up

[Anpheus]

Like I said, run a production system with more than a few dozen LVM snapshots and tell me you're getting good performance.

The abstraction away in one hand is a plus, it makes many things simpler, the problem though is that your disks look like this:

| ext3 volume | ... | ext3 copy on write snapshot volume 1 | ... | ext3 copy on write snapshot volume 2 | ...

With ZFS, your old and stale data is mixed in one volume, and a defragmentation operation would have the opportunity to move the stale data out and move the current data in to make a directory contiguous again.

The problem is, LVM's complete abstraction of the block devices with no means for inter-allocating volumes or making requests to the filesystem to un-allocate small blocks within a volume to assign to snapshots means that performant snapshots on Linux will have to wait until ZFS, BTRFS, NILFS, whatever.

Re:Suck it up

[drinkypoo]

If you told me a third party open source organization, without having full control of the developers and direction of both the ext filesystems and the LVM system, was going to write a service that performed the same function as the volume snapshot service, I would laugh at you. I would laugh and laugh.

It doesn't have to be that way. All that has to happen is the ext (and other filesystem) developers and the LVM developers have to get together and decide what interfaces need to be developed on each side to permit the systems to work together. Having them closely coupled by a single integrator causes problems later, down the road, when the system is to be expanded to permit some new behavior.

Sure, LVM and EXT3 could theoretically work together to provide sane, fast, performant snapshots. But I'd like to meet the person who thinks they can pull that project off.

It's not a person, it's a consensus that must be developed. And so far there is no consensus that a change is needed. If you'd like to see one, I suggest you start stirring up developers.

Re:Suck it up

[Thundersnatch]

And you could still use LVM to make a sane, fast, performant snapshot of it either way.

No, you can't.. Do make sane snapshots, you need application-layer support, especially for databases. Otherwise your snapshots are not clean, they are in a crash-recovery state as far as the application is concerned. This is particularly important for databases, distributed file-systems, and queue-based software (including mail stores).

The beauty of the Microsoft Volume ShadowCopy Service is that it is a stable API through wich applications, backup software, and storage systems can all communicate. Backup software says "I want a snapshot". All the applications who have registrerd themselves with VSS get their writes flushed into a consistent state, and say "I am ready for snapshot". Then the storage device (either a software filesystem like NTFS or a SAN/NAS) says "I am doing the snapshot". This level of coordination is simply not possible on Linux.

Re:Suck it up

[alexborges]

Thats just idiotic. LVM and ext3 perform sane, fast and performant snapshots right NOW and it has been that way for the past few years.

Re:Suck it up

[alexborges]

Not at all. Its a lie. ext3 and lvm snapshots perfectly fine right now.

This guy is an astroturfer.

Re:Suck it up

[alexborges]

Liblvm is right there, clvm works fine and gfs/lvm snapshots perfectly fine.

I think the GP is a MS astroturfer talking out of his ass.

Re:Suck it up

[alexborges]

This is really besides the point. Databases can be told to shut up and flush before snapshoting with a script.

Ah, you perhaps think that the "elegance" of having an OS service tell clients of a volume that a snapshot is or will be done somehow pays for both the money and the downtime of using a filesystem as slow and farked as ntfs?

Do we not have ocfs2, gfs and gfs2 in Linux, XFS and JFS as well, and all of those do very fast, sane snapshoting over LVM2?

Yes, we do. You work for propsoftcorp PR, I get it. Enjoy your monday.

Re:Suck it up

[rawler]

Btrfs is largely working with other kernel Storage subprojects such as the common MD-infrastructure, and block layers to improve atomic consistency and performance in underlying components. (Such as proper barrier-support).

Lately, Ceph came in on top, adding a layer of network-distribution for your storage needs, and they're too working with Btrfs, deprecating their own filesystem for shared benefit.

rsnapshot

[Anonymous Coward]

maybe rsnapshot (http://www.rsnapshot.org) will fit your bill...

Re:rsnapshot

[fearlezz]

rsnapshot has no actual snapshotting functionality. It's just a (very useful) wrapper around rsync that takes care of de-duplication. While making a rsync/rsnapshot backup, files on the system can still be changed.
For example: A LVM snapshot would give you a consistent MySQL dump if you're using innodb. Rsync/rsnapshot does not.

Re:rsnapshot

[TheSunborn]

I newer understod why InnoDB should allow filesystem snapshots. Is this documented anywore or is it just one of "It ought to work"?.

I mean InnoDB does NOT support a normal filesystem copy of the 3 database files as a way to make a backup.

Re:hey retard:

[suso]

Hey, IRC called, they want their asshole back.

"does not yet support my older 2.4 Linux servers"

[tlambert]

"does not yet support my older 2.4 Linux servers"

So upgrade your servers to a supported release instead?

-- Terry

ummmm

[ProfessorKaos64]

dd command? Works well for image snapshots for me, copys raw bit by bit

Re:ummmm

[pavon]

Only works if the partitions don't change while you are copying them. The big advantage of using LVM for this is that you can create snapshots on a live system, without resorting to remounting the partition read-only (and all the problems that will cause).

But really, those are his only options. If you insist on using plain ext3 and won't add a layer of between the FS to allow for this, then you have no choice but to freeze the partition while doing a volume-level back up.

Re:ummmm

[Anonymous Coward]

Put in a couple of new drives and run something like:
dd if=/dev/hdx of=/dev/hdy
overnight, once that's done, repartition the existing drives for LVM and start restoring/rebuilding...

Re:ummmm

[fnj]

You want to take the server out of service overnight? Because if you try that trick on a live running filesystem, you are asking for corruption on the output side. And no, I don't know an alternative in this case.

LVM Snapshots

[theunixman]

I use them on ext3 with no problems. It's true that very early on there was a problem with them and journaled filesystems, but that has long since been solved.

Re:LVM Snapshots

[Anonymous Coward]

WHY o WHY are there always so many people who don't comprehend what they read?

There's some sort of confusion here

[vadim_t]

LVM snapshots work on a block level and don't care about the filesystem. A snapshot of any data in a logical volume should work fine, even if it's not a recognized filesystem.

A nice use for this is using a read/write snapshot to try different strategies for recovering a broken filesystem.

Re:There's some sort of confusion here

[Trepidity]

I think he's saying that he can't use LVM snapshots, because some of his servers have ext3 directly on a plain partition, not on top of LVM (used to be a common setup).

Re:There's some sort of confusion here

[Unoti]

Yes, and that's what he should change. Don't you think?

Not Enough Info

[crow_t_robot]

I'm going to answer your question with a question:

How many servers and what is the approximate amount of data that you need to backup?

Re:hey retard:

[Anonymous Coward]

Your asshole called, it wants the 1990's back.

ZFS?

[corran__horn]

I will admit that I have not tried it on Linux, but zfs is the best of the next gen filesystems. It does cryptographically assured reads and writes (remember that transitory undetected disk malfunctions occur at a rate of ~1/TB of data), it can snapshot changes, it fricken slices bread. If it had a gender, I would probably marry it (well, I guess I can date it for a while and see how things work out). http://zfs-fuse.net/

Re:ZFS?

[Anonymous Coward]

zfs is great under OpenSolaris; if you never tried doing things with fuse under Linux, then suggesting it as a solution is not well informed. pretty much all fuse solutions are lousy. zfs-fuse only serves to put a lousy wrapper around a very strong fs.

Re:ZFS?

[fhuglegads]

I admit I haven't tried this out but there is a native port of zfs to linux:

http://wiki.github.com/behlendorf/zfs/

There is an "issues" tab with 20 open issues but they aren't loading for me.

Re:ZFS?

[icebraining]

Unfortunately it won't ever be included in the Linux kernel thanks to the CDDL.

Re:ZFS?

[carton]

yeah ZFS-FUSE is an old version of the Sun ZFS code, and only the latest versions are really safe & feature complete (ex. zpool import -F). even if it weren't for the performance worry, do not fuck around with ZFS-FUSE. it is only for fun. Nexenta and the OpenSolaris CD's at genunix.org have almost the most recent ZFS. FreeBSD has, meh, newer than ZFS-FUSE version but not really new enough at last glance.

Re:ZFS?

[Anonymous Coward]

Absolute nonsense about fuse. Fuse works dandy. Sshfs-fuse for example works very well. The only problem with zfs on fuse is due to the particular project not having nearly enough resources. And I will readily admit that zfs is not a good candidate for using fuse seriously on a server.

Re:ZFS?

[ArsonSmith]

Might be ok if it weren't so painfully slow. By the time I turn of the hogging features I might as well use a better supported file system.

Re:ZFS?

[swilver]

> (remember that transitory undetected disk malfunctions occur at a rate of ~1/TB of data

Really? I doubt that since every bus and the harddisk itself use extensive ECC to prevent exactly that.

In my experience, the main point of corruption of data is from the use of non-ECC memory. Corruption rates are around 1 bit / 200 GB of data copied. Now, how would ZFS detect that data was corrupted when it was instructed to write a piece of corrupted memory to disk? It won't. It will happily write that crap and provide it with a nice (wrong) checksum. It will checksum its buffers you say? Is that before or after the memory corruption occured? Checked before and after a succesful write?

ZFS checksumming is pretty pointless in my experience as memory corruption can occur at any time, before after and during checksumming/reading/writing of data -- get ECC memory instead and the checksumming will become even more pointless.

rsnapshot

[Anonymous Coward]

It sounds like nilfs is more along the lines of what you're looking for, but something like rsnapshot might be useful to you.

http://www.cyberciti.biz/faq/linux-rsnapshot-backup-howto/ HowTo on setting it up on Debian/Ubuntu

It takes timed snapshots of the filesystem, so you'll only be able to roll back to the last snapshot.

rsnapshot is what you're looking for

[xee]

RSnapshot uses a clever blend of rsync + hard links to do what you want... you can store many incremental backups in just a little more space than a full backup. you can run rsnapshot on a backup server with lots of disk space, and all you need to expose on your target machines is SSH.

you'd create "backup" users on all the target hosts, generate a PKI key pair, and put the private key on your backup server. put the public key in the "backup" account on each target machine so the backup server can securely login without a password. then you just set up rsnapshot to log into your targets and it will use rsync-over-ssh to pull the data.

http://rsnapshot.org/

Re:rsnapshot is what you're looking for

[Anonymous Coward]

Rsnapshot really does work perfectly for this situation.

We use a combination of samba + rsnapshot to make the snapshots available through the shadowcopy interface, if you're interested in doing a similar thing you can find the scripts that you need to do so here: http://sites.google.com/site/evanquirk/Home/shadowcopy-through-samba-with-rsnapshot-backups

Re:rsnapshot is what you're looking for

[dfn_deux]

This is no good for a true snapshot since the rsync operation is non-atomic on a live filesystem.

Re:rsnapshot is what you're looking for

[jgrahn]

[RSnapshot] is no good for a true snapshot since the rsync operation is non-atomic on a live filesystem.

I cannot help wondering when I read stuff like that who *really need* atomic, and who just like it because it sounds cool ... If that 2.4 guy doesn't really *need* theoretical atomicity, and he can do his work with something much more simple, he should.

Re:rsnapshot is what you're looking for

[dfn_deux]

i wasn't trying to guess at what he needed, but his question was about snap shotting. One of (if not THE) key feature of a snapshot is that it is atomic. Anything that rolls through a changing filesystem one file at a time is not going to fit that bill. Also you run the risk of making "backups" that could break things that presume state consistency. If you capture the log of a daemon before the product output then your backup could have no record of the event which created the output for example.

These types of concerns are of increasing importance to professional system administrators in a time where there (to me at least) seems to be an increasing focus on meeting legally mandated audit and retention requirements.

Re:rsnapshot is what you're looking for

[paziek]

Wasn't rsync atomic operation, while - for example - cp wasn't? I can't find source to confirm this and all I have is this http://pl.php.net/manual/en/apc.configuration.php#ini.apc.file-update-protection entry in PHP manual, so...

Re:rsnapshot is what you're looking for

[Tolaris]

Take rsnapshot one step further, and you have BackupPC. Which is what my company uses (mid-level enterprise), and it's awesome.

http://backuppc.sourceforge.net/

Re:rsnapshot is what you're looking for

[X0563511]

The "safest" way to run any kind of backup is to do it with the server offline. In such cases there is -zero- risk of state inconsistencies.

Suck it up. If it's that critical that you can't take it down, well you should have implemented failover redundancy already.

Re:rsnapshot is what you're looking for

[X0563511]

Rsync may tally up a list of files before it operates, but said files are not "locked" at all. Files at the start of the operation can be completely 'outdated' by the time it nears the end.

Re:rsnapshot is what you're looking for

[icebraining]

One file is atomically updated by rsync, but not the entire filesystem. So you can have a DB that uses 2 files, and when it finishes copying the first, the status of the second file has already changed and therefore the two aren't compatible.

LLVM snapshots do it atomically for the whole FS.

Re:rsnapshot is what you're looking for

[hasdikarlsam]

Unison does a better job, in that it checks for changes while it's running and can be configured to retry until there are none. However, it can still be tripped up by changes that touch multiple files; it doesn't give an atomic snapshot either.

In practice, though, it's never failed in giving me working backups.

Re:rsnapshot is what you're looking for

[hey!]

I cannot help wondering when I read stuff like that who *really need* atomic,
and who just like it because it sounds cool ...

Oy. I agree very few personal computers need this, but when you are talking about servers it's quite common. Really any time restoring the version of file A that goes with a different version of file B would be a bad thing. Datasets where debits and credits wouldn't, indexes that might end up pointing to the wrong records, virtual machine disks and other huge data objects that live in multiple files files ... it isn't hard to think of scenarios where a snapshot is really important.

Re:rsnapshot is what you're looking for

[shutdown -p now]

Non-atomic on file (or block) level is still not good enough, actually. You may still have applications that do their own data management, and for which a single file (or block) copied atomically may have inconsistent information within it. It isn't necessarily fatal, but it may mean longer recovery time later. A good example of that are database servers and their transaction journals - you want one to be flushed as much as possible before making a snapshot...

The nice thing about VSC in Windows is that it also offers a userspace API for applications that are affected by this kind of thing, so that they may register for notifications, and ensure that their data is internally consistent before it gets copied. I think that part is actually separate, in fact - e.g. Acronis backup tools don't use VSC, but rather their own custom kernel-space snapshot implementation, but they do use the same notification system to ensure data consistency.

Re:rsnapshot is what you're looking for

[sjames]

It might be better to say that there are a few caveats. For many purposes, this might be perfectly adequate. Often enough, snapshots are used because they are what's there, not because they are actually necessary.

Hobbling Along

[kwalker]

Since the situation is so hobbled (Old Linux kernel, no LVM) about the only thing you will be able to do is learn to use hardlinks. The ext* filesystems support them but you will have to manage them yourself (cp -varl /source/* /destination/version). Yeah it's a huge hack, but unless you can actually fix the problem, it's about your only hope.

Re:Hobbling Along

[ckthorp]

Or use rsnapshot and be lazy...

Re:Hobbling Along

[Nimey]

No need for -r when you're using -a, at least on GNU cp.

Re:Hobbling Along

[TheOrquithVagrant]

Hardlinks won't really work as a snapshot substitute; any files that are modified (rather than deleted/rewritten) will be modified in the "snapshot" as well. A hardlink "snapshot" like that would protect against accidental deletion, but that's about it.

Eh?

[ledow]

If you have backups, then moving to LVM is obviously the way to go if you desire snapshots. The others options are short-term hackery, LVM was designed from the ground up to do such things. And Ext3 has nothing to do with the price of butter.

To clarify, let me rephrase your question for the other way around

"I was asked to manage a number of *Windows* servers at work. I would like to use volume snapshots to improve my backup scripts and keep recent copies of data around for quick restore. I tried Windows Shadow Copy, but most of the servers I manage run MBR partitions with FAT file systems, so Shadow Copy will not work. I found some versioning file systems out there... Those look interesting, but I need something I can use on my existing FAT file systems. I also found --random freeware--, but it does not yet support my older Windows NT 3.5 servers. What are you using to make snapshots on Windows?"

Except, in that case, it makes more sense because the filesystem is the determining factor, not the volume management. If you have LVM, it doesn't matter what the underlying filesystem is, really. Stop faffing about - if you have a server, with backups, that you need snapshots on, take the hit and wipe the drives to a config that supports that... while you're there upgrade that damn kernel already. If nothing else, it will test that the backups you're making are actually worth the effort. It's like complaining that 95 on FAT16 doesn't support Shadow Copy. If you absolutely *can't* take those servers down, or am unable to restore your backups to another machine for testing such changes (whether because of compatibility, software licensing and/or bad backups), you have bigger problems than some random desire for a feature you don't actually *need* at the moment.

Re:Eh?

[gbjbaanb]

If you absolutely *can't* take those servers down

If you can't take those servers down, nature will be getting ready to do it for you. At a time when you least want it's "assistance".

Re:Eh?

[sjames]

NO!

You do NOT wipe your server THEN restore data, you build a new server, load the backups, make sure it works, then use it to replace the old server. The old server can then be wiped and used as a replacement for the next server in line.

If that is simply not practical, at LEAST take the old drives out and set them aside and restore onto new drives. More downtime, but you still maintain a decent way to back out if something goes wrong. Plus you reduce the odds of unplanned downtime a bit by getting the older drives out of production.

rsnapshot

[Nightshade]

Have you looked at rsnapshot?

It's based on this article:
http://www.mikerubel.org/computers/rsync_snapshots/

Re:"does not yet support my older 2.4 Linux server

[0racle]

There are Enterprise Linux distributions that are both supported and still run 2.4, though not for much longer. Not everyone runs Ubuntu.

Rsync, dd

[Anonymous Coward]

Most people just use rsync for backups. LVM is a lower level then the file system... you create a storage pool and then create filesystems on top of that. If you really want to use LVM snapshots you would have to recreate your filesystems, which I gather you would like to avoid. There are lots of options for filesystems that include versioning in some form or another. If you do want to choose one of these filesystems, find the Wikipedia article for any filesystem and down at the bottom will be a list of filesystems you can explore. If you just want to snapshot the entire partition, you can remount it readonly and use dd to copy the block device to a file (this will take up space for every byte, including unused space unless you compress it). I congratulate you on the opportunity to work in a unix environment. Please don't get discouraged because it is different, it really isn't that hard to learn, and many people would say it's easer. Google and Wikipedia will be indispensable.

DRBD much?

[Anonymous Coward]

Weee!

You have 2 good options for snapshot with linux:

[Anonymous Coward]

The more costly, but by far best, is to invest in a small netapp and point the data to it. Netapp has the best snapshot technology of any other out there. If you want whole OS, you can provision some iSCSI on the netapp box and iSCSI boot, and take advantage of snapshots on your OS.

Secondly, build up a solaris box with a large ZFS filesystem hanging off of it (which has snapshots), setup some NFS exports and do the same thing but using ZFS snapshot technology on the backend. Short of those two options, you are going to have to wait for the next-generation linux supportable local filesystems to mature a bit further. There are other options out there, but from my experience, none that don't wont suffer a significant performance hit with the technology. Netapp's filesystem doesn't have that penalty, and zfs doesn't have as bad of a penalty as others.

Wow, where to start...

[CAIMLAS]

Knowing where to start on this is a bit of a miffing point.

First: upgrade your shit. 2.4 kernel systems? Are you running Redhat 6? You know, from the turn of the millennia.

Second: upgrade your shit. Really,

Third: if your kernels are that old and you're using these machines for file storage/backup, chances are the hardware needs to be replaced before you even consider considering messing with them. Seriously: this stuff is ancient. Even Debian hasn't had a 2.4 kernel in 5+ years, I think.

Third: you can do what you're trying to do with rsync 'snapshots'. It works very well, failing filesystem level support. If you're sharing data over samba, this makes it easy: just put a '.snapshot' dir for these 'temporary' backups in their $HOME and hide dotfiles. Then make sure rsync ignores .snapshot. (Of course, there are other ways to do this.)

rsync snapshots (and here).

There are other sources of information out ther on rsync snapshots. There's also rsnapshot.

Chances are you'll have to upgrade before this stuff even works for you, though.

Re:Wow, where to start...

[evilviper]

2.4 kernel systems? Are you running Redhat 6? You know, from the turn of the millennia.

Redhat 6 was a 2.2.x kernel. So was Redhat 7. While the VER FIRST release of kernel 2.4.x was indeed right around 2001, the most recent revision of 2.4.x was just 4 months ago.

you can do what you're trying to do with rsync 'snapshots'.

Not really. There's no atomic rsync snapshots, so while one file in the "snapshot" will be from 12:01, another file might be from 12:15, and depending on the application, the two files might be hopelessly irreconcilable. In fact there are programs like "VSS Backup Helper" to allow your rsync snapshots (on Windows XP or later) to actually get an atomic snapshot, by using said VSS features in Windows, which is being asked about. What's more, rsync operates on the file level, which is hopelessly slow... Run rsync on a mess of small files with hundreds of thousands of folders on a 2TB+ partition, and watch it take (literally) days to complete.

Chances are you'll have to upgrade before this stuff even works for you, though.

Rsync runs perfectly fine on a 2.4.x kernel. It'll run on a 2.2.x kernel, too. Hell, I've ported rsync to ancient proprietary Unix systems (using GCC2.9x) with minimal trouble.

What's with this ageism? You think old software bit-rots if left on it's own? This isn't Windows ME we're talking about here.

Re:Wow, where to start...

[Anonymous Coward]

Lol - got fileservers still running with kernel 1.2.3. Works like a charm ;)

Backup != snapshot != package management

[Alex Belits]

People expect a snapshot to be immediately usable and reliable, however in practice a state of device, even if synchronized with filesystem through its transaction is not a state of all data -- some data may be in buffers, prepared to be written, and rebooting into a restored filesystem may require some cleanup of such state. In particular, SQL databases are completely unsuitable for this kind of backup (this is why they have their own backup and transaction log handling procedures), and database-like applications such as mail servers, may require reindexing.

However for purposes other than those applications, file-level backup is entirely adequate, so utilities like rdiff-backup end up providing more functionality than complicated snapshot-handling procedures -- incremental backups for subtrees, readable trees in backup media, etc.

It also should be noted that backups should not be used as a replacement of package management -- on Linux anything installed through a package manager can be uninstalled through it.

Re:Backup != snapshot != package management

[bertok]

Absolutely not! Snapshots are perfectly safe for capturing your database data.

On real server operating systems snapshot support is integrated into applications, which receive a "snapshot about to occur event" so they can quiesce their writes for a short period to make the snapshot clean.

For example, on a Windows server, a VSS snapshot is a complete restorable backup of everything, including your databases, event logs, the registry, etc... It's the standard mechanism that practically all Windows backup tools use. They take a snapshot, back it up, and then release it. The point in time that the snapshot was taken turns up in the "last backed up on" date field in SQL Server!

Even third party snapshot mechanisms integrate using plugins. If you take a snapshot with, say, VMware or your SAN, then the same quiescing mechanism is triggered.

Some real server operating systems like HP-UX appear to have LVM extensions that are similar to what VSS can do, but I can't find the equivalent in Linux. From what I can see, the closest you can get is to temporarily halt writes from the ext3 filesystem, but that's not the same thing as proper application quiescing.

Re:Backup != snapshot != package management

[swb]

Thanks, you saved me from writing the same thing.

Re:Backup != snapshot != package management

[kriston]

My Windows Home Server disagrees with you completely.
I only wish something similar to WHS with Volume Shadow Copy Provider could possibly exist for Linux. I'm tired of waiting.
You really should open your eyes and see that what you're describing is not only outdated but dangerously untrue.

Re:Backup != snapshot != package management

[Anonymous Coward]

If a database can't survive having the file system snapshotted out from under it, it's not really a database.

Re:Backup != snapshot != package management

[shutdown -p now]

The database will normally survive (just as it survives a hard reset), it will just take time for it to get back online, because it'll need to clean up all the mess (play back journals etc). For large DBs with many concurrent transactions this may take considerable time, especially in the context of "five nines" availability and such.

Re:Backup != snapshot != package management

[Anonymous Coward]

If a database can't sur

I took a snapshot of your post while you were in the middle of writing it, and boy do you sound like an idiot.

Re:Backup != snapshot != package management

[Craig Ringer]

"In particular, SQL databases are completely unsuitable for this kind of backup (this is why they have their own backup and transaction log handling procedures)"

While snapshots aren't ideal for SQL DBs, any real snapshot is equivalent to a point-in-time copy of the state of the file system. Restoring it and starting the database should seem to the database just like it's recovering after unexpected power failure or a process crash.

Any database that doesn't recover properly after a snapshot restore will also fail to recover properly after powerfail or a sudden hardware reset, because it's not ordering its writes properly.

Of course, proper snapshot implementations (ie not LVM) notify apps that a snapshot is about to happen so they can pause their work and enter a stable, easy to recover from state for the moment it takes to make the snapshot. So it's even easier on the database.

Now, I'll grant that for databases it's usually *better* to do incremental block-level copies, SQL-level dumps, etc using the databases own tools because doing so is usually much more _efficient_ than taking a snapshot then archiving it somewhere. But sometimes you just want the snapshot around as insurance before doing a major config change or upgrade, and for that they're just unbeatable.

While I don't much like Windows servers in general, I have a major case of VSS envy (Volume Snapshot Service, not Visual Source Safe - blech!), because it's worlds ahead of anything seen on any open *nix and has been for nearly ten years. Hell, my one and only Windows server maintains in incremental backup of its self on a remote iSCSI volume, including many point-in-time snapshots, that I can just unplug from the iSCSI storage host and boot if I need to for disaster recovery. It's impressive, and VSS is the core of what makes it possible.

Re:Backup != snapshot != package management

[GreyWolf3000]

"Some real server operating systems"... I'm detecting just a tad bit of elitism there.

Honestly, every production deployment I've seen has dedicated database servers that do not use snapshots for backup.. dumping the database itself to a backup is just fine.

I agree that it is a cool feature, but you can be a real server operating system without it.

Re:Backup != snapshot != package management

[Alex Belits]

On real server operating systems snapshot support is integrated into applications, which receive a "snapshot about to occur event" so they can quiesce their writes for a short period to make the snapshot clean.

First of all, that excludes all Unix-based systems from "real operating systems" set because none of them have this as a standard system-wide feature, and no applications actually process such requests in more than rudimentary manner. There were multiple attempts to make such notifications work, all resulted in guidelines to application developers to never rely on such system, and instead use transaction-safe file-update sequences for all non-database applications.

Second, databases usually can't process such requests due to extensive amount of data caching -- they are usually specifically designed to store as much data in memory as possible and write it to disk asynchronously, so in this scenario "make a snapshot" would cause the database to stop processing requests or slow down to a crawl due to literally tens of gigabytes of data rushing to be written to the disks. This is why such databases rely on application-level snapshots and transaction logs instead of momentary snapshots -- they produce predictable rate of write requests, and can be physically separated from the main storage media. Once database is restarted, it combines transaction log with stored data to recover its state to the last committed transaction.

For example, on a Windows server, a VSS snapshot is a complete restorable backup of everything, including your databases, event logs, the registry, etc... It's the standard mechanism that practically all Windows backup tools use. They take a snapshot, back it up, and then release it. The point in time that the snapshot was taken turns up in the "last backed up on" date field in SQL Server!

VSS is famous for being unreliable, and is not something that others would want to emulate. Bonus point to you for implying that Windows Server is a "real server operating system".

Re:Backup != snapshot != package management

[Alex Belits]

"Some real server operating systems"... I'm detecting just a tad bit of elitism there.

No, just Microsoft pride. It would be elitism if it came from Oracle/Solaris fan, but Oracle on Solaris does nothing of the kind he described.

Re:Backup != snapshot != package management

[Alex Belits]

Congratulations -- you are an idiot, too.

Re:Backup != snapshot != package management

[Alex Belits]

My Windows Home Server disagrees with you completely.
I only wish something similar to WHS with Volume Shadow Copy Provider could possibly exist for Linux. I'm tired of waiting.
You really should open your eyes and see that what you're describing is not only outdated but dangerously untrue.

Yesss, we all want to have reliability and performance that Windows is so famous for.

Are you stupid? Linux already has snapshots. And backups that don't interfere with system while they are running. My point is, that to make this work with databases, one has to sacrifice performance or availability, or greatly increase complexity of databases -- something that Microsoft, obviously, has no qualms with, but this is why its products are unreliable. Everyone else just accepted that databases are special, and stopped shoehorning them into filesystems' transaction model.

Re:Backup != snapshot != package management

[Alex Belits]

because it's worlds ahead of anything seen on any open *nix and has been for nearly ten years.

It's "worlds ahead" because it deals with system where the only database that anyone would dare to run, is written by Microsoft itself, and for all other applications it's considered perfectly acceptable to lose data if it was being written at the moment when snapshot was being made.

Hell, my one and only Windows server maintains in incremental backup of its self on a remote iSCSI volume, including many point-in-time snapshots, that I can just unplug from the iSCSI storage host and boot if I need to for disaster recovery. It's impressive, and VSS is the core of what makes it possible.

You can do it with any journaling filesystem by just placing it into data+metadata journaling mode instead of metadata-only that is usually used with, among other things, ext3. However on any system you still risk creating inconsistency at the level of applications that have their own idea of transactions that they don't expose to the filesystem. Ex: "tar cz /some_directory > some_file.tar.gz" -- having its output redirected, tar has no means of ensuring that some_file.tar.gz will not be "snapshotted" in incomplete form. While no one uses tar in this mode directly, things like redirecting output of the same tar over ssh, are very common, useful, and completely impossible to combine with such "transactions". Windows users aren't supposed to do it in the first place so it's easier to impose jackbooted transaction system on everything, however this is hardly an advantage.

Re:Backup != snapshot != package management

[kriston]

You're over three days late. Your unnecessary ad hominem attack, and your sarcasm betrays your obvious ignorance about the Windows Home Server platform and show all of us that you are not qualified for this debate.

Move along; you cannot be taken seriously.

Re:Backup != snapshot != package management

[Alex Belits]

You're over three days late.

How so?

Your unnecessary ad hominem attack

It's never unnecessary to attack Microsoft fanboys.

, and your sarcasm betrays your obvious ignorance about the Windows Home Server platform and show all of us that you are not qualified for this debate.

Windows Home Server three days ago started to works better than Microsoft's own craptacular "enterprise" products?

Move along; you cannot be taken seriously.

If I can't be taken seriously, then why do you care what I post?

Re:"does not yet support my older 2.4 Linux server

[Anonymous Coward]

2.6 came out many years ago. It's hardly bleeding edge.

Re:hey retard:

[Anonymous Coward]

How's he being an "asshole"? LVM is exactly what the topic submitter needs. It's filesystem-independent, can work with existing filesystems, has been supported since the later releases of Linux 2.2, is well-tested in production environments, and has a very minimal performance impact.

Re:"does not yet support my older 2.4 Linux server

[impaledsunset]

I'm troubled why people still run 2.4 server. I remember the time when I was reluctant to upgrade to 2.6, and I used prefer the older 2.4, which felt more comfortable than 2.6, regardless of how tempting the new changes sounded. But now I don't see any reason I would run this anywhere, even my router runs 2.6. Especially on newer hardware, 2.4 is really really too old.

I know there are people who probably still run Linux 2.2, but that are probably systems that are running some task well enough to require any changes, and leaving them as they are is the best. Servers are usually not like that. They need security updates, upgrades to catch up with the times, and many other changes required by the circumstances (for example, adding snapshot abilities, for which some person asked recently on Slashdot). Most production servers are not systems that you just leave running, so upgrades to the kernel are also expected and highly recommended. Not to mention that most recent distributions require 2.6.

rdiff-backup

[xororand]

Take a look at rdiff-backup.

rdiff-backup backs up one directory to another, possibly over a network. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, hard links, dev files, permissions, uid/gid ownership, modification times, extended attributes, acls, and resource forks. Also, rdiff-backup can operate in a bandwidth efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back a hard drive up to a remote location, and only the differences will be transmitted. Finally, rdiff-backup is easy to use and settings have sensical defaults.

I can confirm that rdiff-backup is indeed easy to use.

Re:rdiff-backup

[Gogo0]

i was doing an rsync mirror from one hard drive to another every evening (cron of course). now iv got incrementals out of 5 minutes of work.
thanks for the recommendation!

Next3

[Anonymous Coward]

Funny... I read this question while browsing /. during lunch. Next in line was Linux.com which had a link to an article on Next3: ext3 with snapshots.

Here it is: http://www.linux.com/news/software/developer/317784-next3-ext3-with-snapshots

What is wrong with just plain dump?

[mr_da3m0n]

Seriously, what is wrong with dump(8)? It works on ext3. I use it on FreeBSD. It takes snapshots to do the dump, so you can shutdown your database, start the dump and then immediately start your database again. Of course you have to backup the entire volume, but still...

Re:What is wrong with just plain dump?

[Carl Drougge]

I'm pretty sure the Linux version of dump doesn't do any snapshoting. The FreeBSD version can do it because the FS supports snapshots, but ext3 does not. (Maybe it will do snapshots automatically if you have a setup that will support them, but the original problem is that this is not the case.)

Re:What is wrong with just plain dump?

[mr_da3m0n]

You are right, I researched it a bit better, and it is dependent on the filesystem, and ext3 doesn't do any snapshotting. Oh well :(

Re:What is wrong with just plain dump?

[Anonymous Coward]

ditto, I work for a very large investment bank, and that's how we back up all of our stuff! Works great!

Re:What is wrong with just plain dump?

[kriston]

Linus said repeatedly that filesystem dumps are bad for reasons he is obviously not qualified to evaluate, but people believed him, and ext3 has no real support for dumps for NO GOOD REASON.

http://lwn.net/2001/0503/a/lt-dump.php3

There should be a Godwin's Law against quoting Linus in an argument.

Re:What is wrong with just plain dump?

[Craig Ringer]

I think the "shut down the database" bit is what's wrong with dump.

Re:What is wrong with just plain dump?

[drinkypoo]

Linus said repeatedly that filesystem dumps are bad for reasons he is obviously not qualified to evaluate, but people believed him, and ext3 has no real support for dumps for NO GOOD REASON.

Dumps are dependent on filesystems. You need support for the same filesystem where you take it out. Since many file-based archivers are capable of recording all POSIX filesystem semantics (including ACLs!) there is nothing whatsoever lost by using tar over dump, and meanwhile, much is gained: notably the ability to unpack the backup on a system without support for the originating filesystem.

Filesystem dumps are stupid in every way. tar and cpio can both store just as much information about the files on the filesystem. Snapshots are another matter but you aren't talking about them.

There should be a Godwin's Law against quoting Linus in an argument.

Especially when you totally fail to understand him. I guess Linus quotes get to be filed with Car analogies as "something most users should not attempt"

Re:What is wrong with just plain dump?

[kriston]

I totally understand what Linus says.
The real world fails to agree with him.

Filesystem dumps are good. There is important metadata that is not preserved using cpio and, hahah, you actually said "tar." Now THAT is funny.

Re:What is wrong with just plain dump?

[drinkypoo]

Filesystem dumps are good. There is important metadata that is not preserved using cpio and, hahah, you actually said "tar." Now THAT is funny.

Name some filesystem metadata you cannot preserve with GNU tar and a very small text file containing only volume info, i.e. containing the UUID and any filesystem parameters. There is none in the POSIX standard.

Re:What is wrong with just plain dump?

[kriston]

I'm talking about extents and extended attributes and other data not available to userspace. I was not talking about just POSIX filesystem data.

Re:What is wrong with just plain dump?

[drinkypoo]

I'm talking about extents and extended attributes and other data not available to userspace

Extents? I do not think that word means what you think it means.

xattrs, that's a fair complaint. Fedora apparently has a version of tar with xattr suppor. I suppose if you wanted to store them using tar you'd need to script that stage. I'm looking right now at building Fedora tar for Ubuntu Lucid.

Re:What is wrong with just plain dump?

[kriston]

I dunno, someone I know said extents mattered. *shrug*

Good luck with your project, btw.

Re:What is wrong with just plain dump?

[drinkypoo]

Extents are a mechanism for dealing with large files. ext is not particularly good at it, but it works. They are only relevant at all when talking about sparse files, and even then, compression is the technology we use to efficiently store sparse files in an archive, not storage of extents. Further, extents are simply a way of dealing with file fragmentation, and expanding an archive in an empty filesystem (i.e. restoring an entire filesystem) provides defragmentation in any case.

looks like the fedora xattr patch is nontrivial. It's making tar fail about a third of its tests. I wonder if the tests are lame, or the patch is.

Re:What is wrong with just plain dump?

[drinkypoo]

looks like the fedora xattr patch is nontrivial. It's making tar fail about a third of its tests. I wonder if the tests are lame, or the patch is.

If anyone cares, the tests failed because the patch throws some apparently non-standard-type errors (WARN: blah blah blah) if you don't apparently have ACL support. I don't know why yet. Maybe they break ACL support in their patch, too. I'm not much of a programmer. Can anyone recommend a great program for visually seeing what the results of a patch are on the tar sources if I apply only certain hunks? Prefer GTK but whatever.

Re:"does not yet support my older 2.4 Linux server

[Bratmon]

Can you give us an example?

The obvious one is Redhat. RHEL 3 is the newest version to still use a 2.4 kernel. It may still be supportes, but that expires October of this year, so the OP should already be getting ready to upgrade.

Definitely use an rsync-based program

[apexwm]

Thankfully, Linux is not nearly as complicated as Windows. Every time I've messed with Volume Shadow Copy it's turned into a huge headache. Doing similar functions on Linux just work right out of the box. I'd suggest anything that uses rsync, or you could use rsync directly. I use rsync which can synchronize gigs of data over the network, and is smart enough to parse partial sections of files. It's amazing and very efficient. Windows has nothing that even touches this, that I know of.

Re:hey retard:

[Omnifarious]

In case you hadn't realized this, It is possible to tell people to migrate to LVM without calling them names.

Upgrade kernel + R1Soft

[msh104]

Just upgrade your kernel using a manual build of the 2.6 kernel.
Also install static versions of the modutils ( insmod, modprobe, etc )
Use an external ( a machine with decent software ) for this so your compile doesn't break.
I have done so in the past and it works fine. ( and plan an update for those machines, anything with 2.4 is way to old... )

After that you can just use R1Soft hot copy,
http://www.r1soft.com/tools/linux-hot-copy/

This program is free ( as in beer ) to download and works with every block device.
You can even write to a block device if you really need to.

Their commercial offerings are pretty good as good. ( and DO work with the 2.4 kernel )
We use it here at work.

I heard btrfs supports something like this as well.

Any way, good luck!

Virtualise!

[tomm3h]

Dump the disks to a virtual image, virtualise the machines and then you can snapshot away to your hearts content. Be it via the virtualisation tech, or the host's file systems.

However, as already said, if you're still running 2.4 kernels, you're well over-due an upgrade/migration to newer software AND hardware environments. Why not combine both suggestions?

Understandably, uptime is really golden, but everyone understands that you have to upgrade and/or maintain systems at some point.

Re:Virtualise!

[Bruha]

Uptime egotism is no excuse not to patch or upgrade a system. We've had 2.6 how many years now, and we still have people holding onto their 2.4 kernels like they were prized pets or something.

Re:Virtualise!

[tomm3h]

Exactly! Customers are far more understanding of downtime if they're promised improvements.

Re:Virtualise!

[turbidostato]

"Dump the disks to a virtual image, virtualise the machines and then you can snapshot away to your hearts content. Be it via the virtualisation tech, or the host's file systems."

No, you can't count on the host system for a snapshot of a live guest.

Re:Virtualise!

[Cramer]

See Also: VMware consolidated backup

The guest OS has to have vmware tools running for the backup to work correctly. i.e. the VM has to be prepared to be backed up. And if there are things like a database running inside that VM, they have to be prepared for backup too. Simply copying the files out from under a running VM is a recipe for complete disaster.

Re:"does not yet support my older 2.4 Linux server

[richie2000]

I'm troubled why people still run 2.4 server

They fit in well with his other servers, still running Windows 98 Advanced Server Edition.

But seriously, migrate to LVM already.

rdiff-backup

[gweihir]

Works nicely. I use it for backups over the net. One very nice feature is that it does is reverse diffs, i.e. the nearer to the present time, the faster you get files restored. You also can remove older diffs without any fuss.

Wrong

[Anonymous Coward]

Rsnapshot is great, but it does not create a point in time snapshot of the entire volume or filesystem like an LVM or filesystem snapshot will.

It may be enough for the OP's purposes (it's enough for mine), but it just ain't the same thing as Volume Shadow Copy.

Copy On Write, Where for art thou?

[Anonymous Coward]

Why does Linux still lack this functionality?

Since the early 1990's Novell has had the ability to "Salvage" deleted files and even maintained a near limitless amount of previous versions with a Copy On Write functionality. It still exists, even on Linux in their NSS(Novell Storage Services) Volumes.

Microsoft finally got on board when their Server 2003 product implemented Volume Shadow Copies. This isn't nearly as good as Novell's implementation but, it was better than anything Microsoft had previously offered.

The original poster did mention etx3cow, which offers an awesome feature set. But, etx3cow has been "under development" for a long time without ever catching on.

Ext4 has recently been incorporated into the Linux kernel and there just isn't any excuse for its lack of a Copy On Write version history. Yet here we are, in 2010, yet again answering this question without any good answer. Linux should have a standard Copy On Write file system a long time ago. Its continued absence is shameful.

ext3cow should be merged into ext4 yesterday!

Re:Copy On Write, Where for art thou?

[Anonymous Coward]

SALVAGE.EXE was godly, we seldom needed tape restores.

Veritas Storage Foundation

[Anonymous Coward]

http://www.symantec.com/business/storage-foundation-basic

Corporate advertising?

[Anonymous Coward]

Is this article posted by an employee,
of R1Soft?

Two valid options.

[Lost Penguin]

ZFS and BTRFS, btrfs is included with Fedora 13 (maybe 12 too).

ZFS is found here http://wiki.github.com/behlendorf/zfs/

Upgrade path.

[leuk_he]

Isn't there some upgrade path to convert "old" partitions to LVM partition? Just like windows supports upgrading to dynamic partitions and FAT32 -> NTFS conversions.

Yes, you always need a backup, but having a backup + doing inplace upgrade is far more convinient.

Re:hey retard:

[larry bagina]

that's not as much fun.

Check out RemoteShadow

[Anonymous Coward]

http://www.advsyscon.com/products/rso/

Handles OPENVMS and UNIX

What Are You Talking About?

[Anonymous Coward]

Every time I've messed with Volume Shadow Copy it's turned into a huge headache. Doing similar functions on Linux just work right out of the box.

What are you talking about? Volume Shadow Copy(VSC) is a simple checkbox to enable it and it works! Two snapshots per day by default. Users can restore previous versions directly from their workstations with a right click, select, OK. Or, with newer versions of Windows, drag the desired version from the VSC window to where ever you want it. How is that complicated? How could you screw that up?

Meanwhile, on Linux... Write a bash script with arcane rsync arguments, create a cron job for the script, ??? Does it work as desired/expected or do you have to debug your script, cronjob, permissions? Where are the backups? How do you restore them? How do you maintain permissions and access control, especially with users that are not working local to the machine?

Here's some more information for the deluded. Windows has it's own kind of rsync. It's called Robocopy and it does almost all of what rsync does, including maintaining Windows user information and ACLs, an area where rsync fails miserably in Windows environments. But, the built in Windows features like VSC and Distributed File System(DFS) replication make the use of rsync/robocopy painful, pointless and superfluous.

If a single checkbox is a headache for you, I don't see how rsync can possibly be less so.

So, wheres the 6 bullet point solutions list?

[stimpleton]

I have scrolled to the bottem and replied after a scrolling skim. No answers for this guy yet? Just vague debate. No "use either a,b,c,d,or e".

Thats telling.

I am in this guys situation at work.

Re:So, wheres the 6 bullet point solutions list?

[Anonymous Coward]

No the question has been answered. LVM was made to provide volume management, including snapshotting. The partitions were created without LVM, so there is no snapshotting, it's that simple.
The solution is to create a new partition with LVM and then migrate the data. Yes, it's not a magic bullet, but it's the solution.

Re:So, wheres the 6 bullet point solutions list?

[h4rr4r]

So use LVM. That is the answer. Fix it, don't just whine about it.

Re:So, wheres the 6 bullet point solutions list?

[houstonbofh]

That is what I love so much about Linux... There is only one solution. Just forget rsnapshot, rdiff-backup, next3, r1soft and remoteshadow, also mentioned here. While LVM may be the best for some situations, it is NOT the best for everyone, myself included.

re

[JohnVanVliet]

why not use "dd" it is already installed

Why do mod points stop at FIVE?

[wonkavader]

This is clearly the definitive answer for this article.

We can all stop posting now.

Re:Why do mod points stop at FIVE?

[X0563511]

Indeed. After reading the posed questions, my answer is "suck it the fuck up already and start migrating into the current century"

That's a rough way to put it, but it gets the point across.

Re:"does not yet support my older 2.4 Linux server

[Anonymous Coward]

There are Enterprise Linux distributions that are both supported and still run 2.4, though not for much longer. Not everyone runs Ubuntu.

I'm sure most do not run Ubuntu. I work for a very large company and we only use RHEL and SuSE on physical hardware and mainframe LPAR.

Re: Volume Shadow Copy

[mpapet]

Volume shadow copy might not be doing what you think it is. I know it makes a big file, but restoring a snapshot has never worked as expected.

It sounds like you are just trying to keep an old dinosaur alive. Reboot with a live cd and make a snapshot of the old dog.

dd if=/dev/hda conv=sync,noerror bs=64K | gzip -c > /mnt/sda1/hda.img.gz

That applies to Windows partitions too. It's faster and more reliable *restore* than Volume Shadow Copy on identical hardware. You can also move the Old Dog onto somewhat newer hardware.

gunzip -c /mnt/sda1/hda.img.gz | dd of=/dev/hda conv=sync,noerror bs=64K

You will need to check the disk (fsck) before booting the restored partition.

If you have a say in your priorities, it's time to migrate the services off the Old Dinosaur onto another server with an LVM disk and vaguely modern distro.

KISS

[element-o.p.]

Perhaps you are making this more complex than it has to be. I've had zero success simply copying the files and filesystems from a Windows server to backup and then being able to restore anything but data -- you can't reinstall a Windows OS from the moral equivalent of a Linux "cp -R" command. Linux, however, does not share this feature. You can -- and I have -- use rsync or tar to copy the entire filesystem off of a Linux machine to your backup device, then restore an entire machine -- data and OS. The only caveat is that you will need to partition the disk on the replacement machine to match the partitions on the original machine, or else you will need to modify /etc/fstab to reflect the new partitioning scheme.

So what I do is build a backup server with a huge disk array, big enough for all the data on all the machines you manage, or at least for all the machines you want to write to this backup server. Configure rsync.conf to have a /backup directory. Then on the servers you manage, have them run "rsync -av / rsync://rsyncserver.example.com/backup/rsyncclient.example.com/" (where you replace "rsyncserver.example.com" with the name or IP address of your rsync server and "rsyncclient.example.com" with the name of the client you are trying to back up).

Re:KISS

[amliebsch]

How does that work with, e.g., running databases? Don't you run into a nightmare of consistency issues?

Re:KISS

[element-o.p.]

If your database is processing queries when your backup runs, yes. Ours don't, so it's not really an issue in my environment. But that *is* a good point.

Re:KISS

[rdebath]

There are four answers to this.

1. Yes, you have to backup the database to disk and copy that instead.
2. Yes, but it's fine if the database is idle or stopped.
3. It's fine if you tell the database you're about to do it and when you're done.
4. Nope, it just works.

Number 2 is the most common and the first is almost unknown. But all of them happen with different database tools; check your documentation.

lickwood

[Anonymous Coward]

dd if=/dev/hda | ssh your@mamma.edu "dd of=/abc/remote_hd_image.iso"

rsnapshot.org

[Anonymous Coward]

With judicious use of --link-dest rsync can do this for you without having to use cp at all.

http://rsnapshot.org/

32.5GB? Or Terabytes?

[stefanlasiewski]

Our thumper has 32.5GB alonep>

Did you mean 32.5TB, not GB?

Re:32.5GB? Or Terabytes?

[Crackez]

Yes, 32.5TB.

A method for doing exactly that

[Tolaris]

Indeed. I've done so, and documented it:

http://www.tolaris.com/2010/05/06/moving-an-existing-backuppc-partition-to-lvm/

People smarter than me documented a way to move the data, live, even when the partition is nearly full. See the comments there.

Update?

[Anonymous Coward]

*** I also found the R1Soft Hot Copy command-line utility, but it does not yet support my older 2.4 Linux servers. ***

How about updating to 2.6+?

No such thing

[Anonymous Coward]

There is no such thing. Windows locks files when it opens them, so nothing can touch it. Linux do not do that. Rsnapshot is your friend.

ESXi?

[TheRedDuke]

This might be a bit extreme and would require some infrastructure re-org, but you could use virtualization with VMWare ESXi or XenServer to snapshot your entire box. The software is free and the scheduling is pretty robust. It doesn’t directly help with the restore of individual files, but hell, you could restore entire servers independent of your production servers and pull the files that require restoration. I know, it’s a pretty roundabout solution, but, of course, server virtualization provides all kinds of other benefits beyond this. Plus, it wouldn’t matter what kernel you’re running.

Is it necessary to be a dickhead?

[Beelzebud]

Reading through this thread has brought back the memories of when I first started using Linux. There is a subset of Linux users who seem to think that acting like a giant douche bag will help people adopt the platform.

Don't get me wrong, I've found that there are some amazing people in the Linux community that are more than willing to help out someone genuinely willing to learn, but there still exists this subset of assholes that seem to think ridicule, and basically acting like a dickhead makes them superior. If you're one of those people get over yourself. Linux would be better off without you!

Re:Is it necessary to be a dickhead?

[Atriqus]

Yeah, the unwarranted nerd superiority complex is running a bit higher than usual in this story's comments. I see no other way around this:

* breaks out megaphone *
Alright guys, looks like we all have to hug it out and start over! :D

Re:Is it necessary to be a dickhead?

[cynyr]

yes, it could be phrased a bit better, "to get atomic backups of data, you need LVM with the snapshot module, or be using BTRFS. If the snapshot module for LVM is unavailable in 2.4.x, you will need to upgrade" The OP is basically asking how can i use shadow volume copy to back up my FAT16 partition, running on a windows 98 computer.

Re:Is it necessary to be a dickhead?

[Anonymous Coward]

Dude. You are on the Internet. As far as Internet discussions go this one has been civil and informative. If you're grown up you can handle a few opinionated bastards and pick up the information that is useful for you. If you can't take a few strong words without crying, you need to stop getting unpaid support from Internet forums and arrange a support agreement with a commercial provider.

Re:Is it necessary to be a dickhead?

[Beelzebud]

Yeah silly me, expecting civility.

Re:Is it necessary to be a dickhead?

[drinkypoo]

Reading through this thread has brought back the memories of when I first started using Linux. There is a subset of Linux users who seem to think that acting like a giant douche bag will help people adopt the platform.

You mean like calling a subset of the commenters in a thread douche bags without making it clear which ones you're talking about? I assume you mean me, given a total lack of information on what you think makes someone a douche bag. So, you know, fuck you sideways. If you didn't mean me, disregard that last sentence. Thank you.

there still exists this subset of assholes that seem to think ridicule, and basically acting like a dickhead makes them superior.

Pot, meet kettle.

ZFS-FUSE or VMs

[Luminary Crush]

ZFS is available as a user space filesystem (FUSE) and has lots of goodies including snapshots and snapshot replication for off-box disk-to-disk backup. It's got a much better feature set than LVM/ext3. I've used it and it seems stable - I've even imported and exported Zpools between Linux and Solaris without problems.

You can try to build it from source if not available for all your distros - not sure if it will compile on the 2.4 kernels though.

You are not going to find something you can 'drop on top of' a bare EXT3 filesystem and get robust, modern features like snapshots w/o some data migration.

You could also think about migrating these old physical servers into VMs and snapshot using the facilities available in the hypervisor (eg VMware or Xen).

You could also setup an NFS server or a NAS box with snapshots and migrate your data there, leaving the OS/apps to run via NFS mounts: OpenSolaris has NFS; Nexentastor, built on OpenSolaris, is free for up to 3TB and has ZFS/snapshots/etc, or you could deploy a Linux machine with BTRFS, etc.

Re:ZFS-FUSE or VMs

[carton]

have you actually used any of this stuff, or are you just reading web pages and echoing what you read back into the web? This process reduces S/N. I'm sorry to be unsupportive, but to have some experience is really not too much to ask.

Re:ZFS-FUSE or VMs

[GreyWolf3000]

Who modded this down? ZFS+Fuse is not something I'd recommend anyone put into production unless they really, really knew what they were doing.

There is a difference between, "ooh the mount command just let me mount a ZFS partition on my Linux box" and "I have the ability to take reliable snapshots on my high traffic, high load production system."

Re:hey retard:

[19thNervousBreakdown]

LVM snapshots suck because you can't store the snapshot data on the filesystem you're snapshotting. Sure, there's tons of ways to come up with extra space to store the snapshot data in, but they're all gigantic pains in the ass.

All it needs is the ability to exclude particular blocks from the snapshot, which should be a ridiculously easy option to implement for anyone who's worked on the snapshot code, and then people who aren't experts in kernel hacking can take care of the rest of the layers to make it a user-friendly operation, but nobody will do that because of hurf durfy crap that doesn't account for how people actually use computers.

LVM snapshots for iterative backups?

[dstech]

Depending on how long you're keeping them around, LVM Snapshots are likely to be a bad choice anyway. Their intended use-case is to have a very short lifespan, because they're intended to be used like so:

1. Create snapshot

2. Mount snapshot & copy data to backup server

3. Unmount & destroy snapshot

The point behind them is to create an unchanging version of a live partition so that you can copy the data out without worrying about whether it is being updated while you copy. Since the snapshots keep a diff of all changes to the original volume, they continue to grow in size as you make changes to the original volume. When the snapshot runs out of space, it simply dies (completely... can't mount it or anything, just have to destroy it).

There are some other possibly valid use-cases (e.g., if you have simple throw-away virtual test machines, you can build a gold image, and snapshot it and then mount & use the snapshot, which allows for a quick restore to the gold state), but keeping iterative backup copies on the local volume for quick restoration isn't really the best idea.

dd

[Anonymous Coward]

You can shadow a file, drive, whatever with dd.
'dd if=/volumeaddress of=/whereverYouWantIT/shadowedfile'

Just be very careful when you want to restore the volume, because 'dd if=/whereveritwas/shadowfile of=/' will f-up your partition so bad that you'll have to reinstall everything.

Re:"does not yet support my older 2.4 Linux server

[bertok]

Quoth Wikipedia:

4 January 2001 - Linux 2.4.0 was released (3,377,902 lines of code).

That's about the same as running Windows 2000 Server in a production environment.

Array based snaps.

[Anonymous Coward]

We have our linux systems SAN and NAS attached to EMC and NetApp arrays respectively. We take snaps from the storage array.

GNU TAR

[Narcocide]

*sigh* noobs...

$ man tar

Re:GNU TAR

[coryking]

The GNU folks, in general, abhor man pages, and create info documents instead. The maintainer of tar falls into this category. Thus this man page may not be complete, nor current, and was included in the Red Hat CVS tree because man is a great tool :). This man page was first taken from Debian Linux and has since been loving updated here.

man tar
Yet another reason to switch from Linux. The crappy state of GNU documentation.

Re:GNU TAR

[Narcocide]

My copy doesn't say that. Maybe you shouldn't be running an ancient version of a crappy distro and then bitching about it.

Re:GNU TAR

[Narcocide]

Alternately, you could just maybe look at the info page like it suggests. If you are running an old version of tar you wouldn't want to confuse yourself by looking at a man page for the wrong version of it, would you?

$ info tar

(FYI you'll want to make sure you've got the info pages installed first, of course - otherwise the info reader will just give you the man page anyway)

Re:"does not yet support my older 2.4 Linux server

[carton]

+1.

u r doing it rong.

If you need to keep around such old software, it needs to be running inside Xen/VirtualBox and/or become NFS-booted so that it's insulated from the hardware. That way, you're not forced to keep around old hardware to run your old software. If you insulate with Xen/VBox at the block level you can use LVM2 on the host system to do snapshots but are still constrained by the legacy filesystem. If you NFS-boot, you can use future filesystem-level snapshottable Linux filesystems to do snapshots, or you could buy a NetApp and use proprietary software to do the snapshots, or Solaris (if it sticks around), or any of a variety of things. You can argue about which level of storage insulation is best in the long run: the filesystem level has certain advantages at least with ZFS and probably with whatever future Linux snapshotting filesystem comes along becuase of variable block size: small files get blocks in 512-byte increments, and any file larger than 128kB gets a multiple-of-128kB sized block. Since blocks are the unit of compression and deduplication, you want the largest size block possible, but not too large or you suffer from the read-modify-write RAID5-like tax. A larger block will compress better and takes only one entry in the deduplication hash table. If you insulate at the block layer then all blocks will have to be the same size and a relatively small size, which makes compression and dedup work not nearly as well because they can't give big blocks to big files. however for old software you may want to change as little as possible, and especially with sketchy linux 2.4 NFS clients maybe it's not safe to run certain apps over an NFS root, or maybe your ancient distribution doesn't support NFS booting well.

One way or another, though, you need to find a way to keep the apps running while minimizing the blocks of ancient code on which you're still dependent, and this should be your overriding concern. You need to structure your plans to encapsulate the code subject to bit-rot rather than flailing around on /. for some freshmeat app-of-the-day that claims to solve * with FUSE and some stupid Perl script. This is the difference between a serious professional douchebag who surveys the industry and can smell the difference between high and low quality, and a pathetic flailing do-my-homework-for-me everything-inclding-windows-has-stengths-and-weaknesses-right-tool-for-the-job doomed blinking medicated idiot douchebag. Do not be that douchebag.

Re:hey retard:

[stikves]

Easy on the language...

But LVM is *the* solution on Linux. It's not very difficult to set up, and is very stable. As an additional bonus it supports all file system types (as long as they can exist on LVM volumes).

2.6 is perpetual bleeding edge

[coryking]

The 2.4 series was the last stable series of linux kernels in existence. 2.6 is in a perpetual state of bleeding edge, which makes it a gable to use on a system you care about. It is one of the reasons people (like myself) switched from 2.4 to one of the BSD's.

tar | gzip -1 /...gansta_backup.tgz

[Anonymous Coward]

fdisk, mke2fs, mount, tar -zxvf, ...

fsckin' old skewl.

The problem really is in approach

[Allnighterking]

You are approaching management of the Linux boxes as if they where windows boxes. That is the event causing you the greatest pain. Basically a Linux box can be divided into 3 groups

Configurations

Data

OS

Configurations: Two types Users Configs: this is kept in their home so no need to worry aboutthat as normal backup takes care of it (exception can be /root) System. System Configs: This is /etc/ and key entries in /var for the most part.

Data: By in large this is also maintained in homes, for that too backup software is the key. Data could also include software that either you made the mistake of building locally without some kind of packaging to repeat, or, that is 3rd party. Scripts and Cron Jobs also fall into this realm along with logs.

OS: That's why they make CD's. You just install. To make sure you get all of the same software, both rpm based, and deb based systems have ways of taking an inventory and replicating the inventory to another box.

There is no concept of registry in a Linux box. (unless you run some of the newer gnomes but that registery is by user so it's in the home.) This means you don't have to have restoreable clones to re-create the box. It's sufficient to have a copy of /etc/ a copy of key data, and either a kickstart file, dpkg list or other tool (Like a wiki plan) to re-build the box.

On the cheap tar and gzip are all the tools needed. More involved. Puppet

Re:hey retard:

[Anonymous Coward]

"but nobody will do that because of hurf durfy crap that doesn't account for how people actually use computers."

As long as they are using them *improperly* I wouldn't give a damn.

Hey, please, answer a question: how is it possible to get a filesystem-included snapshot for a full partition? Or is it that you must somehow reserve enough space for the snapshot to happen? And once you need to reserve such space, what's the difference between reserving it within the partition or out from the partition?

BTRFS = somewhat suitable for this

[cwolfsheep]

I tried figuring out how to do something like this a couple months ago: got something going with BTRFS. http://unquietwiki.blogspot.com/2010/03/quick-local-backup-with-rsync-btrfs.html

Re:"does not yet support my older 2.4 Linux server

[omglolbah]

Win2k is still used widely all over the world in production environments.

The problem with systems that work is that you're usually not to touch them until they stop working :(

Why make things difficult.

[ckaminski]

Ext3 on LVM2

Seriously, that's all you need. Why overcomplicate things?

btrfs will have this in a few years... when it's reliable.

ZFS has it now if you choose to run OpenSolaris or FreeBSD.

Re:hey retard:

[19thNervousBreakdown]

Explain why not wanting to reserve space for snapshots is using a computer improperly, because that's a pretty major claim given that nearly every filesystem that supports snapshots doesn't require reserving space for them at filesystem creation time, and a number of block-level systems support it as well.

I can't parse your first question. Your second question makes no sense either. I don't want to reserve space inside the partition, I want to use space in the filesystem.

1. I don't want to try to estimate the amount of space I'm going to use in snapshots and then make a permanent decision before even creating the filesystem.
2. I don't want to unnecessarily fragment my filesystem by filling it to 95%, while 20% of the disk is sitting unused just in case I want to take a snapshot.
3. I don't want to do the above and then still overrun my snapshot space while my filesystem still has free space because I didn't make a perfectly prescient estimate.

It's dumb, there's no reason for it other than a dogmatic clinging to some "right" way to do things. Not every system that needs to be snapshotted is a server with years of usage history and hot-swappable disks owned by a company that can afford to oversize their disks by 50% or more.

Re:"does not yet support my older 2.4 Linux server

[Snowhare]

Even if you are using SUSE or RHEL, support for 2.4 is about to vanish completely. The last version of RHEL to use a 2.4 kernel was RHEL3 - which goes EOL October this year. I actually have one very old system to convert because of this. There is no version of SUSE using a 2.4 kernel and still in general or extended support. Even for 'self-support' (it's a bit of an oxymoron - it just means you can use the forums and the knowledge base but nothing is going to get fixed by Novell. How this is different from 'Googling it' isn't obvious to me), the last SUSE support for 2.4 will EOL in November 2012.

Re:hey retard:

[rincebrain]

The way those filesystems do it is that they implement an allocator of resources beneath the actual "filesystem", so that you can snapshot things by marking blocks CoW and then allocating non-filesystem space for the metadata.

As it happens, ext and friends don't roll that way, so adding that functionality breaks compatibility with those filesystems.

Also, most of the new filesystems which allow snapshots in the way I describe have some awesome problems - like needing to truncate a file in order to rm when the filesystem is full, because the way the allocator structure works means that you can't know ahead of time how much space it takes to atomically delete a file, either...

Re: Volume Shadow Copy

[shutdown -p now]

The whole point of VSS and similar snapshotting services is to reliably make consistent snapshots of live systems - you don't want downtime for your production servers!

rbackup?

[SEWilco]

Maybe rbackup would be useful? You can retrieve individual files, and data transfer/storage are reduced so you can back up more often.

Don't make snapshots, make backups

[pseudonomous]

I'm not reading all of this crap, but there are plenty of methods for storing/restoring data that DON'T involve making backups. Here's a few:

Use dar (search for "dar disk archive" so you don't get daughters of the american revolution)
- dar allows you to get many of the features that snapshots provide by simply backing up files to disk. It supports incremental backups, and supports deleting files on a backups so when you restore from a dar backup it really does restore a given directory to it's state when the backup was made. It's also extremely complex, so you will probably end up writing scripts to automate using it.

Other Options:

-You can also use tar, but it's not as versatile, and only sort-of supports incremental backups, you could probably hack around this by scripting the capabilities yourself, but it's much simpler to use dar instead.

-Use dump/restore (I haven't done this, and this method of backing up is OLD so there may be limitations that make it unusable in this context)

-Use dd to clone partitions/disks to image files, takes a long time and creates huge files, requires harddisks with identical geometry for a restore. (well, not quite, but it makes it alot easier)

There's also some client/server stuff like bacula and amanda, but I don't use them, it seems like much more work than is neccessary. Personally I just do dar backups to an external harddrive (or nfs-mounted share)

Bring out the Rsnapshot..

[tempest69]

I tried Linux LVM snapshots, but most of the servers I manage run regular partitions with ext3 file systems, so LVM snapshots will not work.

Last time I checked they worked fine. I think you mean something else by "snapshots".

suso-- reread it with the assumption he isnt an idiot, you're making an implicit assumption about how he has his servers partitioned.

ok for most of what needs to be done Rsnapshot is awesome.
Rsnapshot uses rsync and some creative hardlinking to make a pretty nice snapshot. I find that it is awesome, until you need to transfer the archives to a new machine. It is packaged, so it installs nicely. If you have live databases you have to get trickier,, one simple way is breaking a raid 1 mirror, and running rsnapshot on the old drive.
just kidding.. the rebuild would jack your system up. And that assumes that you have the machines in raid. For live data that needs atomic control rsnapshot is really the wrong tool.

Storm

Re:"does not yet support my older 2.4 Linux server

[Cramer]

Heh. I still have both in production -- I sometimes wish I didn't, but the feeling passes when I think about how much mess it is to replace them.

(In fact, I still have an NT 4.0 machine and a linux machine with a 2.3 kernel on it; 'tho I'm surprised their hard drives still work -- SCSI for the win.)

Re:"does not yet support my older 2.4 Linux server

[ArsonSmith]

Exactly, about 2 1/2 years ago I worked for a large company that everyone here has heard of that at the time was running ~4000 servers on a modified Redhat 6.2 image. There was a large code base that got lost sometime in the start-up phase of the mid 90s that was much easier to never touch the OS then to re-write the code.

What's your budget? Use a SAN or a NAS

[Lorens]

Maybe your budget is zero, but backups in a work environment should be important.

OK a SAN costs money, but it will work with your ancient 2.4 kernels. Snapshot with ease and power. Add a few terabytes without powering off your systems. Plus you can do a lot of things like synchronous replication (for more $$$) and/or clustering. Virtualization is orthogonal, doing both might be a very good idea since you might manage not to change anything at all in the production systems.

Re:hey retard:

[dotgain]

As an additional bonus it supports all file system types (as long as they can exist on LVM volumes).

I love it when all the supported filesystems also work as well

just don't use snapshot

[Anonymous Coward]

If you don't wanto to switch to LVM, whatever the reason, you need to hack a lot around ext3 to use some form of snapshot technlogy. But to do what? I mean, you need to "keep recent copies of data around for quick restore"? Use rsync, or rsync+git. I used snapshots, and they are a lot more useful as "pristine" image to install new systems, than as daily/weekly backups, not to mention that they are usually a lot bigger than "delta" based backup systems (amanda, just to say one). So, IMHO, you are looking for something you really don't need.

crash & app consistency

[lostsoulz]

It's already been mention, but I'd be looking at snapshots in the array and/or, if your pockets are deep enough, VxFS. Host overheads drop when you're snapshotting in the array (assuming COFW isn't too onerous.) Once you're in the array though, you're operating at the block level (i.e. below any application or filesystem.) Your challenge here is to ensure your snapshots contain useful data and not an arbitrary bunch of 1s and 0s. If you're dealing with file server data, normal snapshots are generally fine. If you're running an app (e.g. Oracle,) then you must quiesce the app before taking a snapshot. If you don't, then your snapshot will be crash consistent. If you quiesce the app, your snapshot will be application consistent. There is a world of difference between the two.

rsync --link-dest

[ranulf]

In that case, using rsync locally would be a good choice.

If you use --link-dest=DIR, rsync will hard link to any files that are identical, so you can have snapshots that are accessible as an entire tree but that consume little more space that a snapshot delta.

Snapshot Software for Linux

[Anonymous Coward]

Not sure it answers the OP needs, but a short googling returns :

Back In Time

DéjaDup

TimeVault (not maintained anymore it seems)

Re:Snapshot Software for Linux

[Chninkel]

what about duplicity ? (incremental snapshots, command-line tool, librsync-based, used by "deja-dup"), rdiff-backup and rsnapshot

Re:"does not yet support my older 2.4 Linux server

[omglolbah]

Some of our systems still use token ring coax networks and OpenVMS servers.

The problem is that the fuckers just work... and keep on chugging along happily in the basement of the plant...

Err what?

[Anonymous Coward]

I tried Linux LVM snapshots, but most of the servers I manage run regular partitions with ext3 file systems, so LVM snapshots will not work

Of course they will! Just fsck the snapshot and mount it read only if needed, it works perfectly well. We back up Oracle databases this way and have never encountered a problem restoring them.

try rdiff-backup

[Anonymous Coward]

Just stick an external drive on and use rdiff-backup. Been working great for me for years.

http://www.nongnu.org/rdiff-backup/

Acronis.

[Anonymous Coward]

Use it for both Linux and Windows. On-line backup works like a charm.

Next3 File System

[Anonymous Coward]

New article here on _Next3: Ext3 with snapshots_ http://www.h-online.com/open/news/item/Next3-Ext3-with-snapshots-1020107.html on adding snapshots to EXT3 file systems.

More is here too: http://lwn.net/Articles/387231/

Perhaps the best article is on sourceforge - http://sourceforge.net/apps/mediawiki/next3/index.php?title=FAQ#Is_Next3_related_to_Ext3cow.3F

Re:Next3 File System

[Anonymous Coward]

I think that this Next3 filesystem is the best solution for the scenario presented: ext3 with snapshots.
Anyone already used it?
I hope it can be ported to ext4 or maybe a future ext5.....

A pay for solution, maybe alternatives

[Anonymous Coward]

I work for a government agency and we currently have no disaster recovery full system restore solution for our Linux servers on IBM P-Series BUT we do have file level backups via IBM's Tivoli Storage Manager product. It would be similar to VSC except you'd invoke it using a command and tell it which file/directories to restore. I'm not sure if you have a budget for this but you may want to look for alternatives to TSM (Tivoli Storage Manager).

Seems like everyone on here has very little experience in an enterprise environment, where end users require quick restoration of their files when they call up complaining they deleted their last months worth of work.

Is it possible to switch to something like Solaris, where you can take snapshots? Although a very expensive alternative AIX v6 provides the new ability to snapshot a specific volume group which can be scripted.

Another option is to look into Amanda, maybe called zAmanda now or something obscure, it's a backup solution for the *NIX(es), it supports virtual tape, so you in effect could install it locally and have a separate disk available to backup the primary volume group/drive, with that you could easily run backups for only changed files periodically through the day. Obviously every minute running a script to scan for changed files with severely affect performance... UNLESS these are just little file servers or DNS and such. With Amanda it saves files as tar and other popular file formats (I think bzip), so you could in effect just extract a single file from the tarball with a script which accepts input.

Good luck, with less and less money we've had to make more from less, as long as you could spend some time you can figure it out.

Which makes me think, if you had an available disk, even a SAN attached disk, gave it to the linux server and made periodic copies of the files that people change most often or are most worried about it would be as simple as scripting a cp, or mv, or even scp to a remote location, you would have to script some sort of front end for any end user but an Adminstrator worth his/her salt should be able to figure it out.

Have fun!

Upgrade... simple as that.

[Anonymous Coward]

Most modern Linux distros either use LVM by default, or offer it as an option during install. They also don't use 2.4 kernels. This is like saying "I can do this on Windows 2003 and 2008, let me shoehorn some other solution onto Windows 2000 and Windows NT to do the same thing, and oh, by the way, some of my filesystems ate FAT32." I think that's just as unrelaistic an expectation as searching for a panacea which will give you snapshots like VSS and VSC do. It has been my experience that there is very little actual reason for keeping an old crusty Linux box going. The system requirements of a console-only (no X) system have not changed all that much. You can run the latest versions on the same hardware you used to run the older ones on. I just finished putting Ubuntu Server 10.04 on a server I bought for less than $150.00, and I expect ti to do everything I need. Maybe your experience will be different. Maybe your Linux boxes are running a lot of custom, hand compiled software, or home-grown admin tools, but if you are a Linux novice, then perhaps biting the bullet and getting that stuff out of your environment should be a priority anyway, as it is most likely will be impossible for you to support when something goes wrong.

Virtualize

[Anonymous Coward]

You could always virtualize and snapshot while up and running.

Myself, I would just use rsync or rdiff-backup depending on whether I wanted version control or not. If live backups of something like a database are needed, there are better ways.

Re : Volume Shadow Copy for Linux?

[ismail.dhaoui]

Assuming the current situation : I don't see a lot of benefits of backing up the OS partitions. You'd better backup any config file that requires a lot of work to restore or version them using cvs/svn repository. If you are very hectic to backup You use rsync considering its mysterious options... Recommendations : - Better migrate your systems to latest supported versions. - Better use lvm for better manageability and more flexibility... Regards,

Re:"does not yet support my older 2.4 Linux server

[vegiVamp]

If it ain't broken, don't fix it.

Re:"does not yet support my older 2.4 Linux server

[pbhj]

>Not everyone runs Ubuntu.

But Fedora, Mandriva, PClinux, SUSE, etc., even Slackware all distribute 2.6 kernels now and have for some time. The fact that support for 2.4 is in it's last throes doesn't appear to be a reason to actively resist upgrading.

Re:"does not yet support my older 2.4 Linux server

[newdsfornerds]

He's probably not even allowed to install security updates without filling out three change request forms and waiting a month for final approval.

Re: Volume Shadow Copy

[Anonymous Coward]

The whole point of VSS and similar snapshotting services is to reliably make consistent snapshots of live systems

1. the old dog he's trying to snapshot doesn't have an lvm. A shutdown is probably required.

2. My claim is vss doesn't work quite like Microsoft's promises. (is anyone surprised?)

3. What's with the uptime obsession? If we're talking about an old dog that's *got* to be up 24/7, then way more time and effort needs to go into it.

Virtual machine

[jolly_rancher36]

VM is the closest thing to what you want to do in practice. Create a virtual server. Snapshot it using the VM tools. End of story.

Re:hey retard:

[Anonymous Coward]

In case you hadn't realized this, It is possible to tell people to migrate to LVM without calling them names.

asshole.

Next3 - a word from the sponsor

[amir73il]

It is a perfect timing to be asking this question. CTERA has just announced Next3 this week (http://lwn.net/Articles/391339/) and Next3 is available (for kernel 2.6). Next3 was designed for this exact purpose: providing Volume Shadow Copies over Ext3. Next3 snapshots are instantaneous and have small performance overhead. With CTERA NAS appliances, Windows clients can actually use the "Previous versions" button to view file history, just as easy as with a Windows file server. Next3 does not try to compete with Btrfs (or ZFS), it is but a complementary feature for Ext3, which people do seem to find missing and will hopefully be merged also into Ext4 some day.

BTFS

[1shot1kill]

Unfortunately EXT type of file systems do not support this as others have posted about there are many other ways to do back ups. You might want to look at BTFS it is still experiential at this time but it have many cool features built in that I think you are looking for http://en.wikipedia.org/wiki/Btrfs

"The network is the machine"

[Colin Smith]

Circa 1980 or so. It's depressing really, how far behind most people are. The really depressing thing is they continually insist they have a clue what they're doing and others have to do it their way too. Then they have an epiphany and re-invent the wheel, all over again, but this time in Ruby (with added expressivity) so it's not really the same thing at all.

 

Just install Bacula

[Colin Smith]

Seriously. Problem solved. You could even shock horror, do it properly and take the backups offsite in case someone spills his Mountain Dew on the "mission critical" servers under his desk.

No, wait. It's much better to spend your time re-inventing backups. Go ahead.
 

Simple

[Anonymous Coward]

XFS + lvm work as expected.
They know each other, creating and removing snapshots is really simple.

Having an ancient kernel and complaining about missing features is just plain stupid

google James Hamilton's Internet-Scale datacentre

[Anonymous Coward]

...stuff:

exactly on this, and excellent.

A Geek's Geek.

http://www.google.com/search?q=%22james+hamilton%22+%22internet+scale%22

Re:"does not yet support my older 2.4 Linux server

[SheeEttin]

2.4 is too old? I'm working on re-setting up a system for my dad's florist shop for the "recipe" for each floral arrangement. The thing runs Red Hat 7--kernel 2.2.
The program is written in COBOL, for crying out loud! I tried installing Ubuntu Lucid and chrooting to the Red Hat partition, but that ultimately failed, so now I have to try installing Red Hat 7 and see if it works better that way.

virtualize

[Anonymous Coward]

if you want vss the only place you are going to get it is under windows

windows + vmware server + vss = solution

linux is too disparate and unorganized - there is too much disconnection between projects, only a small sense of how things should really work

microsoft has an operating system that with all the administration tools built into the system makes for a far superior operating system for the enterprise

well, your out of luck... maybe...

[pjr.cc]

Running ext3 on raw partitions really gives you no latitude for snapshots... but why do you need them?

The conventional wisdom for snapshotting is to create a consistent point in time view of an FS. This is something thats often needed for things like db's. for example, if your running a db, you want all your db files snapshotted at the same point in time (ignoring logging for now) simply for consistency. This is where the rsync ideas fall on their bum, because rsync just wont happen fast enough to create a consistent point in time across the entire database and you end up with a corrupt db. As has been pointed out though, snapshots are NOT necessarily safe ways of backing up database data, the database should be aware of what your trying to do (see hot backup) and most can be made to do just that.

But then again, if you can rsync, you can backup (and typically thats a better idea anyways) - i.e. see bacula, etc. Rsync has its uses, but a backup tool it often is not.

There is one caveat, if you running on a SAN then there is no SAN (that i know of) that doesn't do snap-shotting in this day and age (though, in some cases it comes at a $ cost). Even just about every iscsi san does it.

Your only other option is to be migrating into LVM (see caveat below). IMHO, this is the best idea cause lvm doesnt just give you snap-shotting, it gives you much more besides in terms of flexibility.

Option 3 (and this ones a tough one) is to migrate into DRBD (or LVM is going to support dm-replication soon). But, you can migrate to DRBD now without too much effort (even in 2.4 iirc), and this would allow you to replicate ext3 data to another host running lvm where you CAN snapshot. But, as i said before i think your best choice is to migrate into lvm (drbd will cost you more in terms of sheer volume of work and drbd is not fun at the best of times).

Re:hey retard:

[rakslice]

"nearly every filesystem that supports snapshots doesn't require reserving space for them at filesystem creation time"

I don't think the guy wants to switch filesystems: "but I need something I can use on my existing ext3 file systems"

drbd

[Anonymous Coward]

Possibly you could use drbd to clone disks elsewhere then build LVM on top of that to take snapshots. This might be considered a bit overly redundant if you've already got a disaster recovery / high availability system.

Re:"does not yet support my older 2.4 Linux server

[foofoodust]

This is exactly the point. 2.6 is bleeding edge, it is not for production servers. You don't get it if you run something as critical as a router on Linux. Juniper chose BSD for JuneOS for a reason.

Re:"does not yet support my older 2.4 Linux server

[soppsa]

First of all, it's JunOS, not JuneOS. Second of all, companies like F5 (also very reputable) switched from BSD *to* Linux. Cisco's highend shit runs on QNX for that matter.

However in the case of *all* of these platforms, they're all just being used for management, all of the switching and routing is done by ASICs...