Slashdot Mirror

This is what you want. Anything using rsync rules.

For most of us who aren't running IT departments that equation comes down to something like "ZFS RAIDZ2".

For those who aren't running IT departments, rsnapshot daily, weekly, monthly and yearly is easier and cheaper. Or any versioning backup.

Easier because most such people don't work on BSD / Solaris - so setting up a NAS is the extra step making ZFS difficult.

Cheaper because a lot of data will turn out to be not worth backup - rsnapshot can more easily exclude files that don't look like they need backup. So in a typical scenario, 10 timed versions of 1 TB can stay in 1 TB. Lots of nested mounts can exclude ZFS from RAIDZing too, but it is a hassle far bigger than simple regexp.

There's really no way around it. Storage media is not permanent. You can store your important stuff on RAID but keep the array backed-up often. RAID is there to keep a disk*N failure from borking your production storage and that's it. If you can afford cloud storage, encrypt your array contents (encfs is good) and mirror the contents with rsnapshot or rsync to amazon, dropbox, a friends raid array, whatever. SATA drives are cheap enough to keep a couple sitting around to just plug in and mirror to every weekend but you'll probably find a friend's cable modem and rsync+ssh a very handy alternative (hint: check out --bwlimit option) when run from cron.

Why not do it right?

http://www.rsnapshot.org/howto/

Well, actually probably rsync will be sufficient for your needs. And rsnapshot is probably a little more than your needs. I suppose that only thing you need to configure is a rsync server on windows, a nice writeup you will find here: http://www.stillnetstudios.com/snapshot-backups-howto/

I recently used this to configure my wife's windows PC, so that it will work with rsnapshot, and backup all her projects. After configuring rsync server on windows, the rsync operation works seamlessly. Whether you will use just rsync, or complement it with rsnapshot is up to you.

rsync + hard links, works just like time machine. http://www.rsnapshot.org/

I have tried CrashPlan a couple of times on Windows and Linux, and had to give up. It would either fail to connect, or make very slow progress. It's not my broadband, since Mozy (Windows and Mac only) is fine. I also found that on Linux it would really hammer the system when backing up (4GB dual-core system) so it was barely usable.

Possibly CrashPlan's cloud service is the problem, but I'm not very impressed with the software.

For Linux and Mac backups, it's worth using something like rsnapshot, which is rsync-based and works very well to back up over 1 TB of data. It doesn't do block-level incremental backup, and it makes complete copies of files (rsync plus hard links) but it works incredibly well without writing shell scripts. It can do hourly, daily, weekly, monthly and yearly backups, and has automatic retention i.e. purges the oldest backups in a predictable way (say after 6 months or 5 years).

rsnapshot works well for Linux and (I believe) for Mac, as long as you don't need fully bootable backups, and it should work really well for photos as 99% of them won't change after being created.

rsnapshot is very similar in concept to Time Machine on Mac, but without the nice GUI (in fact, without any GUI). Your files end up in a big file tree and can be restored with any file-copy tool.

http://rsnapshot.org/

rsnapshot seems to work pretty well for incremental rsync'd backups for me. It uses symlinks to maintain the older snapshots, to save on total filesystem usage. It can do rsync over ssh for backing up remote servers/pushing local vital data to a safe remote location.

Local backup server uses Linux software RAID for good measure (5x1TB RAID 5 + 10x2TB RAID 6).

Rsync is not a backup solution, though you can build a backup solution with it.

We've been using rsnapshot to back up data to removable drives (carried offsite every day) and also to a local server with a lot of storage. The local server helps with "oops, I just deleted a file" moments. It uses hard links for identical files to save space.

There is no special recovery software needed. Just go into the directory and look at the files. Simple, easy, and no lock-in.

Just wonder if you've looked at rdiff-backup or rsnapshot and what the advantages/disadvantages might be versus StoreBackup, in your opinion. I'm in the research phase of setting up a home backup solution and had all but decided on one of those, but hadn't come across StoreBackup until seeing your comment.

what about duplicity ? (incremental snapshots, command-line tool, librsync-based, used by "deja-dup"), rdiff-backup and rsnapshot

Check out rsnapshot, which uses rsync and hard links.

Have you looked at rsnapshot?

It's based on this article:
http://www.mikerubel.org/computers/rsync_snapshots/

RSnapshot uses a clever blend of rsync + hard links to do what you want... you can store many incremental backups in just a little more space than a full backup. you can run rsnapshot on a backup server with lots of disk space, and all you need to expose on your target machines is SSH.

you'd create "backup" users on all the target hosts, generate a PKI key pair, and put the private key on your backup server. put the public key in the "backup" account on each target machine so the backup server can securely login without a password. then you just set up rsnapshot to log into your targets and it will use rsync-over-ssh to pull the data.

http://rsnapshot.org/

There's a program that automates what you describe, and it's called "RSnapshot":
http://rsnapshot.org/

If you have a system that isn't always up you want something like this to launch it:
http://bugs.debian.org/cgi-bin/bugreport.cgi?msg=27;filename=run-rsnapshot;att=1;bug=523923

On my system, each incremental backup of a 24GB dataset occupies about 600MB (depending how many files have changed). And each incremental backup is a complete, uncompressed copy of the dataset, making extracting files trivial!

It'll also backup across the network with ssh, so you can back up remote servers; it'll even back up Windows machines. It does proper backup rotation (I store two weeks' worth of daily backups, then a a couple of weekly backups, then monthly). It's totally awesome.

Two things to look into:

rsync snapshots
rsnapshot, for a better rsync snapshot

We even use a script to create versioned backups going back six months using perl as a wrapper.

Kudos for publishing the code! Can you comment on your script vs rsnapshot, which is an established incremental rsync based solution which also uses hard links to factor out unchanging files? Rsnapshot is also a perl script, by the way.

And how do you back it up?

You build two, rsync them together, and give the second to a friend or relative in a different time zone. You can give them their own partition (sized according to how much of the cost they're willing to kick in) and each of you provides the off-site backup for the other.

Of course, that doesn't protect you from accidentally clobbering your own data, as the results would be automatically replicated to the off-site array. In that case, you want to look at something like rsnapshot or ZFS.

This is great advice! A laptop as a server is also great because it uses up very little power!!! Stick one or two 1TB external drives (now $89 each) on top of it, and use it as a NAS. (see http://rsnapshot.org/ for some ideas.)

No, Time Machine is basically a GUI for the great rsnapshot utility. From the aspect of browsing the backups manually, I doubt anyone could tell which system originated them.

Get 4 x 1TB disk and minimum RAID 6. Install Linux. Install rsnapshot, which offers:

* Filesystem snapshot - for local or remote systems.

* Database backup - MySQL backup

* Secure - Traffic between remote backup server is always encrypted using openssh

* Full backup - plus incrementals

* Easy to restore - Files can restored by the users who own them, without the root user getting involved.

* Automated backup - Runs in background via cron.

* Bandwidth friendly - rsync used to save bandwidth

You may also find CentOS or Debian tutorial useful.

Good luck!

rsnapshot?

http://www.rsnapshot.org/howto/

Here's the one thing to remember in terms of rsync. It's going to be the CURRENT snapshot of your data...you're going to want to have made additional local backups on a regular basis so you can roll back to one of those snapshots prior to when you hosed your DB.

rsnapshot may be the tool for you..."Using rsync and hard links, it is possible to keep multiple, full backups instantly available. The disk space required is just a little more than the space of one full backup, plus incrementals."

Here's the one thing to remember in terms of rsync. It's going to be the CURRENT snapshot of your data.

Rsnapshot may be an option. It creates a sucession of snapshot directory's but only one copy of each file (hard linking to make up the difference)

I was in a similar bind not too long ago.

My dilemma was this, I need some kind of NAS/filer that is backed up, but the backup method should be such that every time my "liberal-arts-major" wife deletes something, she does not have to bug me to get the data back. (I don't want to spend all my little free time grabbing tapes, and then hunting and untaring just the right files.)

In a perfect world, someone like NetApp would make a cheep (sub $300) consumer head-end that USB drives could be connected to. I pick Netapp specifically because of how much I love their snapshot utility. But alas, no love for us.

While I was hunting around for solutions, talking to all of my sysadmin friends, and getting all of their cool scripts to automate backups, I came across the utility called rsnapshot. http://www.rsnapshot.org/

Rsnapshot is the same thing as netapps snapshot; it allows you to make a mirror of your current drive, and backup every X amount of time. (read the website, they explain it much better.) Basically, all you do is setup a second drive, and have rsnapshot mirror the data to it automatically. You then mount the second drive read-only, and within it there are directorys that are 5 min, 30 min, 1 hour, 8 hour, 1 week, 1 day, 1 year, etc. copies of the primary drive. So now when the wife wants to get data that was lost, all she has to do is go into the other drive, go into the subdir when the data was backed up, and get the file. - Simple!

Total cost:
Beater PC: free or $500
Extra IDE card: $20
Two "filer" disk drives: (how much do you want to spend?) 500GB for $200 each
A unix distro: free
rsnapshot: free
nfs and samba: free

Easy

When I first heard about "time machine", my first thought was that consumer-grade commercial software had finally discovered rsnapshot. It's packaged for Debian, and available in "sarge" -- that makes it at least three years old.

Rsnapshot is an rsync-and-hard-links based scheme that also doesn't store duplicate data, and provides nice date-indexed browseable full file trees, much like the way both "time machine" and this flyback gizmo are described.

I haven't been this excited since AOL re-invented "ytalk"...

It's easier to just use rsnapshot.

What's important in backup is that whatever system you use meets your needs. My needs are I want full automated snapshots of important filesystems taken at regular intervals (code, /etc, and the like), and I need offsite backups for important, but not necessarily often-changing files (credit card/bank statements as PDF, trade confirmations, my journal, papers from college, etc.) I stress "automated", because I would never remember to do backups on my own.

I use a free tool called rsnapshot to make automatic daily incremental snapshot backups of all machines in my house, plus all hosting accounts. The package conserves space by using hardlinks between snapshots for unchanged files. I keep 1 week of daily snapshots, 4 weeks of weekly snapshots, and 9 months of monthly snapshots.

The package uses rsync to be efficient over the network, and authenticates with remote servers using ssh certificate-based login.

It can backup from any host that supports ssh and and rsync, which means any BSD/*NIX, plus Windows if you install the proper tools (cygwin works well for this. Has an ssh daemon...er... sorry... service, etc.)

The backup server is an old Linux box with a 1.something TB RAID5. Every so often, I tar up and gpg up all the snapshots, burn 'em to DVDs, and leave them in the drawer at my office.

This way, I always have onsite backups that are never more than 24 hours old, and somewhat-regular offsite backups in case my house burns down.

The reason that I typed all of this in, is that your backup needs sound similar to my own. Hopefully I was able to help.

I use rsnapshot to do version control of my entire system. From the description:

rsnapshot is a filesystem snapshot utility for making backups of local and remote systems.

Using rsync and hard links, it is possible to keep multiple, full backups instantly available. The disk space required is just a little more than the space of one full backup, plus incrementals.

Personally, I configure rsnapshot to generate snapshots every 4 hours, and then daily, weekly, and monthly.

In your case, since you only want versioning for your configuration files, you can point rsnapshot at just the configuration directories (probably just /etc).

Why shut down your home system? Why not have it available as a server to make your life easier? I agree with other posters about using "offline" mode of Thunderbird and like clients.

In case you're thinking that you have a particularly repressive ISP...

My ISP blocks ports 80 and 25 - particularly irritating, if you ask me. My ISPs TOS, if read to the letter, would mean that multiple browser windows or tabbed browsing are inappropriate because it's more than one session over the broadband pipe.
 
I agree that it would be ideal if I could use every port I want, block the ones I want to firewall - but I'm too cheap to pay for that kind of access.
 
So I work around it. I use dyndns [dyndns.com] to create a pointer to my dynamic IP address. My ISP does not block https or ssh ports, so I leverage those to get what I want.
 
I use cron, fetchmail [berlios.de],
  procmail [procmail.org],
  spamassassin [apache.org], and
  postfix [postfix.org] to bring mail from my ISP to my local system.
 
I use uw-imapd [washington.edu] to share my mail with other computers on my home network
 
I use ssh and pine, or apache+php+MySQL+https (self-signed cert) with roundcube [roundcube.net] to get remote access to my IMAP server.
 
I use WinSCP [winscp.net] to get access to my files at home when I'm at work. My data is *MINE* and I easily back it up (nightly and offsite qurterly - snapshot backups coming soon thanks to rsnapshot [rsnapshot.org], perl and rsync)
 
Every tool that I use is free of charge and as free as the GPL and apache licenses are free (zealots can feel free to argue with someone else about the relative freedom of the GPL, thanks.)
 
I certainly could pay for more open TOS with an ISP - I could even host my applications at an ISP. I'm cheap, and this solution works well enough for me.
Hope you find a solution that works for you!
 
Respectfully,
Anomaly

My ISP blocks ports 80 and 25 - particularly irritating, if you ask me. My ISPs TOS, if read to the letter, would mean that multiple browser windows or tabbed browsing are inappropriate because it's more than one session over the broadband pipe.

I agree that it would be ideal if I could use every port I want, block the ones I want to firewall - but I'm too cheap to pay for that kind of access.

So I work around it. I use dyndns to create a pointer to my dynamic IP address. My ISP does not block https or ssh ports, so I leverage those to get what I want.

I use cron, fetchmail,
procmail,
spamassassin, and
postfix to bring mail from my ISP to my local system.

I use uw-imapd to share my mail with other computers on my home network

I use ssh and pine, or apache+php+MySQL+https (self-signed cert) with roundcube to get remote access to my IMAP server.

I use WinSCP to get access to my files at home when I'm at work. My data is *MINE* and I easily back it up (nightly and offsite qurterly - snapshot backups coming soon thanks to rsnapshot, perl and rsync)

Every tool that I use is free of charge and as free as the GPL and apache licenses are free (zealots can feel free to argue with someone else about the relative freedom of the GPL, thanks.)

I certainly could pay for more open TOS with an ISP - I could even host my applications at an ISP. I'm cheap, and this solution works well enough for me.

Respectfully,
Anomaly

I have a "system" for backups which includes:

1. Nightly rsync of my iMac and powerbook to a hard disk connected via firewire to my iMac (runs from cron)

2. Plans to install rsnapshot to shorten the window of exposure from 1 day to 1 hour. Used to use this on Linux with great success, fully expect that this will work well on OS X.

3. I bought 2 firewire/USB drive enclosures, and populated them with PIDE drives. I keep one, and gave my sister one. The enclosures are identical and the drives are partitioned with one Windows partition and one OS X partition. When I see my sister (a few times/year) we trade enclosures. I rsync my home movies/pictures/music to the OS X partition, and she uses the "freeware" SyncBack to back up her data

The only down side to this approach is that I'm limited in backup size to ~140GB unless I'm willing to pony up for a pair of SERIOUS TB sized drives. In general it's not a big deal - I suppose I could end up losing my home movies of my kids - that's what really eats up disk space. Guess I need to archive my tapes to the safe deposit box. :)

The idea that I'll be able to easily recover my data if my house burns down brings me great comfort. The cost is fairly minimal and the level of effort is pretty low, too.

I'll second this. rsnapshot can be installed via Darwinports (or manually; it's only two files:, a Perl script and a config file) and works beautifully on OS X.

I've written up an rsnapshot on OS X howto as well as an overview of my own backup system.

I'm now using an external Firewire drive for my backups (the above hasn't been update to reflect this yet) and have written a wrapper script for rsnapshot that mounts the drive before running and unmounts it after. I'll be updating the article soon with details.

I use rsnapshot. It's written in Perl, and uses rsync, so it should work on Mac OS X as well as it does on my Linux box. It's pretty configurable, and rotates backups hourly, daily, weekly, monthly, etc. It uses filesystem hardlinks to do incremental backups.

I moved to Linux in 1994 as my primary desktop and server OS. About three years ago I decided that I wanted to produce some video content. Video editing was theoretically possible in Linux - I hooked up my camcorder to my Linux box and did some editing, but the tools were primitive and cofiguration was unusually difficult.

Eventually I looked at OS X and iLife. I decided to jump to a Mac. What a great move!

I found that Linux made it possible to do some things, but OS X made it simple to do them.

Fast forward a few years. I now have a few macs at home - their licensing policy makes it affordable to have several machines and a five user license for the OS and tools. My family loves the power and usability of the Mac.

Recently my linux server at home began acting a bit flaky. I did some analysis and determined that hardware replacement was needed. After checking prices for CPU/motherboard/RAM (and potentially hard disk) I figured out that I'd need a few hundred bucks to replace the CentOS box with a new one. After thinking about whether to drop a few hundred bucks or not on this server, it occurred to me that I might be able to move all of the services hosted on linux to OS X.

I found that samba,
  hotwayd,
  dansguardian,
  uw-imapd,
  fetchmail,
  procmail,
  spamassassin,
    rsync,
  rsnapshot,
  apache2,
  MySQL4,
  PHP,
  perl,
  java, and
  squid were all available for OS X.

Most of these are "in the box" with OS X. The only ones that I need to compile from source are uw-imapd and squid! Of course I need the bundled developer tools to get a compiler, and the Apple/BSD startup mechanism and the netinfo wierdness require some tweaks - but since when did Linux *not* require any tweaking?

What this means to me is that after more than a decade of running Linux at home (and work) I am *this* close to shutting down Linux for good at home.

Hope your experience is similar.

Regards,
Anomaly

PS - I share your recent comments about the loss of a pet. :(

My Linksys WRT54G (not GS) is a transparent Squid proxy already. I don't see why this ASUS machine can't do the same.

It points to a secondary FreeBSD machine for that, because I have a 5GiB cache on the Squid side. Everything is anonymized through Privoxy + Tor, with no configuration changes on the client side.

Users don't even know (or care) that their traffic is being proxied or anonymized at all.

For user data stored on the FreeBSD machine, I also use rsnapshot to do backups of another disk slice that is GELI encrypted as well, which works out very nicely for the overall solution.

Everything that goes out port 80 (or comes back in on the response) through the Linksys is redirected through the Squid server on the FreeBSD machine. iptables(1) on the Linksys does all the magic for me, as follows:

iptables -t nat -A PREROUTING -i br0 -s ! 10.0.1.6 -p tcp --dport 80 -j DNAT --to 10.0.1.6:3128
iptables -t nat -A POSTROUTING -o br0 -s 10.0.1.0/24 -d 10.0.1.6 -j SNAT --to 10.0.1.2
iptables -A FORWARD -s 10.0.1.0/24 -d 10.0.1.6 -i br0 -o br0 -p tcp --dport 3128 -j ACCEPT

There's more to it, but that should get you started. Its really easy to implement, and I'd trust my FreeBSD machine to process those packets faster than the processor on the Linksys ever could (not even considering the storage requirements for such a caching mechanism).

The Squid cache on the FreeBSD side resides on a partition that is GELI encrypted. Do I have anything to hide? No, but I do have a right to protect the identity of my users, their browsing habits and their data.

Everyone else should do the same (or similar).

See also: rsnapshot.

I use this locally on OS X at home and have implemented it at work to backup all our production Linux systems to a single off-site file server. It works great, and doesn't have the PHP and PEAR requirements that RIBS does. Assuming everyone already has SSH and Perl, all you need is rsync.

It's downsides: it's basically just a wrapper for rsync. It requires a lot of babysitting (if your backups fail for some reason, it'll try to do full backups the next day possibly with disasterous consequences as it tries to jam hundreds of gig down your T1). Also, it has to log in as root on all of your boxes, so there are some very careful sercurity considerations.

But a box with a bunch of disks in it, put it off site, and whamo you have a complete backup solution.

For the windows users, I like backuppc. I have never actually used it, but it allows windows users to choose when their backups are taken, and allows them to recover files themselves through a web interface. It's big downside is the cryptic way it stores files internally, making it really hard to extract files without using the web interface.

in terms of "yet another rsync backup utility" I find rsnapshot to work extremely well.

Does anyone know how Dirvish compares to rsnapshot?

For what it's worth, rsnapshot is based on Mike Rubel's scripts that you refer to.

I use and recommend rsnapshot for taking disk-to-disk backups of unix based servers and PCs. It has a *really* slick directory structure where each daily/weekly/monthly backup directory is a *full* snapshot - but by using hard links, it only saves the changed files multiple times. Also, because it uses rsync, it only copies changed files across the network, and can use ssh no problem.

It's downsides: it's basically just a wrapper for rsync. It requires a lot of babysitting (if your backups fail for some reason, it'll try to do full backups the next day possibly with disasterous consequences as it tries to jam hundreds of gig down your T1). Also, it has to log in as root on all of your boxes, so there are some very careful sercurity considerations.

But a box with a bunch of disks in it, put it off site, and whamo you have a complete backup solution.

For the windows users, I like backuppc. I have never actually used it, but it allows windows users to choose when their backups are taken, and allows them to recover files themselves through a web interface. It's big downside is the cryptic way it stores files internally, making it really hard to extract files without using the web interface.

Check these out:

Easy Automated Snapshot-Style Backups with Linux and Rsync

rsnapshot is a filesystem snapshot utility for making backups of local and remote systems.

RAID-5 for home networks is a solution looking for a problem. RAID-5 is nice for minimizing down time, but for a home network that is very seldom the case.

You see, the problem is usually not that my harddisk failed, but that I need to get an older version of a file, or get a file I deleted by accident. RAID-5 is utterly useless for this. For most home users it's better to use something like rsnapshot and take daily/hourly snapshots of their main harddisk to other hardisks.

Just wanted to provide a link to another great tool.

RAID is a perfectly reasonable backup solution from a hardware point of view. It protects you from drive failure, gives you quick access, and is currently the best bang for your buck if you have non-negligible amounts of data to back up.

Of course if all you do is overwrite a single snapshot every night, all that protects you from is drive failure on the main machine, but that is a software issue. There are plenty of bone-headed ways to back up to tape as well, and as has been established, optical is far from reliable when it comes to long-term storage.

Check it out!

You need rsnapshot and a *nix box.

I have up to 12 months worth of backups, including hourly snapshots of the previous 24 hours, always available to all the users online. Tapes do not have to be hunted down and mounted. The users can easily go browse their own snapshots and get the files they need when they have an oops. The snapshots are maintained on a different machine than the source data, so an individual machine can melt into a lump of slag with the loss of no more than one hour of data.

Since it uses rsync and hard links, only the changed files get added to the backup, so you can size your backup hard drive at roughly 50% larger than the source drive and never have to worry about running out of backup space.

If you have a reasonably fast network connection, you could even have the backups located off site with no extra effort beyond the initial setup.