Slashdot Mirror

I use Bacula for my home computer; it feels powerful enough for a small office, and is very versatile.

It has three main components: a client daemon that you install on the computers you want to back up, a storage daemon that you install on the computer that will write the backup files and/or tapes, and a director daemon which controls the backups. The director and storage daemons only run on unix-like operating systems (BSD, Linux, Solaris) but the client daemon has also been built for MS-Windows.

http://blog.bacula.org/

Part of your choice of solutions will depend on the nature of your data. Is it changing often? At all?

I use Bacula for my backups. My wife has a photography business and her collection of images is about 6TB and is being added to constantly and occasionally edited. The Archive is about 5TB and is stuff that is unlikely to change. Then there's the Working array, which is 1.5TB (max) and generally clocks in around 700GB. This is current work that hasn't been delivered to the client yet (RAW files from recent weddings/portraits, JPEGs where the client is still picking out what they want, PSD files for current album designs, etc). Both are on RAID5 arrays, and the Working array has a hot spare. At the end of each month, the Working folders are gone over for bodies of work that have been delivered to the client and are unlikely to change. This work is then moved to the Archive, backups are burned to DVD and also copied to an external hard drive.

For the Working backups, I have JBOD on another server. I think it's 6 1TB disks. These are set aside for different Bacula pools of volumes. There's two full backup pools, two differential backup pools, and two incremental backup pools.

Every night at 5AM a Bacula job kicks off. On the first sunday of the month, a full backup of the Working array is dumped to the JBOD. Bacuala makes a backup copy of everything (~700 GB). On the other sundays of the month, a differential backup dumps everything that's changed since the previous full back up to another set of volumes on the JBOD (~80GB). On every non-sunday of the month, an incremental backup copies over everything that's changed since the previous incremental backup (or differential or full backup if it's a monday). I have two sets of these pools. On odd months, it uses Full-Pool1, Diff-Pool1, and Inc-Pool1. On even months it uses Full-Pool2, Diff-Pool2, and Inc-Pool2. This way I have two sets of backup copies of everything so I don't have to delete last month's full backup to make this month's full backup.

It works pretty well, and every morning I get an email telling me that all the backups worked fine and the arrays are stable. I know it's a little anal, but well, I couldn't imagine having to tell a bride "Hey we lost your wedding photos. Hard drive crash. Too bad." With the system I've got, unless the house burns to the ground I'm fine. And if the house burns to the ground, I've got bigger problems. I wouldn't mind an off-site solution, but I don't see how I can transfer the several TB of backup data I have at any given time someplace else, except by carrying hard drives out of the house every day, and I don't think that's something I'd be able to stick with for very long.

Not to be a jerk, but RTFM: http://www.bacula.org/en/dev-manual/main/main/Migration_Copy.html

Count Bacula as your friend ;) -> http://www.bacula.org/

http://www.bacula.org/en/

There's even a howto here:

http://wiki.bacula.org/doku.php?id=removable_disk

http://www.bacula.org/en/

There's even a howto here:

http://wiki.bacula.org/doku.php?id=removable_disk

if you want to have backups using deduplication (i.e. 100 identical boxes only get saved once ) you can use bacula.

more about that here : http://www.bacula.org/en/dev-manual/main/main/File_Deduplication_using_Ba.html

Surely you jest? Getting amanda to encrypt your backups. Is just a matter of reading some howto files on amanda's website. And, just peeking over at bacula's website, I can see that they have a similar sort of setup. I don't use bacula, but I'm sure it is a matter of following the directions just like with amanda. It is not clear how anyone can consider encrypting backup tapes as a difficult process. For that matter, with TrueCrypt, OpenSSL, GnuPG, FreeBSD's geli, and linux's dm-crypt encryption in general has become easy and accessible. Add to that the hardware acceleration built into most new systems or just pure computational power of modern processors and organizations are remiss for not using encryption at nearly every turn. If you don't, you should lose your job.

Surely you jest? Getting amanda to encrypt your backups. Is just a matter of reading some howto files on amanda's website. And, just peeking over at bacula's website, I can see that they have a similar sort of setup. I don't use bacula, but I'm sure it is a matter of following the directions just like with amanda. It is not clear how anyone can consider encrypting backup tapes as a difficult process. For that matter, with TrueCrypt, OpenSSL, GnuPG, FreeBSD's geli, and linux's dm-crypt encryption in general has become easy and accessible. Add to that the hardware acceleration built into most new systems or just pure computational power of modern processors and organizations are remiss for not using encryption at nearly every turn. If you don't, you should lose your job.

You know I have had great success in the past with bacula. I think it still uses rsync for the actual transfer of files across the network. It maximizes disk space on the backup server by only storing one copy of any file. I am not entirly sure how it does it, but you can provide network backup for 50+ clients (this was my use case) with disk space of only about twice that of a single client assuming all your clients are running the same os.

Bacula is a good network backup solution. Check it out at http://www.bacula.org/

* what's the right strategy here?
OUTSOURCE. EVERYTHING.

  * What routers or switches or other equipment should I acquire?
Routers and switches: depends on $x connections and @y traffic volumes. You'll also need:
  * A cable tester
  * An 8P8C crimp tool
  * Rollover cables and DB9 adapters
  * At least 1 PDU per rack
  * patch panels (NOT belkin! They suck and your budget allows for anything worth its salt, e.g. Matrix)
  * cable management brackets
  * A KVM drawer
  * USB hubs like these (any decent hub will do)
  * RS232 to USB adapters
  * UPSes
  * A SAS tape autoloader should be sufficient in most cases, otherwise get an iSCSI tape library. Stay away from Veritas and Symantec software, try bacula.
  * If you need more storage, an iSCSI SAN server (there's AoE and it works but I doubt the cost-effectivenes and only CoRaid supports it).
On the networking equipment (and PDUs), connect the rollover cables to the console port and the other end to a DB9 adapter. Connect all the serial ports to the USB adapters and to the USB hubs. Connect the USB hub to the server you operate with the KVM drawer. Require at least public key authentication for ssh access to that box.
Set up the Linux servers to boot on the serial port (in the BIOS, grub and init) so you can easily remote in even when you can't ping it. You could even use a modem and pppd with MS-CHAPv2 to provide a remote getty when the Internet is down. You can further restrict it using e.g. pam_opie. Alternatively, set up a VPN over 3G but that requires a third server somewhere. Or simply cross your fingers and hope you'll never need it (it's all about cost-benefits).

  * What books should I read?
Linux In A Nutshell, ISBN13 978-0-596-15448-6. Using Samba 3rd edition, ISBN13 978-0-596-00769-0. Something on IPv6 as well as both "CCNA 1 and 2 Companion Guide" and "CCNA 3 and 4 Companion Guide" from Cisco press (doesn't cover IPv6, so that's why you'll need a separate book for that). If that's not enough, read up on open LDAP. But most of all, Read The Friendly Man pages.

  * Should I take classes from Cisco, Global Knowledge, my local community college, or somewhere else?
Yes, as a network admin you should get CCNA. LPI is nice to have but not essential and I doubt you'll ever need RHCE.

Don't buy network appliances (e.g. spamfilter/proxy/etc.) unless you have a really good reason to because most of the time these are black boxes hiding Linux with a crap (and potentially vulnerable) userland. As a general advice, Brocade FastIron switches are great for an "unlimited" budget. They support just about every standard under the sun and then some. If the budget is not /that/ unlimited, HP ProCurve will do. Cisco Catalyst is hardly worth the expense.

Lastly, do not accept anything less then 1000Base-T for every port on your network. Do not use UTP for the cabling between the patch panels and wall outlets (use STP or at least FTP instead, STP is perfect for an "unlimited" budget).

AMA!

Cant be more than $10,000? I have single switches worth 6x that.

When I said $10,000 that was clearly tongue in cheek. As in the PHB saying, "Do whatever it takes to get this up and running! Hear me? Whatever it takes. As long as you can do it for free."

You'd be surprised how often some people actually get this as a demand from their boss, myself included. It's amazing how one can build an entire infrastructure for free nowadays with open-source solutions like Zimbra*, Resara*, and Bacula*, along with a few little TurnKey appliances here or there. The only real cost is hardware which, thankfully, is getting cheaper all the time...

* why do all of these things end with "uh"?

Pick a project like bacula (best backup software made to date). Use it adopt it spread the word. After that you can support the project, they have a bunch of items on the to do list ( http://www.bacula.org/misc/Vote-2009.html ). If one of the items would help your work, sponsoring project would be a way to help open source software. -Jason

Well, I don't think there's an ideal solution, but I can tell you what I do.

For software, I find Bacula to be a very effective solution. It's open source, cross platform, and very flexible. Bacula was designed with tape in mind, so it takes a bit of wrangling to make it work well on hard drives - but once you get it set up properly, it works quite well with disk.

Now, I back everything "critical" up using bacula onto my ZFS array of cheap drives on my Solaris box. I just let ZFS itself do the compression (I didn't benchmark this, Bacula's compression may be more effective) and retain the backups on disk for about 2 months. I do nightly incremental backups and monthly full backups, but Bacula gives you lots of different options in this respect.

I then take the Bacula backups and rsync them over to external media weekly. I also take and keep 4 zfs snapshots of these backups on my external media, so that I can go back 4 weeks prior if I need to. I also rsync over to a separate smaller external drive "every now and then." I keep that other drive in the opposite side of the house (better would be to leave it at a friend's house, but I'm lazy - I just have to hope that one end of the house survives a fire / theft unscathed).

I have another class of data - data that I deem important, but also capable of being re-acquired at minimal expense. This data gets no incremental backups, and is only rsync'd around. It gets put on the larger external hard drive, but not on the smaller secondary drive. Beyond that, I have a third class of data, which I deem completely expendable. This is mostly normal recordings from my MythTV machine, which I consider an acceptable loss, and these aren't backed up at all.

At the end of the day, there really is no magic bullet. I really like disks + bacula, but what works best for you will depend on what you're trying to back up (and how much value you place on making sure that this happens properly).

And for those who don't like to pay $10000 for backup software, there's Bacula. Couple that with an LTO-4 drive (~1000) and LTO-4 tapes (800GB uncompressed, ~60/piece) and you're set. Rsync.net is a decent, cheap online provider for those gaps when you haven't rotated tapes.

Bacula is pretty sweet because it lets you backup to disk volumes and then you can schedule a roll to tape. So you can just back everything up incrementally to a disk volume and then copy those backups to tape, and then run rsync on the disk volumes to have an offsite, online backup. When recovering, you ask to recover from whatever's available. If you keep enough disk storage around (and there's really no reason not to) you can recover to any date in the past. In the event of a disaster your tapes come into play.

Now with drives so cheap the temptation is to buy a external hard drive and use that. But tapes have a long history, guaranteed backwards compatibility (planned anyway, LTO drives have to R/W the previous generation and Read 2 generations back), last longer than moving drives, are simpler, lighter, more robust and more portable. Not that I wouldn't keep a external around to dump desktops but tape is the DR standard.

This is considered brief: http://www.bacula.org/en/dev-manual/Brief_Tutorial.html The sixth result involves rsync.I've written batch files in the normal Windows shell, using Winzip with 256 bit encryption that take less time than trying to read the documentation for any Linux backup tool.Nice try with the Net Meeting crap, nobody uses it and you know that.The defrag tool in Windows isn't written by Microsoft so try again. It actually does a better job than watching my XFS tools kill my MythTV storage directo

This is considered brief:

http://www.bacula.org/en/dev-manual/Brief_Tutorial.html

The sixth result involves rsync.

I've written batch files in the normal Windows shell, using Winzip with 256 bit encryption that take less time than trying to read the documentation for any Linux backup tool.

Nice try with the Net Meeting crap, nobody uses it and you know that.

The defrag tool in Windows isn't written by Microsoft so try again. It actually does a better job than watching my XFS tools kill my MythTV storage directory. Oh, and don't forget that most of the Linux defrag is imaginary. Journals my ass.

Remote Desktop, I add a remote user and I'm done. I don't have to worry about the X server and my X client not liking each other. It just works.

About 12 months ago, we stumbled across ClarkConnect and have been using it extensively in deployments of 30 users or less. Felt compelled to reply to the OP since you mention archive and backups.

It has a very simple email archiving module (using MySQL backend).

For backup, we've been using the Bacula module - but the CC team just recently begin offering a remote backup service that is much more elegant.

All in all, one of the best OSS-based platforms for small businesses, IMO.

S.

http://www.bacula.org/

Well for backups you could try this http://www.bacula.org/en/

http://www.bacula.org/

Runs pretty tight (low bandwidth), supports channel encryption and datastore encryption, can even create Bare Metal Recovery disks. I have a server room with LTO3 tape drives that I use to backup my clients' incremental data changes nightly, including Linux, Mac and Windows clients and servers. I have VPN's out to each client, so don't use the built-in channel encryption, but I maintain a keypair for each client.

Backup only, but I /could/ present a maintained volume as a share over the VPN. Bacula supports disk and tape volumes as backup stores. I've personally had no need to do that to date.

We're not talking terabytes here - my ISP would pwn me if that was going on, but I do circa 20G of data changes every night from clients. Some of them are laptops that are not always on or connected. Most are friends and family PC's, so it backs up when it can. I have to do almost no maintenance apart from changing a tape occasionally. The backup client is tiny and unobtrusive, even when running. On Windows it uses VSS, so it is reliable.

I have had a number of panic phone calls (esp from my kids at Uni) who have lost a thesis or the like and are utterly amazed when, after a few clicks over the phone they look at their webmail and yesterday's version is in their inbox. That's what it's all about! I am the god of lost data! Which, of course, works for me.

SQLite doesn't scale very well. I set it up as the backend database for Bacula, and it struggled with a database of only a few hundred megabytes, and that's really not a "big" database. When I replaced it with MySQL, all the performance problems vanished.

SQLite is cool and has its place, but I wouldn't use it for storing any more than a few megs.

Task Scheduler to copy files from client to a network share? Can't be all that complex to set up a basic data backup routine...

Samba has a pretty easy GUI setup, even in Ubuntu. It's also already installed, I believe.

Software RAID is already built-in. If you use Fedora instead of Ubuntu, you can use LVM's GUI tools to do all of the dynamic partition sizing goodness.

Use the Package manager to install rdesktop, which allows remote desktop access to any Windows box. Done.

Ah, now there's one that you've gotten perfectly correct (IIRC), and why I use Bacula on my home network (which is admittedly not something for the casual user).

I'm not so sure I'd want any un-hardened machine to be accessible from the Internet; esp. a Windows one that both streams media and holds all of my personal data in one easy-to-reach location. That's just begging for a first-class arse-pounding from the first script kiddie to see that you've done that.

I'm sure you probably have... but I don't think you had all the facts at hand when you did. Now know that I'm not knocking your choice at all - if you use something as a beta and like it, and it works for you, cool... but I think that you haven't really looked all too deeply into the alternatives, you know?

Personally, I find that spending $169 for just the OS (when I can get at least an extra hard disk with change left over at that price) to be a bit much. There is also the headaches specific to Windows - the high probability of being targeted, the EULA that says I do it MSFT's way or no way at all, the 'phoning home', the DRM, the extra overhead (I stick with runlevel 3 on my home servers), and the fact that there really isn't much I can tweak on it (at least by comparison)... But then, I do the sysadmin thang for a living - so my needs, skillset, and priorities are a lot different from that of the average home user.

And so it goes... :)

/P

I run Bacula (it's not just for the enterprise, folks) and back up all the important data to the disk array.

I think I peek in there once a month or so, mostly to check disk space and see to patching. The box has zero Internet connectivity, so no probs there.

/P

AFAIK bacula supports VSS:
See http://www.bacula.org/dev-manual/Windows_Version_B acula.html:
"In version 1.37.30 and greater, you can turn on Microsoft's Volume Shadow Copy Service (VSS)."

(But I never tried it - I only remembered that I read about VSS support in bacula and then found the mentioned page with google)

I've been using Bacula to backup servers at my agency (300GB+ of data) for the past year and a half. I've found it to be very reliable, flexible, and relatively easy to configure and use. I use an older version (1.36), packaged for sarge, which lacks some features I would like - migrating backups between volumes, ssl support. However, version 2.0 has just been released, and looks like it's got some sweet new features (my wishlist features included). It even runs on windows (in addition to *nix), if you swing that way. http://www.bacula.org/about/press/presskit200.html .en

I'd recommend Bacula. I've installed it for several clients, all have been very happy with the results.

Disclosure: I am a Bacula developer.

Bacula

2.0.0 has just been released, with pretty much full support for Windows. It doesn't have a pretty GUI, but it should be able to do what you want. It does support VSS so it can back up Exchange and SQL for you, and i'm working on an agent to do proper backups of SQL too, and hope to add Exchange support after that.

Finding a drive for a backup tape will be harder than finding an interface for a hard disk.

This is exactly the problem I faced recently when deciding whether or not to stick with tape backup.

Solution:
The next version of bacula will support volume migration, so I can migrate all my old data with minimal pain. Who cares if I can't find a drive for a 10 year old tape? I'll have migrated all the data to a newer one long before.

Use Bacula. Support for HFS+ resource forks, and many other useful features one comes to love about backup software (when one must use it). http://www.bacula.org/

Remoting software: Putty is the best CLI one I've ever seen, TightVNC is good for most of my stuff, but I prefer to use RemoteDesktop when appropriate (when I can lock the screen.. yes I know rdesktop is great, not a server tho)

Check into NoMachine's NX, it's higher quality than VNC, though admittedly not quite as simple to operate.

An OS that supports my eight monitor setup easily, stuck on windows

Whoa. You've got me there, the Xinerama configuration for that likely would be a PITA, assuming that whatever videocard you have is even supported by X. If the card(s) are, then it seems to me that the vast majority of the battle could be taken care of by (X|Xorg|XFree86, depending on distribution, but not startx) --configure, then reading the directions. You'll still have parts to do by hand that wont be easy, for instance, matching PCI addresses of the video cards to your actual cards so you can make sure to number the screens in the right order, and you still have to define the Xinerama layout, but at least all of the Cards and Screens should be configured for you. I doubt there's a run-time way to reconfigure them either, but I'm not really up on all the latest whiz-bang stuff in this field.

Scheduler... sorry Sunbird & the like aren't up to part yet

I have yet to figure out just what the heck it is that people want out of their scheduler. It keeps your schedule. You can make it public or private, so that people can see if you're busy. You can send events to other people and get back whether they accepted or not. Reminders. We're beyond the "groupware won't get you laid" hump here, but I'm not sure what's left on the other side. Maybe it's because I don't manage 50 people from my Outlook or something, since I've never used any features out of it that I couldn't seem to do to a Sunbird calendar with Thunderbird.

Backup solutions: OSS is way ahead of the commercial ones here IMHO

[start rant]IMHO, all of the backup software out there is stuck in the dark ages. I have not seen a single one that can properly support writing backups to files on a disk. Every single one of them assumes that you are backing up to tape, and hey, if it's a file, it's still a tape. Take Bacula which I have pretty much decided is the most awesome backup software yet. Its database indexes the files you store so that it knows exactly where on the tape it's at, and can read forward to that section of the tape to get your file back. My full backup comes out as a 1.5GB file, and is written to two USB drives (for redundancy and redundancy).

One day, I needed to recover a file. Just one. So I loaded up the console, flagged it for recovery, and started the recovery as a background job, assuming that bacula would look at the index in the database, seek to that position in the appropriate backup, and give me back my file. 2 hours later, the file was still not back. An hour after that, the file still wasn't there. Finally, I strace'd the storage daemon, and discovered that it was in fact still reading through the entire 1.5GB file off a 5400rpm USB1.1 harddrive, just as if it was reading forward through a tape. Ridiculous. So ridiculous, it was rediculous all over again. But I haven't found a single thing that can do better.

Oh, and what the hell is it with backup software (commercial and OSS) that shits itself entirely if its catalog gets lost? At least Bacula can back up its own catalog and provides tools for (with some manual difficulty) extracting the catalog from a backup file in the event that, God forbid, something happened to the computer you were backing up and the catalog database sitting on the drive was lost with everything else. Or that you want to extract a file from the backup on some other machine.[end rant]

Visio equivalent... dunno

Pfft. Just use openoffice, it's not like the vast majority of those charts aren't made with WordArt anyway.

I've been using Bacula for a while now. Backing up windows clients, and linux clients. The server side isn't too bad to setup (rpms are available), and configs are standard Linux config style. The nice part, is that it can backup to any media, hard drives, tapes, DVD, CD, etc. There's even a way that it will create bootable CD's that will allow you to become a client, and restore your machine from bare-iron.

I use it and it's prevented some real heartaches caused by deleted/corrupt files.

It has a number of benefits, in particular the windows client can also store permission information and open files:

http://www.bacula.org/

It can also do bare metal restores for some platforms.

Try Bacula, it supports backing up to DVD (though the functionality is pretty new and not as thoroughly tested as the rest of Bacula) and does span accross DVDs. In fact, that's what I use to back up my Debian system at home.

Here's the part from the manual that specifically applies to DVDs:
http://www.bacula.org/rel-manual/DVD_Volumes.html

How does this new version compare to Bacula (http://www.bacula.org/)?

The thing I like about Bacula is that it will allow you to spread a backup job accross multiple tapes, supports backups to disk, has its own scheduling system, and has a native windows client. From what I understand Amanda uses tar and relies upon NFS, SMB, or other network filesystem protocols to work. Bacula on the other hand has a true client/server architecture with a native client running on all of the systems it supports. It also makes use of MySQL to keep track of backup jobs. This made it very easy for me to create a web interface for it (http://raobackup.eas.asu.edu/

If Amanda has been improved to be competitive with Bacula in some of these areas then I'll definitely have to investigate it.

Lee

Bacula on Sourceforge and their own site. works very well as a networked backup system.

I'm running Bacula to back up a small network (11 PCs) of Windows 2k/XP machines (and 2 Linux ones) and must say it's the closest you can get to any commercial backup software. Unfortunately it is very guided towards tapes and many of the interesting features (volume migration, encryption, GUI, ...) are still work in progress, but it is being actively developed and constantly growing/improving (most recent addition: The ability to backup to DVDs automatically). The only real nuisance is that it can't back up open files on Windows (nobody seems to dare and touch that) so you have to circumvent that by using St Bernards Open File Manager or ntbackup/batchfile tricks.

Anyway, I'll happily recommend it if you have a tape drive or library/changer (never tested it with other backup media) and a hybrid Windows/Linux network to back up without wanting to pay for expensive backup software.

Bacula it's the same, but it has a GUI and cataloging. Still testing it.

There's Dar (as mentioned in the article) and also bacula for remote backups - go check them out if they're new to you.

Bacula has a binary to work with windows. Of course, there WILL be some system files that you'll not be able to back up while the thing is running, but the majority of windows files should be backed up flawlessly.

Don't know if it goes through the net encrypted, tough. The security I would use is the (not default, heh) hash pass"phrase", compression, and VPN.

Although the clients do not have built in support for encryption, according to the manual you can run the clients through stunnel to encrypt the traffic between the clients and the backup server. Future versions are supposed to support encryption built into the client.

I use it with Perl all the time, works beautifully. Very fast, too. If you grow a database greater than 500MB, use the VACUUM verb to shrink it up a bit. As others have mentioned, no type checking, though. Also, the bacula project can embed SQLite and use it for its catalog -- I've only used that for testing though, I use a MySQL catalog in production.

I currently use Bacula as my open source backup solution. Clients are available for Windows, Linux and Unix although I believe the server works best with Linux or Unix. It supports most hardware, including some tape robots (something that would be useful for 1TB of data!) and appears to be extremely flexible. It's done everything I've asked of it and more without complaint. Best of all the support from the author via. the project mailing list is second to none. The interface is through a console application although there's also a UI available (still a work in progress). There's also Amanda, you might want to look at that too.

Maybe they should be using some open source backup software like DIBS or Bacula.

Has anybody experiences with Bacula? An NT client is available, the server side is on **ix.

Just found it today, so I can't give any comments, but at least their claim is cool:

"It comes by night and sucks the vital essence from your computers." :-)

Especially interesting would ba a comparison to Amanda.

Bye egghat.

The problem with these distributed files systems seems to be that they're either pretty old and lacking features like disconnected operation (AFS) or seem to be unstable or, even worse, unmaintained (Intermezzo, Coda).
For many simple purposes backups can be done quite nicely using rsync or something like bacular. For laptop/notebook support unison is definitely worth a look. It syncs directories like rsync does, but in both directions. Works nicely for me.

A similan product Bacula performs a similar function.