Slashdot Mirror

← Back to Stories (view on slashdot.org)

BorgBackup 1.0.0 Released (github.com)

Posted by timothy on from the two-is-one-and-one-is-none dept.

An anonymous reader writes: After almost a year of development, bug fixing and cleanup, BorgBackup 1.0.0 has been released. BorgBackup is a fork of the Attic-Backup project — a deduplicating, compressing, encrypting and authenticating backup program for Linux, FreeBSD, Mac OS X and other unixoid operating systems (Windows may also work using CygWin, but that is rather experimental/unsupported). It works on 32bit as well as on 64bit platforms, x86/x64 and ARM CPUs (maybe as well on others, but these are the tested ones). For Linux, FreeBSD and Mac OS X, there are single-file binaries which can be just copied onto a system and contain everything needed (Python, libraries, BorgBackup itself). Of course, it can be also installed from source. BorgBackup is FOSS (BSD License) and implemented in Python 3 (91%), speed critical parts are in C or Cython (9%).

64 comments

  1. Don't rely on this by Anonymous Coward · · Score: 0

    Roll Your Own. It's about your data after all.

    1. Re:Don't rely on this by __aaclcg7560 · · Score: 1

      That's why I ditched the cloud and moved my data by to a local file server. If the Internet is down, I can still access my data and keep on working.

    2. Re:Don't rely on this by ItsJustAPseudonym · · Score: 1

      Unless I've misunderstood the features, it seems to me that Borg will work on local file servers.

  2. Python and C? by Anonymous Coward · · Score: 1

    Couldn't all this be done with some shell scripts?

    1. Re:Python and C? by Anonymous Coward · · Score: 0

      Sure, it could be done even in brainfuck. Unless you wanna retain your sanity.

    2. Re:Python and C? by Anonymous Coward · · Score: 0, Flamebait

      Yeah it most likely could, but you know what, if they had done it in shell then someone would have said something like "couldn't all this be done with python and some c?"

    3. Re:Python and C? by Anonymous Coward · · Score: 1

      Couldn't all this be done with some shell scripts?

      Block-level deduplication, incremental backups and FUSE mounting of the repository? Good luck with that.

    4. Re:Python and C? by Anonymous Coward · · Score: 0

      You could. Python, C and Shell are all Turing equivalent. Would you want to? Probably not. Keep in mind that there are hardlinks and softlinks to deal with, and that invoking shasum over and over could slow things down quite a bit.

    5. Re:Python and C? by jtgd · · Score: 1

      I wrote my backup program in one language but made sure my restore program was in bash, cuz if my house burns down 5 years from now and I have to restore from the cloud, I don't want to have to install programs just to get my stuff back.

      --
      J
    6. Re:Python and C? by Anonymous Coward · · Score: 0

      That's kind of like backshift - you restore by (automatically) assembling a tar archive to stdout, which you can pipe around over ssh as needed. So you don't need an application at all to restore on the system you're restoring to - just a tar program which you'd have anyway.

      http://stromberg.dnsalias.org/~strombrg/backshift/

  3. Re:Slashdot .. Freshmeat at its finest by Anonymous Coward · · Score: 0

    It's also filed under apple?

  4. Re:Slashdot .. Freshmeat at its finest by Anonymous Coward · · Score: 0

    I read this in the firehose yesterday. Regardless... I think it's useless. I just copy my entire home directory over to an external hard drive once in a while. Important stuff gets copied to a few flash drives. Backup software is useless to me. I can reinstall the entire OS and copy over the home directory better and quicker than any backup solution.

  5. Re:Slashdot .. Freshmeat at its finest by OzPeter · · Score: 1

    I read this in the firehose yesterday.

    Damn .. in my anger I missed seeing it in the firehose today. But there are still a lot of better choices of stories to post.

    --
    I am Slashdot. Are you Slashdot as well?
  6. My data... by Nunya666 · · Score: 1

    ...will be assimilated!

    Not for me, thank you. Plain old rsync works great.

    1. Re:My data... by bugs2squash · · Score: 1

      Either that or it will play tennis really well.

      --
      Nullius in verba
    2. Re: My data... by Anonymous Coward · · Score: 0

      Most Slashdotters weren't bjorn then.

    3. Re: My data... by Zontar+The+Mindless · · Score: 1

      Surely you can buy Björn Borg socks and underwear in countries other than Sweden...?

      --
      Il n'y a pas de Planet B.
  7. It definitely has its place... by mlts · · Score: 4, Insightful

    There is a definite place for Borgbackup, attic, bup, obnam, zbackup and other deduplicating backup utilities. The ability to just toss data whenever you feel like it, and only deltas get saved (after being compressed) is a nice thing. Same with having decent encryption.

    I personally have been using zbackup for a while, which is quite usable for backups, especially via SSH, where it can SSH into my NAS, fetch data, and only store what is changed to some media I rotate out for safekeeping. Zbackup has not had much Git activity, but Borgbackup has had an extreme amount of work done with it, so it is definitely a utility to watch and consider using.

    1. Re:It definitely has its place... by Anonymous Coward · · Score: 0

      I went from being able to store a year of daily server backups to several years with double redundancy after adding in deduplication. It's really a god send for that kind of data.

    2. Re:It definitely has its place... by sr180 · · Score: 1

      Ive been using BackupPC with compression and deduplication for well over 10 years now. Current pools stats show 35TB of backups compress down to 4TB in my pool.

      --
      In Soviet Russia the insensitive clod is YOU!
  8. Traditional backup could become irrelevant by macraig · · Score: 1

    Since data capacity has outpaced data rate by many orders of magnitude, anyone trying to maintain terabytes of data can find himself in an awkward situation where the time to create a backup exceeds a desired backup interval. Real-time mirroring or other fault tolerance scheme might become the only reasonable solution to data assurance. If very large numbers of files are involved and an ongoing change log isn't maintained by the file system, then even incremental or differential backups become a time-consuming headache as the backup app needlessly looks at every single file to assess changes.

    1. Re:Traditional backup could become irrelevant by Anonymous Coward · · Score: 1

      Real-time mirroring or other fault tolerance scheme might become the only reasonable solution to data assurance.

      They're not reasonable solutions. An accidental "rm -rf /" will see its effects rippled into the "backup" in real time.

    2. Re:Traditional backup could become irrelevant by Wing_Zero · · Score: 1

      mirroring only protects against hardware failure. if you have a software error, you get hacked, the OS decided to implode that day....etc, the damage will be on both drives at the same time. My boss found that one out.

      He was in the "drive mirror is the best backup" camp, I'm firmly in the "sync and unplug a USB HD". his version of windows deep-6'ed itself, his drives were past recovery with common tools, and had to start over. lost all his business e-mail, and a lot of important documents, thankfully, he was able to restore most of it by asking others about their backups. now, he still has the mirror, but has a USB HD in his filling cabinet that he syncs once a month.

      i find it frustrating that the only cost effective way to backup a current hard disk is via another hard disk.

    3. Re:Traditional backup could become irrelevant by macraig · · Score: 1

      Then put in safeguards against careless foolhardy use. What part of the chasm between capacity and rate didn't you grasp? Don't count on solid state storage to be the messiah, either: we now have a 15TB capacity SSD that again provides a disproportionately smaller increment in rate. Rate will never catch up to capacity.

    4. Re:Traditional backup could become irrelevant by macraig · · Score: 1

      i find it frustrating that the only cost effective way to backup a current hard disk is via another hard disk.

      That at least hints at the problem. RATE MATTERS. If one has a 20TB media server at home, how long does it take to simply make a non-incremental disk-to-disk backup? Don't count on solid state storage to be the messiah, either: we now have a 15TB capacity SSD that again provides a disproportionately smaller increment in rate. Rate will never catch up to capacity. That is the problem. If the chasm gets much larger, backups will play a much smaller role.

    5. Re:Traditional backup could become irrelevant by Anonymous Coward · · Score: 0

      I think filesystems like ZFS and BTRFS will have a large role to play in this arena. Being able to sync two subvolumes/snapshots on a fs level, changes only is much, much faster than going manually file-by-file if the size/atime is different.

    6. Re:Traditional backup could become irrelevant by FaxeTheCat · · Score: 1

      For this scenario, there are continuous backup systems. Any change made to the file system is also written to the backup system. The backup system can then build virtual full backups offline.

    7. Re:Traditional backup could become irrelevant by epine · · Score: 1

      mirroring only protects against hardware failure

      Wrong. A COW mirror with automatic snapshots protects against many other scenarios, and most (but not all) hardware failures. A COW mirror with frequent scheduled incremental snapshot replication to a remote location protects against just about everything, with no USB drives involved.

      Unfortunately, COW mirrors won't win any write performance benchmarks against XFS, as the internal write path tends to be far more complex. But seriously, use the Volvo for 90% of daily errands, and haul out the turbo eggshell only for special occasions.

      Insurance tests show cars have become far more fragile — 2007

      A 6 mph frontal full collision racked up $5,486 in damage to a Mercedes C class sedan. The Infiniti G35, Acura TL and Lexus IS weren't far behind.

      The winner was a 1981 Ford Escort, a sturdy little car that is no longer made. When the institute tested an old model it still had around, the Escort came out of two of the crash tests with zero damage and a fraction of the damage that the best performing new cars had.

      In my books, UFS is the 1981 Ford Escort, ZFS is the mythical Volvo, and Btrfs is the Tesla Model S—totally awesome, but I wouldn't touch it myself.

      Tesla Report: Two-Thirds of all Tesla Model S Drivetrains Replaced by 60K Miles

      Breitbart News reported in October, with Tesla's stock plunging by 10 percent, that Consumer Reports had pulled its "best car ever" rating from Tesla's Model S.

      (BTW, I fixed the -10 percent stock "plunge". Happiness is a left-leaning plungeometer. Somehow I don't think that's what they meant.)

    8. Re: Traditional backup could become irrelevant by guruevi · · Score: 1

      The only solution for that is actual block level backups. This requires file system support to be able to snapshot and/or indicate blocks that have changed. It still takes a week or so to take the initial backup but after that, you could take a snapshot every 5m and replicate just the changes across multiple volumes. If your ingress of data exceeds your capacity of egress to a backup however, then you need to rethink your architecture.

      I have a system that can take hours just to transverse and read the metadata of each file (doing a find or ls) with multiple volumes. Reading the entire dataset to dev null takes 72 hours. Snapshots every 15m complete in under 1m although the initial block level backup takes ~2 weeks over gigabit.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    9. Re:Traditional backup could become irrelevant by Anonymous Coward · · Score: 0

      For raw speed, it's hard to beat a rsync wrapper like http://stromberg.dnsalias.org/~strombrg/Backup.remote.html#rsync

      It doesn't deduplicate or compress that well, but it's very fast.

    10. Re:Traditional backup could become irrelevant by driblio · · Score: 1

      Old cars may be sturdier- but that is not safer. Crumple zones are a thing.

    11. Re: Traditional backup could become irrelevant by jabuzz · · Score: 1

      I suggest that you get a better filesystem them. IBM's SpectrumScale (aka GPFS) while I admit costs money would take minutes to transverse the metadata of even hundreds of millions of files.

    12. Re: Traditional backup could become irrelevant by guruevi · · Score: 1

      Transversing metadata is not my primary concern. It only happens once every few months when someone forgot where they put their files and it's usually deductible in other ways. There are better ways to spend my money than expensive software though, I think a license for this alone would cost as much as the storage array itself, if I wanted to spend that much money, I'd just invest in all-flash storage.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    13. Re:Traditional backup could become irrelevant by tlhIngan · · Score: 1

      Real-time mirroring or other fault tolerance scheme might become the only reasonable solution to data assurance.

        They're not reasonable solutions. An accidental "rm -rf /" will see its effects rippled into the "backup" in real time.

      And what's wrong with that? Because if you wanted to undo that, you just use the backup of the filesystem as it was before you executed the fateful command.

      Are we stuck in the thinking that we can only have one complete copy of something? Must suck for daily backups where we lose yesterday's backup after today's was made.

      Oh wait, no, we can have multiple backups - and even with a continual backup system, it means we have multiple time-separated copies. Heck, Time Machine does this! It backs up every 15 minutes by default. Delete a file, and you simply go back to the last backup before the deletion and recover it. (Time machine aware apps can even use it to see how a document evolves over time, even use it to get back passages that were removed).

      Indeed, a continuous backup starts blurring the lines between backup and versioning.

  9. Re:Slashdot .. Freshmeat at its finest by Anonymous Coward · · Score: 0

    Have fun with that. 25+ years of sysadmin on *ix tells me you will spend weeks/months after you have re-installed that O/S applying all of the tweaks that you forgot you had applied over the years.

    If you cannot bare-metal restore your systems, you are doomed to playing the game described above.

  10. On my 5th backup system by Okian+Warrior · · Score: 1

    A few weeks ago I cut the cord and migrated away from Windows to Linux (mint). Was using SyncBack to backup my files, now I need to find a new solution.

    I'm on my 5th package, because the first 4 were screwy in various ways. The default backup tool doesn't save profiles, so you have to type in the source and destination every goddamn time. (But when you do, it *does* work.)

    "BackInTime" apparently allows multiple profiles, so I created a profile and hit "close" and got the error "default profile source directory invalid". Yep, multiple profiles allowed, but will ONLY run the default profile. Google reports that this is a known issue with the program. "apt-get purge" to the rescue.

    It can't be *that hard* to copy files from one place to another. I like to have multiple profiles that I can just click and let run overnight - sometimes it's copying to my backup server, sometimes it's copying to a thumb drive, and sometimes it's a different subset of files.

    I live in hope that one of the packages (there's like, two dozen) will do what I want: let me set up a 1-click solution that will copy files to a remote location.

    Four down, about 20 to go. I live in hope.

    1. Re:On my 5th backup system by Anonymous Coward · · Score: 1

      Take a close look at 'rsync' and then write a script that uses it to do what you want.

      It even allows you to do versioned backups once you understand how to use the '--link-dest=' option properly.

    2. Re:On my 5th backup system by Anonymous Coward · · Score: 0

      rsync? On windows: cwrsync.

    3. Re:On my 5th backup system by Anonymous Coward · · Score: 0

      there is also tarsnap

    4. Re:On my 5th backup system by Anonymous Coward · · Score: 0

      I remember Unison has often been recommended, including here on slashdot. It uses rsync behind the scenes, has an optional GUI, runs on major OSes not just *nix.

      Also, it's a robust program that only receives boring maintenance updates.
      So, given that you simply want a fancy file copy it should be easy enough to get going.

    5. Re:On my 5th backup system by Anonymous Coward · · Score: 0

      Try FreeFileSync at http://www.freefilesync.org . It has Linux versions, but I've used it only on Windows so far, and not as a regular backup tool yet. Its interface also takes a little getting used to at first.

    6. Re: On my 5th backup system by twistedcubic · · Score: 1

      Unison itself is superior to any of the software mentioned here, but only in the software engineering sense. I've been using Unison everyday for about a decade, but it is not a replacement for a de-duplicating backup system like Obnam. You should use both.

    7. Re:On my 5th backup system by Okian+Warrior · · Score: 1

      Try FreeFileSync at http://www.freefilesync.org/ . It has Linux versions, but I've used it only on Windows so far, and not as a regular backup tool yet. Its interface also takes a little getting used to at first.

      As it happens, that's my next choice. Already installed, will try it this evening.

    8. Re:On my 5th backup system by Anonymous Coward · · Score: 0

      I like http://stromberg.dnsalias.org/~strombrg/Backup.remote.html#rsync for raw speed, and http://stromberg.dnsalias.org/~strombrg/backshift/ for low storage requirements.

      Disclaimer: I wrote them both.

  11. Re:Slashdot .. Freshmeat at its finest by dshk · · Score: 1

    In addition "once in a while" does not seem to be very reassuring :) I have already paid a large sum for hard disk recovery service.

    Since then I am a follower of the Tao of Backup/

  12. Backup software comparison by Anonymous Coward · · Score: 0

    Actually, there are many such utilities available. Here's a table comparing several of them: http://stromberg.dnsalias.org/~strombrg/backshift/documentation/comparison/index.html

  13. duplicity comparison anyone? by short · · Score: 2

    How does it compare to duplicity?

    1. Re: duplicity comparison anyone? by Anonymous Coward · · Score: 0

      I like duplicity a lot: it uses compression and GPG encryption, is efficient and can back up to all sorts of services but there's no dedupe. A simple command line can be scripted and scheduled but I wish there was a decent GUI for it. Deja-Dup is *very* basic but even so, it crashed the last time I tried to restore files to a different directory.

    2. Re:duplicity comparison anyone? by Anonymous Coward · · Score: 0

      Duplicity backs up the original + diff + diff + diff.
      - If you want to reclaim space from old backups, you have no choice but to either (A) retransmit the full original files - sucks to do over the internet - or (B) have the remote server to merge the diffs, which it can't do if it's dumb FTP/S3 storage, nor if the files and diffs were encrypted.
      - If you want to restore the most recent file, it takes a long time to merge a long long diff chain.

      Borg/attic/restic use a different algorithm entirely. They use content-defined chunking to split files up at data-defined boundaries, and only preserve unique chunks. Every snapshot can be restored from in the same amount of time. Compression and encryption work fine to back up to an untrusted storage. There is no downside.

    3. Re: duplicity comparison anyone? by Anonymous Coward · · Score: 0

      Although not a GUI, http://duply.net helps.

  14. Re:Slashdot .. Freshmeat at its finest by MightyMartian · · Score: 1

    Perhaps for a home user that's fine. But where you have to have long term archives and your managing multiple versions of files over time, cp -R doesn't quite cut the mustard.

    --
    The world's burning. Moped Jesus spotted on I50. Details at 11.
  15. Client/server and tape autoloaders? by rklrkl · · Score: 2

    This might be fine where you've got a single Linux machine and, say, backup to an external USB3 hard drive, but what about bigger setups than this? For example, multiple Windows/Linux client machines to backup and a central server with an autoloader/barcoded Ultrium tape drive attached? There's very few open source solutions that deal with this in a heterogeneous environment (Amanda - which is poor with Windows clients - and Bacula - which is ridiculously complex to setup - are just about the only two that spring to mind). Until BorgBackup can do something similar, it's not really useful in a multi-machine/autoloader setup (no, I don't want to install two backup systems on every client...).

    1. Re: Client/server and tape autoloaders? by Anonymous Coward · · Score: 0

      Don't know of any other software of that kind but could you use the backup client of your choice to back up to disc staging and backup from there to tape? You'd just need to erase the backups from the staging once they were on tape. Restore might be tricky although I'm sure there are products which use disc staging (maybe not FOSS though).

    2. Re:Client/server and tape autoloaders? by Anonymous Coward · · Score: 0

      Between backup and restore, so much gets nerfed these days the only strategy I consider sane is trying to make the system as stateless as possible. So I only care about getting user data backed up, and I care less about structure than data, so if I don't preserve links I don't really care. I think users using symlinks is annoying anyway, including when I've done it. Even on OS's where this should always work, like OS X aliases with Time Machine backups, they invariably break at some point for reasons unknown. So I just say, f it, and I don't preserve links, ownership, permissions. Just data and timestamps. The server runs either ZFS or Btrfs for rolling backups. The restore is still a gotcha because it's not automatic, I have to pick a snapshot to restore, and whether pushed or pulled to a newly restored machine, it's data that probably doesn't have ownership and permissions set correctly so that has to be done manually but that's a pretty quick task.

  16. Rsync is not for backup by Dadoo · · Score: 1

    Plain old rsync works great

    Sorry, rsync has a number of issues that make it unacceptable (IMO) for backup. Among other reasons, it doesn't preserve metadata, and the rsync people think that's the correct behavior,

    Don't get me wrong: I use rsync all the time, but never for backup.

    --
    Sit, Ubuntu, sit. Good dog.
    1. Re:Rsync is not for backup by driblio · · Score: 1

      Owner, group, mode, acls, xattrs, times... What metadata doesn't it backup? And what are the other issues? Genuinely interested as I do use it for hardlink incremental backups all the time.

  17. Because "Borg" by davidwr · · Score: 2

    Why the fuck do I care about BorgBackup?

    Because:
    * Saying "Borg" sounds nerdy.
    * Saying "Borg" sounds funny.
    " Because thinking "data" and "assimilated" together sounds funny.
    * Because thinking "assimilated" gives immature nerds the giggles
    * Because what's one more backup system in your collection, er, I mean, assimilation
    * Because ... Star Trek
    * Other [fill in the blank]
    * Because CowboyNeal wants his soul backed up and assimilated

    --
    Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
    1. Re:Because "Borg" by beastofburdon · · Score: 1

      [begins slow clap]

  18. Borg backup kicks butt by linuxguy · · Score: 1

    I have been using Borg backup for a few months. I absolutely love it. Before borg, I had a nightmare backup scheme. I have a lot of data. And I cannot backup all of it every week. It would require too much storage. I got a little taste of deduplicated backup with the backup tool Microsoft includes in Windows Server 2012. I was immediately hooked. But it has severe limitations. I wanted a very flexible backup program that did deduplication well. In my opinion, there is nothing else that even comes close. Borg is exceptionally well thought out and built.

  19. Re:Slashdot .. Freshmeat at its finest by KGIII · · Score: 1

    A long time ago, I lost data. I lost a lot of data. At the time, I lost data that was more than ten years old. That was back around 2000. I do not lose data now. I have not lost meaningful data since. I will never lose data again.

    I do not even store much data locally. Even though I am not home, it is pushed out to my home network. From there, it is mirrored and pushed out to disparate physical locations. I have hardware at friend's houses. I have hardware at other property that I own. I have hardware in my garage/workshop.

    Storage is cheap and connectivity is near ubiquitous (for me). I can not justify not taking the time to create a good backup system and automate it. The only thing that is not automated, entirely, is verification. I do that manually and on a regular basis.

    Even as I'm typing this, I'm using a computer but I'm actually connected to a computer at my home via VNC. Data isn't even stored on that computer but is pushed out over my own network and stored/retrieved from there. Once it is configured, it's good to go. I test it and make sure that backups are not just verified but are actually working as expected when restored. I'd like to automate that part away but I've not really figured out a good way to do so.

    I'm pretty anal about backups. There's not a lot that I'm anal about but backups are one of those things. I never, well almost never, save a single copy of anything. Drives are cheap. I'll buy more. One catastrophic data-loss was unacceptable and, unfortunately, that data loss included my backups - as I wasn't even storing them off-site. I will not lose data again. Yes, I backup things I don't need. That's okay. I'll buy more disk space. I even send out data to be stored with professionals but I only bother with that twice a year.

    --
    "So long and thanks for all the fish."
  20. You may want to take a look at Pukcab by Anonymous Coward · · Score: 0

    It's very similar but doesn't require anything else than a bare unixish OS (i.e. no Python, no non-standard libraries, etc.). It's a single binary, written in Go and probably is much faster, it also has a nice web interface. Homepage: http://pukcab.ezix.org

  21. Re:Slashdot .. Freshmeat at its finest by Anonymous Coward · · Score: 0

    Also, most pro-grade backup systems allow backing up data from multiple machines and on an automated schedule.

    At home maybe you can get away with simple file copies, but once you start dealing with being able to do fast recovery over a set of a half-dozen machines or more, simple manual processes could be a full-time job itself.

  22. Comment by WallyL · · Score: 1

    BorgBackup 1.0.0, so is that Locutus? When will version 7.9 be released?