Slashdot Mirror

← Back to Stories (view on slashdot.org)

Ask Slashdot: Practical Bitrot Detection For Backups?

Posted by timothy on from the error-detected-goodbye dept.

An anonymous reader writes "There is a lot of advice about backing up data, but it seems to boil down to distributing it to several places (other local or network drives, off-site drives, in the cloud, etc.). We have hundreds of thousands of family pictures and videos we're trying to save using this advice. But in some sparse searching of our archives, we're seeing bitrot destroying our memories. With the quantity of data (~2 TB at present), it's not really practical for us to examine every one of these periodically so we can manually restore them from a different copy. We'd love it if the filesystem could detect this and try correcting first, and if it couldn't correct the problem, it could trigger the restoration. But that only seems to be an option for RAID type systems, where the drives are colocated. Is there a combination of tools that can automatically detect these failures and restore the data from other remote copies without us having to manually examine each image/video and restore them by hand? (It might also be reasonable to ask for the ability to detect a backup drive with enough errors that it needs replacing altogether.)"

6 of 321 comments (clear)

  1. PAR2 by Anonymous Coward · · Score: 5, Informative

    http://www.quickpar.org.uk/
    http://chuchusoft.com/par2_tbb/

  2. ZFS filesystem by Anonymous Coward · · Score: 5, Informative

    One single cmd will do that,

    zpool scrub

    1. Re:ZFS filesystem by vecctor · · Score: 5, Informative

      Agreed, ZFS does exactly this, though without the remote file retrieval portion.

      To elaborate:

      http://en.wikipedia.org/wiki/ZFS#ZFS_data_integrity

      End-to-end file system checksumming is built in, but by itself this will only tell you the files are corrupt. To get the automatic correction, you also need to use one of the RAID-Z modes (multiple drives in a software raid). OP said they wanted to avoid that, but for this kind of data I think it should be done. Having both RAID and an offsite copy is the best course.

      You could combine it with some scripts inside a storage appliance (or old PC) using something like Nas4Free (http://www.nas4free.org/), but I'm not sure what it has "out of the box" for doing something like the remote file retrieval. What it would give is the drive health checks that OP was talking about; this can be done with both S.M.A.R.T. info and emailing error reports every time the system does a scrub of the data (which can be scheduled).

      Building something like this may cost a bit more than for just an external drive, but for this kind of irreplaceable data it is worth it. A small atom server board with 3-4 drives attached would be plenty, would take minimal power, and would allow access to the data from anywhere (for automated offsite backup pushes, viewing files from other devices in the house, etc).

      I run a nas4free box at home with RAID-Z3 and have been very happy with the capabilities. In this configuration you can lose 3 drives completely and not lose any data.

      --
      Why, yes I have been touched by His noodly appendage. And I plan to sue.
    2. Re:ZFS filesystem by Guspaz · · Score: 5, Informative

      You don't need raidz or multiple drives to get protection against corrupt blocks with ZFS. It supports ditto blocks, which basically just means mirrored copies of blocks. It tries to keep ditto blocks as far apart from eachother on the disk as possible.

      By default, ZFS only uses ditto blocks for important filesystem metadata (the more important the data, the more copies). But you can tell it that you want to use ditto blocks on user data too. All you do is set the "copies" property:

      # zfs set copies=2 tank

  3. Re:Excellent question by SirMasterboy · · Score: 5, Informative

    Not all cloud storage is expensive. It's only $4 a month for unlimited backups to CrashPlan.

    They also do checksums and versioning and can be set to never remove deleted files from the backup.

    I have 12.8TB backed up to them and it's been working great.

    Other than that, ZFS can't be beat. I use that as well.

  4. Re:Checksums? by Waffle+Iron · · Score: 5, Informative

    I never archive any significant amount of data without first running this script at the top:

    find -type f -not -name md5sum.txt -print0|xargs -0 md5sum >> md5sum.txt

    It's always good to run md5sum --check right after copying or burning the data. In the past, at least a couple of percent of all the DVDs that I've burned had some kind of immediate data error

    (A while back, I rescanned a couple of hundred old DVDs that I burned ranging up to 10 years old, and I didn't find a single additional data error. I think that a lot of cases where people report that DVDs deteriorate over time, they never had good data on them in the first place and only discover it later.)