Slashdot Mirror

I do that as well, but I found out to my horror that all my DVD's had become unreadable over time. So, probably good idea to test your backups from time to time

That's why you should reserve 10-15% of the disk for parity data. While DVD-R format has built in error-recovery at the sector level, by the time you figure out that the disk is going bad it is too late. By adding even more recovery data at the file level, you can treat the disk errors as an early-warning system, then use the recovery data files to get your data off the disc.

The old standard was QuickPAR (PAR3), the new version is called MultiPAR.

With PAR3, you can even write the disc image to an ISO file, make a copy of that ISO file and rename with the PAR3 extension, then let QuickPAR or MultiPAR use the PAR3 filename to search the ISO filename. Which lets you retrieve data from a disc, even if the file system has become corrupted.

(About the only thing you can't easily recover from is a "track zero" error where the TOC of the disc has been broken.)

http://www.quickpar.org.uk/
http://chuchusoft.com/par2_tbb/

The problem with FLAC is that a minor loss of data can result in an archive that cannot be extracted. A single bit error can result in major lossage. PCM does not have that problem - you only lose what you lose.

I'd say you're doing it wrong, then. I'd prefer to have zero loss of data. Format used (FLAC vs WAV) does not really affect this.

For something you can recover from a 3rd party but would rather not (personal rips), make some parity archives – a few MB for a ~400MB album will let you recover from several errors and still be much smaller than a WAV file. Studio masters should be backed up in multiple highly-reliable places – no excuses there.

That said, I've always wondered why FLAC does not have FEC capability built in. It makes sense for a lossless format to have some support for more reliable archiving.

For archival purposes, I've started using WinRAR ( http://www.rarlabs.com/ ) with the file authenticity and recovery options checked. Unfortunately none of this helps you now, but it will help in the future at least...

The other thing to do if you want longish term reliability is to add redundancy to whatever you're storing with a tool like par2, http://www.par2.net/ and http://www.quickpar.org.uk/ are your friend.

Raid5 will help you if you lose a whole drive (e.g. siezes up from sitting still for a long time), the par2 data will both allow you to verify that the data hasn't been corrupted, and if it is (e.g. a couple sectors go bad), it will let you recover the data.

If you have access to the file at the remote end I would run it though QuickPar http://www.quickpar.org.uk/ at that point you can split the file into a group of smaller files (less chance of any one file being mangled), plus add ckecksum files to the set, that way if one of the files is mangled in transport it's 1: identifiable and 2. automatically fixed by reassembling the par files on your end of the connection.

When archiving data to CD or DVD I use a program call Quick Par to generate recovery files that I can use to repair the data on the disc if it becomes damaged.
It is based on the same recovery tech that RAID systems w/ parity drives use, and is mostly used to repair Usenet downloads. I usually put 4GB of data and 400MB of PAR2 recovery files on the disc. This will allow ~10% data loss before recovery is not possible. Also I dont have to worry about the TrueCrypt vol becoming damaged and unusable as well.

I always quickpar files in CD's / DVD's, it takes extra time but re-finding and redownloading data is even more time consuming.

Quickpar
http://www.quickpar.org.uk/

Quicksvf
http://www.quicksfv.org/index.html

For my media center I use quickpar http://www.quickpar.org.uk/.
1. DVD Shrink each disk of a season of a show (IE 7 disks of ST:TNG Season 3) output file to a ISO.
2. Use quickpar to create 6000 megs of par2 files. Use small files so you have about 200 par 2 files to work with.
3. Burn ISO to DVD AS MULTISESSION. In the 2nd session fill up disks with pars so that disk one has pars 0-5, disk 2 has 6-10, ect.
4. Burn 8th disk of all remaining pars that didn't fit on any of disks 1-7. Then fill up any extra pars just to max out the DVD.
Now if you loose ANY one of the 8 disks, the missing disk can be recovered. Also EVEN WITH a missing disk, there is STILL enough par's to recover from minor damage to the other 7 disks. Also note when doing recovery you can copy all the misc files that make DVD (the VOB's ect) to your hard drive, and quickpar will repair them to a ISO.

quickpar especially has been in use on usenet/newsgroups for years....o yea...forgot....they are trying to kill it.

anyways...there's also dvdisaster which now has several ways of "hardening".
one of them seems to catch my attention: adds error correction data to a CD/DVD (via a disc image/iso)

Or choose your acceptable failure rate with par or par2 and the like:

http://www.quickpar.org.uk/
http://www.binaries4all.com/quickpar/
http://en.wikipedia.org/wiki/QuickPar

Rather than hoping that your data exists on 1 of your four 5GB dvds, you hope that one of your three 4GB dvds is mostly ok and everything is dandy. It's a choose your own level of error correction world, rather than a minimal error correction world. Popular among the porn/rom/binary set on usenet.

If you really don't have any other option than backing up to DVDs, then how about this?

Install WinRAR, right-click the directories that you want to back up, click 'Add to archive...', choose the option to split into DVD-sized chunks*, select whichever compression method you'd like, and then wait while it creates the archive files.

Now install QuickPar, perhaps read a parity file tutorial if you're unfamiliar with the concept, and then create a set of parity files to restore any corrupt or damaged data.


* Or perhaps it would be better to create files of just under 45 MB each, and burn 100 of them to each DVD.

i'm too paranoid to use stuff like this for backups.. sure 25 gigs is nice but whats the use if i just burn everything in 35 copies on the disc incase one part becomes unreadable?

as media starts to hold more, i just start creating more copies of the same backup on the disc. bluray/hdvd scares me because if it gets scratched you lose so much more than if a cd gets scratched

For those that don't know, PAR2 files are parity files that can efficiently reconstruct missing or damaged blocks in your archive. If you have more PAR2 recovery blocks than damaged blocks, then you can completely reconstruct all of the damaged files in your archive. The best newbie explanation I've seen is the "PAR & PAR2 files" section from Slyck's Guide To The Newsgroups.

If I'm backing up to a data DVD-R (capacity 4,706,074,624 bytes), I'll leave around 4GB of space for the actual data and fill the rest (to the brim) with the PAR2 files that I created for that data. I name the PAR2 files starting with the letter 'z' so that they get burned on the outer edge of the DVD. When creating the PAR2 files, I choose a block size that is a multiple of 2048 bytes because that is the block size of a DVD sector.

Some easy-to-use tools to create PAR2 files:

• Windows: QuickPar (freeware)
• OS X: MacPar deLuxe (shareware, $15)
• Linux/OS Independent (GTK): GPar2

Some DVD data recovery software (to get every readable block off a damaged disc):

• Windows: IsoBuster (recovery functions are free)
• Linux/OS X (command line): dd_rescue

Thanks, WuphonsReach.

That'd pretty spendy for a DVD. (Probably dual layer.) Anyhow, I would probably give them multiple dvds of different brands, and a couple dvds with the actual ISO's of the dvds on them. I would also include Par2 files for the ISO's so that if there is data loss, it can be reconstructed from the par2 files. This is a longer process, but it much more rendundant than a single dvd.

QuickPar - Par2 windows client
Main site for par/par2 files

Sounds like you need to introduce yourself to the world of par2 ( http://www.quickpar.org.uk/ )

Parity reconstruction

Think of it like the year 2805 where scientists can regrow someones arm if they happen to lose it

You use something like - it basically splits your data into blocks and then you can define the amount of redundancy you like.

So say you have a file and running it through par2 with 50% redundancy gives you 100 blocks. All you need at a later date is 50 (any 50) of those blocks and you can recreate the file.

Pros:

• cheap
• abundant
• easy to transport
• can buy only as much as you need
• easy to set-up and use

Cons:

• doesn't last long
• damages easily
• doesn't hold a lot of information (until Blue-Ray comes along)
• requires splitting your videos into small chunks instead of being stored contiguously

• For DV footage, you will need about 3 4GB DVDs/hour. That isn't a bad price point. Good luck!

For my own personal archives, I have taken the methods from the masters in USENET.

OS X & UNIX: I'm lazy just: tar.gz

For Win32, I back-up a lot more files under win32 than *nix.

Compression
WinRAR
Compression Method: Best
Split to Volumes: 20MB
Parity
QuickPar
With general settings.

I back-up to decent quality DVD media, as I have had a lot of problems with CD media rotting after about a year.

Does this make the case for parity archiving?

But of course! And for the windows folks, there's QuickPar which puts a nice pretty GUI shell on front of it.

There's also been rumor of a PAR3 format in the works, which would not be deterministic, but would calculate roughly an order of magnitude faster. So if you have 10% PAR3 data, you *might* be able to repair up to 10% worth of damaged blocks, but it's not certain. And that's probably a flawed explanation of how PAR3 might work when it's finished.

Bit of a trade-off really, faster calculations or guaranteed recovery for a given number of bad blocks... giving that soon I'll be calculating PAR2 data for 20-24GB at a time (for BluRay discs), I may choose speed. (Prepping 10% PAR2 files for a DVD sized chunk already takes 20-40 minutes.)

A quick perusal of the QuickPar website suggests that at least some Par2 clients can restore based on two damaged files and incomplete recovery files:

At this point you can have QuickPar load additional PAR2 files (to provide more recovery blocks) or scan additional data files (to see if they contain some of the data from the original files).

In the past, however, I've been dealing with getting remote files over a noisy connection where the remote server wasn't so thoughtful to create Par files or even set up Rsync. What I've thought would be a nice is an application that can look for correspondances between three checksum-failed files to try and create one good one. I don't suppose Parchive can do that?

good overview here: PAR2 files

comparison between v1 and 2: here