Slashdot Mirror

The other thing to do if you want longish term reliability is to add redundancy to whatever you're storing with a tool like par2, http://www.par2.net/ and http://www.quickpar.org.uk/ are your friend.

Raid5 will help you if you lose a whole drive (e.g. siezes up from sitting still for a long time), the par2 data will both allow you to verify that the data hasn't been corrupted, and if it is (e.g. a couple sectors go bad), it will let you recover the data.

First: Do NOT store them as playable DVDs. If you transcode them at all (from whatever your camcorder records), put them in h.264, with a decent audio codec -- something like aac, ac3, ogg, anything but mp3.

That alone should drop you from 100 gigs to 20 gigs, without much loss in quality.

More to the point: The way I intend to deal with this is to build a CD-changing robot. You know, something like this -- the cheapest commercial versions are more than I'm willing to pay.

This, combined with scripts to periodically check discs, create parity, and replace/re-burn defective discs, should give me a fair amount of somewhat-reliable storage. Combined with a FUSE filesystem driver, it should also give me relatively easy, sort-of random access (very slow seek time). It would also be upgradeable to Blu-Ray, if that ever gets cheaper per gig than DVD5.

Not that I advocate this to anyone who doesn't have a few dozen (hundred?) hours to kill building a robot, but I think the same basic principles apply -- use parity, check periodically for bad media (and replace it immediately), and DVD5s are about the cheapest storage you can get today.

There is one other possibility worth considering: Just use Amazon S3. 100 gigs = $10 to upload and $10/mo to store.

Or use PAR2? It's free.
Tony

Ok, this has been happening since the beginning of computers.

I still have some punch cards that I need to convert. It doesn't matter that IMSL solved all the same problems better.

I have some:
- 5.25" floppies
- RLL/MFM hard drives with data
- parallel port QIC80 tapes (250MB ea)
- 1/4" tapes (not so important)

All of these need to be converted to useful media. Or, I figure they are probably corrupt by now.

Hence, the addition of par2 http://www.par2.net/ which provides parity protection against partial media failures. Corruption can be handled by home users. For enterprise customers, EMC, Veritas and STK have handled this for years. For home users, the extra effort that par2 requires, http://en.wikipedia.org/wiki/QuickPar, may not be worth it. But if it is your wedding pictures, RAID on disk, off-site storage and optical media are **all** required to save your marriage. That 1GB of GMail space might be worth it?

If it is your corporate date - go ahead, take your chances. How much can it possibly be worth if it is missing? YOUR JOB perhaps?
The cost of time to find all the data and work interuption is nothing compared to gracefully handling a disk failure without a production impact. Where I work, we do backups **very** well and remote vaulting in alternate data centers.

Whomever marked this as offtopic was a little quick on the gun. I believe the coward is referring to PAR files, a method of breaking up data and reassembling it commonly used on newsgroups.

For photos, backups and other archival stuff to CD or DVD I've been adding PAR parity files. They're sort-of like RAID at a file level. Even though the initial use of .pars was for Usenet binary downloads (I think) the tools work great for any situation where partial loss of data is likely. In theory, as long as the unreadable bits of the disc are a small % of the disk, I should be able to rebuild the data. There's an article where this is tested by scratching / drawing on a CDR -- unfortunately, I don't have a bookmark. The chances are good that a disk will not go bad uniformly across the entire disk. So, the parity files should be able to recover from most scratches and a lot of bit rot.

Plus, I feel that .par files are better than making 2 copies of the same data. For example, if you're only backing up a few large files, the odds of having both copies of the file get a few bad sectors is relatively high. I suppose you could patch over unreadable bits with a hex editor or something, but I'll take a nice GUI and relatively common software anyday.

Although, to be perfectly honest, I just don't rely on CDR or DVDR. I keep a copy of all stuff on a hard drive too. DVD's go offsite, HD stays at home.

Re-evaluate what's on your storage. You sound like me - a bit of a hoarder.

Firstly, get a good catalogue system going. Put all your crap in one area. Get a sorted listing of creation time,last access time and categories. Get some hard-and-fast rules going as to when it should be archived offline. Go through once a month (or more often) and work out any new stuff that you need to backup. You do backup, don't you? Of course you do.

Then go through and check the last access times of your categories, and move to offline storage as appropriate.The advantage of getting this sort of regime going is that you've got more chance of having a backup offline somewhere when the inevitable happens and your drive wakes up dead one day.

For example, once you've got categories and last access times sorted:

- Digital Photos, family movies , documents - anything more than 12 months - Offline to DVD. Use par2 for archive copies (sent to distant relatives for storage), but make another set without par2 for normal, semi-random access.

- ISO's of distros? Got a broadband connection? Ditch them.
No broadband? Get them all offline, regardless of age.

- TV Episodes? If it's later than say, 3 months - off to DVD they go.

- Web Site content? 3 months.

And so on. Work the times out for yourself.

You really need a good cataloging system to help find out where the offline files are. Everyone's idea of a good catalog is different - Hell, I just label DVD's and keep them sorted by catagory - so I'll leave it to you.

If you organise your data and find that you still haven't enough drive space to keep all your current data online, then (and only then) go and look at the expensive options.

Who wants to look for part 745 of a 10000 part .rar file?

This is why you have utilities like par2.

Then you don't need part 745 specifically, most of the time you have enough redundancy in the bits you've downloaded that you don't need anything. And if you don't, par2 will tell you that you need x recovery blocks to repair it. *Any* x blocks. So if you can get those blocks off usenet, you're fine.

par2's also good for making redundant backups - if you make a set of par2 files that's got more than 50% redundancy, then all you have to do is put a random half on one CD and again on another. Even if both CD's are fairly corrupted, if you can recover at least half the total data blocks off the CD's, you're fine.

Make sure you have enough redundancy in your population to ensure DNA data integrity

No! Check those backups! I have lost data stored on CD-Rs (luckily I had copies), and many of my discs have started to turn yellow after about 2 years! Also, you can sometimes see these little spots of discolouration on the CDs, which makes me think there's a fungus of some sort that's eating them.

The lifespan of CD-Rs is unknown at this point. Don't trust them for more than a year without inspection, and make fresh copies after 5 or so years.

I'd also recommend using some kind of forward error-correction scheme, like par2.

It'd be cool if they could put in a function in the hardware that would calculate and fill out the media with [standardized] redundancy data. You'd want it do be done in hardware to be fast, compatible and not generate unneccesary bus traffic.

Basically, the burn software would feature a '[X] Fill out with redundancy data and finalize disc'-option box together with the '[X] Finalize disc' one.

I've sometimes done this by hand, but it takes forever to calculate the data, and you don't get it properly distributed over the disc, etc, etc. I think it'd be better done in hardware.

Guess there's no hope though, it'd up the cost a dollar, and we all know that's just impossible to bear. <sigh>

Something along these lines could help with media dropouts. You can build these files with as little or as much redundancy as makes you feel comfortable. Of course, if Timmy Toddler uses the medium as a frisbee or the dog eats it, you're still SOL.

Reed Solomon is the key here: use PAR2-files to protect your investment.

Ok guys, this is the quickest and easiest way (especially if on broadband) to get SP2. Get newsleecher, then go to alt.binaries.wares.ibm-pc.ms-beta, and download the WindowsXP-SP2-2180-ENU-FINAL files (there's 28 of them). Yes, one small file is missing (about 22k). The file ships with a Parchive parity file (.PAR2) to reconstruct and/or correct any data that did not come through. You can get the PAR2 verifier from here. Now what do you do? Well, you will be left with 19 .rar files, so UNRAR them!! You will be left with a filename slightly different from the original, but this is in fact the correct build. MD5 Sum and crc32 verified with fsum (59A98F181FE383907E520A391D75B5A7 & 046F12B1, respectively). Have fun! And best of all?? If you're on broadband, it's fast, especially since you're not leaving the local provider's network (assuming your provider hosts their own news servers).

By the way, I do not normally get software this way, but on Friday it was my mission to get SP2, and get the real one. This worked and was fairly simple. If you've just got to have it, you can use this method. It is the correct service pack, and it works fine! Hope this helps..

One recommendation for the future is to make use of the PAR/PAR2 utilties. (I personally use QuickPAR v0.7)

Basically, you create 'parity' files that you store on the DVD/CD along with the source files. Then, if any of the source files become corrupted, QuickPAR should be able to reconstruct the broken bits.

The amount of redundancy is up to you... 10% is normal, 20% might be worthwhile. (So to store 4Gb with 10% redundancy, you'll need 4.4Gb of space.)