Slashdot Mirror

Except that optical media isn't that durable or reliable. Every DVD or CD I've ever burned has become unreadable after a few years. The inks just don't hold the data for long.

HTL BD-R uses an inorganic phase-change alloy sputtered onto the disc surface. I have media files going back 4 years or so backed up to a bunch of Blu-rays at work, and they've mostly held up pretty well. I recently scanned all of them to see how they were holding up, and out of 300+ discs, 5 had some unreadable areas. They would've been recoverable because I augmented the images before burning with dvdisaster, but it was faster to just mark their contents as not backed up and let them get burned to a newer disc.

LTH BD-R, OTOH, uses the same organic dyes as CD-R and DVD-R, and is just as susceptible to bit rot (though in all honesty, I have plenty of DVD-Rs and CD-Rs kicking around that are still readable.)

Most BD-Rs on the market are HTL. They tend not to be marked as such, but LTH media are. Verbatim seems to be the most prominent of the LTH BD-R brands, though I think I've heard that Taiyo Yuden also produces LTH BD-R. dvdisaster identifies my backup set as a mix of Ritek, Philips, and CMC Magnetics media; they carried a variety of other brands on them (some well-known, some not so much).

If you're not set up for Blu-ray, M-Disc has applied its inorganic recording layer (they describe it as a "rock-like carbon compound") to DVD as well as Blu-ray. You need a drive that can burn them (not just any DVD burner will work), so if you're in the market for a compatible burner, you might as well get one that also handles Blu-ray. Wikipedia says the discs, once burned, are readable in any drive.

That about is my experience.

I have about 1000 burned DVDs that I use somewhat regularity, and about 100 burned BluRays now. Since about 2002 I have encountered exactly *two* discs I couldn't read any more. One got scratched badly when it somehow got loose from the sleeve in my satchel, the other fell onto a tile floor and a corner broke off.

For security I usually make two copies of "important" stuff, and I also fill the media only to about 75%, and fill the rest up with error correction data created by http://dvdisaster.net/en/index..., so that in the event that I one day actually DO run into a bit rot issue, I might still be able to re-construct the data.

Data migration and expanding RAID containers is a major PITA. I absolutely loath the task!

That's why you don't use RAID. Instead, use something more flexible. I've been running Greyhole for a while now. Adding storage doesn't require shifting files around (unless you want to rebalance storage), you can use drives of different sizes, and you can control the level of redundancy you use (more for important files, less for stuff that's easily replaced). You can yank a disk out of a Greyhole installation and read all of the files off of it with standard file-copy tools.

Important stuff that doesn't take too much space (documents, Git repos, etc.) is backed up daily to Tarsnap. Less-important stuff (movies, music) and larger files (photos) get dumped to BD-R and are stored in binders in my desk at work; images are prepared with dvdisaster for added error recovery capability and are burned to single-layer BD-R HTL media.

Using something like dvdisaster, you can burn a CD/DVD/BD with extra recovery information (either on the disk, or as a separate file)

A database is something special

I basically make a "full backup" of my Oracle DBs once a week, and a "incremental backup" in the form of DB change logs every five minutes. (that is, the change logs are pushed "off site" every five minutes, of course they are being written locally continuously with every change.

The thing with backups, though, is not only to make them often but to also *check* them often. With my DBs there is a handy tool where I can check the backup files for "flipped bits" because there are also checksums in the DB files.

For my "private backups to DVD/BR" I only fill them up to ~70%, and fill the rest of the disk with checksum data with dvdisaster., for other "online backups" I create PAR2 files that I also store. With those parity files I can check "are all bits still OK?" now and then, and repair the damage when/if bits start to rot in the backup. In the 10 years I do this, with ~150 DVDs and ~20BRs so far I had 2 DVDs that became "glitchy", but because of the checksum data I was able to repair the ISO and re-burn them.

Basically, IF you go through the trouble of setting up an automated backup system either with software or with your own scripts, It doesn't add much work to also add verification/checksum data to the backup. And that goes a long way into preventing data loss due to bit rot.

RAID-5 uses up 1 disk worth for striping, so net space in an 8-drive array is 7-drives worth (about 27TB using 4TB drives). The problem with RAID-5 is that you are 2 disks away from failure and rebuilds often kill the disks.

RAID-6 uses 2 disks worth for striping, so net space in an 8-drive array is 6-drives worth (about 23TB using 4TB drives). Is able to survive a double-disk failure before data loss. Still has some of the same issues as RAID-5.

I use Greyhole for media and document storage. It handles disks of unequal size (currently running one 3TB and two 1.5TB drives), and you can choose the level of redundancy you need. In my case, movies, TV shows, etc. get a single copy (one file exists on one drive), while documents and photos get two copies (one file exists on two drives). If a drive goes bad, you only lose the files on that drive...and only for the files for which you selected no redundancy. With redundancy, extra file copies are recreated on the remaining drives from the surviving copies; this process is most likely less stressful on the disk set than a RAID rebuild.

My movies, TV shows, and music are backed up to BD-R, stored in a binder at work. They hold ~20GB each, as I'm using dvdisaster to guard against media errors. When a 2TB drive failed, I brought the backup (currently about 190 discs) home and restored the files that had gone missing. Backup and restore are managed by scripts, with information about what files are on what discs held in a MySQL database that gets periodically backed up off-site as well. The initial backup took several months (on and off) to finish, and the last time I needed to restore, it took about a week, but now I just burn a disc when I have about enough new data to fill one. Burning and verifying takes a few hours, but it's something you can start and walk away.

^This.
I go a little overboard* and also build ECC data with DVDisaster into the ISO images before I burn them to disk, but the fact that non-LTH** BD-R media uses non-dye phase change writing tech is a game changer for the utility of optical archiving.

Even better, Millenniata recently released a BD-R media called M-DSIC that has a claimed life expectancy of 1,000 years. We'll see in 1,000 years if that claim holds up. ;-)

*as some would see it.

** LTH dye-based BD media. I want to throttle the person who came up with that idea. "Hey, let's make the media really crappy and we can pass on the 10% savings! Data integrity? Who cares about that!"

Redundancy has to be made intelligently, not just a 1 to 1 copy, which nearly useless. Think dvdisaster, for example. Makes for a pretty good optical archive, if you try to keep a 30% to 40% ECC ratio on the disks, and 2 copies of each ECC-enhanced disk. Of course, it's not perfect either, but what is?

If part of your archiving strategy is to burn data to disc, make sure that you pad those discs with dvdisaster error-correcting data. Optical discs generally fail in a way where only part of the data is unreadable. Without extra error-correcting data, then those parts are gone forever. However, with dvdisaster, you'll:
1) Know when a disc is failing before it's too late
2) Be able to recover the data
3) Be able to migrate the data to fresh media

http://dvdisaster.net/

I should have clarified. Yes, I updated the firmware when the issue became known and never experienced problems before or after, until now. I've run across threads mentioning that the first firmware released to correct the issue...didn't. Perhaps I nabbed that one without realizing it.

1.That's what backups are for.
2. I think that Seagate will fix it for you so that the data is recovered.

1. The video on the drive hadn't been worked with to be backed up yet; this is an additional 1Tb of raw footage. I've got 2Tb of completed video with a 2Tb backup already. And a crapload of DVDs padded by DVDisaster. I'm pretty serious about backups.
2. IF my issue was with the publicized bug, I've heard they offered free data recovery. If self-recovery doesn't work, I'll end up putting in a warranty claim and hoping its covered. Otherwise, it could be $300-500+, which I'd probably not pay. It'd just be a reaaaaaal pain to redo the recordings.

The bug is that if the pointer to the current log entry is evenly divisible by 64 (or some magic number like that), the powerup selftest code crashes and the drive never completes selftest.

Yeah, I remember reading a thread on ./ at the time from one of the Seagate engineers. I think it was something like, if the drive wrote a 360th log entry, but then powered off before writing the 361st, it got stuck. I created worse issues in assembly programming in college a decade ago, but thankfully my mistakes didn't make it to production. D'oh.

I've got a spliceable USB cable arriving tomorrow, so we'll see how it goes. Fingers crossed.

If you are archiving to optical discs, make sure that you use dvdisaster:
http://dvdisaster.net/

It allows you to utilize all of the unused (otherwise wasted!) space on a disc with distributed error-correcting data. It is free, cross-platform, and trivial to use. As an experiment, I burned a dvdisaster-padded CD-R and made a deep scratch on the surface with a key. Dvdisaster was able to recover the data without any trouble.

It's quite brilliant software!

http://sourceforge.net/projects/dvdisaster/
http://dvdisaster.net/en/index.html
?

Or something which will make par files...

I've taken to using DVDisaster to actually pad ECC data into the ISO filesystem, so that there's a good chance of recovering the data, even if a file becomes unreadable. Just another layer / method of protection.

It looks like you want dvdisaster. You only lose 15% of your storage (as opposed to 50-67%) and I suspect that it has a better chance of working that of any of three separate files (especially if they're very large) being 100% whole.

dvdisaster is what I use now...both on CDs and DVDs (it also supports dual-layer)

think of it as a way to embed par2 (parity) onto a disc (it requires an ISO image that you create in your favorite authoring software, then after it's done embedding the parity in it, you can burn it)

alternately, you can create a separate recovery data which you can store on backup tapes or hard drives or on another disc, etc.

dvdiaster has a utility to check for back sectors.

You should probably try dvdisaster. it can test media, and can create (on disk or external) redundancy data, which can be used to recover later.
It's also open source, so you could probably coerce it to export some more information

Someone else linked to this kodak stuff as some that is rated for 100+ years.

But I'm really surprised that no one mentioned data redundancy. Use something like dvdisaster to create a huge amount of extra redundancy data. That way even if there is some bit rot, your chances of recovery go up.

Even with that I agree with the commercial pressed cds since they should last longer than any cdr media. I assume there is special pressed media for archival as well. As others have said I would worry too much, 25 years isn't that long, and cds and dvds are so widespread that you're likely to be able to find a reader for that long at least. But if you're paranoid, try a few of the methods until you run out of funds. Multiple gold cdrs with lots of redundancy is probably your cheapest option, then go from there.

already mentioned, but you should try dvdisaster.
I used it when I left my last workplace, burned a DVD with 1GB of data, 3GB of redundancy :P

quickpar especially has been in use on usenet/newsgroups for years....o yea...forgot....they are trying to kill it.

anyways...there's also dvdisaster which now has several ways of "hardening".
one of them seems to catch my attention: adds error correction data to a CD/DVD (via a disc image/iso)

The cross platform program dvdisaster will add extra information to your DVD as an error correcting code. Alternatively, you can make a parity file for an already-existing DVD and save it somewhere else.

It actually has a GUI too, so it must be user friendly.