TrueDisc Error Correction for Disc Burning?

An anonymous reader asks: "Macintouch has a link to a new piece of software — TrueDisc — which claims to make data burned to record-able discs more reliable. More specifically it uses interleaved redundant cells to rebuild data should part of the disc be scratched. On the developer's blog they say they plan to create an open-source implementation of the TrueDisc system, now that it is not going to be included in the Blu-ray/HD-DVD standards. Have any of you used this software before, and what alternatives are already available?"

Comments

Parchive

[Anonymous Coward]

There's already good parity software available. Parchive will create redundant data that can be burned on the same disc or a separate one. You can create up to 100% redundant data so even if the original disc is lost you can completely restore the files. The software is free and open source. The windows version is called quickpar.

Re:Parchive

[Anonymous Coward]

If you burned two copies of a disc, and both of them went bad on the same file, you're hosed. With error correction codes like par2 uses, you can use any pieces from the par to replace any pieces from the original data, so even if both discs end up scratched, as long as you can get as many par2 blocks off the second one as you can't get blocks off the first, it doesn't matter which blocks they are, you can restore the data.

Or use par2

[bruguiea]

Or use PAR2? It's free.
Tony

Re:Or use par2

[pegr]

For large data sets, I rar to a "block" size one third of that of my media, then put two data blocks and one par block per disk. Yes, it's a pain to restore, even without damage, but it gives me great recoverability, as I can loose up to a third of my disks and still be able to recover. These data sets are typically 50 to 200 GBs, btw...

Re:Or use par2

[dgatwood]

I tried to use it to send my parents a copy of their problematic hard drive that I scraped for them, spread across a handful of DVDs. Turns out that at least on Mac OS X, I couldn't find a PAR decoder implementation that worked correctly if the total data size was over 2GB (or maybe 4GB). It was mistakenly using a 32-bit value for some of its internal math instead of a 64-bit, thinking (incorrectly) that Mac OS X's seek only supported a 32-bit offset. I sent the UnRarX folks a hackish workaround patch to the open source (or was it GPL) tool that they used under the hood. Not sure if they've actually fixed it or not. It also took an eternity to process such a large volume.

Anyway, IMHO, anything that intends to solve the fragility problem of optical media must do ALL of the following:

• Be no worse than half the performance of direct writes. That means that multi-gigabyte Reed-Solomon codes are right out. That also means a very intelligent checksumming mechanism that avoids unnecessary write/seek/settle cycles of the hard disk that will almost inevitably be providing the backing storage for data prior to writing it onto the disk. I'm not sure what the access pattern should look like, but I can think of a good number of examples of what it shouldn't look like.... :-D
• Be BELOW the filesystem level, not above it (on a per-file basis). If a file gets corrupted, per-file protection is okay, but such a protection scheme still leaves lots of very critical metadata without any backup. What happens when a block in the root directory structure fails? You're thoroughly screwed if you just have per-file redundancy.
• Provide backups of data that are sufficiently distributed both in rotational angle and in distance from the edge. Scratches by clumsiness usually occur radially, outwards from the center or inwards from the edge. They may also be arc-shaped. Either way, those are unlikely to destroy a huge amount of data. Scratches due to a piece of sand in your DVD player, spinning the DVD in a roughly manufactured case, etc. happen in a circular pattern, and are, IMHO, one of the major reasons why the mechanisms built into most of the optical disc formats aren't sufficient.... Putting in some n ECC bytes every n*k bytes of data only solves that problem for very large values of n and k so that several entire rotations of the media can be obliterated without losing any data. Unfortunately, that sort of strategy then can quickly turn into a "multi-gigabyte Reed-Solomon code" problem again. Again, on this one, I don't know the solution (though I have some vague notions). I mainly just know a lot of examples of what the solution isn't. :-)
• Take into account that the last few percent of storage on optical media are notoriously unreliable on cheap media and limit the total capacity appropriately to avoid relying on that space for anything important.

After the failure rate I saw trying to do that data transfer (90% immediate verify failure on discs that were over about 98% full, no burn failures ever on discs that were under 90% full), coupled with the failure rate I saw with Retrospect Remote (almost 100% of DVDs stopping with only 15% of the disc used because the craptastic software didn't support burn-safe and wasn't smart enough to pause the burn while it waited for data over the network), at this point, I trust optical media about as far as I can throw it... though maybe not AOL CDs---I can throw them pretty far.... It would be really cool if this sort of tech works and were implemented broadly, as it might make optical media actually useful instead of it just being a nice pretty round disc to slide label-side down against a plasterboard wall in office pachinko.

Just my $0.0195, adjusted for consumer products deflation.

RAR

[mastershake_phd]

what alternatives are already available?

RAR compression has an option for redundancy. You set what % you want to be able to recover if it becomes corrupted.

Re:Sheesh

[Anonymous Coward]

No, you don't understand. The dupes and redundant posts are a required part of Slashdot's backup procedures.

Re:Sheesh

[phliar]

Why not just store the files twice?

Then your overhead is 100%. They promise an overhead of 14%.

There are much better error correction schemes than "duplicate the data" -- look up Reed-Solomon.

Re:Is this enough?

[loraksus]

Your average blank disk out there is pretty poor quality, if anything, this lets you burn on crap disks with at least the chance of reading the data a year or more down the road.
Par does take a while to generate the recovery files though...

What? Huh? Why do we need this?

[Runefox]

In Mode 1 CD-ROM, for every 2048 bytes of data, there's 276 bytes of Reed-Solomon error correcting code and 4 bytes of error detection. Considering we're talking bytes, that's pretty reliable, and as you know, a single scratch often doesn't render a CD totally useless. Usually, a CD has to look like an ice skating rink after an hour of skating for it to fail miserably, and light scratches, even in high numbers, are generally not a factor.

So what the hell? Why is this even necessary, unless you're using a Mode 2 CD (and then, Mode 2 is usually used for videos/streaming data, which requires a more sequential read, where adding ECC would defeat the purpose).

Waste of money.

Re:Sheesh

[ucblockhead]

If you are backing up important data, having the disk go bad is only one issue. I always "duplicate the data"...one disc stays home, one goes to work. No other error correction scheme will work if my house burns down.

That and you can fix discs

[Sycraft-fu]

Get some Brasso (brass polish) and a soft, lint free cloth and you are in business. Really. You just polish the surface so it's all even and thus reflects light equally. If you are nervous about using Brasso, there's a number of products designed just for this purpose, though they are way more expensive and Brasso does just as good a job.

Either way the point is that with error correction as it is now, it's not hard to fix a CD if needed.

Re:That and you can fix discs

[dbIII]

I'll second that - I've used brasso in the late stages of polishing metal specimens to look at under a microscope at 1000x. With a light enough touch you don't see many scratches even at that magnification. A major part of the mechanism is chemical attack on copper alloys, but a lot of suspended really small hard particles that are in there still work with polishing other materials. Silvo has smaller particles again which I've used for the final polish - but you wouldn't need that on a CDROM.

Re:Sheesh

[tenton]

Music CD's already include error correction bytes embedded in each frame of data, so I assume this technology does the same sort of thing for data CD's/DVD's/bluray's/etc..

On music CD's, there's one error correction byte for every three bytes of data. That's a lot more space-efficient than just burning your data twice.....

Music CD's have piss poor error correction, by data standards. CD-ROM and DVD-ROM (which includes the video variant, since it's an application of DVD-ROM) have much more robust error correction. There is more error detection (and correction) per block on a CD-ROM (consequently, less for data) than on a music CD. Music CDs have the additional advantage of not needing to be precise; it can try to guess (interpolate) the missing data it runs into, or, at worse, skip (which may or may not be noticable). Can't do that with a spreadsheet.

Burning your data twice also has the advantage of being able to separate the copies (to different physical locations). Error correction technologies aren't going to help if you CDs and DVDs are roasted in a fire; the extra copy you made and put into storage elsewhere will still be safe.

Re:Sheesh

[peter]

> In general the ECC on DVD is going to prevent you from getting bad data; it's extremely unlikely that you're going to be able to successfully read an ECC block with (say) some bits flipped.

  I burn my movie collection on DVD with par2 blocks and md5sum files. When I verify them, with some disks in some drives I get data errors. So I have seen in practice that you sometimes get silently corrupted data. My NEC-3500A burner is starting to get old, and doesn't read as well as it used to, I guess.

  Par2 is slow to generate, but worth it. I have actually recovered files from slightly bad disks thanks to my par2 files.

  For a disc with lots of small files, par2 would suck because it doesn't know about directories, for one thing. dvdisaster is like par2 for iso images. I don't think it's set up so that you can keep the error correction files on the same disc as the data, though.

ISO9660?

[NekoXP]

From article: "Since the TrueDisc format is open and the master copies stored by TrueDisc are located in the standard ISO 9660 filesystem"

That pretty much fucks up anyone's day when they wanna burn a UDF DVD doesn't it? ISO9660 doesn't support files greater than 4GB, you can only have 8 directories deep (until the 1999 spec but I always had a hell of a time reading this stuff on anything but XP), stupid filename restrictions (and then do you use Joliet or RockRidge or whatever to fix it or not?)..

Free alternative: dvdisaster

[adam1101]

This $89 (or $52 intro price) TrueDisc sounds rather similar to the open source dvdisaster. It builds Reed-Solomon error correction data from CD or DVD iso images, which can be either augmented to the image and burned on the same disc, or stored separately. It's somewhat similar to par2/quickpar, but dvdisaster is more specialized for CDs and DVDs.

Explanation of what's interesting about this

[goombah99]

So far all the comments I've read are way off the mark about what is interesting about TrueDisk. Yes it's true that TrueDisk is just yet another error correction scheme. What is slick about it is it's high usability. This comes from two things
1) It writes the correction bits to a separate partition from the "regular" bits. As a result, the primary partition looks exactly like a regular CD. put it in any computer, even one not equipped with the TrueDisk Software and it can be read normally.

2) The amount of the redundancy is automatically chosen. It just uses any left over space when it finalizes the CD.

As a result the operation of TrueDisk is pretty much transparent. You only need to invoke the truedisk software to read a disk that has been corrupted. Uncorrepted disks can be read normally. So You won't lose your data if you don't have the software or the company goes out of bussiness and it stops working on newer OS's. (All you would lose without the software is the ability to recover from the redundant bits. ).

In comparison to PAR or RAR, you are not compressing the data so it's faster. Now I note that if you compress and then add redundancy you could potentially have higher redundancy for a given amount of data on a fixed CD size. So there could be some theoretical advantages to RAR and PAR. However, those PAR/RAR disks cannot be read in-place (they have to be expanded) nor in "real time" (say if you are playing video). They are very slow to write. They can't be read on any computer without the same verison of par/rar. And if you do lose bits beyond the point of recovery the compressed bits will span a much greater extent in the data space--you might even lose the entire CD with PAR/RAR. So you can see that TrueDisk has usability advantages even if it's redundancy is less and it's uncompressed.

Re:DVDisaster?

[brendan_orr]

I agree, DVDisaster is quite nice, I've my main copies of backups, then a separate disc with error correction files with copies being held on a hard drive and eventually tape, other hard drives and any other medium I can...at least for the really important backups. My ogg collection I'm not too worried about (as I can always re-rip the song/album should corruption occur)