Slashdot Mirror
How To Verify CD-R Data Retention Over Time?
Peter (Professor) Fo writes "I've recently had two CD-Rs reported to me as faulty which are just 3 years old. This is worrying — I suspect the failure rate for this batch could be 10%. When researching CD longevity there is old and unreliable information; pious 'how to cosset your discs so they last 100 years' blurb; and endless discussions of what sort of dye to use, don't use cheap media, burn slower (or don't), but not much by way of hard facts besides there's a lot of data loss going on.
Does anyone know of a generic utility (win or *nix would suit me) that can map sector readability/error rates of CDs? I'd like to measure decay over time in my environment with my media and my other variables; and I expect others would too."
http://www.kvipu.com/CDCheck/
Create a CRC file for the CD. Saved me more than once.
RAID5 for CDs? Is there anything where I can burn 3 CDs with a 'set' of data. When I want to restore my data I just put in each disk sequentially and then it does some RAID5 magic and spits out my data?
Be a cool project, IMHO.
Backups? Use tape.
Optical media is inherently shitty.
If you want to get the best out of it:
Buy good media.
Burn at a slower speed.
Verify the data after burning it.
Store it well. A hard case, and a cool, dry location away from the sun are all you really need.
If you want to test the quality of a disc, go ahead and use any of the tools recommended here.
If you want to harden your discs, go ahead and use any of the CRC tools recommended here.
But really, you shouldn't be using optical media as anything other than a cheap delivery medium. If you need to send stuff to people and you need them to have a copy of it indefinitely, tell them to make a damned copy of it, or give them 2 copies, or keep an ISO and send them a copy when theirs fails.
If it's really important, I always verify the integrity no matter what
I do that even if it's not important. I have a script which creates an md5 checksum file for a directory tree and adds it to the directory, and I always run it before burning a CD or DVD. Once burned, I verify the checksum on two different computers.
There have been a few times that the computer that burned the disc successfully verified a new disc, but a different one didn't. When that has happened, I trashed that disc and made a new one.
Sometimes I wonder if a lot of the reports of "deteriorating discs" are actually cases where someone burned a coaster in the first place, and just never happened to try to read or verify the data until years later.
Interestingly enough, I work for the first company to ever produce and sell pre-formatted floppy discs. We're right down the street from M$ in Redmond, WA. I use many different brands for different uses and client needs. HA, I don't work for TY, and that's why I shamelessly promote them ;) I also suggest using Verbatim, the dye and quality are very similar. In all of my experience, TY is the best. BTW, optical media manufacturers come see us as the perfect testing ground for end use because we do everything you can to optical media, down to print and packaging. (dsgi.com) Cheers!
The request asks no such thing, he simply wants to measure decay rate.
The method of dd+diff can tell only whether the disc has decayed or has not decayed: 1 bit of information. Something that can read C1/C2 error rates, like the program Wanker mentioned, gives much more information that can be used to give a better idea of how much decay has happened before it becomes unreadable. Plotting this over time gives (ta-da) the rate of decay.
Also looking for Taiyo Yuden brand helps (or anything made in Japan...that should be TY just rebranded). Every knowledgable board I've looked over on the subject has recommended them, and I have never had a single one of their disks out of probably 200 now be bad from the start. Unfortunately I don't have any 'old' disks to test. I keep most of my data backed up on hard drives.
As our way of thanking you for your positive contributions to Slashdot, you are eligible to disable Slashdot 2.0.
Does TY make DL discs?
Exactly. I have 60 archival grade CD-R
s from over 10 years ago that are still readable Yes I have tried them, it is a part of our backup integrity testing every year.
Storage and handling is also very important with them as well.
Do not look at laser with remaining good eye.
I use dvd-ram to archive important files. Designed for archival type storage, the slower media has a 30 year designed life, the faster media has something like 5 year. Add in the builtin ecc and cheap cost, it is a good way to save my source code and photos.
Kindness is the language which the deaf can hear and the blind can see. - Mark Twain
RAID5 for CDs? Is there anything where I can burn 3 CDs with a 'set' of data. When I want to restore my data I just put in each disk sequentially and then it does some RAID5 magic and spits out my data?
Be a cool project, IMHO.
3 CDRW drives + direct writing and RAID options turned on in kernel + mdadm to setup the desired RAID level
Just off the top of my head, but it might work...
--
k
This might be a dumb question .. First before: .. does the material slouch or disintegrate over time? For being a hardware guy for awhile, it just seems to me that CDs wouldnt go bad ... not for hundreds of years at least
I have always thought CD media to be the BEST way to backup/archive data, because it doesnt have magnetic retention that can be lost over time. Out of any media, what media is the best way to archive data? And for the dumb Question:
How do CD's actually go bad? They are physically carved
Go go Gadget Nailgun!
CDs and DVDs will last for a long time, it's the (re-)writable versions that degrade (relatively) rapidly. The non -ROM versions aren't physically stamped, they use dyes that change phase when exposed to lasers. They are very susceptible to damage from UV light (i.e. sunlight), and I believe will deteriorate over time even if stored in a dark room.
The storage footprint of data tends to be inversely proportional to its importance.
People lose sight of this fact, and because they don't see a good way to backup the 500 gigabytes of data they have, they somehow fail to backup the 20 pages of documents that they need to protect their assets or limit their liabilities.
Don't underestimate the value of a document printed in archival ink, stored in several fireproof locations. Definitely don't underestimate the archival value of, and potential longevity of, hand-written journals.
This has worked well for me. I had a situation where a RAID 5 lost two drives and my backup had some corruption in a 100 gig video project. Pulled out my 3 year old DVD's and recovered the data fine (and yes there were one or two of chunks that had problems but recovered from the redundancy).
Just my method and 2 cents.
--
So who is hotter? Ali or Ali's Sister?
I have boxes of TRS-80 diskettes, Model I 40 track disks, from as far back as 1978, that have been stored in the most careless manner in very nasty conditions (were even, literally, in a barn for a few years) and they can all still be read. It's unfortunate that none of the information on them is important, which superstition tells me may have something to do with their success rate. Yesterday I burned an Ubuntu 8.10 CD onto a store-brand CDR, and it failed in the second machine I used it in. To be fair, it has a radial scratch that got on it somewhere, and, I burn these things intending to throw them away :-)
-fb Everything not expressly forbidden is now mandatory.
First CD and DVD, including BlueRay, HD, are very different. The CD hardware layer is hugely redundant with each 8-bit byte being written as a 14-bit forward ECC block and each 2048 bit sector is protected by a Cross-interleaved Reed-Solomon code in 304/2048 bits ie hugely redundant. See the Sony/Philips rainbow books.
Thus unreadability means gross damage, dye decay, scratches or thick dirt. Physical scratches/dirt can often be recovered with very fine metal polish and wash. DVDs are _much_ more iffy. A few readers let you read all the 2352 mode 2 data bits of a data CD but most dont and you cannot detect 14 2 8 bit correction which is the first indication of the dye degrading. If you store the CDs in the dark, in scratch resistant envelopes in a strong box you have a good chance of 20+ years. I wrote the one of the first Philips writer drivers and have CDs that old.
Otherwise you are down to replication and data washing, but dont throw the washed CDs out!
Wish I could mod you to +6, sir. As data retention and migration has factored in my career (and hobby) for many years, it's great to see such concise accuracy.
I will vouch that CDs are more fragile than DVDs. I have lost entire Kodak Gold CDs from a single flake off the label, near the disc center (the TOC). Meanwhile last month I experimented with an xacto knife and a DVD-R movie. I could not make a cut deep enough on the label side to disrupt the playback.
I have tried most 24 karat gold media available on the market, and gold media is the only thing that I will consider archival. However not all are created equal, and the Kodak gold media you can buy in 2008 is not the same reknowned manufacturing as in 2000. The advantage now is that DVD sizes are widely available, but still notably expensive for a hobbyist. MAM is still the cream of the crop.