Slashdot Mirror
How To Verify CD-R Data Retention Over Time?
Peter (Professor) Fo writes "I've recently had two CD-Rs reported to me as faulty which are just 3 years old. This is worrying — I suspect the failure rate for this batch could be 10%. When researching CD longevity there is old and unreliable information; pious 'how to cosset your discs so they last 100 years' blurb; and endless discussions of what sort of dye to use, don't use cheap media, burn slower (or don't), but not much by way of hard facts besides there's a lot of data loss going on.
Does anyone know of a generic utility (win or *nix would suit me) that can map sector readability/error rates of CDs? I'd like to measure decay over time in my environment with my media and my other variables; and I expect others would too."
You should probably try dvdisaster. it can test media, and can create (on disk or external) redundancy data, which can be used to recover later.
It's also open source, so you could probably coerce it to export some more information
Nero has a test utility, but I've not really found the results to be all that useful.
I get similar results from both unreadable discs that are 8 years old and stuff that I think is high quality Verbatim discs burned this year.
Archival Grade Media makes a HUGE difference for backing up important data. It is not very expensive and widely available.
I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered. My life is my own.
http://www.kvipu.com/CDCheck/
Create a CRC file for the CD. Saved me more than once.
just hash the cd and you are golden. Like my apples during the fall.
Oh, please we all know cdr is for volatile data. Only the fabric-made discs long enough. Use HDD or SSD.
dvdiaster has a utility to check for back sectors.
cpghost at Cordula's Web.
Whenever I back up important data, I use par2. If the disc has I/O errors, I just make a full image with dd_rescue (skips past bad blocks, whereas dd will just halt operation) and run "par2 verify" on it. If it's really important, I always verify the integrity no matter what (I've even done it on discs 2 days old, and sometimes, due to the reliability of CD/DVD-R media, it even has errors to repair).
The following tool allows you to track the failure rate of your media, and allows you to recover the files and replace the cd/dvd when it starts failing. http://hardware.slashdot.org/article.pl?sid=08/08/03/197254
dd and diff.
Those tools provide no signal-to-noise ratio (Block Error Ratio, BLER) for physical media errors that the drive is just barely correcting. The point of the request, as I understand it, is to detect how likely a correctable medium is to stay correctable.
I use CDCheck. http://www.kvipu.com/CDCheck/ It may be enough for the task.
And suggest that instead of using CD-R's anymore, you buy some 1TB portable drives and keep them backed up. I don't know your application and you don't explain it, so this is more of a statement about how crap CD-R's are for archiving anything at all, ever. If you have important data on a CD-R, back it up asap.
If you consider your data worth it, have enough time and enough money, you should probably re-burn/re-save them to long lasting media.
There was a previous post on askslashdot about this subject.
http://ask.slashdot.org/article.pl?sid=08/08/27/2119252
My suggestion was to use Plasmon "Century-Disc" :
http://slashdot.org/comments.pl?sid=914095&cid=24784787
(even though I have never tried it myself)
Nero has various tools that might be useful. I'm not really into this sort of thing as the people who are into it are generally off the deep end and take it way too seriously. The CD-R Freaks website might have some people who have helpful suggestions.
I have CD-Rs that are 7 years old that still work. In fact, I've never found a failure. You might consider using PAR to make PAR2 files for your CD-Rs so you can recover the data if it's important to you. You'll need to make those PAR2 files at the time of burn. And I would very strongly suggest that you ONLY use Verbatim or Taiyo Yuden media and consider every other brand to be suspect.
A bunch of my old CD's are still playing with no problems (albeit ones I treated well and are virtually scratchless.) Perhaps someone in the know of how CD's are burned in a commercial fashion can give some pointers for higher quality methods at home.
If you have a high failure rate then its because you are using crappy media. Just like buying heaper casette tapes strethed faster than the more expensive, the cheaper the media, the less reliable its going to be. Unless you plan on continually verify disks randomly I don't really understand the point of testing the media other than to generate some useless numbers. Call a company, find the one with the best actual rating for your price and refresh every 75% of that. I guess its conceivable that certain burners are more brutal on media than others, but I think in the end it comes down to quality media you are buying.
Never, ever had a CD die on me due to old age. I have burnt CDs that are probably older than you.
The only 2 reasons I have ever had a CD die.
1. Bad burn.
2. Dropped it/scratched it.
Okay, I really have only had one reason CDs die:
1. I can be somewhat of a dumb-ass.
- I live the greatest adventure anyone could possibly desire. - Tosk the Hunted
When archiving data to CD or DVD I use a program call Quick Par to generate recovery files that I can use to repair the data on the disc if it becomes damaged.
It is based on the same recovery tech that RAID systems w/ parity drives use, and is mostly used to repair Usenet downloads. I usually put 4GB of data and 400MB of PAR2 recovery files on the disc. This will allow ~10% data loss before recovery is not possible. Also I dont have to worry about the TrueCrypt vol becoming damaged and unusable as well.
Print it all out on Papyrus burry in an Egyptian tomb then use an OCR solution.
I use gold disks and burn at slow speed still no guarantee.
Clay tablets work well.
How about bringing back punch cards?
Laser inscribe data on Copper plates or better yet modern stainless steel would last.
The obsessed people at CDFreaks can help. Here's a link to their FAQ on CD-R media:
http://club.cdfreaks.com/f33/media-faq-61943/
In other places in the cdfreaks forums, you'll find links to tools that can read the C1/C2 error rates. One of the simplest is "readcd", part of the "cdrecord" programs on Linux.
In the DVD world, Lite-On and Plextor both make proprietary programs to read the media-level error rates which only work with their own drives. Lite-On has a Linux version of theirs.
Backups? Use tape.
Optical media is inherently shitty.
If you want to get the best out of it:
Buy good media.
Burn at a slower speed.
Verify the data after burning it.
Store it well. A hard case, and a cool, dry location away from the sun are all you really need.
If you want to test the quality of a disc, go ahead and use any of the tools recommended here.
If you want to harden your discs, go ahead and use any of the CRC tools recommended here.
But really, you shouldn't be using optical media as anything other than a cheap delivery medium. If you need to send stuff to people and you need them to have a copy of it indefinitely, tell them to make a damned copy of it, or give them 2 copies, or keep an ISO and send them a copy when theirs fails.
I work with CD/DVD and related technology as a profession. I analyze, QC, and mass duplicate media by the thousands for extreme and critical field use, every day. My best advice to you is to use Taiyo-Yuden (TY) media, always. I've seen mixed results and bad burns from Mitsui and MAM-A gold, Kodak, and the like. The TY dye type has a proven longer longevity than any other so far. I also suggest burning all of your audio CD's at 16X, this affect what's called single-beam readers. Also, it insures higher integrity of the burn. Burn 16X DVD's at 8X to increase the write integrity.
What others say about is CDCheck is true, use it along with this advice. Use Plextools Pro on a PX-716 drive if you can find one. It seems to be more accurate than Nero tools. Use Plextools to check the C1, C2, and CU rates. If the graph is half-way to the top of the reading, back that disc up. As cheap as media is, I suggest burning more than one copy, storing the image on an external archive hardrive. When burning, don't use overburning. You lose some integrity for error correction.
Store your media in a cool dry place, on it's side. Avoid humidity, light, and heat when you can. Remember, the best analysis tools in the industry are very expensive for individuals. Take a look at CATs if you are interested in learning more about optical media testing. Best wishes!
http://www.cdspeed2000.com/
You need certain brands of optical drives, but with them and this program (and others), you can see the PI/PO or C1/C2 correction (I can't remember which is for CD and which is for DVD) rates on a per-sector basis on your disc. As the rates rise, the disc is going bad, becoming marginally readable and you can copy the disc before it becomes unreadable.
You can find out which drives to buy at http://cdfreaks.com/. The terminology on there for a drive that can do this is a "scanning drive".
I have no idea if you will find that your correction rates are rising over time.
http://lkml.org/lkml/2005/8/20/95
NAME
cdck - simple CD/DVD check program
SYNOPSIS /dev/devname] [-i] [-v] [-p] [-o plot-file.dat]
cdck [-d
DESCRIPTION /dev/cdrom
-d CD/DVD device name, default is
-i Print CD/DVD information and quit, perform no timings
-p Save data for gnuplot(1) program
-o specify plot file, ./cdck-plot.dat is default
-V Print version
-v Verbose operations
Are you sure that the discs really did burn correctly 3 years ago? Some burning software, Windows I'm looking at you, doesn't report errors correctly.
For stuff I care about, I always have Nero verify the data when done burning.
If it's something really important, you should back up your critical CD data onto realiable floppy disks and store them in the back seat of your car.
I imagine if it's important, and you don't mind doubling the cost of your media, you could just burn two. Then, you have something to compare against, and if parts of it go bad, hopefully the same parts won't go bad on the other one. CD-RAID? :) You could even go so far as to make sure the two disks are from two different manufacturers.
ON DELETE CASCADE
Short of an electron scanning microscope, the only way to do it is to hook an oscilloscope directly on to the test points within the drive itself and measure signal levels. This will allow you to measure one or both of: Degradation of the laser optics, degradation of the media. It's anyones guess as to which is which :-)
To make things a little more accurate, you should use several drives to test the media. The drives could benefit from being locked away until such a time as they are needed to repeat the tests. Mix in a few new drives when you do actually make your tests in future as well.
readcd -c2scan
You'll need a drive that supports such a scan, although that's true for any such utility.
The best part is you don't even need an md5sum, par, or anything like that. The verification is built right into the disc.
.sfv verification might be helpful.
I'm not sure what utilities are available on *nix, but winsfv has always worked for me.
Generates a checksum file by file, and then dumps the checksums into a file. Simple, fast, and works great.
I know there are .sfv shell scripts all over the web, as I used to use them on my ftp sites, to verify the integrity of files transferred against the list of checksums. Files didn't match checksum, files automagically deleted.
When making up your disk images, you could .sfv checksum the entire dir structure, and have a file in the root of the cd with the sfv checksums. Then when you want to verify, just run the sfv check against the file in question and compare it to the original.
Works great to ensure nobody's changed anything, too. One bit screws up the checksum.
--Toll_Free
CD-Rs have a shorter lifespan from mass produced CDs due to different manufacturing techniques. Mass produced CDs last much, much longer. I have CDs that are 15-20 years old and still are perfectly fine. Regular CDs are mass produced by stamping a pattern into a layer in the CD, this yields something much more reliable than the burn in used in CD-Rs. the average age of CD-R is 3-5 years it seems. CDs can last for decades, maybe even centuries.
Taiyo-yuden media is not labeled as such. What brands are consistently made by TY?
12:50 - press return.
I have many gigs of digital photos and I have also more-or-less moved away from optical media for backup and switched to HDD. As the original poster mentioned, most of the "information" you find on the net about archival longevity of optical media is personal anecdotes or pet theories, and good hard data on archival longevity of CD-R or DVD+-R is hard to find. My own personal experience is that name brand discs do have fewer problems than cheap "house brands", but it's hard to quantify or say much beyond that.
Backing up to hard drives has a number of advantages:
1. It's a heck of a lot easier - in most cases of personal data backup, a few 1TB HDDs will hold all the data you need to back up, so there's no need to manage boxes of 100's of discs. I usually back up the same data onto two HDDs, and store one of them in a firesafe. If you're really worried, you can store one of them offsite.
2. Since no media will last forever, you will *always* need to roll your data over to new media every so many years. With HDDs, its *much* easier to roll your data over to new media every 5 or 6 years. Think of transferring two or three HDD's to a new HDD (by the time you roll over the data, the new HDD will probably hold all the data from those two or three older HDDs), compared to re-organizing and re-burning hundreds (or more) of CDs or DVDs.
The bottom line is that if a few HDDs don't hold enough data for your needs, then backing up to optical media will be totally out of the question anyway, and you will probably need to use tape.
Yes, there are endless discussions regarding type of media, burners, software, rates, dyes, etc. but due to various levels of inconsistency in manufacturing (how dyes are applied, quality control, RoHS, etc.), why bother trying to gather this type of data?
My recommendation? Look towards either Cloud solutions or other technologies (RAID arrays) or PROVEN backup media (LTO) for VERY long-term storage. Unless you plan on keeping your backup hardware, software, and media static over the lifetime of your analysis and storage, tying up tons of man-hours proving that one or two types of media/burners work perfectly, and then find in two years the software/hardware vendors make changes invalidating all of your testing seems rather pointless.
That being said, I do agree with the others. TY media pretty much performs flawless for me, and I try and store a backup copy in the back of my safe inside my Data Center at work to maximize long-term storage conditions. For me long term is less than 2 years on CDs and DVDs due to paranoia. Media is cheap. Data loss is not.
The request asks no such thing, he simply wants to measure decay rate.
Short of an electron scanning microscope, the only way to do it is to hook an oscilloscope directly on to the test points within the drive itself and measure signal levels.
Or the manufacturer of an optical drive could do the SMART thing: provide some sort of self-monitoring, analysis, and reporting tool to let the user see how many errors the drive has corrected per MiB of data. Mobile phones, Wi-Fi cards, and digital TV converter boxes do something like this, showing SNR in "bars" or in percentiles.
http://qpxtool.sourceforge.net - Linux program for performing low-level quality measurements on CDs and DVDs. It only works with some drive models, so check the supported hardware list.
Take a page from the book of Church© of© Scientology©®(TM) and engrave your data on Titanium© Plates© and store it in Gold© Vault©. I apologize in advance to the Church© of© Scientology©®(TM) if I didnt use enough Copyright©®(TM) symbols while referring to Them (©?). Please do not sue me. © (TM) ®
I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered. My life is my own.
1) Get my archive disc out, and re-burn a new copy.
2) Copy whatever files I still care about (some old stuff can die, some can't) to my harddrive, treat as new data, and re-burn on ANOTHER two discs.
Also, when I RAR anything (backups of dvds or large file repositories, for example), I use the option adds 8 percent parity to the file.
This is similar to your fancy-schmancy "par" tools except it is proprietary to WinRAR. But I figure: If 8% of my disc is unreadable, getting 100% of my data back would be pretty damn cool.
What helps the most, however, is having 5TB of LAN space.
-Clio
Karma: Bad (mostly from not giving a fuck)
Blog: http://clintjcl.wordpress.com
I think that the most reliable way to backup your data is to use a USB hard drive changed every 1/2 years.
Actually I use two : one at my parents in Europe, one at my place in Canada.
A hard drive is much more reliable than any CDR/DVDR, and if your data is important it's worth it.
Just my two cents.
David
How well is, say, a USB flash drive suited to archival purposes? They're pretty hardy little things, so I'd expect them to last longer than your average burned CD.
I suggest that you include a plain text file in the root directory with checksums for all the other files on the CD. Any of the MD5/SHA1 utilities would be fine, such as "md5summer":
http://www.md5summer.org/
Despite other comments in this forum, I would suggest that the CD be recorded in a standard ISO/Joliet file format and not using backup/compression programs. For my own use, I wrote a fancier checksum utility that produces XML format with multiple checksums:
http://www.download.com/Compare-Folders-Java-/3000-2248_4-10895789.html
(That's an obvious self-serving promotion.) The advantage to a plain-text checksum file is that years later you can use another utility to verify the checksums.
The request asks no such thing, he simply wants to measure decay rate.
The method of dd+diff can tell only whether the disc has decayed or has not decayed: 1 bit of information. Something that can read C1/C2 error rates, like the program Wanker mentioned, gives much more information that can be used to give a better idea of how much decay has happened before it becomes unreadable. Plotting this over time gives (ta-da) the rate of decay.
What you could do is setup a system that would do the following...
1. Create a CD and then using your favorite hashing algorithm generate either a whole disk hash or a per sector hash and store this information in a database.
2. Take a disk out of storage, analyze the disk and if all the hashes match up, use this disk to make a copy. Then rerun step 1 on the second disk to verify that all it's hashes match up and if they do, store both.
3. Setup up a routine where once a month you pull a volume out of your archive and verify it's hashes. If that volume fails, go to the second disk, hope like hell it's hashes are good and if they are, duplicate off the second disk.
Yes Francis, the world has gone crazy.
Several hard disk drives and ZFS (which has automatic error detection and correction). Backup problem solved.
I also suggest burning all of your audio CD's at 16X
Did you mean 16X, or "16X or slower"? So-called 52X recorders start at about 20X and reach 52X when they reach the outside of the disc with a greater linear velocity. Forcing 16X makes the recorder use a constant linear velocity over the whole disc to minimize the effects of vibration. But is there anything special about 16X that makes it better than 12X?
this affect what's called single-beam readers.
I just wanted to add something to help people understand what your post means. In cheap mechanisms, used in cheap CDDA players, the pickup moves the laser beam back and forth across the data track, centering the head wherever it finds the strongest signal. This is a "single-beam pickup". Because it uses one beam for signal and tracking, the signal level in the parts where less of the beam is over the data is lower, making the data noisier. This strains the error-correcting code more, and uncorrectable errors show up sooner. More expensive mechanisms, used in high-quality CDDA players and computer optical drives, split the beam into three parts: inside, middle, and outside. In such a "three-beam pickup" or "split-beam pickup", the data comes from the middle beam, and the inside and outside beams are used only for tracking, moving the head toward the stronger signal.
Learn more about pickup strategies
When you need storage and reliability for the long term, there's no substitute.
http://www.historyforkids.org/learn/westasia/literature/cuneiform.jpg
You can buy a hard drive that will store 1000 CD's for about $100. Buy a 2 or 3 and use some form of raid and you should never lose any data for as long as you tend to the rare disk failure now and then. This thread should be about making a cheap home raid array for backup and the best methods for doing so.. how often to scrub the array, etc.. Reminds me of a colleague that wanted to buy this expensive tape backup system, robot type that auto-mounted tapes.. when we had this awesome, unused netapp sitting in the data center.. wtf?!
I feel that making backup copies using punchcards will ensure that they last a long time. Especially if you can get them associated with an election controversy like the 2000 Florida Election Results. Just make sure that they have hanging chads. :P
Well, there's spam egg sausage and spam, that's not got much spam in it.
Hard drive space is so cheap that there is no reason not to.
Get a mirrored set of hard drives, and back it all up. CDs are almost as bad as floppies in terms of retention length.
I like the hard drive option - 1 TB for $99 makes it a no-brainer IMHO.
My question is - if you dump all your data to a drive, lock it in a safe, how likely are you to have a problem when you need to retrieve it, say five years later?
Does a HDD need to be powered-up occasionally / re-written / powered continuously / stored in a fridge?
Prove anything by multiplying Huge Number times Tiny Number
I use Nero DriveSpeed (I think it's called) to give me a read on sample archived discs from my storage area.
I have some CD-Rs that are about 13 years old at this point; I wrote them on a 2X writer that the company I work for paid $3000 for - that was the going price at the time.
Anyway, they're Kodak Gold Datalife discs. I recently pulled 5 samples out of a spindle of 50 discs, and there were only a couple of correctable read errors. This is pretty much what you get on freshly-burned discs too since any little piece of dust or fraction of a fingerprint can cause a recoverable read error.
OTOH, in the past I've seen some truly horrendous quality discs. I once bought a box of 250 CDRs from OfficeMax - they were some generic brand. I tried to use about 30 of them, had 18 bad burns (failed to verify) and the silver layer came off just rubbing it with a thumb. I just threw the whole box away.
I've also seen some cruddy CDRs that died after only a few months to a year.
These days, I use Taiyo Yuiden DVD-Rs exclusively.
For backup, have switched to using 1TB WD Green HDs for $120 (the green ones run cooler, which is never a bad thing when you are talking longevity of hard drives), drop them bare into a docking station, update the backup set, then put them in static bag and foam lined box and transport them offsite.
Hard drives are not much more expensive than good quality DVD blanks (I expect them to get cheaper than, pretty soon), they're way the hell faster, they're reusable (so after the 2nd backup they're way CHEAPER than DVD blanks, and only get even cheaper after that), and I've had better long-term luck with hard drives being readable than DVDs or CDs. As far as failures, IMO you need two backups in physically different locations anyway, regardless of your media of choice.
isn't that the dye "degrades" creating error rates. This does happen with CD-RW's and happens really fast. The real problem is the media physically starts to fall off the acrylic and you start getting pinholes where it's missing.
This phenomena is easily seen by holding the disk up to a bright light. In fact, you'll see these "pinholes" out of the box for some cheap cd's.
You can use better CD's that are thicker and have a layer of acrylic over the dye layer but the problem then becomes clouding in the clear plastic on the read side and scratches.
For best results use high quality cd-r (NOT cd-rw) don't expose them to UV, and don't scratch them.
I have good quality CD's which have lasted over 10 years already with no errors. I just used good quality cds and didn't leave them laying around outside the case.
It's not rocket science.
-Viz
Don't kid yourself. It's the size of the regexp AND how you use it that counts.
2'nd Verbatim
That'd be dd and cmp, actually.
(Why the hell am I correcting a troll? It's not even an answer to the right question)
You're probably just stupid. I've recently went through some nine year old CD-R's and only had two 'bad' ones out of well over one hundred, and those were 'fixed' by using a different CD reader.
Your stupidity could be manifesting itself via your shopping habits, though. Try to find name brand CD's and skip the no-name and store brands.
And next time, Mr. Einstein, use the 'verify' option when you burn.
Sadly to say, the 100 years nonsense is for pressed CDs like you get music and programs on. Burnable ones last maybe 5-10 years, tops. When you add in literal bit-rot due to fungi and so on that exist and love to eat worm-trails in the media surface itself... I've had CDs go bad in as little as a year or two. I constantly have to re-burn my media every couple of years. Thankfully the media density gets better, so I can toss an entire collection of CD-Rs onto 2-3 Blu-Rays and be done with it.
If you want it to be secure, the only viable solutions seem to be flash media or an old-school hard drive in storage. Thankfully the prices of both are affordable for your critical data. All of my critical data and installers and so on fits in a single 512MB flash drive. Toss that in a safe deposit box and forget about it.
I use dvd-ram to archive important files. Designed for archival type storage, the slower media has a 30 year designed life, the faster media has something like 5 year. Add in the builtin ecc and cheap cost, it is a good way to save my source code and photos.
Kindness is the language which the deaf can hear and the blind can see. - Mark Twain
AFAIK, the British government appears to have problems with data retention on either media this way..
Insert
I had an original Sony 2x burner in 1992/1993 when the drive cost $2500, the software cost $2500 (QuickTopix) and blanks were $40 each (Tayo Yuden), so I've been using the technology for a while now. Just for the sake of argument I popped a 12 year old CD (Tayo Yuden mfg.) in my drive and did some read tests on it. Sure enough, still kicking. This disc has been in a jewel case inside a desk drawer for 9 years, and prior to that was on a bookshelf in my old apartment. I typically buy name brand media from TDK, Sony, Verbatim and Maxell these days. Some of the older discs I have are Tayo Yuden, back in the days when they were one of the only manufacturers. I have yet to have a problem with non-scratched media. I follow the storage instructions and almost always store them in jewel cases or on old spindles.
My guess is that this person bought some el-cheapo generic brand discs. I made that mistake once and after the first two failed to write I threw out that whole spindle of 25 and went back to a name brand. In my observations of CD and DVD media you do get what you pay for, and not following storage recommendations will *cough* burn you.
CDs incorporate multiple levels of encoding help with data integrity. Reed-Solomon Codes are used at the bit-level on the disc. The data on top is arranged in chunks with two parity levels: PI and PO (Parity Inner and Parity Outer). K-Probe (free) is able to scan discs and report PI/PO results, but it requires drives that allow for access to that data (Lite-On being one line, though they're not the best quality drives). Nero CD/DVD Speed also has the ability to scan PI/PO and supports a wider variety of drives.
Ideally PI and PO should be 0, but it's never 0 in the real world. Single-digit PI/PO is terrific, 20-50 is quite good. Anything above 100 starts to become difficult to recover data from.
Very smart that you test them every year.
But what if you find that they aren't readable? Then what?
One of the biggest problems I've had over the last two years short of disc failure is newer drives simply failing to read good discs. From CD to DVD's, I have a bunch of older discs that can be read from older CD drives or dvd drives but the latest generation fail or take hours for a few hundred meg's. The exception seems to be plextor.
Don't forget if you're using any type of CD recordable media for storage/recovery you should be sealing the sides with a sharpie and periodically checking them for fungus.
Om, nomnomnom...
Ok, I did find some older (10 years) Kodak discs that the dye seems to have faded on that would not mount in my drive. I can still see the data portion on the disc so I'm wondering if it's just the drive. It would be interesting to see if a different drive or Blu-ray Disc reader would do better with older media. Anyone have a BD drive that could test that theory? I don't at present.
First CD and DVD, including BlueRay, HD, are very different. The CD hardware layer is hugely redundant with each 8-bit byte being written as a 14-bit forward ECC block and each 2048 bit sector is protected by a Cross-interleaved Reed-Solomon code in 304/2048 bits ie hugely redundant. See the Sony/Philips rainbow books.
Thus unreadability means gross damage, dye decay, scratches or thick dirt. Physical scratches/dirt can often be recovered with very fine metal polish and wash. DVDs are _much_ more iffy. A few readers let you read all the 2352 mode 2 data bits of a data CD but most dont and you cannot detect 14 2 8 bit correction which is the first indication of the dye degrading. If you store the CDs in the dark, in scratch resistant envelopes in a strong box you have a good chance of 20+ years. I wrote the one of the first Philips writer drivers and have CDs that old.
Otherwise you are down to replication and data washing, but dont throw the washed CDs out!
Kodak doesn't sell this anymore. You can only get authentic MAM-A from a few places, but here's probably the most reliable source: MAM-A's own online store. This is expensive stuff, and will only get more expensive as the price of gold goes up. They also make more standard "silver" discs which are about the same price as genuine TY.
Costco uses MAM-A media for their photo discs and DVD transfers. Good call.
Knowledge is power. Knowledge shared is power multiplied.
Whadya expect, this junk's all made in Taiwan...
So......?
Filter error: Too much repetition.
If you were REAL nerds you'd be able to add the option to not show the mutual masturbaters who rate each other 2 to 5, because at least the trolls aren't boring as snot.
If you want something fast that will tell you if there is any trouble reading the disk go here, or if you want something more complex that'll go by sector go here.
Oh and in the future if the software needed can be Windows software,allow me to suggest Freeware World Team. FWT have hands down the best freeware search engine I have ever seen. You simply type into the search box what you want the software to do and they'll find you a piece of freeware that does it. I use it here at the shop all the time when myself or a customer has a job that needs filling. No spyware,no trialware,just great freeware with a truly great search engine to find it with.
ACs don't waste your time replying, your posts are never seen by me.
You can't use this idea for data that you create or data that you download that was never put onto CD in the first place. This is presumably for backing up data, so it's not a big deal if you lose data that is already pressed onto hundreds of thousands (if not millions) of discs.
Twinstiq, game news
There are discs and then there are discs:
http://www.printfile.com/index.asp?PageAction=VIEWCATS&Category=180
eh'
cdparanoia gives you stats about failures on the disk.
I bought a cheap spindle of 100 DVDs, about two years ago. Worst dye defects I've ever seen, with the naked eye. The first and last discs of a spindle are usually bad, I tossed those and assumed it wouldn't be a problem. I didn't think about it again until some of them started being rejected by computers and players as unreadable.
I was worried all the discs would become coasters in the next year, so I looked for software that could find defects before the media failed. The software existed, but it wouldn't work with my current drive.
I bought a pioneer DVR-112D, which had the features the software needed. Strangely, when I inserted the bad discs I wanted to use as a baseline for testing, the drive read them without a hiccup. I copied the data off the discs normally, made backups again and whistled a merry tune. (all this happened two years ago, I don't remember exactly how bad the surface scan results were - but comparisons revealed it had more to do with where the damaged sectors were than total damage)
Today I own three drives of that model and they all work wonderfully - it's not a fluke, imho. I have around 600 old discs (DVD & CD), and only a dozen have failed since 2001 - and I've used some the worst media there is. (I havent had a disc fail verification after burning in over half a decade, though. I'm confident even the worst media can be usable.)
So, finding a means of testing and dumb luck were a solution.
Usage with ordinary drives, and recovery with an exceptional one. (Your mileage may vary, but I've found not all optical drives are equal. To me, advanced surface scanning features hint that a drive might be better than average, in additional ways. The drive brand and model I mentioned is cheap and exceptional, but not the best or fastest.)
I have sat on my USB keys a few times (in my back pocket) and they have never broken. I might try that on an SD card some day. I suspect they could more easily break than a USB key, but still less easily than a CD-R or DVD+/-R.
now we need to go OSS in diesel cars
It'd be interesting to have some kind of measure of the integrity of a given ammount of data on a HDD versus a writable optical disc all compared against cost. I bet for data integrity HDDs are far far cheaper, even considering the cost of the gear you need to make a NAS box, for example.
Very often I've bought a stack of writeable disks and found about half of them are buggered, either corrupt data fast or simple won't make it through the burn process. Its simply not a good enough method of data preservation, unless you want to fork out for expensive drives and archival media. So taking that into account, what is the real cost per GB? In the enterprise space we use tape/ZIP drives exclusively since they have good integrity and are reusable to some extent. With writable discs I found myself not caring if I bought a stack of duds and not even bothering with the carbon footprint driving back to the store to say WTF is this crap? If I have some compelling reason to burn (new fangled linux distro) I use a rewritable disc, otherwise I've learned to look for good archival media that works with my drives, and not to buy cheap combo drives. I have some writable cds 8 years old that are still readable, strangely but newer stuff seems to be unreliable after 24 months on average. I wonder if the manufacturers know that their burners and media are crap but realise that 90% of people are not using this for anything critical or long term. (Personally a few years back I began to copy EVERYTHING to my raid based NAS on gigabit ethernet with RAID and 5-year warranty Seagate drives. Never looked back. HDD or nothing. I recommend the same)
After logging in slashdot still does not take you back to the page you were on. It's been that way for 20 years.
The new International Standard ISO/IEC 10995:2008 specifies an accelerated aging test method for estimating the life expectancy for the retrievability of information stored on recordable or rewritable optical disks. The test includes details on the following formats: DVD-R/-RW/-RAM, +R/+RW and it may be applied to additional optical disk formats with the appropriate specification substitutions. It covers the following: stress conditions assumptions ambient conditions evaluation system description specimen preparation data acquisition procedure data interpretation Needs to be purchased from ISO.
I prefer the american version: Lev Andropov: [annoyed] Components. American components, Russian Components, ALL MADE IN TAIWAN!
:)
This space is not for rent.
... and easily lost or pilfered. CDs and DVDs are larger and not so easy.
now we need to go OSS in diesel cars
Compare the md5sum of files in the original ISO to the files on the CD-R. You are keeping the original ISOs on spinning disks somewhere, right?
I know CD's seem like a nice cheap but effective technology, but how cheap is your data? Think about it, you have an unknown chemical dye stuck between two plastic platters with a reflective surface. And how many times has Verbatim or some other vendor changed that dye on you? They always seem to be changing.
Regardless of the "reviews", A plain CD-R (or any other similar item) has never been good for long term storage (regardless of what the marketing department says). Now my music CD may last 100 years because it is not the same (or it shouldn't be) as a CD-R. That chemical dye has a finite lifespan and I would only give it a few years before it starts to degrade.
If your data is truly important, start looking at more stringent technologies that are used by financial firms, government, hospitals, etc. We're upgrading from an older DVD-RAM system and have started utilizing UDO2 Plasmon discs (60GB each) for long term (as well as ablative) storage for regulatory requirements. I'll put one of these up against a CD/DVD-R or Blue-Ray disc anyday. It's more expensive, but the data is irreplaceable, just as is my job.
The truth is usually just an excuse for lack of imagination.
I've given up on optical discs except for short term backup. Now I use hard drives. The last one I bought was a 1TB drive for $150.
You can't expect anythig to last a long time. If you use optical media you have to re-burn your data periodically and make a whole new set. Then you store maybe the last 4 of 5 sets and toss out the oldest set.
With a hard drive I can periodically connect the drive and "sync" it to my files. Only what has changed from the last sync is written out to the disk so it goes fast. The "sync" operation is also a test of the hard drive to see that it still works. One day I'll plug it in and it will be dead, I'm sure this will happen. But no big deal. I'll trash (no re-cycle) the drive, then connect another backup and sync that, then open up a new drive nd sync that one. I rotate three backup drives keeping one in my office, one in a fire safe ad one near the computer.
In importent part of this process is actually trashing drives. They fail and they get replaced. Some time I don't wait for failure, they just get to small or to old or have the "wrong" interface so I recycle them to some other use and buy a big, fast new drive. This keeps my set of backup drive new
I usually see about a 30% to 40% failure rate (immediate ability to read back) on CD-Rs. DVD+Rs are doing much better, at less than 3% failure rate. Just a couple years ago that failure rate for DVD+Rs was less than 1% (and back then we were recording on the order of 60 to 80 per day).
So far, I've had ZERO failure for flash memory technology in USB, CF, and SDHC forms. But I haven't done very many (about 2 dozen) of those. And I don't know how well they work leaving them just sitting for 10 years. OTOH, I do see failure rates going high for leaving hard drives just sitting, as well as tape (which I've long abandoned).
BTW, if those drives with the 5-year warranty die and become unreadable in 4.9 years, you get new drives. You don't get your data back.
now we need to go OSS in diesel cars
You mean you all didn't already know this?
Sorry you didn't take action eight (8) years ago.
Read me: http://ask.slashdot.org/article.pl?sid=00/04/14/1418203
People laughed at my tape drives, too.
Kriston
Would vacuum packing the CDs or DVDs have any positive effect? Sealed away from the ravages of dust mites and oxygen, they might last a little longer on the shelf. When you want to use a disc, it would be as simple as piercing the pack where the hole in the disc is and resealing it when you're done (or resealing a new copy of it).
Admit it. You post strawman arguments as AC so you get modded Insightful for refuting them, rather than Troll
Just some comments IMHO. I always go for TY discs, they seem to be best so far for DVD storage. BUT, don't get full face printable (I would hope that stick on labels would be an obvious no no). The partial printable seem to be fine but full face seem to die quite rapidly after printing (< 1 year).
The last time this subject came up, I bitched about a Matrix pressed DVD I bought in 2002 that wouldn't read (circa 2006/7). I take that back. I have tried the disc in another drive since then and found it reads fine - so drives play a significant part in reliability of reading. (hint - don't use pioneer). NEC drives I've found to be quite reliable, and also the newer LG models.
I second the "burn at 8x" posts.
I use CDs like floppies these days, so I don't care as long as they last long enough to do one job. However, I do have CDs from the 90's that still read fine, but they are mostly TDK. I use whatever CD is cheapest these days.
I have in my hot little studio; not only some CDR's but also DVD's that were burnt years ago. I know some of the CD's were made back in 1997/1998 era, and some of the DVD's were made right around 2001/2002. I'll have to try and set some time aside tonight and go grab some of those backups from the basement and see if they're still readable and such. This should be interesting...!
These were not stored with anything more than put into pancake stack cases, and stored on a shelf in the basement of my house which is normally cool. Will be very surprised if they read. Might even be longer going back over some of that crap and thinkin'.... "What in the hell was I thinking when I did this??"... LOL
All content in this message is copyright (c) 2008. All rights reserved. RIAA is prohibited here.
if you want to create a python script for md5 work, it is blindingly fast, approx 60 secs for 30,000 files/1 Gig on my machine.
http://www.digitalfaq.com/media/dvdmedia.htm
There are just too many low quality disc, and there is no way to tell if they are geniune by examining the packaging, the media-ID is just as easy to fake. For example, the "Hyundai" CDRW or DVDRW are all fake, as Hyundai does not produce nor re-brand them.
From the Taiyo Yuden wikipedia:
"Infosmart of Hong Kong, Optodisc of Taiwan and other manufacturers from Hong Kong, Macau and China have been making discs which illegitimately use Taiyo Yuden's mediacodes and are prone to degradation and data loss. If those discs do not have one of these hub codes, they are most likely fakes and should not be used."
So watch out.
Unless you're getting paid to manage all those CDs, bothering with optical media just isn't worth your time. I would never use optical media to back up something that couldn't fit on one disc and only needed to be backed up a maximum of once a week.
My archive solution is a low-end server connected to a gigantic RAID. If a disk dies (as they all do), you swap out the disk and keep moving. Backing up is completely automated and restoring is a snap. If the data is truly mission-critical, then you need a second server in a different geographical location that automatically mirrors the first over a high-speed link.
Yes, it costs more but it's a bullet-proof system and it scales easily. When you figure in all the time wasted handling and managing the optical media, it might actually come out cheaper.
Store your backup drives in a Faraday cage if you're worried about EMP.
That that is is that that that that is not is not.
- Use a reliable brand. There are thousands of comments already about this.
- If you are reading your burned CDs one year on, and finding 10% have read errors, how can you be sure they didn't have these read errors at the time they were burned? Very few people properly verify or scan discs for errors after burning them, and many may be surprised to find that depending on the brand of media, unrecoverable read errors on 'successfully' burned discs can be as common as one disc in 10 (or, for a recent batch of no-brand CD-Rs I was given at work, over half).
- Consider using DVD media rather than CD, as it is more resilient for a couple of reasons. Physically, it is more resilient because the data layer is located half way between the label and data side, whereas on a CD it is located about a hair's width beneath the label, making it much more susceptible to shallow scratches. Secondly, DVD has a more sophisticated error correction algorithm, allowing for full bit-for-bit recovery of much more random error and longer error runs (= resilience to wider scratches).
- Obviously, store your discs out of the sun and away from dust. Don't move them around too much and don't heat them up or cool them down too rapidly (like, by leaving them in a car).
- Always have redundancy in your backup strategy. If you backup to CDs, that's better than nothing, but it is never 100% reliable. Reliability increases exponentially with the number of times you backup the same information in different ways - if you only have a single backup copy of something, the chance of failure is great. Consider backing up to different formats, different brands, and storing the backups in other locations. Online backup is becoming a worthwhile option now, if you can trust the company with your privacy/security.
- I use the software CD/DVD speed (now called Nero Discspeed) to verify discs after burning. On certain burners you can also see statistics from the error correction algorithm (C1/C2 errors) though the reliability of this depends on the burner.
- As others have said, do not assume that burning at slower than the rated speed will increase reliability. On poor quality media, it is a lottery no matter what speed you burn at. On high quality media, the media may have a lower and upper bound of recommended speeds. Going below the lower bound is likely to reduce reliability. Other media may state a lower bound of 1x, but even in these cases I have found that sweet spot in terms of reliability is just one step below the maximum rated speed of the media - so, burning 16x DVD+R at 12x is right for me and burning 52x CD-R at 48x seems good. Your mileage may vary, and you should always stay within recommended bounds and realise that nothing is 100% reliable.
30 years ago, people were using tape backups, but you'd be hard pressed to find a tape drive, machine with the right bus, software to read the tape's error correction, framing, and other algorithms, and software to make sense of your file. Even if you keep your current machine around for this purpose, the electronics within it may not last, such as the FLASH chips that store the firmware that controls your CD-ROM (in tape drive days it was a PROM that was more robust).
For this reason, I believe in using open, standardized, lowest common denominator formats and making several copies (or storing the file in a cloud -- my Yahoo briefcase from more than 10 years ago still has my files in there, assuming I can remember my password). Then again, clouds can go out of business.
I heard that the U.S. Library of Congress is researching this topic -- open formats, long lasting media, readers, etc. Might be worth a look.
I'm also working on developing an open source paper file system, albeit the information capacity of a sheet of paper is small, maybe a megabyte, if that.
Why not use Glad Zip lock bags?
Unless you have the time to waste, better use archival storage media. It will be cheaper. Unfortunately, MOD is basically dead (due to lack of consumer interest in reliable long-term storage). This leaves professional archival tape. Advantages: Known characteristics, not the mix-it-yourself dye you encounter with CDs. Excessive ECC designed for magnetioc degradation. Disadvantages: More expensive and only cost-effective if you have a large amount of data to store.
If you are willing to copy and verify regularly, you can also use removable/USB HDDs or several independent servers.
Personally, I have only about 5GB of data in my long-term archive. That gets stored on DVD-RAM, several servers, and possibly Amazon S3 (currently investigating this option).
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
That's why most shops have to tag them and put theft preventers around them.
Besides, going back to my previous comment, the UK government appears to manage losing CDs and DVDs quite comfortably..
Insert
Unfortunately, this subject is a virtual extended Appendix (or two) to the Tao Of Backup.
There isn't a good final option, apart from (waits for hail of stones)....tape
. And tape has become expensive.
I have seen people burning data to DVD-R and burning PAR2 files along with the data. This is a very good solution but there remains one problem - only 4 Gig at a time (don't get me started on BluRay or HD).
If you go the backup-harddrive route, or external harddrive route, there are a couple of things you have to make sure of.
* How you partition the drive and whether the partitions are real or virtual - ie. do they actually occupy different cylinders, which is what you want. * Have an error-correcting schema of *some* kind. Journaling filesystems like EXT3 or XFS do help, but again PAR2 to the rescue. * Dont rely on RAID, just don't. Its a failsafe, but not a good one.
$18(-ish)/year for my ~12GB of photos/home-vids/documents, simple cron integration via things like https://jets3t.dev.java.net/ JetS3t's "synchronize", daily incrementals with weekly fulls. Couple of hours to setup, never spent time on it since (other than reviewing cron logs, occasionally testing retrieval). 'Nuff said.
"Ahh! I see you're in that indeterminate Schrodinger state where - oh, uh
Comment removed based on user account deletion
Perhaps you should carve the data into stone tablets in 3 different languages?
http://www.yuden.co.jp/us/product/pdf/mdvd_e.pdf
page 5, 4th col... see where it says 8.5GB? see at the top where it says DL?
every day http://en.wikipedia.org/wiki/Special:Random