Slashdot Mirror
CDs, DVDs Eyed For Long-Term Archival Use
Alien54 writes "Computer scientists at the National Institute of Standards and Technology (NIST) are launching an effort to develop specifications for 'archival quality' CD and DVD media that agencies could use to ensure the procurement of sufficiently robust media for their long-term archiving needs (i.e., 50 years and longer). See the press release at the NIST site." The research involves "...enclosed chambers that use temperature and humidity changes to artificially age the media some 20 years in only six weeks."
Now if only we could use these chambers on RIAA CEOs...A couple months at most and we can get on with the important work of finding new ways to make fun of SCO.
"The government of the United States is not, in any sense, founded on the Christian religion."
Age 20 years in 3 months.
That's what waking up at 3:00 in the morning every day to take care of the kid does to you.
With the decay rates of CDs, it seem like about time!
"...enclosed chambers that use temperature and humidity changes to artificially age the media some 20 years in only six weeks."
That makes me wonder if they are considering the use of my apartment for this...
A link to where volunteers can submit Celine Dion, Westlife and New Kids On The Block cds to be included for testing would be greatly appreciated
Wenn ist das Nunstueck git und Slotermeyer? Ja! Beiherhund das Oder die Flipperwaldt gersput.
artificially age the media some 20 years in only six weeks
that's nothing
This ages braincells a solid 20 years in 6 hours !
When will I end this grieving ? When will my future begin ?
The researchers are very welcome to use my workplace, it has managed to age me 30 years in as much time as their machines do (at least I feel like it).
Might I also suggest showing the CDs and DVDs our collection of SCO stories? Those get old so very fast. They are guaranteed to feel like its been 20 years already.
If you outlaw the law, only criminals will have laws
Are they still working?
We also need to think about preserving appropriate equipment for retirieving data from elderly cds and dvds. Resources are finite. What's going to happen if we use misuse our laser reserves and there are none left decades hence with which to read all those carefully preserved discs? These are the kinds of issues I would like to see politicans talk more about instead of just tax cuts.
Honestly,
I've got CDs that I burned just 2 years ago, and my CD drive has trouble reading them - no scratches, it just appears that they age waaaay to quick. I know a lot of people who keep photos on CD, I hope they realise that it's not so permanent.
Maybe something good would come of this. I'd actually be willing to pay out $5 to $10 to get a CD that once burned would stick around for a while.
tom-george.comBecause geeks rate higher t
...enclosed chambers that use temperature and humidity changes to artificially age the media some 20 years in only six weeks.
Temperature and humidity are definitely not the worst enemies of my CDs. My friends are.
It's cool to create media that can hold information for an extended period of time. But - do not forget that you need to have a device that can read the media. I've saved some of my earliest work from the 70's on a paper strip with holes in it, and from the 80's on a 12" floppy disk. Both look like mint condition, and I'm sure they work. But - I haven't got any hardware that can read them.
So - if you plan to store digital information for decades, you need to store the player as well. That means, you need to make hardware that will work after, say, 100 years. This makes me think if we should strive after something that's human readable (microfilm or plain old paper) instead of something that require a computer. This is by far an easy problem to solve. My humble suggestion is to save information on todays media and prepare to copy it to a new media every 10 years.
For this, WORM's have been invented. Currently at 9.2 GB per media, put larger versions are in the pipeline. They are still readable 10 years after, and have been guaranteed to be readable for 100 years, given the software exists.
Just you can't burn them with your run-of-the-mill software, you need some professional software for the whole document and jukebox managment as well, else you'll have some problems to find you archived data in a decade or so when the audit comes.
Sounds like my brother's record collection just before he moved out of the basement. I could lend him a CD for a week and, if I ever saw it again, it had mysteriously accumulated a decade's worth of wear and tear.
The only sure way to archive data is to keep it on a network-attached device - and migrate it regularly with changes in technology. No removable media is foolproof as hardware can break down at a time when it can't be repaired or replaced. Ask anyone with a Betamax video collection or, more relevantly, the BBC, who had great trouble reading their not-very-old Domesday archive on laserdisc. BTW, that's not a really small computer in the photo, it's a really big CD!
When I am king, you will be first against the wall.
the best way is to save this data is by arranging a big ass series of beer bottles. 0's = beer bottle upside down. 1's = beer bottle right side up.
I volunteer to help begin the media preparation process. Anyone want to help?
Why bother when other standards for backup media will come out that will totally put the previous generations into the obsolencense storage shed.
How long does the magnetic bit last on a hard drive? What's all this stuff on slashdot about "Lab Diamonds" that will be used to create CPU's with insane clock speeds? Instead of DVD how about a Diamond disk of lab grown diamonds?
Explore that!
50 years is not long, 500 years is what we should be talking about.
Books, if looked after properly, last for centuries. OK: many modern paperbacks are printed on paper that has not been properly stabilised (still contain acid), but there are plenty of very old books.
In case you think that I am over the top: have you never looked at an old family album with pictures going back to the start of the last century? What will future generations think of us if none of that sort of material survives because we had the lack of foresight to put it onto good media?
There's a number of CDs which have already experienced 10 years of mistreatements, I wonder if any mass producing company has already learned something valuable and if they modified their production accordingly. Polycarbonate-eating fungi were already mentioned here and on Nature as well. Add the aluminum layer oxidation problem and my trust on cd-r as long term storage is reaching zero. I also own a couple of 10+ years old CDs (original, shush RIAA) that don't show any surface problem, but no player I have tried can play it anymore.
Humidity? Temperature? How lame. :)
Just get the CDs to spin fast enough to make their edges reach near c speed and store the data near that edges and you're safe about aging - a year for such data will last ages for us
45 5F E1 04 22 CA 29 C4 93 3F 95 05 2B 79 2A B2
We just had a baby girl (yes, even geeks need to reproduce). So people ask "what can we bring the little gorgeous thing?" (they don't have to sit through nights of "woaAAAHHH!") I've figured that the best thing would be presents that she can open when she's old enough to appreciate them, like on her 18th birthday.
DVD players may still be around in 2021, after all I can still read 3.5" floppies. But DVD media has a shelf life of 5-7 years AFAICT, several older DVDs I've tried recently don't work anymore. CDs may be less delicate, resist better.
But if you wanted to give someone a digital present (say a bunch of their baby photos) for 18 years hence, how would you go about it?
This was going to be an Ask Slashdot, but (a) I'm too tired, and (b) the whole "what can I give a gurgling baby" thing is not really stuff that interests geeks.
Ceci n'est pas une signature
Discs in fact are way better method of storage than traditional tape storage due to ease of accessiblity. However, in order replace tape for long term archiving, more has to be done than making durable discs.
:). I do not know exactly how but they must have something to charge us enoromous amount of money for recovery. :)
The other major fact is Recoverability. It's not unusual to find defective tapes before their end-of-life and we must send them to experts for retrieval of important data in them. They've technologies to recover the data, like baking the tape(yeah, bake them in oven, but please don't do it with your kitchen oven
I'm not sure if existing technology could effectively recover data from aged, defective discs. That's something we must consider before they could replace traditional tape storage for long term archiving.
no, you archive to then you read and rewrite to and use checksumming or similar to make sure they're ok.
I wonder if a glass CD would be appropriate. Or maybe that's not the bit that decays...
I just remember hearing the term 'glass master' when talking about music CDs.
"Derp de derp."
It shows that digital still has a long way to go compared to the current UK practice of printing on vellum... in other words goats skin !!!
Quote: "... we compare longevity of 250 or 500 years [of long-life paper] with the 1,500 years of vellum"
---- The Open Source Record Label : : LOCARECORDS.COM
They tested 30 different brands of CD that had been recorded only 20 months earlier. Here is a picture of one of the CDs.
The red area can no longer be read.
This is pretty hideous.
I use CDs for archival storage.
It looks like it will become necessary to copy everything to new media every year or so, lest it become lost, forever lost, never to be seen again by the eys of mortal man.
It's very annoying.
Those who sacrifice security to condemn liberty deserve to repeat history or something. - Benjamin Santayana
For a start.
A typical backup tape will handle 120Gb to 200Gb these days.
Then you have the problem with getting hardware which will read the disks in 20 years or 50 years.
The real solution to archiving is the ability to move to new formats as they appear and become cheaper than the existing technology, it's an ongoing process, not a product. The hardware itself should be irrelevant.
Government of the people, by corporate executives, for corporate profits.
Yes I still have some CD's and a few of them are from the 80's.
I couldn't tell you if they work or not, because all the music I play is in MP3 format.
Why look for a 50 year solution, when in 20 years the archives will be stored on more efficient media, just like mp3's.
thank God the internet isn't a human right.
I understand that removable media always run a much greater risk of going bad as they're exhibited to all sorts of possibly harmful effects. However, I'd really like to see "CD-like" discs that last at least for around 20 years to give us plenty of time to at least transfer them to more modern media when they arrive. The problem right now is pretty bad since the media degrades much quicker than new technology arrives, with CD's already becoming unreadable when we haven't even fully made the switch to DVD's yet. I'm sure there are other perhaps more reliable removable media available, but they aren't as widely accepted, and I find the problem actually rather silly since reliability on removable media should come as a top priority, with those often being used exactly for storing old data not immediately needed on a hard drive -- as an archival media.
:-P
If this test will lead to an insight in making more reliable CD's or DVD's, where those can be somehow certified with a special "Archival Quality" tag, I'm sure they would sell a lot even to a greater price. I'd completely switch to them at least, since I burn CD's to make them last for a longer time than a year or two. Switching to tapes or something like that isn't very useful, since everyone I might bring my CD's to would need a tape drive.
Beware: In C++, your friends can see your privates!
Hopefully, someone will finally put the hoards of AOL CDs to good use.
I don't know... But I never will, so who cares?
If you're a sysadmin, this problem has already been solved. RAIDed hard drives, always on, read occasionally to check for errors, and drives replaced as they fail. Replace the drives with new models every so often (or as they fail perhaps). Replace the controller and system it is attached to as necessary.
That is to say, that no digital storage that exists outside of a lab is suitable for long-term archival. Luckily, digital data being so easily copied (how easily people forget this!) makes this an easy problem to ignore. If you're developing new types of media, great. Otherwise, there is only one practical solution.
Yes, "my" solution - the solution used by anyone who has digital data they want to store long-term - requires someone to babysit the data. Sorry, most things in this world need some kind of human maintenance.
If you're storing the data on hard drives attached to a working computer, you can mirror the data on the other side of the world to protect it against any catastrophe that humans will survive.
If you don't care about a practical solution that has by far the highest chances of success, feel free to speculate about how long CDs will last based on completely invalid lab testing. (Accelerated aging? Hah! How can they possibly account for every variable?) If you truly care about your data, keep it online and make sure someone is around to maintain the system. If you want something less, it's because you don't actually care about the data that much.
Kodak did some accelerated longevity tests on CD-Rs and found that many disks degraded rapidly when exposed to sunlight, due to the UV components of sunlight.
Mea navis aericumbens anguillis abundat
I may be slipping into tinfoil hat mode here, but I'm rather hoping these don't reach the public market. Why? Because if they do, I could see us ending up with a two tier recordable media market. One range for short term use, the other range for long term use. The former would be lacking in reliability, whereas the latter would last much longer but would be more expensive.
It looks like it will become necessary to copy everything to new media every year or so...
A common mistake is that archives (digital or otherwise) require no maintenance. In the case of digital archives you should be checking them, on an annual basis, not just for physical degradation.
A more common problem is that the applications used to create the data and/or their documentation do not exist any more, rendering the data as useless as if the physical media had been destroyed.
--
This sig is inoffensive.
On the other hand, I'm also a bit concerned with privacy, and the idea of these huge intrusive databases, or archival of all traffic over key gateways of the net bother me. But when I consider the difficulty that I have with huge amounts of data that are just a drop in the bucket conpared with this sort of thing, I breathe a little easier... Unfortunately, duplication and propogation appear to be the surest way to go, and unfortunately there is often a tendency for those who would invade our privacy to share their data for profit or reasons of control...
HaXXXor.com - Naked Chicks Teach You How To Ha
Magnetic tapes leak, don't they? I have a pack of cassettes recorded with old stuff I wrote for a C64, 20-odd years ago, and even ten years ago they were already unreadable. This was not even high-density recording, just normal screeching. My understanding is that each layer of tape has a small effect on its neighbours, and after some time the entire tape is reprogrammed to noise. Presmably if the tapes are played and rewound the effect is less dramatic.
But magnetic tapes do not strike me as particularly stable. Hard disks may be more stable than tapes.
Ceci n'est pas une signature
No need to research this. There's an already exiting technology for this: magneto-optical media (MOD). These have been created for archival purposes. As far as I recall, there life-expectency is around 30 years, whereas CDs last for less than 10 years. Personally, I still backup my home directory with a MOD-drive from 1992.
Global warming will end civilisation in a few years, so what is the point?
I stole this
Here is a project based on peer-to-peer concepts that aims to preserve information over long periods of time without depending on specific media, readers, etc.
Seems far more realistic, after all this is what most of us do with valuable data, we copy it from hard disk to hard disk over time.
Ceci n'est pas une signature
In 296,000 years Voyager-2 will pass Sirius... Do you think the gold video disc on-board will still be readable? :-)
"You lied to me! There is a Swansea!"
one way to preserve data longer than 50 years is still microfilm. Guaranteed to las at least 100 years, if processed and stored correctly. Currently several companies offer the service to store 'digital dots' on microfilm, instead of typeface, improving the data-density considerably. Of course density is still waaaay below that of DVD or CD, but at least you're sure it'l be there in the future, and relatively easy to read (optical-scanning...)
Well it would be nice to have something where I know all my digital photos will be safe in a few years time. I've been using RAID hard disks but will I still be able to get the data off in 5 years let alone 20. It would be nice to have some very long term storage which has some sort of guarentee
Or what about RAID-1 DVD-R's?
Rus
Cheap UK and US VPS
CD and DVD media have a certain lifespan. if you copy them at the end of their lifespan for archive use, will you be subject to the DCMA and can you be pursued by RIAA and MPAA? this would be interesting since the first CD's will reach the age of 20 within a few years/month...
".Sig Stealer" was here
Some things I've discovered so far are:
The biggest problems seems to be that the CDs come and go, so it can be difficult to get the tested products. The tests that has been done has used "accelerated aging", which is just a simulation. That is, there is no real experience in aging CDs.
My advice would be to store valuable information on as many different formats as possible. Continually monitor the quality of these, and transfer to new backups when they start to degrade too much.
Hope this helps!
And acid free paper doesn't turn to ash.
The RIAA must be ROTFLTAO at the thought that the plastic they sell is a perishable good. Only slightly (take the long view, some books are hundreds of years old,) more perishable that the original source which only lasts as long as an echo.
I have vinyl from the '60s and '70s that I played on a good turntable then and (since I still have that turn table,) I can still listen to now.
Since the early 20th century, our industrial processes have been destroying our heritage.
MSBPodcast.com The opinions expressed here are my own. If you don't like 'em... Think up your own stuff.
Um... no. That's a decade-old bromide that's becoming more untrue every year. Today we have downward compatibility and standardized, open data formats. I'm not saying that the problem can never occur, but it's not common and easy to avoid. Besides, even if you were stuck with a proprietary data format, you could most likely reverse-engineer it with moderate effort.
The illegal we do immediately. The unconstitutional takes a little longer.
--Henry Kissinger
Considering the relatively short life expectancy of digital media, the DMCA, and the extensions to copyright term over the past few decades, imagine the Dark Age our children and grandchildren will be facing 50-100 years from now. Only the memories of old men and women and the loot of "pirates" will be available to help fill in the great blank space in our cultural history.
Age 20 years in 3 months.
Dude, reverse that process and you've got a winner.
The coolest voice ever.
I see a lot of people saying that the short CD lifespan is not a problem .. you just copy from one generation of media to the next as you approach the optimum time.
Well that doesn't really cut it for two main reasons
1/ You have now decided that the only information you will hand down to the future is that the stuff that you care about now. As soon as you stop caring about that data, or your descendants stop caring, then that data will lost.
2/ It will only need a skip of roughly 2 generations of technology before you won't be able to recover any digital data that you (or someone else) accidently re-discovers.
If this doesn't seem important, look at what historians and archeologists are finding/learning from poking around things that have survived millenia, compared with the despair of knowing what huge gaps exists from records/items that have been irretrievably lost.
So how do you want to judge the concept of "archival"? As something that is accessible as long as the item is whole, or as something that requires active intervention to maintain its integrity?
I am Slashdot. Are you Slashdot as well?
Yes, lawyers like it too. My father used to take pleasure in showing me some of the old leases and deeds he would come across which were written on vellum. The ink was basically iron oxide after so many years, and therefore still quite legible. The vellum from the late 1700s was in better condition than parchment from the 1900s.
Panurge has posted for the last time. Thanks for the positive moderations.
30 years is considered good enough for day to day use, nowadays. But as a teacher of mine keeps repeating: "we have more problems retrieving data from the 1970's than from the beginnig of this century" Indeed, our future generation will only see a gap...
Yes? What about when it breaks? Can you go out to the shops and buy one which plays 78s?
Government of the people, by corporate executives, for corporate profits.
I have a large collection of 'files' on CD, they've been in a cd case for about a year, and they're already disintegrating, with bits falling off!!! Spose thats what you get from cheap CD's!
.
This was also my conclusion, print out the pictures and stick them in a nice album using starch-based glue. Will be very quaint in 2021.
:-) Flip book?
But what about the videos?
I don't really want to present my daughter with a pack of SCSI DLTs and the hardware to go. Surely something more simple... portable DVD player, maybe, complete with power supplies. If only I could be sure DVDs would last.
Probably the best option would be a link where she can find her data online and live. Idea for a new kind of business? www.everlastingdata.com, guaranteed archival for 100 years!
I threw up a comment about LOCKSS, which does something like this for libraries.
Ceci n'est pas une signature
The difficulty is that these disks cost a dollar or two each. Compare this with the el-cheapo ones that sell by the billions. Few mass consumers bought the good CD's, and Kodak stopped making them entirely and the Mitsui's are now specialty products that are not widely available.
There's also been a major shift in where and how CD-R's are manufactured. At first they were high-spec products, made in a few select factories in the US and Japan. Then manufacturing scaled up and cheaped out as the plants moved to Taiwan. Now a lot of those plants are going even lower-budget and moving to Mexico and mainland China.
The point is that consumers rarely buy for longevity... they go for neat packaging or cheap price or high burn speed or something else. The CD manufacturers have learned that lesson well. That's why it's so hard to buy good archival CD-R's anymore.
But CDs are an interesting case. You could argue that, unless we lapse into complete barbarism or some rejection of science, recovering old CDs should be possible for any future civilisation if the bit pattern is preserved. Provided the encoding and protocols are stored safely somewhere, it should be possible to construct a reader if anything is considered important enough to read. Unlike tape or punch card, the mechanical handling needed for a CD reader is very simple. Small lasers are made in ever greater volumes, and anything that replaces them is going to be more, not less capable. They use little power and there is no environmental reason why they are likely to fall into disuse.
Even so, my best photos are printed on archival grade non-resin coated acid free stock that should last a couple of hundred years. As if anyone is likely to care.
Panurge has posted for the last time. Thanks for the positive moderations.
Hey, we have this great archiving medium - the Internet itself! Plenty of cheap hosting plans around...
(1) It would help the humanity to filter out worthy things, because if your archive is not worthy you will stop paying for hosting one day.
(2) Hosting companies will take care of integrity of your data, because its their job.
(3) Your data will be publicly available and you will see if anyone in the world, besides you, is interested in it.
Huh? Why not?
--
int i = 0;
Why not archive on a high-quality film?
If documents are important to archive, especially for long periods of time, transferring the data to a less technical medium such as film is a much better alternative to the CD.
20 years ago, a great many people owned 8-Track tapes and players, along with record albums on vinyl. Very few of these items are readily available. I certainly know a few people with turn tables, but no one who owns an 8-Track player.
Whos is going to gurantee that the technology in use in 2103 can read a CD created in 2003? By storing data on film, even as a series of light/dark bits, requires very little technology for retrieval. Think about that, a lamp, a lense, and a wall to view an image. Data encoded as a string of bits could easily be read into a recording device.
Many types of film can be stored for much longe periods than CD's, and can be easily copied and in some cases restored.
Why does no one take a Tyrant like approach to this problem?
Many people have talked about older methods of storage as the gold standard. Paper, vellum, papryrus, clay tablets - some documents written on these media have survived thousands of years.
BUT, they have not survived that amount of time without degradation. The reason we can still read them is because of their low information density. Documents can fade - a 1 inch square portion of a document could flake away, leaving the original text still readable. Why? Because 1 square inch of most documents doesn't contain all that much information.
As physical objects I suspect that quality CDs or DVDs would degrade less over 1000 years than just about any of the other media I've previously mentionned. The problem is that we are trying to cram so much data onto them that even the slightest bit of degradation leads to data loss.
So what's the answer? Massive redundancy. Replicate data in 100 different ways across the surfaces of the CD or DVD - this might dramatically decrease the storage capacity, but even 10 MB on the surface area of a CD is a massive improvement over the storage density of vellum. Now you have a chance of lasting 1000 years. Even if the CD is shattered and all some future archeologist can find is a shard, there is a good chance that the entire data set is contained within that shard, perhaps even multiple copies.
Even further, one could imagine using file formats that are resistant to file errors, perhaps uncompressed raster images. Easy for future scientists to decode, and wonderfully resistant to degradation. This is just another way to decrease density.
-josh
I have to disagree.
This heavily depends on the "heterogenity" of your data.
For example in the pharma industry you have a vast amount of different lab equipment with every device writing a different (proprietary) data format.
That's where reverse engineering becomes practicaly unfeasible due to the huge amount of different formats. To make things even worse, retention times are very long (e.g. 30 years) due to regulatory demands.
So here concerns regarding this issue are equaly important as media lifespan.
Just use the Telcordia 1221 standards for passive photonic components for telecom uses.
When I worked at a fiber optic components company, anything that qualified 1221 would last more than 25 years in the field, and much, much longer in a controlled indoor environment.
For many of the devices I worked on, damp heat was the hardest test to pass.
It covers:
Mechanical shock (impact)
Variable frequency testing
Thermal Shock
High Temperature (dry heat)
High Temperature (damp heat)
Low temperature
Temperature cycling
Humidity cycling
ESD testing
Wow! I wonder if I can get one of these for my wine cellar.
"I'd rather be a lightning rod than a seismometer." -Ken Kesey
...that the commercial products actually meet the government specs?
We are already in a situation where every CD-R vendor claims to meet the industry specs, and every CD-R drive claims to meet the industry specs, yet it is not rare at all to find drives that like some brands of CD-R but not others.
It's not just a question of "using name brands" or "avoiding bargain brands," either.
When people raise this issue in e.g. comp.publish.cdrom.hardware, the answer is always "do your own media tests," and when someone complains that a specific name brand of media doesn't work in a specific drive, the answer is always "well, don't use that brand of media in that drive."
Obviously, vendors are NOT adhering to the specifications.
I don't see how promulgating a new set of specifications will change this.
We'll buy and write on these "archival" media, fifty years later wewon't be able to read them, and what will we do? Other than whine "but they were said they were archival?"
What we really need is for CDR drives to have a nice, continuous, real-time indicator that measures signal strength, or quality, or something like that... something that would give us an early warning that a disk, while still readable, was starting to fade.
If you have a problem with cars that run out of gas unexpectedly before reaching their destination, the solution is not cars with bigger gas tanks or cars that get better mileage. The solution is to equip them with gas gauges.
"How to Do Nothing," kids activities, back in print!
Not necessarily true. There were some horribly manufactured brands out there that did fail badly on this test. But that's about poor materials and poor quality control. Read the very end of the register article:
"Unfortunately, the article seems to focus on white label CD-Rs, and doesn't mention any premium brands that performed well."
So there is no indication where those higher end MAM-E, Mitsui, Taiyo-Yuden and Kodak Gold discs will fail like that. So keep in mind: you get what you pay for. Don't store anything you want to keep around for a long time on bargain basement media.
For a little while, yes. They're not made at all anymore.
What good is archival quality media, if you don't have the device to read it with?
Un-news
So, I've got lots of CD-Rs with photos archived on them, how do I tell which ones have been corrupted by data loss?
in a few years, those will be inexpensive enough to use for archival jobs, and they are a hell of a lot smaller and more robust than a CD is.
I am the Alpha and the Omega-3
If only CD and DVD media were enclosed like a minidisc or floppy disk, their life would increase substantially from the standpoint of the consumer (I don't know about the .gov). I manage to burn the same CD at least 2 times a year from wear and tear on the surface (despite my futile efforts to be careful with them).
[[ the only 15 letter word that is spelled without repeating a letter is uncopyrightable: it may soon be, however. ]]
Once I accumulated 5 years worth of backups, I copy the expiring data to new media and throw the old away, this gives me a little protection from 1) aging media and 2) aging i/o devices. I also make sure that as I get rid of older devices, that I convert all backups it might use before getting rid of the device.
Simple, but it works well for me.
Sorry but you're wrong.
Open standards are definitely the way to go, and thankfully we are seeing more use of them, but when it comes to 'mixed elements documents' or 'special use' applications then closed formats rule the roost.
Many apps claim to support the doc format but when you read the fine print it often says something like 'a subset of Word features'. This usually translates into 'basic' support of Word documents.
The same goes for documents created in Adobe applications.
If you've got just a tiff or just some text then you would not expect to have problems 30 years down the line but when you combine your tiff and your text the 'binding' app usually stamps its own format into the thing.
Canvas provides me with a great deal of flexibilty and I can output documents in a wide range of standard formats but doing so means I can never go back and modify those documents.
The solution must come from government and the solution must be real not just lip talk.
The problem of accessing important data over long periods of time just isn't taken seriously enough and is going to give a lot of folks a lot of problems not too far down the road.
Web standards are even guaranteed to be supported 30 years from now.
I think it's even more hideous with the advent of digital cameras. How many people in this world are keeping their children's memories on a disc because it's cheaper than standard film or cheaper than printing out the digital photos? How many people will find out in 10 years that they have zero pictures of the children as babies? As a parent, that's a a very scary thought.
With that said, I always thought DVD-R for Authoring were supposed to be the big bad media that was made for archiving data. Granted that's not a CD-R, but I was under the impression that at least some optical format existed for "consumer" use (it's pretty expensive for the drives right now). Anyone well versed with optical media care to comment?
Also, I have compact discs that are pushing 15 years old that play just fine in my car, computer and home CD player. Is it not possible to make this kind of durable media available to the public?
When I read the previous slashdot post about CDRs rapidly decaying (even becoming unreadable in a few years) I had trouble believing this... but I ran my own tests using the "Nero CD Speed" tool's ScanDisc option.
What I found was that my 3 to 4 year old archival CDs had anywhere between 20% to 50% of their surface damaged; they had (recoverable) errors. This spanned multiple brands, including Memorex, Acer, and HP.
Remember that data correction algorithms can recover from minor errors, but if data is becoming damaged this quickly it will be not too long before data is actually lost on CDRs.
I believe cd and dvd drives already have that at the scsi command set level. It just needs software to display the info.
Stone tablets last thousands of years, and it's off the shelf technology.
--- Ban humanity.
Ten years? Hell, I'd settle for DVDs that work out of the box! Am I the only one seeing more and more brand new movies that just QUIT? It's so damned frustrating that I've about decided to go back to VHS.
Try explaining to a kid why her Monsters, Inc. DVD quits every single time. Yes, we've taken it back to the store and got it replaced time and time again. Each DVD quits at a different, seemingly random, point.
The latest one is the new LOTR DVD. It quits and if we wait several minutes, it starts back, skipping over a minute or two of video.
It's not the player. These problem DVDs have been tried in four different brands of players.
Just damn!
Exactly.
Proper archiving is much like maintaining backups - it's not a passive thing - it's an active, ongoing process.
This space available.
Then there is the problem of uploading. It takes much less time to burn a cd than it does to upload a CD's worth of data, even on a DSL or Cable Modem connection.
So, your idea isn't the best for all cases.
On my current filesrever is my home directory.
It was copied from my old fileserver, and the one before that.
In there is a home dir from my Pentium 90.
In that dir is my home dir from my 386.
Inside that is the one from my Amiga 4000.
Inside that is the one from my Amiga 1200.
All the stuff I kept as text is perefctly readable.
Xix.
"Everything is adjustable, provided you have the right tools"
This problem sounds more like a "not carefully choosing effective standardized software" issue than an archival issue. Sure, scientists, engineers, whatever should be able to use whatever tools they need to get the job done. But part of the job is storing everything in a useful (and agreed upon) format that everyone can use.
.doc, would probably be the first mistake here, given MS track record of munging the format as time goes on. HTML x.x or post-script, even .pdf format (which still has some of Word's problems) would seem much better options for useful archival here.
MS Word
We know little of the great builders of European Cathedrals. Their work may have stood the test of time, as will GPL software model, but the debates, personalities and even the names of the people that built the largest and most sophisticated structures of their age are forever lost.
Pyramid builders of Egypt, carved their stories into the stone walls of buildings and tombs, (they painted and carved plaster as well, but that stuff is deteriorating) with possibly the oldest written language, hyroglyphs.
We have created and lost more data this century than in all recorded history. We need stable storage systems, or everything that we ever knew, will disappear, without nuclear war, without meteor strike, without thought, just neglect.
MacOS refugee, paper MCSE, linux wanna be!
The problem :
:
A paper based collection of 100,000+ maps dating back to 1886 that are slowly decaying with use. They require digitisation for long term ( 200 years +, at least as long again as they have survived already ) archiving but need to be available quickly and easily for viewing on demand. Total storage requirement is in excess of 150 Tb.
The Solution
An archiving setup using Magneto Optical technology in managed jukeboxes in a controlled environment. MO has been around for nearly 20 years now and is a highly refined and proven technology. Current capacities are up to 9.1 Gb per media item with 30Gb coming online in the near future. Jukeboxes handling up to 10Tb per unit are readily available now.
MO doesnt use dye at all. The laser melts a magnetic substrate that is then manipulated by the write head to impart the data in a similar way to a conventional disk, the sustrate cools and the data is permanantly stored. There is no degradation of the media by sunlight, heat etc as compared to DVD or CD formats. It's more accessable and requires less management than tape, its cheaper than conventional disk, off site storage of duplicate media is easily achieved, data throughputs are faster then DVD or CD and capacity is as good or better than either.
any moron would buy TWO devices, ie back up the backup device too stupid!
CDs are easy, as anything can read it now.
Liberty freedom are no1, not dicks in suits.
Ok, agreed, I missed multimedia. Actually what I mean is textual data, which is the most important part of our heritage. Knowledge mainly resides in texts, not on sound tracks or photos.
Multimedia can be somehat informative, but basically it's for emotions and curiosity. It'd be funny, for instance, to see digital photos of Rome back in II century, but it's only an illustration to the heritage of Rome. Still, knowledge and experience are in texts.
In fact, Internet is the archive of our knowledge (though not everything is free, but that's a technical question). And that's why generally I don't care about archiving, simply because I already have it - the Internet itself.
Good tips, I've recently been researching this issue and one huge thing I'd like to add that many people might not have though about is to NEVER EVER use sticky labels. The glue will break down over time and lose it's stick--which on something line photos might be a nuissance but on CDs it can be critical.
Vote Quimby.
Your media needs to be vaccuum sealed with a dessicant bag or two inside and placed in the deep freeze below 0 F. The cold (and darkness) helps preserve the polymers used in the cd's. You have to slowly warm them though when you need your info. I have worked in the coatings industry for 14 years and I havent' noticed any degradation. Heck even pigments don't degrade in the cold. White pigments won't even yellow. Our color standards are kept that way as well.
Is there any reason you couldn't design a "book" that would actaully be a machine-readable optically scanned card deck. You'd get the advantages of a durable paper stock, and the decks could be bound in such a way that they could be mechnically unbound, read, and returned in the way similar to a tape library.
I'm not sure what kind of data density you could get, though, although I suspect it would be slightly more than you might think. It creates a storage problem, but then it has great durability, and the machine to read it would arguably be easier to re-make in the future than the ones used to read traditional optical media, since you could include a card in each deck explaining in human terms how the deck is encoded.
In theory yes, and I would love the setup you described. But last year I had to agree on purchasing a DOS (!) based software because it was the only available device to get the job done. (Believe me, it _was_ the only device, I long enough tried to prove the opposite...)
So at the end of the day it's often: "here's our data, make sure that it's availbale for the next few years for reintegrating, recalculation, etc "
So the whole problem boils down to vendors not being able to standardize their data formats. Sounds familiar, doesn't it?
I think P2P would make the ultimate archival solution. You get several machines dedicated to storing your archival data in different places throughout the world, then share all the data from those machines with everyone else. With everyone duplicating data it will be more resistant to loss.
because in climates not arid desert lichens start eating the stone - go look at tombstones of 100 or 150 years... many are unreadable.
Or try maintaining an audit trail for examination within an application in a plain text file. It's not that simple.
The MAM-E (former Mitsui) Gold Archive and Verbatim Datalife Plus are the best CD/DVD blanks you can get in terms of longevity. In general, these brands along with TDK are what I recommend to folks almost exclusively because they have the best quality control. I've heard good things about Taiyo-Yuden and Mitsubishi's high-end blanks as well. Sadly, Kodak Gold blanks are no longer made, but if you can find some, go nuts on them.
Longevity is the last thing that people are thinking about. That's why crap blanks like Princo and Memorex are usually the best deal price-wise. Whether it's your music collection or home videos, you can't afford the cheap blanks. Of course, I also recommend many of the same things that the original poster (environmentally controlled environment isolated from light). However, using CDs with less storage space really doesn't make a big difference if you've temperature controlled them. The size of the "bits" is not going to be the deciding factor when there's already in-line error correction and redundancy. Just use good archival blanks.
One last note...for the truly paranoid, you should actually go and get a CD mastered by a manufacturing house that specializes in this type of thing. This may not be the cheapest option (probably on the order of hundreds of dollars), but what's your data really worth if you lose it and can't get it back?
My oldest CD-R is from 7 Oct 1996. I just copied its entire contents to my hard drive without encountering any errors, so it *appears* to be entirely readable. The CD-R is a Verbatim "DataLifePlus" CD-R, and it is dark blue. The contents were burnt at 1x speed. The contents are only 250 Mbytes, so it is not close to full.
For those who are interested CDRPlanet sells the Mitsui gold discs
These pits aren't likely to degrade for some time.
In contrast, any current CD-R/RW of which I am aware stores information photochemically; i.e., the information is stored in a dye that changes state due to the application of light from a laser beam.
These chemicals are likely to degrade much more quickly than are physical pits.
I don't believe that this situation will change any time soon.
Those who sacrifice security to condemn liberty deserve to repeat history or something. - Benjamin Santayana
In 2025 I suspect that some CS/IT professor will be delivering a technology history course entitled "The Dark Ages of Data".
...). ...) many criminal/news investigations were impossible and quickly dropped due to lack of evidence and/or no unencrypted information available, and just to expensive and time consuming to decrypt data (of questionable value) by code-cracking. Special interest dictating laws on technology and society rather then elected governments and technology professionals, all these things contributed to the decades of "The Dark Ages of Data".
Start comments, something like this: "The Dark Ages of Data" occurred due, we suspect [after review of data forensics and recent anthropology thesis/studies on available period information] to a total lack of prescience in industry and government on the impact of private special interest policies and laws (RIAA, DMCA, PA-1/2,
The proprietary OSD [Original Software Developers], Copyrights and Patent laws, competing formats (even today in 2025 there are old data file formats that cannot be opened without developing a special application) and storage methods, extremely poor and proprietary archival tools/applications/file-systems; also, a total disregard for adherence to reasonable standards.
Encryption technology and biometrics had no consistence in application across businesses, governments, and institutions. There was no legal requirement for an encryption master-key to be maintained by a corporate and government agencies to decrypt information of employees and agencies for providing task continuity (in case of dismissal, death, accident,
Now may be the best of times for corporate and government criminals, because they can destroy evidence with the click of a mouse button, and encrypt corporate-personal secrets for accomplices eyes-only.
OldHawk777
Reality is a self-induced hallucination.
Unaccountable leaders are masters, and unrepresented people are slaves. How do US and EU fare?
It would be nice to know how medium-scale brands (e.g., Phillips, Imation, etc.) fare.
I have heard some bad things about Memorex CD-RWs (and have experienced some problems with them myself), so it seems that name-brand does not necessarily imply good quality.
("Is it live or is it Memorex?" "It's dead.")
CD-R/RWs are cheap enough that I can make two copies, and I do, but I may have to start buying the more expensive stuff.
Those who sacrifice security to condemn liberty deserve to repeat history or something. - Benjamin Santayana
Paper logs are important too. You don't want to be in a position where you have to read 50 pieces of archive media to find one file from a particular date/study. Ideally you want to keep track of where things are with some sort of database, but what if your machine gets wiped out?
Print out some sort of documentation, keep it in an accessible place, so that if your electronic recordkeeping gets screwed up, you can still make sense of your archives without having to reindex the whole thing from scratch.
This sounds like just the ticket.
So where can I buy vellum CDs?
The idea that name brand does not imply quality in the CD-R market is quite correct. There are only something like 14 companies in the world that make CD-Rs but there are hundreds of brands out there that buy and then resell them under their own name.
So therefore it's important to know which company manufactured the disc as opposed to which is selling it. It's also important to know that some name brands buy from more than one manufacturer. For example, my 'archival' burning is done on FujiFilm discs made by Taiyo-Yuden, one of the best manufacturers. But I have to be careful because Fuji also sells Ricoh media. Typically the "Made in Japan" mark identifies Fujis that are from Taiyo-Yuden. This is how I find the quality.
So how do you tell if some disc on the shelf is good or not? Buy a single disc, bring it home and use an ATIP-reader utility to find the manufacturer name. If it's a good manufacturer (i.e. a manufacturer you have carefully researched and is known to make very good CD-Rs) then go back and buy a truckload. Otherwise keep searching.
Physical CDs (and probably DVDs, too lazy to look it up) are fundmentally different from CD-R media. A store-bought CD with permanent data is an aluminum foil that's had its tracks stamped in at the factory. A CD-R is a layer of phase-change material that will alter its transparency when hit by a write-mode laser to simulate the properties of the 'pits' of a pressed CD. If degradation causes the reflective properties to change, or if the phase-change material decays and loses its recorded state, you have errors.
Supposedly, CD-RWs are worse still due to the material they use. If data can be changed back and forth inside a drive, there's a potential for outside influences to corrupt both the opaque and reflective bits.
Still, I thought that barring scratches, a CD-R could be expected to last 5-8 years and a CDRW for 2 years or about 50 rewrites. I think the major lesson isn't they're necessarily unreliable, they just do better in stable environmental conditions
I use IMATION cdrs frequently, they are supposedly some of the best due to their special patented type of dye used, does anyone have more info on this brand?
NO vendor will provide any assurances on CD content beyond 12 MONTHS. This I know first hand from product meetings. My company has req's to keep stuff for 7 years, CD's will have to be recalled and reburned 1 every 12 months. We used to do the same with magnetic media but they would not provide ANY sort of assurances beyond manufacturer defect....
If it is worth backing up then you need to keep rewriting it until you don't need it anymore...Archfeld
errr....umm...*whooosh* *whoosh* Is this thing on ?
- CD-R: Organic dye is sandwiched between the polycarbonate substrate and the metalized reflective layer of the media. Data is put onto the disc by "melting" pits in the organic dye to give the simulated pits and lands.
- CD: The data layer is part of the polycarbonate substrate, and is pressed into the top side of it by a "stamper" during the injection moulding process. In this process there are actual pits and lands.
The problem is with the organic dye, since CD-R's are basically a sandwich of layers if there is any separation or contamination, the organic dye starts to break down causing data loss. Where as the data layer on CD's are physical, they have a higher tolerance for contamination.This is not the sig line you are looking for... -- Old Jedi Sig Line Trick
The silver starts to fade fairly quickly. When you want Archival Pictures, you print them darker than normal, then "Sepia Tone" them... replace the silver with selenium.
:-)
The Sepia's last a LOT better than the silver.
Plus -- they look lots neater
Yes, but this is mainly a problem because people have completely forgotten about the idea of quality... People hear that there are CDs that last for 100 years, then go out and get the pack of 100CDs for $5, and expect 100 years from them.
Meanwhile, there are several brands of CD-Rs that have been holding data for decades without errors.
Additionally, the lifetimes of CDs depend on how well you take care of them. If you have them lying around, with no cases, in the sun, in 100F degree temperatures, and high humidity, you can expect they aren't going to be around forever.
One more thing while I'm ranting... Most people don't understand that the metallic layer on top is where the data actually resides, and they don't think twice about touching the top, which actually causes damage after a short time. DVDs seem to commonly have the metallic layer between two layers of plastic, which should lessen the problem, but I'll bet there will be many cheap, single-layer DVDs, made just like CDs are, that will still experience the problem.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
I've had the same problem on several DVDs. I've never bought a DVD though, only rented them so I assumed it was because of past abuses of the disc.
So maybe the answer is to create consumer level devices that "stamp" media rather than "burning" discs. I wonder if the hinderance is in the technology or if there is some sort of political reason for not pushing the former method.
Like these DivX-inspired zombie critters from Disney, for example.
"Provided by the management for your protection."
What ever happened to this technology. From what I've seen, a reel could hold TB's of data. I see this being a perfect way to keep archived data online. However, the problem of longevity still exists. Oh well, just my $0.02
see: 35mm ICI-1012 TeraByte Reel Optical Tape
my other sig is a porsche.
My other sig is a Porsche.
Simple, print the key data on machine readable paper, it will last 500 years. Like a barcode primer?
MacOS refugee, Paper MCSE, Linux Wanna Be
Don't know if that's such a good idea...
I want my backups to last a billion years. The cool case is a plus as well.
Many people have already commented on this, so here is my two bits: I would like to use only a digital camera. I would like to scan all of my current photo colletcion. (& save a bunch of space) BUT, if I put these on CDs, how long are they going to last? Paper can last centuries. Other medium has lasted millenium. Whatever the solution is, it will have to be better than that, or I am not going to throw out those photos! We could have a lot of this 100 years from now: "hey kids, lets look at some pictures of when your great grandparents were born... oops looks like all those pictures are gone...."
At Disk Mfg., CDs are produced in a clean room, where a laser beam recorder and nickel-electroplating processes create a metal-stamping tool with the data encoded as tiny projections coming from the flat face of the tool. The polycarbonate CD is formed with this stamping tool on one side of the mold cavity, creating pits in the plastic disc. A thin, reflective coat is applied under a vacuum in an aluminum-sputtering machine and sealed with a UV-curable coating to prevent oxidation and scratching. At the end of the process, labels are screen or offset printed onto the discs. It is the addition of the reflective layer and dye coating that makes inspection especially difficult.
That info might be a bit dated, it came from a page talking about a company who produced Sega CD's as well. I think it gives you an idea on how tough it is to get a machine that creates normal cds.
Don't you think it's interesting that we're always finding ways to destroy things before we figure out how to preserve them?
DX-CDm Manual CD Destruction Device
My solution to this problem is to use distributed parity files. They work a bit like distributed parity in a RAID, but keep the parity data in files instead of outside the filesystem.
You can have up to 50% data loss with parity files, and still get your pr0... er, data back. I'm super paranoid so I sometimes keep two copies of the parity data as well, and distribute it as much as possible. In this respect, CDs are actually better than DVDs because they allow you to spread the data over more physical media for a given amount of data.
There are tools available for PAR 1.0 for just about every OS, but so far PAR 2.0 which is much better seems only to be supported by QuickPAR on Windows.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
A common misconception is that archives should be checked on an annual basis. Digital archives should actually be checked, and the data verified, at least monthly.
Where does one go?
(I have a couple of links to pages of test data, but they no longer work.)
And anyway, how can one be sure that the results on a particular web site are unbiased?
What I do now is make two copies of each archive disc on two different brands of CD-R/RW (in hopes that if one fails, the other won't), but if discs are rebranded, then I can't be sure that I'm not using discs from the same manufacturer (except that, since the two discs appear to have different sizes (e.g., 651 and 657 Mb), I can hope that they are).
Oh, thanks for the ATIP, uh, tip.
It's too bad that we have to do this kind of research.
The discs should just work.
Yeah, I know, that's naive.
Those who sacrifice security to condemn liberty deserve to repeat history or something. - Benjamin Santayana
Good lord, I've never heard of this rationale. I've knew brand-name stuff CDR media could suck, and largely kept to one brand I found rather reliable, but I didn't realize the market was that constrained. Thanks for the info and advice.
Do you have any advice regarding DVDR or RW media?
A common mistake is that archives (digital or otherwise) require no maintenance. In the case of digital archives you should be checking them, on an annual basis, not just for physical degradation.
That's why I'm really looking forward to archival quality CDR media. I'm tempted to use CDRW for long term storage because they are made better than their write-once cousins, but the fact that they are rewritable is holding me back for fear that that just might happen by accident, someday, for some reason.
A more common problem is that the applications used to create the data and/or their documentation do not exist any more, rendering the data as useless as if the physical media had been destroyed.
After several bad experiences, I don't trust any proprietary backup formats with my archives. I just burn files as-is to CDR and that's that. No sector-by-sector backups, no niche compression schemes, no special file systems to worry about.
True, so far. The problem is, CD is not likely to be the archival medium of choice in 5 years, let alone 50. If I had to guess... Terabyte optical flakes the size of a wheatie? Quantum pools in gallium borate thinbooks? Whatever it is, they'll still be storing FORTRAN programs in it.
``Tension, apprehension & dissension have begun!'' - Duffy Wyg&, in Alfred Bester's _The Demolished Man_
That is the matter of much debate and frankly I don't have a straight answer for you. Part of the problems is that CD-Rs are a relatively new technology and no disc has been around for 100 years so we don't know for sure how long they will last.
Lets just do the unthinkable and encode the data into a life form thats viable. Encrypt the data into the dna of a life-form you can raise multiple generations of. -OR- start at the molecular level and use a diamond or silica molecular database. (We'd have to make millions of copies to ensure only one copie advances to the future, but hey, we're talkin molecules here, how much space are you gonna need?) redundancy, redundancy, redundancy.
I guess it can be a death-spiral.
Not carefully choosing good software leads to bad vendors. No good vendors leads to not being able to carefully choose good software...
I find that in such cases, redifining, or properly defining for the first time "get the job done" is:
a) the only useful solution
b) works wonders if you can swing it
Basically, I can't fathom researchers who are allowed to precisely dictate what tools they will use. I would think the requisition process should always include alternatives to any single-vendor solution. ie: "What would we do if this vendor didn't exist?" type of questions. Sounds like you were looking into it. Fundamentally, any company must decide what is important.
If "getting the job done" for the moment, using the easiest tool available, becomes more important than long-term reliability, viability, and support costs... You get an industry that looks a lot like ours, with companies like Microsoft thriving. To me, Microsoft solutions often simply "look useful" at a cursory level, but later on important design flaws and methodology issues combine to create large bottle-necks and high expenses.
Note, this isn't always the case. The Windows 2000 line, and Microsoft SQL server have been coming along nicely. But what are the chances this can last without large price-hikes down the road?
Storing archived data in multiple locations seems like a no-brainer... However, keeping with the subject of CD-Rs, I've found that high burning speeds seem to accelerate aging greatly. This was especially true of the early CD burners that emerged on the market (maybe newer CD burners are better). Anyone else with this experience?
Mitsui makes a gold layer "archival" CD-R they claim will withstand 200 years simulated.
Well I've wrestled with reality for thirty five years doctor, and I'm happy to say I finally won out over it.