Slashdot Mirror
One Way To Save Digital Archives From File Corruption
storagedude points out this article about one of the perils of digital storage, the author of which "says massive digital archives are threatened by simple bit errors that can render whole files useless. The article notes that analog pictures and film can degrade and still be usable; why can't the same be true of digital files? The solution proposed by the author: two headers and error correction code (ECC) in every file."
If this type of thing is implemented at the file level every application is going to have to do its own thing. That means to many implementations most of which wont be very good or well tested. It also means applications developers will have to be busy slogging though error correction data in their files rather than the data they actually wanted to persist for their application. I think the article offers a number of good ideas but it would be better to do most of them at the filesystem and perhaps some at the storage layer.
Also if we can present the same logical file when read to the application even if every 9th byte is parity on the disk that is a plus because it means legacy apps can get the enhanced protection as well.
Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
>>>"...analog pictures and film can degrade and still be usable; why can't the same be true of digital files?"
The ear-eye-brain connection has ~500 million years of development, and has learned the ability to filter-out noise. If for example I'm listening to a radio, the hiss is mentally filtered-out, or if I'm watching a VHS tape that has wrinkles, my brain can focus on the undamaged areas. In contrast when a computer encounters noise or errors, it panics and says, "I give up," and the digital radio or digital television goes blank.
What we need is a smarter computer that says, "I don't know what this is supposed to be, but here's my best guess," and displays noise. Let the brain then takeover and mentally remove the noise from the audio or image.
"I disapprove of what you say, but I will defend to the death your right to say it." - historian Evelyn Beatrice Hall
The PNG image format divides the image data into "chunks", typically 8kbytes each, and each having a CRC checksum. You'd archive two copies of each image, presumably in two places and on different media. Years later you check both files for CRC errors. If there are just a few errors, probably they won't occur in the same chunk, so you can splice the good chunks from each stored file to create a new good file.
It is about time that somebody (hopefully some of the commercial vendors AND the open source community too) get wise to the problems of digital storage.
... I will have to ponder that. Maybe not, my programs seem to ephemeral for that ... Then again, so did people think about their 1960es COBOL programs.
I always create files with unique headers and consistent version numbering to allow for minor as well as major file format changes. For storage/exchange purposes, I make the format expandable where each subfield/record has an individual header with a field type and a length indicator. Each field is terminated with a unique marker (two NULL bytes) to make the format resilient to errors in the headers with possible resynchronisationthrough the markers. The format is in most situations backward compatible to a certain extent as an old program can always ignore fields/subfields it does not understand in a newer format file. If that is not an option, the major version number is incremented. This means that a version 2.11 program can read a version 2.34 file with only minor problems. It will not be able to write to that format, though. The same version 2.11 program would not be able to correctly read a version 3.01 file either.
I have not implemented ECC in the formats yet, but maybe the next time I do an overhaul
... Efficiency is the enemy of redundancy!
Old documents, saved in 'almost like ascii' is still 'readable'. I once salvaged a document from some obscure ancient word processor by opening it in a text editor. I also found some "images" (more like icons) on the same disk (a copy of a floppy), even these I could "read" (by changing the page width of my text editor to fit the width of the uncompressed image).
As long as the storage space keep growing...
I think the problem is more around silent (passive) data corruption and loss.
It does become an interesting exercise when you are dealing with "off-line" type media like tape, DVD and rocks though - the greater the data density the greater the impact of media damage (entropy).
So I guess there are two parts to this problem - how often to you validate your data, and how do you mitigate large scale errors. There are good solutions at the on-line media level (ZFS/RAID, etc..), but relatively weak at the offline level - anyone know of the equivalent RAID model for things like tape?
Cheers,
-I.
Parchive: Parity Archive Volume Set
It basically allows you to create an archive that's selectively larger, but contains an amount of parity such that you can have XX% corruption and still 'unzip.'
"The original idea behind this project was to provide a tool to apply the data-recovery capability concepts of RAID-like systems to the posting and recovery of multi-part archives on Usenet. We accomplished that goal." [http://parchive.sourceforge.net/]
KPH
Ten years ago my old company used to advocate that for individuals who wanted to convert paper to digital, they first put them on microfilm and then scan them. That way when their digital media got damaged or lost they could always recreate it. Film last for a long long time when stored correctly. Unfortunately that still seems the be the best advice, at least if you are starting from an analog original.