Any time you discover corruption from a particular media, I would quickly replace that device (and your multiple copy scenario should save you).
And here is the challenge. I have over 200K pictures spanning a time period of about 15 years.
It would be time prohibitive to manually go through them every few months, or even once a year to verify that all the originals are ok.
I've looked around for an open source program that could at least validate that jpg files are correctly formatted, no dice as of yet. But that will not validate that the content has not changed. By comparing the original "good" version with my oldest backup, I have a program that compares the two versions and removes the identical ones. Then its up to me the human to look at the remainder and determine are the remainder (a) pictures accidentally deleted, (b) "junk" photographs (bad exposure, my kids taking pictures of a wall, etc), or (c) a good photograph, and when I go back to the original I see that that one has been corrupted. If I get a lot of (c) then that is a warning that the original media is becoming suspect.
Let me try to explain this by example. Lets say my main copy is "A".
And that I alternate backups "B" and "C".
At the moment, "B" is the newer backup, "C" is the older one.
Before I delete the old copy "C", I generate MD5 hashes of all files of "C" and of "A". Then I delete all files in "C" that are present in "A".
Normally I will only see a few files in "C".
If my main drive "A" starts getting flakey I will see lots of files in "C".
If I see the same tree path, but different hashes in "A" and "C", I know that something has either modified "A" or "C".
Long term what I want to do with the ECC is to be able to periodically sweep through the directories and verify that all the files are correct and/or correctable.
Well, this is just my personal experience. I've been using digital cameras back to the mid 90's, and have about 200K photographs. Over this time I've seen at least 500 pictures ending up corrupted. Some of these were irreplacable, family members have died, newborn photographs, weddings, etc.
I suspect that sometime in the past one of my drives was starting to go bad, but not enough for the OS to notice. After a while the remaining good copies get overwritten with the bad ones. That is why I put in the compare before delete of the old backups, to make extra sure that I have a good copy of everything that I am deleting.
I create a new directory tree every three months because if I don't, and I loose a file due to corruption, I loose all copies of it as there is only one real copy that is hard linked everywhere. By establishing a new tree periodically I know that I have at least "N" independent copies.
ZFS is an interesting idea, I will look into it when I get ready to actually do it as I'd rather not reinvent the wheel if there is another that is good enough.
An interesting idea. What happens when you upgrade your camera and it uses different media? I have about a hundred digital 8 mm video tapes that I am in the process of copying over to my computer.
My old camera broke, I had to get a used one on ebay.
I'm copying over all the videos before the new one breaks.
My new camera uses SDHC, which will hopefully be around for a while.
What I am thinking of is to have the FUSE drive process run PAR2 to build the PAR2 files for every data file in each directory since PAR2 does not appear to handle recursion. Then, every time a file is modified in a directory, have FUSE create a "need to rerun par" file in that directory. Cron process will scan through FUSE directory tree looking for these files and regenerating the PAR2 files. Another cron process will go through all directories and do a "par2 verify" on all directories not checked in XX days. What I still need to check is to see if PAR2 can handle video files as some of mine are over 4 gig in size.
One more thing. When I expire old backups, I do md5 hashes of all the files, and delete the files that are copies of the current photo directory. I then remove all empty directories as well as.picasa.ini files.I then go through and check out files that have disappeared. Usually they are junk photos, but occasionally I find a good one that appears to have been mistakenly deleted. One annoyance I have found is that the Mac iPhoto program appears to add an Apple copyright mesage to all.jpg files, thus throwing off my MD5 checksums.
Weekly (or so) rsync backup to a Linux box into a date labaled directory, linking to the old one to minimize disk needs.
Every three months on the Linux box establish a new directory tree, not linking to the old one.
Roughly monthly: rsync to a portable usb drive. Rotate with another at work
When I visit my relatives who are a very long distance away, I rotate another usb drive with them.
This one gets updated every year or so.
Long term idea: use FUSE to create a directory structure where every directory has error correction run on the contents.
From the user view, just a normal directory
From the backup view, backup will back up date and ECC files.
In that way the backup can handle limited corruption.
Also this will help detect/correct another problem I have:
Over the years, I find that some files get corrupted and I don't notice it.
Trouble is, all the backups eventually get overwritten.
And when you literally have tens of thousands of pictures, its impossible to go through them all on a regular basis.
So what I hope to do with ECC over FUSE is to run periodic ECC validation scripts to detect and correct these problems.
PS: does anyone know of a utility that will go through and validate JPG or ELIF files? Or any video files?
Yes, gamma ray bursts were originally used by the military to detect nuclear weapons testing.
It was then they found out that gamma ray bursts came from space as well.
I was hoping to see an ironic loop in that gamma ray detectors set up to detect nuclear explosions on earth found gamma ray bursts in space. And that further study of the space phenomena led to discovery of phenomena here on earth.
What gave them the idea to look for these antimatter bursts?
Did some scientist theorize it was possible and ask them to look?
Or did the spacecraft start receiving bursts that they eventually tracked down to thunderstorms on earth?
"Increasing emphasis by schools on catching cheaters"? I can say for a fact that I was doing this back in the 80's with some coworkers. I remember how
difficult some of the classes were. Heck I changed majors because I could not handle differential equations.
But, some people always want the easy way out.
Cheaters hurt themselves, those around them, and eventually the value of the degree they are trying to obtain.
Agreed, there will always be false positives. An easy example is study groups. A group of people who study together will tend to have the same answers, right or wrong.
There are also false negatives.
You will not catch someone cheating if they get 99% of the test correct.
At best statistics point out who to keep an eye on.
It is much easier to bring someone up on cheating if you have a crib sheet, or you confiscate a cell phone with a texting history with the answers.
I did this type of work back when I was in college (the 1980's), let me share a bit about the process, or at least how we did it. What we did was count up the matching incorrect answers between people. For most people who entered in a number of incorrect answers, the same wrong answer count tended to be low.
For those who were copying answers, the matching wrong count was high. However, this by itself was not sufficient to accuse someone of cheating.
What we did was give the names to the exam proctors, who during the next exam would either ensure that these people where physically seperated, or who to keep
an eye on. Note this was in the 80's, before the common availability of wireless communication. But who to keep an eye on would still work.
Some observations from the work we did. One: if your going to cheat, cheat off a smart person (duh!). With fewer wrong answers, you blend more into the crowed.
Second: this technique would false positive on people who study together. That's why we never used it to accuse someone, but only who to seperate or keep an eye on. In addition, it would give no clue of who was cheating from whom. Three: to counteract this, some professors would give out several slightly different versions of exams. It would be very hard explaining how you got all the right answers for an exam you did not take, but all wrong for the one you did take.
AC and DC each have their problems. DC loss to AC during the Edison vs Tesla wars because the cables back then had too much resistence. AC also allows easy conversion to higher voltages to transmit the same power but with less current ( voltage drop is resistance times current, but power is voltage times current). AC has a problem of
capicitance when you get to really large power transfers (the article was talking about A HUNDRED GIGAWATT output). For that you can use superconductors to do the transfer. But superconductors have problems with AC (so I've been told).
Hm, they may be able to borrow a trick from a proposal I read from the nuclear industry. If you situate the plant near a body of water, use surplus power to generate hydrogen and oxygen gas from the water. Sell the liquid oxygen, and use the liquid hydrogen to cool superconducting cables. And at the other end of the transmission line, sell the liquid hydrogen.
My wife used to work at a telecom company that had many long haul lines in desert regions with nobody around for miles.
She says that both copper and fiber optic cables were regularily stolen
Guy shows object to a lady and asks "cigarrette ?". Lady looks it over and says "Yes, it is". I've actually used this one when a lady offered me some doughnuts asking "Doughnuts?". I looked them over and said "Yes, they are".
She looked at me funny and then giggled.
What's next? Are they going to mandate these devices on the doors of every house in the United States? That you would need to go through one to be approved to be outside? After all, if you are outside you could hurt someone.
This presumption of guilt and having to prove your innocence is getting crazy.
If I saw these in the 80's, they could not have been classified, so they were probably 10-20 years old. Sonar from space would be impossible, radar and thermal resolution would probably have been insufficient to differenciate.
But even if they could, a sub in for repairs would probably be about the same temperature as the water it was in.
RADAR signature from a hundred miles down in the 60-70's could be reasonably faked, assuming the US had sufficient resolution to begin with.
You put corner reflectors on the points that would be expected to send back a lot of signal, the prop and conning tower come to mind.
The strong reflections would push down the AGC on the radar, hiding the fact that the rest of the ship is not metal.
Hm, I can see a new weapon under development.
A submunition weapon that will unleash hundreds of steel darts and blanket the area.
Everything still intact after an hour gets a visit from a JDAM.
This is not new.
Back when I was in ROTC (the 1980's), I recall an article where photorecon people found out that they were duped.
They assumed that a set of nuclear subs were berthed for a long period of time for repairs.
A storm came through and bent one of the "submarines".
So the presumption was that the Soviets knew when our sats went overhead and between the times they set sail on one sub and inflated another in its place.
So the Soviets had a sub patroling somewhere unknown because we thought it was in for repairs.
Slashdot Mirror
Any time you discover corruption from a particular media, I would quickly replace that device (and your multiple copy scenario should save you).
And here is the challenge. I have over 200K pictures spanning a time period of about 15 years. It would be time prohibitive to manually go through them every few months, or even once a year to verify that all the originals are ok. I've looked around for an open source program that could at least validate that jpg files are correctly formatted, no dice as of yet. But that will not validate that the content has not changed. By comparing the original "good" version with my oldest backup, I have a program that compares the two versions and removes the identical ones. Then its up to me the human to look at the remainder and determine are the remainder (a) pictures accidentally deleted, (b) "junk" photographs (bad exposure, my kids taking pictures of a wall, etc), or (c) a good photograph, and when I go back to the original I see that that one has been corrupted. If I get a lot of (c) then that is a warning that the original media is becoming suspect.
Let me try to explain this by example. Lets say my main copy is "A". And that I alternate backups "B" and "C". At the moment, "B" is the newer backup, "C" is the older one. Before I delete the old copy "C", I generate MD5 hashes of all files of "C" and of "A". Then I delete all files in "C" that are present in "A". Normally I will only see a few files in "C". If my main drive "A" starts getting flakey I will see lots of files in "C". If I see the same tree path, but different hashes in "A" and "C", I know that something has either modified "A" or "C".
Long term what I want to do with the ECC is to be able to periodically sweep through the directories and verify that all the files are correct and/or correctable.
Well, this is just my personal experience. I've been using digital cameras back to the mid 90's, and have about 200K photographs. Over this time I've seen at least 500 pictures ending up corrupted. Some of these were irreplacable, family members have died, newborn photographs, weddings, etc. I suspect that sometime in the past one of my drives was starting to go bad, but not enough for the OS to notice. After a while the remaining good copies get overwritten with the bad ones. That is why I put in the compare before delete of the old backups, to make extra sure that I have a good copy of everything that I am deleting.
I create a new directory tree every three months because if I don't, and I loose a file due to corruption, I loose all copies of it as there is only one real copy that is hard linked everywhere. By establishing a new tree periodically I know that I have at least "N" independent copies.
ZFS is an interesting idea, I will look into it when I get ready to actually do it as I'd rather not reinvent the wheel if there is another that is good enough.
An interesting idea. What happens when you upgrade your camera and it uses different media? I have about a hundred digital 8 mm video tapes that I am in the process of copying over to my computer. My old camera broke, I had to get a used one on ebay. I'm copying over all the videos before the new one breaks. My new camera uses SDHC, which will hopefully be around for a while.
What I am thinking of is to have the FUSE drive process run PAR2 to build the PAR2 files for every data file in each directory since PAR2 does not appear to handle recursion. Then, every time a file is modified in a directory, have FUSE create a "need to rerun par" file in that directory. Cron process will scan through FUSE directory tree looking for these files and regenerating the PAR2 files. Another cron process will go through all directories and do a "par2 verify" on all directories not checked in XX days. What I still need to check is to see if PAR2 can handle video files as some of mine are over 4 gig in size.
One more thing. When I expire old backups, I do md5 hashes of all the files, and delete the files that are copies of the current photo directory. I then remove all empty directories as well as .picasa.ini files.I then go through and check out files that have disappeared. Usually they are junk photos, but occasionally I find a good one that appears to have been mistakenly deleted. One annoyance I have found is that the Mac iPhoto program appears to add an Apple copyright mesage to all .jpg files, thus throwing off my MD5 checksums.
Its a Mac Mini with two 1T disk drives. One for the pictures/movies, the other for the Time Machine backup software.
Long term idea: use FUSE to create a directory structure where every directory has error correction run on the contents. From the user view, just a normal directory From the backup view, backup will back up date and ECC files. In that way the backup can handle limited corruption. Also this will help detect/correct another problem I have: Over the years, I find that some files get corrupted and I don't notice it. Trouble is, all the backups eventually get overwritten. And when you literally have tens of thousands of pictures, its impossible to go through them all on a regular basis. So what I hope to do with ECC over FUSE is to run periodic ECC validation scripts to detect and correct these problems.
PS: does anyone know of a utility that will go through and validate JPG or ELIF files? Or any video files?
Do these bursts have anything to do with the recently (past 10 years) documented phenomena of lightning that goes from the cloud tops out into space?
Yes, gamma ray bursts were originally used by the military to detect nuclear weapons testing. It was then they found out that gamma ray bursts came from space as well. I was hoping to see an ironic loop in that gamma ray detectors set up to detect nuclear explosions on earth found gamma ray bursts in space. And that further study of the space phenomena led to discovery of phenomena here on earth.
What gave them the idea to look for these antimatter bursts? Did some scientist theorize it was possible and ask them to look? Or did the spacecraft start receiving bursts that they eventually tracked down to thunderstorms on earth?
Assuming you trust the Navajo.
"Increasing emphasis by schools on catching cheaters"? I can say for a fact that I was doing this back in the 80's with some coworkers. I remember how difficult some of the classes were. Heck I changed majors because I could not handle differential equations. But, some people always want the easy way out. Cheaters hurt themselves, those around them, and eventually the value of the degree they are trying to obtain.
Agreed, there will always be false positives. An easy example is study groups. A group of people who study together will tend to have the same answers, right or wrong. There are also false negatives. You will not catch someone cheating if they get 99% of the test correct. At best statistics point out who to keep an eye on. It is much easier to bring someone up on cheating if you have a crib sheet, or you confiscate a cell phone with a texting history with the answers.
I did this type of work back when I was in college (the 1980's), let me share a bit about the process, or at least how we did it. What we did was count up the matching incorrect answers between people. For most people who entered in a number of incorrect answers, the same wrong answer count tended to be low. For those who were copying answers, the matching wrong count was high. However, this by itself was not sufficient to accuse someone of cheating. What we did was give the names to the exam proctors, who during the next exam would either ensure that these people where physically seperated, or who to keep an eye on. Note this was in the 80's, before the common availability of wireless communication. But who to keep an eye on would still work.
Some observations from the work we did. One: if your going to cheat, cheat off a smart person (duh!). With fewer wrong answers, you blend more into the crowed. Second: this technique would false positive on people who study together. That's why we never used it to accuse someone, but only who to seperate or keep an eye on. In addition, it would give no clue of who was cheating from whom. Three: to counteract this, some professors would give out several slightly different versions of exams. It would be very hard explaining how you got all the right answers for an exam you did not take, but all wrong for the one you did take.
Lets hope the "environmentalists" don't get in the way like they did in California.
AC and DC each have their problems. DC loss to AC during the Edison vs Tesla wars because the cables back then had too much resistence. AC also allows easy conversion to higher voltages to transmit the same power but with less current ( voltage drop is resistance times current, but power is voltage times current). AC has a problem of capicitance when you get to really large power transfers (the article was talking about A HUNDRED GIGAWATT output). For that you can use superconductors to do the transfer. But superconductors have problems with AC (so I've been told).
Hm, they may be able to borrow a trick from a proposal I read from the nuclear industry. If you situate the plant near a body of water, use surplus power to generate hydrogen and oxygen gas from the water. Sell the liquid oxygen, and use the liquid hydrogen to cool superconducting cables. And at the other end of the transmission line, sell the liquid hydrogen.
My wife used to work at a telecom company that had many long haul lines in desert regions with nobody around for miles. She says that both copper and fiber optic cables were regularily stolen
Guy shows object to a lady and asks "cigarrette ?". Lady looks it over and says "Yes, it is". I've actually used this one when a lady offered me some doughnuts asking "Doughnuts?". I looked them over and said "Yes, they are". She looked at me funny and then giggled.
"Thanks, I just had it stuffed".
What's next? Are they going to mandate these devices on the doors of every house in the United States? That you would need to go through one to be approved to be outside? After all, if you are outside you could hurt someone. This presumption of guilt and having to prove your innocence is getting crazy.
If I saw these in the 80's, they could not have been classified, so they were probably 10-20 years old. Sonar from space would be impossible, radar and thermal resolution would probably have been insufficient to differenciate. But even if they could, a sub in for repairs would probably be about the same temperature as the water it was in. RADAR signature from a hundred miles down in the 60-70's could be reasonably faked, assuming the US had sufficient resolution to begin with. You put corner reflectors on the points that would be expected to send back a lot of signal, the prop and conning tower come to mind. The strong reflections would push down the AGC on the radar, hiding the fact that the rest of the ship is not metal.
Hm, I can see a new weapon under development. A submunition weapon that will unleash hundreds of steel darts and blanket the area. Everything still intact after an hour gets a visit from a JDAM.
This is not new. Back when I was in ROTC (the 1980's), I recall an article where photorecon people found out that they were duped. They assumed that a set of nuclear subs were berthed for a long period of time for repairs. A storm came through and bent one of the "submarines". So the presumption was that the Soviets knew when our sats went overhead and between the times they set sail on one sub and inflated another in its place. So the Soviets had a sub patroling somewhere unknown because we thought it was in for repairs.