Slashdot Mirror
Ask Slashdot: What's a Good Tool To Detect Corrupted Files?
Volanin writes "Currently I use a triple boot system on my Macbook, including MacOS Lion, Windows 7, and Ubuntu Precise (on which I spend the great majority of my time). To share files between these systems, I have created a huge HFS+ home partition (the MacOS native format, which can also be read in Linux, and in Windows with Paragon HFS). But last week, while working on Ubuntu, my battery ran out and the computer suddenly powered off. When I powered it on again, the filesystem integrity was OK (after a scandisk by MacOS), but a lot of my files' contents were silently corrupted (and my last backup was from August...). Mostly, these files are JPGs, MP3s, and MPG/MOV videos, with a few PDFs scattered around. I want to get rid of the corrupted files, since they waste space uselessly, but the only way I have to check for corruption is opening them up one by one. Is there a good set of tools to verify the integrity by filetype, so I can detect (and delete) my bad files?"
you seem to be surprisingly ok with the fact that your computer crashed and all your documents and media were corrupted, as was your backup. I would have been beside myself. Hulk smash! Please let us know what different set ups you're exploring to avoid this.
If the entire contents of the files are messed up, you could write a quick script that compares the output of file(1) to the file extension. I wouldn't call this high-fidelity - I'd recommend using this to generate a list you go through by hand - but it's at least a starting place.
"What's a Good Tool To Detect Corrupted Files?"
BSOD?
Try running "file" from a command line on a few files you know to be corrupt. If the file command tells you the same, you could run a quick bash script to loop through the files and spit out the names of the bad ones. This is all assuming you know what you are doing with shell scripting.
is urgency. Corrupted files have the ability to detect urgency and your discovery of them will come in a form compatible with the laws of Murphy.
That won't help detect corruption, only truncation of files. You would need an md5 or similar hash.
1. Compare to backup, files that match are ok.
2. AppleScript option others mentioned may help reduce it further.
3. Backup regularly, and verify your backup procedure.
4. Anything else will cost you consulting rates.
make imaginary.friends COUNT=100 VISIBLE=false
2000-2001 MAF-Soft http://www.maf-soft.de/
The version I have is v1.0.3.102
It can scan single mp3s and entire folders structures for defects and logs everything if you wish. It will give you a percentage of how good the file is.
Depending on the damage you may be able to fix headers and chop off corrupted tag info with something like a MP3Pro Trim v1.80.exe
Obama's legacy: (N)othing (S)ecure (A)nywhere and (T)error (S)imulation (A)dministration
Go nuts.
0 = 1 + e^(Alt something)
Suppose your volume is mounted under /mnt/a and your backup is mounted under /mnt/b. Something like this should work:
for f in $(find /mnt/a -mtime -2 | sed -r 's/(^/mnt/a/)(.*)/\2/') ; do
cmp /mnt/a/"$f" /mnt/b/"$f"
done
That should find all files which have been modified within the past 2 days and differ from your backup, which will help narrow down your search. I don't know of a tool that will address your specific question about testing for integrity for particular file formats. For specific file formats, you can automatic this, of course, like using ImageMagick for image files, but I don't know of a tool that will just do everything. It shouldn't be hard to write a script to look at the extension and the output of the "file" command and determine which tool to use to automatically check integrity for that specific file format.
or sha1sum if you prefer. Automate in cron against a list of knowns.
eg: /home/wilbur/Documents/* > /home/wilbur/Docs.md5 /home/wilbur/Docs.md5
$ md5sum
$ md5sum -c
Join the Slashcott! Feb 10 thru Feb 17!
Have some respect, the man just lost his entire porn stash.
Mod me down, my New Earth Global Warmingist friends!
If you're talking about recovery tools, you're already on the wrong track. A Time Capsule costs $300. How much is your data worth? How much are the tools going to cost to recover it? How much is your time worth? I'll bet that the sum of those last three things is a whole pile more than 300 bucks.
If I were you, the thing I'd buy right now is a good backup solution. Re: your existing data, take a full image of your hard disk and take your time recovering it.
Once you've new backup system, you can then sit there with a big smile on your face and comment smugly on all future /. posts about data loss.
Have I lost data? Hell yeah. And it will never happen again.
-bh
You can run jpeginfo -c. I have a script that runs against a directory and makes a list for when I do data recovery for all my friends who don't listen when I tell them their 10 year old laptop may be dying soon.
In the land of the blind, the one-eyed man is kinky.
And even though your last backup is from August, this will still constrain the number of files you potentially have to eyeball.
unix "file" is not the answer. For some formats it does as little as look at a couple header bytes. Its a great tool to guess a format. Its a terrible verifying parser and does nothing to verify content.
An example of what I'm getting at, with some made up details, unfortunately html is not like well formed xml and every viewer is different anyway so the best way to figure out if a html web page file format is corrupt is unfortunately to pull it up in firefox. This only detects corruption in the structure of the file, if the corruption is just a couple bits then you end up with problems like tQis where the only way to see the h got fouled up is to write more or less a IQ 100 artificial intelligence. All "file" is going to test is pretty much does the file begin with or contain a regex something like less-than html greater-than (getting past the filters).
For content you could F around with, for example, piping a mp3 file thru a decoder and then thru an averaging spectrum analyzer and see if there's anything overly unusual in the spectrum. Also some heuristics like is the file only 1 second long, then its F'ed up.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
CRC is faster than md5, and for random corruption just as effective.
Help I am stuck in a signature factory!
Author here:
> Last backup August.
Yes, that was silly of me.
> Thinks there is a way to detect generic file corruption
There is no way to detect generic file corruption. But there is a way to detect specific filetype corruption. For example, I already found mp3val, that is able to scan all my mp3 and check for file integrity, and even fix a few kinds of corruption (such as unmatching bytes in the header and sound chunks). Maybe with the right set of tools, I might also detect (or even fix) my corrupted pictures, movies and books as well.
If I clone myself, can I call it a thread?
If a girl winks to us, can I call it a race condition?
You need good filesystem, with embedded data checksum and self-healing using redundant copies. For Linux - btrfs is fine. For Mac OS X & Linux - ZFS.
:wq
Tech Tool Pro, over on the Mac side, has a "File Structures" check which looks at a lot of different structured file types to make sure that their internal format is valid.
It's already too late, but I keep important files with par2 files. That way, when there's like 5% corruption, I can still fix the file.
I do this with flac files and some datafiles.
Also make sure you keep backups going. I guess this was your warning. Everyone needs one.
Well, don't worry about that. We can get you back before you leave. (Dr. Who)
That seems very strange--the only files that should really be corrupted, unless something extremely rare and catastrophic happened, are the ones that were being written when power went out, or were cached. And even then, a flush usually flushes everything, or at least whole files at once, or areas of disk. Is the partition highly fragmented or something?
I know this doesn't do much for your question, but that kind of failure mode is almost exactly what filesystems do their damnedest to avoid. HFS+, being journaled, should be even more proof against, well, exactly what happened to you. Maybe the Linux driver is poor, but man, if you got silent data corruption on a multitude of files that weren't even being written, that's really bad and the driver should be classified "EXPERIMENTAL" at best, and certainly not compiled into distros' default kernels.
To answer your question, I don't have experience with any tools (I automate my backups, and any archival files go on a RAID volume that does a full integrity scan nightly), but once you find one, you should separate your files into two categories--"must be good", and "can be bad". The "must be good" files (serial #s, source code, etc.), you hand-check, so you know for certain that every one of them is good. It'll also motivate you to replace them now, instead of later when replacements will only get harder to come by. The "can be bad" files (music, pictures, etc.), you do the automated check on and then just delete as you run into ones that the check missed. This has the advantage of concentrating your effort into where it's useful. If you try to check all of your files, you'll just burn out before you finish. You may even want to do more advanced triaging, but you'll have to come up with the categories and criteria there. The main thing is, split this problem up.
<xml><I><am><so><damn>Web 2.0</damn></so></am></I></xml>
Consider the possibility that the backup already contains corrupted files. I once had defective RAM where only one bit flipped occasionally. The machine was quite stable, so the defect went undetected and over a couple of months it silently corrupted hundreds of files. Unless he finds out what caused the crash, he can't be sure that the backup is alright.
That won't help detect corruption, only truncation of files. You would need an md5 or similar hash.
md5 is (relatively) slow. a simple CRC-32 will only fail you for 1 in 2 ** 32 corruptions, and I suspect the guy doesn't even have 2 ** 16 files so the odds are CRC-32 is more than good enough and significantly faster.
Then again, he's probably going to be hard drive speed limited not CPU limited. Then again, no point wasting laptop battery on an overly complicated algorithm. CRC32 is gonna use at least 1/5th the CPU/wallclock time and/or battery of md5.
The tradeoff boils down to you can use md5 and burn at least 5 times more battery/heat/wall clock time (whatever is your limiting reagent) in exchange for (128-32) = 2 ** 96 times lower likelihood of mistake. The problem with accepting 2 ** 96 higher reliability is his dying hard drive probably cannot provide 2 ** 16 reliability so increasing the algorithm is a waste since it's already asymptotically limited.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
You can just run mencoder or ffmpeg on the mp3 and mov on all the files (with a small shell script, probably involving 'find' or similar), just tell it to write the output to /dev/null, that should go through those files as fast at they can be read from disk and abort with error on those that are broken.
For the jpgs, you could try something similar with imagemagick's 'convert', to convert them to whatever format to /dev/null, which also needs to read the whole file content and aborts if they're broken (one should hope).
Those converters are really fast, especially ffmpeg, so that should complete in a reasonable time.
Having dealt with file corruption first hand after bit rot on some older media, the best recommendation I have is to find a solution which will prevent future bit rot. I have been using Great Lakes SAN (http://glsan.com) , who bases their system on zfs. Their system using 256bit check sums on all files and can detect and correct any/all file issues which may occur. Furthermore, backups, off-site disaster recover + system monitoring are built into their system.
I'd be asking myself why lots of files became corrupted from one dodgy file system event. Assuming HFS works like file systems I'm more familiar with, it will allocate sequential blocks for files wherever it can. This means that a random filesystem splat is really unlikely to corrupt loads and loads of files. You might expect a file system corruption to cause a load of files to go missing (if a directory entry is corrupted) or corrupt a few files, but not put random errors into loads of files.
I'd check to see whether files I was writing now get corrupted too. It might be dodgy disk or RAM in your computer.
The above might be complete paranoia, but I'm a paranoid person when it comes to my data, and silent corruption is the absolute worst form of corruption.
For next time, store MD5SUM files so you can see what gets corrupted and what doesn't (that is what I do for my digital picture and video archive).
Every man for himself, all in favour say "I"
Well...
My first suspicion would be that the filesystem is messed up, not the actual files. Unless s/he had a lot of pending writes to all of these files, there is no reason that something should have actually overwritten or garbled them when the power shut down. Much more likely was an impending or in-progress write to the filesystem's tables, which has affected where it thinks all the files' pieces are stored. And if that is the case, date modified and size may be irrelevant because those are going to be reported by the filesystem.
Aside from trying to read back sector-by-sector data and assembling them, however, I don't know that there's a remedy.
I vote based on politicians' actions, unless contrary to my preconceptions. Often wrong, never uncertain. #iamthe99%
I've certainly seen corruption with XP crashes, not a big deal because I do backup. About the same with the other file systems. In this case he was using Mac OS 10.7 Lion, which is a mess, and two others accessing the same partition. Not surprised.
"The ability to delude yourself may be an important survival tool" - Jane Wagner -
Not the BSOD.
If the OP had used open source "tripwire" on known-good files in each filesystem on his Macbook, and saved the resultant data output to a USB thumbdrive formatted with FAT32, the OP would have had a good chance of determining all corrupted files. In this case, an ounce of prevention would have prevented several pounds of "cure".
Check out http://tripwire.org./
Author here:
Ok, I could deal with the loss of some unique videos and pictures from travels... but now that you mention the porn... *weep*
If I clone myself, can I call it a thread?
If a girl winks to us, can I call it a race condition?
An honest question :
I've had several crashes over the years with Windows XP but the files, data and system files were never corrupted. In linux it seems that file systems are not very resilient, and the least crash can corrupt your files. Is NTFS such a good well designed file system compared to linux file systems ?
Linux supports a wide array of filesystems. Which ones have you used? I have used ext3 and ext4 and have never run into file corruption problems. Both of those are journaling filesystems. Journaling filesystems helps prevent corruption in the even of power failure.
Beside the filesystem, one other possibility for corrupted files is a bad hard drive. I know someone who reinstalled Windows on his desktop on a regular basis because key files would go missing or get corrupted. I took a look at it and found out that he simply had a bad hard drive. After getting a new one, he didn't have anymore problems.
I'd recommend running a base OS and then run something like VMware workstation so that you run other OSes inside the main OS. One huge benefit is that you can have access to multiple OSes at the same time and you don't need to reboot into them either. With hypervisor technology getting common on desktop, there probably isn't any need to multi-boot unless you have a specific reason not to use virtualization.
In linux it seems that file systems are not very resilient, and the least crash can corrupt your files.
Is NTFS such a good well designed file system compared to linux file systems ?
I've never had corrupt files after a Unix crash; be it SunOS, Solaris, HP-UX, Linux or any of the other Unix variants I've used.
I've never had corrupt files after an XP crash, but I've often had scandisk delete files, including a multi-gigabyte game installer that I'd just downloaded before it crashed. It regularly deleted Firefox bookmarks before they switched from storing them in big HTML files.
The NTFS approach appears to be 'I'll guarantee file system consistency but won't guarantee any of your files are still there'. I'm sure you can find similar Linux filesystems, but the most common ones don't seem to have any problems.
These comments are full of 'helpful' suggestions to compare to backup or to md5's generated from the backups.
That makes no sense.
If he has a good set of backups JUST RESTORE THE BACKUPS to get known good files back. Why would you read every backup file and every current file, then compare them, then make a list of ones that don't match just to restore the backups. Restore them all. done.
- For the complete works of Shakespeare: cat
Perhaps but I agree with the first post. Going through and simply looking at all the JPEGs or MPEGs is probably the only way to tell if a file is corrupted (I wouldn't trust the CPU to do an accurate job). Also gives you a change to erase a lot of stuff you really don't need anymore. I dumped 300 gig off my drive simply by going through everything... took awhile but it was worthwhile to get rid of old shows/movies I'll likely never watch.
My AC stalker: " I personally agree with your posts most of the time, but that won't keep me from modding you troll"
zfs! Works great. Included with FreeBSD 9, amongst other OSs.
You might also enjoy John Siracusa's exhaustive review of filesystems on one of my favorite podcasts.
Spoon not. Fork, or fork not. There is no spoon.
The JSTOR/Harvard Object Validation Environment:
http://hul.harvard.edu/jhove/
It's specifically designed to first probabilistically identify files, then attempt to verify their format.
Disclaimer: I haven't worked on it directly, but I did spend a number in the digital preservation space, so I probably know some of the people who have contributed to it.
Not true. At all. On modern systems, MD5 is just as fast as CRC-32 because the disk is the bottleneck, not the CPU.
This reminds me of parity and ECC memory battles of decades past. OK, so it detects an error... Then what? Shut off the power? Not really sure what you'll be gaining. The sole example where it works is when you have the policy and budget to replace anything that takes an error. Useless for this situation.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Let me ask a stupid question since I've never run a battery out on a machine running Ubuntu. Why did this happen? Running OSX or Windows, the machine would have hibernated safely before the battery ran out. Does Ubuntu not do this and it just dies? Or is this something you configured to act this way? If it is default behavior in Ubuntu it is something they ought to fix.
Cast Detect Evil, Sense Motive, and Discern Lies on the potentially corrupted files.
mplayer can detect corrupted movie and audio files find . -name '*.mov' -exec mplayer -msglevel all=6 -speed 100.0 -framedrop -nogui -nolirc -cache 8192 -tskeepbroken -ao null -vo null {} \; | grep Warning! > $1.txt Change the *.mov as appropriate.
Alright now I'm afraid I can't help with your verify problem but I do have one piece of solid advice: get rid of Paragon HFS immediately!
It is a truly shoddy piece of software that as of version 9.0 has a terrible bug that will cause it to destroy HFS+ filesystems. Google "paragon hfs corruption" and you will see many many horror stories from people who just plugged a Mac OS X disk into a Windows machine w/ Paragon HFS and then discovered the entire filesystem was hosed. In my dual-boot win/mac setup I replaced my copy of MacDrive with a trial version of Paragon HFS 9.0 from their website and every single one of the six HFS+ disks I had connected internally were damaged. Disk Utility couldn't do a thing and I had to buy a program called Diskwarrior to even begin to recover data. I ended up losing two disks worth of files anyway.
http://www.mac-help.com/t12137-opened-hfs-drive-win7-paragon-hfs-now-wont-boot.html
http://www.wilderssecurity.com/showthread.php?t=299306
http://hardforum.com/showthread.php?t=1677099
http://www.avforums.com/forums/apple-mac/1509344-hfs-super-block-not-found.html
whew! Anyway the pain I went through after that software very nearly ruined my life was so great, I don't want it to happen to anyone else. According to their own website 9.0 has this awful bug but they fixed it in 9.0.1. Evidently the trial download on the main page is still for version 9.0 and still has the disk destroying bug! Any software company that releases a filesystem driver with this terrible a bug (not to mention the numerous reports of BSODs and other relatively minor problems) clearly has terrible quality assurance and simply can't be trusted.
Just have your OSX do a repair - it could be that certain VTOC or directory tables were damaged, and a repair may fix it. The files themselves should be OK, but the pointers to them are fubared.
Also try something like http://www.cgsecurity.org/wiki/PhotoRec or similar to recover deleted files. There's one for OSX. Run it after a repair, and photorec, and you should get most of your crap back.
Look, you're really taking the wrong approach here. The way to deal with corruption is avoidance, backup, and corrective action.
1) Avoidance. This is the generally the role of the filesystem and the underlying hardware, each of which have methods for preventing and correcting data corruption without ever involving the user. The user has a small part to play by doing things like shutting down instead of turning off whenever possible, though journaling filesystems (i.e., all modern filesystems) will know when a file operation was interrupted prematurely and check the integrity automatically. Also try not to put different file systems and OSes on the same drive, since there's the possibility that one OS may not respect the FS or limits of another (typically/historically, Windows has been the culprit here, but not always, and not so much anymore.) Any OS will generally leave an unrecognized drive alone unless you tell it to do otherwise, but the system drive has often been considered fair game.
2) Backups (optional). Once you have a known-good (or believed good) installation, create your backup. Repeat somewhere between often (if your data is important) and never (if it's not).
3) Correction. If and when you come across data corruption, that's not a sign that you're wasting space on your hard drive; it's a sign that something is seriously wrong. The proper course of action is to identify the underlying cause and correct it, not to delete the files to free up space. If you're experiencing corruption on only one drive regardless of channel and cable, replace the drive. If you're consistently having problems on a given channel, then don't use that channel. If you're having random issues across all drives on all channels, then the chipset is bad and the motherboard should be replaced. Basic troubleshooting.
Technically you *could* take a checksum of all of your files and update the database every time a file is changed. Some antivirus systems already do this to detect infections, but it would also detect incidental changes as well. The problem is that constantly verifying the integrity of your files will only hasten the demise of your storage medium. It's a self-fulfilling prophecy.
https://www.eff.org/https-everywhere
You clearly need an image based backup system to prevent this from happening again. It needs to be a chron job (or task scheduler) and run on regular intervals when storage is available. ideally, it needs to be network storage, so that a sudden disconnect (abscence of power) cannot easily corrupt the backup. There is an open source version of Ghost, partd, rsync.. options for you, though I am relatively new to linux so I don't know what the appropriate option is for you. Time machine you could use if you had a separate partition, but I think that isn't what you want. also, fundamentally writing to one partition from three OSes is asking for trouble.
Then what? Restore from last (good) backup, instead of propagating the corrupted file through the backup system until the good version is lost, surely?
Ubuntu pops up a warning window, and if you ignore it the battery light turns orange, and then red, and then it should hibernate. Flat-out dying is not something I've come across under Ubuntu (and I have some flaky old machines with old batteries, and they still warn me and then shut down).
The bad news is I don't know of any (and I don't think you'll find any) easy, one-shot tool to run across the whole lot that gives you a simple "corrupted yes/no?" answer to lots of different filetypes.
The good news is it'd be reasonably easy to lash together something in bash, kick it off overnight and come back in the morning to a list of probably-corrupted files.
In pseudo-bash (because I haven't the time to write it out and check it works properly), something like this would be a good start:
function checkJpeg {
jpeginfo -c $1 || return 1
return 0
}
function checkPdf {
# do something to check a PDF is OK
}
FILETYPE=`file $1` ;; ;;
case $FILETYPE in
"jpeg" )
checkJpeg $1 || echo $1
"PDF )
checkPdf $1 || echo $1
esac
Then run it with the help of find /home -type f -print0 to check every file in /home. This would give you a list of potentially-corrupted files. Up to you how you deal with it - personally I wouldn't run rm against it in case you find files that can be rescued or that your checks aren't as perfect as you'd like.
For extra credit, determine the expected filetype based on file extension and then use file(1) as your first "is it corrupted?" test - that way you'll spot files that are too corrupted for file(1) to work reliably.
That is a good thought, and photorec does an excellent job of finding pictures and videos by searching through your sectors - definitely worth a try.
http://www.cgsecurity.org/wiki/PhotoRec_Step_By_Step
Every man for himself, all in favour say "I"
Ok, forgetting that ECC also corrects random errors that happen on functional hardware... WTF? Of course detecting problems is only usefull if you have the 'policy' of correcting them somehow.
Rethinking email
http://en.wikipedia.org/wiki/Comparison_of_file_verification_software
md5sum is the one I know best, but that's because my computing is unix-centric.
The real reason and it was stated in the summary is that the file system was HFS+ which is far less tolerate to this behavior than ext4.
That's a pretty good idea, if you only want to detect corrupted files (and yes, I know that's what the OP said he wanted), but I can't believe no one's suggested par2, yet. It will not only detect corrupted files, but repair them, too. If he had used par2, he wouldn't have to delete them.
Sit, Ubuntu, sit. Good dog.
I have used pc inspector file recovery with mixed results. It's slow but I have recovered pictures and documents. There's not much you can do if the file has already been stepped on.
We are eternal, all this pain is an illusion.
mplayer can detect corrupted movie and audio files find . -name '*.mov' -exec mplayer -msglevel all=6 -speed 100.0 -framedrop -nogui -nolirc -cache 8192 -tskeepbroken -ao null -vo null {} \; | grep Warning! > $1.txt Change the *.mov as appropriate.
<infomercial>its JUST. THAT. EASY folks!</infomercial>
... yes, this is not what you want to hear at this point, but try to have a positive take on this.
Last year during a routing Windows7 installation, my second hard drive from which I double boot my 90%-of-the-time-in-use Linux was destroyed. Either a coincidence that it occurred during the win7 installation or a nefarious plot, but the hard disk, a 1TB Seageate sata, developed an unrecoverable click of death.
On that hard drive I had my short stories which I had written in college and the intervening years since then, much of my photos, skype history and many other things, seemingly important to me at the time of the "disaster". I was inconsolable for a few days, and felt like I had been bereft of someone very dear to me. Then it hit me -- to hell with the stories, to hell with the photos, to hell with the rest of the digital baggage I had accumulated. I could write my stories again, and do it better, I could take more photos, I could hoard more useless junk. After a month I no longer missed any of the lost stuff.
Learn to view such mishaps more philosophically and learn to shed all the useless garbage you accumulate through the years; realize that almost nothing that you can store on your computer, or up in your attic, has really all that sentimental value you attach to it. Learn what's important, intrinsically important, to you and safeguard that. All the rest, you'll be amazed how little you need it and how even less you'll miss it.
To hell with useless stuff.
Most end users don't have that policy. Is it running right now? Well wait until it breaks completely and is no longer usable in any form.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Seconding the photorec / testdisk suite, they are incredible. I would rate it up with ddrescue as the top 2 data recovery tools.
I used to do that, but found it to be pointless these days. Organizing the stuff is one thing, but deleting is basically pointless unless you can automate it. 300GB may seem like a job well done, but with 3TB drives for $100 these days, you just saved yourself $10 worth of harddrive space and it probably took you a few hours.
My current setup is to have everything on my server box and simply copy over what I need to my laptop as I need it and NFS/SSHFS the rest of it on the fly when home.
It's the HFS+ partition that died, not the Linux one. Mac OS is certainly not a "freetard" OS.
-- Linux user #369862
Lacking not only a backup but also PAR(2) and MD5 files, manual inspection of each and every file is the ONLY way you can determine their integrity. There is no automagic after-the-fact integrity check. If you had MD5 sums for every file, you could at least check their integrity. Some PAR2 files would not only verify but possibly repair if the damage wasn't more extensive than the PAR recovery blocks. Of course if you're willing and able to do all that, you'd probably have had full and differential backups first.
(And yes, the subject was a lyrics reference.)
The Linux HFS+ driver can't even work in write mode unless the journal has been deleted, so the journal isn't working when using the HFS+ partition under Ubuntu and probably Windows as well (author take note). I would not use that filesystem under Linux or Windows on a daily basis. Also, since the journal has been deleted, you are probably missing the safety of journaling under the native OSX as well.
Author should also note that archival backups with md5 or sha256 checksums are probably the most straightforward way to maintain data integrity. If you want something more elegant for day to day use, I would consider setting up a NAS using either BTRFS or ZFS as the filesystem along with a nice 1Gbps LAN (if you don't have that already).
I hate your brace style anyway...
That's a preemptive strategy, though. No help at all if you only think to use it after your kid brother decides it would be fun to slap his Magnet Balls all over your computer case.
AIDE (Advanced Intrusion Detection Environment)
Maybe you didn't mean it this way, but dang if I did not see all the PHBs come out from work with your comment. "I can get 1TB Drives from Fryes for $80.00, why do you say it costs several hundred?".
Oh, you wanted redundant drives to be covered in the event of a failure? You wanted a drive that has some performance so it does not take 32 minutes to open your word file? So much for that 1TB for 80 bucks thing...
The new one is "SSDs are only $150.00, and they are the same as what you get for SSDs without all the writes.
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
George is your best bet. He's not bright enough for most support tasks, but he can certainly handle this one.
My files go through descending levels of staleness. First I copy them off the primary drive to a network drive. They sit there for a while. When a couple years have passed without looking at them, they go on an offline drive and I create a file listing which I keep on the network drive. Once a decade has passed without loading the offline drive, they go in the trash.
Files that still have value get caught in the sweep but then migrate back to the primary drive as needed. Saves having to scrutinize everything before hand.
For the OP: restore from your august backup. Add anything on the drive with a more recent mod time. And then deal with the corrupt files as you come across them. If you come across them.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
Indeed, I used photorec/testdisk to recover mp4 files after they had (all) been accidentally deleted from an HFS+ partition.
But when I first started it in it's default mode, it "found" only rubbish, breaking up the actual mp4s into a mess of .doc, xml, jpg, .whatever files, including totally broken .mp4s.
When I restarted it after configuring it to only look for .mov/.mp4, it did a fantastic job, and as far as I know, all files could be recovered. Of course, that was made easier by the fact that I knew that all the files which needed to be recovered were .mp4.
This guy is using jpeginfo for jpegs http://watson-net.com/blog/checking-the-integrity-of-all-jpg-files-in-a-directory/
I think this is the root of the problem here, he chose the wrong filesystem to share between the three OSes. Sadly there are not too many choices. FAT32 is the only one natively supported by all three, with its well known limitations. He might have been better with NTFS though, using NTFS-3G on Linux and OS X, but that has some performance hit. There's really no perfect solution for this kind of problem.
.sig: No such file or directory
Well, jpeg files have a structure that will generate detectable errors if it's damaged. So simply opening them with something as simple as djpeg from the IJG and piping the output to /dev/null should give you a pretty good start on damaged images. Something like this perhaps:
find . -name "*jpg" -o -name "*jpeg" -o -name "*JPG" -o -name "*JPEG" | while read filename; do if djpeg "$filename" > /dev/null 2> then :; else echo "$filename" is toast; fi; done
You could probably do something similar with mpg123 and mplayer for .mp3 and movies.
If you rarely use Windows, you could partition your disk, and format the Windows part to FAT. Then, on the odd occasion where you want to transfer files, you could just mount its partition in Linux (that's what I do), or OS X. I like the idea of sandboxing Windows, and not letting it touch/corrupt my main (ext4) file system.
HFS+ with journaling enabled is as solid as EXT. Journaling is on by default if the disk was formatted in the past several years. I'm curious to know if he had it turned off.
>>>you just saved yourself $10 worth of harddrive space and [enjoyed watching a ton of TV shows/movies during the process]
Fixed that for you.
My AC stalker: " I personally agree with your posts most of the time, but that won't keep me from modding you troll"
There ought to be an &1 after the 2>.
First, let's presume you're running Linux for what follows.
/foo/bar -name "*.jpg" -print | sort -u > /tmp/files.jpg
/foo/bar for all files suffixed ".jpg" and dump a sorted list of them into /tmp/files.jpg and this one:
/foo/bar -type f -print | sort -u > /tmp/files.all
/tmp/files.all.
(Note that the method by which find traverses filesystem trees won't yield sorted output, hence the
need to pipe these through sort.)
/tmp/files.jpg /tmp/files.all > /tmp/files.all2l
.gif, .mpg,
etc., as you deal with each file type and reduce the remaining list to those awaiting
your attention. /tmp/files.all3, /tmp/files.all4, etc. will each be smaller and eventually,
if you deal with all files, /tmp/files.allX will be zero-length. Note that not all files
have suffixes, of course -- and those without will likely be the ones requiring the most
manual effort. If you want to know which suffixes are most numerous, something like /tmp/files.all | sort | uniq -c | sort -n
1. You're going to want to be familiar with both file(1) and find(1). File(1) is pretty straightforward, but be aware that its heuristics for file type detection vary in accuracy. If you're not find-literate, then at least get used to this construct:
find
which will recursively search directory
find
which will search the same directory, but will return a list of all (plain) files, that is, things which are not directories, devices, sockets, etc., sorted and dumped into file
2. You now have (a) a list of all jpg files and (b) a list of all files. (I picked jpg arbitrarily to illustrate the process, by the way.) You can now generate a list of all files that are NOT jpg with this:
comm -13
The point of this exercise is that you can now repeat steps 1-2 with
sed -e "s/.*\.//"
will give you a rough idea.
3. Now then...you'll need some tools for dealing with each file type. The first tool I'd use is stat(1), to check sizes for plausability. Then things like jpeginfo(1), mp3val(1), tidy(1), will be some help, but of course you'll need to distinguish between "error message emitted because file is corrupt" and "error message emitted because file has minor issues...that it had BEFORE this episode". You may need to check the Ubuntu repository for tools you don't have; you may need to do some searching on the web for "Linux tool to check PDF integrity) and similar.
4. If you have backups of any kind and can restore them, then you could try using sum(1) to compare checksums pre- and post-incident. This is a filetype-invariant method, which is good because it lets you skip the above...but bad because all it wll tell you is "different", not "mildly damaged" or "horribly corrupted" or something in between.
5. I would recommend against deleting anything at this point. Instead, move it to secondary storage, like an external drive. I don't have a specific reason for advising this, other than "many years of experience doing partially-manual, partially-automated things like this and a recognition that sometimes errors in the methodology...or fatigue introduced by the tedium of executing it...lead to mistakes".
6. Good luck.
The identify program is a member of the ImageMagick(1) suite of tools. It describes the format and characteristics of one or more image files. It also reports if an image is incomplete or corrupt.
Last I checked which was about two months ago HFS+ with journaling enabled does not work reliably at all on Linux. I ran into this problem with trying to create a Drobo that everyone could share. The reality is that the only option would have been was HFS+ with journaling disabled in order to have all three. NTFS-3g from my experience works great on Linux and like crap on OS X nevermind the fact that you wouldn't even be able to install OS X on an NTFS partition.
Files can be corrupted by rare spontaneous bit flipping, by mis-writing a block that was intended for another file or corrupting the block list to include data from another file (cross-linked files), by including blocks that don't exist, or by including blocks that have no data or arbitrary data.
Headers or meta data in some file formats can be verified by applications that support that file format, but it's possible for some of those problems to change the file's data such that the data is still valid, but wrong. If you have a large collection of media files or image files, filesystem corruption could potentially cross-link valid data from another file of the same type.
All of that is to say that the only way you can reliably detect corrupt files is to compare them to files that are known good. To anyone with backups in your position, I would simply say that the best option would be to wipe the system and restore a backup that you trust. If you had rsnapshot backups, you might be able to: /backup/ /filesystem/
rsync -avcn
rsync would then tell you which files differed from backup.
According to comment 39919031, Paragon HFS may have serious bugs. It's possible that the problem didn't actually come from the power loss, but from a bad filesystem driver. I'd recommend using something better supported by all systems for your shared space, or using hardware assited virtualization for all but one of the operating systems. On my own hardware, I run Linux with other systems in KVM guests, which works well. The host OS can export shared space over the network (NFS or CIFS) to the guests, which is probably the most stable filesystem configuration possible.
Consider the possibility that the backup already contains corrupted files
In which case, it's not a backup - it's just a waste of storage space.
Not everything that can be measured matters; Not everything that matters can be measured.
That seems not worth it. The thing is, both drive-space and data-volume tends to double every ~18 months or so. You wait first "a couple of years", then on a network drive, then once a decade has passed, they go in the trash.
But a decade ago the cheapest storage was a 40GB drive costing $130 or thereabouts. Today 40GB worth of space is 1.5% of that shiny new 3TB-disk costing $150 or thereabouts.
There's essentially no benefit to deleting old data, because old data is *always* small data, and so copying it to the new disk will use a miniscule portion of the new disc and have essentially no cost. $150/3TB is equivalent to $2 for saving those 40GB.
The only data that's potentially worthwhile to delete is *new* data that you have no need for. There is no such thing as "old but large data".
Avoiding clutter is a different issue, but that's easily solved by copying all the old data to a named folder, then move out of that folder and into the current file-system only those files you actually use.
One way to check JPEG/PNG/GIF files that may be corrupted is browsing the folder with thumbnails on. The files that doesn't show the thumbnail are corrupted.
Simple enough, package up the 'known good' files on your server into a torrent, give torrent file to laptop, have it use bit torrents built in hash checking to verify and, if bad, replace the damaged parts.
...
Silent file corruption is the reason why I now keep all my data on a ZFS filesystem. ZFS has a checksum for every block and if you have redundancy at all (raidz, raidz2, or even just tell ZFS to keep two copies of each file) then it will repair the corruption as well as detect it. I've got a HP Microserver running Solaris but I recommend running FreeNAS instead if you don't know ZFS. This blog is a good place to learn about ZFS.
A latent existence
I don't like losing data so it sits in a raid and gets backed up. This means I'm not just keeping one copy of primary data, I'm keeping many. And reprocessing/recopying it each time I make a backup.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
It is rather unlikely that you have a massive corruption of actual data in multiple files--for that, you'd need a sustained write activity hitting all over the disk. Possible, but not very probable. Instead, I think you have metadata corruption, so that the filesystem points to wrong blocks. The glimmer of hope here is that the actual file data is mostly contiguous, so that you can scan through the image and identify individual files even without the filesystem information. There is a forensic tool called 'foremost' that does exactly that: rips through the binary filesystem image and as it finds headers of known data types (jpeg, gif, doc, mp3, etc. etc.), it tries to find as much following data as is consistent with known file layouts. The result of course has tons of cryptically named files of each type foremost knows about---not all of them are legitimate, even---but it's better than nothing.
but with 3TB drives for $100 these days
Where are you buying your drives? I can't find them for less than about $185...
Karma: Poor (Mostly affected by lame karma-joke sigs)
3TB HDD is less than $150 at newegg. It's faster than or as fast as any 7200RPM drive from previous years as well, even if it's a 5900RPM drive, because technology just gets better. I'm not even sure what you are going on about with SSD.
Are you not American? newegg has them for less than $150.
Since your major partition is HFS+, I recommend DiskWarrior (http://www.alsoft.com/diskwarrior/). I've used it professionally for over ten years, and it still does the best job at finding, fixing, and reporting corrupted files on HFS disks.
Today with Mercurial/Git you can easily create a local repository that contains your text file listing and checkin changes. Your 'hg log' will show all differences over time, which could be useful.
I wrote a disk cataloger years ago and was surprised at the number of CRC32 collisions I found within my own set of files (on CDs at the time - I have a lot more data today). I recall that pairing CRC32 with file size helped a lot (and probably even solved the problem in my case).
I've got a laptop running exclusively Ubuntu and the default behavior when the battery gets to 5% is to just hibernate.
As for a validation of files, this is a truly trivial problem to solve...
Yes Francis, the world has gone crazy.
identification puts much higher requirements on a hash algorithm than corruption detection.
Lets connsider corruption detection.
Say you have c corruputed files.
Now suppose our hash has m possibilities all equally likely (the ideal case for a hash function).
The average number of files that will pass the hash test despite being corrupt is c/m
Now lets consider identification
Say you have n different files, the number of possible pairs of files is ((n^2)-n)/2 . For large n we can approximate this as n^2/2 .
Now suppose our hash has m possibilities all equally likely (the ideal case for a hash function). The probability of any pair files having colliding hashes colliding is 1/m
So the average number of collisions is approximately (n^2)/(2h).
Do the sums and you find you need a much larger hash to get acceptable performance in an identification application than in a corruption detection application.
Note: all the above assumes no malice is involved. If malice is involved then the requirements on the hash get much tighter.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
The real trick is to combine checksuming with mirroring or reconstructive parity.
If you only have mirroring/reconstructive parity you can reconstruct the data if you know which block is bad
If you only have checksumming you can detect which block is bad but you can't do anything about it.
If you have BOTH mirroring/reconstructive parity and things are set up properly so they work together*
Afaict this was/is the killer feature of zfs, I think btrfs now has it too but i'm not positive (though the impression i've got is that btrfs is still too unstable generally to be used as a main fs).
*Having a raid system below a checksumming system doesn't work for this because the raid system may overwrite the good data with the bad during a resync. Having a checksumming system below a raid system would work but runs into the problem that checksumming is difficult to do efficiently below the filesystem layer (because you need to store the checksums somewhere) so the only real way to do it efficiently and correctly is to integrate both checksumming and redundancy into the filesystem.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
If you have BOTH mirroring/reconstructive parity and things are set up properly so they work together*
oops, didn't complete that bit, it should have said
If you have BOTH checksumming and mirroring/reconstructive parity and things are set up properly so they work together* then you can detect which block is bad and then take action to recover the data.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
This happened to me once. At first I thought it was a hard drive problem, it took me a while to figure it out. I lost many pictures in the process, and had rsync'd corrupted files over my backup. To me the problem now is: how to make sure what I'm backing up is not corrupted and that my previous backup, about to be overwritten, is not cleaner than my source?
~~~ Paf. Le chien.
I am, and that was the first place I looked...
Karma: Poor (Mostly affected by lame karma-joke sigs)
Maybe you didn't mean it this way, but dang if I did not see all the PHBs come out from work with your comment. "I can get 1TB Drives from Fryes for $80.00, why do you say it costs several hundred?".
Are you one the the exchange admins that claims that 10,000 emails totalling 500MB is too much to have in your inbox, despite gmail AND hotmail not even sniffing at it (and providing rich-text cross-platfordm email, instant search)?
For enterprise storage I tend to aim high at a capital cost of $500/TB, and running costs of $50/TB/year.
That means purging 300GB saves $20 a year. If it only takes a day that's fast. It also costs the company $800 in lost time. Whoop.
Ubuntu pops up a warning window, and if you ignore it the battery light turns orange, and then red, and then it should hibernate. Flat-out dying is not something I've come across under Ubuntu (and I have some flaky old machines with old batteries, and they still warn me and then shut down).
I have. I have flakey batteries that report 0% when they've still got another 20 minutes or so left. When the warning pops up I often kill gnome-power-manager, save my work, and take my changes.
Sometimes I don't get to power in time.
I also have a problem if I knock my second battery out the CD bay (thinkpad). The machine doesn't like that when on battery power, and just turns off.
mplayer can detect corrupted movie and audio files
find . -name '*.mov' -exec mplayer -msglevel all=6 -speed 100.0 -framedrop -nogui -nolirc -cache 8192 -tskeepbroken -ao null -vo null {} \; | grep Warning! > $1.txt
Change the *.mov as appropriate.
<infomercial>its JUST. THAT. EASY folks!</infomercial>
Yes it is, as the OP was kind enough to tell you advanced mplayer use.
Compare this cut-and-paste job to using "simple" tools.
Click Start
Click Find
type *.mov
wait for the find to finish
double click the top file
watch it
if it breaks, delete the file
double click the next file
continue for all files
The CLI makes complex tasks easy and fast. When someone is kind enough to give you a copy and paste line it's even easier and faster.
How many copies you keep of the data is entirely beside the point.
The point is that a disk full of data from a decade back, is only going to fill 1.5% of a new disc.
Yes, if you keep 2 separate backups plus your primary storage, you're going to need *3* new discs, not one. But the presence of the old data doesn't change this noticeably.
My point is "All my files from the last decade" are always going to be ~2 order of magnitude larger than "all my files from before that", thus deleting the latter is never going to save you more than a tiny percentage.
Even if you keep 100 separate backups, deleting all the decade-old files will *still* only save you 1-2% of your total storage-needs.
Send your drive to the FBI, they will scan it for you.