Slashdot Mirror
Ask Slashdot: What's a Good Tool To Detect Corrupted Files?
Volanin writes "Currently I use a triple boot system on my Macbook, including MacOS Lion, Windows 7, and Ubuntu Precise (on which I spend the great majority of my time). To share files between these systems, I have created a huge HFS+ home partition (the MacOS native format, which can also be read in Linux, and in Windows with Paragon HFS). But last week, while working on Ubuntu, my battery ran out and the computer suddenly powered off. When I powered it on again, the filesystem integrity was OK (after a scandisk by MacOS), but a lot of my files' contents were silently corrupted (and my last backup was from August...). Mostly, these files are JPGs, MP3s, and MPG/MOV videos, with a few PDFs scattered around. I want to get rid of the corrupted files, since they waste space uselessly, but the only way I have to check for corruption is opening them up one by one. Is there a good set of tools to verify the integrity by filetype, so I can detect (and delete) my bad files?"
then you see which are changed. then check, if the file is much smaller or corrupted.
you seem to be surprisingly ok with the fact that your computer crashed and all your documents and media were corrupted, as was your backup. I would have been beside myself. Hulk smash! Please let us know what different set ups you're exploring to avoid this.
If the entire contents of the files are messed up, you could write a quick script that compares the output of file(1) to the file extension. I wouldn't call this high-fidelity - I'd recommend using this to generate a list you go through by hand - but it's at least a starting place.
"What's a Good Tool To Detect Corrupted Files?"
BSOD?
Try running "file" from a command line on a few files you know to be corrupt. If the file command tells you the same, you could run a quick bash script to loop through the files and spit out the names of the bad ones. This is all assuming you know what you are doing with shell scripting.
Compare files hash to known hash of good file.
I'm not entirely sure if this will apply to your setup because corruption is kinda sporadic but....
Way back in my IRC days I ran an Fserv, DCC transfers between different versions of MIRC was a nightmare and quite often ended up corrupting files.
What I did as a half assed method, was write a simply batch script (this was a long time ago) that scanned my folders and created a text file listing of each file, extension, and most importantly, file size. I'd try to run that script daily, but you could easily automate it to run with scheduling.
Then I wrote another simple script to basically parse the current folder contents with the latest list I created, any differences in file sizes would be spit out to another text file. This would be basic command line stuff in linux, but again, this only catches the files that have changed size since the scan, it's not a fool proof corruption method.
is urgency. Corrupted files have the ability to detect urgency and your discovery of them will come in a form compatible with the laws of Murphy.
What ever. Look at /. advertisers. Why are you pestering us with mundane marketing questions?
I know - /. is a marketing company now ... gathering marketing data.
My Bad.
1. Compare to backup, files that match are ok.
2. AppleScript option others mentioned may help reduce it further.
3. Backup regularly, and verify your backup procedure.
4. Anything else will cost you consulting rates.
make imaginary.friends COUNT=100 VISIBLE=false
2000-2001 MAF-Soft http://www.maf-soft.de/
The version I have is v1.0.3.102
It can scan single mp3s and entire folders structures for defects and logs everything if you wish. It will give you a percentage of how good the file is.
Depending on the damage you may be able to fix headers and chop off corrupted tag info with something like a MP3Pro Trim v1.80.exe
Obama's legacy: (N)othing (S)ecure (A)nywhere and (T)error (S)imulation (A)dministration
Go nuts.
0 = 1 + e^(Alt something)
use the 'diff' command between your backups and the originals.
diff -rq /backuplocation /originallocation could work a treat.
Switches:
r = recursive
q = tell only if files differ
Suppose your volume is mounted under /mnt/a and your backup is mounted under /mnt/b. Something like this should work:
for f in $(find /mnt/a -mtime -2 | sed -r 's/(^/mnt/a/)(.*)/\2/') ; do
cmp /mnt/a/"$f" /mnt/b/"$f"
done
That should find all files which have been modified within the past 2 days and differ from your backup, which will help narrow down your search. I don't know of a tool that will address your specific question about testing for integrity for particular file formats. For specific file formats, you can automatic this, of course, like using ImageMagick for image files, but I don't know of a tool that will just do everything. It shouldn't be hard to write a script to look at the extension and the output of the "file" command and determine which tool to use to automatically check integrity for that specific file format.
or sha1sum if you prefer. Automate in cron against a list of knowns.
eg: /home/wilbur/Documents/* > /home/wilbur/Docs.md5 /home/wilbur/Docs.md5
$ md5sum
$ md5sum -c
Join the Slashcott! Feb 10 thru Feb 17!
Have some respect, the man just lost his entire porn stash.
Mod me down, my New Earth Global Warmingist friends!
If you're talking about recovery tools, you're already on the wrong track. A Time Capsule costs $300. How much is your data worth? How much are the tools going to cost to recover it? How much is your time worth? I'll bet that the sum of those last three things is a whole pile more than 300 bucks.
If I were you, the thing I'd buy right now is a good backup solution. Re: your existing data, take a full image of your hard disk and take your time recovering it.
Once you've new backup system, you can then sit there with a big smile on your face and comment smugly on all future /. posts about data loss.
Have I lost data? Hell yeah. And it will never happen again.
-bh
If you see IE come up, followed by several dozen browser tabs in split-second progression, followed by a lot of message boxes proclaiming "Warning! This computer is infected by trojan pirate hacker malware viruses!". And you later open your inbox and see a lot of spam from debt remediation services.
That's a bad sign. I would delete the file.
An honest question :
I've had several crashes over the years with Windows XP but the files, data and system files were never corrupted.
In linux it seems that file systems are not very resilient, and the least crash can corrupt your files.
Is NTFS such a good well designed file system compared to linux file systems ?
You can run jpeginfo -c. I have a script that runs against a directory and makes a list for when I do data recovery for all my friends who don't listen when I tell them their 10 year old laptop may be dying soon.
In the land of the blind, the one-eyed man is kinky.
see man mtree for details.
unix "file" is not the answer. For some formats it does as little as look at a couple header bytes. Its a great tool to guess a format. Its a terrible verifying parser and does nothing to verify content.
An example of what I'm getting at, with some made up details, unfortunately html is not like well formed xml and every viewer is different anyway so the best way to figure out if a html web page file format is corrupt is unfortunately to pull it up in firefox. This only detects corruption in the structure of the file, if the corruption is just a couple bits then you end up with problems like tQis where the only way to see the h got fouled up is to write more or less a IQ 100 artificial intelligence. All "file" is going to test is pretty much does the file begin with or contain a regex something like less-than html greater-than (getting past the filters).
For content you could F around with, for example, piping a mp3 file thru a decoder and then thru an averaging spectrum analyzer and see if there's anything overly unusual in the spectrum. Also some heuristics like is the file only 1 second long, then its F'ed up.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
After one page of nonsense and replies from people who didn't even bother to read the question, finally a useful answer!
Why isn't there even one filesystem for desktop PCs which stores per-block checksums in the metadata? We store the freakin' last-accessed timestamp whenever a file is read, but no checksums?
Well, I don't own nor use any Apple products, but I do have some Linux experience.
It would depend entirely on the type of corruption (fully corrupt vs randomly corrupt), but you could try the find command, combined with the file command to detect a files mime type (which if zerod or garbage, would be obvious).
For example:
find /path/to/start/at -type f -exec file "{}" \;
That would look in "/path/to/start/at" and everything below, for all files ("-type f"), and for each one found, would run the command ("-exec file '{}'") on it. The last part is "\;" since you need to tell the find command that there is "no more", but since ";" already means something to the shell, we need to escape it with a backslash ("\").
The output you get will depend on what it is you are running the command on. If it is an image, file will report such. If it just says "data" or "empty", it is probably corrupt, or worth manually investigating.
Author here:
> Last backup August.
Yes, that was silly of me.
> Thinks there is a way to detect generic file corruption
There is no way to detect generic file corruption. But there is a way to detect specific filetype corruption. For example, I already found mp3val, that is able to scan all my mp3 and check for file integrity, and even fix a few kinds of corruption (such as unmatching bytes in the header and sound chunks). Maybe with the right set of tools, I might also detect (or even fix) my corrupted pictures, movies and books as well.
If I clone myself, can I call it a thread?
If a girl winks to us, can I call it a race condition?
Not sure if it's the HFS+ filesystem, I let my battery run out a lot while working on my ubuntu laptop (ext4 fs), never have a prob.
You need good filesystem, with embedded data checksum and self-healing using redundant copies. For Linux - btrfs is fine. For Mac OS X & Linux - ZFS.
:wq
Tech Tool Pro, over on the Mac side, has a "File Structures" check which looks at a lot of different structured file types to make sure that their internal format is valid.
It's already too late, but I keep important files with par2 files. That way, when there's like 5% corruption, I can still fix the file.
I do this with flac files and some datafiles.
Also make sure you keep backups going. I guess this was your warning. Everyone needs one.
Well, don't worry about that. We can get you back before you leave. (Dr. Who)
That seems very strange--the only files that should really be corrupted, unless something extremely rare and catastrophic happened, are the ones that were being written when power went out, or were cached. And even then, a flush usually flushes everything, or at least whole files at once, or areas of disk. Is the partition highly fragmented or something?
I know this doesn't do much for your question, but that kind of failure mode is almost exactly what filesystems do their damnedest to avoid. HFS+, being journaled, should be even more proof against, well, exactly what happened to you. Maybe the Linux driver is poor, but man, if you got silent data corruption on a multitude of files that weren't even being written, that's really bad and the driver should be classified "EXPERIMENTAL" at best, and certainly not compiled into distros' default kernels.
To answer your question, I don't have experience with any tools (I automate my backups, and any archival files go on a RAID volume that does a full integrity scan nightly), but once you find one, you should separate your files into two categories--"must be good", and "can be bad". The "must be good" files (serial #s, source code, etc.), you hand-check, so you know for certain that every one of them is good. It'll also motivate you to replace them now, instead of later when replacements will only get harder to come by. The "can be bad" files (music, pictures, etc.), you do the automated check on and then just delete as you run into ones that the check missed. This has the advantage of concentrating your effort into where it's useful. If you try to check all of your files, you'll just burn out before you finish. You may even want to do more advanced triaging, but you'll have to come up with the categories and criteria there. The main thing is, split this problem up.
<xml><I><am><so><damn>Web 2.0</damn></so></am></I></xml>
First, copy to an external disk all files that are on your hard disk and not on your backup.
Next, compare the files on the hard disk with the backup, and copy to that same external disk all the files whose MD5 sum is different AND last modified date is later than the backup.
Then, wipe the hard disk.
Then, restore the backup.
Finally, just look at the files on the external disk. There won't be nearly as many of those.
TL;DR Summary. in a quest to be #, Mr. Courteaudotbiz forgoes reading the summary to post Snarky Comment #1(TM)
There Can Be Only One...
You can just run mencoder or ffmpeg on the mp3 and mov on all the files (with a small shell script, probably involving 'find' or similar), just tell it to write the output to /dev/null, that should go through those files as fast at they can be read from disk and abort with error on those that are broken.
For the jpgs, you could try something similar with imagemagick's 'convert', to convert them to whatever format to /dev/null, which also needs to read the whole file content and aborts if they're broken (one should hope).
Those converters are really fast, especially ffmpeg, so that should complete in a reasonable time.
Having dealt with file corruption first hand after bit rot on some older media, the best recommendation I have is to find a solution which will prevent future bit rot. I have been using Great Lakes SAN (http://glsan.com) , who bases their system on zfs. Their system using 256bit check sums on all files and can detect and correct any/all file issues which may occur. Furthermore, backups, off-site disaster recover + system monitoring are built into their system.
I'd be asking myself why lots of files became corrupted from one dodgy file system event. Assuming HFS works like file systems I'm more familiar with, it will allocate sequential blocks for files wherever it can. This means that a random filesystem splat is really unlikely to corrupt loads and loads of files. You might expect a file system corruption to cause a load of files to go missing (if a directory entry is corrupted) or corrupt a few files, but not put random errors into loads of files.
I'd check to see whether files I was writing now get corrupted too. It might be dodgy disk or RAM in your computer.
The above might be complete paranoia, but I'm a paranoid person when it comes to my data, and silent corruption is the absolute worst form of corruption.
For next time, store MD5SUM files so you can see what gets corrupted and what doesn't (that is what I do for my digital picture and video archive).
Every man for himself, all in favour say "I"
I've certainly seen corruption with XP crashes, not a big deal because I do backup. About the same with the other file systems. In this case he was using Mac OS 10.7 Lion, which is a mess, and two others accessing the same partition. Not surprised.
"The ability to delude yourself may be an important survival tool" - Jane Wagner -
Not the BSOD.
If the OP had used open source "tripwire" on known-good files in each filesystem on his Macbook, and saved the resultant data output to a USB thumbdrive formatted with FAT32, the OP would have had a good chance of determining all corrupted files. In this case, an ounce of prevention would have prevented several pounds of "cure".
Check out http://tripwire.org./
Including all of his erotica e-books. Tough life, dude.
Help I am stuck in a signature factory!
Author here:
Ok, I could deal with the loss of some unique videos and pictures from travels... but now that you mention the porn... *weep*
If I clone myself, can I call it a thread?
If a girl winks to us, can I call it a race condition?
An honest question :
I've had several crashes over the years with Windows XP but the files, data and system files were never corrupted. In linux it seems that file systems are not very resilient, and the least crash can corrupt your files. Is NTFS such a good well designed file system compared to linux file systems ?
Linux supports a wide array of filesystems. Which ones have you used? I have used ext3 and ext4 and have never run into file corruption problems. Both of those are journaling filesystems. Journaling filesystems helps prevent corruption in the even of power failure.
Beside the filesystem, one other possibility for corrupted files is a bad hard drive. I know someone who reinstalled Windows on his desktop on a regular basis because key files would go missing or get corrupted. I took a look at it and found out that he simply had a bad hard drive. After getting a new one, he didn't have anymore problems.
I'd recommend running a base OS and then run something like VMware workstation so that you run other OSes inside the main OS. One huge benefit is that you can have access to multiple OSes at the same time and you don't need to reboot into them either. With hypervisor technology getting common on desktop, there probably isn't any need to multi-boot unless you have a specific reason not to use virtualization.
In linux it seems that file systems are not very resilient, and the least crash can corrupt your files.
Is NTFS such a good well designed file system compared to linux file systems ?
I've never had corrupt files after a Unix crash; be it SunOS, Solaris, HP-UX, Linux or any of the other Unix variants I've used.
I've never had corrupt files after an XP crash, but I've often had scandisk delete files, including a multi-gigabyte game installer that I'd just downloaded before it crashed. It regularly deleted Firefox bookmarks before they switched from storing them in big HTML files.
The NTFS approach appears to be 'I'll guarantee file system consistency but won't guarantee any of your files are still there'. I'm sure you can find similar Linux filesystems, but the most common ones don't seem to have any problems.
Perhaps but I agree with the first post. Going through and simply looking at all the JPEGs or MPEGs is probably the only way to tell if a file is corrupted (I wouldn't trust the CPU to do an accurate job). Also gives you a change to erase a lot of stuff you really don't need anymore. I dumped 300 gig off my drive simply by going through everything... took awhile but it was worthwhile to get rid of old shows/movies I'll likely never watch.
My AC stalker: " I personally agree with your posts most of the time, but that won't keep me from modding you troll"
zfs! Works great. Included with FreeBSD 9, amongst other OSs.
You might also enjoy John Siracusa's exhaustive review of filesystems on one of my favorite podcasts.
Spoon not. Fork, or fork not. There is no spoon.
The JSTOR/Harvard Object Validation Environment:
http://hul.harvard.edu/jhove/
It's specifically designed to first probabilistically identify files, then attempt to verify their format.
Disclaimer: I haven't worked on it directly, but I did spend a number in the digital preservation space, so I probably know some of the people who have contributed to it.
Let me ask a stupid question since I've never run a battery out on a machine running Ubuntu. Why did this happen? Running OSX or Windows, the machine would have hibernated safely before the battery ran out. Does Ubuntu not do this and it just dies? Or is this something you configured to act this way? If it is default behavior in Ubuntu it is something they ought to fix.
Yeah right, like a slashdot member could even lift a 15 lb. sledgehammer!
I'm not using Gamemaker until it can also clean the kitty litter, go to the grocery store for me and find porn automatically for me!
Cast Detect Evil, Sense Motive, and Discern Lies on the potentially corrupted files.
mplayer can detect corrupted movie and audio files find . -name '*.mov' -exec mplayer -msglevel all=6 -speed 100.0 -framedrop -nogui -nolirc -cache 8192 -tskeepbroken -ao null -vo null {} \; | grep Warning! > $1.txt Change the *.mov as appropriate.
Alright now I'm afraid I can't help with your verify problem but I do have one piece of solid advice: get rid of Paragon HFS immediately!
It is a truly shoddy piece of software that as of version 9.0 has a terrible bug that will cause it to destroy HFS+ filesystems. Google "paragon hfs corruption" and you will see many many horror stories from people who just plugged a Mac OS X disk into a Windows machine w/ Paragon HFS and then discovered the entire filesystem was hosed. In my dual-boot win/mac setup I replaced my copy of MacDrive with a trial version of Paragon HFS 9.0 from their website and every single one of the six HFS+ disks I had connected internally were damaged. Disk Utility couldn't do a thing and I had to buy a program called Diskwarrior to even begin to recover data. I ended up losing two disks worth of files anyway.
http://www.mac-help.com/t12137-opened-hfs-drive-win7-paragon-hfs-now-wont-boot.html
http://www.wilderssecurity.com/showthread.php?t=299306
http://hardforum.com/showthread.php?t=1677099
http://www.avforums.com/forums/apple-mac/1509344-hfs-super-block-not-found.html
whew! Anyway the pain I went through after that software very nearly ruined my life was so great, I don't want it to happen to anyone else. According to their own website 9.0 has this awful bug but they fixed it in 9.0.1. Evidently the trial download on the main page is still for version 9.0 and still has the disk destroying bug! Any software company that releases a filesystem driver with this terrible a bug (not to mention the numerous reports of BSODs and other relatively minor problems) clearly has terrible quality assurance and simply can't be trusted.
Just have your OSX do a repair - it could be that certain VTOC or directory tables were damaged, and a repair may fix it. The files themselves should be OK, but the pointers to them are fubared.
Also try something like http://www.cgsecurity.org/wiki/PhotoRec or similar to recover deleted files. There's one for OSX. Run it after a repair, and photorec, and you should get most of your crap back.
Look, you're really taking the wrong approach here. The way to deal with corruption is avoidance, backup, and corrective action.
1) Avoidance. This is the generally the role of the filesystem and the underlying hardware, each of which have methods for preventing and correcting data corruption without ever involving the user. The user has a small part to play by doing things like shutting down instead of turning off whenever possible, though journaling filesystems (i.e., all modern filesystems) will know when a file operation was interrupted prematurely and check the integrity automatically. Also try not to put different file systems and OSes on the same drive, since there's the possibility that one OS may not respect the FS or limits of another (typically/historically, Windows has been the culprit here, but not always, and not so much anymore.) Any OS will generally leave an unrecognized drive alone unless you tell it to do otherwise, but the system drive has often been considered fair game.
2) Backups (optional). Once you have a known-good (or believed good) installation, create your backup. Repeat somewhere between often (if your data is important) and never (if it's not).
3) Correction. If and when you come across data corruption, that's not a sign that you're wasting space on your hard drive; it's a sign that something is seriously wrong. The proper course of action is to identify the underlying cause and correct it, not to delete the files to free up space. If you're experiencing corruption on only one drive regardless of channel and cable, replace the drive. If you're consistently having problems on a given channel, then don't use that channel. If you're having random issues across all drives on all channels, then the chipset is bad and the motherboard should be replaced. Basic troubleshooting.
Technically you *could* take a checksum of all of your files and update the database every time a file is changed. Some antivirus systems already do this to detect infections, but it would also detect incidental changes as well. The problem is that constantly verifying the integrity of your files will only hasten the demise of your storage medium. It's a self-fulfilling prophecy.
https://www.eff.org/https-everywhere
You clearly need an image based backup system to prevent this from happening again. It needs to be a chron job (or task scheduler) and run on regular intervals when storage is available. ideally, it needs to be network storage, so that a sudden disconnect (abscence of power) cannot easily corrupt the backup. There is an open source version of Ghost, partd, rsync.. options for you, though I am relatively new to linux so I don't know what the appropriate option is for you. Time machine you could use if you had a separate partition, but I think that isn't what you want. also, fundamentally writing to one partition from three OSes is asking for trouble.
Ubuntu pops up a warning window, and if you ignore it the battery light turns orange, and then red, and then it should hibernate. Flat-out dying is not something I've come across under Ubuntu (and I have some flaky old machines with old batteries, and they still warn me and then shut down).
There is an oldie out there called cleanjpg.exe which does the same for jpg's.
The bad news is I don't know of any (and I don't think you'll find any) easy, one-shot tool to run across the whole lot that gives you a simple "corrupted yes/no?" answer to lots of different filetypes.
The good news is it'd be reasonably easy to lash together something in bash, kick it off overnight and come back in the morning to a list of probably-corrupted files.
In pseudo-bash (because I haven't the time to write it out and check it works properly), something like this would be a good start:
function checkJpeg {
jpeginfo -c $1 || return 1
return 0
}
function checkPdf {
# do something to check a PDF is OK
}
FILETYPE=`file $1` ;; ;;
case $FILETYPE in
"jpeg" )
checkJpeg $1 || echo $1
"PDF )
checkPdf $1 || echo $1
esac
Then run it with the help of find /home -type f -print0 to check every file in /home. This would give you a list of potentially-corrupted files. Up to you how you deal with it - personally I wouldn't run rm against it in case you find files that can be rescued or that your checks aren't as perfect as you'd like.
For extra credit, determine the expected filetype based on file extension and then use file(1) as your first "is it corrupted?" test - that way you'll spot files that are too corrupted for file(1) to work reliably.
http://en.wikipedia.org/wiki/Comparison_of_file_verification_software
md5sum is the one I know best, but that's because my computing is unix-centric.
Short answer:
cfv
Long answer:
How can you be so tremendously stupid to not even know about cfv?
The real reason and it was stated in the summary is that the file system was HFS+ which is far less tolerate to this behavior than ext4.
That's a pretty good idea, if you only want to detect corrupted files (and yes, I know that's what the OP said he wanted), but I can't believe no one's suggested par2, yet. It will not only detect corrupted files, but repair them, too. If he had used par2, he wouldn't have to delete them.
Sit, Ubuntu, sit. Good dog.
mplayer can detect corrupted movie and audio files find . -name '*.mov' -exec mplayer -msglevel all=6 -speed 100.0 -framedrop -nogui -nolirc -cache 8192 -tskeepbroken -ao null -vo null {} \; | grep Warning! > $1.txt Change the *.mov as appropriate.
<infomercial>its JUST. THAT. EASY folks!</infomercial>
I am using Hashdeep for this purpose.
http://md5deep.sourceforge.net/
It is similar to a script that generates an md5 or SHA hash value for every file, but much easier to use.
... yes, this is not what you want to hear at this point, but try to have a positive take on this.
Last year during a routing Windows7 installation, my second hard drive from which I double boot my 90%-of-the-time-in-use Linux was destroyed. Either a coincidence that it occurred during the win7 installation or a nefarious plot, but the hard disk, a 1TB Seageate sata, developed an unrecoverable click of death.
On that hard drive I had my short stories which I had written in college and the intervening years since then, much of my photos, skype history and many other things, seemingly important to me at the time of the "disaster". I was inconsolable for a few days, and felt like I had been bereft of someone very dear to me. Then it hit me -- to hell with the stories, to hell with the photos, to hell with the rest of the digital baggage I had accumulated. I could write my stories again, and do it better, I could take more photos, I could hoard more useless junk. After a month I no longer missed any of the lost stuff.
Learn to view such mishaps more philosophically and learn to shed all the useless garbage you accumulate through the years; realize that almost nothing that you can store on your computer, or up in your attic, has really all that sentimental value you attach to it. Learn what's important, intrinsically important, to you and safeguard that. All the rest, you'll be amazed how little you need it and how even less you'll miss it.
To hell with useless stuff.
I've seen defective memory cause data corruption too, among a lot of other strange problems that seemed to point to another hardware component. Memtest86+ is your friend.
I used to do that, but found it to be pointless these days. Organizing the stuff is one thing, but deleting is basically pointless unless you can automate it. 300GB may seem like a job well done, but with 3TB drives for $100 these days, you just saved yourself $10 worth of harddrive space and it probably took you a few hours.
My current setup is to have everything on my server box and simply copy over what I need to my laptop as I need it and NFS/SSHFS the rest of it on the fly when home.
It's the HFS+ partition that died, not the Linux one. Mac OS is certainly not a "freetard" OS.
-- Linux user #369862
Lacking not only a backup but also PAR(2) and MD5 files, manual inspection of each and every file is the ONLY way you can determine their integrity. There is no automagic after-the-fact integrity check. If you had MD5 sums for every file, you could at least check their integrity. Some PAR2 files would not only verify but possibly repair if the damage wasn't more extensive than the PAR recovery blocks. Of course if you're willing and able to do all that, you'd probably have had full and differential backups first.
(And yes, the subject was a lyrics reference.)
The Linux HFS+ driver can't even work in write mode unless the journal has been deleted, so the journal isn't working when using the HFS+ partition under Ubuntu and probably Windows as well (author take note). I would not use that filesystem under Linux or Windows on a daily basis. Also, since the journal has been deleted, you are probably missing the safety of journaling under the native OSX as well.
Author should also note that archival backups with md5 or sha256 checksums are probably the most straightforward way to maintain data integrity. If you want something more elegant for day to day use, I would consider setting up a NAS using either BTRFS or ZFS as the filesystem along with a nice 1Gbps LAN (if you don't have that already).
Diskwarrior works miracles
http://www.alsoft.com/diskwarrior/index.html
I hate your brace style anyway...
AIDE (Advanced Intrusion Detection Environment)
Maybe you didn't mean it this way, but dang if I did not see all the PHBs come out from work with your comment. "I can get 1TB Drives from Fryes for $80.00, why do you say it costs several hundred?".
Oh, you wanted redundant drives to be covered in the event of a failure? You wanted a drive that has some performance so it does not take 32 minutes to open your word file? So much for that 1TB for 80 bucks thing...
The new one is "SSDs are only $150.00, and they are the same as what you get for SSDs without all the writes.
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
Spinrite may work
I'm not using Gamemaker until it can also clean the kitty litter, go to the grocery store for me and find porn automatically for me!
You might want to program it for the type of porn you like. If you do not, you may get some horrific* results.
*Horrific being the sights may never leave your mind and cause you to have nightmares, day-mares, lose your job, lose your wife, kdis, house, car, computers, pet(s), .
George is your best bet. He's not bright enough for most support tasks, but he can certainly handle this one.
The Digital Preservation Community (e.g. www.openplanetsfoundation.org) use JHOVE http://hul.harvard.edu/jhove/ and JHOVE2 https://bitbucket.org/jhove2/main/wiki/Home for validating some types of files against documented formats. They might be useful for checking which files are valid or not.
Check the md5 sums of the files. I was unsatisfied with md5deep so I wrote my own tool to calculate the md5 sums of all the files. It will tell you if the files have changed directories, been updated or are just corrupt by looking at the files datestamp and comparing it with the one in it's database.
http://pastebin.com/uTTRh4Ws
My files go through descending levels of staleness. First I copy them off the primary drive to a network drive. They sit there for a while. When a couple years have passed without looking at them, they go on an offline drive and I create a file listing which I keep on the network drive. Once a decade has passed without loading the offline drive, they go in the trash.
Files that still have value get caught in the sweep but then migrate back to the primary drive as needed. Saves having to scrutinize everything before hand.
For the OP: restore from your august backup. Add anything on the drive with a more recent mod time. And then deal with the corrupt files as you come across them. If you come across them.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
if the previous mention hadn't been modded down to -1 you would have noticed you were beaten by 2 hours.
Indeed, I used photorec/testdisk to recover mp4 files after they had (all) been accidentally deleted from an HFS+ partition.
But when I first started it in it's default mode, it "found" only rubbish, breaking up the actual mp4s into a mess of .doc, xml, jpg, .whatever files, including totally broken .mp4s.
When I restarted it after configuring it to only look for .mov/.mp4, it did a fantastic job, and as far as I know, all files could be recovered. Of course, that was made easier by the fact that I knew that all the files which needed to be recovered were .mp4.
This guy is using jpeginfo for jpegs http://watson-net.com/blog/checking-the-integrity-of-all-jpg-files-in-a-directory/
I suggest that if you care about your data, you learn the difference between meta data journaling and data journaling. Plus while your having fun, a side trip down proper filesystem/disk fencing is in order.
The problem on linux, is that a lot of filesystems are labeled as "journaling" and not to many people dig into the details, picking filesystems based on popularity or performance. The truth is that nearly all (the linux implementations) of them are susceptible to silent data corruption, and the ones that aren't, don't tend to ship in their safest mode because it kills performance.
Then there are the drivers/controllers with write-back caches, that fail to do proper fencing. Frankly, the state of data integrity over power loss/system crash is pretty bad.
Bottom line, do your homework.
I think this is the root of the problem here, he chose the wrong filesystem to share between the three OSes. Sadly there are not too many choices. FAT32 is the only one natively supported by all three, with its well known limitations. He might have been better with NTFS though, using NTFS-3G on Linux and OS X, but that has some performance hit. There's really no perfect solution for this kind of problem.
.sig: No such file or directory
Well, jpeg files have a structure that will generate detectable errors if it's damaged. So simply opening them with something as simple as djpeg from the IJG and piping the output to /dev/null should give you a pretty good start on damaged images. Something like this perhaps:
find . -name "*jpg" -o -name "*jpeg" -o -name "*JPG" -o -name "*JPEG" | while read filename; do if djpeg "$filename" > /dev/null 2> then :; else echo "$filename" is toast; fi; done
You could probably do something similar with mpg123 and mplayer for .mp3 and movies.
Weeping sores? You'd better use more KY, buddy.
Hail Eris, full of mischief...
E pluribus sanguinem
If you rarely use Windows, you could partition your disk, and format the Windows part to FAT. Then, on the odd occasion where you want to transfer files, you could just mount its partition in Linux (that's what I do), or OS X. I like the idea of sandboxing Windows, and not letting it touch/corrupt my main (ext4) file system.
HFS+ with journaling enabled is as solid as EXT. Journaling is on by default if the disk was formatted in the past several years. I'm curious to know if he had it turned off.
>>>you just saved yourself $10 worth of harddrive space and [enjoyed watching a ton of TV shows/movies during the process]
Fixed that for you.
My AC stalker: " I personally agree with your posts most of the time, but that won't keep me from modding you troll"
There ought to be an &1 after the 2>.
debian has debsums (assume ubuntu includes this too). Can check md5 hash of every file installed via pkg manager and list what changed. Then , just --reinstall of those packages, and back to pristine state as far as OS and applications.
A tiny part of what you want to do, but may help.
The Linux hfsplus driver does not support journaling. If your partition had journaling enabled then it was outdated when linux powered down. After the power failure the only files that should have been affected would be recently written files that linux had in the write cache. The problem is that when you used OSX to repair the disk it would try to use the outdated journal to correct errors in the directory structure, and that may have caused the corruption you found in files that were not open when power was lost.
First, let's presume you're running Linux for what follows.
/foo/bar -name "*.jpg" -print | sort -u > /tmp/files.jpg
/foo/bar for all files suffixed ".jpg" and dump a sorted list of them into /tmp/files.jpg and this one:
/foo/bar -type f -print | sort -u > /tmp/files.all
/tmp/files.all.
(Note that the method by which find traverses filesystem trees won't yield sorted output, hence the
need to pipe these through sort.)
/tmp/files.jpg /tmp/files.all > /tmp/files.all2l
.gif, .mpg,
etc., as you deal with each file type and reduce the remaining list to those awaiting
your attention. /tmp/files.all3, /tmp/files.all4, etc. will each be smaller and eventually,
if you deal with all files, /tmp/files.allX will be zero-length. Note that not all files
have suffixes, of course -- and those without will likely be the ones requiring the most
manual effort. If you want to know which suffixes are most numerous, something like /tmp/files.all | sort | uniq -c | sort -n
1. You're going to want to be familiar with both file(1) and find(1). File(1) is pretty straightforward, but be aware that its heuristics for file type detection vary in accuracy. If you're not find-literate, then at least get used to this construct:
find
which will recursively search directory
find
which will search the same directory, but will return a list of all (plain) files, that is, things which are not directories, devices, sockets, etc., sorted and dumped into file
2. You now have (a) a list of all jpg files and (b) a list of all files. (I picked jpg arbitrarily to illustrate the process, by the way.) You can now generate a list of all files that are NOT jpg with this:
comm -13
The point of this exercise is that you can now repeat steps 1-2 with
sed -e "s/.*\.//"
will give you a rough idea.
3. Now then...you'll need some tools for dealing with each file type. The first tool I'd use is stat(1), to check sizes for plausability. Then things like jpeginfo(1), mp3val(1), tidy(1), will be some help, but of course you'll need to distinguish between "error message emitted because file is corrupt" and "error message emitted because file has minor issues...that it had BEFORE this episode". You may need to check the Ubuntu repository for tools you don't have; you may need to do some searching on the web for "Linux tool to check PDF integrity) and similar.
4. If you have backups of any kind and can restore them, then you could try using sum(1) to compare checksums pre- and post-incident. This is a filetype-invariant method, which is good because it lets you skip the above...but bad because all it wll tell you is "different", not "mildly damaged" or "horribly corrupted" or something in between.
5. I would recommend against deleting anything at this point. Instead, move it to secondary storage, like an external drive. I don't have a specific reason for advising this, other than "many years of experience doing partially-manual, partially-automated things like this and a recognition that sometimes errors in the methodology...or fatigue introduced by the tedium of executing it...lead to mistakes".
6. Good luck.
The identify program is a member of the ImageMagick(1) suite of tools. It describes the format and characteristics of one or more image files. It also reports if an image is incomplete or corrupt.
Last I checked which was about two months ago HFS+ with journaling enabled does not work reliably at all on Linux. I ran into this problem with trying to create a Drobo that everyone could share. The reality is that the only option would have been was HFS+ with journaling disabled in order to have all three. NTFS-3g from my experience works great on Linux and like crap on OS X nevermind the fact that you wouldn't even be able to install OS X on an NTFS partition.
zfs, btrfs or git fsck
I've have a bunch of mods to git to run as a backup utility. Things like all the meta informatation, device files, running across filesystems and so on... :-)
Though, in this case, sounds like you want to get rid of Paragon.
find . -type f | egrep -i jpe?g | xargs -n 1 djpeg etc....
I made a command line tool called Fingerprint for this process: http://www.oriontransfer.co.nz/gems/fingerprint
There is a GUI available for Mac OS X.
Even trolls look stupid when they don't RTFS.
Files can be corrupted by rare spontaneous bit flipping, by mis-writing a block that was intended for another file or corrupting the block list to include data from another file (cross-linked files), by including blocks that don't exist, or by including blocks that have no data or arbitrary data.
Headers or meta data in some file formats can be verified by applications that support that file format, but it's possible for some of those problems to change the file's data such that the data is still valid, but wrong. If you have a large collection of media files or image files, filesystem corruption could potentially cross-link valid data from another file of the same type.
All of that is to say that the only way you can reliably detect corrupt files is to compare them to files that are known good. To anyone with backups in your position, I would simply say that the best option would be to wipe the system and restore a backup that you trust. If you had rsnapshot backups, you might be able to: /backup/ /filesystem/
rsync -avcn
rsync would then tell you which files differed from backup.
According to comment 39919031, Paragon HFS may have serious bugs. It's possible that the problem didn't actually come from the power loss, but from a bad filesystem driver. I'd recommend using something better supported by all systems for your shared space, or using hardware assited virtualization for all but one of the operating systems. On my own hardware, I run Linux with other systems in KVM guests, which works well. The host OS can export shared space over the network (NFS or CIFS) to the guests, which is probably the most stable filesystem configuration possible.
Well yes, it is relatively easy, when compared to the alternative of writing your own program to do the task. Now that you have the command line, all you have to do is copy and paste, no thinking required. I notice you did not offer an alternative, either.
That seems not worth it. The thing is, both drive-space and data-volume tends to double every ~18 months or so. You wait first "a couple of years", then on a network drive, then once a decade has passed, they go in the trash.
But a decade ago the cheapest storage was a 40GB drive costing $130 or thereabouts. Today 40GB worth of space is 1.5% of that shiny new 3TB-disk costing $150 or thereabouts.
There's essentially no benefit to deleting old data, because old data is *always* small data, and so copying it to the new disk will use a miniscule portion of the new disc and have essentially no cost. $150/3TB is equivalent to $2 for saving those 40GB.
The only data that's potentially worthwhile to delete is *new* data that you have no need for. There is no such thing as "old but large data".
Avoiding clutter is a different issue, but that's easily solved by copying all the old data to a named folder, then move out of that folder and into the current file-system only those files you actually use.
exFAT is a good alternative, even if it is propietary. It is what I use to share data between Linux, Windows and Mac. There is a fuse driver for it.
One way to check JPEG/PNG/GIF files that may be corrupted is browsing the folder with thumbnails on. The files that doesn't show the thumbnail are corrupted.
They may have been corrupted before your crash. I had spooky(similiar) things happen using linux ntfs write before. Just may have noticed the bad files after power loss.
Triple booting is the source of the problem. Some things don't mix even if they are technically "supported."
If you must triple boot consider a more compatible file system (fat32 lol?) or disable writing to disk in foreign OS.
Data that is irreplaceable may be worth recovering.
For me formatting and starting fresh -- best option.
The actual files may not be corrupt. Are deeply nested folders the ones corrupted? Something to consider.
Simple enough, package up the 'known good' files on your server into a torrent, give torrent file to laptop, have it use bit torrents built in hash checking to verify and, if bad, replace the damaged parts.
...
Silent file corruption is the reason why I now keep all my data on a ZFS filesystem. ZFS has a checksum for every block and if you have redundancy at all (raidz, raidz2, or even just tell ZFS to keep two copies of each file) then it will repair the corruption as well as detect it. I've got a HP Microserver running Solaris but I recommend running FreeNAS instead if you don't know ZFS. This blog is a good place to learn about ZFS.
A latent existence
I don't like losing data so it sits in a raid and gets backed up. This means I'm not just keeping one copy of primary data, I'm keeping many. And reprocessing/recopying it each time I make a backup.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
http://diglloydtools.com/integritychecker.html
I use it all the time on a daily basis on my post-production job, with daily TB of in/out.
Do the job very well.
It is rather unlikely that you have a massive corruption of actual data in multiple files--for that, you'd need a sustained write activity hitting all over the disk. Possible, but not very probable. Instead, I think you have metadata corruption, so that the filesystem points to wrong blocks. The glimmer of hope here is that the actual file data is mostly contiguous, so that you can scan through the image and identify individual files even without the filesystem information. There is a forensic tool called 'foremost' that does exactly that: rips through the binary filesystem image and as it finds headers of known data types (jpeg, gif, doc, mp3, etc. etc.), it tries to find as much following data as is consistent with known file layouts. The result of course has tons of cryptically named files of each type foremost knows about---not all of them are legitimate, even---but it's better than nothing.
but with 3TB drives for $100 these days
Where are you buying your drives? I can't find them for less than about $185...
Karma: Poor (Mostly affected by lame karma-joke sigs)
If you really did "fix" DarwinSurvivor's post, then perhaps you'd care to explain this part of your prior post?
took awhile but it was worthwhile to get rid of old shows/movies I'll likely never watch.
3TB HDD is less than $150 at newegg. It's faster than or as fast as any 7200RPM drive from previous years as well, even if it's a 5900RPM drive, because technology just gets better. I'm not even sure what you are going on about with SSD.
Are you not American? newegg has them for less than $150.
But OSX IS based on a "freetard" OS
Since your major partition is HFS+, I recommend DiskWarrior (http://www.alsoft.com/diskwarrior/). I've used it professionally for over ten years, and it still does the best job at finding, fixing, and reporting corrupted files on HFS disks.
or just run md5deep on all files and save it to a text file; a full backup would have been the best strategy here though (I use bacula, which also stores the hash of a file it backs up).
So, 1TB for $50 with no backup in case of failure or corruption. Or did you mean 1TB with redundancy for $150? That begins to fall into the "several hundred" category.
It's just a silly joke, calm down Francis.
JHOVE is good, but only recognises a small number of files. It would be better to use the File Information Tool Set (FITS) which wraps several tools, including JHOVE.
http://code.google.com/p/fits/
Years ago, I had a motherboard with a serverworks chipset which had the really fun properrty of corrupting hard drives if DMA was enabled.
I wrote a simple python script that recursed through directories, and generated a md5sum file in each directory, containing the md5sum of each file.
When run again, it compares the current files against the md5sum file and reports differences, including new files, removed files and changed files.
It is GPL3 and works anywhere python 2.6 or newer exists, including windows xp...7 and linux.
It is at http://jdeifik.com/ under "md5sum a directory tree"
Of course, it won't fix broken files, but it will detect all corrupted files (assuming your ran it over good files once).
I've got a laptop running exclusively Ubuntu and the default behavior when the battery gets to 5% is to just hibernate.
As for a validation of files, this is a truly trivial problem to solve...
Yes Francis, the world has gone crazy.
I am, and that was the first place I looked...
Karma: Poor (Mostly affected by lame karma-joke sigs)
Maybe you didn't mean it this way, but dang if I did not see all the PHBs come out from work with your comment. "I can get 1TB Drives from Fryes for $80.00, why do you say it costs several hundred?".
Are you one the the exchange admins that claims that 10,000 emails totalling 500MB is too much to have in your inbox, despite gmail AND hotmail not even sniffing at it (and providing rich-text cross-platfordm email, instant search)?
For enterprise storage I tend to aim high at a capital cost of $500/TB, and running costs of $50/TB/year.
That means purging 300GB saves $20 a year. If it only takes a day that's fast. It also costs the company $800 in lost time. Whoop.
Ubuntu pops up a warning window, and if you ignore it the battery light turns orange, and then red, and then it should hibernate. Flat-out dying is not something I've come across under Ubuntu (and I have some flaky old machines with old batteries, and they still warn me and then shut down).
I have. I have flakey batteries that report 0% when they've still got another 20 minutes or so left. When the warning pops up I often kill gnome-power-manager, save my work, and take my changes.
Sometimes I don't get to power in time.
I also have a problem if I knock my second battery out the CD bay (thinkpad). The machine doesn't like that when on battery power, and just turns off.
mplayer can detect corrupted movie and audio files
find . -name '*.mov' -exec mplayer -msglevel all=6 -speed 100.0 -framedrop -nogui -nolirc -cache 8192 -tskeepbroken -ao null -vo null {} \; | grep Warning! > $1.txt
Change the *.mov as appropriate.
<infomercial>its JUST. THAT. EASY folks!</infomercial>
Yes it is, as the OP was kind enough to tell you advanced mplayer use.
Compare this cut-and-paste job to using "simple" tools.
Click Start
Click Find
type *.mov
wait for the find to finish
double click the top file
watch it
if it breaks, delete the file
double click the next file
continue for all files
The CLI makes complex tasks easy and fast. When someone is kind enough to give you a copy and paste line it's even easier and faster.
How many copies you keep of the data is entirely beside the point.
The point is that a disk full of data from a decade back, is only going to fill 1.5% of a new disc.
Yes, if you keep 2 separate backups plus your primary storage, you're going to need *3* new discs, not one. But the presence of the old data doesn't change this noticeably.
My point is "All my files from the last decade" are always going to be ~2 order of magnitude larger than "all my files from before that", thus deleting the latter is never going to save you more than a tiny percentage.
Even if you keep 100 separate backups, deleting all the decade-old files will *still* only save you 1-2% of your total storage-needs.
Did you even read what you typed before you hit submit?
Send your drive to the FBI, they will scan it for you.