Correcting ext3 File Corruption?

An anonymous reader asks: "I am looking for ext2/ext3 expert. I have a small file (1395 bytes) that appears HUGE when runing ls -l (70368744179059 bytes [yes, that's 70 terabytes]). This causes a problem because tar wants to back up all those extra bytes. We have back ups of the file else where, but I'm afraid to delete it. When I remove it what is going to happen to the file system (Kernal version is 2.4.18 on i686). This seems to be a pretty bad math error on the part of the file system. This is a really weird error, but could just be the issue of a corrupted sector on the drive. Has anyone else seen this before and have any ideas as to whether such files can be recovered? Is this problem just a small glitch or an omen of an impending filesystem crash?

"Here's what the files look like on the system:

[ root@secure parse]# ls -l HTMLFrameSet.class
-rw-rw-r-- 1 root devel 70368744179059 Mar 20 09:05 HTMLFrameSet.class

[root@secure parse]# wc HTMLFrameSet.class
15 58 1395 HTMLFrameSet.class

...and the error message from tar:

tar: HTMLFrameSet.class: File shrank by 70368744169331 bytes; padding with zeros

No wonder my backups didn't finish! :-)"

Comments

And when you run "fsck"?

[Zocalo]

Since EXT3 is just EXT2 with a journal tacked on, there is no reason why you can't run the EXT2 fsck utility accross it in the normal way. You are obviously worried about loosing the entire file system, so you probably want to start by running fsck with the verbose (-V) and interactive (-r) options to see exactly what is going on and have the ability to prevent unwanted changes being made.

Since you appear to use tar for backups, you could also backup the affected filesystem using the exclude (-X [filename]) option first, which might be a *really* good idea. ;)

Re: And when you run "fsck"?

[Omniscient Ferret]

I'd like to add two things:
You can backup the drive image too, so if the file is irreplaceable and corrupted, you can try more than one recovery method safely.
Also, to fsck /, "touch /forcefsck" and reboot.

Re:And when you run "fsck"?

[Linux_ho]

I'd like to add that fsck is ext3-aware. If the journal looks OK, it might not actually check the filesystem unless you tack on the -f option to force the issue.

Before you try to recover....

[bartjan]

Make a copy of the /dev device itself, if you have the space for that on another partion.
Then use that backup-file to try out whatever other posters here suggest.

Have you contacted SCT? The Creator of Ext3?

[LWolenczak]

Have you contacted SCT? The Creator of Ext3?

dd

[Yarn]

As you know how long it is supposed to be:

dd if=[file] of=[new file] bs=1 count=[length]

I strongly suggest rebuilding the affected filesystem, that kinda weirdness can be indicative of deeper problems.

"make test" in Perl builds used to do this..

[pedro]

IE: creating a nonexistent HUGE file that normal measures would not delete.
Try this:
cat /dev/null > /pathto/peskyfile
worked for me in vanilla ext2.
Should (?!?) work in ext3.

Re:"make test" in Perl builds used to do this..

[unitron]

For a moment I misread that as Internet Explorer having created a nonexistent file, and as I'm constantly mistaking IE for (File) Explorer (giving two different programs the same name may not be Gate's greatest sin, but it's right up near the top of the list) it came as no suprise to me that such a thing could happen.

Sparse file?

[Tony-A]

from man tar
-S, --sparse
handle sparse files efficiently

I'm not really familiar with them, but haven't seen any other mention here.
I know it's possible to put a file on a floppy that won't fit on your hard drive.

Re:Sparse file?

[Kredal]

cp would probably preserve the mega-huge filesize, so you wouldn't get anywhere. cat file > newfile *should* ignore all the empty space after the 1500 or so bytes that he wants to keep.

Of course, I have no way to test this right now, so I leave it as an exercise to the reader. (:

hex

[Merlin42]

I don't know enough about filesystems to say what the implications are but:
the reported size in hex is
0x400000000573
and the actual size in hex is
0x573

Looks like a single extra bit got flipped when the size was stored.

This is a sparse file....

[weave]

It has holes in it. We once ran a medical package 10 years ago that did this on purpose. A 40 gig file took about 4 megs on disk.

This is easy to simulate by writing a small program that scribbes a few bytes to offset zero, then does an fseek out to some insane high offset, then scribble a few bytes there. Close, do an ls, see the huge file, but then note it only takes the space of two blocks on your file system. Imagine the fun you can have with this trick at parties!

Every UNIX file system I've ever dealt with handles this the same way.

tar and other programs should have switches to deal with sparse files correctly.

If you're concerned about what's in it, cat it to od. I believe od is smart enough to collapse zero blocks in its display. That way you can see if there is any real data at some pointer far into the file.

If this is a commercial closed-source package where you can't verify what it's doing, I'd strongly suggest leaving it alone and contacting vendor to see if this behavior is normal.

Re:This is a sparse file....

[ivan256]

While what you say about sparse files is generally true, that's probably not what this is. This is probably a single bit error on this guy's hard drive. There's probably more of them, but he noticed this one because it popped up in a noticeable location. The hard drive is probably on the way out, or he's got some faulty memory (if it's ECC, otherwise this could just be a fluke).

Tar does deal with sparse files correctly, and if this were one, he wouldn't be having trouble.

Re:This is a sparse file....

[n9hmg]

He demonstrated that it was not a sparse file, by using the wc command on it. A sparse file treats all the empty space as nulls on reading, so he would have gotten the big size if it were sparse. It's a single-bit error, probably bad media that got past the ECC on the drive, but maybe just a plain corruption. I'd suggest copying it somewhere safe and running an fsck, if you can afford the downtime.

Try the mailing list

[Outland Traveller]

Why don't you try the ext3 mailing list instead of Ask Slashdot? I lurk on the list and I've seen a number of questions extremely similar to yours, with answers. The list gurus will even help you track down the problem.

https://listman.redhat.com/pipermail/ext3-users/20 02-July/thread.html#383

another ext3 question

[superid]

I know this isn't an ext3 help channel...but I haven't gotten a satisfactory answer elsewhere (usually it just consists of a "*shrug*" on the #redhat channel)

I've got a thinkpad running RH 7.3 with two ext3 partitions. Being a laptop it has occasionally had its batteries die and been shutdown improperly. Invariably, there has been a subsequent long fsck .... long....like 10 minutes....once I even was dropped to the maintenance shell to run it manually (yes, yes, yes, yes, yes, yes, yes...when the hell would I *NOT* want to fix the non-zero dtime????)

Isn't the whole point of ext3 so I don't have to go through this pain? This was an extremely generic installation of 7.3, why am I seeing no benefit to ext3?

Thx,

SuperID

Re:another ext3 question

[superid]

Yes, ext3 was (incorrectly) a module rather than statically in the kernel........thx to everyone !

Lossy compression

[MrResistor]

What was that lossy compression scheme mentioned a while back? lzip, I think? Sounds like that's what you need here...

Just delete it.

[Tom7]

Oh, come on, be a man. Backup if you need to, and delete the thing.

Re:EXT3 has failed me as well.

[crisco]

A developer I'm working with on a RH7.3 system had the following to say after seeing the ext3 filesystem perform a fsck after a dirty shutdown:

I looked into the kernel and noticed the ext3 module wasn't statically compiled into the kernel!

No work yet on whether or not that solved things...

Re:EXT3 has failed me as well.

[haplo21112]

I too have been running ext3 since it became part of the kernel, actually since it was fairly stable before it got into the kernel proper, around 2.4.12 I think...I have not once had an issue with since I began using it, and I have the machine go down when power goes out at least once every month or two...ext3 even saved my ass, when after installing a new drive and then having the controller decide on the first boot after install that it didn't like the drive. I had already put a large amount of data on the drive before that boot...the ssystem came up spewed errors, all over...I rebooted put it on a different controller in the system, all came up well, ext3 recoved from the journal and all was well, NO data loss...I asked a firend if the same would ahve happened with ext2 and he said he doubted it...the journal was the saving grace in this situation.

Deletion question

[obtuse]

I think you're right about the flipped bit. Copy the file with dd, specifying the right output size.

I'd bet there are problems with the whole filesystem, but to continue with what he asked:

It seems to me that he should be able to rm the file without any worries, after making a good copy. Only the inode that points to the falsely enlarged file will be removed, and the data blocks won't be touched, right?

If there is other data in the misallocated blocks, that dat should either have its own references, or it's already as good as deleted anyway.

Re:another solution

[unitron]

Apparently one man's irony is another man's flamebait.

Re:EXT3 has failed me as well.

[unitron]

"...and my wife is too cheap to let me get a UPS..."

Is she to cheap to let you get life insurance? Medical? Comprehensive on the car? If not, explain to her that protection from data loss or not having to reboot after a power failure or glitch is just a fringe benefit, the real reason for the UPS is that it protects your expensive-to-replace electronic equipment from damage due to the electrical, thermal, and mechanical shock caused by glitchy power.

You can probably convince her that you need a second one for the TV and VCR.

funny.

[GiMP]

With reiserfs, I had a file that would reboot the system if I read, wrote, or deleted the file. I rebuilt the journal and everything was ok. Imagine if it was a production system!

stick with a real filesystem, get a Sun, HP, IBM, or SGI and use their journaling filesystems.. you'll never want to use ext* again.

Re:that one is EASY to fix...

[Kredal]

I back up my entire network nightly to /dev/null. It takes almost no time at all, and I don't otherwise use the 'mv' command nearly enough.

I read on the interweb that that's how you're supposed to do it... They wouldn't lie to me, would they?