Data Corrupting ext3 Bug In Latest Linux 2.4.20

An anonymous reader writes "Andrew Morton alerted readers of the Linux Kernel mailing list today that ext3 in the 2.4.20 kernel has a new bug that can easily cause file data corruption at unmount time. The bug will only affect people using ext3 in "data=journal" mode, which fortunately is not the default... Full details can be read on KernelTrap."

Comments

Ob Lame Comment

[Trusty Penfold]

I hope this bug doesn't corrupt the Slashdot datab%(@#LJASLO)aojda2

another victory for open source

[tps12]

Forgive me for gloating, but I'm once again elated at how quickly this bug was squashed. Literally hours after the kernel was released, we have a fix available. Meanwhile, Windorks are still getting hammered by the Klez and ILOVEYOU virii. It's a miracle Linux and less popular open source programs like *BSD haven't wiped out the competition entirely.

Of course, I'm sure some of the more bleeding-edge types were bitten by this buglet, but I guess that comes with the territory; backup backup backup! I hope no Slashdotters lost any of their porn collections.

Re:another victory for open source

[The Bungi]

Yes, remarkable, isn't it?

Even more remarkable is the fact that these stories always somehow fail to make the front page, while every 2-cent obscure vulnerability discovered in Internet Explorer and IIS are shoved front and center.

Slashdot needs a bit more balance in the way it covers things. If this had been a problem with the goddamn filesystem (!) in Windows you'd be seeing 900 posts to the tone of "Hah! M$ sucks!!!1!!".

Sad.

Re:another victory for open source

[MattCohn.com]

At the end of the link....

Andrew Morton wrote:>
> ...
> The fix is to only apply the optimisation to inodes which are operating
> under data=ordered.
>

That "fix" didn't fix it. Sorry about that.

Please avoid ext3/data=journal until it is sorted out.

WELL. It seems that the Open Source people ARE on top of it, but please, don't turn a Linux bug into a way to bash Microsoft. A better comment would have been "Hm. Well, they did screw up but they are fixing it".

Klez and ILOVEYOU all have fixes. A lazy person who doesn't update and patch will have an unsecure system regardless of if it runs Windows, Linux, BSD, Mac OS X, or ANYTHING.

And no, people who run Linux ARN'T smarter and WON'T update more consistantly, they just prefer Linux. And yes, newbies are more likely to be running Windows, but they wouldn't update no matter what OS they are on. And while newbies are more likely to run Windows, Gurus are NOT more likely to run *nix. It's getting old. You like Linux? Great. I'm sure that although things could be better you are very happy with your OS. I run Windows. Great. Although things could be better, I'm very happy with mine.

Re:another victory for open source

[the eric conspiracy]

Slashdot needs a bit more balance in the way it covers things. If this had been a problem with the goddamn filesystem (!) in Windows you'd be seeing 900 posts to the tone of "Hah! M$ sucks!!!1!!".

Oh baloney.

The fact is that the open source development process is just that, open. This means that users have access to versions of the kernel at all stages of development. This build is only a few days old. Clearly everyone should realize the amount of testing is too small for widespread production use.

This kernel, and bug have NOT made it into any significant distributions of Linux. The only people using this version are bleeding edge types and testers who routinely compile their own kernels from source.

If this was a case of, say RedHat 8.0 showing up with a file corruption bug, then, yes, it should be a front page article. This is nothing of the sort. This is a kernel version that might have shown up in Red Hat 8.1, say six months from now had it passed the test of time.

I shudder to think what kinds of problems we would be reporting here if Microsoft gave its customers anything like the same level of access to its development process.

After all, Microsoft is the company that shipped Windows ME and MS Smartphone.

Score: -1, Pro-Microsoft

If this is your typical posting, yes.

Re:another victory for open source

[jsse]

Klez and ILOVEYOU all have fixes. A lazy person who doesn't update and patch will have an unsecure system regardless of if it runs Windows, Linux, BSD, Mac OS X, or ANYTHING.

I'm not going to get into pro-some-OS flame war but I'd like to add one thing that you might have missed in the argument.

The OS that was infected with Klez and ILOVEYOU is a production system.

While the kernel which has fs corruption bug is supposed to be used by non-production, testing environment, and for those you like to use bleeding edge release.

Re:another victory for open source

[Tumbleweed]

You'd seem smarter if you didn't use the non-word 'virii'. The correct plural form is 'viruses'.

And this is hardly a 'victory' for open source. Fixing a bug (or not, as the case apparently is) is never a victory. If they'd been able to put out a version of the kernel without a serious bug, now *that* could be considered a victory.

Re:another victory for open source

[shaitand]

It's been my experience that Windows machines don't like anybody ;) As for klez, I can see a "will not", "will so" battle ahead if I go into it. So let's just agree we've had different experiences on this one.

Re:another victory for open source

[Phexro]

"While the kernel which has fs corruption bug is supposed to be used by non-production, testing environment, and for those you like to use bleeding edge release."

Bzzt. 2.4 is the current stable Linux branch, and 2.4.20 is the latest stable version of that branch.

While this kind of thing is not uncommon in the development branch, it's awful to see in a point release of the stable branch.

Re:another victory for open source

[Vlad_the_Inhaler]

Nah.
Back in the days I used to run computers under Windows, Win95b came out with the vfat32 file system as an option. It was new and an improvement over it's predecessor, but bug free it was not.

Coming closer to home, Linus released a 2.4 kernel a year ago (Thanksgiving 2001? The Turkey kernel) with a major data-corruption bug which was far worse than this one and affected configurations used by the majority. I don't use ext3 like that and can live with this new problem.

Re:another victory for open source

[fudgefactor7]

"And no, people who run Linux ARN'T smarter and WON'T update more consistantly, they just prefer Linux. And yes, newbies are more likely to be running Windows, but they wouldn't update no matter what OS they are on. And while newbies are more likely to run Windows, Gurus are NOT more likely to run *nix. It's getting old. You like Linux? Great. I'm sure that although things could be better you are very happy with your OS. I run Windows. Great. Although things could be better, I'm very happy with mine."

Amen, Brother. I wish more *nix zealots thought like this.

Re:another victory for open source

[Cro Magnon]

I still think FILE CORRUPTION bugs should make the front page! Yes, it's in a raw kernel, but the very people dumb enough to use yesterday's kernel are the ones most likely to read /. And while I don't have much sympathy for anyone who uses a new kernel for PRODUCTION, it should be well publisized for the benefit of home users who are fool enough to use it.

Re:another victory for open source

[Cro Magnon]

Bzzt! 2.4.20 has been released. It's officially the latest "stable" version.

From LKM -- GET MIRRORS PEOPLE!

[fire-eyes]

In 2.4.20-pre5 an optimisation was made to the ext3 fsync function
which can very easily cause file data corruption at unmount time. This
was first reported by Nick Piggin on November 29th (one day after 2.4.20 was
released, and three months after the bug was merged. Unfortunate timing)

This only affects filesystems which were mounted with the `data=journal'
option. Or files which are operating under `chattr -j'. So most people
are unaffected. The problem is not present in 2.5 kernels.

The symptoms are that any file data which was written within the thirty
seconds prior to the unmount may not make it to disk. A workaround is
to run `sync' before unmounting.

The optimisation was intended to avoid writing out and waiting on the
inode's buffers when the subsequent commit would do that anyway. This
optimisation was applied to both data=journal and data=ordered modes.
But it is only valid for data=ordered mode.

In data=journal mode the data is left dirty in memory and the unmount
will silently discard it.

The fix is to only apply the optimisation to inodes which are operating
under data=ordered.

--- linux-akpm/fs/ext3/fsync.c~ext3-fsync-fix Sat Nov 30 23:37:33 2002
+++ linux-akpm-akpm/fs/ext3/fsync.c Sat Nov 30 23:39:30 2002
@@ -63,10 +63,12 @@ int ext3_sync_file(struct file * file, s
*/
ret = fsync_inode_buffers(inode);

- /* In writeback mode, we need to force out data buffers too. In
- * the other modes, ext3_force_commit takes care of forcing out
- * just the right data blocks. */
- if (test_opt(inode->i_sb, DATA_FLAGS) == EXT3_MOUNT_WRITEBACK_DATA)
+ /*
+ * If the inode is under ordered-data writeback it is not necessary to
+ * sync its data buffers here - commit will do that, with potentially
+ * better IO merging
+ */
+ if (!ext3_should_order_data(inode))
ret |= fsync_inode_data_buffers(inode);

ext3_force_commit(inode->i_sb);

_
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Update

[fire-eyes]

In fact, there is a reply to that on LKM:

In fact it was reported on lkml on 18th July IIRC before 2.4.19 was
released if that is any help to you. 2.4.19 and 2.4.20 are affected
and I haven't tested previous releases. I was going to re-report it
sometime, but Alan brought it to light just the other day.

Nick

So it was a dumb idea...

[Ayanami Rei]

JUST DON'T SHUT YOUR SYSTEM OFF! MUWAHAHAHAHAA!!

just kiddin'

Fortunately, this bug didn't make it into 2.5 so it won't be propogated forward. Hint: the quick fix ISN'T a quick fix, it doesn't work.
Either stick with 2.4.19, don't use journaled file data, or sync before umounting (I do that anyway... just superstitious I guess ^_^).

It will take a few days to add some extra magic to the umount logic to flush all buffers in an intelligent way. Hopefully this optimization is worth the effort for dudes with high-uptime.

Re:So I'm clueless

[J'raxis]

Unmounts happen at shutdown. You also need to unmount before scanning/fixing a filesystem. The whole bug here pertains to the fact that it isn't flushing ("syncing") the last 30 seconds of cached data to the disk beforehand. A cold reboot without unmounting could potentially cause all kinds of other data inconsistency problems to pop up.

The temporary fix seems to be to run sync manually. Stick "sync" in your /etc/rc.d/init.d/mountfs (or whatever it's called on your system) script right before the "umount" line.

Re:Most Unsecure OS? Yep, It's Linux

[GreyWolf3000]

From the troll that brought you the *BSD is DYING posts (all 5,425 of them) I'm sure. Okay, I'll bite.

Really though, CERT advisories are inadequate tools for measuring vulnerability. Assuming Linux+apache+ssh, etc., all had equal number of bugs, the number of CERT advisories would be dramatically higher for Linux as opposed to Windows, since Microsoft forces people to hush up when a hole is found, and in the case of Linux, the bugs get reported several times, and the same hole in several distros likely becomes different bugs.

Hence, the article draws a similar conclusion to something like "Our army suffered more casualties than our opponent's army; hence, our opponent is the victor."

Greased Turkey, anyone?

[cperciva]

I think the answer here is simply to avoid any Linux kernels released close to Thanksgiving.

Interesting

[droyad]

I just got a similar report of a bug from a Accounting software vendor alerting us to a bug in Windows.

Apparently in W2k SP1 MS broke something that caused data not to be writen from disk cache to the actual disk, which caused data corruption. This was only fixed in SP3.

I just find it interesting that this bug was not common knowledge as it is not really a "security" issue so they can't hide behind that smoke screen.

Re:Why isn't this on the front page?

[walt-sjc]

Um, maybe because regular non-developer type people don't run out and grab the latest kernel that just came out and compile it themselve for the hell of it. Instead, they run whatever version comes with their distro.

Anyone running the latest bleeding edge stuff keeps up with the LKML anyway, and KNOWS what is going on, way before it would hit a news site like /.

The sky is falling! Sheesh...

Re:Why isn't this on the front page?

[Anonymous Coward]

When 2.4.20 was released, the news made it to the front page. Wouldn't it be appropriate to notify the same people who were notified that this new kernel version was released and ready for download?
I suspect that there are many Slashdot readers who will compile the latest kernel, but who do not read the developers section.
I wouldn't consider 2.4.20 "bleeding edge", as it is the latest kernel in the current stable series, and as such is supposed to be safe for running. "Bleeding edge" would be the latest 2.5 kernel or possibly prerelease kernels in the 2.4 series.
Again, this deserves to be on the front page.

More details and a request for information

[DaveAtFraud]

BTW, if you use ext3 with the default mount options, you will not run into this problem. Its only if you override the mount default of data=ordered and use the data=journal option that the problem even occurs.

Hell, it took me several minutes of searching to even find out what the option was to even cause the problem. Something tells me this won't affect many people. Maybe someone who knows ext3 internals will enlighten us with why someone would want to use data=journal.

Re:More details and a request for information

[Clue4All]

Are you kidding? One of the biggest selling points of ext3 was that is journaled both data and metadata, unlike other journaling filesystems that journal only metadata (that's what this option does, and have been discussed in numerous ext3 articles that have appeared on Slashdot). That's why I use ext3. For Andrew Morton to downplay this like no one uses that mode is a big mistake. Surely the release should be pulled or the patch rolled back or SOMETHING.

Re:More details and a request for information

[DaveAtFraud]

Thanks for the info. Internal details of file systems aren't way up there on my list so I appreciate a concise answer. A couple of things though:

1) Journaling both data and metadata may have been a "selling point" of ext3 but journaling of data is off by default. This isn't a distro decision, that's the way it was described in the write up on the LKML. This could be why Andrew downplayed the impact. It takes some digging to even find out about the journaling options.

2) Unfortunately, most of my experience with a journaling file system has been with reiser. With journaling file systems, my impression is that people ask too much of an operation that is inherently physically limited. Writing the data and writing the meta-data are two separate operations. reiserfs tries to keep small I/O in the journal but ended up with a complicated scheme that fails all too frequently (also, this was more for performance than robustness if I remember correctly). I fear the data=journal option for ext3 has simply demonstrated the same flaw: if you can write the data to the journal, why not write it where it belongs? If the answer is that the journal is simpler and thus faster to write to then you have incurred the complexity of having two separate file systems. You will note that the ext3 error occurred when an optimization was applied to the "data=journal" case that should not have been.

Robust and fast usually are alternatives and are not usually compatible.

Re:So I'm clueless

[iggymanz]

hey, I like that! Like the good old BSD days when you did sync;sync;sync;halt; right after kicking all the (l)users off!