Reliability of Journalling Filesystems Under Linux?

chrysrobyn asks: "Every write-up I see about journalling file systems under Linux discusses efficiency (embedded) or speed (desktop/server). Have any studies been done on reliability? I've used Linux since Slackware 96 (and kernel 2.0.0), and put it on 9 or 10 machines over the years (Slackware on x86 and Debian on PPC), but I've never strayed from ext2. Always, when the uptime gets high, 20-50 days, the filesystems start to get minor fsck errors. Not that I repair the system and expect it to stay live, I just use the fsck -n to help me decide when a repair is in order. Since the same thing has happened on a variety of hardware (386-PII and every interface in between and 601 and 750 processors with Apple hardware), I'm leaning on blaming the ext2 filesystem for these, the slightest of problems. I typically keep my servers up for as long as possible because 95% of my hardware problems have happened during resets and cold power-ups. It's time for my every-other-year rebuild of my personal server, with another on its way, so I was hoping to incite some anecdotal Slashdot conversation on the journalling file systems available for Linux. Personally, I'm most interested in hearing about the file systems supported under Debian stable for ease of administration for this machine which is a 5 hour drive away from home. I've been around the block a few times, so I'm not fearful of patching the kernel with better patches, but I'm respectful of the work the Debian assurance teams have done."

Comments

If you are fscking a live filesystem

[Pathwalker]

you have to expect some errors to show up from time to time, because the filesystem may change while fsck is running, and if so it will not be internally consistant.

Few links for you

[Rubbersoul]

Not knowing the answer to this myself I present to you a few links that may be helpful. Hope this helps.

This link has some good benchmarks of Ext2, ReiserFS and XFS.

And here is a fairly good news group discussion relating to what you are talking about.

/something/ is wrong here.

[Eivind]

ext2 is a quite stable fs. It is not journaling, so crashing at an inconvenient time can lead to an inconsistent fs, but other than that there is no reason why an ext2 fs should magically develop inconsistencies after 3-6 weeks of runtime.

You should be aware that if you are running fsck -n on the fs while it is mounted in rw-mode, then it can and will report inconsistencies which are not real, simply because the fs has changed between the passes in fsck, something which it does not expect.

For this reason, I suggest you try again with remounting the fs in ro-mode before running fsck -n. I am fairly sure you will find that your errors go away. Especially since you state this has happened on diverse hardware and presumably diverse kernels.

That said I would recommend going with a journaling fs for that extra safety that comes from never getting inconsistent even if the power goes out at the worst moment. ext3 and reiserfs are both good, my preference would be ext3 for the simple reason that it can be mounted as an ext2-fs, which means that you will be able to read it with any old rescue-disk or whatever. Reiserfs typically requires you to redo all your rescue-disks, and make sure that your backup-restore-scheme handles it rigth.

If the remounting in ro-mode does *not* make the reported errors in fsck -n go away, and you are somehow able to reproduce this, please report the bug to linux-kernel.

Re:/something/ is wrong here.

[chrysrobyn]

/something/ is wrong here.

I think I may be able to point to the answer.

You should be aware that if you are running fsck -n on the fs while it is mounted in rw-mode, then it can and will report inconsistencies which are not real, simply because the fs has changed between the passes in fsck, something which it does not expect.

I'm doing my fsck -n's in RW mode. From the less file system experienced Linux user's perspective, I wonder what ext2 does when going from RW to RO that cleans up for fsck. I can understand the value of delaying some writes, but shouldn't that get flushed when the box is not active? Would fsck -n work on a RW mounted ReiserFS, JFS, XFS or ext3?

I'm not being argumentative, this sounds like one of those typical Unix behaviors, but learning why may help me with other potential issues.

Re:/something/ is wrong here.

[Zeio]

I have evaluated file systems of late, and wish only to express the need for more attentiveness in one's file system. Being nonchalant about this can lead to "bad situations."

I just finished evaluating JFS 1.0.24 for Linux. My opinion of 1.0.24 and JFS is IBM is doing the port as a courtesy to AIX and OS/2 migrators. It is extremely robust, but slow, 2x slower than XFS or Reiser. I had maximal R/W activity (tar untar create deletes in while loops, Xwin started, downloading via ftp, scp, etc) and power off hot several times, never saw anything but "file system clean."

I am in process or evaluating XFS 1.2pre3. 1.1 XFS for Linux is unreal. It does "everything," it has done it for years, its high performance, has a robust heritage and is all around very good. I have cold killed it, inserted and removed hot swap drives while running, while doing fairly absurd amounts of activity on the test box. Not using this file system is a shame. The release patched kernels, one catering to the Redhat droids and the other is a vanilla with their magic patched in. This isn't a Marcelo kludge either, these are professionals who care greatly in the stability of their product and do a great job in their little cornel of the kernel. The Mandrake and SuSE kernels have this stuff patched in, along with extended attributes and ACLs, and the XFS kernel only has ACL and DMAPI support, and the JFS patches won't apply clean to their kernel, but on thing is true of SGI's version: It actually compiles. The Mandrake 9 and SuSE 8.1 kernels seem not able to compile outside of their proprietary environments. I am upset about this. Typical second tier vendors who fail to bring coherency to fragmented set of projects loosely and informally known as the nebulous "Linux."

EXT3 is a dirty hack (EXT2 with fake journaling). I don't know how EXT3 gets high performance marks - ever - my experience has suggest awful and inconsistent performance with several nasty changes made to e2fsprogs in succession to address potentially severe problems. Its insulting to enterprise customers that RedHat touts this garbage as a journaling filesystem. Reiser is a UFO, and is easily corruptible, and I fail to understand its wide use and early integration in the kernel - my only guess is its simplicity required the least cleaning up of the kludged Linux file system underpinnings. I also get sick to death of Hans blaming everyone and their mother while the guys at XFS and JFS quietly patch away the problems, while Hans whines. Hans did have a good point about the broken RedHat compiler back when it was an issue. I base my opinion of EXTx, and Reiser based on experience. I am appalled, and disappointed at the lack of respect the Linux kernel maintainers have given to XFS. The best of the litter being the last to go in - typical, and Appalling.

UFS+logging on Solaris and UFS+S on FreeBSD are both superior. I have never seen these go haywire. Ever. Interestingly, UFS+S is apparently the 'softcore' journaling method that EXT3 uses, but its far less damageable by empirical determination, and its clearly faster and runs more smoothly. Anytime Veritas appears, which ironically is included in SCO, and is available for Solaris and NT based OSs, things come along quite nicely.

Recently OS X added journaling to the already pathetic HFS+ filesystem. My experience with Mac OS 10.X, including 10.2 has been horrible. I think its inferior, the Mach kernel was deprecated by its progenitors, CMU, in 1994. I think the FreeBSD userland is outdated. I think HFS+ is a pathetic file system and fail to understand why they don't use UFS, but if you have ever tried using it with OS X you know it's not "finished." [defined as: nothing work if UFS is used - don't try and say otherwise] Adding journaling to HFS+ will only slow down an already horrifically bloated and underpowered platform. I find it laughable Apple hardware does not get submitted to www.spec.org, but I have CPU2000 results for PPC 1.25GHz, and of course it is so horrible they can't submit - everything including the SPARC beats it hands down. I also though having to have OS 9 installed on a separate partition as OS X for classic to work properly laughable. I base my deprecation of the Apple efforts on real life experience and objective comparison. I only have to convince myself, but for those who can't easily see where the truth lies on the speed of a Max vs. a PC, my condolences to any significant other you might be lucky to have.

FreeBSD 5. UFS2 will probably be one of the best filesystems to ever see the light of day, and vinum will be there as well.

[I hate Eugenia Dork Loli and her horrible crap "editing" and "journalism," but there are interviews with Steve Best [JFS],Hans Reiser, and Nathan Scott [XFS], held prisoner on OS"News" (more like OSCrapConjecture), very informative; http://www.osnews.com/story.php?news_id=69 ; with some more Journaling info here, http://www.linuxgazette.com/issue55/florido.html showing how Robust XFS is]

When examining the facts, the superiority of XFS becomes clear, and I advocate its use, it's the responsible thing to do. I have recently beaten heavily on a 2.4.19 stock + XFS pre3 of release 1.2 merged in. I can tell you my experience with the Dell 1650 and constant filesystem abuse that the filesystem is that last thing I would worry about in that kernel. I am eagerly awaiting the release of the 2.4.20 kernel, typically long over due as we seem to have an absentee maintainer that rarely speaks, however, upon its release I believe the XFS 1.2 stable will be merged in or completed and I will have a configuration good to go for use on the order of years.

While I may have harsh words from certain practices and sometimes people, I find XFS and the 2.4.19 kernel to be acceptably stable. I ran that 1650 through the washing machine fairly rigorously, and besides the idiotic spurious " Warning - running *really* short on DMA buffers" errors (which caused a flame war on LKML), it seems to be a useful kernel. The RedHat 2.4.18-17.7x kernel, by the way, is the worst most untested pile I have ever seen. What is wrong with these people? Several net drives with no working promiscuous mode, kernel panics, the list is endless.

old hardware

[wotevah]

I had a couple of these systems, they would develop filesystem errors out of nowhere. It always turned out to be faulty old hardware (memory, cables, motherboard etc). The PC components get old really fast, my plan has been to get new hardware at least every three years and get real server hardware (ServerWorks mobo etc).

For the last series, I have not noticed any unexpected filesystem errors after 200-300 days of uptime (they need to be rebooted from time to time for kernel upgrades).

To conclude, always suspect your hardware first, especially if it's at least a couple years old.

Ext3 vs ReiserFS

[DarkDust]

I don't know about XFS and other journalling fs's since I've only used ReiserFS and Ext3 so far.

My experience so far is that Ext3 is more reliable (read: repairable) than ReiserFS simply due to the fact that Ext3 is a kind of "extension" to Ext2, so you can just run the good old well tested and known to work fsck.ext2 on a Ext3 partition should it screw up.

But I have yet to see a Ext3 partition screwing up, I've set up several PCs and servers with Ext3 and it works fine, no single problem to date.

Unlinke ReiserFS. I have to admit, my only experiences with ReiserFS were about one and a half years ago or so, but at that time I had set up a home PC with ReiserFS and somehow I f***ed it up beyond all repair. I don't remember what I did then but I just got scared of ReiserFS :-)

On the other hand I have still another home PC, running SuSE Linux 7.2 updated to 7.3 with ReiserFS which just runs fine, and this is my home server, running 24-7.

So I guess until you don't do anything stupid like I did both ReiserFS and Ext3 are pretty reliable today, given their widespread use you would probably have heard of any major glitches/problems ;-) The decision whether to use on or another is more performance/religion-wise, IMHO :-)

Re:Ext3 vs ReiserFS

[GigsVT]

Well I can fill in your gaps on XFS. XFS has run flawlessly for me for over a year now, on a 1.9TB RAID volume. EXT2/3 seems to have rough edges in regard to large file systems, for example, by default mkfs will set aside 5% of your disk for root use. That's 97GB wasted! Another problem is that you need to specify -T largefile4 or it will try to create way too many inodes, taking forever to create the filesystem, or fsck if you ever need to.

XFS is a very mature file system, and file systems that are many TB work fine with its defaults. Performs more consistantly too. EXT2/3 was very sensitive to RAID stripe size, and things like that. Even setting the special stride option, you had to recreate the filesystem many times to make sure things worked right. XFS performed consistantly at any stripe size, with no strange dips in performance if boundaries didn't line up just right.

In all, if you are building a large RAID, I would go with XFS. For day-to-day use of 200GB or less on a single disk, EXT2/3 is fine. (You probably still don't want to let it waste 5% of the disk, that is such a retarded default, use -m 1 to help reduce it)

Re:There's more (thanks for crediting me)

[Per Wigren]

Yes, this is a known troll but I still want to comment on this particular line:

On other unices, crashes usually are caused by external sources like power outages. Crashes in Linux are a regular thing, and nobody seems to know what causes them, internally. Linux advocates try to hide this fact by denying crashes ever happen. Instead, they have frequent "hardware problems".

Crashes in Linux are NOT a regular thing, unless you want to be extremly bleeding edge and/or use NVidia's drivers and/or ALSA (at least up to 0.90rc5) on 2.4 with lowlatency- and preemptive-patches. Especially if the above stuff are used on SMP-systems.

My system used to crash (freeze) frequently (every 2nd or 3rd day).. But after I sold my GeForce4-card and got a Matrox G450 instead, and switched back to using OSS instead of ALSA (I've got a SB Live..), I've not had a single crash! It has been running for several months without a single reboot, and everything is super-stable! I've used it heavily every day, burnt more than 150 CD-Rs, been on Direct Connect and Freenet 24/7 etc.. That's despite I run the heavily patched 2.4.19-gentoo-r8 kernel, and my whole system (including the kernel) is compiled with gcc 3.2 "-march=athlon-mp -O3 -mfpmath=sse -pipe"..

So my conclusion is: Linux IS stable! Extremly stable! The cause of 99% of the "linux crashes!"-bullshit is because of NVidia's crap-drivers (fast but unstable) and drivers still not "preemtive"-safe (ALSA on SMP for example).. But those things are not used on servers anyway.

And about the "hardware problems": Yes, you DO get hardware problems MUCH MUCH more often on cheap PCs than on multi-million-dollar Unix-servers from Sun/HP/IBM.. Cheap PCs uses the cheapest-of-the-cheapest variant of all components to cut down the price. Expensive Unix-servers use expensive components and have a lot of redundancy, so you don't have to have downtime just because a CPU, a harddisk som RAM or something else failed.

No problems with Reiser

[inkfox]

I'm afraid you won't get much more than anecdotal evidence on this question. Here's mine.

We had a bad network adapter which would fail when other DMA devices were busy. This meant that whenever disk I/O was heavy, using the network adapter was likely to cause a complete system lockup. This took a while to diagnose as the problems took upward of two weeks to reproduce.

Despite the equivalent of having the power cable yanked randomly a dozen times when the machine was at its busiest, we never had a single problem with Reiser. The file which was being written to existed as the old version, and there wasn't even a lengthy fsck. Integrity was 100%.

ext3 on woody

[Anonymous Coward]

I've been running Debian3.0 with ext3 on a few machines here. None of them have extremely high loads, but I haven't had any problems. For me it seems the biggest advantage of ext3 is that if something happens you can always use something without ext3 support and mount it as ext2.

ext3 is simple to install or uninstall. I promise.

[Cecil]

Aside from everybody telling you "that shouldn't happen, you're doing something wrong", which is probably true, I just wanted to chime in with my support of ext3. I think you're making a mountain out of a molehill.

You obviously haven't looked very closely into ext3, because it's an extremely simple layer on top of a standard ext2 filesystem. Essentially, all it is, is an extra file in /, a daemon to do journalling, and a bit or two toggled on the disk itself.

the FAQ has one question that lists the two steps required to install a journal on a stock ext2 filesystem (provided you've got a 2.4.16+ kernel, or have patched your older kernel).

Not only is it very simple to install, but it's very simple to uninstall too. Blindingly easy, in fact. Mount your filesystem as ext2. Done. No journal. If you want to do it permanently, there's an answer about that in the FAQ too.

So really, you have nothing to lose by trying ext3. I've had 0 problems with it, and I use it on a laptop that gets a lot of abuse WRT being turned off at random times (I can't view my battery level in Linux, but I can in Windows. Thanks broken ACPI BIOS...)

The only downside is that the filesystem will sync every 5 seconds or so, which completely destroys any possibility of ever letting the disks spin down for power saving, but that's more of a laptop issue than a server issue.