Linux File System Shootout

IpSo_ writes "Finally an extensive, human readable Linux file system benchmark has been unleashed upon us. Originally posted on the Linux Kernel mailing list, using two of the most popular benchmarking tools available, it compares all the major file systems, including their different mount options. The results are surprising."

Comments

human readable ?

[gokulpod]

I am sorry..all I see are numbers floating around. Does someone have a "human readable" summary of this ?

Re:human readable ?

[Mr_Perl]

Green is good (fastest) and red is bad (slowest)

HTH

Re:human readable ?

[arivanov]

The human readable result is you need to know what you want. There is no silver bullet.

It looks like xfs wipes the floor for all but temporary (loads of create/delete) file usage. Jfs looks like the best all-rounder. Reiser looks like something that can be tuned to the specific usage, but eats CPU for breakfast, lunch and dinner and EXT3 "surprise, surprise" sucks rocks. The other "surprise, surprise" is that EXT2 is still very good for many uses.

Frankly, I do not see anything new and fascinating in the results, but they are good to throw at people who keep asking me "why not EXT3" and "Why XFS or EXT2". Here is why!

Re:human readable ?

[Tet]

EXT3 "surprise, surprise" sucks rocks.

Really? You must be looking at a different set of benchmarks to me, because as I see it, ext3 is running a close race with XFS to take second place behind JFS. Remember, ext3's journalled mode is journalling data as well, and hence it isn't fair to compare it to other filesystems directly as it's doing much more work (equally, ext2 comes out on top for a number of things because it's doing far less work). Others like reiserfs, XFS and JFS are journalling metadata only (c.f., ext3's ordered mode).

I tend to run ext3 on all of my servers, because while it's not necessarily the absolute fastest, it's fast enough, and more importantly, it's rock solid in terms of stability. I wouldn't touch reiserfs with a 10 foot bargepole for any of my machines, mostly because I don't trust it (or Hans). Now it seems even the touted performance benefits aren't really there either. I've been considering JFS for a while, and have had a test JFS filesystem running for the last few months. Maybe I'll switch, but even if I don't, ext3 is more than adequate.

Re:human readable ?

[blixel]

eally? You must be looking at a different set of benchmarks to me, because as I see it

So much for the "human readable" aspect of these benchmarks. Everyone seems to be walking away with a different idea of what the results are supposed to be showing.

Since there was no legend explaining what the colors meant, I couldn't figure out anything from looking at them. Is the high number good? As in did the most work? Or is the high number bad? As in took the longest amount of time to do something?

Re:human readable ?

[budgenator]

The other "surprise, surprise" is that EXT2 is still very good for many uses.
especialy if the file system can be mounted read-only; you could do this in partitions like /boot, /opt, /lib, and /usr that hold programs and are not changed often.

I wonder if the kernal version makes a difference, are the xfs and jfs better on the 2.6 as compared to say the 2.4.19 that I'm running now; or how about with files that are much smaller like on the typical web server?

also partioning schemes can make a big difference in overall performance, in the old days i placed the swap partition in the center of the disk (most accessed in the center where the heads are most likely to be) the next most likely like /usr and /var beside the swap so the heads didn't have to move to far to get to them ect.

also does the disk make a difference, such as is any performance differences consistant between scsi and ide drives?

These are things that need to be looked at before make a decision as to which is best, but it definately appears that we need to do some looking now

Re:human readable ?

[Psiren]

Since there was no legend explaining what the colors meant, I couldn't figure out anything from looking at them. Is the high number good? As in did the most work? Or is the high number bad? As in took the longest amount of time to do something?

Depends on the column. For K/sec, higher is better, so red cell shows lowest, and green shows highest. For %CPU, lower is better, so green shows lowest and red shows highest. It's not that complicated really if you take a few minutes to look at it. What you get from the data depends on what you were looking for in the first place.

Re:human readable ?

[TheCrazyFinn]

Never heard of Read-only filesystems?

Mount static filesystems read-only, and make them EXT2 for performance. Use a journalling FS for dynamic filesystems. Reap the benefits of both.

Cheaters!

[borgdows]

NTFS has been removed of the benchmark results because it was the best performer in every test!

Re:Cheaters!

[davFr]

Indeed It would be very interesting to see the results of the microsoft fs supported by linux (fat32, ntfs), as well as more exotic ones (BSD, Netfs, minix, etc).

Re:Cheaters!

[kasperd]

Like I said, who cares about NTFS or FAT performance?

FAT performs well as long as you just do sequential access to large files. But don't access too many files at the same time, because there are only eight entries in the fat_cache. If you run over this limit or do random access FAT is going to be the worlds slowest filesystem. And that is so bad it has really caused me trouble some times.

The reality is minix is faster, cleaner, simpler, and more flexible than FAT. Just take a look on the source minix 47KB, fat 131KB. And keep in mind that minix use a a fast tree structure to locate blocks while FAT use an extremely slow linked list.

Huh?

[bazik]

Woah, looks like JFS performs really well!
Anyone has good/bad experience using JFS?

Hmm... I think I'll setup my test box with JFS...

Re:Huh?

[matticus]

Well, here's IBM's page about it.

From what I've seen poking around USEnet, JFS seems to have the too little, too late problem. I've never seen it pwn a benchmark like it did today though.
I'm a little confused-I have been told XFS is the best designed, highest performing file system, and I would hate to think SGI is getting into a lot of this crap with SCO for a relatively slow journaling file system...

Re:Huh?

[ATitan]

I've got a 120 gig WD disk with JFS in my fileserver at my home network work. It's fast and since it's sharing multimedia, there is a lot of moving around, deleting and copying files around and still it's very fast writing on it ( according to my experience ).

Re:Huh?

[Frodo420024]

I'm a little confused-I have been told XFS is the best designed, highest performing file system, and I would hate to think SGI is getting into a lot of this crap with SCO for a relatively slow journaling file system...

IIRC, XFS is more about guaranteed performance under various stressful conditions than about getting the absolute peak speed in calm conditions.

Re:Huh?

[Anonymous Coward]

Time to rethink things. The generally accepted opinions between the two are that JFS is faster for small files and XFS has a bit of an edge with larger files. Both perform very very well.

I don't know how JFS falls in the "too little to late" catagory, both file systems have been available for a long time on Linux, however very few Linux distributions embrase them during installs so they have gone unknown to a great deal of the non-storage geeks out there. Mandrake, much to their credit, has for a long time included these file systems as an install option for your root filesystem, which has always made me appreicate what Mandrakes doing.

I will say that even though JFS is probly the best high-performance choise for user workstations due to smaller file sizes on average, I prefer XFS because it has a much more robust toolset and you can get alittle more hands on with the filesystem thanks to the tools provided. But most users just don't care that much for low level tools so give JFS a twirl.

Throughput benchmarks only...

[pe1chl]

There is have focus on throughput in these benchmarks. Reading and writing lots of data, seeking in files and reading data, etc.

Notably missing are more day-to-day useful operations such as the creation and deletion of lots of files, parallel action on many open files,
lots of files in a directory, etc.

When I want to select a filesystem, I do not want to know how fast it can read a 3GB file sequentially. I want to know how well it performs on a fileserver, mailserver etc.

Re:Throughput benchmarks only...

[zurab]

Have a look at Hans' benchmarks at namesys.com. Although he only compares Reiser4 to ext3, and may not be an objective party. But I'm surprised how well JFS performed anyway and that Reiser4 is unusually CPU-intensive.

Re:Throughput benchmarks only...

[sql*kitten]

When I want to select a filesystem, I do not want to know how fast it can read a 3GB file sequentially. I want to know how well it performs on a fileserver, mailserver etc.

Horses for courses, my friend. If you are running a database or doing video editing then reading large files is exactly what you do want. This is exactly what SGI's customers do, and that is why XFS is IRIX default filesystem.

Re:Throughput benchmarks only...

[pe1chl]

Well, if you run a mailserver with maildir storage format, you are interested in the performance of directories with 5000 small files.
You want to quickly do readdir operations, quickly open many of the files and read some data from them, etc.

When you run a fileserver and don't want to explain to your users that it is not a good idea to have 100.000 files in a single directory as a storage format for filled-in entry forms, you want that situation to be handled well by the filesystem.

When a nightly backup operation needs to read the tree that includes that directory and write it out to a directory on a different disk (and remove the copy of a few days before), it should be able to do that without spending ridiculous amounts of time or excessively wearing out the diskdrives.

Those are not hypothetical situations, those are situations that I encounter in real life.

Short summary

[mst76]

iozone benchmark
best: jfs
worst: ext3_journal

bonnie++ benchmark
best: ext2
worst: reiser4/reiser4_extents, ext3_ordered/ext3_journal

Re:Short summary

[triptolemeus]

Outperforms all the others, might be a bit over the top. If you look at the numbers, it really doesn't outperform any of the others. It is a good one, in the top ranges, but considering the fact that the outperforming fields are green and not red...

Re:Short summary

[Spy Hunter]

bonnie++ benchmark
worst: reiser4/reiser4_extents

You might think that just based on the amount of red in Reiser4's row, but if you look all the way over to the right, you'll notice something interesting: Reiser4 often completes the benchmark in significantly less time than the other filesystems. Reiser seems to be caching a lot of flak for the CPU usage (certainly it gets a lot of red boxes in this benchmark because of it). Personally, though, I've got CPU to spare. Disk seek times aren't changing drastically anytime soon, unlike CPU speeds. If I can trade some CPU cycles for less wasted disk seek time, I think that's a great trade.

Re:Short summary

[Afrosheen]

Christ, where are my mod points when I need them most, you deserve a +1 Insightful for this.

I totally agree. CPU cycles are alot less important on my box than disk seek times. Then again, I'm guessing that the people this will be most relevant for are those running servers. Mine are all running reiserfs and ext2.

Re:Short summary

[Anonymous Coward]

Reiser4 is now better in some areas, mainly speed. It is not fairing well because it is still unfinished. The Reiser4 team are now focusing on two things before releasing a more final version: increased stability and reduced processer usage (which is what currently kills Reiser4 in benchmarks).

From: Comment 7167683

We allocate a "jnode" per unformatted node in the filesystem. The traversing of these jnodes consumes more CPU than performing the memcpy from user space to kernel space when doing large writes. I don't yet really understand on an intuitive level why this is so, which is a reflection on my ignorance as it is consistent with stories I have heard from other implementors of filesystems who found that eliminating per page structures was an important part of optimizing large writes. We will fix this by creating a new structure called an extent-node that will exist on a per extent basis, and this will probably cure the problem. This will greatly simplify parts of our code for reasons I won't go into, and it will also take us 6 weeks to do it. I don't think users should wait for it, and so we will ship without it.
. . .

Our dbench performance was poor, has improved due to coding changes, and we need to test and analyze again. Perhaps more fixes will be needed, we can't say yet.

* Our fsync performance is poor. We will pay attention to this next year, frankly, after we have fully implemented the transactions API. At that point we will say something like, if you care about fsync performance you should be using the transactions API and/or sponsoring us to tune for NVRAM, users will say back "but our legacy apps on hardware without NVRAM matter!", and we will grudgingly but effectively tune for this because we care about real users too.;-)

Benchmarks can be found at www.namesys.com/benchmarks.html

Re:Short summary

[Hard_Code]

The question is, I think, does CPU load for non-disk-IO related tasks tend to increase when disk IO tends to increase. Is there a correlation? I would argue against it. Typically programs fetch data, and THEN perform operations with it. IMHO, in order to derive a cpu load/disk load correlation, you would apparently have to be doing LOTS of SMALL disk IO while simultaneously using lots of CPU. I don't think many programs operate this way. Many programs access lots of disk in small pieces, and many programs are CPU-bottlenecks, but I don't think the intersection set is all that large (if you are reading tiny amounts of data, you can't possibly being doing THAT much processing on the data right?).

Furthermore, the question becomes "is the CPU usage amortized as filesystem functionality increases". I think this is EXACTLY what we are seeing. Logic is being offloaded from individual applications and bundled into the filesystem (at least with Reiser), as more applications require more sophisticated and database-like filesystems. I think this trend will increase, and we will see more "smart" filesystems being used as databases, and less "dumb-but-fast" filesystems. Sure, applications which don't specifically use the new sophisticated features of the filesystem will see the hit, but I'd wager applications that DO use the new features would actually see a net CPU usage drop, not increase. One immediate example I see is security/ACLs and how they can be embedded in Reiser.

Re:Short summary

[JanneM]

Actually, at least in our case we thread the app, with one thread handling disk IO and other threads handling other aspects (such as CPU intensive stuff), precisely to squeeze out a bit more performance and so disk accesses do not interfere with and stall other stuff. You get this as soon as you try to do something in soft realtime (such as video applications). On one hand, you want to stream video to/from a drive as quickly and efficiently as possible; on the other, you want to do some CPU-intensive operations (filtering, resizing) on the video stream at the same time.

I'm not saying that trading CPU for filesystem speed is a bad idea; it isn't. What I'm saying is that it's not a simple "more is better" function, and that the cutoff for when it no longer makes sense does depend a lot on the application you intend it for. Again, to take an extreme, you would not want to have a system where the filesystem eats so much CPU the rest of the system essentially blocks, starved for CPU time, when the disk is used.

To take an even more extreme way of doing the tradeoff: you could compress and uncompress all data on the fly. That way you would increase transfer speed (and increase it quite a bit in the case of text files and similar) as well as decrease disk usage. It is not often done, though, because the tradeoff is not worth it in general.

For us, and our app, Reiser is on the wrong side of that cutoff point (and Reiser4 is not even on the horizon yet).

Re:Short summary

[Lodragandraoidh]

To me the most important issue is response time. I replaced the standard EXT3 with ReiserFS and have seen a marked improvement when accessing files on disk. I have alot of small files on my workstation - so reiser really works for me.

Now, if I had a few large database files, then I might think of changing to something else - but I don't have that situation on my workstation, and probably never will. To echo what Spy Hunter said, I am willing to trade some CPU cycles for more efficient data retrieval - my cpu is 90% idle most times anyway (when I am not playing video games) so I have cycles to spare.

On my server, on the other hand, I am running Zope, initially using Berkely DB for the database. It is currently running redhat with EXT3 - and this will be the box where I will really need to think about performance in terms of large files as my application grows. I am also running Seti@home on this box, so CPU cycles are at a premium here.

On another note, the report itself was difficult to understand because there was no key that explained the colors or the fields. While it may be 'human readable', it is not 'human decipherable'.

these are narrow tests, not comprehensive tests

[hansreiser]

Still, they are interesting in showing areas of performance where something is a bit amiss.

It would be nice if exactly what they did was explained. You know, things like how you can get both the lowest total elapsed time and the worst overall score on one of the runs (because of CPU usage? ), what task was measured by each of the numbers printed, what the different settings on the different runs mean.....

Sigh, time to go read the source code for them.

Re:these are narrow tests, not comprehensive tests

[mst76]

Still, they are interesting in showing areas of performance where something is a bit amiss.
It would be nice if exactly what they did was explained. You know, things like how you can get both the lowest total elapsed time and the worst overall score on one of the runs (because of CPU usage? ), what task was measured by each of the numbers printed, what the different settings on the different runs mean.....

Just looking at the table of results, it seems clear that the bonnie test has a penalty for cpu utilization, while the iozone test only considers total run time.

Sigh, time to go read the source code for them.

I just hope we're not going to get benchmark-optimized filesystems.

Re:these are narrow tests, not comprehensive tests

[hansreiser]

Think of different benchmarks as being like x-ray vs. infrared photographs. Each of them gives a different insight into the subject of the photo.

In this case, I think that this 200Mhz CPU benchmark is not highly worth optimizing for, but generally more views of a design are interesting.

One of the things reiser4 needs to do is not have a structure per unformatted node for large files, and you can see the need for that if you look at our CPU consumption when writing a large file using dd. We'll probably adjust that aspect of the design sometime in the next two months, and have a structure per extent instead of per unformatted node. If I hadn't been running benchmarks, I never would have understood that misdesign decision of mine so clearly. The nice thing is that the code will get simpler as a result of the change.

Size on disk too?

[NicenessHimself]

Are there any similiar bakeoffs that work out efficiency with regards to different file sizes?

It would be nice if non-Linux filesystems (FATxx, NTFS etc) were also benchmarked.

histogram, please!

[glenkim]

Hey, if somebody could organize this data into histograms, it'd be a lot easier to interpret the results..

Results question

[DaEMoN128]

I see that JFS won in the bonnie test, but EXT2 put up one hell of a fight and won the other roundup. I didnt think EXT2 was a journaling file system. Is it fair to thow in a Non Journaling FS in a benchmark against a bunch of Journaling ones? If it isnt journaling, then I gess Im going with JFS.

If I am wrong, please either resopond to correct me or email me.

scythefwd@yahoo.com

Re:Results question

[NickFortune]

Is it fair to thow in a Non Journaling FS in a benchmark against a bunch of Journaling ones?

Yes. Of course. The ext2 numbers provide a baseline for the comparison comparison. Any journaled FS that could match it would have to be very good indeed. This isn't explicity stated anywhere - but this was posted to the kernel list. They can reasonably be expected to know the difference between ext2 and the rest. It's all data. Data is good.

I know we're used to seeing "benchmarks" used as corporate propaganda, but let's not forget what they're supposed to be used for

DeFacto Standard

[Bios_Hakr]

I'm not trying to be an asshole or a troll; just hear me out.

I love Reiser. I also love Gentoo and adore Debian. Myself and another guy, Joe, are the main "linux geeks" in our computer group (cugy.net). When it came time to decide what to support at our group, we had to choose RedHat.

If I'm in a message board or IRC channel, I need to know some things about the guy I'm helping. We reccomend RedHat because that is the biggest US company behind Linux (IBM and SUN notwithstanding). If I am teaching people about Linux, then it is to both our advantages to teach/learn about what we will see "in the field". Therefore, we only support RedHat.

What does this have to do with anything? Well, RedHat 9 and Severn do not allow the creation of Reiser by default. I could probably boot from a Gentoo disk and format a partition to Reiser, then install RedHat to it. But, by default, only ext* is allowed.

I love to do things that improve performance. I love testing new things on my laptop or on a offline box in our test lab. But unless RedHat offers it, it will remain in the shadows of the linux world, which is, in turn, in the shadows of the user enclave. Hell, of every important box on my network, they are either RedHat or Win2k.

More on topic, Joe got a lot of recognition when the "internet got a lot faster". Did he upgrade the firewall? Did he install another OC-3? Maybe he reconfigured services on the proxy?

Nope, he installed a hard drive, formatted it to Reiser, and moved the proxy cache to the reiser disk. I couldn't belive it. Just changing the filesystem caused an increase that was noticable across our network. At no cost!

Good work, Joe.

Re:DeFacto Standard

[nosfucious]

Um, why would you want to put squid on a journaled file system?

Squid is a cache of (parts of) the internet. It can be rebuilt pretty easily. For example, the next time a user goes to a page. It might cost you a fraction of a second the first time, but after that you're sweet. Journalling transitory data just adds unnecessary overhead.

If it's quite a large cache with a number of binary objects, why don't you just mirror it up front?

A mail spooler or news spooler that has to keep somewhat static data I'd happily put on a journaled file system. However, Squid and things that maintain their own data integrity, (say an sql db), I'd put on Ext2.

If you're looking to restart quickly after a power failure you can always set a partition to ignore file system checks at startup, "0 0" options in /etc/fstab. /var/spool/squid (or whatever) is on its own partition right? Perhaps on it's own disk?

Re:DeFacto Standard

[warpSpeed]

Um, why would you want to put squid on a journaled file system?

For the same reason you would want to have email, or a file server, on a journaled system, recovery speed.

I have some clients with servers (that run squid) and when they take a power hit, long enough to drain thier UPS, the last thing I want to have to do is deal with a call saying "how come the server did not come back up..." Meanwhile the fsck is still running and they are hitting the power switch trying to "reboot" the problem away before the ext2 fsck can finish checking though the cache partition...

Thats just on reason.

Re:DeFacto Standard

[hackstraw]

Um, why would you want to put squid on a journaled file system?

If you're looking to restart quickly after a power failure you can always set a partition to ignore file system checks at startup, "0 0" options in /etc/fstab. /var/spool/squid (or whatever) is on its own partition right? Perhaps on it's own disk?

You have never waited over an hour to fsck 3 harddisks while over 100 people have no "internet".

Re:DeFacto Standard

[drinkypoo]

he installed a hard drive, formatted it to Reiser, and moved the proxy cache to the reiser disk. I couldn't belive it. Just changing the filesystem caused an increase that was noticable across our network. At no cost!

He installed a hard drive. He didn't just format to reiser. The hard drive costs money.

If the proxy cache was formerly on a disk that was also doing other things, it would have sped up no matter what filesystem he used.

You will have to give us more information if you want your claims to have any merit.

Reliability?

[hofer]

We did a lot of testing with various file systems for a product earlier this year. After a couple of terabytes of intensive reads/writes (and a couple of days...) the JFS kernel processes randomly locked up and blocked all disk I/O operations (1.1.0 and 1.1.1 versions). JFS was indeed the fastest of the file systems we tested, but we had to drop it for being unreliable.

I wonder if anyone has some experience with the reliability of the current version?

Re:Didn't jfs do well

[NTmatter]

SCO has never claimed to own JFS. But then again, tomorrow's an entirely new business day, and there's a whole million lines of code that SCO hasn't yet lay claim to. ...is anyone else afraid that SCO will try to use Quantum Mechanical principles to gain ownership of the entire Linux kernel? Now, I am not a quantum physicist, but if they don't show which 50% they claim to own, won't the system be in an undetermined state? Wouldn't that mean that SCO could own both halves of the kernel at the same time?

Perhaps SCO has only lay claim to one line? This would account for the manner in which the number of lines claimed has grown from 80 to a million. This can be explained through the uncertainty principle, and compound error. The one line in question has not been determined, but has a probability of being located within certain files.

Could someone with a better grasp of mathematics please aid in identifying the SCO constant of ownership uncertainty?

Summary

[samj]

Use XFS unless you want to do lots of deletes (as they are slow and expensive) in which case ext2 is probably a better bet since the files are probably temporary (Squid caches for example).

Re:Summary

[samj]

And if you want to use 2.4 kernels without compiling your own then you probably want to consider the 'all rounder', JFS, as 1.1.0 (or thereabouts) has been included since 2.4.20. I have a feeling XFS modifies things which weren't to be touched until 2.6.0 so you'll need a custom kernel for it. While some vendors ship 2.4 kernels with XFS support, I only really care about debian and it only ships the patch.

Re:Didn't jfs do well

[kfg]

You are incorrect. The whole "SCO thing" started with their claim that JFS is, as a UNIX derivitive work, their IP.

It is the core issue of their suit against IBM and everything that has followed out of it.

KFG

Note on ext3

[moZer]

Now, before everyone goes "I knew it! ext3 sux!!!!111!!1", remember that the default mode for ext3 is ordered, and not journal. Compare the numbers for ext3_ordered and ext3_writeback with reiser/xfs/jfs, and you'll see that ext3 is very very close in most cases.

"linux reiserfs"

[bani]

type "linux reiserfs" when booting the installer, and you will have access to reiserfs during redhat install.

i've been using this method for ~2 years now.

Re:"linux reiserfs"

[Mark Wilkinson]

Although it can leave you with a system that the Red Hat installer won't upgrade. If your root partition is actually an LVM logical volume or a RAID device, and it's formatted with reiserfs, the installer won't find your existing system and won't offer the upgrade option.

Here's my $699 now.

[clusterix]

Wow, it looks like SCO has the best filesystems for Linux with JFS and XFS.

Re:i am a human

[1u3hr]

please explain - in lamens terms,

Is that "layman's" or "lamer's"?

Re:Other's don't do journaling?

[altamira]

There is a difference between journaling DATA and METADATA. Don't get confused by that...

Where's the deviation?

[rufusdufus]

Filesystem benchmarks can be remarkably inconsistent. These tables do not display average difference between runs. Usually this means that the methodology used to do the benchmarking is lax, and thus, untrustworthy.

For example, consider that harddrives do their own error correction. Depending on the location of marginal blocks on the media, different file systems can score dramatically for no other reason than the drive's re-mapping or error correction logic is kicking in at a bad point. Alignment of data can also be a factor in performace which depending on the formatting procedure may be completely random when compared to the file system sitting on top of it.
For these reasons and a host of others, it is not reliable to do filesystem performance comparison on a single machine.
Bottom line is that there is a good chance that these data are not fair representations of the relative merits of each filesystem.

Odd

[ThoreauHD]

Why do the benchmarks seem to be completely opposed to the other. The bonnie says reiser4 is faster, and the iozone says jfs is faster, and reiser is the slowest. This isn't making a great deal of sense to me.

Ext3

[rf0]

Well ext3 might suck but when you've only got a resuce system that can read ext2 it can really save your neck. I would be intrested to see what is best in terms of stability though..

Rus

Re:Ext3

[dr.Flake]

Exactly,

I'm also very interested in the ease and ability to repair (or auto-heal) a corrupted file system after a hard crash.

Also, simple undelete would also be on my wish list. just in case.

Anyone here who's got those features ready?

Re:Ext3

[srussell]

Augh. I can't stand it any more.

I had ext3 on my wife's laptop for a while, and it failed twice. By "fail", I mean that, due to Linux crashes, the filesystem had errors that had to be recovered by hand. By "fail", I mean actual, significant data loss.

When I got my new laptop (from QliLinuxPC), they formatted the HD into one big partition (well, one for /boot and one for /), and formatted those as ext3. I didn't switch to ReiserFS, because QliLinuxPC said they'd had good luck with ext3. In the past year, I've had three seperate filesystem corruptions of / on that machine.

On my older laptop (now functioning as a print server), I have had ReiserFS for the past three years, and ReiserFS again on a desktop system for the past 5. I've only had one problem with ReiserFS, and that was three or four years ago, and I don't remember what it was -- although I remember it being a real pain to recover, and I think it involved LVM.

Considering I've tried it on two systems with entirely different hardware components, I'm faulting ext3. My conclusion is that ext3 sucks, but my opinion is based on the fact that, for a journelled filesystem, ext3 seems to be terrible at surviving sudden power failures, and has given me as many, if not more, filesystem failures than ext2 ever did.

Re:Ext3

[Anime_Fan]

While I have experienced most of what you say, I've also had my share of Reiserfs corrupting filesystem. Granted, it was not root partitions, but I had two separate IBM Deathstars (120GB) that both failed within a week (during normal usage, no power failure). I couldn't access certain directories, and a rebuild-tree saved some more files.

That said, I've not had ANY problems with Reiserfs on good hardware (Maxtor 160GB 8MB Cache/Western Digital 120GB 8Mb cache).

I would have used XFS if it wasn't for the fact that the kernel wouldn't mount the damn things ^^. But I'm now stuck with 2x 120GB ext3 partitions.

Is this realistic?

[nemaispuke]

Is it me, or is there a lack of information about the machine the tests were run on such as why is almost all of the memory used up? Second, a system with a single disk and swap is in use? What was this guy trying to test anyhow? All this tests is basically a typical Linux box with a single drive. I wouldn't base any decision to go from one filesystem to another based on these tests!

Re:Is this realistic?

[nemaispuke]

I think you are missing the point. In any test (especially one that is going to come in under scrutiny) all information about the target for the test should be published. There is no information about the OS (other than its Linux), what running software, what tweaks have been applied, etc. Based on what little information is provided, I doubt if anyone could duplicate the results. And the information should be available so that it can be tested independently. Everybody complains about benchmarks, but few actually put up any useful information other than the results. Hardly scientific.

Can't wait for Novell Storage System on Linux

[thehunger]

Personally, I'm happy to wait for yet another file system: Novell Storage System. It certainly is feature packed. Now before you all start banging on me, remember that Novell for years was the king of file system services. Just some of the features:

• Compression and fast decompression
• Hiearchical storage system integration
• Advanced access control model, with granular access control with inheritance and inheritance filters
• Copy on Write
• File system snapshot
• Journaling
• Transaction tracking
• DFS, Junctions and yes! symbolic links!
• Disk, directory and user level quotas
• Fast mount and repair times
• Name spaces for MAC, NFS, NCP
• Native CIFS, NFS, AFP and WebDAV protocol support
• Clustering support
• Software mirroring and RAID0 striping
• Fast! State of the art caching and read-ahead algorithms
• Low memory requirements
• Scalable: 64-bit, 8 terabyte sizes, pooling etc

I could go on... About the only thing it is missing is encryption. Of course it remains to be seen whether the port to Linux will be successful, and whether Novell has the sense to make it open source.

Re:Can't wait for Novell Storage System on Linux

[Anonymous Coward]

You forgot one thing. As an enterprise filesystem Novell was absolutely bulletproof long before RAID systems were in vogue. It took me awhile to even figure out why we needed one after years of running Novell as our main storage controller (flawlessly).

Novell could give *nix systems windows like (don't bash if you don't know) fine granularity over user access at the enterprise level along with true enterprise scaling. Again, if you have never worked in a cross enterprise environment, don't start bashing because you really can't appreciate some of Novells strengths until you need the features.

Re:Can't wait for Novell Storage System on Linux

[kasperd]

Copy on Write and directory quotas are stuff I'd really like to see in Linux filesystems. The combination of hardlinks and directory quotas is however very tricky, any pointers to implementations that made this work in the right way?

This is kinda weird

[insomaniac]

I used to have a gentoo install with a JFS partition for the system on my laptop. The laptop ran so slow that I reinstalled the whole thing with reiserfs, now it runs so much faster, so how could JFS come in so high on the benchmarks while my experience has been that its dog slow?

Irrelevant numbers

[krorvik]

These benchmarks were performed on relatively old hardware, with a slow cpu and a disk only running UDMA2. And, as others have already pointed out, the data are statistically not really reliable.

Myself, I'd be much more interested in seeing numbers made on a setup more like my own.

Static benchmarks are never good for deciding "which is best".

there is more to a filesystem that speed.

[msh104]

every filesystem has its own purpose, for example reiser4 has atomic operations, database like capabilities, journalizing and metadata. now how are you going to say that ext2 is better because it performed better in brenchmark xyz. this is just the same thing as people buying a graphics card because it scored 1 or 2 fps more then an other card but forget that the other card has a build in mpeg2 or for the same price.

Re:there is more to a filesystem that speed.

[LNX Flocki]

Remember that this was a "perfomance test" not a "which FS is better" test. Benchmarks only show you which one is faster at specific tasks, it doesn't necessarily tell you which one's better.

Sort of on topic...

[blixel]

I have a new 200GB hard-drive on the way that will be here any day now. I plan on using this new drive as a storage drive for music, digital camera images, documents, bookmarks, settings, game save data, e-mail messages, backup data, and so on. If WinXP or Linux irreparably crashes on me, this storage drive (and it's mirrored backup) will contain all the data I care about.

I have two different physical drives in this machine now and I dual boot between them. Linux (for just about everything I do) and then WinXP (for things that absolutely require Windows.)

The new drive I'm getting will be hooked up to my machine externally via Firewire. (I don't need help with the external setup. I already have another drive hooked up this way and it works just fine.)

Now my question is - what is the best file system to use for compatibility between Windows XP and Linux. I require full read/write access to this drive whether I'm in Linux or WinXP. I know NTFS is out. (Even with the 2.6 kernel, write support from Linux to an NTFS partition is limited [can't create new files or directories] and Linux NTFS writing is not considered completely safe.)

I'm guessing VFAT is my only option but I thought I would ask around first.

I do have another machine laying around but I don't want to set it up as an NFS/Samba server for a few different reasons. #1. I don't want to leave the machine on 24/7. #2. I don't want to tie up that machine. I like experimenting with new things so if I turned that machine into a full time server, I wouldn't have a test bed machine any more. #3. I don't like NFS.

I have also thought about one of those Network Area Storage systems. Maybe someday, but at this point in time that idea is out too.

Does anyone have any experience with this? What solutions have you come up with?

Re:Sort of on topic...

[Jameth]

If you can afford a little extra partitions, try reiserFS. Although it is not natively a part of windows, there are tools which let you use it. This means you cannot install windows on it, but you could get read-write access with it.

Back when I still dual-booted, I had this layout:

5 gig NTFS WindowsXP partition
5 gig XFS Slackware partition
1 gig swap parition
45 gig reiserFS shared storage partition

This also made me feel a lot safer in using the systems: Neither ever mounted the other system's Root directory, so all that was actually shared was what I used as my home directory. No matter what I did on one system, even if data corruption happened, it would never be so bad that I couldn't boot it. (Note: I never had data corruption, it was just a mental comfort issue)

Re:Sort of on topic...

[angle_mark]

There are some free and some commercial products which can offer full read/write + journalling access for ext3 partitions from Windows. I'd definitely recommend you pick ext3 over fat32.

Some examples..

Free: Explore2fs allows you to read ext2 and ext3. Limited write support is available.

Commercial: Ext2FS Anywhere don't let the name put you off as it has full read/write support for ext2, ext3 and I think reiserFS is supported now too.

file size a problem?

[twitter]

If 30 33.3MB files (bonnie test) are not representative of your needs, please download the scripts. You can then modify the parameters for thousands of 2k files and post the results. Lots of people would be intersted, you know.

Worth Noting

[MajroMax]

It's worth noting here that the benchmarks were all run on files >= 1GB, if I'm reading the table correctly; this stresses the raw IO of the system, and doesn't really take into account the differences in tree-structure between the filesystems.

As for complaints about Reiser's performance -- last I heard, it was more optimized for many small files -- precisely the domain that this thing didn't test.

bonnie++ comparison between reiser4 and ext3

[nikitad]

I should probably add that I am getting quite different bonnie++ results for reiser4 vs. ext3.

They are available at reiser4 benchmarking page along with
hardware specifications.

http://www.namesys.com/benchmarks.html#bonnie++.20 03.09.30

Some suggestions for future tests

[cluge]

These numbers are great, but only tell us a little about reliability or "real world" performance. When I did testing on these file system I used all the benchmarks here, plus a benchmark called postmark. This benchmark utility was released into the public domain by Net App and has to be one of the better "real world" benchmark suites.

The problem that we had with JFS during testing is that we had kernel panic with very large files. Thus we chose XFS - which has done an excellent job. I'm sure glad that the XFS file system has been merged into the 2.6 kernel, no more patching the 2.4's!

For more benchmarks on other file systems using postmark check out This

this benchmark was performed using a 200Mhz CPU

[hansreiser]

which makes the whole thing pretty questionable in my view, especially when you consider that Nikita got completely different results on his more modern hardware (see www.namesys.com/benchmarks.html)

I don't really target 200Mhz CPUs in my performance tuning....;-)

Hans

Re:this benchmark was performed using a 200Mhz CPU

[Deagol]

How true a difference the hardware makes.

I took an old PII-350 w/ 128MB RAM and benchmarked ext2, ext3, jfs, reiserfs, and xfs on an old 5GB IDE drive. ext2 was the winner by a margin (raw throughput).

Now I'm beating up various hardware and software RAID configs on a dual Athlon MP 2200+ system w/ 2GB RAM and dual 3ware 8-port 7500 controllers w/ 180GB WD drives. JFS rises above the rest in terms of throughput (I didn't test XFS on this new machine), and, of course, reiserfs simply spanks everything in terms of file creation/deletions. The thing I noticed was the JFS had much lower CPU utilization for file creations/deletions and was twice as fast at it than the ext2/3 filesystems (it still got spanked by reiserfs, though).

If anyone's interested, the "best" overall was reiser w/ the mount options noatime,notail,nodiratimeall. Also, if anyone cares, on this machine, the Linux software RAID code at no less than twice the performance numbers over the 3Ware hardware RAID. Running RH9 with all RH updates applied.

NTFS read/write in Linux

[sethadam1]

Don't count on this. The write (and maybe read) drivers will always be "experimental." Why?

Because NTFS specs are locked in a dark closet in near Seattle never to be seen by the evil Linux developers. These developers, fearing for their lives, will never have the guts to deem their NTFS write stable - there will always be a slight chance you'll corrupt your entire disk table - and no one wants their so-called "stable" driver to corrupt people's data.

In NT4, NTFS was at version 1.1, aka NTFS 4.0 (to align with NT4.0). In Windows 2000, it was version 3.0, aka NTFS 5. And in XP, it's version 3.1, also known as NTFS 5. The point is, NTFS is a moving target, so it's unlikely we'll see effective NTFS abilities in Linux anytime soon.

Re:The only one that matters

[ktulu1115]

I'm wondering if anyone either has real-world experience or benchmarks on optimium filesystems/partitions for a Linux workstation (the usual: web browsing, games, multimedia, etc)...

I'm thinking about the following (planning on switching back to Linux sometime soon hopefully):

/home - JFS (fast but still has journaling support)
/etc, /usr - ext2 (mostly read-only operations, correct?)
/var - ext3, maybe JFS... still thinking. Like to run a database & webserver but only for my own use (practically zero load)

Any comments?

Well... typical?

[non-poster]

Ok, so where's the "human readable"? There is no analysis, conclusions, or anything else except raw numbers. There are no rankings, no sugestions, just numbers. Also, the test was run on a Pentium II, 450MHz, 512MB of ram and a 6.5GB IDE disk, 5400 rpm, 256kb cache, and 3 heads (=4gb/platter). This is nowhere close to current technology. You can't even buy this hardware any more! I would have much rather seen a 1-1.5GHz CPU with a 40GB hard disk used in the tests. The amount of memory is adequate, though.

Re:Eh.

[drfreak]

Really, those pesky filesystems just get in the way. Just cat file >>/dev/hda and be done with it.

Re:The only one that matters

[Newander]

If you read carefully, you'll notice that the name of the filesystem is SMB. Samba is software that interfaces with the SMB filesystem. Of course, SMB isn't really a filesystem either. When you want to share something you don't have to create and format a partition as SMB.