Linux File System Shootout

IpSo_ writes "Finally an extensive, human readable Linux file system benchmark has been unleashed upon us. Originally posted on the Linux Kernel mailing list, using two of the most popular benchmarking tools available, it compares all the major file systems, including their different mount options. The results are surprising."

Comments

human readable ?

[gokulpod]

I am sorry..all I see are numbers floating around. Does someone have a "human readable" summary of this ?

Re:human readable ?

[arivanov]

The human readable result is you need to know what you want. There is no silver bullet.

It looks like xfs wipes the floor for all but temporary (loads of create/delete) file usage. Jfs looks like the best all-rounder. Reiser looks like something that can be tuned to the specific usage, but eats CPU for breakfast, lunch and dinner and EXT3 "surprise, surprise" sucks rocks. The other "surprise, surprise" is that EXT2 is still very good for many uses.

Frankly, I do not see anything new and fascinating in the results, but they are good to throw at people who keep asking me "why not EXT3" and "Why XFS or EXT2". Here is why!

Re:human readable ?

[blixel]

eally? You must be looking at a different set of benchmarks to me, because as I see it

So much for the "human readable" aspect of these benchmarks. Everyone seems to be walking away with a different idea of what the results are supposed to be showing.

Since there was no legend explaining what the colors meant, I couldn't figure out anything from looking at them. Is the high number good? As in did the most work? Or is the high number bad? As in took the longest amount of time to do something?

Re:human readable ?

[budgenator]

The other "surprise, surprise" is that EXT2 is still very good for many uses.
especialy if the file system can be mounted read-only; you could do this in partitions like /boot, /opt, /lib, and /usr that hold programs and are not changed often.

I wonder if the kernal version makes a difference, are the xfs and jfs better on the 2.6 as compared to say the 2.4.19 that I'm running now; or how about with files that are much smaller like on the typical web server?

also partioning schemes can make a big difference in overall performance, in the old days i placed the swap partition in the center of the disk (most accessed in the center where the heads are most likely to be) the next most likely like /usr and /var beside the swap so the heads didn't have to move to far to get to them ect.

also does the disk make a difference, such as is any performance differences consistant between scsi and ide drives?

These are things that need to be looked at before make a decision as to which is best, but it definately appears that we need to do some looking now

Re:human readable ?

[Psiren]

Since there was no legend explaining what the colors meant, I couldn't figure out anything from looking at them. Is the high number good? As in did the most work? Or is the high number bad? As in took the longest amount of time to do something?

Depends on the column. For K/sec, higher is better, so red cell shows lowest, and green shows highest. For %CPU, lower is better, so green shows lowest and red shows highest. It's not that complicated really if you take a few minutes to look at it. What you get from the data depends on what you were looking for in the first place.

Re:human readable ?

[TheCrazyFinn]

Never heard of Read-only filesystems?

Mount static filesystems read-only, and make them EXT2 for performance. Use a journalling FS for dynamic filesystems. Reap the benefits of both.

Cheaters!

[borgdows]

NTFS has been removed of the benchmark results because it was the best performer in every test!

Re:Cheaters!

[davFr]

Indeed It would be very interesting to see the results of the microsoft fs supported by linux (fat32, ntfs), as well as more exotic ones (BSD, Netfs, minix, etc).

Throughput benchmarks only...

[pe1chl]

There is have focus on throughput in these benchmarks. Reading and writing lots of data, seeking in files and reading data, etc.

Notably missing are more day-to-day useful operations such as the creation and deletion of lots of files, parallel action on many open files,
lots of files in a directory, etc.

When I want to select a filesystem, I do not want to know how fast it can read a 3GB file sequentially. I want to know how well it performs on a fileserver, mailserver etc.

Re:Throughput benchmarks only...

[zurab]

Have a look at Hans' benchmarks at namesys.com. Although he only compares Reiser4 to ext3, and may not be an objective party. But I'm surprised how well JFS performed anyway and that Reiser4 is unusually CPU-intensive.

Re:Throughput benchmarks only...

[pe1chl]

Well, if you run a mailserver with maildir storage format, you are interested in the performance of directories with 5000 small files.
You want to quickly do readdir operations, quickly open many of the files and read some data from them, etc.

When you run a fileserver and don't want to explain to your users that it is not a good idea to have 100.000 files in a single directory as a storage format for filled-in entry forms, you want that situation to be handled well by the filesystem.

When a nightly backup operation needs to read the tree that includes that directory and write it out to a directory on a different disk (and remove the copy of a few days before), it should be able to do that without spending ridiculous amounts of time or excessively wearing out the diskdrives.

Those are not hypothetical situations, those are situations that I encounter in real life.

Short summary

[mst76]

iozone benchmark
best: jfs
worst: ext3_journal

bonnie++ benchmark
best: ext2
worst: reiser4/reiser4_extents, ext3_ordered/ext3_journal

Re:Short summary

[Spy Hunter]

bonnie++ benchmark
worst: reiser4/reiser4_extents

You might think that just based on the amount of red in Reiser4's row, but if you look all the way over to the right, you'll notice something interesting: Reiser4 often completes the benchmark in significantly less time than the other filesystems. Reiser seems to be caching a lot of flak for the CPU usage (certainly it gets a lot of red boxes in this benchmark because of it). Personally, though, I've got CPU to spare. Disk seek times aren't changing drastically anytime soon, unlike CPU speeds. If I can trade some CPU cycles for less wasted disk seek time, I think that's a great trade.

Re:Short summary

[Afrosheen]

Christ, where are my mod points when I need them most, you deserve a +1 Insightful for this.

I totally agree. CPU cycles are alot less important on my box than disk seek times. Then again, I'm guessing that the people this will be most relevant for are those running servers. Mine are all running reiserfs and ext2.

Re:Short summary

[JanneM]

Actually, at least in our case we thread the app, with one thread handling disk IO and other threads handling other aspects (such as CPU intensive stuff), precisely to squeeze out a bit more performance and so disk accesses do not interfere with and stall other stuff. You get this as soon as you try to do something in soft realtime (such as video applications). On one hand, you want to stream video to/from a drive as quickly and efficiently as possible; on the other, you want to do some CPU-intensive operations (filtering, resizing) on the video stream at the same time.

I'm not saying that trading CPU for filesystem speed is a bad idea; it isn't. What I'm saying is that it's not a simple "more is better" function, and that the cutoff for when it no longer makes sense does depend a lot on the application you intend it for. Again, to take an extreme, you would not want to have a system where the filesystem eats so much CPU the rest of the system essentially blocks, starved for CPU time, when the disk is used.

To take an even more extreme way of doing the tradeoff: you could compress and uncompress all data on the fly. That way you would increase transfer speed (and increase it quite a bit in the case of text files and similar) as well as decrease disk usage. It is not often done, though, because the tradeoff is not worth it in general.

For us, and our app, Reiser is on the wrong side of that cutoff point (and Reiser4 is not even on the horizon yet).

Re:Huh?

[matticus]

Well, here's IBM's page about it.

From what I've seen poking around USEnet, JFS seems to have the too little, too late problem. I've never seen it pwn a benchmark like it did today though.
I'm a little confused-I have been told XFS is the best designed, highest performing file system, and I would hate to think SGI is getting into a lot of this crap with SCO for a relatively slow journaling file system...

these are narrow tests, not comprehensive tests

[hansreiser]

Still, they are interesting in showing areas of performance where something is a bit amiss.

It would be nice if exactly what they did was explained. You know, things like how you can get both the lowest total elapsed time and the worst overall score on one of the runs (because of CPU usage? ), what task was measured by each of the numbers printed, what the different settings on the different runs mean.....

Sigh, time to go read the source code for them.

Results question

[DaEMoN128]

I see that JFS won in the bonnie test, but EXT2 put up one hell of a fight and won the other roundup. I didnt think EXT2 was a journaling file system. Is it fair to thow in a Non Journaling FS in a benchmark against a bunch of Journaling ones? If it isnt journaling, then I gess Im going with JFS.

If I am wrong, please either resopond to correct me or email me.

scythefwd@yahoo.com

Re:Results question

[NickFortune]

Is it fair to thow in a Non Journaling FS in a benchmark against a bunch of Journaling ones?

Yes. Of course. The ext2 numbers provide a baseline for the comparison comparison. Any journaled FS that could match it would have to be very good indeed. This isn't explicity stated anywhere - but this was posted to the kernel list. They can reasonably be expected to know the difference between ext2 and the rest. It's all data. Data is good.

I know we're used to seeing "benchmarks" used as corporate propaganda, but let's not forget what they're supposed to be used for

Re:Huh?

[Frodo420024]

I'm a little confused-I have been told XFS is the best designed, highest performing file system, and I would hate to think SGI is getting into a lot of this crap with SCO for a relatively slow journaling file system...

IIRC, XFS is more about guaranteed performance under various stressful conditions than about getting the absolute peak speed in calm conditions.

DeFacto Standard

[Bios_Hakr]

I'm not trying to be an asshole or a troll; just hear me out.

I love Reiser. I also love Gentoo and adore Debian. Myself and another guy, Joe, are the main "linux geeks" in our computer group (cugy.net). When it came time to decide what to support at our group, we had to choose RedHat.

If I'm in a message board or IRC channel, I need to know some things about the guy I'm helping. We reccomend RedHat because that is the biggest US company behind Linux (IBM and SUN notwithstanding). If I am teaching people about Linux, then it is to both our advantages to teach/learn about what we will see "in the field". Therefore, we only support RedHat.

What does this have to do with anything? Well, RedHat 9 and Severn do not allow the creation of Reiser by default. I could probably boot from a Gentoo disk and format a partition to Reiser, then install RedHat to it. But, by default, only ext* is allowed.

I love to do things that improve performance. I love testing new things on my laptop or on a offline box in our test lab. But unless RedHat offers it, it will remain in the shadows of the linux world, which is, in turn, in the shadows of the user enclave. Hell, of every important box on my network, they are either RedHat or Win2k.

More on topic, Joe got a lot of recognition when the "internet got a lot faster". Did he upgrade the firewall? Did he install another OC-3? Maybe he reconfigured services on the proxy?

Nope, he installed a hard drive, formatted it to Reiser, and moved the proxy cache to the reiser disk. I couldn't belive it. Just changing the filesystem caused an increase that was noticable across our network. At no cost!

Good work, Joe.

Re:DeFacto Standard

[drinkypoo]

he installed a hard drive, formatted it to Reiser, and moved the proxy cache to the reiser disk. I couldn't belive it. Just changing the filesystem caused an increase that was noticable across our network. At no cost!

He installed a hard drive. He didn't just format to reiser. The hard drive costs money.

If the proxy cache was formerly on a disk that was also doing other things, it would have sped up no matter what filesystem he used.

You will have to give us more information if you want your claims to have any merit.

Summary

[samj]

Use XFS unless you want to do lots of deletes (as they are slow and expensive) in which case ext2 is probably a better bet since the files are probably temporary (Squid caches for example).

"linux reiserfs"

[bani]

type "linux reiserfs" when booting the installer, and you will have access to reiserfs during redhat install.

i've been using this method for ~2 years now.

Here's my $699 now.

[clusterix]

Wow, it looks like SCO has the best filesystems for Linux with JFS and XFS.

Where's the deviation?

[rufusdufus]

Filesystem benchmarks can be remarkably inconsistent. These tables do not display average difference between runs. Usually this means that the methodology used to do the benchmarking is lax, and thus, untrustworthy.

For example, consider that harddrives do their own error correction. Depending on the location of marginal blocks on the media, different file systems can score dramatically for no other reason than the drive's re-mapping or error correction logic is kicking in at a bad point. Alignment of data can also be a factor in performace which depending on the formatting procedure may be completely random when compared to the file system sitting on top of it.
For these reasons and a host of others, it is not reliable to do filesystem performance comparison on a single machine.
Bottom line is that there is a good chance that these data are not fair representations of the relative merits of each filesystem.

Ext3

[rf0]

Well ext3 might suck but when you've only got a resuce system that can read ext2 it can really save your neck. I would be intrested to see what is best in terms of stability though..

Rus

Can't wait for Novell Storage System on Linux

[thehunger]

Personally, I'm happy to wait for yet another file system: Novell Storage System. It certainly is feature packed. Now before you all start banging on me, remember that Novell for years was the king of file system services. Just some of the features:

• Compression and fast decompression
• Hiearchical storage system integration
• Advanced access control model, with granular access control with inheritance and inheritance filters
• Copy on Write
• File system snapshot
• Journaling
• Transaction tracking
• DFS, Junctions and yes! symbolic links!
• Disk, directory and user level quotas
• Fast mount and repair times
• Name spaces for MAC, NFS, NCP
• Native CIFS, NFS, AFP and WebDAV protocol support
• Clustering support
• Software mirroring and RAID0 striping
• Fast! State of the art caching and read-ahead algorithms
• Low memory requirements
• Scalable: 64-bit, 8 terabyte sizes, pooling etc

I could go on... About the only thing it is missing is encryption. Of course it remains to be seen whether the port to Linux will be successful, and whether Novell has the sense to make it open source.

Re:Can't wait for Novell Storage System on Linux

[Anonymous Coward]

You forgot one thing. As an enterprise filesystem Novell was absolutely bulletproof long before RAID systems were in vogue. It took me awhile to even figure out why we needed one after years of running Novell as our main storage controller (flawlessly).

Novell could give *nix systems windows like (don't bash if you don't know) fine granularity over user access at the enterprise level along with true enterprise scaling. Again, if you have never worked in a cross enterprise environment, don't start bashing because you really can't appreciate some of Novells strengths until you need the features.

there is more to a filesystem that speed.

[msh104]

every filesystem has its own purpose, for example reiser4 has atomic operations, database like capabilities, journalizing and metadata. now how are you going to say that ext2 is better because it performed better in brenchmark xyz. this is just the same thing as people buying a graphics card because it scored 1 or 2 fps more then an other card but forget that the other card has a build in mpeg2 or for the same price.

file size a problem?

[twitter]

If 30 33.3MB files (bonnie test) are not representative of your needs, please download the scripts. You can then modify the parameters for thousands of 2k files and post the results. Lots of people would be intersted, you know.

Worth Noting

[MajroMax]

It's worth noting here that the benchmarks were all run on files >= 1GB, if I'm reading the table correctly; this stresses the raw IO of the system, and doesn't really take into account the differences in tree-structure between the filesystems.

As for complaints about Reiser's performance -- last I heard, it was more optimized for many small files -- precisely the domain that this thing didn't test.

Re:Sort of on topic...

[angle_mark]

There are some free and some commercial products which can offer full read/write + journalling access for ext3 partitions from Windows. I'd definitely recommend you pick ext3 over fat32.

Some examples..

Free: Explore2fs allows you to read ext2 and ext3. Limited write support is available.

Commercial: Ext2FS Anywhere don't let the name put you off as it has full read/write support for ext2, ext3 and I think reiserFS is supported now too.

this benchmark was performed using a 200Mhz CPU

[hansreiser]

which makes the whole thing pretty questionable in my view, especially when you consider that Nikita got completely different results on his more modern hardware (see www.namesys.com/benchmarks.html)

I don't really target 200Mhz CPUs in my performance tuning....;-)

Hans

Re:this benchmark was performed using a 200Mhz CPU

[Deagol]

How true a difference the hardware makes.

I took an old PII-350 w/ 128MB RAM and benchmarked ext2, ext3, jfs, reiserfs, and xfs on an old 5GB IDE drive. ext2 was the winner by a margin (raw throughput).

Now I'm beating up various hardware and software RAID configs on a dual Athlon MP 2200+ system w/ 2GB RAM and dual 3ware 8-port 7500 controllers w/ 180GB WD drives. JFS rises above the rest in terms of throughput (I didn't test XFS on this new machine), and, of course, reiserfs simply spanks everything in terms of file creation/deletions. The thing I noticed was the JFS had much lower CPU utilization for file creations/deletions and was twice as fast at it than the ext2/3 filesystems (it still got spanked by reiserfs, though).

If anyone's interested, the "best" overall was reiser w/ the mount options noatime,notail,nodiratimeall. Also, if anyone cares, on this machine, the Linux software RAID code at no less than twice the performance numbers over the 3Ware hardware RAID. Running RH9 with all RH updates applied.

NTFS read/write in Linux

[sethadam1]

Don't count on this. The write (and maybe read) drivers will always be "experimental." Why?

Because NTFS specs are locked in a dark closet in near Seattle never to be seen by the evil Linux developers. These developers, fearing for their lives, will never have the guts to deem their NTFS write stable - there will always be a slight chance you'll corrupt your entire disk table - and no one wants their so-called "stable" driver to corrupt people's data.

In NT4, NTFS was at version 1.1, aka NTFS 4.0 (to align with NT4.0). In Windows 2000, it was version 3.0, aka NTFS 5. And in XP, it's version 3.1, also known as NTFS 5. The point is, NTFS is a moving target, so it's unlikely we'll see effective NTFS abilities in Linux anytime soon.