Linux File System Shootout

IpSo_ writes "Finally an extensive, human readable Linux file system benchmark has been unleashed upon us. Originally posted on the Linux Kernel mailing list, using two of the most popular benchmarking tools available, it compares all the major file systems, including their different mount options. The results are surprising."

Comments

human readable ?

[gokulpod]

I am sorry..all I see are numbers floating around. Does someone have a "human readable" summary of this ?

Re:human readable ?

[arivanov]

The human readable result is you need to know what you want. There is no silver bullet.

It looks like xfs wipes the floor for all but temporary (loads of create/delete) file usage. Jfs looks like the best all-rounder. Reiser looks like something that can be tuned to the specific usage, but eats CPU for breakfast, lunch and dinner and EXT3 "surprise, surprise" sucks rocks. The other "surprise, surprise" is that EXT2 is still very good for many uses.

Frankly, I do not see anything new and fascinating in the results, but they are good to throw at people who keep asking me "why not EXT3" and "Why XFS or EXT2". Here is why!

Cheaters!

[borgdows]

NTFS has been removed of the benchmark results because it was the best performer in every test!

Throughput benchmarks only...

[pe1chl]

There is have focus on throughput in these benchmarks. Reading and writing lots of data, seeking in files and reading data, etc.

Notably missing are more day-to-day useful operations such as the creation and deletion of lots of files, parallel action on many open files,
lots of files in a directory, etc.

When I want to select a filesystem, I do not want to know how fast it can read a 3GB file sequentially. I want to know how well it performs on a fileserver, mailserver etc.

Re:Throughput benchmarks only...

[zurab]

Have a look at Hans' benchmarks at namesys.com. Although he only compares Reiser4 to ext3, and may not be an objective party. But I'm surprised how well JFS performed anyway and that Reiser4 is unusually CPU-intensive.

Short summary

[mst76]

iozone benchmark
best: jfs
worst: ext3_journal

bonnie++ benchmark
best: ext2
worst: reiser4/reiser4_extents, ext3_ordered/ext3_journal

Re:Short summary

[Spy Hunter]

bonnie++ benchmark
worst: reiser4/reiser4_extents

You might think that just based on the amount of red in Reiser4's row, but if you look all the way over to the right, you'll notice something interesting: Reiser4 often completes the benchmark in significantly less time than the other filesystems. Reiser seems to be caching a lot of flak for the CPU usage (certainly it gets a lot of red boxes in this benchmark because of it). Personally, though, I've got CPU to spare. Disk seek times aren't changing drastically anytime soon, unlike CPU speeds. If I can trade some CPU cycles for less wasted disk seek time, I think that's a great trade.

Re:Short summary

[JanneM]

Actually, at least in our case we thread the app, with one thread handling disk IO and other threads handling other aspects (such as CPU intensive stuff), precisely to squeeze out a bit more performance and so disk accesses do not interfere with and stall other stuff. You get this as soon as you try to do something in soft realtime (such as video applications). On one hand, you want to stream video to/from a drive as quickly and efficiently as possible; on the other, you want to do some CPU-intensive operations (filtering, resizing) on the video stream at the same time.

I'm not saying that trading CPU for filesystem speed is a bad idea; it isn't. What I'm saying is that it's not a simple "more is better" function, and that the cutoff for when it no longer makes sense does depend a lot on the application you intend it for. Again, to take an extreme, you would not want to have a system where the filesystem eats so much CPU the rest of the system essentially blocks, starved for CPU time, when the disk is used.

To take an even more extreme way of doing the tradeoff: you could compress and uncompress all data on the fly. That way you would increase transfer speed (and increase it quite a bit in the case of text files and similar) as well as decrease disk usage. It is not often done, though, because the tradeoff is not worth it in general.

For us, and our app, Reiser is on the wrong side of that cutoff point (and Reiser4 is not even on the horizon yet).

Re:Huh?

[matticus]

Well, here's IBM's page about it.

From what I've seen poking around USEnet, JFS seems to have the too little, too late problem. I've never seen it pwn a benchmark like it did today though.
I'm a little confused-I have been told XFS is the best designed, highest performing file system, and I would hate to think SGI is getting into a lot of this crap with SCO for a relatively slow journaling file system...

these are narrow tests, not comprehensive tests

[hansreiser]

Still, they are interesting in showing areas of performance where something is a bit amiss.

It would be nice if exactly what they did was explained. You know, things like how you can get both the lowest total elapsed time and the worst overall score on one of the runs (because of CPU usage? ), what task was measured by each of the numbers printed, what the different settings on the different runs mean.....

Sigh, time to go read the source code for them.

Results question

[DaEMoN128]

I see that JFS won in the bonnie test, but EXT2 put up one hell of a fight and won the other roundup. I didnt think EXT2 was a journaling file system. Is it fair to thow in a Non Journaling FS in a benchmark against a bunch of Journaling ones? If it isnt journaling, then I gess Im going with JFS.

If I am wrong, please either resopond to correct me or email me.

scythefwd@yahoo.com

Re:Results question

[NickFortune]

Is it fair to thow in a Non Journaling FS in a benchmark against a bunch of Journaling ones?

Yes. Of course. The ext2 numbers provide a baseline for the comparison comparison. Any journaled FS that could match it would have to be very good indeed. This isn't explicity stated anywhere - but this was posted to the kernel list. They can reasonably be expected to know the difference between ext2 and the rest. It's all data. Data is good.

I know we're used to seeing "benchmarks" used as corporate propaganda, but let's not forget what they're supposed to be used for

Re:Huh?

[Frodo420024]

I'm a little confused-I have been told XFS is the best designed, highest performing file system, and I would hate to think SGI is getting into a lot of this crap with SCO for a relatively slow journaling file system...

IIRC, XFS is more about guaranteed performance under various stressful conditions than about getting the absolute peak speed in calm conditions.

DeFacto Standard

[Bios_Hakr]

I'm not trying to be an asshole or a troll; just hear me out.

I love Reiser. I also love Gentoo and adore Debian. Myself and another guy, Joe, are the main "linux geeks" in our computer group (cugy.net). When it came time to decide what to support at our group, we had to choose RedHat.

If I'm in a message board or IRC channel, I need to know some things about the guy I'm helping. We reccomend RedHat because that is the biggest US company behind Linux (IBM and SUN notwithstanding). If I am teaching people about Linux, then it is to both our advantages to teach/learn about what we will see "in the field". Therefore, we only support RedHat.

What does this have to do with anything? Well, RedHat 9 and Severn do not allow the creation of Reiser by default. I could probably boot from a Gentoo disk and format a partition to Reiser, then install RedHat to it. But, by default, only ext* is allowed.

I love to do things that improve performance. I love testing new things on my laptop or on a offline box in our test lab. But unless RedHat offers it, it will remain in the shadows of the linux world, which is, in turn, in the shadows of the user enclave. Hell, of every important box on my network, they are either RedHat or Win2k.

More on topic, Joe got a lot of recognition when the "internet got a lot faster". Did he upgrade the firewall? Did he install another OC-3? Maybe he reconfigured services on the proxy?

Nope, he installed a hard drive, formatted it to Reiser, and moved the proxy cache to the reiser disk. I couldn't belive it. Just changing the filesystem caused an increase that was noticable across our network. At no cost!

Good work, Joe.

Summary

[samj]

Use XFS unless you want to do lots of deletes (as they are slow and expensive) in which case ext2 is probably a better bet since the files are probably temporary (Squid caches for example).

"linux reiserfs"

[bani]

type "linux reiserfs" when booting the installer, and you will have access to reiserfs during redhat install.

i've been using this method for ~2 years now.

Where's the deviation?

[rufusdufus]

Filesystem benchmarks can be remarkably inconsistent. These tables do not display average difference between runs. Usually this means that the methodology used to do the benchmarking is lax, and thus, untrustworthy.

For example, consider that harddrives do their own error correction. Depending on the location of marginal blocks on the media, different file systems can score dramatically for no other reason than the drive's re-mapping or error correction logic is kicking in at a bad point. Alignment of data can also be a factor in performace which depending on the formatting procedure may be completely random when compared to the file system sitting on top of it.
For these reasons and a host of others, it is not reliable to do filesystem performance comparison on a single machine.
Bottom line is that there is a good chance that these data are not fair representations of the relative merits of each filesystem.

Can't wait for Novell Storage System on Linux

[thehunger]

Personally, I'm happy to wait for yet another file system: Novell Storage System. It certainly is feature packed. Now before you all start banging on me, remember that Novell for years was the king of file system services. Just some of the features:

• Compression and fast decompression
• Hiearchical storage system integration
• Advanced access control model, with granular access control with inheritance and inheritance filters
• Copy on Write
• File system snapshot
• Journaling
• Transaction tracking
• DFS, Junctions and yes! symbolic links!
• Disk, directory and user level quotas
• Fast mount and repair times
• Name spaces for MAC, NFS, NCP
• Native CIFS, NFS, AFP and WebDAV protocol support
• Clustering support
• Software mirroring and RAID0 striping
• Fast! State of the art caching and read-ahead algorithms
• Low memory requirements
• Scalable: 64-bit, 8 terabyte sizes, pooling etc

I could go on... About the only thing it is missing is encryption. Of course it remains to be seen whether the port to Linux will be successful, and whether Novell has the sense to make it open source.

file size a problem?

[twitter]

If 30 33.3MB files (bonnie test) are not representative of your needs, please download the scripts. You can then modify the parameters for thousands of 2k files and post the results. Lots of people would be intersted, you know.

Re:Sort of on topic...

[angle_mark]

There are some free and some commercial products which can offer full read/write + journalling access for ext3 partitions from Windows. I'd definitely recommend you pick ext3 over fat32.

Some examples..

Free: Explore2fs allows you to read ext2 and ext3. Limited write support is available.

Commercial: Ext2FS Anywhere don't let the name put you off as it has full read/write support for ext2, ext3 and I think reiserFS is supported now too.

this benchmark was performed using a 200Mhz CPU

[hansreiser]

which makes the whole thing pretty questionable in my view, especially when you consider that Nikita got completely different results on his more modern hardware (see www.namesys.com/benchmarks.html)

I don't really target 200Mhz CPUs in my performance tuning....;-)

Hans

Re:this benchmark was performed using a 200Mhz CPU

[Deagol]

How true a difference the hardware makes.

I took an old PII-350 w/ 128MB RAM and benchmarked ext2, ext3, jfs, reiserfs, and xfs on an old 5GB IDE drive. ext2 was the winner by a margin (raw throughput).

Now I'm beating up various hardware and software RAID configs on a dual Athlon MP 2200+ system w/ 2GB RAM and dual 3ware 8-port 7500 controllers w/ 180GB WD drives. JFS rises above the rest in terms of throughput (I didn't test XFS on this new machine), and, of course, reiserfs simply spanks everything in terms of file creation/deletions. The thing I noticed was the JFS had much lower CPU utilization for file creations/deletions and was twice as fast at it than the ext2/3 filesystems (it still got spanked by reiserfs, though).

If anyone's interested, the "best" overall was reiser w/ the mount options noatime,notail,nodiratimeall. Also, if anyone cares, on this machine, the Linux software RAID code at no less than twice the performance numbers over the 3Ware hardware RAID. Running RH9 with all RH updates applied.