Benchmarking Linux Filesystems In New 2.6 Kernel

An anonymous reader writes "KernelTrap has an interesting article about a recent benchmark conducted to compare five journaling filesystems available with the current 2.6.0-test2 Linux development kernel. The tests were conducted with a very simple shell script, mainly timing how long it takes to copy, tar, and remove directories. Looks like reiser4 is the fastest filesystem at the expense of consuming much more CPU, with ext3 trailing a ways behind."

Comments

fs issues

[Tirel]

speed is nice, but I think the more important question is; how stable is it?

Re:fs issues

[BrokenHalo]

I don't know about 2.6 yet, but Reiserfs hasn't let me down so far.

For a long time I resisted moving away from ext2, as I didn't want the journalling overhead and didn't mind the occasional e2fsck. After I got a few power outages I changed my mind :-) but I haven't been disappointed with Reiserfs' speed.

karma for Anonymous Coward

[Anonymous Coward]

The first item number is time, in seconds, to complete the test (lower
is better). The second number is CPU use percentage (lower is better).

reiser4 171.28s, 30%CPU (1.0000x time; 1.0x CPU)
reiserfs 302.53s, 16%CPU (1.7663x time; 0.53x CPU)
ext3 319.71s, 11%CPU (1.8666x time; 0.36x CPU)
xfs 429.79s, 13%CPU (2.5093x time; 0.43x CPU)
jfs 470.88s, 6%CPU (2.7492x time 0.20x CPU)

What's interesting:
* ext3's syncs tended to take the longest 10 seconds, except
* JFS took a whopping 38.18s on its final sync
* xfs used more CPU than ext3 but was slower than ext3
* reiser4 had highest throughput and most CPU usage
* jfs had lowest throughput and least CPU usage
* total performance of course depends on how IO or CPU bound your task is

Re:karma for Anonymous Coward

[dustman]

I think another interesting metric to look at would be cpu time used. If one of them took 90% CPU for 10 seconds, that would be a big winner imo.

reiser4 171.28s @ 30% CPU = 51.384s CPU
reiserfs 302.53s @ 16% CPU = 48.4048s CPU
ext3 319.71s @ 11% CPU = 35.1681s CPU
xfs 429.79s @ 13% CPU = 55.8727s CPU
jfs 470.88s @ 6% CPU = 28.2528s CPU

Speed is not of the essence

[mnmn]

There are other things about filesystems us sysadmins to know about. Which is the most stable and crashproof filesystem? Ive suspected it to be XFS from which I recovered data after doing dd if=/dev/zero of=/dev/hda1 count=2048576

Also what filesystem would require the lest or no syncing at all? Befs?

In server environements with stripped 15K cheetah SCSI drives, you'd worry more about stability than speed.

Re:Speed is not of the essence

[Molina the Bofh]

Pick one of the three answers:

With the new and improved Magnetic Forensics Tunnelling Electronic Microscope, used by the FBI

He had a backup

XFS takes so long to sync that by the time it would start to sync, he turned his computer off

Re:Speed is not of the essence

[mnmn]

I did a dd to dump the whole filesystem to a file in a larger filesystem for recovery purposes. The XFS tools that came with RedHat 7.1 seemed to crash so I did a small slackware install and went from there. xfs_restore and other tools sounding like xfs_sync or something found my third superblock. I used it to reconstruct the other superblocks. The root and early directories were gone, and the files I needed were in the root directory, well the xfs_restore linked them all to numbered directories and files. A few greps later I was in a directory that had those files all intact. I recovered around 80% of all the files, and 100% of the ones I was seeking. Just because I succeeded in one major recovery op from an XFS filesystem, I feel confident with it more than the ext3 or reiser guys. I havent really compared them.

Re:Speed is not of the essence

[Bernie]

In server environements with stripped [sic] 15K cheetah SCSI drives, you'd worry more about stability than speed.

I'm not convinced. ext3 under 2.4 on SMP can be cripplingly slow entirely for software reasons--lock_kernel() being the biggest culprit.

Having said that, my new big fileserver is going to be ext3 on 2.6 (eventually!); the data volumes being on a 12-way RAID-5 set and the journals on a RAID-1 pair. It seems to perform adequately with dir_index and sparse_super.

The main thing that swings it for me is the brilliant e2fsck.

Incomplete comparison?

[Futurepower(R)]

From the article:
"reiser4 171.28s, 30%CPU (1.0000x time; 1.0x CPU)
reiserfs 302.53s, 16%CPU (1.7663x time; 0.53x CPU)

What's interesting:
* reiser4 had highest throughput and most CPU usage"

The comparison seems incomplete to me. Reiser4 took about half the time, with twice the CPU usage. The

Total Work Done by the CPU = Percent * Time.

Reiser4 did the work in half the time, but the total work was roughly equal. Actually, ReiserFS was more efficient considering total CPU cycles.

Re:Incomplete comparison?

[makapuf]

Except that if you have a mostly idle CPU and your task is more & more waiting for the disk to complete, you don't care about 11 or 30 % if the other 89 or 70 % of the CPU are idle.

Comparing CPU cycles needed is NOT a fair benckmark, unless your task is CPU _AND_ IO Bound.(if it's not io bound, take whatever fs you have, it doesn't matter.)

The benchmark was right in giving results as TWO parameters : CPU used and time spent. Might be interesting to see how it depends of the drive type or the CPU arch, though)

Re:Incomplete comparison?

[p3d0]

I disagree. Consider:

• FS1: 1sec CPU time, 1sec IO time. CPU usage=50%.
• FS2: 2sec CPU time, 6sec IO time. CPU usage=25%.

FS1 is undoubtably better. It consumes less CPU and IO time. Yet noobs would complain about the high CPU usage of FS1. The truth is, CPU usage is higher only because IO is more efficient.

I can think of no corresponding pathological case in which CPU load would be a more appropriate predictor than CPU time.

if you have a mostly idle CPU and your task is more & more waiting for the disk to complete, you don't care about 11 or 30 % if the other 89 or 70 % of the CPU are idle.

Exactly. That's why CPU load doesn't matter.

Comparing CPU cycles needed is NOT a fair benckmark, unless your task is CPU _AND_ IO Bound

No, CPU cycles matters for CPU-bound tasks. I think that's pretty self-evident.

(if it's not io bound, take whatever fs you have, it doesn't matter.)

Wrong. In a compute-bound task, CPU time adds directly to the total execution time of the task, so you should choose the FS with the lower total CPU time.

Another thought...

[BrokenHalo]

It seems to me to me that these benchmarks would probably have been more meaningful (and useful) if they had run the same tests against 2.4.21. Anybody got any thoughts on this? I know there are low-latency features and so forth with the new kernel, but I would be curious to know if that has any real impact on disk I/O.

Kerneltrap on new server..

[molo]

FYI, kerneltrap just moved to a new server (this week!). It used to be run on Jeremy's DSL line, which was why it would get shot to shit whenever it got slashdotted.

Now its in a colo on new (fast) hardware (paid for by users' donations), and upgraded to drupal 4.2 (faster here too).

So slashdot away!

-molo

Blah... This was on lkml...

[j.e.hahn]

And a number of people complained that it wasn't a great benchmark. Hans Reiser admitted it was just a quickie, and I forget who it was that said it, but ext3 has some performance enhancements that are on the cusp of merging into Linus' tree from the -mm kernels.

Wait until 2.6 is out folks. These numbers are still open for mass fluctuation.

Re:Stability will be fixed

[T-Ranger]

Speed problems can be solved by throwning hardware at the problem. Faster disks, more ram, more servers. A filesystem not desigined with stability in mind wont be stable regardless of the hardware.

Stability should be at the top of the list when desigining a filesystem. If its a quetsion of stability or speed, stability should win.

Oracle posted some stats

[Stone316]

With respect to filesystems and database performance. EXT3 came out on top.

Re:Oracle posted some stats

[nyteroot]

In that article, 4 filesystems are used: ext2, ext3, ResiserFS, and JFS. However, Reiser4, which is the clear leader in these benchmarks, was not tested.

CPU Usage?!

[metalmaniac1759]

30%-40% CPU Usage - whoa. What happens to poor me - with a PIII 550 MHz, 128 MB SDRAM and KDE running all the time.

I switched from RedHat to Gentoo and KDE stopped crawling. God knows what will happen with ReiserFS4.

Nandz.

Stupid question

[Hard_Code]

Why do we need fast file systems? Or rather, why should we spend so much effort on performance differences of such small magnitude (take out reiser4 and its high cpu usage)? The only things I can think of are: swapping memory - this will really slow the system down, because disk is orders of magnitude slower than main memory; and databases that absolutely require high performance. But both of these typically *already* use custom file systems (or raw partitions) tailored to their exact needs. As normal user (somebody who is not constantly copying Mozilla source tries around), why should this matter? I can't remember the last time I thought "boy, this file access was slow, I wish I had a faster file system".

Re:Stupid question

[Arandir]

When you're streaming a file, you're streaming a single file. All file systems will be roughly equivalent at this point. After all, once the harddrive has seeked to the starting sector, odds are very high that the data will be contiguous.

Where you want the efficiency is when you have to deal with a large number of files. For example, I use UFS on FreeBSD. Before I used softupdates, untarring the ports tree severly bogged down the system, because there were tens of thousands of very small files. But after I turned on softupdates, the same operation was extremely fast. But moving a single large file around (tarball of my homedir) didn't make any difference if softupdates was on or off.

quick question

[perlchild]

quick question, what mount options were used on each? Wouldn't e3fs appear slower than the others if it was data-journaling? (I read the article a bit fast, but I didn't see any "how we tested") Also wouldn't a good test take at least two hours on each filesystem?

Re:quick question

[perlchild]

Some of that however, has nothing to do with FS design though, browser start speed has dns components, thread starting, tuning. OS booting also has lots more factors than just the filesystem(on one of my systems, using XFS, even a dirty start and fsck of five filesystems, the part before the fsck is only 1% of the start of the machine, why? dns resolution by the daemons I need started near the end of boot)

Because people use Linux at work too.

[tdyson]

I care about performance because I have a few hundred thousand files on servers that 100 employees need to access. Caching will only get you so far when you have a lot of people going after a lot of files through out the day. Since we have the choice of which fs to use, it is nice to have more info to pick between them.

There is more to this world than home users and databases.

One word: Maildir

[Gothmolly]

If I have 2500 4k files in a directory, and my filesystem uses a stupid algorithm to store that information, it might take 2 or 3 times longer to retrieve a list of files than it does on a decent filesystem. When you've got a webmail frontend to your Maildir, and your webmail code has to grope through all directories, all files, even small differences are HUGE. Clever FS design will win the day.

Some XFS performance tweaks

[Simon Kongshoj]

Greetings gentlebeings,

Since I recently reinstalled my Debian system, I decided to put some effort into implementing my filesystems right. I decided on XFS for various reasons, mainly that it has always been rock solid for me (I've had some problems with ReiserFS causing heavy data corruption -- it's a long time ago and they've undoubtedly improved the system since, but still I prefer XFS since I've never even once had a problem with it, and know nobody who has), and that its good large-file performance is more useful to me than Reiser's kickass small-file performance. Also EAs and ACLs are neat. What sucks about XFS is poor small-file performance and abysmal delete performance.

Anyways, I made a few bonnie++ runs and messed around with some of the many mkfs and mount options of XFS. In the end, my tweaked XFS filesystems beat ext3 (mode=ordered) for delete performance, which was a substantial improvement over a standard XFS mount.

I made a writeup about the whole procedure at Everything2. Go slashdot the poor bastards. Warning: The language is tailored to the fact that not all E2 users are geek hardcore.