Benchmarking XFS, ext2, ReiserFS, FAT32

blakestah writes: "Well, it looks like someone on the LKML has taken upon himself to do some benchmarking of ReiserFS, ext2, and XFS using the 2.4 kernel series. It is not a real benchmark test, but kind of interesting nonetheless. See the results (in Spanish) at this LUG in Mallorca. Simple runs of dd, tar, and rm are shown, and for most of the tests XFS is pretty dern fast, beating all the others. The exception is removal of a large source tree (the kernel source), for which XFS is the slowest by a fair amount. See this kernel post for the translations of important words. It will be nice to see more such open benchmarking posted, because benchmarks provide developers goals." The contrast between FAT32 and XFS is particularly interesting to see.

Comments

Re:Does it bother anyone...

[Anonymous Coward]

Deleting files on a FAT* partition doesn't actually delete the files. It just marks the first character of the filename with an invalid character (0 ASCII IIRC) so the filesystem thinks the space is free. That's why undelete utilities work -- they just search for all files with the invalid first character and then change the character to something else (which the program prompts the user for). Assuming the space the file had before hasn't since been used by another file, you get the file back.

My guess here is that FAT32 just marks the root source directory as deleted and then moves along. (This is how it works under DOS/Windoze at any rate.) If this were so, FAT32 is hardly fast here -- it takes 6.7 seconds to do just one write to the FAT table!

To be honest though, I'm not sure how the various Linux FSes work, so maybe they're doing it the same way, though I doubt one FS entry would take 10 - 20 seconds.

Those "tests" aren't good benchmarks

[Anonymous Coward]

Benchmarks... bah. I don't think the ones linked to on that Spanish site are worthy of the name. For those that can't see it because it's /.ed, it makes out XFS to be quite a bit faster. Some relevant comments from the LKML... [the last one showing that XFS is not always faster] Alan Cox: "reiserfs seems to handle large amounts of small files well, up to a point but it also seems to degrade over time. ext3 isnt generally available for 2.4 but is proving very solid on 2.2 and has good fsck tools. Ext3 does not add anything over ext2 in terms of large directories of files and other ext2 performance limits. XFS is very fast most of the time (deleting a file is sooooo slow its like using old BSD systems). Im not familiar enough with its behaviour under Linux yet." Andi Kleen: "On one not very scientific test: unpacking and deleting a cache hot 40MB/230MB gzipped/unzipped tar on ext2 and xfs on a IDE drive on a lowend SMP box. XFS (very recent 2.4.4 CVS, filesystem created with mkxfs defaults) > time tar xzf ~ak/src.tgz real 1m58.125s user 0m16.410s sys 0m44.350s > time rm -rf src/ real 0m50.344s user 0m0.190s sys 0m13.950s ext2 (on same kernel as above) > time tar xzf ~ak/src.tgz real 1m26.126s user 0m16.100s sys 0m36.080s > time rm -rf src/ real 0m1.085s user 0m0.160s sys 0m0.930s ext2 seems to be faster and the difference on deletion is dramatic, so at least here it looks like Alan's statement is true. The test did not involve very large files, the biggest files in the tar are a few hundred K with most of them being much smaller. The values stay similar over multiple runs. I did not do any comparisons recently with reiserfs, but at least in the past reiserfs usually came out ahead of ext2 for similar tests (especially being much faster for deletion)"

Re:Those "tests" aren't good benchmarks

[norton_I]

I just did some similar benchmarks on XFS vs. ReiserFS with and without the notail option, and ext2 (for reference -- I wanted to switch to a journaled filesystem).

My tests were 1) copy a several gigabyte file tree (/home) to a temp filesystem, 2) run du on the tree, 3) run Bonnie++, 4) build a kernel, and 5) rm -rf the whole tree.

My results were that ext2 and xfs had high read/write throughput, resierfs-notail was slightly lower, and resierfs without notail was a fair bit slower.

All metadata intensive operations (rm -rf, du, bonnie++ small files, etc) were blazing fast on ReiserFS, slow as molasses on xfs, and fast on ext2, but scaling poorly to high numbers of files.

xfs could do slightly more random seeks than either ext2 which in turn beat out Reiserfs, by about the same amount.

reiserfs doesn't have an fsck -- at least the one I have is a noop.

I switched to reiserfs-notail, and was proptly annoyed by the lack of fsck, but otherwise it is running well. I ran a pair of make -j6's, some directory copies, and an updatedb, then hit the power and it is running fine.

My personal preference would be to have ext3 in 2.4 along with Daniel Phillips' hash-tree directory patch stabalized--I think that would smoke reiserfs and still give me journaling.

Re:Whatever happened to Tux2?

[V.]

http://kernelnewbies.org/~phillips/

I've always been pulling for XFS.

[Omega]

I guess I've always been partial to XFS and I hope that it becomes the new default filesystem for Linux.

This guy Dave (I forget his last name now), from sgi gave a presentation to the DC-LUG back in 1999 and talked about XFS and how sgi wanted to release it as GPL to become a core component of Linux. He also talked about the history of XFS and how they had to invent a new size prefix to describe how large a filesystem XFS could accomodate ("exo-byte" = 1024 Gb). XFS has been used by sgi for their MIPS and Cray machines ever since 1984, and now that sgi has donated it to the Linux community, I think we'd be remiss if we didn't welcome it with open arms.

But that's just MHO. ;)

Re:I've always been pulling for XFS.

[rangek]

they had to invent a new size prefix to describe how large a filesystem XFS could accomodate ("exo-byte" = 1024 Gb)

This is a slight exaggeration. A kilobyte is 1024 bytes, a megabyte is 1024 kilobytes, a gigabyte is 1024 megabytes, a terabyte is 1024 gigabytes. Oh wow, it is not even an exaggeration, it is just wrong, I guess. Anyway, if we keep going up the table of SI prefixes in my CRC, we could have petabytes, which would be 1024 terabytes, and then exabytes = 1024 petabytes, and so on. The is no need to invent things, it is all taken care of, from 10E-24 (yocto-) to 10E24 (yotta) (now thats a lotta stuff) ;)

Re:I've always been pulling for XFS.

[Dragonmaster Lou]

I believe 1024 TB is a petabyte. 1024 PB is an exobyte, and 1024 EB os a yottabyte.

Re:I've always been pulling for XFS.

[enrico_suave]

"I believe 1024 TB is a petabyte. 1024 PB is an exobyte, and 1024 EB os a yottabyte."

Actually I believe 1024 EB is a lottabyte (*badoom tch!*) =P

e.

Re:I've always been pulling for XFS.

[foobar104]

While I agree with the substance of your message, you've got a couple of facts wrong.

First, the maximum filesystem size that XFS can handle is 18 exabytes. Since exabyte is also the name of a brand of tape drive, it's more common to hear of people talking of 18 million terabytes.

(Aside: an intresting statistic found on this page says that as of 1995, 5 million terabytes was about enough data to store all words ever spoken by humans, ever. Cool.)

Also, XFS was never used on Cray systems. XFS made its first appearance (if I remember correctly) on IRIX 5.3 back in 1994. Other than being off by four orders of magnitude in your sizing and by ten years in your dates, I think you're exactly right. ;-)

Re:XFS size

[jd]

Size of Linux-2.4.4, with no patches: 3,239,950 lines (as calculated by doing a wc -l on all .c's and .h's in the tree, then summing the result), and 131,448,832 bytes (for the whole tree, using du).

Size of Linux-2.4.4 with XFS (as taken from SGI's CVS tree): 3,612,541 lines (same algorithm) and 145,367,040 bytes.

Either the site's slashdotted already...

[jd]

Or they've a serious PostgreSQL bug. :)

Seriously, XFS looks interesting. I did some stats on source size, and posted them to K5. :) Dropped in score so rapidly, it'll reach the Earth's core sometime in the next half hour.

Essentially, XFS is 10% of the entire kernel size, making it perhaps the single-most sophisticated driver available. On the flip-side, I can't help but feel that that much code is going to have -some- impact on the rest of the system.

Talking of benchmarking, how does IBM's "Next Generation PThread" code stand up? And how on Earth are you supposed to install it? It clashes with glibc, making an RPM install, ummm, of questionable safety. And once you start with RPM (or tarballs, or any other system), it's unwise to mix-and-match. Either you keep track of where things are, or the system does. Half-and-half is NOT good.

Lastly, anyone found a way to get XFS, JFS and AFS into the same kernel? (Without using a sledgehammer, preferbly.)

I managed to get a copy of the page

[Kiwi]

Since I know a little Spanish, here is my very crude translation of the core benchmarks:

First benchmark:

Writing, reading, and deleting a fairly large file (256MB). The commands used are as follows (omitting the redundant 'time' command used in all cases):

• dd if=/dev/zero of=./prova bs=1M count=256
• dd if=./prova of=/dev/null bs=1M count=256
• rm -f ./prova

And the time these commands took (in seconds):

Filesystem Write Read Delete
ReiserFS 18.5 23.41 0.4
Ext2FS 20.3 21.38 0.57
XFS 16.32 19.42 0.26
FAT32 43.65 27.98 1.59 Second benchmark: This benchhmark was donw with the source code of Linux 2.4.4. The .tar.gz file was first uncompressed, so that all the work was done on the tar file (which is larger then 100MB). The commands used on the uncompressed .tar file are as follows (with the time command ommitted again):

• cp linux-2.4.4.tar prova.tar
• tar xf prova.tar
• rm -f prova.tar
• rm -rf linux

The times these commands took (in seconds):

FS Copy Extract rm file.tar rm -rf dir
Reiser 38.48 58.44 0.45 10.09
Ext2FS 21.31 59.19 2.88 11.12
XFS 16.21 35.44 0.18 21.96
Fat32 39.76 134.19 1.2 6.7

Give the poor site a chance...

[m2]

What the gut got:

$ddif=/dev/zeroof=./provab s=1Mcount=256

$ddif=./provaof=/dev/nullb s=1Mcount=256

$rm-f./prova

FSWriteReadrm

ReiserFS18.523.410.4

Ext2FS20.321.380.57

XFS16.3219.420.26

FAT3243.6527.981.59

$cplinux-2.4.4.tarprova.tar

$tarxfprova.tar

$rm-fprova.tar

$rm-rflinux

FScpextractrmrmdir

ReiserFS38.4858.440.4510.0 9

Ext2FS21.3159.192.8811.12

XFS16.2135.440.1821.96

FAT3239.76134.191.26.7



[1] ReiserFS has a high (reproducible) variance when copying the kernel tarball.

All test ran three times with the exception of the reiserfs kernel tarball copy, which ran six times. Machine is running in single user mode. Results timed with time(1).

XFS size

[Booker]

I can't help but feel that that much code is going to have -some- impact on the rest of the system.

Although XFS is big, it doesn't stomp on the kernel that much.

If you're looking at it from a pure "volume of code" point of view, here's some info:

The XFS patches are split into 2 parts, one which contains kernel changes, and one which is the filesystem itself.

The "core-kernel" patch is about 190k, while the actual filesystem code patch is about 4.3M.

Bear in mind that a 190k patch to the kernel does not mean 190k of new code, either, since you have to take out the context lines, the headers, and the delete lines.

Overall, the impact on the kernel isn't as big as you might think, looking at the overall size.

Now, whether or not executing code in the filesystem slows down the rest of the system, I don't have any real data on that, although I have not noticed any detrimental effect on my system.

Re:No ext3?

[sl70]

I'm using ReiserFS now and I'm really happy with performance/crash recovery. I have a lot of cat-related outages...

I installed xfs a week ago. Yesterday I wrote about it to my ex-office mate, saying I couldn't wait for a power failure. Got my wish this morning at about 4:00 a.m. My machine took 15 seconds to reboot. My colleague's machine with ext2 took nearly 3 minutes. Cool messages in the logs:

May 10 03:55:11 musuko kernel: Start mounting filesystem: ide0(3,5)
May 10 03:55:11 musuko kernel: XFS: WARNING: recovery required on readonly filesystem.
May 10 03:55:11 musuko kernel: XFS: write access will be enabled during mount.
May 10 03:55:11 musuko kernel: Starting XFS recovery on filesystem: ide0(3,5) (dev: 3/5)
May 10 03:55:11 musuko kernel: Ending XFS recovery on filesystem: ide0(3,5) (dev: 3/5)

FAT32 vs. XFS

[MSG]

The differences between FAT32 and XFS may be interesting, but keep perspective. What you're seeing is the difference between the Linux driver for XFS and the Linux driver for FAT32, and not necessarily the inherent properties of either filesystem.

Don't get me wrong, I'm not comparing FAT32 to XFS by a long shot! But FAT32 is a fs that not a lot of hackers care about enough to improve the performance under Linux. Personally, I've always found that FAT32 access under Linux has been abysmal compared to access to the same filesystem under Windows.

No ext3?

[Jethro]

I used ext3 about a year or to ago. It worked a lot better than it should have. Or so I thought.

It claimed that it'll still have fsck run when you crash, but actually nothing was getting run... I'm pretty sure that it was just not repairing itself after crashes. I'm really surprised I never lost any data until, well, one day I lost the whole damn partition.

I'm using ReiserFS now and I'm really happy with performance/crash recovery. I have a lot of cat-related outages...


--

SI Prefixes

[rangek]

This isn't specifically for bytes, but the list of SI prefixes is here

The new prefixes for binary units (which nobody uses (the prefixes, not binary units)) are here

Re:Why FAT32?

[Scudsucker]

Maybe because this was a test of filesystems that work under Linux, not filesystems in general. To have a reliable test of NTFS, you'd have to use it under NT or Win2k, unless you wanted to do some shady benchmarking and use the developing NTFS driver for linux. Also, the point was do use the exact same commands for each filesystem...dd, rm, etc, which you couldn't do in Windows.

Fat32 was probably included because it's its so well supported and easy to do.

ntfs vs fat32

[LordXarph]

fat32 is an interesting control, but in an ideal benchmark, ntfs would have been used, as it is designed closer to other filesystems, as opposed to fat32, which is more like Baby's First Filesystem(tm).

Though I will grant that NTFS would have been hard or impossible to benchmark in this test, given the lack of robust drivers for it.

-Lx?

Re:ntfs vs fat32

[SuiteSisterMary]

The other problem being that if the Fat32 test was run under Linux as well, all it's showing is how good the Linux driver is, not the filesystem itself.

Slashdotted Already

[sid crimson]

Maybe they should benchmark their DB system.

;-)

-sid

XFS deletion performance.

[be-fan]

Yes, XFS does kick that much ass. Still, the deletion performance surprises me. Of course, the other speed things make up for it, but its still a puzzle. If you don't know, rm -rf has historically been a slow operation on XFS.

Here's my theory. Ext2 uses a bitmap to track free blocks, and I'm pretty sure ReiserFS does as well. Free block runs on XFS is managed by two B+trees, one keyed by address, one keyed by size. Thus, allocating space is very fast on XFS, and it is easy to keep things contiguous. However, inserting runs of blocks into both trees is a slower operation then simply clearing the appropriate bits. This would explain the difference between the file creation speed (extraction test) and file deletion speed (rm -rf test.) If this is the case, I think it is quite a good tradeoff, given that space is allocated much more often.

Of course, IANAXE (I am not an XFS engineer) so this is just my theory. I'd appreciate it is someone more informed about XFS could tell me the reason for the performance delta.

Re:XFS deletion performance.

[vidarh]

Or if you never delete the full contents of your drive - which most people likely won't.

It still holds for most cases: You'll copy a bunch of data to the drive over time, and some of it will be deleted, but most of the time a lot of the data you accumulate will be data that you'll keep around for the lifetime of the drive.

Which equals less deletion that allocation...

Perhaps not as significant as the original poster thinks, but his claim still holds true for most users.

More significant, I'd assume, is that a lot of deletion often is maintenance that won't take place during peak usage of the system, and that often may just be left running in the background, to complete whenever it completes.

Allocation is in many applications much more time critical, as allocation is mostly done by applications that the users of the system are much more likely to care about performance for.

Re:But reiserfs wastes 1/3 of disk space!

[Convergence]

Did you look through the mailing list for ReiserFS?

If you had, you'd have seen that Reiserfs is journaling (Well, actually log structured)

It reserves 32mb for the log... Ergo, 32mb of missing space. If I remember right, you can decrease this when formatting it.

Which eventually yields. . .

[kfg]

a 'Yottafied' filesystem.

"Hmmmmmm,large it is. Strong is the file system in this one, hmmmmm."

KFG

Does it bother anyone...

[electricmonk]

...that FAT32 actually beat ext2 on two tests, and even beat EVERYONE ELSE on the test that involved removing the kernel source tree? Could an FS hacker with some time and patience on his hands please explain what's going on?

--

I AM THE SYSADMIN OF BULA

[gallir]

And it's hard to maintain postgress running.

You should tell ne before you try to slashdot us ;-) So, I could have time to increase the PG backend.

Hope we can keep it running now... (poor PIII 500Mhz)

--ricardo

Whatever happened to Tux2?

[ortholattice]

Possibly off-topic, but what's the story on the Tux2 http://slashdot.org/features/00/10/13/2117258.shtm l file system? It sounded like a great idea, now it seems even the links are broken.

Well, I guess..

[hhg]

That their webserver can not keep up with their filesystem anyway...

Slashdotted!

[Scoria]

Mirror here.

No idea why they're including something as outdated as FAT...

Re:Are the benchmarks on the Reiser site rigged?

[vidarh]

I can assure you that whatever the quality of Reiserfs, it beats ext2fs hands down in the cases it's designed to handle particularly well: Large (huge) directories and small files. The company I work for operate a mail server for about 1.1 million people, and we have quite a few million files, of an average size of less than one kilobyte (we're using maildir format for the accounts, and also store a lot of small XML objects as separate files for each user), and some of our directories has tens of thousands of entries.

Running something like that on ext2fs would (apart from the agony of fsck'ing 800GB of storage) be completely hopeless.

When it comes to setups with a few large files, though, the advantage isn't that great, and the numbers in the article makes reasonable sense.

Reiserfs should be your filesystem of choice if: a) you want to be able to put gazillions of files in a single directory when there is no logical hierarchy to the data (our tests indicate that Reiserfs handles shallow directory hierarchies with many files pr. directory faster than the opposite), and b) you want to be able to efficiently store a 100 byte file when grouping your data logically would give you file sizes of that magnitude, instead of grouping many pieces of data together.

We've been running Reiserfs for well over a year now, and it works great. It's important that you keep up to date on bugfixes, though, and that you're very careful about your recovery procedures - reiserfsck is really a last resort, and you should always ensure you copy out as much as possible of any data on a damaged volume, and preferrably take a raw copy of the entire volume, before running it. That said, I wouldn't hesitate to recommend Reiserfs to anyone with the specific needs I mentioned above.