Slashdot Mirror
XFS Merged into Linux 2.4
Alphix writes "As noted on KernelTrap Marcelo has merged XFS into 2.4 after a code review by Christoph Hellwig. The mail from Marcelo on LKML is here. Apparently it touched very little VFS code so people not using XFS shouldn't see any ill effects from this (it's even supposed to fix some VFS bugs).
XFS is described by SGI as '...a journalling filesystem developed by SGI and used in SGI's IRIX operating system. It is now also available under GPL for linux. It is extremely scalable, using btrees extensively to support large and/or sparse files, and extremely large directories. The journalling capability means no more waiting for fsck's or worrying about meta-data corruption.' Let the stability vs. new-features flamewar begin."
back in the days when ext3 was still in our dreams i downloaded the SGI XFS kernel from their site and installed it on my wife's laptop. it was extremely stable and had the advantage, that her "oops, i have to run off and just close the lid"-atacks would not corrupt the filesystem (which i would have to clean up...).
nowadays i use ext3 on my machines because it comes default with RH (by the way EL is now available for academia, woohoo!). hence my question:
can someone offer a nice comparison of ext3 versus XFS?
Any up-to-date comparisons between the 3 main journalling filesystems (ext3, xfs, reiserfs), for both speed and reliability ?
:-)
I like xfs on the SGI - it's never let me down yet. I have to admit I'll be sorely tempted to try out xfs now that it's passed the 'seal-of-approval' and made it into the kernel - surely the best benchmark of all
Simon
Physicists get Hadrons!
Let the stability vs. new-features flamewar begin.
It's already been stable for years, since VERY early in the 2.4.x cycle. It's just a detail in the naming that makes it merged as part of 2.4.x itself.
Be careful those of you who still use lilo
Q: Does LILO work with XFS?
This depens on where you install LILO. For MBR installation: Yes. For root partitions: No, because the XFS superblock goes where LILO would be installed. This is to maintain compatibility with the Irix on-disk format. This will not be changed. Putting the Superblock on the swap partition is reported to work but not guaranteed.
SGI has an overview on the XFS filesystem, just briefly pointing out some highlights. I also recall reading somewhere that it was possible (moreso than ext* filesystems) to undelete files on an XFS filesystem, although I'm skeptical.
Trolls lurk everywhere. Mod them down.
For all those that are looking for a filesystem comparison, I found this story to be quite interesting...or go here for the test details and results.
I use XFS on serveral different servers, mainly because I belive it performs better then ext3, or any other fs. Also because Alot of the servers I run are samba servers and the ACL support is built native into XFS. And last I looked ACL support was still not quite stable in ext2/3 it has been awhile so it could be stable by now.
SGI released XFS into "the wild" and has ensured its longevity with little to no support on their part and increased the number of "out of box" coders they can hire to work on FS projects.
Microsoft....hasn't. Heck, MS is preparing to charge media makers (CF, SM, MMC, etc) to use FAT.
I say media makers switch to using XFS or another GPL'd journaling file systems. Won't take long for other platforms to support it in bulk (make/ config.....) and for stuff like flash where corruptions can occur often, I'd like a bit of journaling to minimize the impact.
I've been on slashdot so long I'm starting to get out of touch with the cool stuff if it ain't on slashdot.
Faster, More trusted (SGI's been using it for how many years now?), not sure how it compares cpu-wise.
The main thing that keeps me on ext3 is ext2 backwards compatability. You dont have to worry about having custom repair/bootdisks to recognize your install, and its easier to do stuff like mount under windows (great for dual booting)
Pain lasts, kid. Its how you know you're alive. Sometimes I think this growing up thing is just pain management-TheMaxx
Back in the day I would always just use ext2. Then I realized, hey there are other filesystems to choose from, so I decided to shoot in the dark and try reiser and xfs and ext3. xfs has always been the most awesome.
Just one thing. Now that we've got the source code for dealing with xfs, can someone write a driver so I can mount my xfs partitions from windows xp? It would really help out a lot of us dual booting types. I would do it myself, but I don't know jack about how filesystems work. I just know which ones do what.
I hope they put the xfs into 2.6 also. Maybe it wont be necessary to have seperate xfs-sources in gentoo anymore and xfs will finally be included in the gentoo-sources.
The GeekNights podcast is going strong. Listen!
I have been using it for for almost 3 years now. It has never let me down unlike some of the other journaling filesystems. No corrupt superblock like jfs. Kinda slow for small files when the parttions reach around 80% full. Plus its sure to piss SCO off :) but for video editing or files over 100kb it can't be beat. Reiserfs is great for directories like /var since 2.4.18 but I have lost too much data in the past to the bugger.
Just wondering, why does everyone get so excited about journaling filesystems? Many distros default to ext3/reiserfs now for even home boxes, but it's like a big band-aid.
If your box is crashing enough to make fscking a chore, you already have bigger problems. Sure, I can see where JFSs are sometimes useful, but on dekstops and most other machines the better-performing ext2 is a much more appropriate choice.
I agree with Marcelo's action on this. We're still in a very early stage of the 2.6 (stable) branch to feature freeze 2.4.
:)
I know we need the maximum user base for 2.6 testing, debugging and to recieve those "My TV stopped working when I installed kernel 2.6" messages. But we have to take it easy.
2.6 rocks. And a lot of distros have plans to release 2.6 based releases in the first quarter of 2004, which will greatly improve the user base.
IMHO, a good feature freeze, as Marcelo said somewhere in LKML, is 2.4.24 or even 2.4.25.
It's no time for a flamewar to begin. The Beaver is in the building.
.. what SCO will have to say about this...
Fun for the whole family with guess where the patch was applied From the snapshot directory
bk6 - 424K
bk7 - 964k
bk8 - 1.2M
Well thats increased the kernel by about another 5-10%. However I would say I do like xfs and its proven quite stable now.
Rus
Cheap UK and US VPS
after a code review by Christoph Hellwig
Incidentally, this is the Christoph Hellwig who contributed code to the kernel on Calderas behalf. His contributions may become an important point in the SCO-IBM-RedHat battle.
Any sufficiently advanced libertarian utopia is indistinguishable from government.
After patching every single kernel thats come out since the early 2.4s, I now have a kernel that I don't need to patch. WOW, about darn time!! Perhaps I'll even get lucky enough that RedHat and others that do not support XFS yet will build it into their kernels. That will make MY life easier, and updates go faster.
We chose XFS after lots of serious testing. It beat all comers at the time and we've been using it ever since. The only downside to XFS is file deletion times are a bit long, especially compared to Reiser, but when you have a server that is uner HEAVY load (Databses, mail servers) and with LARGE files (log server) nothing beats XFS.
Thanks guys, this is one of those merges that has made me estatic!
Angry People Rule
"Science is about ego as much as it is about discovery and truth " - I said it, so sue me.
yeah, they probably sent Vito down to visit Marcelo with a suite case full of jacksons and a simple message to deliver. "Marc... look we've got this fine filesystem here. you've seen the code. its nice code. but, you've got our competitor filesystem code in your kernel. now before we get our brawls all in an up roar, all we're asking is you HIGHLY consider putting this here XFS filesystem into that there kernel you're maintinging. Now, be a good boy, Marc, I know you'll do the right thing"
me i always hate seeing commercial quality products ending up on free software. the disgust.
Mandrake has offered XFS since at least 9.0, my first Linux distro. I've been using XFS (at the suggestion of my friend who helped with the install) for at least 6 months now, with only instance of a problem (not sure if it was a fault in the filesystem itself): lost or corrupted an inode or two, and fixed very easily once I knew what to do.
It works with both GRUB and LILO, is reasonably speedy, and has enormous partition and file size limits.
Count me a happy customer.
~~LF
This "big merge" has nothing to do with vendor pressure. The XFS patches have been available and well-tested throughout most of the 2.4 kernel's life cycle and since its (XFS') stability has already been proven to play nicely with the rest of the kernel, it's quite appropriate to do a merge so late in the 2.4 tree's live cycle. The team at SGI that handles merging the XFS code into the kernel have done a very good job of keeping up with bug reports and changes in the kernel vfs code.
Marcelo probably shares my opinion in that the current XFS code has been around long enough, demonstrated stability, and successfully merged with every recent 2.4 kernel back to at least 2.4.1x, that it's more than suitable for inclusion in the main kernel source without risk of introducing instability.
The only clashes I've ever seen with XFS and other code was with other 3rd-party patches, such as the ACL support in grsecurity. Those are now "switchable", anyway.
That is not true. The biggest hold back during the past 3 years has been the fact that the VFS layer needed a number of alterations and so far Marcello did not merge XFS because of this.
It wasn't untill Cristoph OK'd the VFS changes that Marcello merged the XFS core.
SGI as a vendor has had nothing to do with it. Buy a altix 3000 and they would happily maintain any special patch you would need for that (IA64) machine.
I think I know what I'm talking about since it's my name on the XFS FAQ. And no I don't work for SGI.
Where's the light switch. Seth
This was just mentioned here on /. the other day, but according to this article on Groklaw, Christoph Hellwig is (was?) a Caldera (SCO) employee.
SCO is going after SGI for XFS, when one of their own employees was working on it.
Extended ACLs, btree filesystem structures to facilitate huge files, fast sparse files, large directories, fast deletes, and a couple other niceties that would have required huge functional changes to ext2/ext3 to implement. It's also completely 64-bit clean, as it has from its conception.
The btree-based storage structure is already employed by reiserfs in a similar manner, but XFS' implementation has been stable (used in IRIX) for quite a bit longer.
I've been using XFS in production servers for more than two years already without any problem, it was time that it was merged into 2.4 kernel...
Maybe this way RedHat begins to support it for their installations
May the source be with you!
from the journals-are-for-girls dept.
:-)
Huh? I always thought it was diaries that were for girls... at least, that's what I told my friends when they made fun of my journal.
Laugh at stupidity: mod idiots +1 Funny.
Iirc, isn't he a former SCO/Caldera employee who was heavily involved in developing SMP and the Linux port of JFS? Iirc, Groklaw has a thing on this.
Now he works for SGI. The question I have is this-- it seems as if he has a conflict of interest to give his employer a beneficial review of the driver in order to ensure that it is included. I wonder how independent he is in his review.
That being said, he has been an important contributor to the kernel, and so I will give him the benefit of the doubt. I just wish that some sort of third-party review would have been done.
OTOH, there has been some speculation that Marcelo was biased against the inclusion, so maybe this balances things out.
LedgerSMB: Open source Accounting/ERP
From what I could find out XFS is the only Linux filesystem which stores quota information as meta-data-- there's no risk of an XFS filesystem getting its quotas "out of sync" with the contents of the disc and having to run a tedious quotacheck. We recently deployed it as a backup server and it's working very well!
Matthew @ Bytemark Hosting
I've been patching XFS support into my distro for a few months now. Let me get you caught up briefly. We had ext3 patched with acl support. That patch tends to lag behind kernel versions a little, which is not a problem if you're running a distro's standard back-ported kernel. However, I just undertook a migration on a live production box from ext3+acl to XFS on a second attached disk array, so I've been in Patch Hell for the last few weeks. I will say up front that I am not a k3rn31 h4ck0rz, but I get more done with it than probably anyone I personally know. That I got as far as I did in this story amazes me even today!
:-D
SGI ports their patch up to the latest kernel within a few days, but they have a nasty habit of removing older versions from their downloads when newer versions come out. When I only had the ext3+acl patches for kernel 2.4.20, and acl.bestbits.at was down for over a week ('grumble'), SGI only had XFS patches out for kernel 2.4.22. Andreas was kind enough to personally provide me some 2.4.22 ext3 patches. By the hardest, I got my 2.4.22 kernel built on my file server with ext3+acl and XFS.
The next DAY, I read of a root exploit in 2.4.22. The patch from kernel.org rendered my ext3+acl patches incompatible, and I'm not the type of guy yet to divvy up patches into even smaller pieces on any sort of schedule. I had to either forego backward compatibility or maintain a shell exploit in an environment where people do have shells.
I found, just yesterday, that Red Hat's newest kernel package includes xattrs and acls for ext3 and the 2.4.22 exploit's bugfix. It won't accept my xfs patch for 2.4.23 over some posix_acl and kdb conflicts. I found yesterday that SGI's latest kernel image has XFS and ext3+acl, but not the bugfix. The 2.4.23 patch broke my build. I find today that XFS is about to be added onto the native kernel tree, which just received both the bugfix and the ext3+acl extensions.
It's about TIME!
-j
http://epoxy.mrs.umn.edu/~minerg/fstests/results.
Of course your mileage may vary but I generally got results consistent with those cited.
My own experiences (I have used both reiserfs and xfs with 2.4.20 kernel:
You can defy gravity... for a short time
(from http://www.sgi.com/software/xfs/overview.html)
Guaranteed Rate I/O
XFS is the only file system available that provides a guaranteed rate I/O system, which allows applications to reserve specific bandwidth to or from the file system. The file system can determine the available bandwidth and guarantee that a requested level of performance is met for a given time. This functionality is critical for media delivery systems such as video-on-demand or data acquisition.
Expanded Dump Capabilities
Unlike traditional file systems, which must be dismounted to guarantee a consistent dump image, you can dump an XFS file system while it is being used. The XFS dump utility, XFSdump, can dump an entire filesystem, a directory tree, or specific files. XFSdump is restartable, which allows a large dump to be spread over an extended period of time or to be resumed after a system restart.
-->tech stuff
Not a whole lot, really. I haven't used XFS and haven't read the article, so I don't have anything useful to say on the subject.
Plus its sure to piss SCO off :)
;-)
That is not the half of it. You see-- Hellwig is a former SCO employee who when he worked there, worked with IBM closely on their port of JFS to Linux. He was also heavily involved in the SMP development process too. Just do a search for his name and SCO and Caldera on your favorite search engine. I think it will be hard for him to avoid a deposition
Now he works for SGI.
LedgerSMB: Open source Accounting/ERP
GRIO is not available on Linux, because it requires a lot of other support in the kernel proper, in the various I/O subsystems etc.
however, the realtime subvolume, which is a component of GRIO, is available for use on Linux.
When using the shred utility, I get warnings about not using the utility for journaled filesystems.
So what utility can I use to securely delete data on a journaled file system?
And being a previous windows user, I really liked the BCWipe utility that securely wiped unused areas of a partition. Is there an equivalent in linux systems?
I guess you missed the article on Using the Real ntfs.sys Driver Under Linux, eh?
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Now that the VFS layer has been stabilized and supportive of such FS drivers as XFS, very little needs to be changed to add XFS support. It's almost completely "additive", rather than modifying the existing code.
One of the XFS authors said anyone who wants to undertake such a port -- "go for it".
Considering the difficulty in ensuring data integrity and support for B-tree arranged data, Microsoft would not look kindly upon XFS being ported to NT, since their next generation OS is supposed to include database like features to speed up indexing and accessing data like XFS already has built-in. It would really rain on their parade. Also, benchmarking shows NTFS is considerably slower than XFS (or FAT32 for that matter) for large files and NTFS has no support for Real-time I/O partitions or journals being located on separate disks.
NTFS also requires (according to ad-copy) constant defragmentation due to their primitive block allocation scheme while XFS does quite well even without the XFS FSR (File System Reorganizer). XFS's FSR was created for 1 specific customer who had a particular application that generated excessively fragmented disks. Before that, an FSR (/defragmenter) wasn't considered necessary because XFS is intelligent about how it lays out files when they are written and how it stores free space (with free space also stored in ordered B-tree's by powers-of-two size of the free space blocks.
The only benchmark I've seen XFS run noticeably slower on linux, on is deleting large numbers of small files -- something one doesn't notice on IRIX, since the space deallocation happens in background on IRIX, and only the inodes need be marked deleted before the user prompt returned. I seem to remember on Linux the space had to be deallocated synchronously for some reason or another.
Makes sense given the way free space is managed -- when files are deleted, free blocks are recursively combined with adjacent free blocks to create the largest possible 'free block size' (I think up to 128k blocks, default=4k block size) (my numbers may be a bit rusty). Free space blocks were combined asynchronously, under IRIX (as I understand it), in a system thread after the last reference to an inode was released. Linux, if I remember correctly, didn't support the facilities for such a background thread -- thus the block combining happens synchronously, explaining the performance hit for file tests that delete lots of small files: there are many small free blocks that are candidates for being merged with adjacent free space.
I'm not entirely sure why a special "XFS_del" process couldn't be started at system run time who's sole purpose was taking unreferenced inodes and doing the space combining in background, allowing foreground programs to continue asynchronously after simply marking the inode as unusable and enqueing it to the XFS "free space" combining process. It is quite possible some of this has been implemented and my information is dated. But free space combining on cleanup is one of the main reasons why, historically, XFS file systems, didn't need to have _continually_ running programs like Executive Software's, _DiskKeeper_, running, full time in background: because XFS had it's own built-in defragmentation every time a user did a file-delete.
For the degenerate case -- *one* customer was not getting sufficient speed for real-time, uncompressed video recording to disk (back in the early to mid 1990's when disks were much slower). The swat team, assigned to the problem, found that the customer's particular use kept many small files around while deleting some files in a way that prevented automatic space consolidation. This odd usage was just enough to slow down direct-to-disk video recording (something quite difficult on systems in the early to mid 90's when disks were not so fast and SCSI-2 was still state of the art). To solve this problem for *one* customer, the "xfs_fsr" util was written.
To make the most of the efforts spent on the one customer, SGI incorporated xfs_fsr into the general OS to be run occasionally to stave multi-month/year buildup of possible, similar degenerate cases. I.e. XFS customers considered fragmentation such an unlikely / non-issue, that the X