ZFS Confirmed In Mac OS X Server Snow Leopard

number655321 writes "Apple has confirmed the inclusion of ZFS in the forthcoming OS X Server Snow Leopard. From Apple's site: 'For business-critical server deployments, Snow Leopard Server adds read and write support for the high-performance, 128-bit ZFS file system, which includes advanced features such as storage pooling, data redundancy, automatic error correction, dynamic volume expansion, and snapshots.' CTO of Storage Technologies at Sun Microsystems, Jeff Bonwick, is hosting a discussion on his blog. What does this mean for the 'client' version of OS X Snow Leopard?"

Comments

What does this mean for 'client'?

[daveschroeder]

Nothing, in particular. It means that ZFS isn't going to be officially supported and/or promoted on client. But, since Mac OS X and Mac OS X Server are essentially the same OS with some different/additional pieces on the top of Server, and like other filesystems that were exposed via the GUI tools and supported on Mac OS X Server, but not on Mac OS X, in the past -- such as Mac OS Extended (Journaled, Case-Sensitve) -- it will likely be available via the command line tools, and usable by people savvy enough to work with other boot devices to format the volume in the desired fashion, etc.

Re:What does this mean for 'client'?

[Goaway]

You should perhaps consider taking the internet less literally.

Re:What does this mean for 'client'?

[Just Some Guy]

From what I understand, ZFS is fast not memory efficient.

Nitpick: it definitely likes a lot of RAM, but it doesn't necessarily use it inefficiently. Car analogy: a semi truck is fuel efficient, even though it's gas mileage is a lot lower than a sedan's.

Re:Finaly

[jandrese]

8TB is rapidly becoming "not that much stuff" these days. You can already buy 1TB HDDs, so we're just three doublings away from hitting the limit with a single drive (not to mention RAID arrays).

Re:How will I benefit?

[fishdan]

This is very informative

Re:How will I benefit?

[profplump]

What is it with you people and filesystem-level snapshots?

I'd much rather have volume or block level snapshots, like with LVM and other similar systems. Those systems provide RO and RW snapshots, dynamic partitioning, drive spanning, etc., and can be easily layered with other block-level components to provide compression, encryption, remote storage, etc. as well. All that without tying you to a single file system (though that may be a moot point on OS X, as it will only boot from HFS/HFS+ AFAIK).

If you really wanted to you could even write a script that takes no arguments other than a path name and automatically created a series of volumes of an appropriate size for the folder you selected, setup software raid to mirror them into a single device, mount the device with a compression filter, format it (with any file system) mount it normally, move the data over, drop the old data, rebind the mount point to the old path name, and update fstab. The only thing you miss here that ZFS may be able to do (I didn't check) is avoid closing the files that are moved.

I'm not saying the features ZFS has are useless -- I think they are great -- they just aren't all that new and exciting. They might be new OS X, or repackaged in a way that's easy to consume, but they are things that anyone with big disks has been doing for years.

Re:How will I benefit?

[Cyberax]

SMART sucks. That's just a fact - very often it kicks in when your drive has failed.

Also, there are lot of real cases where malfunctioning drive can silently write incorrect data. ZFS will help you in this case.

Re:How will I benefit?

[Lars512]

For home users, it would let you simply plug a new drive in your Mac, press a button, and have it just add space to your main drive. You wouldn't need to specifically setup a RAID. No resizing. No "external drive" if you don't want it that way. Just buy a drive, plug it in, and it's all handled for you.

I'm not sure you'd want it to work this way for external drives. Will they be available at crucial parts of boot time when some important files are striped across them? Even if they are, you're basically unable to ever remove the external drive again. If there's a problem with the drive, all your data is lost. Probably the way these drives work now is better. Maybe mirroring onto an external drive would work ok, but it would then be an undesirable write bottleneck.

Re:Standardizing file systems

[larry bagina]

1. The GPL prevents ZFS integration (without a complete and total reimplementation of all the code... which won't happen since everybody prefers to write their own filesystem)
2. MS DOS/FAT is the universal file system for operating systems.

Re:THIS:

[Anonymous Coward]

I was referring to MacFUSE itself, not what it does -- it is a native pluggable filesystem. Without such support in OS X, MacFUSE wouldn't be possible. (Being an add-on itself is its goal: many interesting research-level filesystems are written to the FUSE interface, which runs code in userland, rather than to several OSes' individual kernel interfaces.)

But if you don't like that example, try Apple's own ZFS kext.

It seems apparent that you aren't a kernel programmer, but could you at least find one with even a passing familiarity with OS X before claiming it doesn't have a feature it obviously does have?

Re:How will I benefit?

[MrMacman2u]

Actually S.M.A.R.T. is an amazing tool and is utterly invaluable in monitoring drive health... IF and ONLY IF you have the appropriate software (windows: google it, *nix: smartmontools) AND know how to read the resulting output.

The reason many people think SMART sucks and I say to check SMART manually is because 95% of drive manufactures set the threshold or "fail" values WAAAAY too high or low!

I use SMART constantly (about once every other week) to "check in" on how healthy my drives are and knowing how to read the values the software returns has saved my data many times. In fact, due to certain SMART values, I KNOW my Thinkpad hd is on currently on it's way to failing even though it is still working fine, for the moment. The SMART information has let me know it's on it's way out the door and therefore I have taken precautions to safegaurd the data on that machine and have the drive replaced.

Granted, SMART can't inform you ahead of time about sudden and complete mechanical or electronic failure, it can warn you if your drive is slowly the kicking the bit bucket. (With a small amount of know how until drive manufactures wake up and set more appropriate values)

Don't knock something you know nothing about! Kthnxbye!

Re:How will I benefit?

[Cyberax]

Maybe I'm unlucky, but I had three notebook HDs die on me without any warning. Even though I'm using 'SmartMon' program which should warn me about worsening drive condition.

Also, Google's on hard drive survey seems to come to the same conclusion: "One of those we thought was most intriguing was that drives often needed replacement for issues that SMART drive status polling didn't or couldn't determine, and 56% of failed drives did not raise any significant SMART flags"

http://www.engadget.com/2007/02/18/massive-google-hard-drive-survey-turns-up-very-interesting-thing/

Re:I don't have to be...

[Anonymous Coward]

Okay, then, would the phrase "out of the box" clarify matters for you? Meaning you want Apple to ship more filesystems with OS X? I can understand wanting that, but it seems odd to claim it's a requirement for a modern OS. Usually it's better to provide support for loadable filesystem modules, no matter how many filesystems you ship yourself, so that third parties can add whatever you don't do yourself. Even Linux's wide range of filesystems are mostly supplied by third parties, not by the primary kernel maintainers.

Perhaps I could have made this clearer, but from the context I thought it was obvious that I was referring to native, out-of-the-box support for pluggable filesystems. NOT some kind of add-on you can get later and have to compile yourself, then tweak a bit before you can even get it to work (which is what all the examples here have stated they require). Apple's ZFS is provided in binary form. The installation process is manual because it's developer/alpha quality, and you're replacing the read-only ZFS support that ships with 10.5, but no compiling is required. This is what will eventually ship with Snow Leopard.

MacFUSE provides installable binaries. Once installed, you can use any of the FUSE filesystems that their developers provide in binary form for OS X, such as sshfs or NTFS-3g.

If you install MacPorts, you can choose from their collection of FUSE filesystems, without dealing with MacFUSE etc yourself. This is similar to using package repositories on popular open source OSes.

Hell, even Apple's developer demo MFSLives ships with PPC binaries, although I haven't the foggiest idea what anyone would practically use it for.

All without compiling or tweaking anything.

The bottom line is that Apple does, in fact, provide native, out-of-the-box support for pluggable filesystems in OS X. Apple does not provide a wide variety of filesystems itself, but neither do most other OSes, and that's a separate issue from support for pluggable filesystems. After all, there's no need for pluggability if you ship all the filesystems yourself...

Re:I am not sure where you are failing to understa

[Anonymous Coward]

First, if you have to INSTALL SOMETHING (even a small something) other than the filesystem itself, BEFORE other filesystems can be used, then by definition it is not "out of the box"!

Sorry, but you're the one who isn't understanding -- you don't have to do any such thing! Apple's own ZFS is an example of that. The developer samples they provide are examples of that. Inactive open source projects like ext2fsx are examples of that. The various commercial (e.g. Paragon NTFS) and in-house filesystem extensions in use in obscure places (e.g. VirtualPC, Parallels, Symantec's filters, etc) are examples of that.

It's just that not many non-commercial developers are providing filesystem modules for OS X. Filesystem development is some of the most demanding work there is. That's why FUSE is popular: the filesystems are implemented as userland applications, where the demands of the programming environment are much less intensive.

Everything that uses FUSE (Linux, NetBSD's PUFFS, MacFUSE for OS X, etc) takes the form of installing a framework module in the kernel, which in turn uses the FUSE interface for the userland filesystems. This has absolutely nothing to do with out-of-the-box support for anything; it's just an abstraction that's made it easy for people experimenting with filesystems across several OSes, thus more filesystems are available with it. If you don't want to use that abstraction, that's fine, but it has nothing to do with an operating system's support for pluggable filesystems (other than showing it exists in the first place).

Ignoring the first link I gave you, which details the level of support available in OS X according to Apple itself, and instead trying to play semantic games using a single sentence from the MacFUSE homepage in order to justify the incorrect position that OS X does not support pluggable filesystems is the height of arrogance.

Second, according to TFA, ZFS will be provided for the server product, not for plain off-the-shelf, out-of-the-box Mac OS X. And that was part of my point.

ZFS is provided by Apple for OS X Leopard right now; it's just not officially supported for production use because it's a work-in-progress and still contains bugs. It will continue to be available for Leopard, and will therefore also be available for the non-server version of Snow Leopard.

Apple may, of course, choose to not officially support it on the non-server version, much like they did for both the journaled and case-sensitive versions of HFS+ in the beginning. If that's the case, then it will likely be supported at some point after Snow Leopard.

But what exactly is your point over that? And what on earth does it have to do with pluggable filesystem support? It's just a question of what Apple chooses to provide itself, which you already said was not an issue.

Third, MacPorts (like the other examples you gave) is yet another outside-party add-on; it is NOT an "out-of-the-box" feature of OS X!

MacPorts is a package management system and repository for OS X. Of course it's an add-on, just like all the package management systems for every other OS out there. I mentioned it because it's a convenient method of obtaining software and may be familiar to you if you like other package managers. But what does that have to do with pluggable filesystem support?

And fourth, I have an excellent grasp of what "pluggable" means. That has no bearing whatever on whether it is supported out-of-the-box, versus a self-compiled add-on or the like!!!

I remind you of your assertion in your original post:

If Apple really wants to start calling OS X a "modern" operating system, they are going to have to start supporting pluggable filesystems.

You don't get to change the argument now. You're absolutely ri

Re:No, I am understanding just fine.

[gutter]

It is pretty hard to tell from your tirades whether you are talking about the ability to support pluggable filesystems, or the availability of those pluggable filesystems. You seem to be conflating the two. You start by complaining that OS X lacks the ability to support pluggable systems, but the first link from the AC's post proves you wrong:

http://developer.apple.com/qa/qa2001/qa1242.html

In fact, every filesystem OS X supports is written using this mechanism, out of the box:

[gutro:~/] gutter% ls -1 /System/Library/Filesystems/
AppleShare
URLMount
afpfs.fs
cd9660.fs
cddafs.fs
ftp.fs
hfs.fs
msdos.fs
nfs.fs
ntfs.fs
smbfs.fs
udf.fs
ufs.fs
webdav.fs
zfs.fs

Your most recent tirade seems to be a complaint about the lack of available filesystems, which I guess is a reasonable complaint, but that's not what you orignally asked for. Then you asked for a simple package you could download and install, and again, the original reply contained one (MacFUSE). Granted, that's a poor example, because it hides OS X's native pluggable FS support behind the FUSE pluggable FS support, but that doesn't mean that the AC was wrong. You can go and download the MacFUSE package, and the sshfs package, install them using the standard installer, and begin using a filesystem that works over SSH, no compiling necessary. (Incidentally, that one is super handy).

In short, the original reply by the AC was 100% correct, and you were 100% wrong, (and seemingly unable to comprehend his reasonable explanations) and somehow by sheer bluster, you seem to have convinced everyone of the opposite.

Re:No, I am understanding just fine.

[Anonymous Coward]

First, the ARTICLE we were supposed to be talking about mentioned ZFS for their future server product. It was not mentioned that it would work as-is in already-distributed OS X. If it does, then great! But you (and some references I have seen) mention that it has bugs. So... it is not a supported product yet. That was part of my point. Because it is not yet supported, it is not yet an example.

The fact that it works at all makes it a perfectly good example of OS X's support for pluggable filesystems. It's not a good example of a finished product, no, but that's not what your original post was about. The issue was whether OS X supported pluggable filesystems, not whether people have created complete, polished filesystem plugins for OS X.

However, I went back and looked at your first post to see what you meant. And indeed, there was a link there that I missed the first time. I did not "ignore" it, I simply did not see it the first time around. And yes, it does say that 10.4.x and later support VFS plugins. My bad on that one. But I did not see that ANYWHERE else, and the examples you were giving were NOT examples of that: they were examples of plugins that worked through a third party add-on. Thus that misunderstanding.

Yes, MacFUSE was a bad choice for a first example on my part.

MacPorts, for example, is an easy to get add-on for OS X. But it is NOT a "feature" of out-of-the-box OS X.

For clarity's sake, note that MacPorts itself is essentially a collection of open source software packaged specifically for OS X. In the context of filesystems, it's simply an alternate way to get MacFUSE and a few filesystems that use FUSE, with the MacPorts people shouldering the burden of making sure they install properly on OS X without hassle. MacPorts doesn't provide any new filesystem support itself, and makes a lot of other software available.

As far as I have been able to tell with brief looks, the filesystem examples I have seen all work through some other add-on that has to be installed first. (Aside from ZFS, which as mentioned does not count). And that was the point I have been trying to make.

But the only relevance here would be to argue that the OS X kernel has no support for pluggable filesystems, and that a third party had to add that support through some other means. As I've been trying to establish for several posts, OS X does indeed have that support. The prerequisites of some projects are therefore beside the point.

I brought up kernel extensions that are separate from the filesystems themselves for precisely this reason, which you seem to keep evading: THAT IS NOT "OUT OF THE BOX"! That is an add-on. It doesn't matter if it comes from Apple, or Google, or Paul Reiser. You have to go get it and ADD IT ON. I do not understand why you have had such trouble with this simple concept. You may not care whether you have to compile it first or not, but whether YOU find it easy is NOT the point! A lot of people do care, and won't go to the trouble.

In the context of the FUSE filesystems, MacFUSE is simply a prerequisite, which is fairly common for software. Being an add-on has nothing to do with compiling; as I pointed out earlier, getting the popular NTFS-3g working merely takes 2 downloads. No compiling involved.

FUSE is popular simply because it's easier for developers than doing kernel extensions.

You are a bit late quoting about "out-or-the-box" and saying that I "changed my argument". I did nothing of the sort. Some posts back I clarified it for you, by stating that I felt that the context made it quite clear that I was talking about "out of the box" functionality... because it seemed clear to me at the time that you did not understand what I meant. But I do not understand, now that I have explained that many times now, why you have kept sidestepping that issue.

I sidestep it because it's a specious argument. Your original statement was that Apple n

Re:ZFS is fast, but not lightweight

[OrangeTide]

hehe.. very funny, WAFL is what is used in NetApp filers. You can read the patents on it and get a pretty good high level idea of how it works, the patent documents lays it out in pretty easy to understand terms. As for the mystery filesystem (starts with a D), it's like XFS but has more features to make it worth comparing to ZFS. The performance is similar to XFS too (a little better when it can hold transactions in NVRAM).

Even XFS is on par with ZFS (with lower cpu utilization) for nonclustered performance. XFS is so close to the theoretical maximum performance of a raw unclustered disk group (when tuned for the workload) that there is no "an order of magnitude or more" in those cases. When get get 99.5% of the raw disk I/O performance you're not going to reach an order of magnitude by making it .5% faster :)

Of course for a distributed cluster of disks the free version of XFS is nothing compared to ZFS, because XFS can't even really do clustering. And few people benchmark SGI's commerical CXFS, despite my experience with XFS even I haven't used CXFS.

Also it's hard to compare most filesystems to ZFS because most filesystems that have RAID just live on top of a hardware/software RAID implementation. XFS (and others) do a little bit of optimizations when they are on a RAID, but it doesn't really reflect in the performance numbers in a major way. ZFS has the advantage of RAID-Z, which is faster(usually) and far more flexible.

ps - I like to think of XFS as the base line of what a good filesystem should be, and you're either better than XFS or worse. I'm not trying to claim XFS is the fastest filesystem in the world or anything. Although if tuned for your workload the performance is quite impressive, when not tuned there is a lot of head scratching why all your RAM is gone and your performance is worse than FAT.