Apple Discontinues ZFS Project

Zaurus writes "Apple has replaced its ZFS project page with a notice that 'The ZFS project has been discontinued. The mailing list and repository will also be removed shortly.' Apple originally touted ZFS as a feature that would be available in Snow Leopard Server. A few months before release, all mention of ZFS was removed from the Apple web site and literature, and ZFS was notably absent from Snow Leopard Server at launch. Despite repeated attempts to get clarification about their plans from ZFS, Apple has not made any official statement regarding the matter. A zfs-macos Google group has been set up for members of Apple's zfs-discuss mailing list to migrate to, as many people had started using the unfinished ZFS port already. The call is out for developers who can continue the forked project." Daring Fireball suggests that Apple's decision could have been motivated by NetApp's patent lawsuit over ZFS.

Comments

Github project taking up the slack

[KillNateD]

Dustn Sallings put the code on Github and has already hacked some basic Snow Leopard support and a minimal installer:

http://dustin.github.com/2009/10/23/mac-zfs.html

Code's here, fork away:

http://github.com/dustin/mac-zfs

Correction

[toby]

A lot of confusion has resulted from labelling ZFS a "filesystem". It actually combines both volume management and filesystem layers to achieve unique levels of performance, manageability, and data protection. Merits close study, as the concepts of ZFS overtake current best practices, conventional filesystems and RAID. You can get this taste of the future today, if you're using Solaris 10/OpenSolaris/FreeBSD.

Re:Correction

[BitZtream]

Warning: I have become a ZFS fanboy, my statements may not be entirely rational.

Use ZFS for a week on something like a file server, you'll be more than willing to ignore traditional Unix logic (which I firmly believe in) in order to use the goodness that is ZFS.

I'll admit, I'm a ZFS fanboy now, and I've only used it on FBSD 7.2, which is using a much older version of ZFS (v6, current is 13 I think). In 7.2 its not considered production ready and its still awesome to work with for me.

I shoved 4 PATA and 4 SATA drivers in a case, 2 gigs of ram (really the minimum for FBSD and ZFS on a 64 bit machine, which is where it sings). The 4 SATA drivers are in the zraid vdev, 2 of the PATA drives are in another mirrored vdev, and the final 2 in a second mirror vdev. So all the drives have redundancy and they are all in one big zpool with the total space available as one big chunk.

Now thats all well and good, but heres where it gets awesome. Want more space? Add some drives, create a new vdev, add it too the pool, instant space available. Okay, so thats not that impressive in and of itself. I can also add in a SSD as a 'cache' drive, which the ZFS system will then populate with a read only copy of data that gets accessed often in a random way, for a speed increase for your random IO needs.

Okay, so I've got a few terabytes of data available, but I need a to create a share for backup using TimeMachine on my Mac. Well time machine will be happy to consume all the space on the drive. No problem, create a new mount point in the zpool, limit it to 1TB. It will only consume up to 1TB, IF space is available in the zpool. I can also reserve the space if I want to ensure its there for the backups. You can use the same system for quotas of individual users.

I also have a bunch of ISOs with uncompressed data on them that I mount from virtual machines for various reasons, full of essentially text data. Welp, for that create another mountpoint on the same zpool, turn on compression, now my 5TB of text gets compressed into a few hundred gigs automatically and transparently. Its read only, and very important. I have backups, but they are in another state and transfering them back would be a painfully slow process which would cost me a lot of time. So I set the mount point to read only and set copies=2. Now the mount point is read only, and the data is stored twice, on 2 different vdevs. So not only is the data on a raid or a mirror, its on BOTH. If I was really worried I could set copies=3 and it would be on all the vdevs, so as long as one is usable I have my data, assuming all the vdevs have enough space to store the data. One of them doesn't, so copies=2 is the only useful option to me.

I also have a software archive of all my commercial windows software, I want to keep it safe from any sort of infection or modification, so I set that mount point readonly.

So far, I've done nothing that can't be done already with existing methods, but what I've done it across a common shared data pool. Free space is shared across all of them.

I thought ZFS was a waste of time until I started using it. I am by no means a ZFS master, but I've learned that it can do some pretty powerful things. My setup is small, I only use it at home until FBSD 8 is released, which will have what is considered to be a production ready implementation, but I will be moving to it at that point in our internal office servers.

I know I haven't listed any life changing reasons to use it, but its turned me into a fanboy.

Technically, it IS modular, even if it doesn't have well defined zones. If you look at ZFS in a slight different way, it can be just one layer of the system. You can create virtual devices in a zpool and treat them as a block device (its just another /dev entry). Just to see how it worked, I created a virtual device on top of the zpool, formatted it with UFS, put some files on it, expanded the virtual device, ran growfs and had a larger UFS mount point.

Re:Correction

[TheRaven64]

ZFS is modular, it just doesn't have layers in the same places at the conventional block device, filesystem, vfs layers.

At the bottom layer, you have the pooled storage layer. This is comprised of several modules. The lowest layer is responsible for combining virtual devices into storage pools. This handles things like stripping and mirroring, but it does so with dynamically sized block ranges, rather than static ones (i.e. partitions). On top of this you have the transactional I/O layer. This handles things like checksums, encryption and compression, and provides support for transactions. Every disk access goes via this layer. That means that every single block device access from the upper layers gets transaction support, so you can guarantee that the block devices are always in a recoverable state; every group of low-level I/O operation either worked or failed, it never partially worked. Above this is the adaptive replacement cache, which handles caching and, with the L2ARC can also perform caching in Flash as well as RAM.

On top of that, you have the transactional object layer. While the pooled storage layer works with a flat 128-bit address space, the transactional object layer works with objects and sets of objects. Any higher-level I/O is expressed in terms of transactions containing modifications to sets of objects. This is a very powerful abstraction. An object can be a complete filesystem, some metadata, or a file; there is no difference at this layer. Anything that you can do with one, you can do with any of the others. There are a few other things in this layer, like the ZFS intents log, which handles a higher-level (finer-grained) transactional model.

On top of this is the user interface layer. There are two things here, the ZFS POSIX Layer (ZPL) and ZVOL. The former produces something that looks like a UNIX filesystem, with NFSv4 ACLs and a few other nice features. This is close to a filesystem in the traditional model, but also a bit like a VFS. It provides something that looks like UFS on top of lower-level features, but these are not on-disk structures. ZVOL is simpler; it provides something that looks like a block device. You can export this via iSCSI, for example, and put any kind of filesystem you want into it. Unlike a real block device, this one supports copy-on-write, snapshots, transactions, and so on. You can, for example, have a Windows VM (or real system) with a FAT or NTFS filesystem in a ZVOL mounted via iSCSI. You can snapshot it with the same O(1) snapshots that the rest of ZFS has, and then restore it later. You can also use zfs send and zfs receive to stream the changes across the network to another machine, irrespective of whether you're using ZPL or ZVOL + some filesystem. You can also use ZVOL for things like UDF images for burning, creating a skeleton image, cloning it, and using the native copy-on-write support to make small changes to it.

Re:The straight dope

[setagllib]

You really need to subscribe to the mailing list. The rate of development is only growing -- it's just now moved on to a lot of smaller features and improvements, now that most of the work is already done.

Re:The Reason is Probably Technical

[4iedBandit]

I'm sure it will, but I'm afraid that doesn't mean that made it practical for Apple to integrate into OS X or that it fitted the use cases they needed for many desktop scenarios.

Um, the technical work was already done. It could have shipped with Snow Leopard. Again, the reason it didn't has nothing to do with the technical feasibility of it.

Ultimately, the only way to deal with silent data corruption or 'bit rot' is to have multiple levels of redundancy several times over for your data - which ZFS has and deals with. No desktop Mac can ever have that.

Why? Because you say so?

Anyone who thinks that is anywhere near being practical to deal with on a desktop system is an idiot

While I may be an idiot, you have to convince me that ZFS is not practical for a desktop. Again, just because you say so is not reason enough. I stand by my statement that ZFS is the only file system with enough benefit to make me explicitly choose it for building servers. You may argue that there's a difference between a server and a desktop but those really are nothing more than abstract concepts. A file system that has too much overhead for my desktop has too much overhead for my servers. Performance matters. ZFS may not be the fastest, but it is no slouch either and the other benefits it brings to the table far outweigh miniscule performance concerns.

By no stretch of the imagination does ZFS handle this 'magically'. There is a severe price to be paid.

What exactly is this severe price? Can you spell it out? Exactly? "Any sufficiently advanced technology is indistinguishable from magic." In that respect yes I will say it is magic because it is head and shoulders more advanced than anything else I've had the pleasure of working with. File systems have not had this kind of improvement in decades.

I'm afraid that hardware, bad sector and disk issues are far, far more prevalent problems than data corruption at an OS level...but I'm afraid it's just not a primary concern for everyone else or for those developing desktop operating systems.

How do you know? It's not a significant problem till the data you need is unavailable when you need it. At home my own modest media library sits on just 500 Gig with no guarantee that any of it will still be whole in 6 months. Sure I back it up. Routinely. But until you access the file you don't know if it's been corrupted. Then how long as it been corrupted? Do your backups go back far enough to compensate? Yes you can checksum everything routinely and maintain a database of checksums to validate file change. Part of the beauty of ZFS is it does this with every thing you put in it, at the block level, and it validates the checksum every time you read the data. If a block fails the check, it not only sends you the valid block from the mirrored copy (You do have redundancy right? Even ZFS won't save you if you only have one copy.) but also replaces the bad block with a copy of the good one.

Storage capacity is skyrocketing. Going to backup to fix problems is a real problem in itself. Are the tapes on-site? Do we have to go to the vault to find an uncorrupted copy? Did the media pool get recycled and now there is no uncorrupted copy? Do you enjoy explaining to an executive why the data they want is unavailable despite spending millions on enterprise class storage and backup solutions? The problems of enterprise storage are becoming problems of home users. I have three terabytes of storage just to backup my home system in a replication layout I'm okay with, but I really would have loved the protection ZFS offers against bit rot to top it off. Stick your head in the sand if you want, but I consider my data and it's availability a little more important. ZFS handles it elegantly, in the background, with negligible performance hit.

Re:This is devastating...

[mysidia]

I tried the zfs commands you suggested but there aren't any of those available in my path,

The two programs used to work with zfs are 'zfs' and 'zpool'. Both of them are normally located in /usr/sbin

If your PATH is broken so you can't easily access the tools, it doesn't matter what filesystem you're using, you'll have a bad time. I would suggest adding /usr/sbin to your PATH if it's not, or else use "/usr/sbin/zfs list" etc.

the user's, life is made harder?

The user's life is not made harder. It's no harder to learn "Zfs list" than to learn "df"

What is all so totally worth it? I haven't seen any advantages of zfs over hfsplus.

Copy on write filesystem, performance is excellent..

Unlimited impact snapshots which have no I/O performance impact, and has the ability to rollback to most recent snap, clone snapshot, etc; makes backup, replication type tasks easy. The ability to implement transactional unbreakable system upgrades, ala apt-clone.

RaidZ, to protect against drive failure

Ditto blocks and checksums to ensure data integrity, protect against silent data corruption, heal bad disk blocks.

No need to 'fsck'

No need to decide the size of each file system at creation time, 'quotas' are soft and can be changed, instead of having to re-format/fdisk to change sizes of a dataset.

No need to manage individual disks. No limits on number of disks imposed by the filesystem layers.

No arbitrary limits in zfs. ZFS can scale to (in theory) 256 quadrillion zettabytes. Other filesystems have a max file size limit, and a max filesystem size limit.

Sharing a folder is a simple invokation of a "share" command.