Apple Removes Nearly All Reference To ZFS

Roskolnikov writes "Apple has apparently decided that ZFS isn't really ready for prime time. We've been discussing Apple/ZFS rumors, denials, and sightings for some years now. Currently a search on Apple's site for ZFS yields only two hits, one of them probably an oversight in the ZFS-cleansing program and the other a reference to open source. Contrast this with an item from the Google cache regarding ZFS and Snow Leopard. Apple has done this kind of disappearing act in the past, but I was really hoping that this was one feature promise they would keep. I certainly hope this isn't the first foot in the grave for ZFS on OS X."

Comments

Well fuck it, we're going to 128 bits

[QuantumG]

cross-meme joke completed.

Death knell

[siloko]

I certainly hope this isn't the first foot in the grave for ZFS on OSX.

More like the last nail in the coffin . . .

Re:Death knell

[Macrat]

Or it's not ready for a consumer OS yet.

Re:Death knell

[speedtux]

The data loss and corruption that the parent is talking about is the fault of crap hardware. In almost every case, USB is involved, or more rarely the lack of ECC ram. It is true that ZFS is less tolerant of bad hardware.

What good is a fault tolerant file system if it isn't tolerant of faults?

With such hardware, it is impossible for any filesystem to function reliably.

Quite incorrect.

USB and Firewire bridges are notorious for this. If you care about your data, you should run the other way if you happen upon one.

Well, golly, those only happen to be the way 99.999% of Apple's customers attach exernal drives, not to mention 99.9% of all of the rest of the world.

Re:Death knell

[udippel]

I have been following the zfs-discuss list for years, and almost no one has lost data.

That's not good enough for the likes like me.
For the rest of your post, I am simply too lazy to prove you wrong. For a beer each I could fiddle out those that were confirmed to lead to data loss, including unrecoverable data loss, as I mentioned in my post.
But I won't do this (except for that beer each), because you know that best yourself:
The data loss and corruption that the parent is talking about is the fault of crap hardware. In almost every case, USB is involved, or more rarely the lack of ECC ram
Because this is exactly, word for word, the usual excuse given in the mailing list.
And I didn't add the one in my original post, when it was 'confirmed' that you need RAID if your data are valuable to you; and now, read this in bold: irrespective of hardware failure. I for one accept the need for RAID, in case of a hard disk really and effectively dying. Not for manhandling the data. Read the postings carefully.

Of course, the other person answering your flawed arguments about 'crap hardware' is right to the point: What good is a fault tolerant file system if it isn't tolerant of faults?
May I remind you, the premise and promise of ZFS was the atomic write, the always consistent state on the drive. I do think and believe this is true, and all blocks are either written and confirmed or just not. As far as I can make out, the problem has only been shifted: to the problem of metadata. Again, refer to the mailing list. Those exist in four-fold. Why? It seems the consistency of blocks on the drive being guaranteed, the layer of actually having the links to those correct data is more vulnerable. Think of a pool: if you jank the structure of a pool by janking a USB, you have 100% correct data (contrary to any other file system, I agree), but alas no more structured access to reassemble them (compared to inodes).

(The mods opting for 'informative' of your post obviously don't read the ZFS mailing list, and nobody blames them.)

Re:Death knell

[KonoWatakushi]

What good is a fault tolerant file system if it isn't tolerant of faults?

This is not about faults; ZFS handles those fine. This is about hardware which behaves badly.

You should not talk so authoritatively when you are so obviously ignorant of the subject. What you are implying, is that a filesystem does not care about the ordering of writes, and that is absolutely absurd. The ordering of writes is more critical for copy on write filesystems like ZFS, but in neither case is your data safe on bad hardware.

Like you point out, there is lot of bad hardware out there. What you overlook is that existing filesystems have no facilities to catch, much less correct such errors or corruption; that is why it is called silent data corruption. Even filesystems like HFS+ only journal metadata, so this is a lot more common than you realize.

ZFS can be improved on misbehaving hardware, but that still won't fix bad hardware. Those improvements will only allow the possibility to recover to a consistent state, but data will still be lost.

Re:Death knell

[wereHamster]

Every disk will corrupt eventually, it's just a matter of time. Not even the best hardware will help you there. So the question is, how well does the filesystem catch these errors and correct them. It turns out, ZFS is really bad at this, as it can get into a state where you can't even import the pool (where zpool either stops with an error and in worse cases causes a kernel panic). There have been numerous bug reports on the zfs mailing list and the opensolaris bug tracker. So far nobody seems interesting in fixing those. My pool got corrupted in such way. I had to manually poke around the filesystem and invalidate metadata until zpool was able to import the pool. Something that a 'fsck' could have easily done, but Sun refuses to create such tool because, according to them, ZFS is robust enough. All credits go to this guy who had the idea to invalidate the uberblocks directly on the disk: http://opensolaris.org/jive/message.jspa?messageID=318457#318457

Re:Death knell

[Charan]

What good is a fault tolerant file system if it isn't tolerant of faults?

Any time you read about a product that guarantees perfect fault tolerance, there is always a list of constraints that must be met for that claim to hold. You probably won't ever see this list marketed, but it's there somewhere.

I haven't looked into this, but it sounds like ZFS is fault-tolerant given a system model where data can change once it's on-disk, but otherwise system components are fail-stop. So if you ask a hard disk to perform a write barrier and flush its data to disk, the disk will either do so and report success, it will issue an I/O error, or it will catch on fire. Any way, ZFS will handle the situation correctly.

Of course, the immediate next question is whether that failure model is realistic. Turns out it isn't. Some hardware will report 'write barrier complete' when it still has unwritten data in its buffer.

If you can't count on your hard disk to flush its caches or even order writes correctly, I don't think it's possible to build any fault-free file system on top of the drive.

Re:Death knell

[Sancho]

The problem is that one flipped bit can cause ZFS to think that the whole pool is unusable - even though it keeps redundant copies of the metadata which it then completely ignores! What for does it keep the copies then?

Where are you getting this from? It's not in the link you posted.

In the link you posted, the admin found three uberblocks (there are supposed to be four). ZFS correctly made multiple uberblocks, per design. It appears that all three were corrupt. Why? Who the hell knows. Could be that the disk was going bad. Could be the problem I mentioned. Could be a ZFS bug. There's not enough information in that thread to say with certainty.

Now I'd say that ZFS "isn't ready for the desktop" no matter what the reason for the faulty uberblocks. I'm not supporting ZFS by disagreeing with what you're saying--rather, I'm railing against drives that do bad things and developers not being willing to work around them. One of these is the fault of Sun--and it's a major one. The ability to recover from hardware misbehaving is important.

However, the posting you're discussing doesn't have anything to do with "silent corruption"--it's explicitly about an unexpected halt, which causes highly unpredictable results when using drives that misbehave. Hence my bringing up that possibility in the first place. As I said, I don't know how the blocks got corrupted--but neither do you.

Take it from another perspective. If all of the copies of the superblock on an ext3 filesystem were simultaneously corrupted (it doesn't matter how), what would happen? Won't the drive fail to mount, or even fsck? (fsck requires a good superblock, I think.)

Perhaps it will BE ZFS just not BE CALLED ZFS

[KenCrandall]

WIth the impending purchase of Sun by Oracle, I'm thinking it could be one of 2 things:

1) ZFS will be killed and/or de-emphasized and/or re-licensed in such a way that Apple is not comfortable/happy with putting it into Mac OS

2) It will still be ZFS just not called ZFS anymore (either re-branded or forked by Apple or re-named by Oracle/Sun)

Re:Perhaps it will BE ZFS just not BE CALLED ZFS

[Bill, Shooter of Bul]

1) Oracle hasn't publicly said anything of that nature, nor is even any rumors to that effect.
2) They aren't mentioning the features that zfs provides under any kind of name

Most likely, they've been focusing too much on the embedded space with the iphone and didn't have the man power to integrate a complex third party FS into their OS. As it was only going to be for the OSX Server for "production servers", they probably thought that was the easiest thing to drop. I mean, lets be honest no one really uses OSX Server for anything really mission critical that relies on it for the kind of storage capabilities ZFS would provide. Do they? Feel free to correct me with real world usage senarios of OSX Server ( I haven't heard of much).

I want a universal filesystem

[Anonymous Coward]

Which one can you mount on Linux, MacOS and maybe even Windows without precarious hacks, and with journaling, long filenames, and maybe extended attributes? So far FAT and HFS+ without journaling seem to be about the only choices. ZFS would have been it if MacOS and Linux both ended up supporting it, but now neither of them do (without precarious hacks!)... so Solaris is off in the corner by itself again. Bah humbug.

When I dual-boot my Mac (Linux & Leopard) I'd like to have the same partition for home directory on either system. A better FS for thumb drives than FAT would be nice, too.

The situation is utterly pathetic.

Re:I want a universal filesystem

[JohnFluxx]

Why, it's almost as if Microsoft don't want to inter-operate. Ext3 is fully documented with viewable code, yet MS don't implement it. NTFS on the other hand has to be reverse engineered.

Re:I want a universal filesystem

[drsmithy]

Ext3 is fully documented with viewable code, yet MS don't implement it.

What's in it for them ?

Re:I want a universal filesystem

[JohnFluxx]

Exactly how they think. They don't care about making it easy for users to have a decent cross-platform filesystem. They want lock in and control.

That would be a new feature

[code4fun]

Snow Leopard is about performance and optimization. A new file system would fall under new features.

ZFS still needs more miles under the belt

[BlackSabbath]

I've played around with ZFS on the Mac a little bit. I've also played with ZFS at work (Sun UltraSPARC platforms) where we went from true believers to backing away rapidly (let's just say that there are certain Oracle workload profiles for which ZFS causes some massive performance hits especially when the disks are close to full).

I'm guessing that ZFS failed to meet at least one of (what I imagine are) Apple's criteria:
1. has to be simple to use
2. has to be rock solid

There's a good chance it failed at both. I'm not saying that ZFS is crap. Personally I think its a brilliant design, however it needs a bit more sunlight before its ready for the Steve.

Re:ZFS still needs more miles under the belt

[BlackSabbath]

We never got a reproducible cause/effect explanation from the Sun engineers (which is one of the main reasons we started backing away).

Our particular problem seemed to occur when free-space shrank to below 20% and we had workloads with large numbers of connections doing lots and lots of very small write transactions in Oracle (using Oracle AQ as the backing store for our ESB/BPEL implementation). It wasn't 100% reproducible but seemed to be linked to those configurations more often that not.

Having said all that, we never used ZFS for production systems (we are far too conservative a company). We used it for dev/test/UAT environments where the ability to clone large numbers of test environments cheaply, quickly and with very little disk space cost was of great benefit. Its still used in some circumstances, just not all of them. Horses for courses.

Re:ZFS still needs more miles under the belt

[BlackSabbath]

> you are running a database engine on top of ZFS you have to tune it to that specific database engine

Been there, done that.

> any IT worth their salt would not have let the disk get that close to full that it causes issues

I don't consider 80% a threshold which any FS should start to cause issues

> As for 1, ZFS is extremely simple to use

For unix admins perhaps. For the remaining rather large subset of Mac OS X users perhaps not.

> I don't think that Apple has had the time to make sure that everything fits in with their way everything has to work

Totally, 100% agree.

Re:ZFS still needs more miles under the belt

[chefmonkey]

Wait... what? You open by telling us that you have to perform custom tuning on the filesystem for certain applications, and later assert that "ZFS is extremely simple to use."

[cue the car analogy]

That's like claiming you have to open the hood on a car and tinker with the engine depending on what kind of road you're driving on, and then asserting that the car is "extremely easy to use." If you mess with your engine on a regular basis, it might seem that way -- but if you're a normal user, it's an unspeakable pain in the ass.

Re:Larry effect again?

[rsmith-mac]

Larry Ellison, the Oracle CEO. Oracle just recently purchased Sun (makers of ZFS), so the OP is postulating whether Apple pulling ZFS is a product of Cisco not working on/opening up ZFS to Apple like Sun did.

I see no problem with that

[erroneus]

If something isn't "good enough" to make a solid product, then don't include it. This is how Vista got whittled down the way it was. The list of features that were pulled is longer than those remaining by my estimation.

Re:One less "feature"

[LingNoi]

Unfortunately it's incompatable with the linux kernel license (not to claim which is more free, or who's to blame). There's a fuse project, but fuse projects don't tend to go in places where zfs does.

Given how 90% of the comments on here are about how many problems there are with ZFS I don't think "unfortunately" is the correct word.

BTRFS vs. ZFS

[DrYak]

So, no, Larry's company becoming ZFS owner ain't the reason Steve's company would drop it.

Unless you keep in mind that Larry's got his own filesystem under the hood : BTRFS was Oracle's GPLed answer to Sun's ZFS (BSD licensed).

So perhaps Oracle is thinking that developing 2 competing filesystems with the same feature-set is maybe too much ? (Specially since Oracle tends to target slightly more often Linux than BSD - and thus could make sens to put more resource into a file system with a GPL-friendly license rather than a files system whose license makes it incompatible with Linux)
And perhaps they would like to drop ZFS in the long term ? So suddenly there's less incentive for Apple to support ZFS.

Or maybe, indeed, it has nothing to do with Oracle acquiring Sun and perhaps ZFS is to Apple what WinFS is to Microsoft: an eternal "Sorry, we didn't have time to implement it in the current version of the OS, but we promise we will put it into the next iteration. Trust us, that time it'll be for REAL !" vaporware.

Who's Larry?

[fm6]

If you mean, has Oracle management quietly told Sun to back off the ZFS evangalism, I kind of doubt it. It's hard to see why they would even care, at least not enough to risk getting caught doing something that could have nasty consequences — Oracle's acquisition of Sun still hasn't had federal approval, and illegally interfering with Sun's management would be just the thing to get it turned down.

The whole ZFS-on-MacOS thing is part of Sun's broader efforts to fight the marginalization of its technologies by open-sourcing them and then evangelizing everybody in sight to adopt them. This has happened not just with ZFS, but also with Solaris, the Sun implementation of Java, and even the Sparc CPU.

One aspect of this effort has been to push OpenSolaris and ZFS at desktop users. Pushing Apple to fully support ZFS (right now, they only provide a read-only driver) is part of this, as is a big push to get CS students and other hackers to download and use OpenSolaris on their personal PCs.

There's a certain amount of wishful thinking here. Solaris and ZFS do have very real and important technical advantages over their alternatives. But for a desktop user these advantages are pretty minimal. And to get them, you have to pay a big price in learning to use more complex tools and in not being able to participate in in bigger user communities.

Apple's response to Sun's ZFS evangelism was initial polite interest, but little positive effort over the long term. Not at all surprising: what use is ZFS to the typical Mac user? If servers were a bigger part of Apple's business it might be different.