Slashdot Mirror
Sun CEO Says ZFS Will Be 'the File System' for OSX
Fjan11 writes "Sun's Jonathan Schwartz has announced that Apple will be making ZFS 'the file system' in Mac OS 10.5 Leopard. It's possible that Leopard's Time Machine feature will require ZFS to run, because ZFS has back-up and snapshots build right in to the filesystem as well as a host of other features. 'Rumors of Apple's interest in ZFS began in April 2006, when an OpenSolaris mailing list revealed that Apple had contacted Sun regarding porting ZFS to OS 10. The file system later began making appearances in Leopard builds. ZFS has a long list of improvements over Apple's current file system, Journaled HFS+.'"
But they killed that project.
Mmmm... Boiled Oceans!
More
Well, not in THIS forum. But elsewhere.
/ 06/sun-ceo-jonathan-schwartz-zfs-to-be-the-file-sy stem-in-leopard
5:1 that it's not the default root file system in Leopard.
The first bootable release of ZFS (not "BUILD," but "RELEASE") isn't even due until the Fall.
I'm not alone in this skepticism. See this Ars story, for example.
http://arstechnica.com/journals/apple.ars/2007/06
When ZFS was first mentioned in the same breath as OS X it was pointed out that at the time you couldn't boot off ZFS file systems, so people were thinking it would power external (or secondary) timemachine devices. If it's replacing everything, I'm assuming you can now boot from a ZFS drive? When was this functionality added?
He's already taken it back, more or less:
"I don't know Apple's product plans for Leopard so it certainly wouldn't be appropriate for me to confirm anything. [...] There certainly have been plenty of published reports from various sources that ZFS is in Leopard, I guess we will all have to wait until it is released to see if ZFS made it as the default, or if they simply announce that it will become the default in a future release."
"There is no time, sir, at which ties do not matter," Jeeves, (Jeeves and the Impending Doom)
Not anymore, it ain't... Now, Apple will go with NTFS just to spite them...
"I like systems, their application excepted", George Sand (French)
Jobs is probably not happy about his thunder being stolen right before for the June 11th keynote
I strongly doubt he didn't know about it. This is Jonathan Schwartz, not a OS X rumors blogger. At any rate, ZFS in OS X is Sun's thunder; Time Machine is Apple's thunder, and that's already announced. How many OS X users (other than slashdot readers) will care in the slightest about the underlying filesystem? What they care about are the features, like Time Machine, that it enables.
Yes I'm sure it will be worth it in the long run but I'm not looking forward to yet another hiatus in which: no industrial-strength disk-recovery tools are available, in which accidentally running the wrong disk-repair tool on the wrong partition hoses it instead of fixing it, and in which yet more legacy software suffers breakage due to subtle incompatibilities in implementation.
(Yesyesyes, I know, ZFS is reliable that disk-recovery tools are not needed. And if you believe that, then you probably believed Microsoft when they said NTFS volumes never needed defragmentation).
Dear Apple:
Please let HFS+ still be an option.
Please let Classic still run on Power Mac processors.
Please let reasonably well-behaved software that uses resource forks still work.
Please let it be case-insensitive and case-preserving.
"How to Do Nothing," kids activities, back in print!
Maybe some in the know (not me) could fill us (people like me) in... Are there other benefits that will come from moving to ZFS? I'd guess that for the average consumer any performance gain, or loss, won't really make a difference, but what about those running servers or doing heavy video/audio work? Or are there other aspects of this filesystem that will make it that much better than HFS+?
Once we're sure it's stable, because it looks like a massive improvement over the 1970s-style file systems we're using now. ZFS is now part of FreeBSD, Solaris will have ZFS "soon" and many Linux distros are also considering it. Good. Let's get to a common standard that's excellent and forget the tedium of these past, less effective file systems.
technical writing / development
If ZFS is the default file system, it will mean that Time Machine (i.e. the snapshot feature) of 10.5 will be able to take snapshots without requiring a secondary file system to keep the copied (recoverable) blocks, as it does now with HFS+. To me, the secondary filesystem requirement makes Time Machine essentially useless on a laptop.
We had a guy come in a few months ago to give a class on upgrading to Solaris 10, highlighting the differences between Solaris 9 and 10. When he got to the ZFS portion, he really did talk about it like that. He basically described ZFS as the filesystem to end all filesystems, the killer app that would revolutionize computing, end file corruption, and bring about world peace.
I'm not sure if that's the way they talk about it internally at Sun, but that's how their instructors portray it out in the field.
I think Slashdot would benefit from adopting some of K5's approach to story submissions. The Firehose is a great start, but instead of simply saying yes or no, users should be able to give feedback to the submitter. The summary for this article is a great example. The submitter typed "build" instead of "built," resulting in an annoying distraction in an otherwise concise description of the story.
Newspapers have Copy Editors (at least they used to; most seem not-too-bothered by spelling these days). It would be nice if interested Firehose users were given the opportunity to help make sure the summary was fit for publication before it hits the front page.
I guess this should have been a journal entry, but it seemed like an opportune time to bring this up.
I don't care why you're posting AC
I know its not on Mac but this shows how easy and powerful ZFS is. I have heard directly from Sun that by Solaris 10 will soon have bootable ZFS either in update 4 or update 5. Remember that the big problem with Sun hardware is that they need firmware support for bootability and that it may be much easier on OS X to make ZFS bootable. http://video.google.com/videoplay?docid=8100808442 979626078
This will make going from earlier versions of OSX to the new one more of a pain because the whole disk will have to be reformatted.
Its worth noting that most Sun instructors do not work for Sun. As someone who has implemented and is using ZFS, it really is as good as they say. I use it at home for storing video files and have not suffered any data loss.
There is no need to 'stick the knife' in! ;-)
That's quite a change from about a year ago, when I took the "new features in Solaris10" class; at that time the instructor I had was in no uncertain terms saying it's "not ready for production, wait until later". Apparently we have reached "later"? Or it could be that people have opinions and express them, and aren't all speaking for Sun; I suppose that's possible...
ZFS came out in Solaris 10 Update 2. (Sun is days away from releasing update 4.) It is currently bootable in OpenSolaris.
It still has memory hogging issues as well as performance issues in certain areas. More kernel tuning will be needed to tame the beast that is ZFS. It is good for many things but it does not replace EVERYTHING just yet.
as a worker at sun and having used ZFS and playing with it constantly , it is a good File system , I appreciate the little things it has and it has brought data stability to a whole new level. I think personally that this will be a defining moment for ZFS , it will be linux ready soon ( at the same level of stability that the mac will enjoy ) and it will take off and become more of a standard for unix and linux boxes.
To bad no windows port is available. It would be nice to see my unix drives from windows.
This package Does Not Contain a Winner
Some drink at the fountain of knowledge. Others just gargle.
Apparently you've never used ReiserFS.
You misunderstood him. He said that it would bring about "War and Peace", not world peace.
Every ZFS volume has a copy of the Tolstoy classic embedded for internal benchmark purposes.
He wrote 'data loss', not 'wife loss'.
Sorry, but your description of the lookup process isn't right. First: lookup depends on the way directory entries are store. On UFS, it's a non-sorted array; in order to do a lookup, you need to (worst case) scan the entire directory. On VxFS, they use a hash, so first you hash the input, and then do run through the entries that have a matching hash. On HFS+, the catalog is stored as a B-tree, so you do compares to get to the right node, and then look through the node until you either find it or reach the end of the node. Second: None of those is affected by case-insensitivity. You simply do a case-insensitive compare each time. This is the difference between HFS+ and HFSX on Mac OS X: in the former, the key-compare function is a Unicode case-insensitive comparator; on the latter, it's just a memcmp. Third: your comment about "i" is a glyphing issue, not a character issue. Apple has a pretty good technote up on their HFS+ impelementation, and it describes the way the case insensitivity works. I recommend reading it.
Aren't there still licensing issues to iron out?
Ext2/3 and ReiserFS have all been ported to Windows, so I don't see there would be any problem porting the Linux ZFS implementation as an IFS driver for Windows Vista/XP/2000
My blog
I must have some nonstandard use case, because I've lost data with both ReiserFS (multiple times, as I optimistically tried newer versions) and XFS.
Ext3, on the other hand, has been rock solid for me.
it will be linux ready soon ( at the same level of stability that the mac will enjoy ) and it will take off and become more of a standard for unix and linux boxes.
Depends - whatcha building? An app server, a web server, a database server, or a file server? Different strokes for different folks, and I'm not clear yet if I'd like the overhead of ZFS on a database server. The jury's still out on ZFS+Oracle...
I'm not 100% on which file system I'd like. Certainly the integrity of ZFS is quite pleasing for a DBA, questions is if the overhead is worth it...
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
I married Brent Spiner, you insensitive clod.
Seems like Apple take some of the best ideas from the Unix world. Really shows the potential of Unix systems if the people who wrote them thought a little more about usability.
On linux FUSE is all we can use because of the license restrictions. I use it on a Solaris box with a vmed red hat install it's not quite ready for prime time. And the licensing sucks because it won't get put into the kernel.
;) yet and I don't see any intent internally to even attempt it. as it stand I think I am maybe 1 of 8 people using ZFS on linux here. And that is only in the VM.
Which I find really deplurable , I would love to have it in the kernel.
And yeah id like an ifs driver but none exist
This package Does Not Contain a Winner
ZFS does indeed look like the greatest thing since sliced bread, but if all I cared about `wasn't losing data', then I'd just stick with ext2 or fat. (But I care about more than that, which is why I use xfs, ext3, hfs+ and ntfs for various things.)
I am using ZFS-FUSE right now. On my gentoo system, many partitions are zfs, including /home, /var/tmp, /usr/share, /usr/portage, and /opt
Because I have suffered some random corruptions in the past, even with ext3 ("This mp3 didn't used to have a skip there!"), I wanted the checksumming so that I can tell when I need to restore something from a backup.
As a filesystem, it works completely, including creation of new filesystems, compression, checksums, etc. However, I've noticed a decrease in my system's general performance since installing zfs (probably due to it holding my home directory). Memory usage and mysterious CPU usage (I don't think it's checksumming) are the current disadvantages, but the author says it's still completely unoptimized.
Should you try zfs-fuse? Definitely. But right now don't expect a performance gain.
A cat can't teach a dog to bark.
From what I understand, the API that Windows filesystem drivers interface with is an undocumented nightmare that's entirely different (but not necessarily 'worse') from the way the rest of the world does it.
So porting a filesystem as complicated as ZFS could take some time.
-- If you try to fail and succeed, which have you done? - Uli's moose
Apple is "well known" for massive backwards compatibility updates... except they aren't... They always handle transitions over a couple of versions, intelligently bringing people along. They swapped processor architectures twice and each time brought people along with emulators, in the Intel case it wasn't faster than the fasted G5 machines, but those of us upgrading 3+ year old machines (Powerbook G4 1Ghz -> Macbook Pro in my case) found our PPC apps running faster and Intel code flying.
We all expected the Intel migration to happen with 10.5, they shocked us when they did it off the 10.4 base.
While they did abandon Mac OS to move to OS X, they provided a migration strategy (Carbon) and a compatibility layer (Classic). Classic support shipped with 10.0/10.1, 10.2, and was supported in 10.3 if you already had it, as well as 10.4 I think, but they kept classic for around 5 years, which gave everyone time to migrate to Carbon. Its unfortunate that there is no long-term Classic via Rosetta just from a classic application point of view, but they didn't leave anyone in the lurch.
I expect 10.5 to introduce this OS, which will be useful for new installs, or for external drive arrays, especially for the Video market, but I wouldn't expect it to be the default. OS X has supported a Unix filed system, but defaulted to HFS+, because HFS+ was compatible with Mac OS, so you could dual-boot OS 9 and OS X for a good 2 years on new hardware to maintain compatibility. If they hadn't done that, they would have lost the Pro-Audio and Pro-Video markets that took a few years to get native OS X applications.
Getting it in the wild and for professionals would help that market, while not breaking ANYONE's compatibility. Sometime in 10.5's lifetime they may ship new computers with it, or they may wait for 10.6 in two years. But giving everyone two years is plenty of time to get utilities and applications compatible with the new file system.
The flashy consumer features are touted for the OS, but the underlying architecture has always followed a 2-cycle release. If you've used OS X Server for 10.2/10.3/10.4, you'd notice that they introduced stuff in one version with limited exposed functionality (with the rest via the Unix layer), enhanced the functionality in the next rev, and polished thereafter.
The Apple Mail Server -> Cyrus migration was someone poorly handled, but mostly because AMS was garbage. But the 10.4 Mail tools are night and day beyond the 10.3 ones.
They are actually far more careful than people give them credit for.
The different is, they don't keep backward compatibility as a long-term goal, they do a two-stage migration, giving people 2-4 years to transition.
Discuss.
Website Hosting
Shh, that is an inside secret. They were originally going to use The Brothers Karamazov, but the developers couldn't agree on a pronunciation.
"To Do Is To Be" - Socrates, "To Be Is To Do" - Sartre, "Do Be Do Be Do" - Sinatra
Windows filesystem kernel API (it's called IFS - Installable File Systems) is fairly well documented, and you can get free GPL2 headers for it (http://www.acc.umu.se/~bosse/ntifs.h) or buy IFS kit from Microsoft for about $109 (http://www.microsoft.com/whdc/DevTools/IFSKit/def ault.mspx). Unfortunately, IFS is a very complex API and there's only ONE good book about it.
:)
You definitely can port FS to Windows using only documented API, but it's a long and tedious process. I'm currently porting FUSE to Windows, so I know it
For something that's only a year or so old (production wise), I don't trust it worth shit.
/dev/null > /path/to/file', then, once you have some blocks free, rm works (so does unlink).
We run Netbackup Enterprise on Solaris 10 - during our last round of upgrades we installed ZFS on our disk staging storage units. It replaced VxFS. The way disk staging storage units (DSSUs) work in Netbackup, the disk is always near 100% full form a unix perspective. Basically, any time more disk is needed, the oldest image that has been copied to tape is expired from disk, thus freeing up more room. However, ZFS's most prominent bug from our perspective is that during periods of high activity, if all blocks become allocated, it becomes impossible to unlink(2) a file. This causes the application to no longer be able to make space for new backup images.
Going down the shell, try to rm a file and it comes back: rm failed, disk is full.
Well, if the disk is full, and you can't rm because the disk is full, how do you free up space?
Sun's response, truncate an unnecessary file using 'cat
Ok - so how do you tell a compiled application to truncate an unnecessary file before unlinking it? You can't! How can you determine what an unnecessary file is? If you delete the image before expiring it from the catalog you get errors when you try to expire, so you end up with catalog corruption.
All in all, this is a problem that should never have been introduced, let alone still exist after months of sending trace outputs and reproducing it in multiple environments. ZFS isn't ready for the real world.
ZFS is not ready for prime time - at least not on Solaris.
I setup ZFS on some SAN storage in a new system. The internal boot disks were mirrored UFS. When one of the HBAs fried, the SAN storage disappeared - and the system panic'd.
Every reboot thereafter stopped in a panic. The ZFS subsystem panic'd the system at every boot when it couldn't find all its volumes. After calling Sun support, I found out that they need to do a massive code redesign to catch that issue, and it wouldn't be out for at least 6 months.
I'm sure ZFS will be great - once they clean up these type of showstopper bugs.
Jeremy Baumgartner
there are no licensing issues with linux/zfs. The "issue" is that Alan Cox and Linus Torvaldes have a hot iron up their asses about compartmentalizing everything. ZFS is vertical -- it's in the fs layer, the driver layer, and the vfs layer.
You can't restore the original case of a string afterwards, but you can always make it lowercase. This is called "case folding." You can fold two strings to a lowercase form, and then compare them for equality or whatnot. Works with Unicode, too. Then there is the issue of internationalization. For example, consider "I" and "i". Some places have an uppercase with the dot, and other places have a lowercase without the dot. The rules for uppercasing and lowercasing differ from what most people are used to. Oh crap! This issue doesn't exist on a case-sensitive filesystem. While folding Unicode chars is frequently presented as an unsolvable problem ("what do you do with the letter with the squiggly thing above it? Or converting that German capital 'B' thing to two lowercase 's' chars? There are MILLIONS OF THESE!")
Here's the whole list of characters that need to "folded" to a lowercase form, accounting for instances where it will cause the string to grow (like that German 'B' thing):
http://www.unicode.org/Public/3.2-Update/CaseFold
(And you can hash those chars too, so folding a string doesn't involve hundreds of conditionals.)
If you don't care about Unicode, case folding an English ASCII char is 2 lines of C code, and a few more if you want extended ASCII.
Once you have a filename, you can store it in the filesystem as the specifically-entered characters, so you don't lose the original casing, but also store with it a hash of the case-folded version. Now whenever you need to look up a specific filename, you case-fold it, hash that folded string, and look it up that way against the hash you previous calculated when creating the file. Now it's as fast as the case-sensitive filesystem, minus the overhead of folding a small string. Because of the way directory listings are done (read then look up stats) you can generally square the above numbers. Ouch. The way directory listings are done doesn't change...readdir() is the same in all cases, and your lookup is still a hash. If you had to scan, the first run is slow anyhow due to disk bandwidth and seek speeds, but then a modern OS can cache the inodes to speed this up for the next run. App needs to make a file. App sees that file does not seem to exist. App writes file. Complex international case rules mean that no, the file DOES exist, and it gets clobbered. I would think that stat(filename) would not report the file doesn't exist if open() would then clobber it, at least not for case-sensitivity issues.
If your app decides about a file's existence by using readdir() until it finds it, and doesn't properly case-fold, and didn't call open() with O_EXCL, then not only did you go the long way about it, you got what you deserved for clobbering the file.
Actually, if you don't just open(O_CREAT | O_EXCL) to check for existence and create if missing in one step, then you'll have an atomicity problem anyhow. Use the services the OS provides, they are there for a reason.
--ryan.
Don't say, "don't quote me," because if no one quotes you, you probably haven't said a thing worth saying.
Nope. ZFS WAS going to be the file system until Sun's CEO leaked it before Jobs could announce it. No one knows what it will actually be NOW. Jobs would rather use FAT32 than let someone leak info about one of his announcements.
Time Machine is already fully functional (apart from a few gui glitches) in the current leopard developer builds, but ZFS isn't even available in Disk Utility (yet?). This doesn't mean ZFS won't be added at the last minute, but it certainly isn't required for Time Machine.
they shocked us when they did it off the 10.4 base.
It did not shock those of us who know that NEXTSTEP was transparently portable to at least four architectures.
you had me at #!
That was user error.
If you mod this up, your slashdot background will turn into a beautiful sunset!
Yes there is. ZFS is licensed under CDDL, which is not GPL2 compatible. Linus has so far refused to move to GPL3 when it comes out, so before that happens, there is a licensing issue.
Michel
Fedora Project Contribut
No, no and no.