Slashdot Mirror
ZFS Shows Up in New Leopard Build
Udo Schmitz writes "As a follow-up to rumours from May this year, World of Apple has a screenshot showing Sun's Zettabyte File System in "the most recent Build of Mac OS X 10.5 Leopard". Though I still wonder: If it is not meant to replace HFS+, could there be any other reasons to support ZFS?"
How will I store my 32 exabyte "movie" files?
I hope they make it a default.
Isn't the term 'Zettabyte File System' actually inaccurate now? I thought they dropped that and ZFS now only remains as a pseudo initialism
I never get used to these constant resurrections
Now that Vista is finalized, expect Apple to show more and more of the 'secret' features of leopard!
--jeffk++
ipv6 is my vpn
oh yes, for servers! server gurus may prefer it.
Because if Apple showed them before, there was a risk that Microsoft tried to announce them as future features in their soon-to-be-released perfect Windows Vista ?
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
The letter "Z" is pronounced "Zed" in Canada, UK, and other funny British Colonies, former funny British Colonies, and Colony-wannabies
The letter "Z" is properly pronounced "Zee" in the USA and Iraq (after 2003)
Stop feeding the trolls.
Not enough good acronyms with Z in them. This adds an aire of leetness to the whole thing.
I'm a fiscal conservative, it's a pity we don't have a political party anymore
"It's too bad that stupidity isn't painful." - Anton LaVey
Here's their reason: They are just showy!
When my Karma level reaches 0 I feel in piece with the Universe
Stop trolling the feeders.
"Though I still wonder: If it is not meant to replace HFS+, could there be any other reasons to support ZFS?"
Duh... It's called compatibility.
I will be soon converting my Linux server to Solaris just for ZFS. Although ZFS may not terribly useful on a normal desktop, on a server, it's very powerful.... The idea of parity data actually being used actively to ensure data isn't corrupted is brilliant imho. So is the idea of on-the-fly recovery (I remember a video of some guy writing 30 megs of junk to a partition using dd, ZFS detecting it, and repairing it). *ends rant since all this can be read up about online*
In undeveloped countries, the consumer controls the market. In capitalist America, the market controls you.
...the words "Time Machine" are jumping up and down in front of my face trying to attract my attention. I can't think why that might be.
It's not you: I'm just this horrifically socially awkward with everybody.
"Though I still wonder: If it is not meant to replace HFS+, could there be any other reasons to support ZFS?""
Geek cred.
Never ask for directions from a two-headed tourist! -Big Bird
A clicky to the Wiki article on ZFS.
Though I still wonder: If it is not meant to replace HFS+, could there be any other reasons to support ZFS?"
Apple Developer Connection released this VFS plug-in to support MFS (the original Macintosh File System). Does this mean Apple is going to replace HFS+? No. It means Apple is happy to employ its programmers working on making its operating system more useful for more people.
- They had implemented everything I thought they should, and
- That only accounted for about 40% of the features of ZFS.
Calling it the last word in filesystems might be hyperbole, but I expect ZFS to last a good 10-20 years, which is quite respectable for a filesystem, and I wouldn't be surprised if it lasted longer. Is it a replacement for HFS+? Not yet.HFS+ is a very nice filesystem for single user systems with a single disk. It implements journalling, has reasonable performance, and has good metadata support. For the average users at the moment, the only real advantage of ZFS would be snapshots, and these are not too difficult to implement for other filesystems.
ZFS, however, is much better when you have multiple physical disks. At the moment, only the top-end Macs have more than one disk. This is likely to change in two ways:
- Cheap flash,
- Network storage
For a home user, ZFS could handle backups trivially by plugging in a large flash drive and adding it to the pool. I suspect this will be one mechanism Time Machine will use. Due to the way ZFS works, you can just mirror a part of the directory tree (e.g.ZFS is not needed as a replacement for HFS+ in 2007, but it probably will be in 2008-9. ZFS is a 128-bit filesystem, which means it is designed to last for a long time. We will probably never need a 128-bit filesystem (unless we actually want to build hard drives the size of planets with single-atom sectors), but we will need a 65-bit filesystem once we get to around 10 Exabytes. This won't happen with single drives for a while, but it will with RAID arrays.
I am TheRaven on Soylent News
I thought, "That'd be powerful"...
I'm such a geek.
Cake or Death? Cake Please!
The tech behind ZFS at least sounds very impressive, but I have to wonder how useful it is for workstation drives.
I've never found plain-Jane posix permissions to be all that useful on anything other than the most basic of server environments.
HFS has going for it all the fun stuff we've come to love apple for, such as transparent file customization like icons, labels, meta data, and whatnot through resource forks. I assume that these can be made to work with ZFS by making hidden files.
What I'd really like to see is both that kind of functionality along with NTFS's really excellent ACL permission system implemented. ACL permissions are a godsend for people responsible for running a file store that's used by humans as opposed to automated processes. NTFS also has a great deal of capacity for meta-data, although not to the same level as HFS.
NTFS is one of the few worthwhile things that's ever come out of Redmond. I wish more people would spend a bit learning from it without throwing it away simply because it's MS bloat.
The next Slashdot story will be ready soon, but subscribers can beat the rush and slashdot the links early!
So it's about the snapshot ability of ZFS, and that's exactly what will be needed for Time Machine.
In this forum they duiscuss build 9A326 and ZFS. Some posters mention, that you can choose ZFS but it doesn't work (yet) and/or that you can't install OS X on it.
And then I notice that the official name of ZFS is ZFS these days: "The name originally stood for "Zettabyte File System", but is now a pseudo-initialism." Someone should tell Apple.
P.S.: What about my rewritten in cocoay goodness Finder? Pretty please?
With ZFS we might be able to get some very powerful backup features into OSX. Most binaries files don't change most of their content, ZFS makes it possible to due meaningful differential backups on large binary files. So for example 200 versions of a word doc with sounds and pictures that got revised over 6 months get stored in maybe 3x the space of the last revision. Emails with the same attachments get stored in just a few k rather than taking a meg each.... If Apple has this all working together by 10.5 then TimeMachine will work far far better then people currently expect it to. A 50g drive will be backing up a terabyte of worth of files.
The answer is that it probably is meant to replace HFS+, but since ZFS is not bootable yet (including for Solaris 10) Apple can take the time to introduce ZFS, build tools for easier management, and let people get familiar with the FS before they have to drop HFS+.
HFS' lifetime has already stretched far beyond what it should have, it's time for Apple to think of its next generation FS, and ZFS is an extremely promising FS with heaps of amazing features Apple has already started to integrate into its UIs with Leopard (Time Machine + ZFS Snapshots anyone?).
ZFS also shows strong promises as both a home and a server FS.
"The way we can tell it's C# instead of Haskell is because it's nine lines instead of two." -- wadler
... because they want an extra edge in the server market? ... because they wanted to grab some free headlines?
... because an engineer was bored?
And last but not least:
On a side note, I've read a lot about how ZFS is supposed to be reliable and flexible as hell. Nowhere, however, have I found information about read/write performance. I guess it's a great filesystem for huge mission-critical datasets, but for anybody else? I have my doubts.
Or can somebody enlighten me?
Misleading titles? Inflammatory blurbs? Keep in mind that Slashdot is a tabloid.
I really don't see much use for this in workstations, but for us having a real-life need for this, I'd like to know how it compares in performance to Linux LVM, Veritas File System, AIX LVM with JF2, and NTFS Dynamic Volumes.
I'm actually very curious why Apple does not support ext2/3 "out of the box"? Currently you can "try" to use ext2fs, but that only works from time to time. I would never trust it with real data that cannot be corrupted.
What is the rational behind not supporting ext2/3 natively in OSX? No demand? It would make our lives a lot easier...
ZFS is an impressive filesystem, and Apple have every reason to be interested in it over and above, say, ext3. Of course, Apple aren't the only ones interested in ZFS - I have seen a great deal of excitement in the Linux community, too. I am pretty confident that a Linux driver for ZFS will emerge, and in the long run, I wouldn't be at all surprised if it ended up being a very common filesystem on Linux systems. For anyone wanting (read-only, ATM) access to ZFS partitions on Linux today, there is a ZFS FUSE driver available.
No, they just wanted something that could implement "time machine", their backup-done-right proposal.
I expect Solaris to be switchd to GPL some time in the GPLv3 era, at which point there won't be a problem porting ZFS to Linux. Not that it was technically difficult to port it to Apple/BSD (;-))
--dave
davecb@spamcop.net
A RAID array won't survive someone dding 40 megs or so to a partition, and reiserfs can hardly be called reliable, seeing as sometimes it just decides to hose the entire partition for no apparent reason. But glad to see you're using them for that, one more retarded uninformed clueless ignorant asshat to deal with.
Going off on a tangent, it's a little funny that everyone is always going on about how "free" the GPL is, and yet ZFS is open-source, but not useable by Linux because of the GPL. I'm liking the BSD license more every day.
While ZFS might be useful for supporting Time Machine, it's not necessary and I would think that if Apple were including it by default on all Leopard distributions (as they'd have to if it were the underlying technology for Time Machine), they'd probably be talking it up a little bit more right now.
ZFS is still overkill for most home computing needs, so I'm not sold on it being the default filesystem for OS X installs. My first guess is that it's going to be an option for network-acceccible storage. Something like this would be really killer on an Xsan volume.
From the article:
"Finally the most important feature of data integrity os check summing and everything on ZFS is checksummed meaning zero data corruption."
Umm , last time I looked checksumming merely told you if there WAS data corruption , it doesn't prevent it. Does ZFS has some other method to prevent data corruption (eg for a gone bad sector etc) or does it just flag a file as corrupted and leave it up to you what to do next?
They already have UFS and don't make it really usable, even after making a big deal about it being updated to the latest version from FreeBSD in Panther. It's a shame, too, because while HFS+ has a lot of nifty features all of them could be emulated over UFS or ZFS or any other file system (by putting the hooks for applications like Spotlight in the vnode layer rather than the file system - the vnode layer already has most of the hooks Spotlight needs), it falls far behind UFS in terms of reliability.
In fact HFS+ is *so* bad that if it wasn't for a couple of apps that absolutely freak out if they don't have their pet un-emulated feature I would have gone to UFS long since... even if I lost Spotlight completely. Until my Mac I had never run into a file system that wasn't so badly damaged as to be unbootable that coudn't be repaired by fsck... but apparently with HFS+ just running it "too full" can trash it, and I lost my system disk on my old G4 three months running because of that!
So I wouldn't hold out any expectations of ZFS being implemented in any useful way. They already have better file systems than HFS+ and they're not using them.
Not really.
http://zfs-on-fuse.blogspot.com/
No, they just wanted something that could implement "time machine", their backup-done-right proposal.
There are several open source file systems that support time-machine like functionality. Nor is there anything particularly new about "time machine"--it's a well-known approach that works fairly well in some circumstances and not at all in others.
I expect Solaris to be switchd to GPL some time in the GPLv3 era, at which point there won't be a problem porting ZFS to Linux.
Perhaps, or perhaps not. Either way, it won't make a big difference to Linux. Ext3 is actually a good design; its deliberately limited feature set is what makes it good.
could there be any other reasons to support ZFS
Could it be that the guy was raised on Windows ? Or what ? I mean, in the real world, there are more file systems that the one which your OS uses by default. And adding native support for them is not a questionable move, it's an applaudible move.
Hell, I remember what a happy day it was when I found crossmeta's xfs reading tools for Windows. Things like this shouldn't be a thing to raise an eyebrow for. What you all should raise your eyebrows for is why some OSes repetitively do not want to add native support for widespread file systems.
I am putting myself to the fullest possible use, which is all I can think that any conscious entity can ever hope to do.
For anyone wanting (read-only, ATM) access to ZFS partitions on Linux today, there is a ZFS FUSE driver available.
Yes, that's the typical Apple solution: you can sort of use it, but if you really want to use it, you have to commit to using OS X. It's not a good proposition.
I am pretty confident that a Linux driver for ZFS will emerge, and in the long run, I wouldn't be at all surprised if it ended up being a very common filesystem on Linux systems.
I seriously doubt there will be an independent implementation of ZFS; that work would probably go into ext5. Even if there were (or if ZFS becomes GPL compatible), I doubt it will get much traction: Linux has had more powerful file systems than ext3 for many years, and people choose not to use them. Impressive feature lists don't make a better file system.
...but if I were on a closed alpha or beta program, I'd be real cagey about posting screen shots. I think it wouldn't be that hard to embed a steganographic code on the screen so the company doing the beta would know who leaked. Is there some sort of random noise filter that would remove anything like that?
The next Cmdr Taco duplicate will be ready soon, but subscribers can beat the rush and see it early!
I, for one, would really like to see the Apple guys create an enterprise-class server. The XServe gets close, but fails as it is missing a OOB management processor (see HP's ILO for an example of how to do it right) and requires funky button pushing operations to do some things. No, ILO is not the same as an IP-KVM or serial console, which only handle 50% of all the functionality needed.
We might be seeing the first stages of a Sun/Apple alliance. I don't suspect we'll see a merger anytime soon, if ever, but increasing cooperating between the two would be a sensible thing to do. It's the only way they can rival Microsoft.
Sun offers superior server performance, while Apple of course handles desktops and workstations with ease. With increased interoperability between the two, corporate customers can take advantage of high-performance Sun servers running Solaris, connected to by usable and reliable Mac OS X desktops from Apple. In short, they get everything they get from a Windows network, but with better performance and security.
the X11 stuff is BS. what is apple's X server missing? Hardware acceleration, aqua integration, exporting? It works a heck of a lot better than Xming or other X-server on desktop servers. In fact, there was just an update about a month ago to improve stereoscopic applications.
Apple needs X11 to get people building scientific apps on Linux and Solaris. Its actually one of the best X implementations I've used (X.org, XFree86, Irix X Server)
Mac OS X is really lacking a modern volume manager. ZFS adds this plus a whole lot of other data integrity/portability features. And it's open source. I hope this isn't just a rumor.
I wonder if this was one of the "new APIs" they were talking about at the Time Machine session I attended...
There exists no way of exchanging information without making judgments. --Bene Gesserit Axiom
I've been wandering about this and am insanely curious: if ZFS really does intelligently copy on write how far can it take it?
e tc.) then this could probably happen quite easily.
for starters, does the FS "know" that i've just clicked "Save As" in my word processor? what about copy and pasting a file back into the same directory to make a local copy? Also? is it just within variations on the same file? if i have a particular setup exe on my system but forget, and download it again to the desktop surely the FS has no initial way of knowing that they are one and the same, does some funky heuristic happen?
basically: does the OS's read/write/copy/delete functionality have to invoke copy-on-write via a FS API or is it built in for every single sector-sized chunk that gets stuffed into the FS?
the next question is the one in my subject: how therefore do you define "capacity"? if i've got a bunch of files that take up 700mb on a ZFS device and try to back up to a (Joliet) CD will i get a message telling me that the CD doesnt have room? i can imagine this scenario being unlikely with optimised binary data (jpegs and mpegs) but if i'm backing up a dev environment with autobackups (main.c,main.c.bak.001,main.c.bak.002,etc.) and manually created and dated directory tree "snapshots" (dev,dev_backup_2006-12-18,dev_backup_2006-12-01,
If you don't risk failure you don't risk success.
ZFS has a data storage capacity of 256 quadrillion zettabytes
"256 quadrillion zettabytes ought to be enough for anyone..."
I can imagine that using zfs send/receive to export/import pools would be an extremely efficient/safe method of replicating data. Perhaps some sort of ".mac mirror" could work. This would make Time Machine exceptionally useful, and I'd definitely commit extra $ for .mac services (if reasonable) for this.
There exists no way of exchanging information without making judgments. --Bene Gesserit Axiom
I now realize what a "karma whore" is.
It sounds good, but I think migrating for it just a tad extreme given that it will be implemented for Linux pretty quickly. We're talking about neat new features here, it'll re-enforce or make easier backups and redundancy, but it's not a to-die-for solution that will solve all your problems. There's no way I'd drop a fully configured server installation which does what I need for a new filesystem.
By the way it's nice to see dtrace, open source Java, and now ZFS coming out of Sun recently. I almost feel sorry for how little they get out of a lot of their innovations, they remind me of Bell Labs just before they died.
// MD_Update(&m,buf,j);
ZFS is a very nice and promising filesystem but it still lacks one very important feature: decent support for offline backups. While it is possible to make an offline backup ('zfs send ...') restoring such a thing is extremely tedious since your only option is to import the entire backup back into your ZFS pool after which you can restore the snapshot (or parts of it).
/export/home slice (where all the userdata resides) this way on my company server!
Not sure about you but I really wouldn't like to try and restore data from the
what is apple's X server missing?
Desktop integration, pre-installation, automatic on-demand launching, consistent window management, keyboard layout integration, to name just a few. Performance could also be improved further.
Apple needs X11 to get people building scientific apps on Linux and Solaris. Its actually one of the best X implementations I've used (X.org, XFree86, Irix X Server)
Yes, and that's pretty much all they are supporting: scientific apps. If you want to use X11 desktop apps, you're out in the cold because they don't integrate well with the rest of OSX.
ext2/3 is a child's toy filesystem.
why would you dd to a partition? Oh cuz you run as root and are a smacktard losemongerer. I use RAID as an automatic reliability system, and rsync/cds as a backup method. if I did dd for whatever reason over the partition I could just restore from the BACKUP. I don't see how ZFS would remove the need for backups though. So the move away from Linux to whatever else to use ZFS because it's oh so much better doesn't make sense.
As for reiserfs, I've been using whatever comes with the 2.6 kernel for over a year and I have yet to lose a single bit even amongst various physical failures (and power outages). Maybe you're using the experimental branch v4 reiser or just are, well, how do I put this delicately, a retard?
Tom
Yes, that's the typical Apple solution: you can sort of use it, but if you really want to use it, you have to commit to using OS X. It's not a good proposition.
The GP is talking about how to (partially) use a Sun-created filesystem under Linux; why are you bringing Apple into it?
ZFS: because love is never having to say fsck
I agree that ext3 isn't the best thing out there for Linux, and I don't even use it myself. However, I would suggest that the reason for so many people still using ext3 is that most other filesystems aren't better enough to encourage people to move away from ext3. Look at some of the posts under this story - you will find stories of people moving from Linux to Solaris, really just because they want the features of ZFS. There is demand for ZFS in the Linux kernel, and if it becomes a common filesystem on OSX, I predict the demand will only increase. I don't expect ext4 will satisfy this demand, either.
Damnit, replied to the wrong post :-S
Aargh no I didn't! I am very hung-over.
Apple's attitude towards many FOSS standards vacillates between indifference and hostility.
For example, they view X11 as a dead end and want to actively convert people to using Cocoa; hence, Apple's X11 support sucks and they have no interest in making it better.
You mean Quartz, not Cocoa. And Quartz is unarguably far more powerful than X11 as well as faster; I think they're absolutely correct in promoting it over X11. I'm not sure what "sucks" about their X11 support, as it certainly seems to work just fine on my system.
OpenDoc compatibility would be natural for Pages and Keynote, but I suspect apart from being a lot of work, they don't want it: Microsoft Word matters commercially, and they likely view OpenDoc as supporting FOSS competition (since NeoOffice is actually quite sweet on MacOS).
People still use OpenDoc? Wow. I rather thought it was an experiment gone horribly, horribly wrong.
As for ext3, I think there are two reasons. First, using ZFS, Apple has a feature-list advantage over ext3 (never mind that that doesn't translate into any real world advantages).
Are you seriously -- SERIOUSLY -- suggesting that ZFS has no real-world advantages over ext3? That crosses the line from "Linux advocate" into "bat-shit insane".
Second, I think Apple really doesn't want to make it easy for people two switch between OS X and Linux; despite the bluster, they must recognize that Linux is a serious alternative to OS X, in particular given the current and upcoming releases of Gnome and KDE.
Yep. I'm sure Apple is quaking in fear at the threat of Gnome and KDE. You nailed it.
In a nutshell, the company just isn't committed to FOSS, they are just using FOSS when they see a short-term business advantage.
Apple is the only organization that has managed to make Unix completely appealing and usable to average non-technical users. Yes, they chose to use a better graphics library than X11, and a better (also open-source!) filesystem than ext3 -- and good riddance on both counts, I say. Why all the hostility?
ZFS: because love is never having to say fsck
ZFS sounds very interesting, although I don't see a lot that would be useful on my MacBook Pro (a lot more would be useful on my Linux server / MythTV backend).
The main benefit I saw is the 'snapshot' support (copy on write). I used this a long time ago on AFS, and found it very handy.
So, are there any compelling reasons _not_ to use ZFS?
The hell are you smoking? How do they not integrate well? I use GIMP on OS X on a nearly day to day basis...
Wouldn't that pose a problem for mmap?
I think it would pose a problem for secure deletes. Try to obliterate a file by overwriting it with garbage, you end up writing somewhere else instead? Would the next overwrite attempt get the original location or would you have to write enough garbage to cycle over all the free space of the volume? Considering how large these volumes can get, that's a lot of boiled oceans for a multi-pass secure delete.
Oh, say does that Star-Spangled Banner entwine / The myrtle of Venus with Bacchus's vine?
Is this what you want?
4 870
http://www.apple.com/xserve/management.html
Or this
http://docs.info.apple.com/article.html?artnum=30
There are two types of people in the world: Those who crave closure
Yes, they chose to use a better graphics library than X11, and a better (also open-source!) filesystem than ext3 -- and good riddance on both counts, I say. Why all the hostility?
Typical: when someone points out that Apple keeps pursuing proprietary solutions and when someone disagrees with your view that Apple has the best of everything, you call them "hostile". It's you who is hostile, not me.
Desktop integration
Dunno what you mean by this. Perhaps copy/paste ? That works for me just fine. I always use xterm/nedit (it's a hangover from a previous life) and apple-C will do a copy in nedit, apple-V will paste into (whatever).
pre-installation
You get to choose whether to install it when you install the OS. It's a package that 99% of the user-base won't need or want, but it's a choice.
automatic on-demand launching
Oh dear. Open up the System Preferences panel, click on the Accounts icon, click the Login Items tab, add X11 to the list of apps installed on login. Problem solved. Why would they *not* use the standard way to start apps on login ? It's not as though it's the default window server or anything...
consistent window management
Consistent with what ? X11 doesn't have a consistent window management policy (even if you just consider X11 apps, there are different ways of managing windows). Apple's way is simply to make an X11 app respond just like a native window AFAICT. Works just fine for me.
keyboard layout integration
Since I've never had any problems with keyboard layout (I can type Option-3 to get a £ symbol in nedit, for example, even on my US keyboard, which is usually a problem for me in Linux) I don't know what you're talking about...
Performance could also be improved further
When couldn't it ?
If you want to use X11 desktop apps, you're out in the cold because they don't integrate well with the rest of OSX.
As someone who uses X11 apps all day every day, this is bullshit.
Simon.
Physicists get Hadrons!
Stop feeding the troll feeders!
Look at some of the posts under this story - you will find stories of people moving from Linux to Solaris, really just because they want the features of ZFS
There are always some. The question is whether the mainstream needs or wants ZFS features, and I don't see it; that's not where storage is heading.
If I did dd for whatever reason over the partition I could just restore from the BACKUP.
If you were aware that the data was corrupted. Hard drives can silently lose data for a variety of reasons, ZFS will discover that damage whereas Linux filesystems will silently ignore it.
Honestly, read up on ZFS. Your ignorance is palpable.
I wish that was the case. I really don't want to set up my linux root partition on fuse.i d=95110224#Platforms)
"Porting ZFS to Linux is complicated by incompatibilities between CDDL, the license its source is released under, and GPL, the license which governs the Linux kernel. To work around this problem the Google Summer of Code program is sponsoring a port of ZFS to Linux's FUSE system[10] so the filesystem will run in userspace instead. However, running a file system outside the kernel on Linux has significant perfomance impact." (from http://en.wikipedia.org/w/index.php?title=ZFS&old
We will probably never need a 128-bit filesystem (unless we actually want to build hard drives the size of planets with single-atom sectors)
:-)
Spoken like someone who obviously has not done much work with digital video - I could at the very least use a small moon sized drive at the moment.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
The hell are you smoking? How do they not integrate well? I use GIMP on OS X on a nearly day to day basis...
So do I. I needed to install X11 by hand. It got blown away by upgrading the machine. Drag-and-drop and cut-and-paste don't work. Focus doesn't quite work. Window groups/icons don't quite work. And I suspect what does work works because MacGimp folks already put in platform specific stuff.
Let me tell you a cautionary tale. Some six months ago I decided to do exactly what you're suggesting for exactly the same reason. I got OpenSolaris and installed it on a spare drive on my fileserver. It was a pain in the ass figuring out how to administer a Solaris system, which despite all the documentation seemed opaque and baroque, but I persisted. Unfortunately, the damn thing wouldn't configure the network. I couldn't figure it out. I was reduced to tracing through the init scripts line by line figuring out where they failed. It turned out that the file which contained the hostname had a trailing carriage return. I'd pressed "enter" an extra time when I was editing it. As a result the system decided it's hostname was the empty string, which it couldn't find in the hosts database and gave up. At that point I nerfed the partition in a fit of pique and went back to Linux + software RAID + reiserfs. Any software which is that brittle and administrator-hostile honestly does not deserve to live, certainly not on my server, no matter how nice its filesystem.
Hard drives silently losing data is a problem solved by RAID. I really don't see what additional protection this provides against hardware failures over any other filesystem on a RAID disk -- AFAICT it only protects against writes that bypass the file system without bypassing RAID.
There may be other benefits to ZFS, but I don't want to bog down my filesystem with data integrity calculations -- I have dedicated hardware for that.
"Yes, that's the typical Apple solution: you can sort of use it, but if you really want to use it, you have to commit to using OS X. It's not a good proposition."
What does this have to do with what you quoted, or what Apple is doing? Apple's solution is using the ZFS. They don't own ZFS, they didn't create it, SUN did. And sun's solution isn't that if you want to use ZFS you have to use Solaris.
Again, wtf are you talking about?
a man, a plan, a canal, panama
"ZFS On Leopard: How Cool Is That?"
You can probably guess the author's opinion about the move from the title of the post"Great men are not always wise: neither do the aged understand judgement." Job 32:9
As far as I know Mac OS X supports X11. That doesn't mean that Apple will replace Aqua with it. Same thing with file systems.
Well, there's spam egg sausage and spam, that's not got much spam in it.
Indeed, we may see Mac OS X Server supporting default ZFS before Mac OS X proper. It would make sense to deploy it first in a limited market with technical expert users as your target market.
The Rise and Fall of Online Community
On an HP, I can:
Can I do ALL those things on the mac with it's management processor? Didn't see the remote graphical console - do I need a separate IP KVM to get that? Can I even get a text console with it? The original xserve didn't have a video card - is that now embedded? Don't see it. If I have to add one, it wastes one of two available slots. Given the GUI centric design of the mac, remote graphical console is important. ARD is not enough (which is what I use now) and I end up having to manually restart the service all the time via ssh because it keeps locking up.
The HP ILO is a truly awesome remote management system.
"given that it will be implemented for Linux pretty quickly."
Why is that a given? ZFS has been out for more than a year, and there's no sign that the Linux people are even attempting to copy it yet. Remember how long it took them to integrate XFS? In that case, they had the source, and it still took them years.
Suggesting that I base my deployment decisions on something that some theoretical Linux developer might decide to do at some unspecified point in the future, is just stupid.
- Old Man of the Mountain ---- "I want to disturb my neighbor"
Hard drives silently losing data is a problem solved by RAID.
_ data
That is profoundly wrong. Vanilla RAID will not discover and cannot automatically correct silent data loss. The reason is that RAID has no way of knowing which data is correct. For example, if two mirrored copies disagree on the contents of a block, the data is unrecoverable without manual intervention or external knowledge. Furthermore, in normal operation your RAID subsystem will simply read data from whichever drive is idle at the time the read request comes in; it does not ordinarily compare the two mirrors. The data will remain corrupted until the user notices a problem, at which point they have no practical recourse. Essentially the same problem occurs with parity RAID.
There is no dedicated hardware in your system that provides the end to end data integrity that ZFS does. I honestly suggest you learn more about it before airing your opinions. Here is a start:
http://blogs.sun.com/bonwick/entry/zfs_end_to_end
If two mirrored copies disagree on the contents of a block, the data is unrecoverable without manual intervention or external knowledge.
Or, you know, a checksum. Or more than one level of redundancy.
I agree that RAID-1 cannot, by itself, correctly recover from error-free reads of mis-matched data. But RAID 5 and 6 are both capable of verifying the primary data source against the parity data and transparently correcting errors that occur on less than the critical number of disks. In the common configuration this is only done when a hardware-level error is detected to keep things fast, but it's quite possible to verify every read if so your system is so configured. Mutli-layer RAID also provides this same sort of protection.
Reasons zfs isn't popular among "Linux people" are e.g. license issues and the lack of need for zfs in Linux (yep yep).
old
Old format of ufs default value. Supported as read-only.
44bsd
Used in FreeBSD, NetBSD, OpenBSD. Supported as read-write.
ufs2
Used in FreeBSD 5.x. Supported as read-only.
5xbsd
Synonym for ufs2.
sun
Used in SunOS (Solaris). Supported as read-write.
sunx86
Used in SunOS for Intel (Solarisx86). Supported as read-write.
hp
Used in HP-UX. Supported as read-only.
nextstep
Used in NextStep. Supported as read-only.
nextstep-cd
Used for NextStep CDROMs (block_size == 2048). Supported as read-only.
openstep
Used in OpenStep Supported as read-only.
I have a feeling all of those high end features will make zfs really slow for ordinary desktop users
Firefox Power http://firefoxpower.blogspot.com/
Not sure why you'd want to do bullet point 1. What do you want to install to a console? Or has HP expanded the definition of LOM?
I can kind of see why for #2, but then that's just for techs - who most likely shouldn't be messing with LOM anyway.
It can do #3.
Again, not sure why you'd need #4.
Apple's LOM is about equivalent to HP's LO Standard. Considering that it's Apple's version 1, and HP's (probably) version 20, I think they're coming along nicely. Not perfect, not overkill featured, but not useless by any means.
There are two types of people in the world: Those who crave closure
Ditto to that. I've toyed around with it, and the insides are painful to work with when I'm used to Linux (not to mention the vague hardware support). I'll still have to see if I can get everything I want on it working. If I can, I will switch. If not, I'll probably wait.
In undeveloped countries, the consumer controls the market. In capitalist America, the market controls you.
[ZFS] will be implemented for Linux pretty quickly.
*sigh*. I wish. ZFS is being implemented on FUSE. This automatically creates limitations in performance and function (no root ZFS). IMO ZFS on FUSE will be a no starter in production.
I don't think we'll see ZFS in the kernel proper either, given the history of incorporating XFS and ReiserFS 4. Along the same lines, DTRACE will probably never make it in. It's being cloned in the form of Systemtap.
Meanwhile, FreeBSD has been porting ZFS and DTRACE. MacOSX is (partly) based on FreeBSD and DTRACE has shown up in MacOSX.
I agree that ZFS is a good reason to convert a file server to Solaris from Linux. FreeBSD may become a good candidate also. I'm a Solaris admin and haven't done much with FreeBSD so I'll lean that way. I'd love to see ZFS in the Linux kernel, but I'm not waiting for it.
Perhaps the way to go is Solaris x86 with ZFS file server then a BrandZ zone running Linux to provide other functions?
From the description of Time Michine on apples site, it looks like it requires an external drive on which all the backup data is stored. I would infer from this that ZFS would only be used on the backup drive. This way they get all the snapshot tricks from ZFS without needing to implement it for boot/root in their kernel. Seems pretty smart to me, most of the benefit without any real hard work:)
But RAID 5 and 6 are both capable of verifying the primary data source against the parity data and transparently correcting errors that occur on less than the critical number of disks.
:)
With RAID-5, as with RAID-1, the critical number of disks is one. RAID-5 cannot transparently correct errors that occur on even a single stripe, unless you know a priori which stripe is affected.
With RAID-6 you can automatically correct errors that occur on a single stripe, but it still does not automatically detect such errors on read the way ZFS and RAID-Z do.
Or, you know, a checksum.
That's a great idea. Too bad those ZFS guys didn't think of it. Oh, wait.
http://maczealots.com/articles/acl/
It's there but it's not really supported at the interface level yet. FYI
"I don't think we'll see ZFS in the kernel proper either,"
I'd have to agree. Personally, while the ZFS featurelist is nice, cramming everything into the filesystem might not really be the brightest architecture choice yet.
For technical merit, I'd consider block device layering like the linux devicemapper and management ala EVMS to be far more powerful and flexible. Want a new feature? Put it in a layer, neither the devices above or below need to know about it. It leaves each component to be good at what it does (lvm for volume management, various filesystems for file management, block encryption layers to do their thing, etc) while keeping them apart and online/live replaceable and interchangeable.
Over past months, I've read a lot of people commenting on ZFS who have no idea what it is. What it is, is the next generation of filesystems, not a "tweak" of current fs technology. It just happens to "look like" an ordinary POSIX fs, from a distance (if you ignore the administration/pool stuff...) But inside, it's something new under the Sun, folks.
RAID experts don't grok it, because it does things RAID can't do (end-to-end).
Devotees of ext2fs, reiserfs (yay!), NTFS (LOL!), or HFS+ don't grok it, because none of those filesystems do what ZFS does.
Read about it before you write it off as old wine in a new bottle. To ask the question, "Does OS X need a new filesystem?" is a perfect example of missing the point. Once you've looked at what ZFS really brings to the table, you'll see why it's an inevitable future, sooner or later, and you'll stop looking foolish.
Some links I posted this week:
- http://www.osnews.com/story.php/16739/Screenshot-Z FS-in-Leopard
- http://mac4ever.com/news/27485/zettabyte_sur_leopa rd/
(older rumour http://www.osnews.com/story.php?news_id=14473)
For OS X people wondering why the fuss about ZFS - summaries include: - http://www.sun.com/2004-0914/feature/ - http://www.sun.com/bigadmin/features/articles/zfs_ part1.scalable.html
"Why ZFS for home": - http://uadmin.blogspot.com/2006/05/why-zfs-for-hom e.html
"Here are ten reasons why you'll want to reformat all of your systems and use ZFS.": http://www.tech-recipes.com/rx/1446/zfs_ten_reason s_to_reformat_your_...
And some more technical explanations from Chief Engineer: - http://blogs.sun.com/bonwick/entry/zfs_end_to_end_ data
- http://blogs.sun.com/bonwick/entry/smokin_mirrors
you had me at #!
I couldn't agree more.
And which parallel universe did you crawl out of?
Time machine + ZFS seems to be quite the perfect combination. A file system that supports not only snapshots internally, but also allows the addition of entire new disks (zpools) with next to no effort, added together with a backup solution that's exactly what most people need.... seems good to me :D
If you don't realise where ZFS goes beyond ext3fs, you got some readin' to do, Lucy.
you had me at #!
Apple's LOM is about equivalent to HP's LO Standard.
Except for the missing remote console which IMO is critical for dealing with equipment that is in a different physical location, or headless in a closet / rack. Not everyone works on the data-center floor (in fact, you generally try and spend as little time as possible there, being noisy and cold.) In apple's case, they really need a remote graphical console.
All I'm saying, is learn from HP. They did it right. If Apple cares about the enterprise at all (which I'm not sure if they do to be honest) they need to compete a little better and start making sure their boxes are more enterprise friendly. I'm sorry you can't see the utility of all the enterprise-class features HP offers in their remote management system. When you deal with as many servers as I do, it matters as it greatly improves productivity and gives me and my staff a lot more flexibility. I'll will say that I won't buy another server that doesn't have at least 95% of their feature set however.
There is a lot more to being "designed for the enterprise" than being rack-mountable.
With RAID-5, as with RAID-1, the critical number of disks is one. RAID-5 cannot transparently correct errors that occur on even a single stripe, unless you know a priori which stripe is affected.
:)
Well, cannot correct errors that occur on a single stripe without being detected by the hardware. It's not like the underlying hardware won't notice bits being swapped about. But for the sake of argument I'll limit the scope to filesystem-bypassing block writes or the transformation of bits into so other configuration such that the disk still reads without error.
In that case you're right of course -- there's still less than 2 independent copies of the data in a standard RAID-5 array. I got excited with my RAID-6 an multi-layer RAID line of reasoning.
With RAID-6 you can automatically correct errors that occur on a single stripe, but it still does not automatically detect such errors on read the way ZFS and RAID-Z do.
What makes you say that? The choice to verify every read is purely an implementation decision -- RAID-6 provides all the necessary data. And in multi-layer RAID you likewise have enough copies of the data to transparently reconstruct after a single (or sometimes multiple) filesystem-bypassing write.
That's a great idea. Too bad those ZFS guys didn't think of it. Oh, wait.
I never suggested that ZFS didn't use a checksum. I was just arguing the novelity you seem to think ZFS has -- using checksums to verify data integrity is not exactly cutting-edge computer science.
I'm also not arguing the usefulness of having sufficiently redundant data storage so that you can transparently recover from a variety or errors. I'm just saying ZFS is neither the first nor the only way to accomplish this, and other solutions provide both maturity (at least on non-Sun platforms) and hardware acceleration.
And ZFS might have lots of other useful features that make it a great choice. I just don't think this particular feature is unique or superior to other available solutions for the same problem.
What makes you say that? The choice to verify every read is purely an implementation decision -- RAID-6 provides all the necessary data. And in multi-layer RAID you likewise have enough copies of the data to transparently reconstruct after a single (or sometimes multiple) filesystem-bypassing write.
Because it seems to be universally true in practice. I don't know of any hardware or software implementation of R5 or R6 that reads parity in order to either verify or even spit out an error on read, only on some sort of dedicated parity check/scrub process. I hope some do that I haven't heard of.
I'm not sure that you know what Apple's LOM does. It does provide remote console login.
And I agree, Apple could learn a lot from HP too.
There are two types of people in the world: Those who crave closure
Have you read the link I posted? It is getting old to keep repeating Jeff Bonwick's arguments when you can just read what I already pointed you to.
The underlying hardware will not necessarily notice errors. Hard drives are only designed to error protect the magnetic domains on the disk. There are all sorts of other places in the increasingly long datapath to disk where data can be lost, and, in fact, routinely does get lost.
The choice to verify every read is purely an implementation decision
RAID-6 does not verify every read because it is a stupid way to achieve data integrity. It wastes two thirds of your aggregate IO read bandwidth when you can just use checksums virtually for free. CPU cycles for checksumming is dirt cheap whereas IO bandwidth is extremely expensive.
I was just arguing the novelity you seem to think ZFS has -- using checksums to verify data integrity is not exactly cutting-edge computer science.
Yet strangely there aren't any other widely available storage solutions that provide transparent data integrity from the filesystem down.
I just don't think this particular feature is unique or superior to other available solutions for the same problem.
Then name another one. I think we've already shown that vanilla RAID does not qualify.
Not sure what Linux distro you are using but when implementing Rehdat 8/9 at Cisco I used our Solaris install as the reference for things like NFS+ configuration and had little trouble translating between the two systems. Config files were generally in the same place, or close enough that figuring it out wasn't that hard. Perhaps things have changed since Solaris 8/9 and RH 8/9 but at that time the differences were fairly minimal for someone with a passing understanding of each.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
You admitted yourself that you
a) didn't know what you were doing
b) didn't understand what was going on
c) thought that you traced the problem (empty line), when that wasn't the problem at all, however you thought it was
...I used to be a Linux engineer. And I hated every second of it. Having come from Solaris, HP-UX and IRIX, working with Linux had been a very miserable existence. I went back to Solaris after a while, and I never regretted it.
In fact, I have never had a problem that you just described... but then again, I've been doing this Solaris gig for almost 12 years now.
I'd guess ZFS would be pretty handy for those of us with a lot of XRaid enclosures -- use them as JBOD with ZFS handled by the mac host, and you eliminate all sorts of annoying problems (e.g., each xraid is actually two separate arrays of 7 drives each, so raid-5 with a hot spare means losing 4-drives worth of capacity on each shelf.)
Also as xraid transitions (it must, right?) from 3.5" SATA to 2.5" SATA or SAS drives, they'll be able to pack 10 into a 1U enclosure, or 40 into a 4U enclosure. If the host supports ZFS then they could probably skimp on the "raid controller" contained in this hypothetical xraid replacment even more than they already have.
"But actually trying to use m4 as a general-purpose langage would be deeply perverse" --ESR
Though I still wonder: If it is not meant to replace HFS+, could there be any other reasons to support ZFS?
Of course there is! How about simply supporting ZFS?
Don't blame me, I didn't vote for either of them!
I'm not sure about most RAID cards, but I know that HP's SmartArray and IBM's ServeRAID have a background process on the adapter that continually tries to read and re-write each sector one block at a time in an effort to maintain data integrity. While not a complete solution for data integrity issues it is a strong step in that direction.
As a rock-in-roll Physicist once said, No matter where you go, there you are.
You get to choose whether to install it when you install the OS.
No, you don't "get to choose it", you have to track it down on the install CD. Furthermore, it gets blown away on upgrades. What does this mean? Nobody can actually ship a mass-market product for OS X that's based on X11. Apple treats X11 purely as a compatibility product for UNIX geeks.
Oh dear. Open up the System Preferences panel, click on the Accounts icon, click the Login Items tab, add X11 to the list of apps installed on login. Problem solved. Why would they *not* use the standard way to start apps on login ? It's not as though it's the default window server or anything...
Yes, and that's a problem: X11 should just "be there". You don't start up Quartz in System Preferences, it's just there and runs.
Consistent with what ?
Consistent with Macintosh.
Apple's way is simply to make an X11 app respond just like a native window AFAICT. Works just fine for me.
Apple's way is to treat all of X11 as a single app; that's not consistent.
Since I've never had any problems with keyboard layout (I can type Option-3 to get a £ symbol in nedit, for example, even on my US keyboard, which is usually a problem for me in Linux) I don't know what you're talking about...
When you change keyboard layouts in OS X, the X11 layouts don't change.
It's a package that 99% of the user-base won't need or want, but it's a choice.
Yes, and Apple's policies are working towards keeping it that way by setting up things such that X11 apps always remain foreign to OS X. It would be easy for Apple to put X11 on equal footing with Classic, Carbon, and Quartz/Cocoa and to ship Gtk+ (and maybe even Qt) with the system. Thousands of high quality Linux desktop apps would instantly run on OS X with a native look and feel. The fact that that isn't happening is Apple's choice. Apple, instead, wants to ensure that X11 remains a special-purpose tool for workstation users "switching" to OS X. That's not my guess, Apple employees have told me so.
It's not improper... it makes the alphabet song rhyme...
The underlying hardware will not necessarily notice errors. Hard drives are only designed to error protect the magnetic domains on the disk. There are all sorts of other places in the increasingly long datapath to disk where data can be lost, and, in fact, routinely does get lost.
Routinely does get lost is a bit of strech at best. If my data routinely got lost between the disk and the motherboard I'm pretty sure I'd notice. There are occasional transmission errors along physical data paths; if only someone had used your magical ZFS checksums when designing Ethernet or IP or SCSI to deal with these sorts of errors.
I know your link talks about how terrible firmware in SAN devices can be, but I have a hard time beliving that data integrity errors in a data storage device would go undiscovered for too long. If you're complaining about that you might as well whine about bad hard drive firmware or bad Ethernet firmware or bad RAM. In theory you could add protections for all those things, but it doesn't seem very useful to me; why not just test the system an to empirically discover such faults?
RAID-6 does not verify every read because it is a stupid way to achieve data integrity. It wastes two thirds of your aggregate IO read bandwidth when you can just use checksums virtually for free. CPU cycles for checksumming is dirt cheap whereas IO bandwidth is extremely expensive.
I don't know what kind of RAID-6 you use, but mine creates parity data that's significantly smaller than the original. I can't even guess at where you got the 2/3 number; the one I come up with is more like 1/3 worst-case-scenario, and that's only at the physical disk level, not at the host-interface level. And that worst-case scenario assumes that you haven't recently and weren't going to read the partiy data in the first place, and that reading parity data prevents you from completing other operations. I'll grant you that checksums are potentially more efficient from a data throughput standpoint, but let's try to stick to realistic numbers.
It's also worth noting that, while the checksum in ZFS is small, the entire data set much be read from memory and processed to calculate the checksum, in addition to whatever you plan to do with the data after you've done the integrity check. I/O starvation is not limited to disks; almost any busy system spends a huge amount of time waiting for data to come across the system bus from the RAM to the CPU. This is also not a task that can be entirely offloaded, even if you added a dedicated checksumming card or processor; given the end-to-end nature of ZFS, data would still have to come across the system bus to get to the card.
I'll admit there are things ZFS protects against that other systems do not, but those things don't seem any more likely to me than the possibility of a similar error in the ZFS system. For example, Mr. Bonwick sights bad firmware as a possible error. But how is an implementation error that causes a ghost write in my hard drive any more likely or difficult to detect than an implementation error that introduces a ghost write at the ZFS level?
If the data-integrity benefits of ZFS require that ZFS be implemented perfectly and only protects me from implementation errors in other, generally simpler systems, I'm not seeing a lot of practical benefit here. Belt-and-suspenders I guess, but nothing very pragmatic, since we're already talking about the very small percentage of the time that A) an error occurs and B) cannot be detected by all the existing data-integrity checking already in place. Not to mention the fact that, at least until ZFS is mature on non-Sun systems, I see the possibility of an error in the implementation of ZFS as much greater than the possibility of an error between my physical disk and my motherboard.
Yet strangely there aren't any other widely available storage solutions that provide transparent data integrity from the filesystem down.
No, but I can name several that provide it at both a higher and
You are correct. Apple wants X11 to remain a special-purpose tool for using UNIX apps in OSX. Obviously, they want to encourage the use of Aqua for anyone targetting the Mac platform, as they should. I would hate for companies to start releasing general-use apps for OSX using X11, as, like you mentioned, they don't play well with regular OSX-native apps. They will never have all the niceties of a native Aqua app, and so Apple is right to force developers to code for Aqua if they want to release a Mac app. X11 is there as an option if you really need to use an X11 app, but it will never be on the same level as Aqua as a vendor-supported deployment option.
"I like systems, their application excepted", George Sand (French)
ReiserFS should handle something dying quite well!
....raid-{1,5,6,10}.... + reiserfs == survive crash at anytime including a drive physically dying.It won't survive a marriage to Hans Reiser!
Do you even lift?
These aren't the 'roids you're looking for.
I am not a computer technical person, but reading about ZFS and similar i wonder one question.
Could i use a pool of disk images and Burnt CD's?
OK so maybe DVD's would be better or there might be Write once technology turn up sometime.
It seems to me if the information is not overwritten on copy then does it matter if that drive space never comes back in to the pool?
Have a pool of disk images, (50gig of hard drive space would give 10-12 DVD images) as the disk image gets full burn it, put it in a stack keep the image on the computer for a while as well then just drop them off as you need to free up the allotted drive space.
Why not burn a couple have one on hand put the other one elsewhere?
For a small bit of disk space, and on going cost of DVD's, you'd get a massive time machine backup disk.
Ideally you'd be burning a disk once a week at least, even once a day would still be good.
Is this at all feasible using ZFS or another FS?
"Call us when the New age is old enough to drink" Beck
Don't any of you guys use Macs?
.Mac? ZFS enables all kinds of coolness and I, for one, can't wait to get it on my laptop.
Here are five good reasons for Apple to go to ZFS:
-No more Disk Warrior. The entire data store is self-validating. No bit rot.
-No RAID controllers needed: ZFS gives you fast RAID for free. Just add drives. Why would anyone care? See #5.
-No more volumes and, therefore, no more volume management. ZFS eliminates the whole volume concept. Add a disk to your system and it joins your storage pool. More capacity. Not more management. What home user would want that?
-Continuous Data Protection out of the box. Time Machine could give you a view of your data every time you update a file.
-ITV, or whatever it is going to be called. Multi-GB files that each cost $10-20, that can't be backed up - thanks DRM! - and therefore need a cheap and highly reliable RAID. ITV, two firewire drives, ZFS and you are in business.
-Not to mention the existential pleasure of having great technology that Vista doesn't have. In fact, since consumer technology is driving the enterprise, expect ZFS on Mac to raise the bar for every OS and file system.
I suspect that Time Machine is simply the first of several beautifully designed storage utilities that we'll see on Leopard. How about automatic synchronization when you plug in an external drive? Snapshots automatically exported to
Read more at ZFS On Leopard: How Cool Is That? Means, Motive & Opportunity: Apple Kills the Media Center PC and the latest ZFS On Mac: Now All-But-Official.
And you heard about the native iSCSI support in Leopard, right?
Mainly so you can (re)install a system remotely... Very useful in a lights-out datacenter, especially one which is miles from where you are.
I have a number of sun systems which were physically installed into a rack by the fairly clueless on-site techs, they did little more than plug cables in, everything else i did remotely, that is power the systems up, install the OS, configure the OS, etc..
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
In theory you could add protections for all those things, but it doesn't seem very useful to me; why not just test the system an to empirically discover such faults?
Yes; that's exactly what ZFS does: test the integrity of your data to empirically discover faults. On the other hand, most RAID implementations and filesystems don't test the integrity of your data; that is left up to the application layer and hardware.
I can't even guess at where you got the 2/3 number; the one I come up with is more like 1/3 worst-case-scenario, and that's only at the physical disk level, not at the host-interface level.
I came up with 2/3 because you need to read the data and both parity stripes. Actually it's worse than that, you need to read all the stripes to validate any given block, just as you do in a write operation. For example in the minimal 4 disk RAID-6 array, reading and verifying will be 4 times slower than reading without verifying. That's why nobody does it. (Also because in any level lower than RAID-6 you are still screwed; you don't know whether the data or the parity is correct.)
It's also worth noting that, while the checksum in ZFS is small, the entire data set much be read from memory and processed to calculate the checksum, in addition to whatever you plan to do with the data after you've done the integrity check.
ZFS calculates checksums on a (variable size) block level. Compared with IO bandwidth, CPU and memory bandwidth are virtually free and getting cheaper all the time.
If the data-integrity benefits of ZFS require that ZFS be implemented perfectly and only protects me from implementation errors in other, generally simpler systems, I'm not seeing a lot of practical benefit here.
You know, there are so many problems with that statement that I don't even know where to start. If you believe that implementing a reliable filesystem is on the same order of magnitude of difficulty as perfectly error-hardening every piece of firmware and hardware that will ever be used with that filesystem, there's nothing I can do to convince you otherwise.
md5sum comes to mind -- that provides file-level data integrity checking, which is superior even to the filesystem-level checking that ZFS provides. And it's pretty commonly used in an automated fashion to verify that executables and archives are intact.
ZFS has an enormous advantage over md5sum: it is completely transparent to the user. I'd say that md5sum is the problem, and ZFS is the solution.
(Of course md5sum is primarily used to solve higher level problems.)
I would hate for companies to start releasing general-use apps for OSX using X11, as, like you mentioned, they don't play well with regular OSX-native apps.
They "don't play well with" other OS X apps because Apple doesn't invest time to make things work together well. With a modest amount of work, Apple could make X11 and Gtk+ apps better integrated into OS X than Carbon apps are.
They will never have all the niceties of a native Aqua app, and so Apple is right to force developers to code for Aqua if they want to release a Mac app.
Aqua is just the UI; there are many toolkits and APIs implementing Aqua. Apple themselves has Cocoa and Carbon, two completely different APIs and libraries. Making an Aqua-integrated version of Gtk+ is fairly straightforward, in particular given Cairo and the existing work on KDE/Gnome integration. Even non-Gtk+ X11 apps could integrate much better with the Apple desktop than they do, through improvements to Apple's X11 server implementation.
The fact that Apple keeps choosing to maintain proprietary APIs, and that X11 apps are so poorly integrated on Macintosh, is a business and marketing choice, not a technical one; Apple deliberately keeps it that way. And that brings me back to my original point: ZFS is an analogous choice. It's not that it's technically better in any meaningful sense, Apple just doesn't want too much Linux compatibility.
Apple has a choice between lots of file systems to succeed HFS+, among them ext3 and ZFS, and they chose ZFS, a file system that has no good Linux implementation and a Linux-incompatible license. That tells you how much they value Linux interoperability. How hard is that to understand?
It's my understanding that the developers at DragonFlyBSD are already in the process of porting ZFS to DragonFly. Maybe they're already done. Either way, my point is that I keep wishing there was a way for Apple to dump the Mach microkernel in favor of something a bit speedier. That's where DragonFly comes in. From my limited understanding of what they're doing it sounds like they've been adding a message passing system similar to what most microkernals use. So Mac OS X uses a Mach microkernel with FreeBSD wrapped around it, maybe they could dump that and just adapt everything to the DragonFly kernel which sounds like it will be much faster and was also based upon FreeBSD 4.8.
I came up with 2/3 because you need to read the data and both parity stripes.
You only need to read one parity set unless you detect on error, and you need to read the original stripe regardless. You would only need the second set of parity data to determine which set of data is accurate in case of a conflict; in the general case the first parity set will match with data, and the second parity set can be ignored.
Actually it's worse than that, you need to read all the stripes to validate any given block, just as you do in a write operation. For example in the minimal 4 disk RAID-6 array, reading and verifying will be 4 times slower than reading without verifying.
When I read from RAID-6 without verifying I need to read the stripe itself. When I read with verification I need to read the stripe and one of the parity sets, which is 1/2 the size of the stripe. I come up with 100% + 50% = 150%. That's only 1/3 of the disk bandwidth going to parity data. Obviously you've got some other read/verify algorithm in mind, but I don't know why.
And that 50% overhead is still assuming that the parity data isn't cached and that the parity read (which is on a physically seperate disk) blocks other operations.
ZFS calculates checksums on a (variable size) block level. Compared with IO bandwidth, CPU and memory bandwidth are virtually free and getting cheaper all the time.
I don't know what systems you've been using, but it's been a long time since my memory bandwidth has jumped up any considerable amount. Overall bus speed is slowly gaining, but processor speed is still pulling away from bus speed, and unless there's some breakthough in bus technology that's like to continue to be true.
ZFS has an enormous advantage over md5sum: it is completely transparent to the user. I'd say that md5sum is the problem, and ZFS is the solution.
In many applications, md5 is completely transparent to the user. Run any modern package management system -- it fetches archives and their checksums and notes errors and in many cases refetches automatically in an attempt to correct the error. And it actually provides higher data integrity than ZFS's filesystem-level checking, because it actually knows what's supposed to be in the file, not just what the file system last thought was supposed to be in the file. As others have noted, it's not really end-to-end unless one of the ends is "program consuming data" -- with md5 that's possible, with ZFS it's not.
One of the two links at the bottom of the article describes the addition of HiDPI, making the UI resolution-independent. It explains why they're changing its name.
This is not the signature you're looking for.
My Prediction: Apple and Sun will come to an agreement whereby Solaris improves it's OSX support to help OS X on the corporate desktop.
Do you even lift?
These aren't the 'roids you're looking for.
So, if I were to stick a USB thumbdrive into my computer, how would I copy files to it? The way I understand ZFS, the thumbdrive would be added to my storage pool, and the file system would somehow know what I want on it. And when time came to eject the thumbdrive...?
Could someone clear this up for me?
I never used it myself but read that Plan9 had offices use a WORM drive that basically captured everything as you typed it and would let you undo basically forever one character at a time. If ZFS takes 1/3 second to do a snapshot of a 10 gig mp3 partition like the link someone posted above, how about something that would let it store my entire workflow at a very high granualarity say for 10 minutes, a day, a week. So I could undo whole paragraphs, redo the filters in photoshop or whatever.
Apparently Time Machine will have an api for applications to use. I think the time has come for people to realize that while time progresses forward in a single line (at least in this universe we share) workflow does not really. It is lots of little projects interleaved in time, and in a single project maybe you go down one path of processing and then back up and realize you should be doing something different - but were you paranoid enough to back up your file at that point you need to revert to?
Generally I am very paranoid in terms of saving lots of versions, but really the OS should handle all this. I would also like to be able to type in a label or even a short paragraph that describes what I think is a certain milestone (i.e. solved that problem, break for lunch, fixme, function x works, come back to this later). It would be saved by the OS system-wide, and could have an alarm to tell me to check if it is finished (if I say "come back in 2 hours" then it should tell me calmly in 2 hours to do so).
To me this would be such an earth-shattering advance that it would release (most) humans from the shackles of the computer and start making computers help us be more productive and smarter. This doesn't really require ZFS though it would be extra. I'm trying to imagine a world where this works in Linux but I don't know, I think Apple has a shot at it.
Well, they made it as good as they had to, and they're still adding to it. By the way, nothing's preventing anyone from writing their own X server if they feel Apple's is inadequate...
"I like systems, their application excepted", George Sand (French)
That would correctly read:
...
"The letter 'Z' is improperly pronounced 'Zee' in the USA and Iraq (after 2003)"
It's improper to place double quotes (") within double quotes -- single quotes (') are used instead
I see the confusion. You are thinking about sequential reads, where you are correct, the asymptotic bandwidth penalty is 1.5x for a 4 disk RAID-6. With small random reads the penalty is 3x because you have to read both data chunks and one parity chunk from the stripe even though you only need one of the data chunks. On any practical workload the penalty is likely to fall somewhere between those limits.
Memory bandwidth isn't growing as fast as CPU power, but they are both growing so much faster than practical drive throughput that they have become effectively infinite by comparison. The only way you can become memory bound doing disk IO is to have a pure sequential access pattern and dozens of spindles per CPU. And as long as we are stuck with hard disks for mass storage, that imbalance is only going to get more extreme. Memory bandwidth is one of the last things a modern filesystem designer needs to lose sleep over.
The bottom line is that ZFS establishes and maintains true data integrity from the filesystem layer down at a very reasonable cost. Neither RAID of any flavor, nor any widely used filesystem can say that. This has been an interesting discussion but I don't think there's really any more that needs to be said to defend this position.
What if they just chose the one that was best for what they where looking for?
Relating this to some kind of slight on linux is just plain stupid, and seems like
a sad way of almost searching for a reason to draw the conversation away from
two other companies ( SUN and Apple ) back to Linux.
Give it up.
a man, a plan, a canal, panama
One solution is reverse Engineered, hoping for the best, and with the Damocles sword of MS patent litigation hanging over anything that smells remotely related to MS.
The other is an open solution provided by a company that, finally, is betting in openess and colaboration.
Sorry, but they simple ain't the same things.
IANAL but write like a drunk one.
Just for the record...
IANAL but write like a drunk one.
Is it a graphical console? If so, it doesn't say so in the marketing lit. In fact, the marketing lit is VERY thin.
It's CLI, but my point was that it's remote, which the GP didn't seem to notice.
There are two types of people in the world: Those who crave closure
Apple's LOM does not provide a remote console login.
The only supported remote login solution is Apple Remote Desktop. This is not available until the system has come up completely, and is not capable of tasks like "repair the boot filesystem in single-user mode."
There may be some limited, unsupported serial port functionality, not via the LOM. This, too, cannot allow you to select a boot device or otherwise interact the boot process.
If you believe otherwise, I'd love to hear specifics?
Abstraction and factoring aren't always the right choice.
ZFS wins big in several ways (and will likely win more in the future) because it shatters the walls between the various layers of the stack that, to the end-user, is simply "the disks". This is, at least in part, because the hardware balance has changed. (CPU vs. memory latency vs. disk latency vs. disk bandwidth vs. network bandwidth etc.) But it's superior for other reasons too. For example, the way it (or NetApp's WAFL) does snapshots is dramatically superior to volume-management-based snapshots, for most (nearly all) purposes.
If I remember correctly, Carbon apps would not run on drives formatted with UFS on the Mac. (10.2 IIRC). I wondeer if only Cocoa apps will be supported on a ZFS formatted drive?
What if they just chose the one that was best for what they where looking for?
Obviously, it is. That is, they choose a few extra features (useless ones, as far as I'm concerned) over Linux compatibility.
Relating this to some kind of slight on linux is just plain stupid,
Companies don't "slight" open source software, they attempt to compete with it.
and seems like a sad way of almost searching for a reason to draw the conversation away from
two other companies ( SUN and Apple ) back to Linux.
What's there to "draw away"? In the long term, there are going to be two operating systems: Windows and Linux. What Apple and Sun do is relevant only to the degree that they are compatible with those two operating systems. That's why I'm looking at how Apple's choices relate to Linux. Going with ZFS was a stupid mistake on their part.
I think ZFS won't get into the Linux kernel for similar reasons that ReiserFS 4 is having problems. There are licensing issues too.
I'd say putting RAID, Volume Management and the filesystem into one unit is a good idea. Right now, the file system touches the data. Then the VMS. Then RAID. Then the device driver. Maybe a compression layer. And encryption.
This is the old BSD vs Bell Labs. ls -r vs ls | sort -r. Or monolithic vs micro kernel OS.
BSD's FFS depended on knowledge of the underlying disk geometry to do some of its speed. The only filesystem I can think of that is optimized for RAID is WAFL in NetApp. ZFS has taken some ideas from WAFL.
When you use RAM in your system, do you have to think about the 1st slot is a 512MB, the 2nd is a 256MB, etc? And there's another 1GB in swap. But when you go to malloc, you don't care where one device ends and another begins. You just get some memory from the pool. There might be ECC on it too. If it was on swap, it gets migrated to RAM as it gets used. ZFS does this for file systems.