ZFS Set To Eventually Play Larger Role in OSX

BlueMerle writes with the news that Sun's ZFS filesystem is going to see 'rudimentary support' under OSX Leopard. That's a stepping stone to bigger and better things, as the filesystem will eventually play a much larger role in Apple OS versions. AppleInsider reports: "The developer release, those people familiar with the matter say, is a telltale sign that Apple plans further adoption of ZFS under Mac OS X as the operating system matures. It's further believed that ZFS is a candidate to eventually succeed HFS+ as the default operating system for Mac OS X -- an unfulfilled claim already made in regard to Leopard by Sun's chief executive Jonathan Schwartz back in June. Unlike Apple's progression from HFS to HFS+, ZFS is not an incremental improvement to existing technology, but rather a fundamentally new approach to data management. It aims to provide simple administration, transactional semantics, end-to-end data integrity, and immense scalability."

Comments

Clearly ZFS is superior...

[psiogen]

It has a "Z" in it!

Re:Clearly ZFS is superior...

[Trespass]

It gives new meaning to the term 'Killer App'.

Re:Clearly ZFS is superior...

[Anonymous Coward]

Uh... that is 'Killer.app' to you sir!

Re:Clearly ZFS is superior...

[hey!]

Add support for the Andrew File System and you'd have the bases covered from A through N on to Z.

Re:Clearly ZFS is superior...

[TheVelvetFlamebait]

I dunno... I'm still waiting for iFS.

Re:Clearly ZFS is superior...

[Neanderthal Ninny]

I think it is the application that Charles Manson and the people at SuperMax uses.

Does anyone proofread these articles?

[tomRakewell]

It's further believed that ZFS is a candidate to eventually succeed HFS+ as the default operating system for Mac OS X

Macs are really going to stink if Apple changes their default operating system to ZFS. ZFS is a file system.

Re:Does anyone proofread these articles?

[rueger]

Macs are really going to stink if Apple changes their default operating system to ZFS. ZFS is a file system.

Yes, but the fanboys will still rave about how superior the whole thing is the Windows....

Re:Does anyone proofread these articles?

[McFadden]

Does anyone proofread these articles?

They haven't done for 10 years. You're expecting them to start now?

Re:Does anyone proofread these articles?

[Anonymous Coward]

And, sadly, they will of course be correct.

Re:Does anyone proofread these articles?

[aliquis]

Tell me about it:
http://www.youtube.com/comment_servlet?all_comments&v=zTlHKXbBqFQ&fromurl=/watch%3Fv%3DzTlHKXbBqFQ (1wellhowdydoo1)

Re:Does anyone proofread these articles?

[hackstraw]

Macs are really going to stink if Apple changes their default operating system to ZFS. ZFS is a file system.

Right, and emacs is a text editor.

Re:Does anyone proofread these articles?

[stefanlasiewski]

Silly man. Everyone knows that Emacs is an operating system. It even runs VI.

Re:Does anyone proofread these articles?

[TheBrutalTruth]

A good question for the Ask Rob Malda interview!

Re:Does anyone proofread these articles?

[GPL Apostate]

It probably won't happen anyway. ZFS wasn't invented at Apple, and thus will never become an important part of 'The Mac.' Unless, of course, Apple buys Sun Microsystem. Then it's a certainty. (and of course, if that happens, Darwin will be ditched for OpenSolaris (which will be closed.)

Re:Does anyone proofread these articles?

[phliar]

It probably won't happen anyway. ZFS wasn't invented at Apple, and thus will never become an important part of 'The Mac.'

And Unix wasn't invented there either, but I'd call OS X "an important part of 'The Mac'".

Re:Does anyone proofread these articles?

[Durandal64]

Apache wasnt invented at Apple. Neither was Samba. Neither was SSH. Neither was gcc. All of these things are very important in Mac OS X.

Re:Does anyone proofread these articles?

[GPL Apostate]

What does OS X have to do with Unix, other than the fact that it's based on NeXTStep, and NeXTStep happens to use a POSIX based API as it's primary interface?

No, it can be said that Apple's new OS is even more formed on a proprietary base because of it's direct lineage to NeXT's OS. They could have based it on Linux or a BSD Unix if they'd wanted. Or even the 'real' Unix codebase. They chose none of these.

"Open Source" was a popular buzzword during the launch period for OS X, so they chose to borrow components from the OSS community, and launch forth a brouhaha marketing blitz. That's typical for Apple. They leveraged the hype (while ignoring the substance) behind RISC in a similar fashion about a decade earlier.

Re:Does anyone proofread these articles?

[cthulhu11]

"emacs" is a group of text editors. There have been a number of implementations over the years. PDP-10/TECO. Gosling. Unipress. MicroEMACS (smaller binary than vi). And of course the bloated GNU emacs that punchcard fanboys like you enjoy pretending is "the" emacs.

So.... BSD or Solaris???

[queenb**ch]

Even though OSX will still be Unix, will they'll move away from BSD and toward Solaris?

I'm hoping not, since many things behave very oddly on Solaris. Non standard tools and such, but it would be one way to keep it from running on cracked PC's.

2 cents,

QueenB.

Re:I hate the new comment system

[ArcticFlood]

Turn your highlight threshold down to whatever you normally browse at. That is a workaround for the time being.

Buzz compliant

[suv4x4]

end-to-end data integrity

You can't talk about end-to-end data integrity when this is just a filesystem. It's only one tiny place where the data you store in said file system can wreck its integrity. Are there memory bus or in-memory check for integrity of data read from ZFS? What about applications?

Also stop talking to ZFS. Very secret internal sources told me ZFS was supposed to be a bigger event in Leopard but Steve killed it because Sun scooped him. It has happened before folks!

Don't scoop the Steve. You scoop the Steve and business is over.

Re:Buzz compliant

[Anonymous Coward]

There is a in-memory checksum check for all data that is read, yes. If the checksum doesn't match ZFS tries to read the same data from another disk, in a mirror/RAID-Z setup.

Re:Buzz compliant

[lorenzino]

>It has happened before folks! Tell me more, argument about it please. Thanks.

Re:Buzz compliant

[kaiwai]

"If you care about my opinion, you have no girlfriend." - Miguel de Icaza, Novell VP & founder of GNOME and Mono

Well of course I have no girlfriend - I have a boyfriend!

Re:Buzz compliant

[Anonymous Coward]

So APPLE wants to use SUN'S ZFS, eh?

I'll be back in coupla weeks.

Re:Buzz compliant

[caseih]

That's Steve's loss then. Too bad his own ego often gets in the way of things that could benefit the customer. Honestly, why should Sun really care what Jobs does with ZFS in the long run. Sure it'd be good for Sun in terms of publicity, and maybe even some royalties. But in the long run, I can't see it being that big of a deal for Sun.

Re:Buzz compliant

[jadavis]

Are there memory bus or in-memory check for integrity of data read from ZFS? What about applications?

They have defined what they mean by that claim already: they have a checksum (256-bit, I think) on every block, and that checksum is checked from the OS when the block is read.

This will catch some errors that might otherwise go uncaught, which is important for servers that move a lot of data around.

It will not catch a memory error at the wrong time, or a processor error that stores the wrong value, or an error in the brain of the person who reads the data from the screen.

Re:Buzz compliant

[DAldredge]

Why should business trust a platform from a company that lets petty crap like that drive product development?

Re:Buzz compliant

[mikeee]

IIRC, the block checksums are stored in the inode, not with the individual blocks. It turns out that one of the main failure modes of modern disks isn't reading a few bits wrong, but missing slightly on a seek and actually returning the wrong block! Block-included checksums won't find this, since it's still a valid block...

Re:Buzz compliant

[suv4x4]

Tell me more, argument about it please. Thanks.

There you go

Re:Buzz compliant

[Anonymous Coward]

You can't talk about end-to-end data integrity when this is just a filesystem. It's only one tiny place where the data you store in said file system can wreck its integrity.

It's called a "filesystem", but it does more than any other filesystem, e.g., it has its own RAID. It does so much more because it isn't constrained by the old abstraction boundaries.

there memory bus or in-memory check for integrity of data read from ZFS?

Yes.

What about applications?

It's copy-on-write, so even if your application decides to "delete" all your data you can still get it back.

Do you have any complaints that can't be answered by looking at wikipedia's article for 10 seconds?

Re:Buzz compliant

[Anonymous Coward]

"You can't talk about end-to-end data integrity when this is just a filesystem."

ZFS is NOT just a filesystem, it is also a volume manager.

It is currently possible to create a zpool and use a different file system on top of it (like UFS, PCFS, UDFS, or any other FS Solaris currently supports).

Please read up on the subject before you post about ZFS.

Re:Buzz compliant

[amRadioHed]

Fuck'em. It's Apples loss if Steve throws a tantrum and ditches ZFS for petty reasons.

Re:Buzz compliant

[kithrup]

Not quite... ZFS stores a checksum with each block pointer. So wherever you have a structure that indicates where the data is, there's also a checksum of that data. This also means that the block pointers themselves are checksummed with their pointers. And so forth. The only one that doesn't have a checksum with the pointer is the top-level root pointer, and they have multiple copies of that for redundant checksumming.

And yes, for true integrity, you need ECC memory, and ECC CPUs. I don't know if the Intel CPUs have any sort of internal error checking; I have worked with processors that have.

Re:Buzz compliant

[Anonymous Coward]

And yes, for true integrity, you need ECC memory, and ECC CPUs. I don't know if the Intel CPUs have any sort of internal error checking; I have worked with processors that have.

They do. They have to: at current process nodes it's more or less impossible to build functioning CPUs without internal ECC protection, since the SEU (single event upset) rate gets worse as gates get smaller. In fact, as I understand it, high performance logic at 65nm and 45nm often requires error correction for logic paths, not just memories.

Re:Buzz compliant

[FueledByRamen]

And that is why the block checksum is a function not only of the contents of the block but of its address on disk. If the disk picks up the wrong block, the address you input into the checksum won't match and the verify will fail.

Re:Buzz compliant

[Anonymous Coward]

integrity

We've heard of it.

Re:Buzz compliant

[Jeremi]

Why should business trust a platform from a company that lets petty crap like that drive product development?

They shouldn't, if that is indeed the case. OTOH, you are only asking that question because you are trusting the words of some pseudonymous poster on Slashdot. Why do you trust that what he said happened, actually happened?

Re:Buzz compliant

[DAldredge]

Because such rumors are not new. It has been reported by multiple, realworld sources, that Apple did the exact same thing in regards to ATI.

Re:Buzz compliant

[Lars T.]

Because such rumors are not new. It has been reported by multiple, realworld sources, that Apple did the exact same thing in regards to ATI. According to similar rumors, they also did it to Nvidia - and we all know that Apple uses neither ATI nor Nvidia cards, right?

Time Machine

[JayPee]

This is awesome and I knew there had to be something more interesting behind Time Machine. While I'm not that impressed with how it appears it's going to work in 10.5, later versions of OS X, with full ZFS support, will make Time Machine damned near magical.

Re:Time Machine

[wile_e_wonka]

I notice ZFS has read-only support in Leopard--does this mean only that it is able to read a drive formatted in ZFS, if you happened to plug in such a drive? Or does this mean that the time machine data is being stored in a read only ZFS format? I would think the former (which would definitely be no big deal), because I don't see how the OS would be storing data in ZFS format unless it writes to it. Only having read support for ZFS, I would think that there is no way for the OS to record information in ZFS.

Someone please clarify for me, please...?

Re:Time Machine

[aliquis]

No, time machine probably doesn't use ZFS atm and is implemented in some other way, read-only ZFS support are rather useless but probably easier to implement, and if Apple had got it all working (and Sun got it bootable, maybe they had now, it was a long time since I read about it) I guess they might had switched filesystems, or offer it as an option, or use it in timemachine.

Anyway we will hopefully see it in a minor release update, I just hope they don't call it beta just to remove it later and not release it for real in 10.6 =P

Re:Time Machine

[LKM]

Except Time Machine does not use ZFS.

Re:So.... BSD or Solaris???

[khb]

BSD and Solaris have compatible licenses. So new tecchnologies developed in either can potentially migrate to the other. That is, of course, the point of Open Source isn't it?

A filesystem isn't a kernel, so leaping from the incorporation of ZFS into Darwin to a replacement of Mach and/or the BSD bits with Solaris is a bizarre one.

So...

[Anonymous Coward]

Where's my linux boot support?

Re:So...

[Anonymous Coward]

It's here here, and here.

Re:So.... BSD or Solaris???

[Anonymous Coward]

Non standard tools and such

Uh, Mac OS X is certified standard UNIX. Solaris is also certified standard UNIX. And they're both fully POSIX compliant.

What are some examples of non-standard tools?

MOD SLASHDOT -1 ANNOYING

[argent]

I can't even see my preferences link in at all with this system, the floaters do not work in Safari 3. I had to go straight to my user page by URL.

In addition, what's the point of having a checkbox "to turn off the excesses of the user interface" if it doesn't actually DO that?

Damnit!

[pi_rules]

Alright, who broke the comments? Seriously, I'm stuck in this "new" version and it doesn't make fuck-all of any sense to me.

Re:Damnit!

[colourmyeyes]

The new version works alright, but I have no idea how to post a new comment (i.e. I only know how to reply). The other day I was a first poster, and I went back to the old comment system just to make the initial post. I'm probably just overlooking the obvious.

Re:Damnit!

[Corporate Troll]

Yes, to the side you should see a box containing something like "40 Comments" as a title. Then you have white area, that with the text "nn Full" with a white background, then a white slider which you can use to adjust the amount of full comments, then a grey area, with the text "nn Abbreviated" followed by a slider (again to adjust the visible posts) and finally a dark-grey area containing "nn Hidden".

That said, this box only appears when you scroll to the end of the story... (Not the comments, obviously)

Below that, you'll find three links: "More", "Prefs" and "Reply".

To reply to a story, click "Reply". Once you get used to it D2 is better.

Re:Damnit!

[99BottlesOfBeerInMyF]

I believe the function you describe is still broken (completely absent) on Safari, which could also be the cause of confusion. I reported the problem months ago but always have to switch back to the old comment system to post a comment that is not a reply to some other post.

Re:Damnit!

[colourmyeyes]

Wait, a helpful response and not mere invective? This is Slashdot; are you sure you're in the right place?

Thanks.

Re:Damnit!

[Anonymous Coward]

So did I. It works now.

Re:Damnit!

[tholomyes]

Oddly enough, it was your comment here that made me finally curious enough to try the new comment system. Maybe I'm just sadistic.

Re:Damnit!

[Corporate Troll]

Sorry, I must have forgotten that I'm a Troll... Won't happen again, promised!

In Soviet Russia, Trolls help YOU!

Hmmmm, that didn't turn out how I intended. ;-)

a true end

[gEvil (beta)]

Unless I'm mistaken, this will mean the true end to resource forks on the MacOS. For those of you who aren't familiar with them, resource forks were a part of a file under the Classic MacOS (OS 9 and before) that contained icon information, filetype and creator codes, etc. This part of the file was only supported under the HFS and HFS+ filesystems, meaning the resource fork would get lost if you copied a file to a non-HFS/HFS+ filesystem (this is why files copied to FAT filesystems in the old days often wouldn't reopen on a Mac. It also explains the "Mac OS X" folder with underscored-dot files from archives created with OS X's built-in zip utility). With OS X, Apple rolled the resource fork into the "data fork" portion of the file, meaning the information was still there for legacy purposes. However, this is only supported under apps that know where to find the information. This change has the potential to cause some headaches for shops that have legacy files spanning several decades. OTOH, I'll be glad to see it finally go...

Re:a true end

[Just Some Guy]

For those of you who aren't familiar with them, resource forks were a part of a file under the Classic MacOS (OS 9 and before) that contained icon information, filetype and creator codes, etc.

I'll be happy to see them kill that obsolete feature. It's hard to implement everything-is-a-file semantics when some things are files, and others are combinations of random amounts of metadata.

Re:a true end

[nine-times]

With OS X, Apple rolled the resource fork into the "data fork" portion of the file, meaning the information was still there for legacy purposes.

That doesn't sound right to me-- or at least I'm not sure what you mean by that. OSX still has resource forks, but Apple basically told developers not to put important information in them anymore because they get lost so easily. They can't just push the resource fork into the data fork of the file, because in many formats there's essentially no space for that information. How do you store an icon, metadata tags, and a default application setting into a normal .txt file without making it into something that isn't really a normal text file?

So right now, developers are only really supposed to use resource forks for things that don't matter much. So if they're stripped, you lose a little metadata, but nothing catastrophic. It used to be that some files would carry all their data in the resource fork, and that was a bit of a nightmare. You'd lose important information all the time. In order to protect resource forks a little while transferring them to other file systems and such, Apple made those little files that you mentioned that begin with dot-underscore. So copy a file to another filesystem, and it'll often dump the resource forks into those dot-underscore files so that OSX can recombine them later.

I don't see how ZFS will change the situation greatly. Resource forks are just how HFS+ supports metadata. ZFS supports metadata too. I'm sure the technical implementation is different, but I bet you'll still risk losing your ZFS metadata if you move your file to a FAT partition. Or am I missing something?

Re:a true end

[rabtech]

Not only can you use ZFS extended attributes for this but NTFS also supports true alternate data streams and has since its inception; this is how NT fileservers supported Mac clients natively (Services for Mac). You could copy a file to the NT share and back without losing any of the resource fork data.

I'm sure there are other filesystems that have done the same thing in the past as well.

Re:a true end

[Anonymous Coward]

NTFS also supports true alternate data streams

You mean that 'feature' that's only used to hide spyware and rootkits? No thanks, Apple.

Re:a true end

[noidentity]

HFS+ supports a separate resource fork; in fact, it supports up to 256 (I think) separate forks for a file. For a while now Apple has been recommending that developers not store anything critical in the resource fork, since it gets lost easily when transferring to another system (and isn't supported on UFS I believe). With the demise of the Classic emulation environment on Intel Macs, the major source of resource-using applications is gone.

Re:a true end

[Guy Harris]

For a while now Apple has been recommending that developers not store anything critical in the resource fork, since it gets lost easily when transferring to another system (and isn't supported on UFS I believe).

Apple's UFS doesn't support additional data streams such as a resource fork, so the resource fork would be stored in a ._ file. Sun's UFS, however, supports multiple data streams in sufficiently recent versions of Solaris (I think they might've first appeared in Solaris 9), just as ZFS does.

They said the same thing about UFS.

[argent]

They made a big deal about the import of the latest UFS from FreeBSD in Panther, and their support for UFS was actually reduced in Tiger because they put the Spotlight hooks into HFS+ instead of using the hooks already in the vnode layer in Darwin.

So don't do anything that would depend on them supporting ZFS.

Re:They said the same thing about UFS.

[erikvcl]

Is their support for UFS better now? The last time I used OS X (v10.2) their support for UFS was crap. I chose UFS to have a true Unix-like filesystem with case-sensitivity that didn't corrupt itself. Boy did I make a mistake! UFS was dog-slow and lots of Mac OS X software won't run on it. Nice.

Re:They said the same thing about UFS.

[Ilgaz]

I don't think UFS using community would be happy about Spotlight anyway.

Spotlight in current form tries to index every single source file, huge framework headers and there is no practical way to stop it. I have tried the Privacy pane as suggested and no, it doesn't explain my 130 MB of spotlight metadata after installing Developer tools and couple of GNU libraries.

If they have checked the NeXT history, they would figure the UFS is the default,supported Filesystem on NeXT. As OS X is a mix of NeXT with FreeBSD and Cocoa/Carbon, it is pretty natural that UFS gets into it a bit lately but finally.

I can imagine what Apple needs for supporting ZFS on startup volumes. Complete metadata and resource support. They could be happy with their ext3 plain filesystem but Apple using professionals REALLY label their files, sometimes change their icons, sometimes has to FORCE OS to open a file with a different version of suite (e.g. Quark 7 vs 6), add comments to them and professional software developers like Adobe still stores critical data on resource forks.

If there is a way to make ZFS support all those features without huge hacks (like the ZIP _resource stuff), they would give up their HFS+. Another thing is, it must support every serious software (non hack) backwards. You may find yourself using a application from 2001 written in Carbon under OS X and only it can provide the tool you require.

I am saying these since some elitists think Apple is backwards and stupid still supporting resource forks and implement special features to OS X just to give minimum compatibility with old applications.

Before critising HFS+ and suggesting Apple to use plain, Unix filesystems, they should sit around in a professional environment such as a DTP house, Movie studio and see how all those "childish" "backwards" features are used by professionals in job.

This is not a post against ZFS, I am just trying to explain why Apple can't magically move to another filesystem just because it has better features. Not even mentioning the "overhead" required by ZFS and the fact that there are some 2k/4k (Cinema) edit environments which you can't even enable journaling let alone adding another layer of overhead.

Also while writing these, if I only used plain Unix tools without any "native Mac" Application, e.g. use OS X as Darwin with X11, UFS would be my choice of filesystem.

Re:They said the same thing about UFS.

[flaming-opus]

True, but the capabilities of UFS don't really exceed HFS+. ZFS, on the other hand, is a thoroughly modern filesystem. UFS is just as rusty as HFS+.

Re:They said the same thing about UFS.

[kithrup]

You can disable Spotlight on a volume with sudo mdutil -i off /Volumes/foo (or simply / for the root volume).

As has been commented above, ZFS already has support for the metadata stuff.

Re:They said the same thing about UFS.

[kelnos]

Pardon my potential ignorance, but I was under the impression that there was nothing you couldn't store in a resource fork that you couldn't also store in a 'normal' extended attribute that file systems like ext3, xfs, reiser, etc. have supported for some time. Is this not the case? Obviously there would need to be some 'conversion mechanism' in OSX to preserve your resfork/extattrs when moving between file systems, but that's just a detail.

Re:They said the same thing about UFS.

[Anonymous Coward]

They made a big deal about the import of the latest UFS from FreeBSD in Panther, and their support for UFS was actually reduced in Tiger because they put the Spotlight hooks into HFS+ instead of using the hooks already in the vnode layer in Darwin.

Are you certain about this? I haven't ever tried it myself, as I have zero need to use UFS for anything, but I've seen people report success with Spotlight searches on UFS volumes.

It would surprise me if the Spotlight hooks were in HFS+ rather than the VFS layer. Apple's kernel guys have generally seemed quite strict about layering violations, at least in their interaction with KEXT developers on mailing lists who propose doing things which violate layering.

Re:They said the same thing about UFS.

[argent]

I don't think UFS using community would be happy about Spotlight anyway.

I'm an old-school UNIX guy who qualifies as part of "the UFS-using community" and Spotlight is Tiger's "killer app" for me.

Apple already implemented support for resource forks and almost all the other metadata in HFS+ in arbitrary file systems. Yes, if you go down to the command line you can see some of that metadata exposed at that level. But that's the way it should be... if you're working at that level you need all the data to be enumerable through the normal file system operations. At the Finder and application level, you don't see any of that... you don't see the "._" files and you don't see ".rtfd" and ".app" and ".pages" bundles as directories.

Apple's even using these tricks to apparently add features to HFS+: most spotlight metadata, for example, is stored in a regular file, and bundles are still implemented as folders. They even implement aliases in HFS+ on OSX using a real secondary directory tree, presumably because the vnode layer in UNIX won't work any other way!

They've also extended UFS to handle the most critical parts of their dual-fork model inside the file system... but they've stopped short of completing the job. They could do the rest of it, for UFS or for ZFS, if they wanted to... after all, HFS+ on OSX really *is* using these tricks too.

Re:They said the same thing about UFS.

[argent]

Is their support for UFS better now? The last time I used OS X (v10.2) their support for UFS was crap.

It got a lot better in Panther with the import of FreeBSD's latest UFS, though they never implemented even as much metadata support as they fake in foreign network filesystems. They didn't go anywhere with it, either.

Like you I wanted to use UFS to get away from the "PC-quality" HFS+, but no dice... so I'm really dubious about them doing anything for desktop users with ZFS. I suspect it's only going to be useful, if at all, for folks using XServes. :(

Re:They said the same thing about UFS.

[argent]

Are you certain about [the spotlight hooks being in HFS+]?

Pretty certain.

They use the fsevent mechanism in the vnode layer for some of it, and the application-level stuff should work on any file system because that's not actually in the file system... it's in the .Spotlight-V100 subdirectory along with the indexes created by mdimport, but the extensions they created to speed up searching and indexing by tracking metadata in file system reads and writes is apparently all inside HFS itself.

I thought they they'd work off vnode operations too, esecially since the indexes are in a regular file tree, and I speculated that they'd be doing that before release... and got "authoritatively corrected". :)

Re:They said the same thing about UFS.

[Ilgaz]

You can disable Spotlight on a volume with sudo mdutil -i off /Volumes/foo (or simply / for the root volume).

As has been commented above, ZFS already has support for the metadata stuff.

Yes it is really easy to stop a volume getting indexed. The hard part is adding per directories, e.g. /opt/src

I wished I was a developer actually helping the source of those projects but I am not, all I get is a 200mb useless (for me) metadata which makes entire spotlight slow down as a crawl even on a very high speed HD running on a Quad G5 Mac.

I can't trust to stuff I read on web by NDA breaking people but I heard Leopard Spotlight won't have such problems and per Folder settings are really coming (without Privacy trick).

What I need is mdutil -i off /Volumes/foo/SRC . That is what realtime mpeg compressor people (e.g. El Gato EyeTV) needs too.

Re:They said the same thing about UFS.

[Ilgaz]

I understood there could be some real life issues when changing file systems this way:

While I was experimenting with open source tools built from source, I figured the "case sensivity" really matters and the people doing a great job converting them (e.g. Fink) are all getting around these issues.

As I thought one day, I may have to hand-compile a large software from source and my Unix (grep etc) experience is horrible, while formatting my startup as a raid 0 striped, I also set "Case sensitive HFS+"

Now I know nothing from Apple will fail (I bet they work on UFS too) and I am using fairly new software on this workstation, I could do without "ancient" Carbon/OS 9 coded software if they failed.

Nothing from Carbon/OS 9 era failed! All the software having serious issues (one refused to launch from HFSX) are all modern, OS X Frameworks using and even Spotlight metadata using software.

I think Apple is thinking about 3rd party, their own stuff already runs on UFS or (future) ZFS.

An example: The language support scheme. Normally, you should be able to clean or remove each unneeded language from .app's /Resources and they should happily work. If you do this to Adobe Professional Applications, you may create a havoc which may even cost you money in some cases. These are the tools which are developed in modern times, using OS X Frameworks. Adobe simply refuses to use language support as intended on OS X and every single "language cleaner" has special work arounds to stay away from Adobe Professional apps.

I think I know the primary suspect companies which prevents Apple from changing filesystems. Imagine Leopard shipped with ZFS support, every professional switches to it (especially DTP) and some unpredicted horrible failure with Adobe and Quark apps happen. These are the tools which may actually fail when a basic issue with filename text encoding (coming from ancient times) happens.

Re:They said the same thing about UFS.

[argent]

I've been there and got the t-shirt too.

The point is that in the UFS extensions they've already done, and in the vnode layer, they already have the tools to perform all the transitions necessary to make the underlying file system behave the same way whether it's UFS, HFS+, or ZFS underneath.

They have chosen not to do so.

That is, they don't consider the problems in HFS+ sufficient to make it worth the effort of even making it possible to switch away from it.

I've had this bite me three times already, so I hope you understand why I don't agree with them.

Re:They said the same thing about UFS.

[Ilgaz]

I remember seeing a post here on slashdot from one of OS X Desktop core authors, possibly the guy who lead the entire project (Quartz etc.) explaining why they decided some methods, what was the reason for deciding them and why some sort of PDF used on OS X.

If one filesystem/kernel developer posts something like that and explains step by step why HFS+ isn't that "horrible" (as people do raw HD editing on very same fs) and why they can't go with a "plain" filesystem, the whole discussion will end for good.

There are lots of junk data even made into Wikipedia as reference information.

As you probably know, the actual power of HFS+ spec is not yet used even by Apple. They have even had to go back in time and add extensions as explained there:
http://arstechnica.com/reviews/os/macosx-10-4.ars/6

That schizophrenic approach (both resource and extension matters) eventually cost them with that mp3 virus (?) comedy which people should be still glad that it ended up in hands of a security company (one way or another) rather than some really bad guys.

Funny is, MS didn't use the power of NTFS too (it has alternate streams etc.) and yet tried to implement a new filesystem (WinFS) which may have been dropped for same reason.

The promise of ZFS is to be the filesystem which nobody should find any need to add features or enhance so if Apple really decides to give up HFS+, I bet it will be ZFS. The overhead of it is not suitable to current home or even workstation machines though. If flash/magnetic hybrid drives go down to current standard SATA2 drive price levels, overhead wouldn't be issue of course. That time, the redundancy and extra ease of scalability would matter which is provided by ZFS.

Re:They said the same thing about UFS.

[argent]

(1) They wouldn't be going to a "plain filesystem". They've already extended UFS for multiple forks, so the argument that they need HFS+ because a traditional UNIX file system isn't "good enough" is moot.

(2) The fact that people are using HFS+ for demanding tasks doesn't mean that HFS+ is needed for those tasks or even that HFS+ is necessarily well suited to them.

(3) Historically, embedding application level metadata in the file system has had at best mixed results. It's occasionally good for OS manufacturers if applications are dependent on OS-specific features, but it's not necessarily good for the users or developers, and it makes it hard to adapt to new technologies. There's a reason that complex file types have increasingly been replaced by UNIX flat files and "smarts" get moved back to the applications.

So while there may well be technical advantages to staying with HFS+ to maintain application-level metadata in the file system, that doesn't mean it's necessary or that maintaining application-level metadata there is even the right thing to do. Given that the cost of using HFS+ is that the file system is unstable in the face of normal file activity... you don't even need to have hardware problems or hack on the file system below the driver level to trash the catalog... I think that Apple needs to either fix HFS+ properly or abandon it. There is nothing they could write that would justify retaining it in its current form, because there's no justification for ignoring a known flaw that can so easily lead to irrecoverable file system corruption.

Finally, regarding this article, let us say that it's entirely possible for competent people of good will to disagree on issues such as this, and this particular one has proven particularly polarizing.

Re:They said the same thing about UFS.

[Guy Harris]

Pardon my potential ignorance, but I was under the impression that there was nothing you couldn't store in a resource fork that you couldn't also store in a 'normal' extended attribute that file systems like ext3, xfs, reiser, etc. have supported for some time.

In theory, you can store an "arbitrary" mount of data in a resource fork (the Resource Manager might impose a limit, but the underlying file system doesn't). I think at least some file systems that support extended attributes (including HFS+) impose a limit on the size of extended attributes that's lower than the maximum file size. ZFS (and Solaris UFS) "extended attributes", however, are full-blown alternate data streams, and can get as large as a file.

Correcting errors in AppleInsider ZFS article

[kuma]

The AppleInsider article is largely vacuous...

Please do not bother with this debunking (via Macjournals) unless you are truly interested. Thanks.

http://www.macjournals.com/news/2007/10/04.html#a79

Re:Correcting errors in AppleInsider ZFS article

[Sandor at the Zoo]

This wasn't a debunking, it was more of a whiny "AppleInsider gets way more traffic than we do, so we'll dump on what they have to say, even though we don't have anything to add to it."

Despite the macjournals piece, ZFS is cool, and it is better than HFS+journaled, at some things. Deal with it.

Re:Correcting errors in AppleInsider ZFS article

[kuma]

I see you are unfamiliar with the Macjournals profiles of HSF+ and ZFS, which appeared in print, and informed their *debunking* of the AppleInsider piece.

Only if you are SERIOUSLY, honestly interested, refer to the following:
    MWJ_20070611 "the reality of zfs"
    MWJ_20060417 "Uniform type identifiers and their place in Tiger"
    MWJ_20030531 "DiskWarrior explained"
    MWJ_ 20030525 "HFS and HFS Plus Complete"

Macjournals *agreed* with you about ZFS being "cool" and better than HFS+, at some things.

But it is not a better startup volume for current and foreseen versions of Mac OS X.

Please go away, or backup your *despite*... Your sarcasm does not impress me, you shit.

Re:Correcting errors in AppleInsider ZFS article

[Sandor at the Zoo]

But it is not a better startup volume for current and foreseen versions of Mac OS X

Well, duh. Since booting under ZFS is still an unreleased feature on Solaris, let alone Mac OS X, it's not any kind of startup volume.

From the macjournals page (which looks like ass, by the way), pretending to quote AppleInsider:

"We don't find HFS Plus administration to be complex, and we can't tell you what those other things mean, but they sound really cool, and therefore we want them. On the magic unlocked iPhone. For free."

That isn't debunking, that's just whiny dumping. I'm all for informed discussion, which is why I was on the ZFS discussion list for a while, but the macjournals piece doesn't cut it.

Re:Correcting errors in AppleInsider ZFS article

[IvyKing]

I thought the Macjournals article was at least as vacuous as the Apleinisider article. Furthermore, the author of the Macjournal article is obviously not very aware of how file systems interact with hard drives (e.g. soft errors) and not very up on the innards of ZFS.

If HFS or HFS+ are so great, then why isn't there more interest in porting HFS or HFS+ to other OS's such as Linux or the BSD's?

Re:Correcting errors in AppleInsider ZFS article

[Anonymous Coward]

"Your sarcasm does not impress me, you shit."

Wow - that's just what I thought about your whining.

Re:So.... BSD or Solaris???

[abigor]

Even though OSX will still be Unix, will they'll move away from BSD and toward Solaris?

I'm hoping not, since many things behave very oddly on Solaris. Non standard tools and such, but it would be one way to keep it from running on cracked PC's.

2 cents,

QueenB. Please go and stare at this page for a while: http://en.wikipedia.org/wiki/XNU

If by "non-standard tools" you mean non-GNU, yes, but they are hardly odd.

I have no idea what your "cracked PCs" comment is all about, and what it has to do with Solaris and ZFS.

Re:So.... BSD or Solaris???

[Trigun]

killall

How many people have learned that one the hard way?

I maintain:

[teknopurge]

Sun is the new Bell Labs.

Watch for the robotics coming out, very quietly, from Sun in the next 10 years.

Re:I maintain:

[anilg]

Have you seen the SPOTs?

Re:I maintain:

[cain]

There will be robots coming out of Sun?!?! Holy crap, will they look like Arnold? Will they use us for batteries? Everyone stay away from teh stairs!!! We should nuke Sun from orbit. It's the only way to be, um, pretty sure...

Re:I maintain:

[tholomyes]

Robots coming out of the sun?! I think that nukes will only fuel that thing, we need a plan B!

Re:I maintain:

[cain]

Crap! You're totally right. What about some kinda freeze ray?!?!

Re:I maintain:

[OriginalArlen]

Thanks! I'll take two raspberry flavour ones, and a Mivvi please.

Re:I maintain:

[IvyKing]

Robots coming out of the sun?! I think that nukes will only fuel that thing, we need a plan B!

Not Plan 9?

Re:I maintain:

[moosesocks]

Not even remotely valid.

Bell Labs had a near infinite source of money, and concentrated much of its efforts on hardware development, as well as pushing the envelope on general science.

Sun manufactures nice servers (increasingly using off-the-shelf components), and writes software. I'm also not going to forgive them for Java.

Re:So.... BSD or Solaris???

[Hijacked Public]

I have been playing around with ZFS on FreeBSD since the middle of this year or so.

I wonder if I should be concerned that FreeBSD is moving toward Solaris and away from FreeBSD.

Non-Standard my ass!

[kaiwai]

I'm hoping not, since many things behave very oddly on Solaris. Non standard tools and such, but it would be one way to keep it from running on cracked PC's.

What are you smokeing - what ever it is, pass it this way. Non-standard or 'does not conform to the bastardised standards which GNU have embraced and extended'. Case in point, look at the number of nimrods who assume gnu grep and use gnu specific switches for their make scripts.

It isn't Solaris that it is non-standard, it is those who insist on using GNU tools and their extensions to the standard which are the non-standard.

Re:Non-Standard my ass!

[ArsonSmith]

Hmm,

Solaris tools - specific to Solaris, good interpretation of Posix standard, and mostly consistent across releases

GNU tools - ubiquitous across nearly all OSs, good interpretation of the Posix standard plus usability extensions, and nearly identical across all releases and platforms

Re:Non-Standard my ass!

[Anonymous Coward]

Have you tried a recent Solaris version, a lot of GNU tools extentions seem to be in solaris 10.
Although the GNU tool's are not always the published "Standard", their tools do set the usability "standard", but I think you'll find that the good extentions are incorporated into solaris in good time ( e.g. to think of one item I've noticed in solaris 10 ( the -h option to du and df )

A merge of MacOS Apple and Solaris Sun would make perfect sense.
A) they have complementary customers ( enterprise / servers/data center) and ( home/smaller company / desktops ).
B) they both have a similiar focus on quality ahead of cost.
C) they both derive a certain amount of their strength from their ability to control from top to bottom of the stack ( hardware, Operating System , Software )

Re:Non-Standard my ass!

[rsidd]

look at the number of nimrods who assume gnu grep and use gnu specific switches for their make scripts.

Or, for that matter, assume that /bin/sh means /bin/bash. (Which causes their scripts to break on Ubuntu too, since Ubuntu uses dash for /bin/sh.)

Re:Non-Standard my ass!

[Sentry21]

Hear hear. I'm getting more than a little tired of Linux migrants who complain over and over that 'everything's different on OS X' or 'why did Apple have to change everything', when OS X is more UNIXy than Linux and its GNU userspace ever was. Don't get me wrong, I grew up on Debian and the GNU userspace, but at least I'm aware that it's not the original gold standard of compliance.

Re:Non-Standard my ass!

[GPL Apostate]

Case in point, look at the number of nimrods who assume gnu grep and use gnu specific switches for their make scripts.

That's no surprise. The GNU Project competes with Microsoft in the 'Embrace/Extend/Extinguish' derby.

Re:Non-Standard my ass!

[Jeremi]

The GNU Project competes with Microsoft in the 'Embrace/Extend/Extinguish' derby

Who has the GNU Project extinguished so far?

Re:Non-Standard my ass!

[ashayh]

Case in point, look at the number of nimrods who assume gnu grep and use gnu specific switches for their make scripts.

It is wrong to assume GNU grep is always present, but Solaris and Aix do go out of their way to provide official support for GNU utilities. This is obviously better for them than to retrofit GNU capabilities to their utils. I think, therefore GNU should be made the standard on all UNIX. Even if GNU utils are not present, any Sysadmin should use "uname" to test the OS type.

SUN/IBM provide official GNU packages because GNU IS better. Do you think the sh/ksh that comes with Sun/IBM is better than bash or pdksh? For scripting, GNU greps' -C, -A, -B , -P switches are so much more useful. GNU finds' "-iname" switch is pure gold.

Re:Non-Standard my ass!

[GPL Apostate]

Virtually all of the proprietary Unixes.

Re:Non-Standard my ass!

[WinterSolstice]

GNU utilities are frequently superior to the Solaris ones, but I'll keep my smitty, TYVM :)

AIX is still the best Unix IMHO. Best tools, best support. Linux is also very nice. Years of supporting Solaris (99-2007) have taught me nothing but hatred for it. Most of those years were also spent supporting AIX, which I found was better in almost every respect.

Re:Non-Standard my ass!

[ashayh]

I agree. My new job is teaching me AIX and I have found it impressive. smit/smitty is quite cool.(Apart from some weirdness for me in older smitty versions) Nothing quite that comprehensive, consistent and simple exists in the Linux world.

Translation:

[Troy Baer]

"We don't have an LVM layer to speak of, so we're going to build it into the file system."

There are a lot of things to like about ZFS. The built-in LVM isn't one of them IMHO, but I can see where it might be attractive if either you don't already have an LVM subsystem or your existing LVM subsystem is complete crap.

Re:Translation:

[lauwersw]

Way easier to manage: only 2 commands! While now with an LVM you have to place your disks in the desired topology inside your LVM (RAID0, 1, 5, ...), format them, put a filesystem on, mount, file check, repair, whatever. With zfs you place disks in your pool and kinda mount part of it, that's it.

There are some other things you could complain about: it makes less sense on hardware RAIDs with good management tools. They missed a chance to make it a distributed or clusterable file system (though they bought Lustre lately, who knows) and it's not possible to boot from it yet, but all in all it's a major step forward.

Bad Technology Journalism

[StCredZero]

eventually succeed HFS+ as the default operating system for Mac OS X

It would appear that Appleinsider's writers can't tell the difference between an Operating System and a File System. Bad Science Journalism is about the same thing as Bad Information Technology Journalism. Is one more widespread than the other? If Science Journalism is worse, then I wonder what we do differently with Tech jornalism that we can apply there.

I once worked for a company where the CTO couldn't tell the difference between NNTP and HTTP. (He clicked on a link that brought up Usenet in Outlook Express, and was up in arms about how so many negative comments were "on our website!")

Re:Bad Technology Journalism

[aliquis]

Or they just thought about/typed the wrong word since they are atleast sort of related. I wonder what is more likely? ...

Re:Bad Technology Journalism

[StCredZero]

Either way, it's there in print, wrong.

Not so. ZFS could handle resources

[Henriok]

Hardly! ZFS have provisions for any number of "forks" in the file system, called "extended attributes" in ZFS. If Apple migrates to ZFS they have every chanse to use these attributes to provide for quite a seamless integration with previous filsystems. The file system is open source and Apple can prettymuch do what they like or need. Even NTFS have these features but MS seems to ignore them due to backwards compatability issues with FAT filsystems and Windows APIs

You know.. Wikipedia is very handy to look these things up. Please do. http://en.wikipedia.org/wiki/ZFS

Re:Not so. ZFS could handle resources

[Trailer Trash]

You know.. Wikipedia is very handy to look these things up.

Dude, we're still trying to get people to read the linked article. Let's not get too crazy.

Re:Not so. ZFS could handle resources

[Anonymous Coward]

then I question why they don't already use xattrs in 10.4. See this to see how you can use them with the present OS.

Re:Not so. ZFS could handle resources

[EvanED]

Even NTFS have these features but MS seems to ignore them due to backwards compatability issues with FAT filsystems and Windows APIs
They are starting to do some stuff with them. The first major use I know of was with XP SP2. With that, when you downloaded a file from the internet, IE would mark it as such in an alternate stream. When the program was run, someone (I don't know who) would check for the presence of the stream and if it was there, would display a "this program came from an untrusted source, would you like to run it?" dialog.

I would expect more uses as we move into the future, as Vista is pushing even heavier for NTFS (for instance, IIRC the installer didn't ask which file system I wanted to use and just formatted NTFS), and MS doesn't have to worry about, for instance, some 98 or ME user who upgraded to XP but is still running FAT so he didn't have to reformat. For my large partitions (~100 GB), I can't format as anything but NTFS. (I don't know about smaller ones; I have a 24 GB system partition but if I try to bring up the format dialog there it complains that I'm trying to reformat the drive with the OS and I don't want to do that.)

Personally, I think that there's a lot of awesome stuff that you could use extended attributes and alternate streams (WHY are these separate concepts on some file systems?!) if only they would be preserved when you move stuff around systems, upload them, etc., and am somewhat resentful at Unix and POSIX for the fact that for ages they didn't do this stuff and hence it's really hard to move to using them because no one supports them because there's no demand because people haven't thought of what to do with them because they haven't seen what can be done with them because no one uses them because no one supports them because... :-) I've often wondered what operating systems would be like if we kept the knowledge of the last decades, but threw out everything that we had now and started from scratch without worry of backwards compatibility, and this is one of the things I would like to see change.

Re:Not so. ZFS could handle resources

[Henriok]

I think you fint the answer in that same article. It's being field tested and is not incorporated in the higher level APIs like Cocoa or Carbon. Yet. Apple isn't an army of developers that can implement every possible feature with no effort. Be patient, young grasshopper.

Re:So.... BSD or Solaris???

[Deagol]

So how goes the evaluation? I've been patiently waiting for the 7.0 release (it's killing me) so I can get my hands on ZFS. Kind of ironic that you're worried about BSD moving towards Solaris, as Solaris was preceded by SunOS which was based on BSD.

ZFS is still missing 1 very important feature

[Anonymous Coward]

Sure thing, ZFS supports huge storage, easy administration and the ability to add more data sets into the storage pool to easily increase available storage.

I was amazed to discover though that ZFS can't increase the size of a RAID5 or 6 dataset. Given the ability to dynamically add storage is various other ways it is extraordinary that something as common as resizing RAID5 is missing.

Re:ZFS is still missing 1 very important feature

[quantum bit]

Are you talking about adding drives to the pool?

I know you can definitely expand it with the whole replace one disk with a bigger one, wait for rebuild, repeat until all your disks are bigger trick.

Re:ZFS is still missing 1 very important feature

[Sancho]

Most RAID controllers let you add a disk to a RAID5, and the controller takes care of restriping so that you can use the new disk space. ZFS does not support this kind of function in its raidz. The only way to increase storage of a raidz is to upgrade each disk, one at a time.

This is probably the single biggest feature request for ZFS.

HFS+ as the default operating system for Mac OS X

[theNetImp]

Uh folks, I hate to break it to you but ZFS is a FILE SYSTEM not an OPERATING SYSTEM....

I am sure Apple isn't going to replace the core of their OS with Solaris. ZFS may be a better choice the HFS+ I don't know a lot about ZFS, but if it is better I will have no problem migrating my stuff over to it.

Re:So.... BSD or Solaris???

[OddlyMoving]

I wonder which one you feel is nonstandard.

Re:So.... BSD or Solaris???

[porkchop_d_clown]

What's so bad about killall?

It's good enough for Linux but not good enough for you?

built in LVM == Win

[mikeee]

Actually, a built-in LVM makes a lot of sense if you stop to think about it; many of the things a LVM does could benefit from information only the filesystem has.

How to post a new comment

[henrikba]

Click "Reply" in the "n Comments" box at the left hand side. Not extremely intuitive, but it works as advertised.

Re:So.... BSD or Solaris???

[Ilgaz]

Does FreeBSD's implementation of DTrace makes it away from BSD? No, that tool is very critical and useful for their needs/focus and they implement it just like Apple planning sort of ZFS support for specific needs.

folders are even worse

[SuperBanana]

Resource forks are far better than the idiotic "everything is a folder" model.

Want to upload that Keynote project to your friendly CMS via a web browser? Can't, because it's not a file, it's a #@$!ing FOLDER. You have to zip it first. Words cannot accurately describe how tiresome this becomes.

It also makes data recovery (should the file get accidentally deleted) nearly impossible- the files inside the folder are not named uniquely or in any identifiable manner.

ZFS isn't nearly all it is cracked up to be- among other things, you can't expand RAID-Z...absolutely moronic. I'm not even sure you can expand a simple mirrored pool. Users have been repeatedly asking for growing abilities, and the developer reaction was "just create a larger pool and move it over". That's hilariously stupid advice given that you usually don't have that kind of storage hanging around- not even in enterprise environments.

There's simply no comprehension amongst the ZFS developers that virtually EVERY raid card on the market supports such an operation. Even more shocking was when one developer said (paraphrasing) "gosh, how would one even go about doing that sort of thing?"

Don't get me wrong- checksumming and automatic disk scrubbing are features long overdue, but ZFS is not magic bullet.

Re:folders are even worse

[kithrup]

It's true that you can't expand a RAID-Z set (I think, anyway -- if you replace all of the drives, one at a time, does that work?), but you can add another RAID-Z set, and expand the pool.

That's the big thing in ZFS, combining all of the resources into a pool, rather than treating disks (or groups of disks) as part of a volume. The other part of this was making filesystems nearly as light-weight as directories.

My plan is to use twinned drives, adding them as a mirror to the pool. I can replace each drive individually, let it re-silver, and then do the same with the other, to expand it, or I can simply add another pair of drives to the pool, and get more space that way. There are advantages and disadvantages to each.

Oh, as for resource forks -- the model that Sun is choosing (as are some others) is that the extended attributes are treated as sub-files to a directory. I'm not sure that simply going to a directory is not a better idea, but that has a whole slew of its own problems. It's a bit ironic, really -- Apple had an idea from the beginning, and every application was prepared to deal with it, but nobody else did the same thing. Then, when Apple went with the flow, everyone else started trying to do what Apple did... and none of the applications are prepared for it.

I'm not sure how it'll all turn out.

Re:folders are even worse

[EvanED]

Resource forks are far better than the idiotic "everything is a folder" model.

Want to upload that Keynote project to your friendly CMS via a web browser? Can't, because it's not a file, it's a #@$!ing FOLDER. You have to zip it first

Personally, I'm not a MacOS user so don't know excatly what you're talking about, but from what I've heard about this I think the "everything is a folder" model is a really good idea. I said in another post a couple up that I wish that we were free to innovate without worrying about backwards compatibility and that one of the things I would like to see change is better metadata support, and this is at least very similar to the best model I've come up with for how to handle it. (Reiser4 does something fairly similiar.)

It just sounds like it's implemented wrong on MacOS. For instance, you could have "file-folder aware" applications and "file-folder unaware" applications, distinguished by the API they use or a flag they pass to open or something like that, and if your web browser doesn't explicitly say "I want the folder view" it will automatically zip it during the read calls. Keynote can then use open_folder and get the contents as a directory.

(Of course, there may be consequences of this that I haven't thought of.)

Re:folders are even worse

[EvanED]

Since you seem to know something about ZFS, here's a question:

One of the issues with extended attributes is that they are easily destroyed on a Unixish system. (This isn't a problem on Windows perhaps somewhat by accident and through a big hack that happens to help a lot.) If you modify a file and save it in, for instance, either Vi or Emacs, at least under ext3 it will kill any extended attributes associated with the file.

The reason it does this is because the system calls made by Emacs are something along the lines of "move(foo.txt, foo.txt~)", "open(foo.txt, O_CREAT)". In other words, after you save foo.txt, the foo.txt that now exists is a /different/ foo.txt from what existed before.

Does ZFS do anything to address this problem?

(FWIW, on Windows, if you make a series of API calls similar to that so a new file is created with the same name as one that was just deleted, if it happens within 15 seconds the metadata is kept from the old file. This was essential in the land of Win95's long file names back when people still used DOS programs. Imagine editing "c:\some file with long name.txt" in a non-LFN-aware application, saving it, and now having your file called "c:\somefi~1.txt". Ewww! So they needed a way to keep the file name past that save. I said it was a hack. ;-))

Re:folders are even worse

[g0at]

Resource forks are far better than the idiotic "everything is a folder" model.
Want to upload that Keynote project to your friendly CMS via a web browser? Can't, because it's not a file, it's a #@$!ing FOLDER. You have to zip it first. Words cannot accurately describe how tiresome this becomes.

True, but you'd have had the same problem with a multi-fork file, since the web process would upload the data fork... HTTP forms don't know anything about multi-fork files.

Better would be for the web browser to auto-zip the file or something prior to uploading (or have the OS present the bundle [i.e. folder] as an opaque pre-zipped file to the browser).

b

Re:folders are even worse

[nine-times]

It just sounds like it's implemented wrong on MacOS. For instance, you could have "file-folder aware" applications and "file-folder unaware" applications, distinguished by the API they use or a flag they pass to open or something like that, and if your web browser doesn't explicitly say "I want the folder view" it will automatically zip it during the read calls.

It kind of works that way within OSX and treats them as files unless you explicitly try to look at the contents. However, it's not smart enough to zip the files automatically when dealing with systems that aren't aware of the arrangement. For example, if you upload the "file" to a Windows file server, it gets uploaded as a directory (IIRC). Also, I think it will sometimes treat it as a folder unless you have to correct viewing application already installed.

Personally, I'm not sure why OSX doesn't use tar files or zip files (with or without out compression, depending on file type) to contain these "files" by default. You could still do some voodoo in the OS to allow users to browse these packages like folders, but at least their default state would still look like a file to all operating systems.

Admittedly, I'm no expert here, so forgive me if there's something naive about the idea.

Re:folders are even worse

[Anonymous Coward]

ZFS isn't nearly all it is cracked up to be- among other things, you can't expand RAID-Z...

Yet.

ZFS is not perfect, but it is a relatively new system and thus a work in progress that is being improved, e.g., the first alpha code for built-in encryption was released recently.

Re:folders are even worse

[Jeremi]

Personally, I'm not sure why OSX doesn't use tar files or zip files (with or without out compression, depending on file type) to contain these "files" by default. You could still do some voodoo in the OS to allow users to browse these packages like folders, but at least their default state would still look like a file to all operating systems.

Admittedly, I'm no expert here, so forgive me if there's something naive about the idea.

I think it's a good idea... however the likely reason why they don't do it that way is that it would make adding/removing data from the "archive" very inefficient. i.e. instead of just writing another hidden file into the Package folder, you'd have to rewrite most or all of the .tar file every time you made a change.

Re:folders are even worse

[SuperBanana]

It's true that you can't expand a RAID-Z set (I think, anyway -- if you replace all of the drives, one at a time, does that work?), but you can add another RAID-Z set, and expand the pool. That's the big thing in ZFS, combining all of the resources into a pool, rather than treating disks (or groups of disks) as part of a volume. The other part of this was making filesystems nearly as light-weight as directories.

I understand all this, and don't care.

If I have a 10x500GB drive RAID-Z pool in a twelve drive chassis and I need to add roughly another 1TB of storage, I'm fucked unless I go and buy 1TB drives, and I just wasted ~$400 (or whatever they cost these days) because I had to add them as a mirror to the pool instead of restriping the array.

Same problem if I want to upgrade those drives to 750's. Not a problem with 100% of the raid arrays and raid controllers on the market. Impossible with ZFS and RAIDZ.

If RAIDZ supported re-striping and odd disk sizes (ie, yank a 500, insert a 750 and bam, get the extra space), it would be hands down perfect.

Re:So.... BSD or Solaris???

[Anonymous Coward]

Only Linux weenies get burned by that one.

Real Unix admins just get harmlessly pissed off that pkill results in command not found whenever they use a toy Unix.

Expandable storage

[kimble3]

I doubt that Apple is going to switch the default file system to ZFS anytime soon, but one situation where I think it might be very useful right away is in the Apple TV or possibly iPods. A lot of people were dissapointed with the small size of the disk when the Apple TV was released and it did have that mysterious USB port on the back. Could ZFS be used to make plugging disks into Apple TVs easier? Just curious...

Re:Expandable storage

[Wesley Felter]

Could ZFS be used to make plugging disks into Apple TVs easier?

No, I think ZFS would make that use case unreliable, since unplugging the external disk would fault the entire pool (aka you lose access to all your data until you plug the external disk back in).

Re:Expandable storage

[Sancho]

ZFS is REALLY memory intensive. I highly doubt that current AppleTV hardware will ever support it.

Re:So.... BSD or Solaris???

[MightyYar]

Go here if you need this tools... at least it works on Mac.

Don't die. 7.0 is fine for non-critical uses.

[Kartoffel]

In my experiences, anyone who runs FreeBSD for srs business also has a few machines kicking around for testing and development. Why not unleash 7.0 on one of those right now? HEAD is currently frozen (and about as stable as HEAD could ever be), pending official creation of RELENG_7.

Evolve or die

[apachetoolbox]

The GNU tools can also be seen as the next evolutionary step.

Re:Evolve or die

[jhol13]

I am certain you would not say that when Microsoft "extends" something.

Re:So.... BSD or Solaris???

[memfrob]

Uh, Mac OS X is certified standard UNIX.

According to the Single Unix Standard, only Mac OS X 10.5 (Leopard) can be considered "Unix". And only when deployed on Intel-based Macs. Previous versions must be considered like Linux: "Unix-like".

FWIW, Sun's operating system (SunOS) has been fairly close to Unix standards over its lifetime. In fact, the official version of System V release 4 was written by Sun and called SunOS 5, integrated into Solaris 2

Why is anyone even having this argument? GNU means "Gnu's NOT Unix" for a reason...

Just wondering about the implications...

[bennomatic]

My question is, why ZFS for the Mac? I mean, for 99% of people's uses, it seems like the most enticing features of ZFS are overkill, unless implementing it does not imply any load on the system if all the features are not being used, and they want to synch up FS development between all of their products, from iPod to XServe.

That being said, they may have something up their sleeves, and forgive me if the connection between ZFS and my idea is tenuous. If it seems like a silly idea, I blame the overdose of coffee I had this morning.

My understanding is that one of the features of ZFS is effectively infinite virtual device size, spread over effectively infinite numbers of physical volumes in a RAID configuration.

Since the introduction of iTMS, especially with TV and movies, Apple is now very much in the business of pushing bits, and the costs of that bit-pushing grow--maybe not linearly, but they do grow--as demand for those bits grows.

People have been suggesting that Apple might be building some sort of BitTorrent client to facilitate distribution of content, and I'm thinking that ZFS might be a key to this.

Perhaps--and this is where my understanding of the technology may be leading me down the wrong path--they could build some sort of ZFS hooks into iTunes such that, if the user chooses to do so, they could mount their purchased library as a network-shared ZFS partition and register an IP address and port with an Apple server. If someone wants to buy a TV show that 10 people have already bought, they get a magic read-only volume mounted which is effectively a network-mounted RAID1 partition striped across those 10 drives, with access only to the TV show in question.

The iTunes hosts which are providing the data shake hands and agree on some sort of wrapper that is provided by the Apple servers, and encode their data appropriately. The buyer then gets their content with minimal data flow from Apple's infrastructure.

To provide incentive for people to do this, perhaps Apple offers lower-cost or even free content to regular bandwidth contributors.

Is this feasible, or even a likely path that they would be thinking of with ZFS, or am I just on crack?

Re:Just wondering about the implications...

[rahvin112]

Ask those large gentlemen in uniforms breaking down your door.

Re:Just wondering about the implications...

[bennomatic]

OK, are these RIAA/MPAA uniforms, FBI uniforms, or Looney-Tooney-land uniforms?

Re:So.... BSD or Solaris???

[aliquis]

He's arguing that it would be much harder to run a cracked OS X on a regular PC if it ran more Solaris userland (?) than BSD. Thought noone on the planet can understand why that would be the case. Also why would they switch other things from BSD to Solaris just because they add support for another filesystem?

The whole post is just weird, I wonder how it even got +2 =P

Re:So.... BSD or Solaris???

[memfrob]

What's so bad about killall?

You should get out more: (from the Linux manpage)

Be warned that typing killall name may not have the desired effect on non-Linux systems, especially when done by a privileged user

...and from the Solaris manpage:

killall is used by shutdown(1M) to kill all active processes not directly related to the shutdown procedure

...which might explain why many linux distributions have begun including the solaris-like "pgrep" and "pkill" commands...

You left out D...

[swb]

D) They're both run by megalomaniacal zealots.

I thought that Apple and SGI should have merged at one time. It no longer makes sense, but at the time (say, 2000 or before) both were clear leaders in graphics and visualization. It would have been very interesting to have a common software platform from true desktop to true datacenter.

The party is kind of over now, as Apple has decided they are a consumer electronics company and not a computer company and Sun is less interested than they used to be in the desktop-type workstation market (at least from a hardware perspective, Java kind of keeps them there in a software basis).

Aliases

[pizzach]

I wonder how Mac OS X aliases will function under ZFS. With HFS no matter where you move a file or its alias, they will be correctly linked. This is because the filesystem supposedly stores everything with an abstracted unique name. Will this be the end of Mac OS style aliases?

Re:Aliases

[kelnos]

You can do something very similar using hard links on just about any unix/linux filesystem. Though I know that doesn't work across devices -- you can only make a hard link to a file (actually, an inode) if it's on the same filesystem. I'm not sure if the same is true for MacOS aliases. The cool thing about hard links is that, after you create one, you essentially have two 'files' that point to exactly the same data on the disk, and you can't even tell them apart afterwards. One isn't an alias of the other, they're just the same file with different names.

Though, really you could probably implement cross-device hard links (in a non-portable way) by storing both the target device name and inode number in the file entry. Or maybe some kind of UUID assigned to the device to protect against device names changing due to moving the disks around on the cable or whatever.

Re:Aliases

[pizzach]

Looking up my man pages, a Mac OS X alias appears to be somewhere between a hard link and a soft link. Yes, an alias links directly to the file like a hard link, but it is _not_ actually the file. If you delete the orignal file, the alias will point nowhere and error. My guess is that with a change ZFS, Apple will exchange aliases for soft links.

Re:Aliases

[Anonymous Coward]

In ZFS a hard link works across the entire zpool.

oh, puhleeeze

[m2943]

Even though OSX will still be Unix, will they'll move away from BSD and toward Solaris?

OS X is a heavily hacked Mach kernel with a bit of BSD code thrown in. Its architecture and codebase are completely different from UNIX. So, apart from a bit of UNIX compatibility and a lot of marketing hype, OS X is not UNIX.

Will they "move towards Solaris"? I have no idea what that even would mean.

even then

[m2943]

According to the Single Unix Standard, only Mac OS X 10.5 (Leopard) can be considered "Unix".

Even conforming to the standard means that it is "UNIX" only in one sense; in terms of its internal architecture, OS X is still completely different from a traditional UNIX.

Re:even then

[Anonymous Coward]

"Flamebait"? These Macintosh zealots are really going completely nuts.

The 1984 commercial was right, except it's Macintosh users sitting in the audience. Time to break their screen.

Re:even then

[Guy Harris]

Even conforming to the standard means that it is "UNIX" only in one sense; in terms of its internal architecture, OS X is still completely different from a traditional UNIX.

So what's the internal structure of a "traditional UNIX"? The internal structure of V7? If so, most if not all UNIXes and UNIX-like systems are "completely different" from a traditional UNIX.

Re:So.... BSD or Solaris???

[isj]

FWIW, Sun's operating system (SunOS) has been fairly close to Unix standards over its lifetime

It is still a long way from following IEEE Std 1003.1
Solaris still defaults to non-compliant tools (/usr/xpg4/bin is not in PATH, /usr/bin/sh points to a non-compliant shell, etc). I understand Sun's reason for doing it that way (don't make old customers afraid of upgrades), but it is a pain when trying to make cross-platform shell scripts.

Pure BS

[LKM]

Like most Appleinsider articles, this one is pure bullshit. Macjournals.com takes it apart. Choice quote:

"(...) its battery-chomping, disk-eating storage hog nature that makes it fantastic for 20TB disk arrays and entirely, completely unsuitable for a Mac OS X startup disk, now or in the foreseeable future."

Re:Pure BS

[Anonymous Coward]

My favorite quote from macjournals response is this one:

"... but can only correct them if you're using ZFS's RAID-like capability to store duplicate copies of information, in which case a standard RAID system could fix the error too. Oops."

I love it because it is factually false. No "standard RAID system" (1, 5, or 6) can recover bit-wise errors. This is because there is not enough data to determine which data is "good" and which is "bad." the RAID functions of ZFS add enough information to determine this, and therefor support something that standard RAIDs do not. Standards RAIDs only support recovery of completely failed drive, which is not always the failure case, then again if the guys at Macjournals knew anything about RAID, they would know this.

*I* am using zfs, and am quite pleased with the data integrity it provides, which is far better then my experiences with RAID. Yes, this comes at a price, but how much is your data worth? My data (and time in backup-restores) is worth the additional processing power. Don't get me wrong, I don't know or care if Apple is going to start using ZFS. It's not like it would be the first time they made a decision because Steve Jobs was throwing a temper-tantrum.

Frankly I'm tired of the lack of professionalism from the head of companies in tech, and Jobs is almost as bad as Balmer.

Oninoshiko

Re:Pure BS

[LKM]

There are RAID systems whch do error correction. RAID-2 can correct one-bit errors, I think, although I don't think anyone uses it anymore. So yeah, the article is at least somewhat incorrect in that area.

ZFS is great for many situations, but it's no good as a default OS X file system, and Apple knows it. For example, it could not be used as the basis of Time Machine because it would use way too much disk space. ZFS is great. It's just not magic, as some people seem to think

I find it amusing that my post got rated Flamebait, by the way. I generally tend to avoid the "fanboy" defense, but I must say there seem to be a lot of ZFS fanboys here who kind of try to avoid reality.

Re:Pure BS

[Anonymous Coward]

Your post was modded flaimbait because it linked to a "ha ha you suck" article with only a tiny bit of content and no references to back up the claims. That's pretty much the definition of flaimbait.

Re:Pure BS

[LKM]

You could read it that way, if you had no clue about ZFS. The claims which aren't backed up in the blog post are backed up in an exhaustive article MacJournals has written on ZFS a few weeks ago, although I don't think that article is available anywhere for free.

So what you're basically saying is that the post got moderated as Flamebait because moderators who have no clue about ZFS were to lazy to inform themselves about it and preferred to just moderate the post as Flamebait.

Re:Pure BS

[Anonymous Coward]

I lost interest in what Apple does when stopped working for a school district, so I'll acknowledge my ignorance of how Time Machine works. I believe it allows you to restore files to previous states. (correct me if I'm wrong). This should be fairly trivial to implement using snapshots and clones. Which ZFS does quite efficiently.

ZFS is a vary good File System, and its not magic, its a trade-off. I would probably not recommend it for actively accessed files by Adobe products (admittedly something that seems to be common amongst Macintosh users) for performance reasons (although thats only speculation based on what I've read on how Adobe's products (Photoshop in particular) use the drive). I also would probably not put it on any machine doing processor-bound tasks (seriously, how many tasks are processor bound right now?).

But yes, as a general purpose FS, I do think it is a solid choice. If you happen to be preforming one of the tasks it is not good at, you should know it, and configure your machine accordingly. (NOTE: these situations are all things I would expect only from fairly advanced users, not from my mom)

You are correct on RAID-2, but most implementations required (note the past tense here) a large number of spindles (14-32). If anyone was wondering, Apple's Xserve RAID *just* meets that at 14. I would hardly call RAID-2 remotely close to standard RAID, not to mention these things were expensive and performance probably would be no better then ZFS.

Oninoshiko

Re:What about NTFS?

[Anonymous Coward]

So, Apple is supposed to start supporting a file system which nobody but some research geeks use

ACCORDING TO A RUMOR SITE.

Let me say that again.

ACCORDING TO A RUMOR SITE.

If you take their word for it that Apple is actually planning to make ZFS a big deal for the everyday user of MacOS X, you are a fool. They might, they might not, but it's just a rumor. Mac rumor websites have been notorious for inventing things on slow news days.

- but they STILL refuse to fully support NTFS, which half of the disk drives on the planet are using. What kind of sense does that make?

Are you stupid?

No really, are you?

Have you not noticed by now that full read/write support for NTFS has been a very painful thing for all non-Microsoft operating systems? Even when write support is available, it's typically a relatively recent addition, and also typically slow. There's a reason for this: NTFS is a very complicated filesystem, and there's no reference implementation available.

ZFS, on the other hand... you can download Sun's own code. It is being added to other operating systems such as Linux and MacOS X by porting Sun's own code. It should not be a surprise that full functionality will eventually be available.

(For those living in a closet: MacOSX supports reading NTFS, but not writing).

If Apple would stop being so dysfunctional and start to play nicely with others, maybe it would get invited to more offices.

Maybe you'd have a chance at being more convincing if you didn't try to spin it as Apple REFUSING to support NTFS write. Where is it they've said they don't want to do it? What's that, you don't know? Of course you don't, because they never said it.

Apple originally didn't support NTFS at all. Then they added read-only support (a typical first step in writing a complex FS driver, see: the history of Linux NTFS support) and you interpret that as them deliberately denying you functionality and refusing to play nice with others? Good lord, what a tool.

Re:So.... BSD or Solaris???

[rsidd]

First of all, the comparison would be between BSD and SysV, not BSD and Solaris. Second, ZFS has nothing to do with either SysV or BSD. Third, Mac OS X is not the first BSD-based OS to include ZFS -- FreeBSD has already done so.

Extended metadata and the UNIX programmer

[kithrup]

No, and that's not just due to ZFS -- that's any filesystem with EAs. rename() works, in that the EAs stay with the file, but any other method for moving or copying files around needs to be modified.

Mac OS X has a couple of different ways of handling this, based on the API set you're using. Carbon has their own method, that I don't know too much about; the POSIXy API has copyfile(), which allows you to copy data, extended metadata, or security information. (There's a man page for it in the Leopard seeds.)

The version of vim that comes with it seems to copy the extended data; I don't know if emacs has been modified to do so.

It's a sad fact, but UNIX tended to be so low-level that there weren't any routines provided to copy files around (well, you could always do system("cp src dst"); and that'll still work 8-)), and as soon as filesystems started presenting more data, that became a problem.

In addition to using a new system-provided API (as Mac OS X does with copyfile()), the system can also present the objects as directories; this works for letting tar et al copy them, but it then means that anything that examines a filesystem object type before manipulating it won't let you read or write it.

Re:So.... BSD or Solaris???

[Kymermosst]

What exactly do you find "non-standard" about the tools included with Solaris (and which standards are we talking about)? Or did you really mean "non-GNU" tools?

Re:What about NTFS?

[Anonymous Coward]

http://en.wikipedia.org/wiki/Ntfs-3g

Re:So.... BSD or Solaris???

[phliar]

FWIW, Sun's operating system (SunOS) has been fairly close to Unix standards over its lifetime. In fact, the official version of System V release 4 was written by Sun and called SunOS 5, integrated into Solaris 2

Not exactly, all Sun operating systems are called SunOS. SunOS 4.1 == Solaris 1 and SunOS 5.x == Solaris 2.x. (Of course then Sun decided they needed some big numbers -- MacOS was up there in the 9s and 10s, and Windows was at 2000! Solaris 2.8 was renamed Solaris 8 and voila! instant credibility!)

When most people say "SunOS" they usually mean SunOS 4 which is BSD -- from a user's standpoint SunOS 4.1.3 on a Sun workstation was pretty much the same as BSD 4.3 on a VAX.

It's good...

[silverdr]

... that MacOS X will receive a better default operating system! Odds are good that it'll be Vista not ZFS though...

Barbie says "Proofreading is hard!"

[jbn-o]

It depends on whom one listens to; the term "operating system" is also confused with kernel when some refer to a "Linux operating system" despite that Linux has always been an important part of a complete operating system but not the entirety of it. It's not much of a stretch to understand how one could confuse a file system and an operating system, at least if one has no clear idea what the two are to begin with. Heaven forbid if another major contributor gets a share of the credit for the complete system.

UFS has an FSCK that really works

[argent]

True, but the capabilities of UFS don't really exceed HFS+

In one way, at least, UFS is far better than HFS+.

The internal redundancy in UFS means that so long as the basic file system structures (directories, inodes, and indirect blocks) are intact, it can be repaired. The idea of having file system damage in a bootable file system that can't be repaired by FSCK is all but inconceivable for UFS or any of its precursor file systems. In nearly 30 years working with UNIX, once FSCK was introduced I *never* had a file system so damaged that FSCK couldn't completely restore the structure to working order. Three times now I've had HFS+ file systems require a backup and restore because of some obscure damage that even rebuilding the catalog wouldn't fix. A friend of mine is currently booting his Mac Pro off the second drive because the original installed file system was trashed.

ZFS claims "you'll never have to fsck again". That's what every journalled file system proponent says. That's what they said about XFS... until they came back with tools to do repair and you still had to reinstall to recover sometimes. I'll believe it when I've seen it in practice for a decade or so. What does ZFS do when it hits unrecoverable data in the file system structure itself?

UFS's fsck deals with it by rebuilding the file system structures so that they're valid, and tells you what you lost.

HFS+ tells you that you have some obscure catalog problem and you go out and buy DiskWarrior and hope your backups are in good shape.

XFS apparently gives you a chance to do a final backup.

What does ZFS do? The write-ups on ZFS indicate that they stop short of testing that case, and that's the most important one.

Re:UFS has an FSCK that really works

[Anonymous Coward]

The internal redundancy in UFS means that so long as the basic file system structures (directories, inodes, and indirect blocks) are intact, it can be repaired. This has nothing to do with 'internal redundancy', it has to do with filesystem metadata not being as easy to damage. UFS maintains a kind of free list to allocate new blocks, whereas HFS+ and JFS and XFS use bitmap allocation. If you stomp on part of a bitmap it's way worse than stomping on part of a list. ZFS on the other hand has a tree of blocks and can keep a configurable number of redundant copies on each drive and there are sometimes older copies that exist depending on how full the filesystem and how much has been written.

In nearly 30 years working with UNIX, once FSCK was introduced I *never* had a file system so damaged that FSCK couldn't completely restore the structure to working order. That's nice. In 20 years working with Unix and Unix-like systems I have had several UFS filesystems be unrecoverable with fsck. I've had a JFS be *completely* destroyed just by doing sync() calls too often. Probably you were using the good hardware that UNIX (as opposed to Unix or Linux) typically runs on.

Three times now I've had HFS+ file systems require a backup and restore because of some obscure damage that even rebuilding the catalog wouldn't fix. A friend of mine is currently booting his Mac Pro off the second drive because the original installed file system was trashed. Can I suggest not filling your drive up completely with porn and movies and music or whatever else you are filling up the drive with? HFS+ has a problem when more blocks need to be allocated for the catalog and there are no blocks which can be used, it ends up with files and catalog sharing blocks and the catalog getting destroyed catastrophically. It could be fixed but it looks sufficiently annoying that they would rather have it occasionally fail instead of messing with it.

Conceptually HFS+ is far better than UFS, the problem is just one of implementation.

What does ZFS do? The write-ups on ZFS indicate that they stop short of testing that case, and that's the most important one. It is written up in many places, you just missed it. ZFS keeps a configurable number of backup copies of any metadata and all metadata is checksummed separately, so even on a single drive it's almost impossible for the data to not be recoverable. Because the filesystem itself can detect all errors (other than the 1 in 2^160 chance of an error *and* hash that perfectly matches it) it does the repairs itself (other fs don't know if a block is wrong so they can't fix it easily, which is why you need a separate tool). There could be a use for a separate tool that would recover partial files, ones that had been deleted a long time ago and part of its blocks reclaimed, but there's no need for a fsck and this is not possible with other filesystem anyway (since they rewrite in place
).

ZFS claims "you'll never have to fsck again". That's what every journalled file system proponent says. ZFS is not a journaled filesystem, it's a log-based one. "Journaled" filesystems only journal the metadata, because they have to do everything twice: first write to the journal anything that should be guarenteed, then wait for it to sync, then write the data to the 'disk'. A log-based filesystem basically writes the 'diff' to the disk then says 'this is the new state'. It's more like a (good) kind of subversion or version control that forgets old revisions from time to time.

Re:UFS has an FSCK that really works

[argent]

Can I suggest not filling your drive up completely with porn and movies and music or whatever else you are filling up the drive with

You can suggest it, and I can suggest that you keep your damn hands off my porn.

HFS+ has a problem when more blocks need to be allocated for the catalog and there are no blocks which can be used, it ends up with files and catalog sharing blocks and the catalog getting destroyed catastrophically.

I know. I consider that completely unacceptable behavior from a file system.

It could be fixed but it looks sufficiently annoying that they would rather have it occasionally fail instead of messing with it.

Which is why I would rather use a file system designed by people who give a damn about file system reliability.

In 20 years working with Unix and Unix-like systems I have had several UFS filesystems be unrecoverable with fsck.

I've never had that happen for a file system that wasn't so trashed it couldn't even be mounted readonly. If there's a file system there, FSCK will get everything out of it that it can, and it'll put everything it can't link into lost+found, and when it's done that file system is clean.

I'm not saying that other file systems (like ZFS) can't be as good, I'm saying that there WAS a legitimate reason to go to UFS if Apple had actually cared about file system stability and reliability. As you noted, they don't.

Which brings me back to my original point, that just because Apple's providing support for ZFS, don't assume that they'll actually make it into something that can be used as a replacement for HFS+ on the desktop.

My point wasn't "UFS pwns ZFS"... that was an aside. It's "there's a legitimate reason to want to use UFS on a desktop or laptop Mac, and Apple made a big deal about the improved UFS support in Panther, and they ended up not going anywhere with its, so don't depend on them going anywhere with ZFS either".

BTW, and this is (still) just an aside... I know about the difference between log based file systems and journalled file systems... I read the Sprite papers, I've installed several NetAPPs, and I know the advantages of treating the journal/log as the primary storage and checkpointing references to active metadata. I would still like to know that the file system has a tool to dig through that mess and rebuild all the metadata that it can and actually repair the thing if worst comes to worst.

Re:UFS has an FSCK that really works

[Guy Harris]

UFS maintains a kind of free list to allocate new blocks, whereas HFS+ and JFS and XFS use bitmap allocation.

No, UFS uses a bitmap. You're probably thinking of the old V6 and V7 file systems, which did use a free list.

ZFS is not a journaled filesystem, it's a log-based one. "Journaled" filesystems only journal the metadata, because they have to do everything twice: first write to the journal anything that should be guarenteed, then wait for it to sync, then write the data to the 'disk'. A log-based filesystem basically writes the 'diff' to the disk then says 'this is the new state'.

From everything I've read, ZFS isn't a log-based file system, it's a "copy-on-write" file system, similar to WAFL. If a block is modified, the changed copy is written to a new block rather than overwriting the existing block; this includes indirect blocks, so you have the new copy of the data block pointed to by a new copy of an indirect block - at least in the case of WAFL, this goes all the way up to the root inode, which is updated last.

And, yes, WAFL has an fsck equivalent. On occasion, you need it, but on a typical reboot, you either have the new root inode, from which can be found a version of the file system with all updates for that consistency point finished, or the old root inode, from which can be found a version of the file system with no updates from that consistency point finished, plus the NVRAM log, from which you can reply all the changes that caused that update. In either case, the file system is consistent if no disk problems or software bugs corrupted the data.

Re:So.... BSD or Solaris???

[Wolfrider]

--I've been anxiously awaiting FBSD7 myself; playing around with it in Vmware Server with a 7-disk SCSI rack (physical disk access) for ZFS. So far it's nice, but when a disk died I found out a huge limitation with Server - continuous prompts on the host that FREEZE the ENTIRE VM until the disk got cut from the ZFS pool and the VM was rebooted.

Psst...

[Anonymous Coward]

It is based on BSD. It's called Darwin.

Re:So.... BSD or Solaris???

[IvyKing]

killall is used by shutdown(1M) to kill all active processes not directly related to the shutdown procedure

...which might explain why many linux distributions have begun including the solaris-like "pgrep" and "pkill" commands...

IMO, the Solaris use of "killall" makes more sense than the Linux use - i.e. Kill all

On a related note, I had a weird moment reading the manpage for sshd_config on OpenBSD 4.1 - thought I'd made a mistake and was reading the same manpage from Solaris - the 'maxstartups' line appeared in sshd_config for Solaris before it showed up in OpenBSD.

Pure BS, the follow-up post

[LKM]

Macjournals.com clarifies its earlier post.

Snapshots vs. Backup

[LKM]

I lost interest in what Apple does when stopped working for a school district, so I'll acknowledge my ignorance of how Time Machine works. I believe it allows you to restore files to previous states. (correct me if I'm wrong). This should be fairly trivial to implement using snapshots and clones. Which ZFS does quite efficiently.

While Time Machine's UI looks like Snapshots, Time Machine really does not implement Snapshots. It implements Backup. There's a difference: Snapshots preserve the current state of a disk locally, while Backup stores the current state of a predefined set of documents remotely (usually on another computer or on an external disk). In other words, Snapshots take up a lot of local disk space and are thus not suitable for something like Time Machine, where you want (ideally) each state of a file preserved (i.e. backups occur at regular intervals).

What Snapshots do in ZFS is simply telling ZFS to not delete any of the files currently on the disk. If a file is then changed, a copy is created (or a delta, but I think current implementations simply create copies). If it is deleted, it remains on the disk but becomes invisible. As you can see, this implies that you'll quickly use a huge amount of disk space for Snapshots, so it's simply not an option for Time Machine.

Re:Snapshots vs. Backup

[Anonymous Coward]

I understand the workings of snapshotting in zfs, and the difference between a snapshot and a backup (normal snapshots are no substitute. I will admit that I was under the impression that Time Machine was closer to a snapshot then a backup, but as I said earlier, I don't follow what Apple is doing in-depth). That said, snapshots can be easily exported to an external device, and if that is your only purpose, take the snapshot, send it to the external device, then delete the snapshot (I have done this, it works nicely). I stand by my point that the implimentation is trivial.

It is a delta (it has to be due to zvols).

I really have never understood where this whole "it uses too much space" argument comes from. Snapshots only take up room as long as they exist and in so far as they differ from the active dataset. This leaves the obvious solution: GET RID OF ONES OLD ENOUGH THAT YOU NO LONGER NEED THEM. On the second point, how often do you REALLY flip every bit on the drive? I only ask because that is the worst case between snapshots, and most data access patterns don't follow that (they come nowhere close). The most changed data on your drive should be your virtual memory, which should not be on a dataset you are snapshotting.

From experience, unless you do something terminally stupid, like snapshotting virtual memory, it really doesn't use as much as you think it implies (unless you keep hourly snapshots around for eternity, then it might eventually. I happen to be one eternity short of being able to test this, and its a Camelot-level silly thing to do anyway). FTR I *AM* using this as the only filesystem on machines already. I just wish we had support for it in the installer.

So let's compare. You have what you think the papers you've read imply, to what I have actually seen on multiple machines taking into account real world writing patterns. Yes, I will acknowledge that there are theoretical access patterns for which zfs would be impractically huge, but to date I have never encountered one of these patterns and, short of synthetic tests, highly doubt I will.

Oninoshiko

Re:Snapshots vs. Backup

[LKM]

I understand the workings of snapshotting in zfs, and the difference between a snapshot and a backup (normal snapshots are no substitute. I will admit that I was under the impression that Time Machine was closer to a snapshot then a backup, but as I said earlier, I don't follow what Apple is doing in-depth). That said, snapshots can be easily exported to an external device, and if that is your only purpose, take the snapshot, send it to the external device, then delete the snapshot (I have done this, it works nicely). I stand by my point that the implimentation is trivial.

Okay, now you have me confused. Are you talking about the specific implementation of Snapshots in ZFS, or about the idea of Snapshots in general? What does "take a snapshot and send it to an external device" mean? Does that mean storing the whole FS state at the time of the snapshot on the external disk? If so, how is that superior to basically versioning each single file, as Time Machine does? If you do this and then create a second Snapshot and store that on an external disk, does ZFS somehow know to only store the difference between the first and the second Snapshot? Can the OS easily get the different versions of single files as they exist in different Snapshots?

Frankly, I simply don't see the upside Snapshots provide. When trying to use Snapshots as Backup, they simply make everything much more complicated.

Why use Snapshots if you aren't going to use them as Snapshots?

It is a delta (it has to be due to zvols).

Doesn't that create a huge overhead? If I work on a video project with a number of multi-gigabyte files, create a Snapshot, and then continue working on them, ZFS somehow only writes deltas of the changes of each single file?

I really have never understood where this whole "it uses too much space" argument comes from.

Oh, no. The argument is not really that it uses "too much space". The problem is that Sun claims that Snapshots are free. They are not. They are free at the specific time when you create them, but as soon as you start changing files, they aren't free anymore.

On the second point, how often do you REALLY flip every bit on the drive?

Obviously hardly ever. However, in certain environments (video production being an obvious one with regards to Macs), you flip a huge percentage of all bits on the drive pretty regularly.

Re:Snapshots vs. Backup

[Anonymous Coward]

NTFS allows for something similar. Take the snapshot, run the backup against the snapshot (I believe windows calls this a "shaow copy"). So while the concept is not intrinsic to the concept of a snapshot, it is also not a ZFS specific implimentation (and giving credit where it's due, is a good idea, and AFAIK was first in NTFS). Getting at different versions isn't that difficult, although it would require some footwork to make it practical for the lay-person via the GUI (if i can script it, a GUI can be made for it).

Using a snapshot freezes that data in a state quickly (close enough to instantly that any difference is irrelevant). It can then continue to be written while backups are occurring. This is probably not an issue so much on a workstation as a loaded server (although if your taking them hourly, it might become one).

The only thing written is the differences between the states. Maybe I'm missing something, but thats the most-efficient way to store both. Now that I'm really thinking about it, it is probably block-granular. I know it cannot be file-granular because the zfs has no understanding of how the data within a zvol is structured (ergo, if it were file-granular it would have to treat the entire volume as a single file). If you were to store two full copies of your multi gig files, THAT would be huge.

No one, including SUN (despite what the wonderful and highly knowledgeable folks at MacJournals seem to think http://www.macjournals.com/news/2007/10/07#a80 ), is claiming is that they take up no disk space ( http://docs.sun.com/app/docs/doc/819-5461/6n7ht6qs9?a=view : "Snapshots can be created almost instantly, and initially consume no additional disk space within the pool. However, as data within the active dataset changes, the snapshot consumes disk space by continuing to reference the old data and so prevents the space from being freed."). What I am claiming (I cannot speak for SUN) is that they do not, under normal usage patterns "eat disk space like crazy" (both of these lies MacJournals has claimed at least twice now (once verbatim on the latter)). Further MacJournals' article in response to their statement goes on to completely mischaracterize an comment from Drew Thaler (the thought that 20GB was huge 10 years ago now 200GBs are in things we don't even normally think of as computers, so it is likely that 20TB will be commonplace in the not-to-distant future). While they occasionally have a point (yes,it does increase processor usage, so if your machine is processor starved (or not 64 bit (has OSX gone fully 64 bit yet?)) it might introduce battery issues), most of MacJournals' comments are so ill-informed they do not even qualify as FUD. This source is less reputable then the AppleInsider (and that is saying alot, there were numerous errors in the AppleInsider piece, but common these guys can't even grasp how "standard RAID" (and, as before, 2 is far from standard) works.

Oninoshiko

Re:Snapshots vs. Backup

[LKM]

Ah, I see. The advantage that Snapshots provide over "plain" backup is that you can run snapshot backups in the background while you keep changing files. You generally do not want to do that with plain backup because you could end up with an inconsistent backup. That's actually a good point.

The "eat disk space like crazy" comment is hardly a lie. Even when only storing deltas, creating snapshots means that the data in the snapshot can never be released unless the snapshot is deleted, which will quickly fill your disk if you're working with large files that change often.

Also, I was wrong when I said that MacJournals claimed that Sun said that backups were free. Here's the quote from MWJ Jun 11 07:

Signatures only make sense for backups, since you want the
  signature to verify that no one changed the data since it was
  backed up. You may recall that ZFS proponents insist you get
  "backups for free," which is why they keep screeching that it's
  the only sensible foundation for Time Machine, the new backup
  feature in Mac OS X 10.5. That goes back to the copy-on-write
  methodology. And no, Tiger isn't fully 64 bit, but Leopard is supposed to be, except for Carbon apps.

Re:Snapshots vs. Backup

[Anonymous Coward]

no, Mac Journals did make that claim, just in a different article (emphisis mine): "Sun is fond of saying that snapshots take up no disk space, but that's only true when they're created, just like an empty file." -- http://www.macjournals.com/news/2007/10/07#a80

I am not presently seeing ZFS snapshots "eating diskspace like crazy." MacJournals' claim is that it does "eat diskspace like crazy." Their wording is absolute; it will always happen. While I will acknowledge that there might be access patterns that cause it to happen, I would hardly call these common (let alone "EVERY" pattern).

I am using ZFS now, I do not see what is claimed on my system. I wouldn't be calling it a lie if it weren't for the adamant nature of their claim, but they don't say "under some access patterns", what they say is "it does this." If someone makes a absolute claim that tests show to be factually untrue (in a not insignificant percentage of cases), is that not a lie? I suppose it IS after 1984, so maybe its "truth."

Professional editing of full-length video is NOT a normal access pattern. Not even close. The majority of people who are doing digital media work have a level of technical expertise which would allow them to change filesystems, or to adjust the settings appropriately to what they do (this is not uncommon of users of Adobe products now). MOST usage does not follow the pattern of "massive changes to massive files," small changes are far more common then massive ones.

I do not claim that it is perfect for all conditions, but to claim that it is completely inappropriate for normal desktop usage because of how it preforms under a professional grade digital media creation workload is silly.

Oninoshiko

Re:Snapshots vs. Backup

[LKM]

Professional editing of full-length video is NOT a normal access pattern.

A lot of people using Macs make use of iMovie. It comes for free with new Macs, after all. And even cutting a half-hour vacation movie easily eats up dozens of gigabytes of often-changing disk space.

FileIDs?

[anarkhos]

What about aliases and FileIDs? I see no mention of this anywhere.