ZFS, the Last Word in File Systems?

guigouz writes "Sun is carrying a feature story about its new ZFS File System - ZFS, the dynamic new file system in Sun's Solaris 10 Operating System (Solaris OS), will make you forget everything you thought you knew about file systems. ZFS will be available on all Solaris 10 OS-supported platforms, and all existing applications will run with it. Moreover, ZFS complements Sun's storage management portfolio, including the Sun StorEdge QFS software, which is ideal for sharing business data."

Comments

billion billion?

[michael path]

From the article:

Unlimited scalability
As the world's first 128-bit file system, ZFS offers 16 billion billion times the capacity of 32- or 64-bit systems.

Microsoft immediately countered by saying WinFS will now support "twelveteen million billion times" as much storage as Sun's ZFS, and is "a bazillion times" more secure.

When reached for comment, Sun CEO Scott McNealy replied "neener neener". Microsoft CEO Steve Ballmer responded by putting gum in Sun President Jonathan Schwartz's hair.

Re:billion billion?

[Paulrothrock]

Billion billion is a perfectly valid number. Or would you rather they say 6.0 × 10^18? Most people can't imagine that. But people can (kind of) visualize a billion, and then multiply that by a billion, and see it's really, really big.

Re:billion billion?

[michael path]

How about quintillion?

Re:billion billion?

[backslashdot]

a file systems for trillions of billions bytes of data?

What's it for?

Installing Windows ?

Re:billion billion?

[TedCheshireAcad]

16 billion billion times!!

pinky in corner of mouth.

Re:billion billion?

[HerculesMO]

Given the fact there are an infinite amount of numbers, any murmur from your mouth or any written gibberish can be conveyed to a number.

For example, 'sassdfadef' is a number I think is a 2 with one thousand 3s after it. It's really moot :)

Re:billion billion?

[escher]

You don't do much video editing, do you? ;)

Re:billion billion?

[poot_rootbeer]

As the world's first 128-bit file system, ZFS offers 16 billion billion times the capacity of 32- or 64-bit systems.

A 64-bit (unsigned) binary number can already store values up to 16 billion billion (actually, closer to 18, but who's counting). That's roughly 2.5 billion individually addressable locations for every man, woman, and child living on Earth.

Shouldn't that be enough to hold us for a few generations at least?

Re:billion billion?

[hackwrench]

In fact, this page is just one big number.

Re:billion billion?

[Shadowlion]

Even including all the world's porn.

I dunno, man. I've got a lot of porn...

Re:billion billion?

[Dazza]

Hmm... another one who doesn't know that there's a fair amount of land outside the US borders.

Nope. He said he'd never been outside the UK, so I'd be fairly certain he's aware of land outside the US.

Also living in the UK, I can attest that whenever you hear '1 billion', '1000 million' is meant. The UK converted to this for accounting purposes during the 70's.

The same I suspect is true for most of previously Europe-dominated countries (say India for example).

India, in particular, is toally different. They don't rely on millions and billions but 'crore' and 'lakh' which are 10million and 100k respectively.

Re:billion billion?

[Just Some Guy]

Our filesystem goes to eleven.

Open source

[Splinton]

And it looks like it's going to be opensourced along with most of Solaris 10!

Presumably a 32 bit machine will be able to handle a 128 bit file system, in the same way as Solaris 10 is currently destined for (at most) 64 bits.

Re:Open source

[balster neb]

Yes, it does look like it would be open-sourced as part of Solaris 10 (it was mentioned as one of the major new features).

Assuming the Solaris 10 will be true open source (not like Microsoft's "shared source"), as well as GPL compatibile, would I be able to use ZFS on my GNU/Linux desktop? Will ZFS be a viable alternative to ext3 and ReiserFS? Or is the overhead too big?

Re:Open source

[tolan-b]

I suspect that whatever open source license Sun release Solaris under, they'll be careful to make sure it's incompatible with the GPL.

Re:Open source

[CrkHead]

It looks like Microsoft may have its new WinFS after all...

Re:Open source

[GileadGreene]

From the article:

More important, ZFS is endian-neutral. You can easily move disks from a SPARC server to an x86 server. Neither architecture pays a byte-swapping tax due to Sun's patent-pending "adaptive endian-ness" technology, which is unique to ZFS.[emphasis mine]

So while it might be open-sourced, you're not likely to see it migrating to Linux or the BSDs any time soon.

Two things...

[rincebrain]

1) Even Sun has succumbed to recursive acronyms, now.

2) Is it just me, or is the post surprisingly bereft of unique details? I mean, integration with all existing applications is rather assumed, given that it's a file system and all...

Re:Two things...

[InadequateCamel]

I especially liked:
"Neither architecture pays a byte-swapping tax due to Sun's patent-pending "adaptive endian-ness" technology"

Adaptive endian-ness? What a stupid thing to include in a press release...there has to be a better way to say that.

Just announced by Sun:
"ANMF, our new file system (Ambiguous Nomenclature FS) will be filled with file cataloguing technology stuff that allows faster-ish operations that result in application goodness".

Hmf.

[BJH]

Logically, the next question is if ZFS' 128 bits is enough. According to Bonwick, it has to be. "Populating 128-bit file systems would exceed the quantum limits of earth-based storage. You couldn't fill a 128-bit storage pool without boiling the oceans."

So, what was the point of creating a 128-bit filesystem?

-1, Marketing Hype.

*Yawn*

Re:Hmf.

[Kenja]

"So, what was the point of creating a 128-bit filesystem?

Getting rid of file/drive size limitations for the foreseeable future?

Re:Hmf.

[elmegil]

"You'll never need more than 640K of memory". The point would be to be ready as storage densities increase. In the last 8 years we've gone from a terabyte filling a room to a terabyte on a desktop, and I'm sure there are more density breakthroughs coming.

It's your density, Luke.

Re:Out of letters.

[Saint Stephen]

[fs, natch

There already is a ZFS.

[TheLoneGundam]

IBM has ZFS on their z/OS Unix Systems Services (POSIX interfaces on z/OS) component. ZFS was developed to provide improvements over the HFS (Hierarchical File System) that they ship with the OS.

Re:not alphabetically

[laird]

Nah, the ultimate filesystem has to be xyzzyfs! Your data magically appears... :-)

Why don't they just describe the capacity in

[wiredog]

Sagans?

Just better than the old stuff from Sun

[Ewan]

Reading the article, all I see is Sun saying how bad their old stuff was, e.g.:

Consider this case: To create a pool, to create three file systems, and then to grow the pool--5 logical steps--5 simple ZFS commands are required, as opposed to 28 steps with a traditional file system and volume manager.

and

Moreover, these commands are all constant-time and complete in just a few seconds. Traditional file systems and volumes often take hours to configure. In the case above, ZFS reduces the time required to complete the tasks from 40 minutes to under 10 seconds.

Compared to AIX or HP-UX, 28 steps is shockingly bad, both have had much simpler logical volume management for several versions now (AIX for 5 years or more? certainly as long as I have used it). The existing Solaris 9 logical volume infrastructure is years behind the competition, this is bringing it up to date, but not putting it far ahead.

Ewan

What sort of crap is this?

[LowneWulf]

COME ON! It may be a slow day, but how is this news? There's only one link, and it's to Sun's marketing info.

Can someone please provide a link to some technical details other than it being 128-bit? What does this file system actually do that is even remotely special? What's under the covers? And, more importantly, does it actually work as described?

-1,Uninformative

That's a lot of storage

[Gentoo Fan]

But of course you'll still have to have your boot image within the first 1024 cylinders.

Another quote to cherish

[AsciiNaut]

I broke the habit of a lunchtime and RTFA. According to Jeff Bonwick, the chief architect of ZFS, "populating 128-bit file systems would exceed the quantum limits of earth-based storage. You couldn't fill a 128-bit storage pool without boiling the oceans."

Who else instantly thought of, "640 K ought to be enough for anybody", uttered by the chief architect of twenty years of chaos?

Re:Another quote to cherish

[mdmarkus]

From Bruce Schneier in Applied Cryptography: Thermodynamic Limitations One of the consequences of the second law of thermodynamics is that a certain amount of energy is necessary to represent information. To record a single bit by changing the state of a system requires an amount of energy no less than kT where T is the absolute temperature of the system and k is the Boltzman constant. (Stick with me; the physics lesson is almost over.) Given that k = 1.38x10^-16 erg/Kelvin, and that the ambient temperature of the universe is 3.2K, an ideal computer running at 3.2K would consume 4.4x10^-16 ergs every time it set or cleared a bit. To run a computer any colder than the cosmic background radiation would require extra energy to run a heat pump. Now, the annual energy output of our sun is about 1.21x10^41 ergs. This is enough to power about 2.7x10^56 single bit changes on our ideal computer; enough changes to put a 187-bit counter through all of its values. If we built a Dyson sphere around the sun and captured all of its energy for 32 years, without any loss, we could power a computer to count up to 2^192. Of course it wouldn't have the energy left over to perform any useful calculations with this counter. But that's just one star, and a measly one at that. A typical supernova releases something like 10^51 ergs. (About a hundred times as much energy would be released in the form of neutrinos, but let them go for now.) If all of the energy could be channedel into a single orgy of computation, a 219-bit counter could be cycled through all of its states. These numbers have nothing to do with the technology of the devices; they are the maxiumums that thermodynamics will allow. And they strongly imply that brute-force attacks against 256-bit keys will be infeasible until computers are built from something other than matter and occupy something other than space.

Different architecture, same functionality?

[perseguidor]

With traditional volumes, storage is fragmented and stranded. With ZFS' common storage pool, there are no partitions to manage. The combined I/O bandwidth of all of the devices in a storage pool is always available to each file system.

Until now it does sound just like raid, but:

When one copy is damaged, ZFS detects it via the checksum and uses another copy to repair it.

No competing product can do this. Traditional mirrors can only handle total failure of a device. They don't have checksums, so they have no idea when a device returns bad data. So even though mirrors replicate data, they have no way to take advantage of it.

I guess I just don't get it; I know they are talking about logical corruption and not a physical failure, but this is kind of like raid with somethink like SMART, or isn't it?

And what kinds of corruption can there be? Journaling filesystems already work well for write errors and such, or so I thought.

I know the architecture seems innovative and different (at least for me), but is there really new functionality?

Sorry if I seem ignorant this time. I don't know if I was able to get my point across; the things this filesystem does, wouldn't they be better left on a different layer?

What I really want to see in a file system...

[kcbrown]

...and that I haven't seen in any file system announced to date, is a way of bundling multiple filesystem operations into a single atomic transaction that can be rolled back. This would clearly require an addition of four system calls (one to begin a transaction, one to commit it, one to roll it back, and one to set the default action, commit or rollback, on exit).

Such a feature would rock, because it would be possible to make things like installers completely atomic: interrupt the installer process and the whole thing rolls back.

Re:What I really want to see in a file system...

[dominator]

Reiserfs will apparently soon have what you're looking for. Already, all primitive operations are atomic, but they plan on exporting a user-space transaction interface soon.

http://www.namesys.com/benchmarks.html

"V4 is a fully atomic filesystem, keep in mind that these performance numbers are with every FS operation performed as a fully atomic transaction. We are the first to make that performance effective to do. Look for a user space transactions interface to come out soon....

Finally, remember that reiser4 is more space efficient than V3, the df measurements are there for looking at....;-) "

Apparently...

[qtone42]

... ZFS will also make you forget everything you knew about English grammar.

"We've rethought everything and rearchitected it," says Jeff Bonwick

Rearchitected? WTF? Howsaboot "Redesigned?"

I'm still wrapping my brain around "adaptive endian-ness" as well.

--QTone

Sounds really nice

[mveloso]

Looks like Sun went out and redid their filesystem based on the performance characteristics of machines today, instead of machines of yesteryear.

Some highllights, for those that don't (or won't) RTA:

* Data integrity. Apparently it uses file checksums to error-correct files, so files will never be corrupted. About time someone did this.

* Snapshots, like netapp?

* Transactional nature/copy-on-write

* Auto-striping

* Really, Really Large volume support

All of this leads to speed and reliability. There's a lot of other stuff (varying blocks sizes, write queueing, stride stuff which I haven't heard about in years), but all of it leads to above.

Oh, and they simplified their admin too.

It's hard to make a filesystem look exciting. Most of the time it just works, until it fails. The data checksum stuff looks interesting, in that they built error correction into the FS (like CDs and RAID but better hopefully).

It might also do away with the idea of "space free on a volume," since the marketing implies that each FS grows/shrinks dynamically, pulling storage out of the pool as needed.

Any users want to chime in?

Re:Unlimited scalability

[szo]

How did you check it? Count up to it and then add one and see if you could? Just asking.

Thanks

Szo

64 bits is awfully big already

[pslam]

Getting rid of file/drive size limitations for the foreseeable future?

It would take over 500 years to fill a 64 bit filesystem written at 1GB/sec (and of course 500 years to read it back again). 64 bits is already an impossibly large figure. There's absolutely nothing special or clever whatsoever about doubling the size of your pointers aside from using up more disk space for all the metadata.

64 bits is enough for today's filesystems in much the same way that 256 bit AES is enough for today's encryption - there are far bigger things that will require complete system changes than that so called "limit". I suspect a better filesystem will come along well before those 500 years are up... I agree with grandparent:

-1, Marketing Hype.

Re:64 bits is awfully big already

[Jeff DeMaagd]

It would take over 500 years to fill a 64 bit filesystem written at 1GB/sec (and of course 500 years to read it back again).

One product already can transfer a Terrabyte per second, so that would cut the transfer down to half a year. And I imagine that transfer rate would continue to increase.

I don't see how one would necessarily argue against such a thing for products that will go for cluster and supercomputer use. I say might as well get the bugs out so when you can so that once the 65th bit is needed, the supercomputer suppliers are ready.

http://www.sc-conference.org/sc2004/storcloud.ht ml

Re:64 bits is awfully big already

[pslam]

Yeah, its probably marketing hype now, but in 5 years, what about 10? Just because we can't do it now doesn't mean that we should stop progress.

No, precisely because we can't do it now, and for the very predictable future, we shouldn't be wasting all that disk space, access and CPU time for a boundary that no production system is likely to ever reach before they get upgraded. That's just practicality.

Seagate apparently sold 18.3 million desktop drives last year. Assuming they're all about 120GB (which is generous of me), that would be about 17.6*10^18 bits. Guess what, that's 2^64 bits. Yes, you would have to buy every single desktop hard drive Seagate shipped in the last year to have the capacity to fill a 64 bit filesystem. And find space for 18 million drives. And a power station to deliver the several hundred megawatts you'd need.

Even at 2 times drive capacity growth per year that's still a ridiculously unattainable figure. In 14 years time you'd only need to buy 1000 drives (which are now 2000TB each). But 14 years is a geological time scale when it comes to computers. You'd have wasted 14 years of CPU time and disk space devoted to those extra 64 bits.

If you still think 64 bits isn't enough, how about 96 bits? It would take 46 years before hard disks were big and cheap enough so you could fill the filesystem by buying 1000 of them. But no, they chose 128 bits because it sounded good.

Re:Out of letters.

[badriram]

I just wonder how many people on slashdot would even understand that....

To those who dont know.. [ comes after Z in ASCII and unicode-latin

There already is an HFS as well.

[tepples]

Then why didn't IBM call its improved HFS "HFS Plus"? No wait, that would collide with Apple's HFS and HFS Plus, used in Mac OS.

It would appear that there can be only twenty-six distinct file systems. Then Microsoft went and innovated NTFS with Four-Letter-Word File System Technology, which actually was just a copy of IBM's HPFS, the first to introduce File System Named After a Competitor Technology.

Shared data pools...

[vspazv]

So what are the chances that someone could accidentally wipe the shared data pool for an entire company and how hard is recovery on a volume striped across a few hundred hard drives?

Patents and other Bad Signs.

[twitter]

Opensource is useless when it's patent encumbered. While it's nice that the details will be available, it sucks to think that I can't use them except to serve Sun for the next 17 years. Such disclosure, of course, is what the patent system is supposed to provide but does not. What the patent is providing is ownership of ideas. How obvious those ideas are and if there's prior art is impossible to say from the linked puff piece.

This article is shocking. I'm used to much less hype and far more technical details from Sun. Software patents and bullshit are not what I expect when I follow a link to them.

I don't like any of this.

Last Word?

[tedgyz]

It's a 128-bit filesystem, so doesn't that make it the last 8 words?

There are a lot of cluster file systems

[anzha]

Right now there are a lot of file systems that do somehing not all that different than what Sun is proposing. The project I am on is evaluating them as we speak for a center wide filesystem. I've had the fun (no sarcasm, honestly) of setting up a number of different onces and helping to run benchmarks and tests against each. All of them have strengths. Every single one of them has some nasty weaknesses.

If you are looking for an open source based cluster file system, Lustre is what you want. It's supported by LLNL, PNNL, and the main writers at ClusterFS Inc. It's a network based cluster FS. We've been using it over GigE. However, we've found that there needs to be a ratio of 3:1 for data server:clients for a ratio. Wehave only used one metadata server. Failover isn't the greatest. Quotas don't exist. it also makes kernel mods (some good and bad) to do a mild fork of the linux kernel (they put them into the newer kernels every so often). It only runs on Linux. Getting it to run on anything else looks...scary.

GPFS runs on AIX and Linux. Even sharing the same storage. It runs and is pretty stable. it has the option to run in a SAN mode or network based FS. In the latter form, it even does local discovery of disks via labels so that if a client can see the disks locally it will read and write to them via FC rather than to the server. It, however, is a balkanized mess. It requires a lot more work to bring up and run: there is an awful lot of software to configure to get it to run (re: RSCT. If you haven't had the joys of HATS and HAGS, count yourself very, very lucky).

ADIC's StorNext software is another option. This one is good if you are interested in ease of installation, maintanence, and very, very fast speeds (damn near line speed on Fibre channel). I have set this one up for sharing disks in less than two hours from first install to getting numerous assorted nodes of different OS's to play together (Solaris, AIX, Linux). It freakin on virtually everything from Crays to Linux to Windows. It's issues seem to be scaling (right now doesn't go past 256 clients) and it has some nontrivial locking issues (righting to the same block from multiple clients, and parallel I/O to the same file from multiple clients if you change the file size).

There are some others that are not as mature. Among them are Ibrix, Panasas, GFS, and IBM's SANFS. All of them are interesting or promising. Only SANF looks like it runs on more than Linux though at this point. Our requirements for the project I am on are to share the same FS and storage instance among disparate client OSes simultaneously. This might not be the same for others though and these might be worth a look. Lustre dodges this because its open source and they're interested in porting.

Technically....

[jolyonr]

The last word in file systems is "systems".

Thank you.

Re:What is their disk allocation scheme?

[majid]

I was in a chat session with their engineers yesterday. It looks like they have adaptive disk scheduling algorithms to balance the load across the drives (e.g. if a drive is faster than others, it will get correspondingly more I/O). The scheduler also tries to balance I/O among processes and filesystems sharing the data pool.

This is a good thing - queueing theory shows a single unified pool has better performance than several smaller ones. People who try to tune databases by dedicating drives to redo logs don't usually realize what they are doing is counterproductive - they optimize locally for one area, at the expense of global throughput for the entire system.

ZFS uses copy-on-write (a modified block is written wherever the disk head happens to be, not where the old one used to be). This means writes are sequential (as with all journaled filesystems) and also since the old block is still on disk (until it is garbage collected) this gives the ability to take snapshots, something that is vital for making coherent backups now that nightly maintenance windows are mostly history. This also leads to file fragmentation so enough RAM to have a good buffer cache helps.

Because the scheduler works best if it has full visibility of every physical disk, rather than dealing with an abstract LUN on a hardware RAID, they actually recommend ZFS be hosted on a JBOD array (just a bunch of disks, no RAID) and have the RAID be done in software by ZFS. Since the RAID is integrated with the filesystem, they have the scope for optimizations that is not available if you have a filesystem trying to optimize on one side and a RAID controller (or separate LVM software) on the other side. Network Applicance does something like this with their WAFL network filesystem to offer decent performance despite the overhead of NFS.

With modern, fast CPUs, software RAID can easily outperform hardware RAID. It is quite common for optimizations like hardware RAID made at a certain time to become counterproductive as technology advances and the assumptions behind the premature optimization are no longer valid. A long time ago, IBM offloaded some database access code in its mainframe disk controllers. It used to be a speed boost, but as the mainframe CPU speeds improved (and the feature was retained for backward compatibility), it ended up being 10 times slower than the alternative approach.

Re:fileless systems

[Tony]

After years of everyone saying that the relational model was the answer to all data organziation needs... the hierarchical model reappeared in the form of XML, and people realized that it is convenient to organize some types of data hierarchically.

Convenient, and flawed.

XML isn't designed to handle changing data. It's designed to be a data markup language, which indicates it's used for presenting data, not managing data.

So far, the relational model is the best mathematically-rigorous method of managing sets of data. There are many advantages to hierarchical data representation, but for manipulation, the relational still trumps.

Do I want to use SQL to access my files? Not if I don't have to. There are perhaps better methods, even some transparent methods.

But, do I want to continue to self-organize my data? Hell, no! There's just too much information stored on my computer, and on my network, these days. And, considering that much of my data has multiple relationships, the hierarchical model is growing a bit long in the tooth. Many of my documents belong in multiple hierarchies.

But, there might be a real solution soon:

Gnome Storage looks to be a good first step.