A Short History of Btrfs

diegocgteleline.es writes "Valerie Aurora, a Linux file system developer and ex-ZFS designer, has posted an article with great insight on how Btrfs, the file system that will replace Ext4, was created and how it works. Quoting: 'When it comes to file systems, it's hard to tell truth from rumor from vile slander: the code is so complex, the personalities are so exaggerated, and the users are so angry when they lose their data. You can't even settle things with a battle of the benchmarks: file system workloads vary so wildly that you can make a plausible argument for why any benchmark is either totally irrelevant or crucially important. ... we'll take a behind-the-scenes look at the design and development of Btrfs on many levels — technical, political, personal — and trace it from its origins at a workshop to its current position as Linus's root file system.'"

Comments

oh wee sun's sloppy seconds.

[Anonymous Coward]

So, let's get this right. the BEST that Linux can get is a murderer and a SUN REJECT? lol quality.

Excuse me, I think I'll stick to a REAL OS that features a REAL FS backed by a REAL company!

Re:oh wee sun's sloppy seconds.

[Anonymous Coward]

Will you also be enjoying your media in REAL PLAYER?

Re:oh wee sun's sloppy seconds.

[mysidia]

That argument isn't actually based on the technical merits, and thus doesn't make any sense..

Just because a Real OS features a Real FS backed up by a real company, doesn't necessarily mean the FS or OS are any good on technical merits compared to a REAL project licensed under a REAL free software license backed up by a REAL community and supported by a REAL foundation.

Re:oh wee sun's sloppy seconds.

[ioshhdflwuegfh]

That argument isn't actually based on the technical merits, and thus doesn't make any sense..

So what's your point? that ext4 is better than ZFS on technical merits!?

Re:oh wee sun's sloppy seconds.

[Runaway1956]

Let us all get this straight. AC would reject any and all medical knowledge that resulted from experimentation by the Nazi's during the second world war. Going a step further, he might wish to have any medical knowledge gained by the allies during WW2 purged from human knowledge. After all, such knowledge is "tainted", because it was motivated by violence.

If it can't be patented and commercialized it can't be much good.

Don't anyone inform AC that there is no patent on his arse, or his reproductive organs. He'll be removing them as "junk".

Re:oh wee sun's sloppy seconds.

[dna_(c)(tm)(r)]

I doubt it, but he will have to hand over REAL MONEY using nothing but REAL WHEEL BARROWS.

Re:oh wee sun's sloppy seconds.

[mrsteveman1]

Maybe someday you'll be a Real Boy

Re:oh wee sun's sloppy seconds.

[ultranova]

Don't anyone inform AC that there is no patent on his arse, or his reproductive organs.

AFAIK there are many patents on human genome, and thus his arse and reproductive organs would, in fact, be patented as derivative works. Furthermore, said reproductive organs would also qualify as a patent violation facilitation device (unless he has a license from patent holders), and thus possession of said organs would imply that he intends to break patent law - and of course his mere existence proves that his parents already did.

Disgusting criminal scum, his like drags the quality of this site down.

Re:oh wee sun's sloppy seconds.

[mysidia]

No. I was alluding to GP's failure to make a good argument for supporting ZFS, using sound reasoning. I'm saying the truth of the premises doesn't imply the truth of the consequent.

ZFS on supported hardware is actually superior to ext4 on certain technical merits; primarily data integrity (checksumming), random write performance, and read performance (when massive amounts of RAM are available), and more advanced features (snapshots).

On the other hand, ext4 works on 32-bit processors (ZFS is only recommended to be used on 64-bit procs) with small amounts of RAM available, less than a GB; the minimum amount of RAM one should use ZFS with is 2gb, and 4gb or more is strongly recommended, above and beyond any RAM required by apps running on the machine.

But that has little to do with the OS being produced by a large corporation.

Re:oh wee sun's sloppy seconds.

[OrangeTide]

Maybe you should get a REAL DOLL

Re:oh wee sun's sloppy seconds.

[Runaway1956]

Ahhh, but, it seems that you have assumed AC to be human. ;-)

Re:oh wee sun's sloppy seconds.

[Anonymous Coward]

Don't make me laugh.

Solaris is a hacked together version of BSD and System V and its uptime used to be measured in hours because it was leaking memory like a sieve.

Since the NFS disaster (data corruption, complete lack of security, and poor performance, all in one file system!), Sun can't be taken seriously on file systems.

And Oracle's own product requires its own dedicated database monkey to keep it running.

When it comes to technology or producing decent code, Sun and Oracle are a joke; the only people they impress are people aspiring to pointy hair.

Re:oh wee sun's sloppy seconds.

[phoenix_rizzen]

No. I was alluding to GP's failure to make a good argument for supporting ZFS, using sound reasoning. I'm saying the truth of the premises doesn't imply the truth of the consequent.

ZFS on supported hardware is actually superior to ext4 on certain technical merits; primarily data integrity (checksumming), random write performance, and read performance (when massive amounts of RAM are available), and more advanced features (snapshots).

On the other hand, ext4 works on 32-bit processors (ZFS is only recommended to be used on 64-bit procs) with small amounts of RAM available, less than a GB; the minimum amount of RAM one should use ZFS with is 2gb, and 4gb or more is strongly recommended, above and beyond any RAM required by apps running on the machine.

ZFS can be used on low-memory, 32-bit machines. Or, at least, the ZFS support in FreeBSD works quite well on 32-bit machines with as little as 768 MB of RAM. There are a lot of FreeBSD users with laptops using ZFS, as well as a lot of test boxes setup with

Yes, ZFS works best on 64-bit systems with 4+ GB of RAM, but that's mainly due to the aggressive caching that ZFS does. The more RAM you have, the more caching it does, and the faster things are.

But that doesn't mean that it can't be tuned to work in RAM-constrained situations.

Re:oh wee sun's sloppy seconds.

[DaVince21]

btrfs != reiserfs.

That is all.

Re:oh wee sun's sloppy seconds.

[mysidia]

It's more than "works best on 64-bit", there are real issues utilizing ZFS for any production load on a 32-bit proc. It's probably not that bad on a laptop, as there's really no issue if your system crashes from time to time; Windows users are quite used to it.

The issue is ZFS caching causes free memory address space fragmentation on a 32-bit architecture, resulting it being unstable on 32-bit procs. Running out of allocatable virtual address space is basically equivalent to running out of memory; you may have plenty of RAM free, but your system can't service allocation requests larger than the biggest free fragment.

ZFS is as address space hungry as it is memory hungry.

For example, if you have a large filesystem shared out via NFS, at least in some versions you could cause a kernel panic on a 32-bit architecture by running a "find /" command.

First shill!

[Anonymous Coward]

OHAI! I used Micrososft Windows Live(tm) Visual Team System Express Enterprise Vista Ultimate Edition (R) to get this first post!11!one

Re:First shill!

[Anonymous Coward]

Deliberate? I wonder...

Re:First shill!

[kdemetter]

Poor guy, he could have been first , if not for the long post .

We could implement a locking system , so that no one can start posting until the first person to begin writing posts . That way, it doesn't matter what you write , you will be first if you start first.

It may also lead to more interesting first posts , as the writers won't be stressed that they won't have a first post.
Offcourse , we need to implement a sort of timeout , to prevent locking the thread in case the first poster decides not to post.

Re:First shill!

[GigaplexNZ]

Offcourse , we need to implement a sort of timeout , to prevent locking the thread in case the first poster decides not to post.

No we don't, if it's locked and no-one is posting we won't get spam and people might actually read TFA.

Re:First shill!

[x2A]

"Offcourse , we need to implement a sort of timeout"

Nah, it's all about lockless algorithms now, don't ya know. This one's simple; you get a cookie with the post form, and posts are displayed in cookie time order, not message post order. That way, person who hits 'reply' first gets first post as soon as they post it, pushing down any posts that've been made before first post is posted but after first post was initiated. Simple :-)

Re:First shill!

[Anonymous Coward]

Success! At least for anyone that doesn't block cookies; or modifies them to ensure FP.

Re:First shill!

[x2A]

Yeah just like you can modify a cookie from an online shop to make it look like you've paid for something to get them to post it out to you... oh or is it that the cookie is just a code used to lookup the actual values on the server?

And you'd also just put the cookie value in the form, so even with cookies blocked it would work.

You didn't think this one thru huh? :-p

Looks promising

[PhunkySchtuff]

This looks like a promising filesystem - as ZFS on linux is, at present, doomed to die an ugly death, btrfs looks to address a lot of the shortcomings of other filesystems and bring a clean, modern fs to linux. It goes beyond ZFS in some areas too, such as being able to efficiently shrink a filesystem, and keeps a lot of the cool things that ZFS made popular, such as Copy-On-Write.

It looks like Btrfs also addresses some decisions that were made with the direction that ZFS would be going in, or how it would handle certain problems that now with hindsight behind the developers, they possibly would have done things differently.

Apple are really struggling with ZFS, with it being announced as a feature in early betas of both Leopard (10.5) and Snow Leopard (10.6), as well as being there in a very limited form in Tiger (10.4) - maybe development on Btrfs will leapfrog ZFS for consumer-grade hardware and Apple can finally look at deprecating HFS.

Re:Looks promising

[dirtyhippie]

... but btrfs is GPL. Therefore Apple can't use it, unless perhaps they are able to work out licensing from Oracle.

Re:Looks promising

[Anonymous Coward]

... but btrfs is GPL. Therefore Apple can't use it

sorry, i don't get it.

Apple already uses gpl software. did you meant "won't use" instead of "can't use"?

Re:Looks promising

[am 2k]

Or implement it themselves. It's only a spec after all.

Re:Looks promising

[PhunkySchtuff]

Apple has, and does, use GPL'd code and complies with the terms of the license.

Take, for example, WebKit, which is a fork of KHTML. It's now released as LGPL:
http://webkit.org/coding/lgpl-license.html

This code powers the browser that Apple ship with Mac OS X, Safari - which is arguably one of the most important pieces of code in the whole OS.

As a result of it's quality, speed and standards adherence, it's now used by companies like Nokia and Adobe...

Re:Looks promising

[Anonymous Coward]

What, really? Who the fuck writes a file system implementation in GPL!? That's as bad as Sun!

Re:Looks promising

[TheRaven64]

Apple are really struggling with ZFS, with it being announced as a feature in early betas of both Leopard (10.5) and Snow Leopard (10.6), as well as being there in a very limited form in Tiger (10.4)

It's also available on 10.5/6 with some limitations. It's not marketed because it's not quite feature-complete, but it works in OS X and on FreeBSD. There's almost no chance of Apple adopting btrfs in the OS X kernel though, because the GPL is incompatible with the license of a number of other components.

Re:Looks promising

[Anonymous Coward]

A browser is probably easier separated from the rest of the OS than a filesystem. (Well, i'm not talking about M$IE).

Re:Looks promising

[aj50]

I think that since it's a part of the kernel, it would count as a derivative work which would mean the whole kernel would have to be GPL'd as well.

This is similar to the reason that ZFS can't just be ported to linux, the code is under CDDL which is incompatible with GPL.

Re:Looks promising

[BitZtream]

IE is easy as shit to seperate from the OS. Delete iexplore.exe.

The rendering engine on the other hand, isn't. It is used by MANY apps, this applies to Windows, OS X, and lots of other OSes.

Re:Looks promising

[Anonymous Coward]

I see what you did there.

Re:Looks promising

[TheRaven64]

The GPL and LGPL are very different. The LGPL does not affect any code beyond that originally covered by the license. You can link LGPL'd WebKit against proprietary-licensed Safari with no problems.

Apple also ship GPL'd software like bash, but they don't link it against any of their own code.

Linking GPL'd code into the kernel would require the rest of the kernel to be released under a license that places no restrictions that are not found in the GPL. That's not a problem for Apple's code; they own the copyright and they can release it under any license they choose. It would be a massive problem for third-party components. DTrace, for example, is heavily integrated into OS X's Instruments developer app and is CDDL (GPL-incompatible). Various drivers are under whatever license the manufacturers want, and are mostly GPL-incompatible. A GPL'd component would need to be very compelling to make Apple rewrite DTrace, most of their drivers, and a lot of other components. Btrfs is not this compelling. Even if Btrfs were sufficiently good, it would take less effort for them to just completely rewrite it than to rewrite all of the GPL-incompatible components.

Re:Looks promising

[PhunkySchtuff]

Damn! That sucks then...

Re:Looks promising

[ppc_digger]

Correct me if I'm wrong, but I thought runtime linking wasn't covered by the GPL. Otherwise, you wouldn't be able to use bash in OpenSolaris, as its libc is CDDLd.

Re:Looks promising

[Runaway1956]

Someone has little understanding of the GPL. Apple need not open source it's kernel in order to use a kernel module. Things might be simpler if MacOS were GPL'd, but the use of GPL'd code within the OS is quite possible. As someone else already points out, Microsoft makes use of GPL'd code.

Re:Looks promising

[Shoe Puppet]

But in this case bash links against libc, not the other way. AFAIR, GPL programs can themselves still link against anything else irrespective of the library's license.

Re:Looks promising

[TheRaven64]

Correct me if I'm wrong, but I thought runtime linking wasn't covered by the GPL.

Okay, you are wrong.

Otherwise, you wouldn't be able to use bash in OpenSolaris, as its libc is CDDLd.

The GPL contains a special exemption for linking against system libraries. This would not apply to linking in the kernel.

Re:Looks promising

[cowbutt]

Apple are also welcome to study the GPLed source of btrfs and develop their own independent, but compatible implementation.

Re:Looks promising

[Haeleth]

It's unclear. The FSF's position is that dynamic linking against a GPL library does produce a derivative work that is covered by the GPL. Some other people disagree. The question has never been tested in court, so the safest option is to assume that the FSF are right.

Re:Looks promising

[ubersoldat2k7]

Even when released under the GPL, I don't think the FSF would go after Apple for including Brtfs on Darwin or MacOSX. They would reach an agreement on licensing and get over it. Well, at least that would be the smartest move so everyone would be happy with, right?

Re:Looks promising

[spitzak]

You can combine GPL code with BSD code no problem. You can still release all the rest of the code BSD. So nothing legal prevents Apple from doing this, unlike the CDDL situation.

However the btrfs code would remain GPL and in effect a Darwin source with it included is GPL. They could probably make it a loaded module so that it is trivial to separate, but if it is the main file system and you assume the main file system is an important part of the kernel, then yes the kernel would be GPL.

Re:Looks promising

[ingenuus]

If linking with the kernel creates a derivative, how can Linux link non-GPL drivers, but a non-GPL kernel cannot link a GPL driver?

Re:Looks promising

[russotto]

It's unclear. The FSF's position is that dynamic linking against a GPL library does produce a derivative work that is covered by the GPL.

Suppose it does. The GPL V2 allows creation of derivative works; the restrictions are on distribution of them. Since nobody distributes the dynamically linked work, there's no copyright violation. The GPL V3 is even more clear on this: "You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force."

So I can link a non-GPL work against a GPL work without restriction. If I then distribute the non-GPL work, without the GPL work, the GPL cannot apply (because I haven't attempted to do anything restricted by its copyright). If the end user then re-links my work against another copy of the GPLed work, there's no violation there either. Distributing or conveying an image of the combined work (e.g. a dump of the process's address space) would be a violation, but that normally doesn't happen.

(and before anyone starts yelling "secondary infringement", there can be no secondary infringement if there is no direct infringement)

Re:Looks promising

[TheRaven64]

Because the GPL is a distribution license. If you link the Linux kernel with a GPL-incompatible kernel module, you then lose the license to distribute the combined work. A company that does not distribute Linux may distribute a binary-only kernel module, as long as that module is not a derived work of the kernel (e.g. it does not use any kernel code, just implements some published interfaces).

Re:Looks promising

[TheRaven64]

What does the FSF have to do with anything? They are not the author of the Btrfs code and so have no standing to do anything. And releasing XNU under the GPL would be complying with the terms of the GPL, so none of the authors of the Btrfs would have any (legal) grounds for complaint either. The fact that complying with the GPL is prohibitively expensive is the problem for Apple, not the FSF.

Re:Looks promising

[dgatwood]

Unless something changed very recently, Mac OS X does not support using filesystem kernel extensions as boot devices, and the booter has no mechanism for extending it to read new filesystems, either. Thus, if it isn't linked into the kernel and understood by EFI, you can't boot from it. That pretty much puts the kibosh on ext4 except as a shared data partition, and if you do that, you might as well use MacFUSE and never have to touch the Mac OS X kernel at all....

Re:Looks promising

[Ilgaz]

I am afraid it will be never feature complete until it does things as Resource forks, Icons, "Open With", 4 char creator codes.

You know, the things "flat unix" people hate but OS X users and especially professionals use. Yes, they do like to change their icons.

If they work on a thing which will be super advanced UFS and nothing else, that is a problem. A serious problem. OS X is a hybrid OS, it is Unix, NeXT and MacOS same time. Anything designed for it must contain things which will make all 3 happy. I will get killed for saying this but case insensitive operation compatibility is a must for any OS X filesystem. I said "compatibility".

The issue with ZFS is the culture of the company, products of them and their focus. I can't imagine explaining a Sun developer about how would user would want to change his application icon and even pay for apps that does it safely.

While I speak about this, I am considering to format an external drive as ZFS but I won't launch any apps from it or do the other Mac things. It will be an expandable pure data drive.

Re:Looks promising

[TheRaven64]

A nice rant, but you clearly have absolutely no idea about the ZFS architecture. All of the filesystem metadata in ZFS is stored at the very top layer. In the Sun implementation, there is ZPL, which provides something that looks like Sun UFS. Apple can use this and store resource forks and so on in POSIX extended attributes, or they can modify it to add more fields that are allocated with every file. It's not like a traditional filesystem, where adding another attribute type requires invasive changes to the inode structure and consequently everything above the block layer. Oh, and you can get a lot of the benefits of ZFS (transactions, snapshots, and so on) by creating an HFS+ volume inside a ZVOL (something designed to look like a block device, backed by ZFS's storage pool rather than a single device). When ZFS was ported to FreeBSD, the first version ran FreeBSD UFS inside a ZVOL.

Re:Looks promising

[larry bagina]

ZFS isn't a traditional file system; it's also an LVM and raid system. So it doesn't fit into the linux model and making it work with linux would require other kernel changes.

Anyhow, file systems are not part of the kernel. They're separate binaries that communicate with the kernel (or with FUSE) through a documented API (ie, the vfs). So if btrfs doesn't require kernel changes, Apple (or a 3rd party) could use the GPL btrfs source to create a GPL OS X btrfs file system.

Re:Looks promising

[Randall311]

Butterface
Btrfs, Btrfs, Everything looks good on her Btrfs!

Re:Looks promising

[Urkki]

Suppose it does. The GPL V2 allows creation of derivative works; the restrictions are on distribution of them. Since nobody distributes the dynamically linked work, there's no copyright violation.

If you distribute both the library and the application, then (under GPL) you're distributing a dynamically linking derivative work of the library, and the entire work must be GPL'd. Also, if you're #including the header files of the GPL'd library when you compile the application, you're putting GPL'd code into your application, with obvious licensing consequences.

So if you write your own function prototypes etc, and don't distribute the libraries with the application, I think there's no legal problem, just the gnashing of teeth coming from GPL zealots, and frustration of users when installing. Quite unpractical, not to mention that distributing an entire operating system but leaving out a library needed by some applications would be rather stupid, but possible... :-)

Then again, IANAL.

Re:Looks promising

[bytesex]

My feeling is that it depends on the availability of other (non-GPL) libraries that provide exactly the same calling interface. That's why it makes no sense to have the (GNU) C library be GPL: I could link against twenty bloody other C libraries if I wanted to. Very specific libraries that are only available under the GPL OTOH, there you have a point. The problem lies with the libraries in between - ssl-related stuff comes to mind; can you (morally) claim a calling interface that is a de facto standard ?

Re:Looks promising

[JesseMcDonald]

I'm no expert, but I believe the FSF's position is that the derivative work is created when you design your software to be linked against a GPL library (making it a derivative of the library's API), not when the actual dynamic linking happens at runtime. If true, that would mean that even the un-linked binaries could only be distributed under the terms of the GPL. The only exception would be in cases where you could potentially link against a non-GPL library with the same API (e.g. motif vs. the GPL'd lesstif), since in that case the API isn't specific to the GPL library.

Note that the FSF's position on dynamic linking is very much on the fringe, and far from unopposed.

Re:Looks promising

[salimma]

Darwin code is under APSL, not BSD. It is incompatible with the GPL.

Re:Looks promising

[gd2shoe]

Anyhow, file systems are not part of the kernel. They're separate binaries that communicate with the kernel (or with FUSE) through a documented API (ie, the vfs).

You've never compiled your own kernel, have you? Yes, some file systems are implemented entirely using FUSE in user space, but key file systems (and a number of misc) are implemented in the kernel. They can optionally be compiled as loadable modules or compiled right into the kernel binary. The article mentions that BTRFS has been accepted into the official kernel tree, implying that it does not rely on FUSE nor run in userspace.

Re:Looks promising

[Ilgaz]

Well, nice information I am hearing for the first time.

You know HFS+ is also fully documented right? The only undocumented part is .DS_Store flat file which Finder uses and nobody really cares too much.

So, if a person reading filesystem specs for entertaintment didn't know about these and only knew about enterprise features of ZFS, perhaps there is a problem with how ZFS is advertised to OS X community and the stuff you talk about not being implemented at all.

Re:Looks promising

[TheRaven64]

You know HFS+ is also fully documented right?

Yes, I've read the documentation. Fully is a bit of a stretch; it's just about enough for an alternative implementation, but it's horribly written. ZFS is also fully documented. If you want to read about it, start here and follow the links. You'll find a complete cross-referenced and explained copy of the source code as well as high-level descriptions.

So, if a person reading filesystem specs for entertaintment didn't know about these and only knew about enterprise features of ZFS, perhaps there is a problem with how ZFS is advertised to OS X community

There's a massive difference between reading filesystem specs and reading marketing material. You'd know about the things I described if you'd read the introduction to the source tour which discusses the three layers in ZFS and what goes in each one. Even if you just looked at the pretty picture (top-right), you'd see that the POSIX compatibility code is in the interface layer, right next to the block-device emulation code, and the transactional object layer and storage pool layers both go under this.

and the stuff you talk about not being implemented at all.

Well, obviously. The port to OS X is not yet finished. If you look at the TODO section on macosforge, you'll see a number of missing bits. Generally, you don't add new features to a port until after you've completed the actual porting part of the job. OS X doesn't really need resource forks anymore; the VFS layer maps them to hidden files if they are not supported by the underlying filesystem and their use is discouraged in favour of bundles by new apps. Creator codes and so on can easily be stored in POSIX extended attributes, so modifying the ZPL may not be the best option; they may just add some transparent mappings in the top of the VFS layer.

Re:Looks promising

[True Grit]

how can Linux link non-GPL drivers

*Legally*, it can't.

For drivers that are built *into* the kernel at compile time, they must be GPL-compatible.

For a non-GPL driver built as a kernel module, however, the issue is more complicated/controversial. For example, the proprietary graphics drivers from NVIDIA/AMD do something like having 2 parts: the part which links with the kernel, and a second non-GPL part, a binary blob, that is loaded at runtime by the first part. Non-GPL drivers are therefore always built as external kernel modules and use this kind of "technique", to avoid using any linkage mechanism between the kernel itself and the binary blob. There are, ah, "purists", though, who consider even this to be a GPL violation, hence the controversy (and the reason for kernel modules having a "Tainted" flag).

Re:Looks promising

[larry bagina]

OS X file systems are not compiled into the kernel.

Re:Looks promising

[Anonymous Coward]

...it's now used by companies like Nokia and Adobe...

...and Google and Palm, and...

Re:Looks promising

[jbolden]

Lots of copyright computer stuff has been tested in court but indirectly. In general the courts have tended not to consider mechanism all that important but to consider intent very important. In other words a dynamic library designed only to be linked with a GPLed work would be GPLed while a library that could work on multiple "platforms" would not be.

If the driver structure for Linux is mostly similar to the one for Windows my guess is that it is fine. If it was designed solely to bypass GPL restrictions the court is going to construe things very narrowly.

Re:Looks promising

[jbolden]

If I then distribute the non-GPL work, without the GPL work

Agreed but Linux distributions would like to distribute both.

total gay

[Anonymous Coward]

i've never lost a file on NTFS, but i have on linux file systems

Re:total gay

[Calydor]

Well good for you. I've had an entire HD decide that moving files to another drive, while deleting other files, meant that I wanted the entire filesystem hosed.

Re:total gay

[Anonymous Coward]

Since you didn't specify what filesystem you were using I'll just assume it was NTFS.

Re:total gay

[pdusen]

I've lost files on lots of different systems, including NTFS and ext3 and the kitchen sink. It's rarely the fault of the filesystem itself.

Re:total gay

[Runaway1956]

I wish the parent hadn't been modded down. He makes a point that should be addressed.

I've lost data on every file system that I've ever used, including NTFS, and the highly touted ReiserFS. Nothing guarantees the security of your data. The nearest you can come to data security, is to backup, backup, and backup again. Those people and organizations that keep regular backups seldom lose data. However, even those people can lose data in the event of a physical disaster (fire, flood, theft, being hit by a humongous meteorite) which is why off-site backups are important.

That said - IMHO, a journaling file system is an important first step to data security. NTFS and Ext3 are about equal, in my experience. Turning off caching features is an important second step. A power outage before data is written to disk, and/or while data is being written results in corruption in all current file systems. The important thing is, if data is mission critical, you want it written IMMEDIATELY, not floating around in RAM.

And, finally, you NEED redundant backups. Anyone who fails to make backups WILL LOSE data, eventually.

Re:total gay

[evilviper]

That said - IMHO, a journaling file system is an important first step to data security. NTFS and Ext3 are about equal, in my experience.

Journaling does NOTHING to improve "data security". Absolutely nothing, either in practice, or in theory. What it does is obviate the need for fsck. It doesn't even do that particularly well, as non-journaled filesystems like FreeBSD's UFS2 can be used much more quickly after an unclean shutdown, and is extremely reliable.

Turning off caching features is an important second step.

No, disabling on-drive caching has to be #1. Without it, no file system can possibly act sanely.

The important thing is, if data is mission critical, you want it written IMMEDIATELY, not floating around in RAM.

Unless the RAM in question is battery-backed, as many higher-end RAID controllers are.

Re:total gay

[jbolden]

Turning off caching is a terrible idea. What you want though is information being written in a way that is recoverable after an outage quickly and rectified slowly. A database based filesystem for example like you see on i-os.

Linus', not Linus's.

[Anonymous Coward]

It's not that hard!

Re:Linus', not Linus's.

[WillKemp]

It's not that hard!

No, it's not - but you're wrong. :"Linus's" is correct. The 's' after the apostrophe only gets dropped in plurals.

Re:Linus', not Linus's.

[aCC]

It's not that hard!

Not necessarily true: http://www.bartleby.com/141/strunk.html#1

Re:Linus', not Linus's.

[Paul Jakma]

It's fine to drop the 2nd s, at least in British english. Though, there doesn't seem to be any hard and fast rule..

Re:Linus', not Linus's.

[compro01]

Both are correct, depending on who you ask. It's a British English vs. American English thing. Up here in Canada, just the apostrophe seems to be the preferred form.

Re:Linus', not Linus's.

[satoshi1]

I was taught in my American schools (grade school, even) that you drop the second S on names that end in S. "Linus'"is how I was taught.

Re:Linus', not Linus's.

[WillKemp]

There doesn't seem to be any hard and fast rules about anything in British english! ;-)

In Fowler's Modern English Usage, which is generally considered to be the bible of english usage by UK journalists and writers, there's an article called "Possessive Puzzles". In that, he says it was "formerly customary" to drop the last 's', but not any more.

If it was formerly customary in Fowler's day, i reckon it must be well and truly archaic now.

Re:Linus', not Linus's.

[WillKemp]

That's interesting. I think in Britain the preferred form of anything is to put an apostrophe anywhere it can possibly be put and drop any syllables that can possibly be dropped.

It seems the version with the final 's' is more modern than the version without it. Considering their efforts to update the language, it's perhaps not entirely surprising that the Americans prefer the more modern form.

Re:Linus', not Linus's.

[WillKemp]

Yeah, that's what Fowler reckons, too.

Re:Linus', not Linus's.

[Anonymous Coward]

gor blimey guvnor you aint arf stereotypin us brits'.

Re:Linus', not Linus's.

[Anonymous Coward]

Gut feelings are no good at all when it comes to language. USians wouldn't have much trouble talking to Edgar Allen Poe, but Brits would have a very hard time hanging with John Keats.

Re:Linus', not Linus's.

[Runaway1956]

The Revelation of Saint Slasher Dotter, chapter 12, verse 19, says, "Verily, in the deepest dungeons, there is a place reserved for grammar nazis. There shall be weeping and gnashing of teeth, as the grus gnaw upon the bones of the grammar nazis for all time. Lo, there shall be no death, no resets or restarts for the grammar nazis, nor will there be relief for them in the lakes of fire. Beware, grammar nazi, or you shall feed the gru!"

Re:Linus', not Linus's.

[Anonymous Coward]

> It's not that hard!

That's what she said!

Re:Linus', not Linus's.

[kdemetter]

Can i have a copy of that book ?

Re:Linus', not Linus's.

[Bacon Bits]

Except that English is defined by how it's used not by some arbitrary authority, in spite of what those boys up at Oxford will tell you. The fact that people use both "Linus's" and "Linus'" means that both are acceptable. By the same token, "ain't" and "y'all" are now recognized words because people use them and nobody is confused by meaning. "Google" is now a verb, as well.

If I say something in what I believe is English and you understand it unambiguously as English, then it's English or an accepted dialect.

Re:Linus', not Linus's.

[siride]

In spoken English, you generally pronounce the second 's' (unless you are a pedant of some sort), so it would stand to reason that the second 's' should remain. There is another motivation: the "'s" is actually a clitic that attaches to phrases (usually noun phrases) and is thus a separate word, not a part of the word it is attached to. As such, it should always be spelled out (as it is always pronounced).

Re:Linus', not Linus's.

[gardyloo]

Except that English is defined by how it's used not by some arbitrary authority, in spite of what those boys up at Oxford will tell you.

Txt spkrs rejoiced. Thus started the revolution. Everyone left after the MegaWar was a freak, except for a small band of roving former English professors. These kept the language alive with campfire stories, in the vein of The Canterbury Tales, until one day, the world could rebuild . . ..

Re:Linus', not Linus's.

[larry bagina]

Elements of Style says:

Form the possessive singular of nouns with 's.

Follow this rule whatever the final consonant. Thus write,

Charles's friend
Burns's poems
the witch's malice

This is the usage of the United States Government Printing Office and of the Oxford University Press.

Exceptions are the possessives of ancient proper names in -es and -is, the possessive Jesus', and such forms as for conscience' sake, for righteousness' sake. But such forms as Achilles' heel, Moses' laws, Isis' temple are commonly replaced by

the heel of Achilles
the laws of Moses
the temple of Isis

The pronominal possessives hers, its, theirs, yours, and oneself have no apostrophe.

---

I think we can all agree that Linus is on par with Jesus and therefore a single apostrophe is correct.

Re:Linus', not Linus's.

[Bacon Bits]

In spoken English, you generally pronounce the second 's' (unless you are a pedant of some sort)

This is Slashdot, sir. Pedantry is what we serve.

Re:Linus', not Linus's.

[evilviper]

In spoken English, you generally pronounce the second 's'

No, you pronounce 'es', yet nobody is advocating the use of es instead of 's everywhere just because it's pronounced that way...

It stands to reason that what stands to reason doesn't determine the rules of written english.

Re:Linus', not Linus's.

[Paul Jakma]

Well, I was taught to drop the second s, and I'm not /that/ archaic.

So,

[Josh04]

Is this ever going to replace ext4? The ext series of file systems are 'good enough' for most people, so unless it has some epic benchmarks I can't imagine a huge rush to reformat. Maybe that's what drives file system programmers insane. The knowledge that for the most part, it's going nowhere. FAT12 is still in use, for Christ's sake.

Re:So,

[PhunkySchtuff]

Aside from Copy on Write, one other feature that this filesystem has that I would consider essential in a modern filesystem is full checksumming. As drives get larger and larger, the chance of a random undetected error on write increases and having full checksums on every block of data that gets written to the drive means that when something is written, I know it's written. It also means that when I read something back from the disk, I know that it was the data that was put there and didn't get silently corrupted by the [sata controller | dodgy cable | cosmic rays] on the way to the disk and back.

Re:So,

[borizz]

Snapshots are nice too. Makes stuff like Time Machine and derivatives much more elegant. ZFS has built in RAID support (which, I assume, works on the block level, instead of on the disk level), maybe Btrfs will get this too.

Re:So,

[Anonymous Coward]

Our storage consumption is exponential, so even if you just format *new* drives with Btrfs, it will become a significant player very soon.

Re:So,

[Josh04]

It's not about getting devices formatted with it, it's about getting platforms to support it. Will there be a Windows driver for btrfs? Can I put it on an external hard disk and actually be able to use it?

Re:So,

[joib]

ZFS has built in RAID support (which, I assume, works on the block level, instead of on the disk level), maybe Btrfs will get this too.

Yes, btrfs currently has built-in support for raid 0/1/10, 5 and 6 are under development.

Re:So,

[Anonymous Coward]

If that's so, then why did you say "I can't imagine a huge rush to reformat" instead of "I can't imagine many platforms supporting it"?

Re:So,

[zsau]

People buy new computers often enough. For Btrfs to replace ext4 (I'm still using ext3 and didn't even realise an ext4 had been released!), I think all it will take is for major distributions to change the default file system for new installs. Obviously the number of people who replace existing file systems different ones will be comparatively low.

Re:So,

[am 2k]

Doesn't help against RAM issues though, because those will just get into the checksum as well.

Re:So,

[aj50]

I had this exact problem very recently.

If my data was important, I should have been using ECC RAM.

Re:So,

[BrentH]

What I'd like to know if btrfs does continuous checking of these checksums, preferably when there's not a lot of activity. Checksums are an excellent idea, but unless you check your files every now and again (automagically), you still don't know anything.

Re:So,

[xororand]

Waiting for native Windows filesystems is mostly hopeless, backed up by this incomplete list. As far as I know, the Windows IFS development kit is not free, neither as in speech nor as in beer.
A pragmatic solution would be to use virtualization instead of badly rewriting filesystems again and aigan. You'd need a small virtual machine:

• The VM should contain a minimal Linux setup that shouldn't need than 64 MB disk space, using cramfs or squashfs, like LiveCDs do

• 64 MB of RAM ought to be more than enough as well.

• The hostp passes the raw drive/partition that you want to mount through to the VM
• The filesystem is mounted inside the VM, supporting even LVM2, dm-crypt / LUKS, ... whatever you want.
• The mounted filesystem gets exported back to Windows via CIFS (SMB successor)

I'm sure one could make this process very easy to use, with a neat GUI to set it up. The VM could run in the background, so that the user only sees its rough status in the GUI.
Maybe there's hope for a fast native solution, as Microsoft just recently released Linux guest drivers for their own virtualization solution under the terms of the GPL2.

Re:So,

[PhunkySchtuff]

What you do know is that when you read a block of data back from the disk, that block is what was supposed to be written to the disk.

If a file that is never read is corrupted somehow, then you will only discover that corruption when you read the file.

Having checksums is very good if you have a RAID-1 mirror. With full block checksums, you can read each half of the mirror and if there is an error, you know which one is correct, and which one isn't. At present, if a RAID-1 mirror has a soft error like this, due to corruption, you don't know which half of the mirror is actually correct.

With ZFS, for instance, you can create a 2-disk RAID-1 mirror and then use dd to write zeroes to one half of the mirror, at the raw device level (ie, bypassing the filesystem layer) and when you go to read that data back from the mirror, ZFS knows that it's invalid and instead uses the other side of the mirror. It then has an option to resilver the mirror and write the valid data back to the broken half, if you so want.

Re:So,

[joib]

Would would a windows driver be a requirement? I, and presumably many others, currently use ext3 regardless of the existence of a windows driver (such a thing supposedly exists, but I have never needed it). For windows access there is samba, after all (via a virtual machine if it needs to be done on the same piece of HW).

For external drives the situation is indeed slighly different. I typically use either ext3 or ntfs depending on whether I want to access them from windows or not.

Re:So,

[swilver]

If the checksumming is done by the CPU, on non-ECC memory, you might as well not use it as the data is most likely going to get corrupted at the source (your memory) not in transfer.

The biggest source of bit errors at the moment is non-ECC memory as far as I can tell. Most busses are already protected in some form due to their high-speed nature. Hard drives themselves use many forms of error correction routinely to even read any sane data at all.

On my own system I noticed a problem when copying large amounts of data (which was checksummed in some other fashion) that I'd get about 1 bit error for every 100 GB of data copied. This came up fairly often, and is the reason I now always use mv/cp tools that can do a verify. After I replaced the memory with ECC variants however these tools haven't reported an error since.

Re:So,

[swilver]

Surface scans with SMART already catch these problems (blocks going bad).

There's no need to check for single bit errors in files / blocks themselves as that's never gonna happen when the block was either fully error corrected by the drive or simply not returned at all due to a checksum failure (in which case it's a bad block).

If a checksum DOES fail in such a setup (with the filesystem doing the check), it is actually more likely that it was damaged in memory (which could be fixed by simply reading it again). Damage in memory is sinister though as you never know when a (cached) block gets damaged, which could be never or right after the checksumming process declares it "correct".

If you are really worried about these issues, I'd first invest in an ECC capable motherboard and ECC memory.

Re:So,

[AzureDiamond]

As far as I know, the Windows IFS development kit is not free, neither as in speech nor as in beer.

You can download the Windows Driver Kit for free, and that includes the Installable Filesystem Kit headers and libraries and the source code to FASTFAT.SYS, the Windows FAT driver and CDFS.SYS, the ISO9660/Joliet filesystem.

http://www.microsoft.com/whdc/DevTools/WDK/WDKpkg.mspx

That being said writing a performance filesystem for Windows is much less easy than for Linux.

Re:So,

[borizz]

Odds are the checksum then won't match anymore and you'll be notified. It's better than silent corruption.

Re:So,

[borizz]

At least with a checksumming FS you'll be notified of the error, even if it happens in RAM. That way you atleast know something is not right.

Re:So,

[Anonymous Coward]

ZFS has built in RAID support...maybe Btrfs will get this too.

I would hope not. Software RAID does not belong in the filesystem. Putting it there is a layering violation. Linus would unlikely accept such a ham-fisted implementation.

Re:So,

[Anonymous Coward]

incorrect. just last week i helped a customer who suffered exactly
this fate. her drive miswrote the data and did not return a ure on
reread. it just returned bad data. needless to day, it took a long
time to find the true problem. (i have a similar drive which at least
had the decency of returned ure after the block had been flushed
from the cache, but writes appeared to be successful.)

Re:So,

[asaul]

Read the ZFS White Paper. Just because the disk checks its blocks, doesn't prevent other sources corrupting, overwritting or generally tampering with data. For example, say your el cheapo fibre card corrupts one bit in every 2 billion writes - on disk its fine, SMART never sees it, it never complains. When ZFS reads the corrupted blocks it will see a checksum error, and repair if necessary.

It also doesn't cover the case of deliberate/administrative corruption such as accidentally overwriting the wrong disk etc. With a normal mirrored device you could read off either side, it simply returns the data and the data would be blockwise correct. With ZFS again you would see failures and if possible it could correct. In fact this is how the early demonstrations of ZFS would work - simply using dd to clobber one half of the mirror and watching it fix itself.

And for ZFS I would absoultely recommend ECC memory. For the exact case that I had a blown capacitor cause random memory errors on a motherboard I had, and any new or modified file would throw up checksum errors when re-read. Without ZFS I probably would not have known until I got some weird panics from corrupted metadata or something.

Re:So,

[ioshhdflwuegfh]

Aside from Copy on Write, one other feature that this filesystem has that I would consider essential in a modern filesystem is full checksumming. As drives get larger and larger, the chance of a random undetected error on write increases and having full checksums on every block of data that gets written to the drive means that when something is written, I know it's written. It also means that when I read something back from the disk, I know that it was the data that was put there and didn't get silently corrupted by the [sata controller | dodgy cable | cosmic rays] on the way to the disk and back.

You mean like ZFS does it?

Re:So,

[PhunkySchtuff]

Yes, exactly. I'll happily use ZFS wherever it's supported to do so. Btrfs is looking like a promising alternative and looks to be happening on Linux long before ZFS will be.

Re:So,

[Anonymous Coward]

Just the name is enough for me. "Butterfly system" goes so well with my "Pink ponies!" theme ^_^

Re:So,

[Macka]

The layering in Btrfs is done internally. As the Wiki says:

Does the Btrfs multi-device support make it a "rampant layering violation"?
Yes and no. Device management is a complex subject, and there are many different opinions about the best way to do it. Internally, the Btrfs code separates out components that deal with device management and maintains its own layers for them. The vast majority of filesystem metadata has no idea there are multiple devices involved.
Many advanced features such as checking alternate mirrors for good copies of a corrupted block are meant to be used with RAID implementations below the FS.

And Linus has obviously already accepted this, as Btrfs is part of the Linux kernel now.

 

Re:So,

[Macka]

The author of the article thinks so. He's predicting btrfs will be the default Linux filesystem 2 years from now.

Re:So,

[anilg]

ZFS raid works on disk level. When you create a pool.. you have to specify the disks (either as mirrors or parity). You can offline a disk if required (due to disk error or if you need to back it up somewhere).. these make them very interesting in certain applications. Nexenta, for example, has Devzones, where the root filesystem is on it's own ZFS dataset.

Re:So,

[toby]

Who was talking about single bit errors?

SMART is irrelevant here. You need to study ZFS a lot more.

Re:So,

[am 2k]

No, I was talking about alterations that happen before the checksum is calculated.

Re:So,

[mysidia]

There doesn't have to be. Many popular UNIX filesystems do not have Windows drivers.

For example: UFS/FFS are some of the most popular file systems.

There was also ext2/ext3, which did get a Windows program available for read-only access, but long after the filesystem was widely used.

Re:So,

[Anonymous Coward]

Yes, obviously. GP is a mong.

Re:So,

[Mad Merlin]

Will there be a Windows driver for btrfs?

No.

Can I put it on an external hard disk and actually be able to use it?

Yes.

Re:So,

[adolf]

Such alterations would be caused by either a coding issue, or a RAM/CPU issue, and would slip right through. Checksums in this context only work to protect (or at least advise upon) data once it is on disk.

If have reason to be paranoid enough to worry about such things, you should at least be using ECC RAM as well. CPU bugs (on stock clocks) almost never happen these days, and when they do, they're generally a fault which affects all CPUs of that type and are patched around pretty quickly in software.

Re:So,

[phoenix_rizzen]

What you do know is that when you read a block of data back from the disk, that block is what was supposed to be written to the disk.

If a file that is never read is corrupted somehow, then you will only discover that corruption when you read the file.

With ZFS, you can do a manual zpool scrub at any time, and it will go through all the data on the disk(s), check the file/metadata/block checksums, and repair any corrupted blocks using redundant data from elsewhere in the pool. This is a low-priority task that can run in the background, although running it at a time when there is no other I/O will allow it to complete quicker.

Thus, you can check for corrupted data in files that aren't being used, or haven't been used in a long time, or that won't be used for a very long time. No need to wait.

Re:So,

[True Grit]

Is this ever going to replace ext4?

FWIW, Ted T'so, the maintainer of ext3/ext4, himself said somewhere he expected, or could envision, btrfs replacing ext4 when the time came.

Only question is timing, btrfs isn't at v1.0, and isn't stable, but it seems to have a lot momentum behind it, compared to all the other advanced Linux file systems out there.

Re:So,

[BrentH]

That's exactly what I mean, and like about ZFS. Does BTRFS have such a thing too?

mod parent up

[Anonymous Coward]

Parent is truthful. Eat it linuxtards!

So, what is the status of btrfs?

[MMC Monster]

Is it Beta? The fact that Linus runs it as his root fs doesn't tell me much. Now, if you told me that's what he uses for ~/, I would be more impressed.

The important question to me is, how long 'til it gets in the major distributions?

Re:So, what is the status of btrfs?

[joib]

The important question to me is, how long 'til it gets in the major distributions?

The article predicts a couple of years until it's safe enough as default in new distros.

Re:So, what is the status of btrfs?

[Anonymous Coward]

As the article notes, there has been a disk-format change pushed for the 2.6.31 kernel release. I'm not sure if Chris has announced that format to be final. If it is, I think you could consider btrfs in beta from the 2.6.31 release onwards. In its current form, I'd say it's still alpha-code.

As for major distributions, Debian is shipping btrfs-tools in Squeeze (lenny+1). Not that you can use it, since squeeze is still shipping a 2.6.26 kernel. Don't know about other distributions.

Re:So, what is the status of btrfs?

[BlackCreek]

Chris Mason said that this should be the last unmaintained binary format change http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg02502.html

Re:So, what is the status of btrfs?

[TheRaven64]

Meanwhile, FreeBSD and OpenSolaris are shipping with a version of ZFS that is usable now...

Re:So, what is the status of btrfs?

[joib]

Yeah, but those operating systems suck in other ways, so no thanks. I'll get by with ext3/4 and xfs until btrfs is production ready. YMMV.

Re:So, what is the status of btrfs?

[jonaskoelker]

The article predicts a couple of years until it's safe enough as default in new distros.

It's phrased as an upper bound, "Btrfs will be the default file system on Linux within two years."

Note that it's the prediction of a single person, Valerie Aurora-formerly-Henson, who doesn't try to explain any rationale behind the prediction.

She even says "Check back in two years and see if I got any of these predictions right!" ... So take it with an appropriate amount of salt*, whatever that means to you.

(* please consult your doctor before consuming large amounts of salt if you have high blood pressure or a pre-existing heart condition.)

Re:So, what is the status of btrfs?

[TheRaven64]

Yeah, but those operating systems suck in other ways

Yes, in a great many of other ways, that somehow can't be put into words...

Re:So, what is the status of btrfs?

[BitZtream]

Name one so everyone can rip you a new asshole for being wrong.

Re:So, what is the status of btrfs?

[joib]

Just because a replied to your snarky message with another equally snarky one, doesn't mean I'm not able to put it into words. For instance, a few reasons why I prefer Linux over *BSD or Solaris:

- better package management

- better hw support

- better ISV support

- the uncertain future of Solaris (after all, Sun got bought because they were bleeding red ink left and right, will the Solaris devs escape the inevitable layoffs and Oracle continue pumping money into Solaris development just to try to keep up with Linux?)

- Lack of tier-1 commercial support for *BSD.

- Much larger community

- Better availability of qualified Linux sysadmins

Re:So, what is the status of btrfs?

[Enleth]

"Real men don't use backups, they post their stuff on a public ftp server and let the rest of the world make copies."

  - Linus Torvalds

Re:So, what is the status of btrfs?

[ioshhdflwuegfh]

[...]why I prefer Linux over *BSD or Solaris:
[...]
- better hw support
[...]

To you then file system is not something that supports hardware!?

Re:So, what is the status of btrfs?

[asaul]

For hardware support it really depends what segment of the market you are arguing about. If you are talking white box, low end mostly self supported stuff then no doubt, Linux wins hands down. But as a sysadmin I find Linux to be the of the most painful platform to work on compared to Solaris or AIX - predominantly because of the lack of standardised, stable and properly supported management interfaces.

Fibre channel support is a joke. Sure, for the most part you can dynamically bring stuff in and out, and udev goes a short way to bringing some consistancy. The problem is when something goes wrong you are left with pretty much just rebooting - messages tell you nothing - is the device there or not? Usable details are buried away in /proc and /sys and typically are only useful for developers. Solaris and AIX had cfgadm/cfgmgr and lsdev and friends to tell you what state things are in or what has happened. There are useful and informative error messages (typically). So far on RHEL 3/4/5 all I ever see is odd octal dumps from drivers when errors occur, and wierd hangs and IO errors when devices get broken. It gets worse as you change fibre drivers and versions. Options which exist in one disappear in others. Vendor drivers add customisations which cause other issues.

The lack of stablity in terms of being able to do things between versions gets me as well. On AIX/Solaris you write a script for Solaris 8, and it just works going forwards to other versions. Solaris 10 changes things a bit, but for the most part you can still poke around the same places or the same way to get info back. In short they tend not to break things that work.

Linux goes the other way - a change is made, and thats that, it seems to be up to you to either track or figure it out. You find yourself having to customise things for many many variations of platform - not just major versions, but minor versions as well. Changes to config file locations, the ways those files are defined etc.

Don't get me wrong, I got into UNIX on Linux and I wont dispute its strength in drivers or community, but that community is not "Enterprise" focused. Its why I use it for my PVR and not my file server. The rapid changes in Linux are why the DVB-T cards I got became supported so quickly after the hardware changed. I get the differences, but its not one size fits all.

Re:So, what is the status of btrfs?

[Anonymous Coward]

Just because a replied to your snarky message with another equally snarky one, doesn't mean I'm not able to put it into words. For instance, a few reasons why I prefer Linux over *BSD or Solaris:

- better package management

...

BWAAAHAAAAHAAAHAAAA!!!!

Yeah, right.

I want to see you back out a series of patches on Linux and revert to the previous configuration because the updates broke something.

Re:So, what is the status of btrfs?

[Anonymous Coward]

- better package management

Value judgement. Personally I like FreeBSD's Ports better than anything else out there. You can do either binary installs (pkgadd) or from-source builds with the exact functionality you need. YMMV.

- better hw support

For ("enterprise") servers I would it would be a wash between Solaris and Linux.

- better ISV support

For BSD probably, for Solaris much more debatable.

- Lack of tier-1 commercial support for *BSD.

This is a check box requirement for the suits IMHO. We have a Red Hat contract and they're completely fucking useless. Besides hardware repairs, I've found vendor support to be not useful regardless of which company I'm dealing with.

- Better availability of qualified Linux sysadmins

Debatable. I would say a good sysadmin is OS agnostic. Unix is Unix, it's methodology and mindset that are most important IMHO.

Disclaimer: I have admined all three and currently do mostly Solaris and Linux. My perfect system nowadays would be the underlying OS of Solaris 10, FreeBSD's Ports, with the polish / usability of Mac OS X. If I can't have that, I'd have Solaris 10 on the server and OS X on the desktop. Linux is good, but there are little things here and there that often make it really annoying to work with (e.g., man pages that are missing or completely useless). Make your bed and sleep in it.

Re:So, what is the status of btrfs?

[ion.simon.c]

Yeah, right.

I want to see you back out a series of patches on Linux and revert to the previous configuration because the updates broke something.

# echo =package-cat/package-offending-version >> /etc/portage/package.mask
# emerge -C =package-cat/package-offending-version && emerge package-cat/package

Rinse and repeat for any other packages which may be borked.

Re:So, what is the status of btrfs?

[Anonymous Coward]

Yeah, right.

I want to see you back out a series of patches on Linux and revert to the previous configuration because the updates broke something.

# echo =package-cat/package-offending-version >> /etc/portage/package.mask
# emerge -C =package-cat/package-offending-version && emerge package-cat/package

Rinse and repeat for any other packages which may be borked.

Fail.

1. Not available on the majority of Linux installations
2. Removing a package is not the same as reverting to an earlier version of the same package.

Re:So, what is the status of btrfs?

[Wonko the Sane]

1. The point is that the requested feature is available on Linux. Just because a majority of Linux users don't (yet?) use Gentoo does not invalidate this.

2. The commands he posted <em>will</em> revert to an earlier version of the same package.

Re:So, what is the status of btrfs?

[Anonymous Coward]

I still put more stock into this prediction than I do into Gartner reports.

Re:So, what is the status of btrfs?

[FooBarWidget]

"I want to see you back out a series of patches on Linux and revert to the previous configuration because the updates broke something."

The updates *don't* break things, unlike on FreeBSD where I had to fight breaking updates throughout the entire year in which I used it. No thanks, I'll stick with Linux.

Re:So, what is the status of btrfs?

[ion.simon.c]

Fail.

1. Not available on the majority of Linux installations

Something similar seems to be available in APT:
http://www.debian.org/doc/manuals/apt-howto/ch-apt-get.en.html
Check section 3.10.
And here's the rough equivalent for RPM:
http://www.linuxjournal.com/article/7034

So, what distro is no longer covered?

2. Removing a package is not the same as reverting to an earlier version of the same package.

I guess that you missed the latter half of the last command that I posted:

# emerge -C =package-cat/package-offending-version && emerge package-cat/package

An English translation of that command is

Remove the offending package and install the latest available that's not masked if the removal was successful.

I could have written that command as:

# emerge package-cat/package

and -as I had previously masked the offending package version- Portage would have done the right thing.

So, in summary:

No, you're a towel.

:D

Re:So, what is the status of btrfs?

[mysidia]

Except what happens when you don't necessarily know which package is broken, or perhaps emerge or the kernel itself is what is broken?

Possibly the update of a package to the broken version didn't entirely complete -- some of the files were getting changed, and the system crashed while the update was in-flight.

OpenSolaris, and specifically Nexenta with 'apt-clone' can deal with these situations, emerge cannot.

Re:So, what is the status of btrfs?

[mysidia]

- better package management

Take Nexenta's implementation of OpenSolaris, for example: the package manager is 'dpkg' and 'apt-get', just as with Debian Linux. There's also a Debian GNU/NetBSD port of Debian. If you want to keep Linux-style package management, you have good options.

BSD also has decent package management... NetBSD has pkgsrc, FreeBSD has ports, which allow you to very easily install software from source code, lots of software is available in this form that you'd have a hard time finding on Enterprise Linux distros.

Of course, binary packages are available as well.

- better hw support

If this is the most important thing to you, then perhaps you should stick with Windows server when using commodity processors, because Windows is what nearly all hardware vendors say their product is designed for.

Choose hardware your OS works on then; I never had the least bit problem doing that with BSD or Solaris; it's not as if hardware required to run Solaris or BSD is necessarily more expensive. Sun publishes a compatibility list OpenSolaris (HCL). FreeBSD/NetBSD work just fine on most commodity hardware.

In fact, NetBSD has been known to have ports for more platforms than others, it even runs on a fair number of processor architectures that Linux hasn't been ported to.

- Much larger community

If this is the most important thing to you, then you should run Windows, because Windows has the absolute largest community.

The Linux community having grown so much does not necessarily mean that the other OSes suck, or that Linux is magically better.

- better ISV support

This is only useful if you are actually running independent software. Most UNIX users are running common open source software, that they've installed from package repositories, or compiled themselves.

- Better availability of qualified Linux sysadmins

I'll agree there's better availability of Linux sysadmins, not necessarily that they are best-qualified, and this is a valid reason to choose Linux, but not a reason that other options suck.

A well-qualified sysadmin should be able to work in Linux, FreeBSD, Solaris, etc, once they have done some basic reading up on the differences and the OS of choice's documentation.

The administration of Solaris is pretty similar to the administration of BSD, which is also very similar to the administration of Linux.

Much of the software is the same. The main differences are in some system monitoring and diagnostic tools that OSes other than Linux have more of, and Linux users rarely use.

In fact, a major reason Linux sysads aren't automatically qualified Solaris sysads, is because Solaris provides tools that are a lot more useful for effective system administration.

Other OSes provide fault management, troubleshooting, diagnostic, and monitoring capabilities that Linux does not.

E.g. Linux has 'strace' that is very dangerous to use in practice (if you can help it, you don't trace or debug processes on production systems); Solaris has 'truss' or 'dtrace' that are fairly safe to use: you can transparently instrument a production system, whereas with Linux you cannot.

Solaris has 'fmadm' for fault management, plus 'smf' for service management, including fault-managed services and e.g. automatically restarting apache, and many kernel tools for tuning settings without rebooting; on Linux, there is no fault management, and if you want to tune things, you're probably going to need a kernel recompile.

Oh yeah, and on Linux, if you want to automatically re-start BIND or Apache should it crash, you will probably be hacking together a non-standard cron script, or installing some random third-party package and hacking together a script to make it happen.

Re:So, what is the status of btrfs?

[ion.simon.c]

Except what happens when you don't necessarily know which package is broken,

If you don't know what package is broken, then no, you can't ask emerge to select a particular package to remove. I suppose that you're either looking for a tool that would analyze emerge's logs and roll back the builds that happened during a particular time period, or you're looking for whole-system snapshotting. The former would not be a hard tool to cook up. The latter can be achieved with LVM snapshots. Neither technique is (IIRC) currently integrated in emerge.

or perhaps emerge or the kernel itself is what is broken?

a) Are you *seriously* saying that you would perform a kernel upgrade without keeping a copy of the previous revision around to boot with in case of emergency? 'Cause I can't see any other way that a broken kernel update could cause any problems beyond the downtime needed to reboot twice.
b) You really don't know how package updates are handled in Gentoo, do you? Lemmy explain a thing:

If avoiding broken packages is your goal, then you're running what the Gentoo folks call a Stable system. All potential upgrades and changes to a package have to sit in the Testing branch for at least a month before they become available for Stable users. If an update to emerge completely breaks emerge, then the only folks who will get nuked are those who are using the Testing packages. [0]

Possibly the update of a package to the broken version didn't entirely complete -- some of the files were getting changed, and the system crashed while the update was in-flight.

The fix for this is simple:

# emerge package-which-failed-to-merge-completely

But really, the only way that a package could fail to merge is if the system lost power during the install phase of the emerge... that is the phase where emerge does a "cp" of all of the files that it has just built to their final destination on disk. [1]

Aaaaaannnnnd, if your system is so broken that it can't boot, then you can boot from a CD and fix things from there.

Also, I read the manpage for apt-clone. We can duplicate that behaviour with LVM2 snapshots. I've never needed this behaviour, though.

[0] Folks running the Testing branch of Gentoo *don't* get catastrophically broken versions of emerge pushed down to them... it doesn't happen. The first rule of changing your package manager is "Don't break your package manager."
[1] And I'm not 100% sure that even this is catastrophic. emerge might have some way to determine if an Install was interrupted in the middle of things and be able to recover. I've never needed this functionality, so I've never had a chance to find out.

Re:So, what is the status of btrfs?

[mysidia]

Most servers have all disk space automatically allocated during installation, so there's no free LVM2 space; by contract, Opensolaris / Nexenta pre-upgrading snapshotting does not require installation-time planning or early reservation of disk space for upgrades.

Linux doesn't include any tools to use LVM2 snapshots to assist with this, or perform the updates to 'grub' to allow booting into the old OS version; as a result, it would not be a trivial task to use LVM2 snapshots to offer upgrade rollback, or to perform the rollback if needed, not in an automated manner suitable for repeating during an upgrade rollout on many servers that may have varying disk space allocation choices.

The existence of a LVM2 snapshot of a volume seriously degrades the disk write performance of the live filesystem, it also has the problem that disk space has to be set aside for it in advance, and the snapshot "deactivates" / ceases to be usable, if the estimated amount of required disk space is exceeded.

Re:So, what is the status of btrfs?

[jafo]

The thing that makes me sad is that Linux is basically the only OS out there that isn't shipping with a root file-system that supports snapshots and other advance features. dm snapshots are useful for some things, but are a far cry from what you get from having it in the file-system.

btrfs sounds great and all, but it's going to be a few years before it gets to the point where I think it'll be usable for production systems. It would have been nice if we could have had ZFS in the interim, but that just isn't going to happen.

Sean

Re:So, what is the status of btrfs?

[WuphonsReach]

btrfs sounds great and all, but it's going to be a few years before it gets to the point where I think it'll be usable for production systems. It would have been nice if we could have had ZFS in the interim, but that just isn't going to happen.

Yah, sometimes you don't get the pony for Christmas.

(On the upside, I'm glad that there are new filesystems coming, such as ext4 and btrfs. A lot of the stuff that ZFS does, such as checksumming, could probably be handled by the LVM layer. 80-90% of our server's filesystems are already setting on top of LVM anyway.)

Re:So, what is the status of btrfs?

[Anonymous Coward]

What a trollish statement to make. Hardware support is a much, much broader subject than file system support. If I can't use my video/sound/network/telephony/etc card, then the file system is immaterial. On the other hand, there are alternatives to ZFS. You might not like them as much, but they exist and work.

Besides, file systems themselves are purely logical entities, not hardware ones. It's making them efficient and reliable that requires supporting hardware. (It's a fine distinction, but a relevant one.) You could, for example, mount a filesystem from a loopback file living in a ramdisk (no hardware involved beyond the CPU and RAM). It could be EXT2/3, NTFS, or (theoretically) ZFS. See, it's not really a hardware support thing after all.

Re:So, what is the status of btrfs?

[jipn4]

The thing that makes me sad is that Linux is basically the only OS out there that isn't shipping with a root file-system that supports snapshots and other advance features.

I don't really care. Ext2/3 does everything I need.

It would have been nice if we could have had ZFS in the interim, but that just isn't going to happen.

I don't really care. I tried ZFS and it seemed unnecessarily complex.

Re:So, what is the status of btrfs?

[jipn4]

OpenSolaris, and specifically Nexenta with 'apt-clone' can deal with these situations, emerge cannot.

It also protects against unexpected attacks by killer penguins and having all your clothes disappear spontaneously while speaking in front of a crowded room. And it is about as useful.

Re:So, what is the status of btrfs?

[mysidia]

In fact, it's very useful to be able to completely reverse an upgrade to an automatically captured image if the worst should happen.

It's a highly useful feature, especially for enterprises. And ignorance of its usefulness only shows how ignorant some Linux users are to the needs of the enterprise. For Linux to be taken seriously by an entire class of user, it should provide some additional features, INCLUDING online fully image-revertible upgrades.

It eliminates almost all the risk of updating. I have had too many Linux systems that I tried to update using the package manager only to find that the update process completely b0rked and broke lots of things, or that the system wouldn't start up properly after the update.

Mistakes happen. New software deployments don't always work like they should.

Re:So, what is the status of btrfs?

[deek]

For Debian APT systems, the website you listed does not describe rolling back versions. Pinning a package is not the same as reverting a package.

There is a way, but it is a manual process. Firstly, locate the previous package in /var/cache/apt/archives/ . Then run the command "dpkg -i --force-downgrade" with that package as an argument. The package should then be downgraded.

This will not work with downgrades like moving back database major versions, or other systems that significantly change data or config files on upgrade. But then again, you really should be backing up data before doing an upgrade of this scale. If the upgrade does not work, stop the service, revert the package, and restore the old data.

Re:So, what is the status of btrfs?

[DarkEmpath]

For instance, a few reasons why I prefer Linux over *BSD or Solaris:

- better package management

Really? Ports is one the best package management systems I've ever used. I hated the linux version of "dll hell" I always ended up with on Debian (yet never on Windows!)

- better hw support

Again, I've always had better hardware support on Free/PC-BSD. Most linux distribution have an irrational aversion to binary drivers/blobs, or legally can't because of GPL issues. The BSDs don't suffer from that, and I've always had an easier time with drivers for that reason.

- Much larger community

You'll be wanting Windows, then ;-)

- Better availability of qualified Linux sysadmins

I've never needed so much help as when I was on linux. BSD always seemed more consistent, and therefore intuitive once I got started.

Diff'rent strokes, I guess.

BTRS vs NILFS

[Anonymous Coward]

How do they compare?

LWN -- it should be noted...

[Slartibartfast]

That this is a "service" provided by LWN so that non-subscribers can read premium content; this story would be free for all come Thursday, but apparently "diegocgteleline.es" didn't feel compelled to mention that, that LWN's weekly page is premium content, and that premium content subscribers help LWN stay afloat -- when it's almost gone under a couple times.

Re:LWN -- it should be noted...

[joib]

It's already free; it's from last week's LWN weekly news.

Looks quite robust

[Enleth]

With the COW-enabled b-tree storing everything including metadata and packing it in the same block as the data it describes, the atructure looks quite similar to reiserfs (v3) in terms of error tolerance and recovery. Should this get a tool like the reiserfsck --rebuild-tree, I'm switching - this single feature (well, and some quite sensible performance) is keeping reiserfs on my systems. Saved me a lot of grief several times, when an ext filesystem would be a totally lost cause (or lots of $$$ for a data recovery company), without any built-in utilities, and really no way to write them, that search the entire disk for anything that looks like filesystem metadata and try to make sense of it, even with all superblock structure completely missing and the rest spotted with garbage.

Re:Looks quite robust

[physburn]

Similar to reiserfs in terms of tolerance. Oh great, thats safe, just don't marry it. :-), Look up reiserfs on slashdot, if you don't get that. Seriously though thanks for all the hard work from Linus developers.

---

Linux Feed @ Feed Distiller

Re:Looks quite robust

[Anonymous Coward]

...the atructure looks quite similar to...

I feel like that word should mean some combination of "atrocity" and "structure".

Beta, indeed.

[Slartibartfast]

I asked when it would be usable for "people who backed up their data" about a year ago -- which is about how long I've been using it -- and the answer was, "No firm date." If you load up a 2.6.31 kernel, the commits have reached the point where not only shouldn't you see significant on-disk format changes, but that the bulk of non-RAID tweaking to occur is probably performance related. (RAID is coming, but it's only just started.) Grub still doesn't know about btrfs, and that's semi-back-burnered functionality, so in order to get it to boot, you need a /boot partition on something Grub does know about.

Linus runs btrfs, on a spare laptop?

[jonaskoelker]

The fact that Linus runs it as his root fs doesn't tell me much. Now, if you told me that's what he uses for ~/, I would be more impressed.

It gets even worse. FTFA:

Linus Torvalds is using it as his root file system on one of his laptops.

Maybe one of his spares?

I'm speculating, but note that the article doesn't say "his main laptop", which it could, and which would be a better "seal of approval", so it probably would if it was true...

Re:Linus runs btrfs, on a spare laptop?

[Anonymous Coward]

Does this make it even worse?

Whether it does or not, I think it's still a fun read. "Double-plus-ungood" indeed :)

Oh great

[teslatug]

As if fsck wasn't bad enough to use in business talks, now I have to get prepared for btrfsck

Re:Oh great

[Anonymous Coward]

I suddenly feel the urge to create a new fs called buttfs.

Re:Oh great

[Anonymous Coward]

so that would be "butter fuck"?

Re:Oh great

[Rocketship Underpant]

I propose a network filesystem called ClusterFS, which would be fixed using the ClusterFSCK command.

Re:Oh great

[anilg]

If btrfs is following ZFS's lead here then there will be no fsck for it. End-to-end checksumming makes it redundant.

Re:Oh great

[diegocgteleline.es]

No it doesn't - it avoids most of the rare cases where you need it, though

Re:Oh great

[toby]

I'd rephrase that. It eliminates the common cases where you'd need fsck on a conventional filesystem.

ZFS' design makes consistency failure extremely unlikely. I understand why they claim it doesn't need fsck ("always consistent on disk"). There is controversy over whether there should be a scavenging tool. Some people want one for peace of mind.

But again, most cases of ZFS pool loss where some believe a scavenger may have saved them, may actually have been solved by more aggressive rollback (I believe work is being done on this).

Anyone interested in this issue should follow the ZFS mailing list.

Re:Oh great

[Simetrical]

I'd rephrase that. It eliminates the common cases where you'd need fsck on a conventional filesystem.

ZFS' design makes consistency failure extremely unlikely. I understand why they claim it doesn't need fsck ("always consistent on disk").

That's assuming that when you have a power outage, either you were using battery-backed RAID, or the disk was kind enough to commit the writes to the physical medium in the order the OS gave them. If you don't have battery-backed RAID, the only way you can have the faintest clue what's on the disk in the event of a power outage is to configure the OS or applications to flush data to the actual physical medium constantly, skipping the write buffers on the device. Which is incredibly slow. So nobody tends to do it very aggressively. If you don't take that step, you can't guarantee anything; a fsck is still required after a power outage. (Not to mention other types of hardware failure.) The only time you won't need fsck is after a mere OS crash; but you usually don't need a fsck then anyway, on journaled filesystems.

Choice of filesystems...

[Bert64]

You can't even settle things with a battle of the benchmarks: file system workloads vary so wildly that you can make a plausible argument for why any benchmark is either totally irrelevant or crucially important.

As pointed out, filesystem workloads vary massively, which is why it's good to have a choice of different filesystems which can be chosen based on individual requirements. Only offering a single filesystem like many other OS's do is extremely inefficient. One size does not fit all.

Porn stars

[BitZtream]

File systems written by porn stars are bound to fail. If this isn't a porn star, a name change is in order.

Re:Porn stars

[MLease]

If "Chris Mason" makes you think "porn star", you have a very strange mind. I can't even begin to follow your reasoning there....

-Mike

Re:Porn stars

[Anonymous Coward]

I do cocaine and JWH-018

Yet another "modern" FS without undelete...

[redirect 'slash' nil]

Allow me, if I may, to open the can of worms here.

Why is it that in 2009 we can't get a filesystem that allows easy undeletion? It all happened to us one time or another: You typed that 'rm' command you shouldn't have, and now your work of the past few hours is gone. If only there was a simple way to undelete those few recent removed files...
We all know that the data is not zeroed on deletion, so why can't we have a File System that (preferably after fs umount) can scan the blocks and retrieve any file whose data blocks have not been overwritten yet, even if it takes a lengthy whole disk surface scan.
Hell, in 1989, I could do just what I described above very easily on the Amiga (granted, it was floppy based, but still...), and I never lost a single file apart from disk errors. I can even do that fairly easily with external tools on NTFS. So Why are all the UNIX filesystems I know (and btrfs is apparently not going to be an exception) such a pain in the butt for undeletion?

Re:Yet another "modern" FS without undelete...

[asaul]

Simple - use ZFS + snapshots. There are tools out there allready that to periodic and rolling snapshots - bad delete, go into the snapshot and copy it back, that easy.

The real trick with undeletion is making sure you dont overwrite the data before you are really sure it doesnt need to undeleted....

Re:Yet another "modern" FS without undelete...

[cnettel]

Yes, snapshots is the way to go, and it is there for NTFS and btrfs as well. (Volume shadow copies being enabled with a shortage for free space is a direct route to terrible performance in Vista, but oh the joy of realizing that you actually have a shadow copy when things go really bad by a human error...)

Re:Yet another "modern" FS without undelete...

[Lemming Mark]

There was some work on making undeletion work with ext4, with support integrated into the filesystem. AFAIK that's not yet merged into Linux but I do remember at least one developer being serious about it I imagine they could do the same with Btrfs, I'd be surprised if they haven't thought about it.

As others have noted, the snapshotting ability of Btrfs mitigates the lack of undelete somewhat but doesn't quite replace it.

Re:Yet another "modern" FS without undelete...

[FlyingBishop]

There are a lot more reasons to want data to disappear than there are to want it to stick around.

On any FS with easy undelete, I guarantee a large number of your passwords will end up scattered around the disk for those who know where to look. Also social security number, and so on.

It's far better to handle it in userspace, where you can see what you're backing up.

Re:Yet another "modern" FS without undelete...

[ultranova]

We all know that the data is not zeroed on deletion, so why can't we have a File System that (preferably after fs umount) can scan the blocks and retrieve any file whose data blocks have not been overwritten yet, even if it takes a lengthy whole disk surface scan.

Why would you use such shenigans? Simply make the filesystem mark deleted files as "hide from directory listing, and really delete only if you need the space". Then add a couple of syscalls to examine these "recyclable" files and restore them to normal status.

Now, there are a number of corner cases that need to be thought out - such as what happens if you delete a file/directory and then create a new one with the same name - but the principle is simple enough: don't really delete files, merely mark them as deletable/recyclable/harvestable/condemned/dying.

Re:Yet another "modern" FS without undelete...

[toby]

NILFS2 (recently integrated into Linux mainline) also supports snapshots.

Re:Yet another "modern" FS without undelete...

[OrangeTide]

I undelete stuff all the time on Linux. you just open the trash and pull the stuff out. Once you empty the trash it is gone though. If you're using a command-line and 'rm' stuff though, that's entirely your fault for using such a low-level power-user interface for file management.

There are serious performance consequences and fragmentation consequences of supporting undelete at the filesystem level. But supporting snapshots is something high performance filesystems do, and snapshots are way more useful than undelete. Especially if snapshots are cheap enough to make them automated. Imagine having 24 revisions of your filesystem of the last 24 hours. This is done all the time on real Filers. I love it when my home directories at work are snapshot this way, makes it super easy to recover screwed up source code due to my inability to check in source before I make huge changes to it.

I think we should demand that Linux get snapshot support that is generally available (like default on RHEL,SuSE,Ubuntu,etc.) It's a feature that has been missing from Linux. While things like FreeBSD have had it standard for many years now.

Re:Yet another "modern" FS without undelete...

[redirect 'slash' nil]

Simply make the filesystem mark deleted files as "hide from directory listing, and really delete only if you need the space". Then add a couple of syscalls to examine these "recyclable" files and restore them to normal status.

You, sir, hit the nail on the head.
That would be my preferred solution as well, but then it needs to be implemented in the FS, and all the discussions I've seen on recent and future filesystems tend to indicate that the developers see little value in adding that kind of undelete future ("not worth it", "too complex", "undelete belongs in userspace"...), which brings us back to the original point...

Re:Yet another "modern" FS without undelete...

[raylu]

If you're using a command-line and 'rm' stuff though, that's entirely your fault for using such a low-level power-user interface for file management.

No, it's not. 'rm' is by no means a "low-level power-user interface." That doesn't even make sense. While it is arguably for "power-users," it is not anyone's fault for using this high-level userspace tool for file management. The perception that it is low-level annoys me.

Re:Yet another "modern" FS without undelete...

[terremoto]

Simply make the filesystem mark deleted files as "hide from directory listing, and really delete only if you need the space". Then add a couple of syscalls to examine these "recyclable" files and restore them to normal status.

Netware has a Salvage utility that relies on a filesystem with those attributes. It used to be called the Netware File System (yes, NFS). More recent versions are now called Novell Storage Services. Ported to SuSe now according to Wikipedia http://en.wikipedia.org/wiki/Novell_Storage_Services/

Re:Yet another "modern" FS without undelete...

[Anonymous Coward]

Ditto. And to think, many people see C++ as a low level programming language too. (kid you not)

Re:Yet another "modern" FS without undelete...

[sowth]

If nobody intends to implement this feature, then why is there a listing in the linux chattr man page?

When a file with the 'u' attribute set is deleted, its contents are saved. This allows the user to ask for its undeletion.

I don't know if the current kernel implements this flag or not, since I have not tried it. However, there is this patch from 1996 which appears to implement it...

I suppose someone could play with setting this flag and deleting files on different filesystem types to see.

Re:Yet another "modern" FS without undelete...

[Simetrical]

I undelete stuff all the time on Linux. you just open the trash and pull the stuff out. Once you empty the trash it is gone though. If you're using a command-line and 'rm' stuff though, that's entirely your fault for using such a low-level power-user interface for file management.

There are serious performance consequences and fragmentation consequences of supporting undelete at the filesystem level. But supporting snapshots is something high performance filesystems do, and snapshots are way more useful than undelete. Especially if snapshots are cheap enough to make them automated. Imagine having 24 revisions of your filesystem of the last 24 hours.

Filesystems like NILFS plan to do even better: potentially a snapshot on every write, with better performance than existing filesystems. Of course, they only work on solid-state drives.

Re:Yet another "modern" FS without undelete...

[OrangeTide]

The perception that it is low-level annoys me.

too bad, 'rm' is just a wrapper for unlink() system call on unix.

Re:Yet another "modern" FS without undelete...

[OrangeTide]

potentially a snapshot on every write

Now that is exciting! But then you need a service just to manage all those snapshots and discard things before you run out of space.

Re:Yet another "modern" FS without undelete...

[Simetrical]

Now that is exciting! But then you need a service just to manage all those snapshots and discard things before you run out of space.

If I understand correctly, it takes "checkpoints" automatically, and discards the oldest checkpoints automatically as needed unless they're transformed into snapshots by the appropriate command. If something is explicitly transformed into a snapshot, it will take up space until explicitly discarded. Something like that. It's still a ways off, even if you have solid-state drives handy.

Re:Yet another "modern" FS without undelete...

[OrangeTide]

It's probably something you'll see in the mainstream in less than 10 years. possibly less than 5.

LVM

[Anonymous Coward]

This looks like a promising filesystem - as ZFS on linux is, at present, doomed to die an ugly death, btrfs looks to address a lot of the shortcomings of other filesystems and bring a clean, modern fs to linux.

If I have to put up with the LVM I'm not interested. It was a great idea back in the early '90s, but we really have to move on. Personally I like ZFS' idea of pools.

It goes beyond ZFS in some areas too, such as being able to efficiently shrink a filesystem, and keeps a lot of the cool things that ZFS made popular, such as Copy-On-Write.

The lack of pool shrinking is an acknowledged and documented feature request (as is going from a single- to dual-parity configuration on the fly). They're working on it, but if it was really desired by a lot of large customers, it would have been a higher priority (Sun is often "coin operated" on things like this). They've also publicly talked about de-dupe functionality.

Apple are really struggling with ZFS ...

Given Apple's secrecy, how do you know what's happening with ZFS in Apple? ZFS is very portable, shown by the fact that the developer who got ZFS onto FreeBSD (Pawel Jakub Dawidek) had basic things working in only ten days:

http://blogs.sun.com/eschrock/entry/zfs_on_freebsd

Given that it went into OpenSolaris on 2005-11-16 and commited into FreeBSD on 2007-04-06 (15 months later) by a single developer (?), I would hazard to guess that it's not a technical issue that's preventing Apple from having it.

Re:LVM

[Pinky's Brain]

http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices

triple-parity RAID

[Anonymous Coward]

ZFS has built in RAID support (which, I assume, works on the block level, instead of on the disk level), maybe Btrfs will get this too.

Yes, btrfs currently has built-in support for raid 0/1/10, 5 and 6 are under development.

On a related note, ZFS recently had triple-parity ("RAID-7" ?) added:

http://blogs.sun.com/ahl/entry/triple_parity_raid_z

Re:triple-parity RAID

[heson]

But this goes to 11.

Re:triple-parity RAID

[Hucko]

well played! very nice.

doesn't belong in the filesystem.

[Anne Honime]

It's my layman opinion that undeletion doesn't belong to the filesystem, but to userspace ; kde and gnome already provide an undelete wastebasket. It is even conceivable on some systems to forbid users from truly deleting files (think White House staff, for instance). And you already can alias rm * to 'mv * ~/wastebasket/' for your regular users.

But what you are really asking for is a magical shield for your mistakes as root. Either write yourself a false rm script as you would for normal users, or pay attention. But really, you shouldn't delete anything as root unless you've tested by moving away what you intend to delete that the system is working without this specific file you want to erase. Only after testing, delete to your heart content.

Meh

[Dachannien]

Who cares? In a few years' time, this will be obsoleted by its successor, icantbelieveitsnotbtrfs.

Re:Meh

[Lemming Mark]

When Fedora added btrfs support they didn't allow it as a root filesystem at installation time ... unless you passed the special command "icantbelieveitsnotbtr" on the kernel command line ;-)

Short Version

[Anonymous Coward]

Don't be stupid. Problem solved.

Re:Short Version

[Anne Honime]

Everybody can make a typo without being stupid. I'd rather sum up my words by "don't be careless, problem solved".

Re:Short Version

[redirect 'slash' nil]

Alright, I'll bite some more:

The "don't be careless" position would be fine, but it's akin to never making any mistake, which, considering the amount of background cursing noise experienced in any IT profession, is the exception rather than the rule. If you've never been careless when working with computers then kudos to you, but I don't think that's the general rule. Oh, and I don't think I mentionned using rm as root (but of course, you want to be root to umount the filesystem and revent further writes after you realized your mistake).
For the sake of what follows, let's consider that I, stupidly or not, as a non root user, innadvertently deleted that 10 GB worth of irreplaceable data I just generated (eg that 10 GB HD video file of an UFO landing on my front lawn). This was my only copy. Now let's consider the options:

- 'rm' alias, a.k.a. as the Windows-like trashcan: That solution would be fine if I didn't (usually) want to have that file space freed straight away. If I remvove a 10 GB file from my system, I want 10 GB more disk space available => not the best option. Plus what if that file just got deleted from an UI (eg. the video editing app I was using under X) rather than from an 'rm' command. Also defuses the careless argument with "what if I am the most careful person in the world, but I am using an application (which I carefully picked), that has a bug, which noone foresaw, that deletes large files using system calls"

- snaphots: This looks like the best compromise at the moment, but for that to work, snapshots would need to occur in real time. I might be wrong, but I don't think this is the case on ZFS right now => if I recorded that UFO landing video and lost it in the interval between snapshots, it doesn't really solve my problem

- undelete in userspace through something different than aliases: I've seen this advocated again and again, and it is my layman's opinion that undeletion does indeed belongs to the filesystem rather than userspace. All undeletion takes is a chained list of blocks (so let's say 8 bytes for 64 bit block addressing of the partition, to be safe). Give me 8 bytes/block of file data, and I can now scan the whole partition, retrieve all data that has not been overwritten, and recover it. If I umounted my partition right after I realized my UFO video was deleted, it won't be much of a problem to recover it. Don't care much about filenames and perms as long as I can peek into recovered files (or parts of files), but if you insist, I'll be creative an flip a single bit instead of 64 for blocks that are sequential, so that I can use the remaining 63 bits, over multiple sequential blocks, for stuff like unique ID, original filename, perms, etc. Won't do much for small files, or if a lot of write operations have intervened in the meantime, but if that wasn't the case, there's a good chance you'll be able to rebuild it almost in full, even with a few missing blocks here and there
Now, I'll agree that not everybody wants to lose 8 bytes/block, but then mkfs could give undelete as an option on FS creation, and as long as people have a choice, everybody's happy...

Re:Short Version

[Anne Honime]

I think you are not understanding me ; when I wrote "don't be careless", I meant in the light of my previous post, don't be a careless administrator. I understand your concerns, and I think strongly you're handling them backward. That's because you have, like most everyone, two hats and you are confused between when you have only your user hat on, and when you, as root, must treat your user double with a clue bat.

To follow on your video example, the main failure of it is : as a user, you want your 10GB disk real estate back. Dealing with disk space is not your problem as a user. That's when you put your root hat on, and become a BOFH against yourself. Not putting safeguards in this case, as an admin, is carelessness ; it means basically that you are servant to your users tyranny. Even as an admin, you should treat yourself with paranoia and don't fall into the almighty god power over everything trap.

That's when alias and scripts come handy, as a first line defense ; most of the time, removing a file to another location won't cause problem. After a certain delay, you, as root, can remove it for good. But you should stand firm to your own impetous, and, should you need 10GB on the spot, either tell yourself "nocando" or go buy a spare disk.

Re:Short Version

[redirect 'slash' nil]

After a certain delay, you, as root, can remove it for good

That's the problem I have with userland based solutions. They require precognition-like ability so that you setup the scripts/cron jobs that will save your life in advance. Unfortunately, when you go around a large number of systems, or, as I mentionned above, use an application UI rather than the commandline for file manipulation, this kind of "I'll do some additional work against the very rare case where I really might need to undelete a file" doesn't really cut it (did I mention I was also lazy?)
What I am saying is, a crude undelete is so simple to implement at the FS level, it should be a part of any modern filesystem, so that lazy bums like myself can get it at the flick of a switch when creating a new FS (mkfs --allow-undelete ...).
But don't worry: I never blame anybody but myself when I inadvertently delete something (which hasn't happened that often, fortunately). I'm just annoyed that, as far as the end user is concerned, some of the native capabilities of filesystems seem to have regressed in the past 20 years. 20 years ago, I could easily undelete a file on my favorite O/S. Today I cannot...

Linux Filesystems IX

[Anonymous Coward]

Don't worry, the next one will live up to the hype! In the meantime use this...

You forgot the most critical advantage:

[Anonymous Coward]

GPL licensed.
Without it, companies like IBM and Red Hat would not be willing to put so much time and money into the platform. Just look at Apple. They went with BSD code and instead of simply building a new "distribution" of BSD, they forked it *AGAIN* and made something that is largely incompatible (sure you can run BSD apps on OS X, but you cannot run OS X apps on BSD, this is BY DESIGN). This kind of thing happens time and time again in the BSD world and is IMHO the primary reason why BSD has to date largely failed despite its technical advantages.

Re:You forgot the most critical advantage:

[thenextstevejobs]

GPL licensed. Without it, companies like IBM and Red Hat would not be willing to put so much time and money into the platform. Just look at Apple. They went with BSD code and instead of simply building a new "distribution" of BSD, they forked it *AGAIN* and made something that is largely incompatible (sure you can run BSD apps on OS X, but you cannot run OS X apps on BSD, this is BY DESIGN). This kind of thing happens time and time again in the BSD world and is IMHO the primary reason why BSD has to date largely failed despite its technical advantages.

I'm sorry, but that is actually the best part of the arrangement.

If there were no BSD, Apple would not have thrown in the towel and used the Linux kernel. They would have gone the Microsoft route and created software that is completely proprietary. With the way things stand, Apple has created a platform that is relatively easy to port software to.

The BSD license has brought us amazing levels of compatibility in computing. Little thing called the network stack that I'm sure we're all happy was BSD licensed?

Re:You forgot the most critical advantage:

[True Grit]

They would have gone the Microsoft route and created software that is completely proprietary

Instead, they just stole what they wanted from *BSD and created a system that is partially proprietary. That you think that is somehow better than "completely proprietary" is where the disagreement is. Three points still remain:

a) Apple took from the *BSD world, made a profit, and never returned in kind, and

b) Apple's OS is still proprietary: you can't write an "Apple OS application" using just the underlying BSD plumbing, and thus

c) The GP's points still stand, in particular: the *BSD's can't garner any momentum in the corporate space because most companies will never consider contributing back for fear of their competitors using their code against them. Since it isn't a level playing field, most commercial entities are only ever going to *take* from the *BSD world, not give back.

Little thing called the network stack that I'm sure we're all happy was BSD licensed?

Well, at least MS was happy they could snag that for free (then turn around and get others to pay *them* for it).

But, hey, if you think thats a great "arrangement", no problem, you're free to choose that path, just don't be too surprised that, having watched all this *taking* over the years, others have decided to go in a different direction...

Re:You forgot the most critical advantage:

[DarkEmpath]

GPL licensed.

That's a bug, not a feature. Keep your politics out of my PC, and let me get on with what I want to do. I prefer "Free" software licences, "free as in freedom" (BSD, MIT, Mozilla, CDDL), not "free as in Follow My Rules" (GPL).

Bit Rot Filesystem?

[Dwedit]

I don't really like the name. The first time I look at the name, I thought it was short for "Bit Rot Filesystem".

Duh...

[coryking]

Who the fuck writes a file system implementation in GPL!?

Zealots, that is who. People so blinded by their ideology that they cannot think about the long-term issues with their license choice.

GPL is okay for end-user stuff, but it is a horrible choice for protocol libraries, programming languages and well, basically any kind of developer used library.

Oh, actually I'm only half right. Sun and other companies use the GPL as a way to tease you into buying their real stuff. Since you can't modify their GPL stuff*, they hope you'll eventually pony up and by their big-boy stuff so you can modify it.

*(well unless you are okay with your own work becoming GPL'd)

Re:Duh...

[caseih]

Wow. FUD flies fast and hard on slashdot. Zealots? Are you serious? Rather than mod your post as +1 Funny, I think I'll blow some karma and respond, just to set the record straight.

Laying aside misconceptions about the GPL, the main reason BtrFS is GPL is because it's part of the Linux kernel which is also GPL! How hard is it to grasp that? If Apple or anyone else wants to license Oracle's BtrFS code, they are welcome to negotiate and get the code under a different license than the GPL. It's that simple. BtrFS is an implementation of an idea, a specification. If Apple wants to write their own BtrFS driver, they are welcome to do that. Or Microsoft.

Why are developers who don't want their code to be ripped off (used without payment in a closed product) by companies and incorporated into a product are labeled zealots? How is this different than software companies requiring code to be licensed by third parties? So a company who creates some really cool technology that they license for a fee to others for use in products zealots? There really is no difference.

While I haven't written any software of note, I also use the GPLv2 (evaluating v3) since I want my software to be able to be freely used by those that want to use it, but if my code is that valuable to a company, I want to get paid for my trouble. If no one is willing to pay me, then that's fine. They are welcome to use my software without restriction, but if they redistribute it, to do so under the terms of the GPL. Guess that makes me a zealot.

Re:Duh...

[Anonymous Coward]

Why are developers who don't want their code to be ripped off (used without payment in a closed product) by companies and incorporated into a product are labeled zealots?

For the same reason any and all anti-piracy tactics are horribly wrong ?

Ooooh ! Shiny free stuff

Re:Duh...

[evilviper]

Why are developers who don't want their code to be ripped off (used without payment in a closed product) by companies and incorporated into a product are labeled zealots?

Perhaps because they are writing software which is by FAR most useful when it is used as far and wide as possible, while using a license which makes that goal extremely difficult to achieve, unnecessarily.

Honestly, the only reason anyone cares about Btrfs is because the license on ZFS is too restrictive for inclusion in Linux, and NOBODY has opted to write their own implementation under a GPL or other, freer, license.

Re:Duh...

[True Grit]

Why are developers who don't want their code to be ripped off (used without payment in a closed product) by companies and incorporated into a product are labeled zealots?

Perhaps because they are writing software which is by FAR most useful when it is used as far and wide as possible, while using a license which makes that goal extremely difficult to achieve, unnecessarily.

Um, "most useful" to *who*, exactly? Why is your, or anyone else's, convenience/preference more important than that of the software authors? By this logic, *all* software should be released into the public domain.

I know its an old-fashioned idea and all, but geez, whatever happened to letting software *authors* decide how their software may be used?

Re:Duh...

[Anonymous Coward]

Laying aside misconceptions about the GPL, the main reason BtrFS is GPL is because it's part of the Linux kernel which is also GPL!

A lot of of the Linux kernel is BSD/MIT licensed, mostly because they've gotten it from BSD projects. There's nothing stopping anyone from writing a BSD licensed part of the kernel, and there can be good reasons to, especially if you want the BSD guys to be able to enjoy your work.
It's cute that you hurt inside when other people benefit from your work, but some people actually enjoy their code being as useful as it can be, and enjoy having their files to the maximum extent possible stored in open standarized formats readable by anyone. They further see that the best way to encourage this goal is to have reference implementation that absolutely anyone can use, even those evil, proprietary code-making bastards.

If libpng had been GPL, PNG would never have taken off. IE support? Hah! It'd only be supported in Debian's custom Firefox build, because they wouldn't include support for whatever patent encumbered Microsoft backed format that turned out to succeed GIF. A freely available reference implementation is the only way to get any sort of compatibility for a complex format.

I don't see why anyone would store their data on a file system that only worked in one operating system, any more than they'd store it in a format only read by one program. BSD and Linux should be a minimum goal for a free file system, and that should be a goal from the start, as it's ridiculously easy to accomplish. Just don't arbitrarily wall off the code to make other people have to duplicate work so you can get ahead of them in some imaginary competition of yours, and things will write themselves.

Re:Duh...

[evilviper]

I know its an old-fashioned idea and all, but geez, whatever happened to letting software *authors* decide how their software may be used?

Oh no! People are being CRITICIZED! Horror of horrors!

Look, software authors DO have the LEGAL right to decide what license to release their code under, which is, by far, the most important part. That doesn't change the fact that their choice in this case is foolish. And it certainly doesn't preclude them from criticism.

Re:Duh...

[True Grit]

Oh no! People are being CRITICIZED!
...
That doesn't change the fact that their choice in this case is foolish

Responses of the form: "Duh, your stuuupid!" might technically be "criticism", but that doesn't go very far here, at least not for readers for which content actually matters.

Hint: You've yet to make any rational argument for why they're being "foolish", especially since you already admit:

software authors DO have the LEGAL right to decide what license to release their code under

This is especially weird here, as the software is an implementation of (AFAIK) an open specification, and can be reimplemented by anyone else if they want it under a different license. Can't stand GCC because of its license? No problem, use another C compiler.

So, don't like the license? Fine, then attack that, but just calling its users "foolish" only makes you sound like a troll. Or was that what you intended all along?

Its also a little amusing, given that the people you are calling "foolish" are actually employees of Oracle, IBM, Hitachi, and the usual Linux suspects, e.g. RedHat, SUSE, et. al. Best of all, the originators of BTRFS are the people that now significantly control ZFS's destiny, e.g. Oracle, and they were the ones who gave it the GPL license. Do you really want to continue calling the new masters of ZFS/Solaris/OpenSolaris "foolish"?

Maybe whats really got you irked is that the audience for BTRFS (the Linux ecosystem) will be more than large enough to ensure its long term success, whereas ZFS's future depends a lot on whether Oracle chooses to continue it into the long term. Its not clear yet what they're going to do with ZFS, but the Oracle dev who created BTRFS, Chris Mason (who BTW, was one of the ZFS devs too), and who is still leading its development (along with the others mentioned above) told its dev ml recently that everything was still a go with BTRFS as far as Oracle was concerned. So do you *still* want to continue calling Oracle "foolish"? :)

Re:Duh...

[jbolden]

I wouldn't call that teasing that's been a time tested model for open source development contribute or pay. Trolltech used that for QT and everyone was OK with it.

BTRFS apparently supports auto-defrag

[Zantetsuken]

First paragraph here describes how ZFS works in contrast - FTFA:

In my opinion, the basic architecture of btrfs is more suitable to storage than that of ZFS. One of the major problems with the ZFS approach - "slabs" of blocks of a particular size - is fragmentation. Each object can contain blocks of only one size, and each slab can only contain blocks of one size. You can easily end up with, for example, a file of 64K blocks that needs to grow one more block, but no 64K blocks are available, even if the file system is full off nearly empty slabs of 512 byte blocks, 4K blocks, 128K blocks, etc. To solve this problem, we (the ZFS developers) invented ways to create big blocks out of little blocks ("gang blocks") and other unpleasant workarounds. In our defense, at the time btrees and extents seemed fundamentally incompatible with copy-on-write, and the virtual memory metaphor served us well in many other respects.

In contrast, the items-in-a-btree approach is extremely space efficient and flexible. Defragmentation is an ongoing process - repacking the items efficiently is part of the normal code path preparing extents to be written to disk. Doing checksums, reference counting, and other assorted metadata busy-work on a per-extent basis reduces overhead and makes new features (such as fast reverse mapping from an extent to everything that references it) possible.

Now for some personal predictions (based purely on public information - I don't have any insider knowledge). Btrfs will be the default file system on Linux within two years. Btrfs as a project won't (and can't, at this point) be canceled by Oracle. If all the intellectual property issues are worked out (a big if), ZFS will be ported to Linux, but it will have less than a few percent of the installed base of btrfs. Check back in two years and see if I got any of these predictions right!

Cow friendly butter FS?

[flyingfsck]

Hmmm, just keep meat products out of it so that it can be used in the Middle East without causing a religious problem and needs to be shut down once a week... ;)

Yeah, that's fine, UNTIL IT HAPPENS.

[toby]

The point of ZFS is it is *end-to-end*. Any part of the chain can silently introduce errors. Even if you believe "most likely an error will be in the memory buffer written by my application", I do not see how you can extrapolate that "checksumming file blocks is pointless."

Can you not imagine other failure modes? Even leaving aside unlikely (but possible) silent failures such as "wrote correct data to the wrong sector," what happens in an ordinary powerfail when only half of a mirror has been written? Ordinary RAID systems have absolutely no recourse. ZFS does.

Checksumming is of great protective value. You can verify this by following the ZFS mailing list. Silent write failures are more frequent than you might guess (for example, an often-reported cause is marginal power supplies).

Val Aurora?

[RaymondKurzweil]

So I wondered if Val Henson had ended up getting married. And I was going to use the opportunity to make a lame remark (Slashdot calibre) and advise her to try not to kill her husband (filesystem design can be stressful). But it turns out she just has daddy issues.

Whatever.

Uhhh... D'oh! Mea culpa...

[Anonymous Coward]

Mea maxima culpa. Throw in the stock "a thousand pardons," and you'll get the drift.

Sorry!

-K

Why should I care?

[mcrbids]

I'm not running Ext4, I'm running Ext3, having switched over the last of the Ext2 FSs to Ext3. Now, I've got to deal with not only EXT4, but BTRFS? What kind of changes does it bring that I would/should give a !@# about?

Ext2 worked well. EXT3 was basically EXT2 but with journalling. What does BTRFS give me over EXT4, or for that matter, EXT3?

Sorry, just having trouble giving a damn....

Re:Why should I care?

[PhunkySchtuff]

Ummm, if you read the linked article, you'd see it has things that the ext family lack - such as advanced storage pool management, full checksumming on all data, efficient b-tree style database structure, copy on write, atomic commits (so no need to fsck, ever) more efficient use of disk space, better handling of lots of small files, extensible metadata, cleaner modern design... Need I continue?

Re:Why should I care?

[SST-206]

What kind of changes does it bring that I would/should give a !@# about?

I found this informative Heise article (linked to from the comments in TFA) gave helpful explanations, for example:

• Btrfs uses Copy on Write: Modified data is written into new data blocks to preserve the old data. Only once the data has been written is the tree to the data updated - until then, the file entry points to the old data blocks.
• Checksums verify metadata and data blocks, which allows flawed data to be detected
• Files can be compressed when they are written (mount option compress)
• The file system can be up-sized or down-sized as well as de-fragmented during operation

It's not quite ready for prime time yet, but looks good.

Unclear summary

[Anonymous Coward]

WhoTF is this Linus character?

Why no replacement filesystem for consumer XP?

[QuestionsNotAnswers]

That being said writing a performance filesystem for Windows is much less easy than for Linux.

I have often wondered why there is not an XP replacement filesystem for consumers with better performance characteristics. XP degrades over time and one of the causes appears to be file access speeds degrading over time (it happens on XP systems without viruses or cruft).

Perhaps there is too much "noise" on the internet spamming optimising systems, so a working system can't get known? But perhaps a well known brandname could do it? Or is it due to non-file-system causes like registry-cruft and patch-cruft?

ion.simon.c is a convicted child rapist

[Anonymous Coward]

ion.simon.c is a convicted child rapist who was caught several years ago raping and molesting little children.