Slashdot Mirror
A Short History of Btrfs
diegocgteleline.es writes "Valerie Aurora, a Linux file system developer and ex-ZFS designer, has posted an article with great insight on how Btrfs, the file system that will replace Ext4, was created and how it works. Quoting: 'When it comes to file systems, it's hard to tell truth from rumor from vile slander: the code is so complex, the personalities are so exaggerated, and the users are so angry when they lose their data. You can't even settle things with a battle of the benchmarks: file system workloads vary so wildly that you can make a plausible argument for why any benchmark is either totally irrelevant or crucially important. ... we'll take a behind-the-scenes look at the design and development of Btrfs on many levels — technical, political, personal — and trace it from its origins at a workshop to its current position as Linus's root file system.'"
This looks like a promising filesystem - as ZFS on linux is, at present, doomed to die an ugly death, btrfs looks to address a lot of the shortcomings of other filesystems and bring a clean, modern fs to linux. It goes beyond ZFS in some areas too, such as being able to efficiently shrink a filesystem, and keeps a lot of the cool things that ZFS made popular, such as Copy-On-Write.
It looks like Btrfs also addresses some decisions that were made with the direction that ZFS would be going in, or how it would handle certain problems that now with hindsight behind the developers, they possibly would have done things differently.
Apple are really struggling with ZFS, with it being announced as a feature in early betas of both Leopard (10.5) and Snow Leopard (10.6), as well as being there in a very limited form in Tiger (10.4) - maybe development on Btrfs will leapfrog ZFS for consumer-grade hardware and Apple can finally look at deprecating HFS.
Specialist Mac support for creative pros, Melbourne
Well good for you. I've had an entire HD decide that moving files to another drive, while deleting other files, meant that I wanted the entire filesystem hosed.
-=This sig has nothing to do with my comment. Move along now=-
Is this ever going to replace ext4? The ext series of file systems are 'good enough' for most people, so unless it has some epic benchmarks I can't imagine a huge rush to reformat. Maybe that's what drives file system programmers insane. The knowledge that for the most part, it's going nowhere. FAT12 is still in use, for Christ's sake.
Since you didn't specify what filesystem you were using I'll just assume it was NTFS.
No, it's not - but you're wrong. :"Linus's" is correct. The 's' after the apostrophe only gets dropped in plurals.
Will you also be enjoying your media in REAL PLAYER?
It's not that hard!
Not necessarily true: http://www.bartleby.com/141/strunk.html#1
It's fine to drop the 2nd s, at least in British english. Though, there doesn't seem to be any hard and fast rule..
I use Friend/Foe + mod-point modifiers as a karma/reputation system.
Both are correct, depending on who you ask. It's a British English vs. American English thing. Up here in Canada, just the apostrophe seems to be the preferred form.
upon the advice of my lawyer, i have no sig at this time
Is it Beta? The fact that Linus runs it as his root fs doesn't tell me much. Now, if you told me that's what he uses for ~/, I would be more impressed.
The important question to me is, how long 'til it gets in the major distributions?
Help! I'm a slashdot refugee.
I was taught in my American schools (grade school, even) that you drop the second S on names that end in S. "Linus'"is how I was taught.
There doesn't seem to be any hard and fast rules about anything in British english! ;-)
In Fowler's Modern English Usage, which is generally considered to be the bible of english usage by UK journalists and writers, there's an article called "Possessive Puzzles". In that, he says it was "formerly customary" to drop the last 's', but not any more.
If it was formerly customary in Fowler's day, i reckon it must be well and truly archaic now.
That's interesting. I think in Britain the preferred form of anything is to put an apostrophe anywhere it can possibly be put and drop any syllables that can possibly be dropped.
It seems the version with the final 's' is more modern than the version without it. Considering their efforts to update the language, it's perhaps not entirely surprising that the Americans prefer the more modern form.
Yeah, that's what Fowler reckons, too.
That argument isn't actually based on the technical merits, and thus doesn't make any sense..
Just because a Real OS features a Real FS backed up by a real company, doesn't necessarily mean the FS or OS are any good on technical merits compared to a REAL project licensed under a REAL free software license backed up by a REAL community and supported by a REAL foundation.
How do they compare?
gor blimey guvnor you aint arf stereotypin us brits'.
I've lost files on lots of different systems, including NTFS and ext3 and the kitchen sink. It's rarely the fault of the filesystem itself.
Gut feelings are no good at all when it comes to language. USians wouldn't have much trouble talking to Edgar Allen Poe, but Brits would have a very hard time hanging with John Keats.
That this is a "service" provided by LWN so that non-subscribers can read premium content; this story would be free for all come Thursday, but apparently "diegocgteleline.es" didn't feel compelled to mention that, that LWN's weekly page is premium content, and that premium content subscribers help LWN stay afloat -- when it's almost gone under a couple times.
With the COW-enabled b-tree storing everything including metadata and packing it in the same block as the data it describes, the atructure looks quite similar to reiserfs (v3) in terms of error tolerance and recovery. Should this get a tool like the reiserfsck --rebuild-tree, I'm switching - this single feature (well, and some quite sensible performance) is keeping reiserfs on my systems. Saved me a lot of grief several times, when an ext filesystem would be a totally lost cause (or lots of $$$ for a data recovery company), without any built-in utilities, and really no way to write them, that search the entire disk for anything that looks like filesystem metadata and try to make sense of it, even with all superblock structure completely missing and the rest spotted with garbage.
This is Slashdot. Common sense is futile. You will be modded down.
I asked when it would be usable for "people who backed up their data" about a year ago -- which is about how long I've been using it -- and the answer was, "No firm date." If you load up a 2.6.31 kernel, the commits have reached the point where not only shouldn't you see significant on-disk format changes, but that the bulk of non-RAID tweaking to occur is probably performance related. (RAID is coming, but it's only just started.) Grub still doesn't know about btrfs, and that's semi-back-burnered functionality, so in order to get it to boot, you need a /boot partition on something Grub does know about.
The fact that Linus runs it as his root fs doesn't tell me much. Now, if you told me that's what he uses for ~/, I would be more impressed.
It gets even worse. FTFA:
Linus Torvalds is using it as his root file system on one of his laptops.
Maybe one of his spares?
I'm speculating, but note that the article doesn't say "his main laptop", which it could, and which would be a better "seal of approval", so it probably would if it was true...
As if fsck wasn't bad enough to use in business talks, now I have to get prepared for btrfsck
You can't even settle things with a battle of the benchmarks: file system workloads vary so wildly that you can make a plausible argument for why any benchmark is either totally irrelevant or crucially important.
As pointed out, filesystem workloads vary massively, which is why it's good to have a choice of different filesystems which can be chosen based on individual requirements. Only offering a single filesystem like many other OS's do is extremely inefficient. One size does not fit all.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
That argument isn't actually based on the technical merits, and thus doesn't make any sense..
So what's your point? that ext4 is better than ZFS on technical merits!?
Deliberate? I wonder...
I doubt it, but he will have to hand over REAL MONEY using nothing but REAL WHEEL BARROWS.
Allow me, if I may, to open the can of worms here.
Why is it that in 2009 we can't get a filesystem that allows easy undeletion? It all happened to us one time or another: You typed that 'rm' command you shouldn't have, and now your work of the past few hours is gone. If only there was a simple way to undelete those few recent removed files...
We all know that the data is not zeroed on deletion, so why can't we have a File System that (preferably after fs umount) can scan the blocks and retrieve any file whose data blocks have not been overwritten yet, even if it takes a lengthy whole disk surface scan.
Hell, in 1989, I could do just what I described above very easily on the Amiga (granted, it was floppy based, but still...), and I never lost a single file apart from disk errors. I can even do that fairly easily with external tools on NTFS. So Why are all the UNIX filesystems I know (and btrfs is apparently not going to be an exception) such a pain in the butt for undeletion?
Looks like these truths are not so self-evident after all...
Maybe someday you'll be a Real Boy
This looks like a promising filesystem - as ZFS on linux is, at present, doomed to die an ugly death, btrfs looks to address a lot of the shortcomings of other filesystems and bring a clean, modern fs to linux.
If I have to put up with the LVM I'm not interested. It was a great idea back in the early '90s, but we really have to move on. Personally I like ZFS' idea of pools.
It goes beyond ZFS in some areas too, such as being able to efficiently shrink a filesystem, and keeps a lot of the cool things that ZFS made popular, such as Copy-On-Write.
The lack of pool shrinking is an acknowledged and documented feature request (as is going from a single- to dual-parity configuration on the fly). They're working on it, but if it was really desired by a lot of large customers, it would have been a higher priority (Sun is often "coin operated" on things like this). They've also publicly talked about de-dupe functionality.
Apple are really struggling with ZFS ...
Given Apple's secrecy, how do you know what's happening with ZFS in Apple? ZFS is very portable, shown by the fact that the developer who got ZFS onto FreeBSD (Pawel Jakub Dawidek) had basic things working in only ten days:
http://blogs.sun.com/eschrock/entry/zfs_on_freebsd
Given that it went into OpenSolaris on 2005-11-16 and commited into FreeBSD on 2007-04-06 (15 months later) by a single developer (?), I would hazard to guess that it's not a technical issue that's preventing Apple from having it.
I wish the parent hadn't been modded down. He makes a point that should be addressed.
I've lost data on every file system that I've ever used, including NTFS, and the highly touted ReiserFS. Nothing guarantees the security of your data. The nearest you can come to data security, is to backup, backup, and backup again. Those people and organizations that keep regular backups seldom lose data. However, even those people can lose data in the event of a physical disaster (fire, flood, theft, being hit by a humongous meteorite) which is why off-site backups are important.
That said - IMHO, a journaling file system is an important first step to data security. NTFS and Ext3 are about equal, in my experience. Turning off caching features is an important second step. A power outage before data is written to disk, and/or while data is being written results in corruption in all current file systems. The important thing is, if data is mission critical, you want it written IMMEDIATELY, not floating around in RAM.
And, finally, you NEED redundant backups. Anyone who fails to make backups WILL LOSE data, eventually.
"Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
ZFS has built in RAID support (which, I assume, works on the block level, instead of on the disk level), maybe Btrfs will get this too.
Yes, btrfs currently has built-in support for raid 0/1/10, 5 and 6 are under development.
On a related note, ZFS recently had triple-parity ("RAID-7" ?) added:
http://blogs.sun.com/ahl/entry/triple_parity_raid_z
It's my layman opinion that undeletion doesn't belong to the filesystem, but to userspace ; kde and gnome already provide an undelete wastebasket. It is even conceivable on some systems to forbid users from truly deleting files (think White House staff, for instance). And you already can alias rm * to 'mv * ~/wastebasket/' for your regular users.
But what you are really asking for is a magical shield for your mistakes as root. Either write yourself a false rm script as you would for normal users, or pay attention. But really, you shouldn't delete anything as root unless you've tested by moving away what you intend to delete that the system is working without this specific file you want to erase. Only after testing, delete to your heart content.
If "Chris Mason" makes you think "porn star", you have a very strange mind. I can't even begin to follow your reasoning there....
-Mike
I'm sorry; I don't know what I was thinking!
Who cares? In a few years' time, this will be obsoleted by its successor, icantbelieveitsnotbtrfs.
Don't be stupid. Problem solved.
Poor guy, he could have been first , if not for the long post .
We could implement a locking system , so that no one can start posting until the first person to begin writing posts . That way, it doesn't matter what you write , you will be first if you start first.
It may also lead to more interesting first posts , as the writers won't be stressed that they won't have a first post.
Offcourse , we need to implement a sort of timeout , to prevent locking the thread in case the first poster decides not to post.
Slipping shoelaces ?
> It's not that hard!
That's what she said!
Can i have a copy of that book ?
Slipping shoelaces ?
Don't worry, the next one will live up to the hype! In the meantime use this...
GPL licensed.
Without it, companies like IBM and Red Hat would not be willing to put so much time and money into the platform. Just look at Apple. They went with BSD code and instead of simply building a new "distribution" of BSD, they forked it *AGAIN* and made something that is largely incompatible (sure you can run BSD apps on OS X, but you cannot run OS X apps on BSD, this is BY DESIGN). This kind of thing happens time and time again in the BSD world and is IMHO the primary reason why BSD has to date largely failed despite its technical advantages.
Except that English is defined by how it's used not by some arbitrary authority, in spite of what those boys up at Oxford will tell you. The fact that people use both "Linus's" and "Linus'" means that both are acceptable. By the same token, "ain't" and "y'all" are now recognized words because people use them and nobody is confused by meaning. "Google" is now a verb, as well.
If I say something in what I believe is English and you understand it unambiguously as English, then it's English or an accepted dialect.
The road to tyranny has always been paved with claims of necessity.
I don't really like the name. The first time I look at the name, I thought it was short for "Bit Rot Filesystem".
Zealots, that is who. People so blinded by their ideology that they cannot think about the long-term issues with their license choice.
GPL is okay for end-user stuff, but it is a horrible choice for protocol libraries, programming languages and well, basically any kind of developer used library.
Oh, actually I'm only half right. Sun and other companies use the GPL as a way to tease you into buying their real stuff. Since you can't modify their GPL stuff*, they hope you'll eventually pony up and by their big-boy stuff so you can modify it.
*(well unless you are okay with your own work becoming GPL'd)
AFAIK there are many patents on human genome, and thus his arse and reproductive organs would, in fact, be patented as derivative works. Furthermore, said reproductive organs would also qualify as a patent violation facilitation device (unless he has a license from patent holders), and thus possession of said organs would imply that he intends to break patent law - and of course his mere existence proves that his parents already did.
Disgusting criminal scum, his like drags the quality of this site down.
Forget magic. Any technology distinguishable from divine power is insufficiently advanced.
In spoken English, you generally pronounce the second 's' (unless you are a pedant of some sort), so it would stand to reason that the second 's' should remain. There is another motivation: the "'s" is actually a clitic that attaches to phrases (usually noun phrases) and is thus a separate word, not a part of the word it is attached to. As such, it should always be spelled out (as it is always pronounced).
No. I was alluding to GP's failure to make a good argument for supporting ZFS, using sound reasoning. I'm saying the truth of the premises doesn't imply the truth of the consequent.
ZFS on supported hardware is actually superior to ext4 on certain technical merits; primarily data integrity (checksumming), random write performance, and read performance (when massive amounts of RAM are available), and more advanced features (snapshots).
On the other hand, ext4 works on 32-bit processors (ZFS is only recommended to be used on 64-bit procs) with small amounts of RAM available, less than a GB; the minimum amount of RAM one should use ZFS with is 2gb, and 4gb or more is strongly recommended, above and beyond any RAM required by apps running on the machine.
But that has little to do with the OS being produced by a large corporation.
Offcourse , we need to implement a sort of timeout , to prevent locking the thread in case the first poster decides not to post.
No we don't, if it's locked and no-one is posting we won't get spam and people might actually read TFA.
Maybe you should get a REAL DOLL
“Common sense is not so common.” — Voltaire
Except that English is defined by how it's used not by some arbitrary authority, in spite of what those boys up at Oxford will tell you.
Txt spkrs rejoiced. Thus started the revolution. Everyone left after the MegaWar was a freak, except for a small band of roving former English professors. These kept the language alive with campfire stories, in the vein of The Canterbury Tales, until one day, the world could rebuild . . ..
Hmmm, just keep meat products out of it so that it can be used in the Middle East without causing a religious problem and needs to be shut down once a week... ;)
Excuse me, but please get off my Pennisetum Clandestinum, eh!
Ahhh, but, it seems that you have assumed AC to be human. ;-)
"Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
Elements of Style says:
Form the possessive singular of nouns with 's.
Follow this rule whatever the final consonant. Thus write,
Charles's friend
Burns's poems
the witch's malice
This is the usage of the United States Government Printing Office and of the Oxford University Press.
Exceptions are the possessives of ancient proper names in -es and -is, the possessive Jesus', and such forms as for conscience' sake, for righteousness' sake. But such forms as Achilles' heel, Moses' laws, Isis' temple are commonly replaced by
the heel of Achilles
the laws of Moses
the temple of Isis
The pronominal possessives hers, its, theirs, yours, and oneself have no apostrophe.
---
I think we can all agree that Linus is on par with Jesus and therefore a single apostrophe is correct.
Do you even lift?
These aren't the 'roids you're looking for.
The point of ZFS is it is *end-to-end*. Any part of the chain can silently introduce errors. Even if you believe "most likely an error will be in the memory buffer written by my application", I do not see how you can extrapolate that "checksumming file blocks is pointless."
Can you not imagine other failure modes? Even leaving aside unlikely (but possible) silent failures such as "wrote correct data to the wrong sector," what happens in an ordinary powerfail when only half of a mirror has been written? Ordinary RAID systems have absolutely no recourse. ZFS does.
Checksumming is of great protective value. You can verify this by following the ZFS mailing list. Silent write failures are more frequent than you might guess (for example, an often-reported cause is marginal power supplies).
you had me at #!
So I wondered if Val Henson had ended up getting married. And I was going to use the opportunity to make a lame remark (Slashdot calibre) and advise her to try not to kill her husband (filesystem design can be stressful). But it turns out she just has daddy issues.
Whatever.
This is Slashdot, sir. Pedantry is what we serve.
The road to tyranny has always been paved with claims of necessity.
Mea maxima culpa. Throw in the stock "a thousand pardons," and you'll get the drift.
Sorry!
-K
I do cocaine and JWH-018
"Offcourse , we need to implement a sort of timeout"
Nah, it's all about lockless algorithms now, don't ya know. This one's simple; you get a cookie with the post form, and posts are displayed in cookie time order, not message post order. That way, person who hits 'reply' first gets first post as soon as they post it, pushing down any posts that've been made before first post is posted but after first post was initiated. Simple :-)
The revolution will not be televised... but it will have a page on Wikipedia
No, you pronounce 'es', yet nobody is advocating the use of es instead of 's everywhere just because it's pronounced that way...
It stands to reason that what stands to reason doesn't determine the rules of written english.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
Journaling does NOTHING to improve "data security". Absolutely nothing, either in practice, or in theory. What it does is obviate the need for fsck. It doesn't even do that particularly well, as non-journaled filesystems like FreeBSD's UFS2 can be used much more quickly after an unclean shutdown, and is extremely reliable.
No, disabling on-drive caching has to be #1. Without it, no file system can possibly act sanely.
Unless the RAM in question is battery-backed, as many higher-end RAID controllers are.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
I'm not running Ext4, I'm running Ext3, having switched over the last of the Ext2 FSs to Ext3. Now, I've got to deal with not only EXT4, but BTRFS? What kind of changes does it bring that I would/should give a !@# about?
Ext2 worked well. EXT3 was basically EXT2 but with journalling. What does BTRFS give me over EXT4, or for that matter, EXT3?
Sorry, just having trouble giving a damn....
I have no problem with your religion until you decide it's reason to deprive others of the truth.
Don't make me laugh.
Solaris is a hacked together version of BSD and System V and its uptime used to be measured in hours because it was leaking memory like a sieve.
Since the NFS disaster (data corruption, complete lack of security, and poor performance, all in one file system!), Sun can't be taken seriously on file systems.
And Oracle's own product requires its own dedicated database monkey to keep it running.
When it comes to technology or producing decent code, Sun and Oracle are a joke; the only people they impress are people aspiring to pointy hair.
No. I was alluding to GP's failure to make a good argument for supporting ZFS, using sound reasoning. I'm saying the truth of the premises doesn't imply the truth of the consequent.
ZFS on supported hardware is actually superior to ext4 on certain technical merits; primarily data integrity (checksumming), random write performance, and read performance (when massive amounts of RAM are available), and more advanced features (snapshots).
On the other hand, ext4 works on 32-bit processors (ZFS is only recommended to be used on 64-bit procs) with small amounts of RAM available, less than a GB; the minimum amount of RAM one should use ZFS with is 2gb, and 4gb or more is strongly recommended, above and beyond any RAM required by apps running on the machine.
ZFS can be used on low-memory, 32-bit machines. Or, at least, the ZFS support in FreeBSD works quite well on 32-bit machines with as little as 768 MB of RAM. There are a lot of FreeBSD users with laptops using ZFS, as well as a lot of test boxes setup with
Yes, ZFS works best on 64-bit systems with 4+ GB of RAM, but that's mainly due to the aggressive caching that ZFS does. The more RAM you have, the more caching it does, and the faster things are.
But that doesn't mean that it can't be tuned to work in RAM-constrained situations.
btrfs != reiserfs.
That is all.
I am not devoid of humor.
It's more than "works best on 64-bit", there are real issues utilizing ZFS for any production load on a 32-bit proc. It's probably not that bad on a laptop, as there's really no issue if your system crashes from time to time; Windows users are quite used to it.
The issue is ZFS caching causes free memory address space fragmentation on a 32-bit architecture, resulting it being unstable on 32-bit procs. Running out of allocatable virtual address space is basically equivalent to running out of memory; you may have plenty of RAM free, but your system can't service allocation requests larger than the biggest free fragment.
ZFS is as address space hungry as it is memory hungry.
For example, if you have a large filesystem shared out via NFS, at least in some versions you could cause a kernel panic on a 32-bit architecture by running a "find /" command.
WhoTF is this Linus character?
That being said writing a performance filesystem for Windows is much less easy than for Linux.
I have often wondered why there is not an XP replacement filesystem for consumers with better performance characteristics. XP degrades over time and one of the causes appears to be file access speeds degrading over time (it happens on XP systems without viruses or cruft).
Perhaps there is too much "noise" on the internet spamming optimising systems, so a working system can't get known? But perhaps a well known brandname could do it? Or is it due to non-file-system causes like registry-cruft and patch-cruft?
Happy moony
Turning off caching is a terrible idea. What you want though is information being written in a way that is recoverable after an outage quickly and rectified slowly. A database based filesystem for example like you see on i-os.
ion.simon.c is a convicted child rapist who was caught several years ago raping and molesting little children.
Well, I was taught to drop the second s, and I'm not /that/ archaic.
I use Friend/Foe + mod-point modifiers as a karma/reputation system.
Success! At least for anyone that doesn't block cookies; or modifies them to ensure FP.
Yeah just like you can modify a cookie from an online shop to make it look like you've paid for something to get them to post it out to you... oh or is it that the cookie is just a code used to lookup the actual values on the server?
And you'd also just put the cookie value in the form, so even with cookies blocked it would work.
You didn't think this one thru huh? :-p
The revolution will not be televised... but it will have a page on Wikipedia