Slashdot Mirror
File System Design part 1, XFS
rchapman writes "Generally, file systems are not considered "sexy." When a young programmer wants to do something really cool, his or her
first thought is generally not "Dude, two words... File System." However, I am what is politely termed "different." I find file systems very interesting and they have seldom been more so than they are right now. Hans Reiser is working on getting Reiser4 integrated into the Linux kernel, the BSD's are working on getting a journaled file system together, and Sun Microsystems just recently released a beta of ZFS into OpenSolaris. "
gonna party like it's 1979zzzzzzzzzzzzzzzzz
Oh, snap. Somebody's not running Soft Updates.
(Yes, I understand that Soft Updates is not technically metadata journalling as practiced by the Linux people. No, I don't believe there are a significant number of practical situations where the results will differ.)
If you're interested in this, you'll probably also be interested in Practical File System Design with the Be File System (PDF), by Dominic Giampaolo, the designer of the Be file system. There's also a Slashdot review of this book.
Bogtha Bogtha Bogtha
Sector size on hard disks is 512 bytes, not 512kbytes. WTF, don't act like an authority and be a dumbass. Imagine the data waste if we actually had 512k physical sectors on disks.
Also the scaling numbers are completely hokey.
--Brandon
You've not played much with Ext3, then, have you? =)
He didn't say it very eloquently, but the difference between 512 bytes and 512 kilobytes is pretty sigificant. 512KB is a half of a megabyte. Can you image if a hard drive sector was .5 megabyte?? You'd end up with tons of wasted space.
So constructing a complier from stratch is no longer sexy?
With everyone and their parrot talking about RAID these days, it would've been fun if some sort of dual array would work as ONE filesystem; where one(++) redunant set took care of the balancing/tree'ing, etc., (separately,) and the other(s) kept the actual files. If there was _yet_ another set (a ++third), with the relevant META-information belonging to the files, you would imagine it to be a step forward to what is now, well; I can, anyway..
A horse can't be sick, you know, even if he wants to.
If you like on disk file systems you should read Venti: a new approach to archival storage.
Plan9's primary on-disk storage is Fossil, which runs in user mode. (Plan9 doesn't have a super user)
You can run arbitrary programs in Plan9 that present a file/folder directory structure by using the common 9P protocol. All devices look like files and folders and can be manipulated like any other, even at the permission level.
For instance, I have an image mounter that takes a tga file and presents 1 folder containing 4 files, red, green, blue and alpha.
I can then use any tool I like to manipulate those files using the file semantics we are all familiar with. I even have a flag that mounts the files as textual rather than binary, i.e :
00 00 ff ff
00 00 ff ff
ff ff 00 00
ff ff 00 00
and I can do image processing with awk !
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
from TFA:
... I mean:
"There is a minimum size you can write to or read from the disc. This minimum size is called a "sector," and is usually around 512k. So, unless you really like 512k files, it is very likely that you will end up either wasting space or cutting off the end of the file if your file system doesn't deal with this."
This is clearly not a typo - which is what I was certain I would find when I did RTFA. This guy has a basic, fundamental flaw in his understanding of the very thing he's writing an article about. This is a non-starter, IMO. Combine that with poor sentence structure and bad scansion
"Note: My ibook has a "30 gig" drive. This is bullshit and I'll tell you why: Drives are defined by the binary definition of mega, kilo and giga. For example, a kilobyte is not 1000 bytes, but actually 1024 bytes. However, your HD manufacturer uses the metric definitions, even up to gigabytes. Now I can see you thinking..."But Wait Mr. Mad Penguin Person...Thats patently ridiculous and means they are lying on the box." Yah... "
If I'd written something like that, I'd delete it right away and start from scratch.
Thinking outside my Head
NSS has been ported to Linux too. That's an another modern industrial-strength filesystem with features sorely needed by Linux.
What compels people to make the leap from "I've grasped the basics of a large and complex field" to "I think I'll write an article about it for the Slashdot crowd" via "I'm sure it doesn't matter that I'm not a good writer" and "I think I'll go with a self-satisfied tone"?
From the article:
Small difference there. It is also a very fast file system, allowing reads of up to 7 GB/sec.
An assumption which could only be made by a newbie. Maximum throughput of a filesystem is not filesystem architecture dependent, but hardware dependent.
I could give you 7GB/sec out of a FAT drive, given the proper hardware.
Several other quotes suggest a bit of 'newbieness' like "B+trees are insanely complex".
The concept was designed by a human, therefor it is clearly understandable by a human. It's not say, some potentially impossible-for-humans-to-really-comprehend law of nature.
The author should just acknowledge that he or she does not know enough about B+trees, or does not know them well enough to illuminate the subject sufficiently.
There's nothing wrong with that, but trying to scare people off of them just because he or she doesn't understand them well enough is damaging to potential readers.
I want to postscript this with an evaluation that the article is not bad. If those things are changed, I would even say it is good.
-fooburger
Sorry, this article didn't really teach me anything interesting about filesystems. In general, the article was poorly written. For example, taking two sentences to say: "B+Trees are complex. Let me rephrase that. B+Trees are very, very complex." Readers of all types appreciate their time and don't want to have to waste it.
You were lost at points between trying to sound like an expert to trying to sound like a grandfather explaining the grande old days of filesystem development. Are you a storyteller or a teacher? Pick one.
Content-wise, there wasn't really much there for me. You spent a lot of time explaining the problems of a binary tree, but I think that your target audience already understands the time complexity of a binary tree. Then, you glaze over the B+ tree because its complicated.
Sorry if I sound harsh. I hope that this comes off as constructive criticism.
"An inode is quite simply a data structure that stores a link to the file data, the file name and metadata."
Last time I checked, filenames were separate from inodes. Wikipedia agrees with me.
XFS and ext3 can have their metadata logs/journals on other physical devices, seperate from the actual filesystem blocks.
Sometimes filesystems are RAID aware, in that they choose to allocate blocks at the beginning of RAID strides and stuff like that, But that's about as flexible as filesystems get.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
Sorry, but if you really like file systems maybe you should try learning something about them before deciding to write this kind of articles. I've had less than 10h of classes on file system design, do not consider myself to know anything about the subject and still got the impression that I knew quite a bit more than you.
Come on, the guy did most of his research on Wikipedia. I think that expecting any sort of familiarity with or understanding of the material he is discussing is unreasonable.
Who gives the fuck what Microsoft names it's kiddie boxes and assorted file systems? XFS (as in filesystem) has much longer history than "XFS on Xbox".
No, I was thinking more along the lines of when(/if) META-data becomes big, and you'd get further throughtput by having it on its own drives, so as to speed things up.
By all the three examples I provided, I tried to "account" for both speed and reliability, even though it's only a vague theory..
--No wonder (_real_)things keep standing still for fscking 10 years at the time, and only Disney features are implemented; people turn down theories just as snappy as they turn down webdesigns (50ms, or whatever)..
A horse can't be sick, you know, even if he wants to.
No, XFS is the filesystem created by SGI. You might want to reconsider your post.
No, it's not. That filesystem is called 'FATX' on Xbox1 and 'XTAF' on Xbox360 (endianness).
Here: http://labs.google.com/papers/gfs.html. Abstract: "We have designed and implemented the Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients. (...)" Pretty damn cool stuff, very advanced but perhaps a bit too tuned for Google's needs. See also the papers on their own clustering technology and distributed programming framework (MapReduce), which go hand in hand with GFS. These are some of the technical secrets behind Google's search engine and other apps, it's funny how little you hear about them. For massive tasks like Google search, I don't think anybody can compete with standard technology, i.e. big machines (vertical scalability), off-the-shelf clusters and SQL servers, and conventional programming techniques.