Slashdot Mirror
The Linux Filesystem Challenge
Joe Barr writes "Mark Stone has thrown down the gauntlet for Linux filesystem developers in his thoughtful essay on Linux.com. The basic premise is that Linux must find a next-generation filesystem to keep pace with Microsoft and Apple, both of whom are promising new filesystems in a year or two. Never mind that Microsoft has been promising its "innovative" native database/filesystem (copying an idea from IBM's hugely successful OS/400) for more than ten years now. Anybody remember Cairo?"
Hans Reiser has written a white paper containing his thoughts on the design of the next major version of ReiserFS.
...wrote "Open Sources", which you can read/buy here. He's a fairly savvy fellow...
The Army reading list
Hans Reiser has some interesting ideas about the role of a modern file system. Here's a recent USENET post describing some of the immediately visible features of reiserfs v3. Some people have said that there was corruption in the past, but I think there are no longer any problems in recent 2.4 kernels. Namesys is now developing Reiser4, which appears to be more flexible (still needs time to stabilize though). If I had to put down my money on a future filesystem though, it would be ReiserFS.
dtrace, due with Solaris 10 does that. So it's not quite a top equivelent, but it does laet you answear your questions ("What processes are kicking the shit out of the disk", and "By how much"), and long with the also useful "In what way" i.e. many small writes, hugh seek to read ratio, or what have you.
It is, however, expert driven, unlike top, which is simple to use. Still, I think that dtrace shows the furture of performance monitoring apps.
Note that dtrace lives partially in the kernel - it's not portable to Linux.
Notice the plugin feature. This will create endless possibilities for what you can do with the file system. Want to tie a DB/SQL search function in to it? Write a plugin, want special security? Write a plugin. Tons of possibilites with ReiserFS4 and it is _very_ fast. This is hands down better then the MS "a filesystem as a DB" approach. ReiserFS4 will be like Firebird, lean-n-mean-n-fast. Want more features, grab _your_ favorite plugins!
If Tyranny and Oppression come to this land,
it will be in the guise of fighting a foreign enemy. -James Madison
And neither of whom have a journaled filesystem yet, while Linux has many to choose from.
... you get the point.
What are you talking about? NTFS has had journalling for over a decade. And Unicode. And ACLs. And streams. And reparse points (these are amazingly cool). And compression. And encryption. And
Now, MS doesn't use most of this good stuff, but it's all in there. Even three-letter file extensions on Windows are obsolete, since everything on NTFS can be an OLE server. There's nothing on Linux that comes close to the capabilities of NTFS. About the only major thing NTFS is missing is versionning, which VMS has.
If you load everything on the filesystem to memory on boot, you end up wasting a lot of memory, since you typically use only a very small subset of your filesystem at any given time.
The solution would be to load things "on demand," as you've suggested.
Linux already does this, and it does more.
If you've ever looked at the output of free(1) after your system has been running for an hour or so, it will appear as if almost all your memory is in use. See those last two columns, "buffers" and "cached"? That's your "on-demand ramdisk" at work.
Linux will use memory that applications aren't using to cache filesystem data (including executables and metadata) to speed future accesses. If your applications need more memory than is currently free, the kernel will drop cached data rather than swap out application memory to disk. That way, you get the benefits of having your executables on a ramdisk, with the flexibility of not having to sacrifice running application performance in the process.
Man, you totally miss the point. NFS is not a file system (don't be fooled by the name), it's a network protocol. The files provided by a NFS server have to be physically stored on some (real) filesystem, like ext3 or reiserfs.
This is very much like saying "the future of filesystems is apache2, local filesystems are already good, now we have to concentrate on apache2".
Offsite backups are your friend. No matter what your filesystem's software, or the coolness of your raid array, or your battery-backed redo-logs; if a fire or a burglar takes your disks holding your filesystem you're hozed.
Personally, instead of a raid, I do a nightly "rsync" to a "yesterday" drive on a separate server (hense protecting myself from stupid-user failures as well as filesystem/disk failures); a "every time I did something significant" rsync to an encrypted filesystem removable drive kept in my car; and a "once in a blue moon" copy to DVDs in a safe.
An added benefit - upgrading an OS, or a computer is trivial, because the live backups are just that - live, and tested every night.
(Back to the filesystem topic, Reiser's whole naming idea is so much cooler than a heirarchy or a relational system I really hope this is the next big advance for Linux).
Apple is simply adding functionality to HFS+. Everything you've read about Spotlight describes a services that sits above the file system. It takes advantage of HFS+, but there is NO database driven FS coming out from Apple.
Their solution is to build a service that can interact with individual files, including their native metadata (ID3 tags, pdf metadata, MS Office metadata, email headers, etc.) through metadata importers and to store the metadata indexes in a separate database. This is relatively similar to how iTunes does it's thing. The services will have lots of APIs open to apps to incorporate the functionality locally.
The obvious clue that HFS+ isn't going away is that Apple is finally pushing full HFS+ support back up to the command line utils like cp to support resource forks and whatnot in 10.4, so hopefully we can stop needing OS X specific tools like ditto.
They've been adding improvements steadily over the years, such as journaling and most recently case sensitivity. The more obvious question to me is why doesn't the Linux community just jump all over HFS+ and build off of Apple's work since they seem more than willing to give the HFS+ support back anyway?
Sorry, but you are wrong here. Reiser4 is atomic and you can pack as many operations into one transaction as you like, you just have to use the reiser4 system call. This is, because there is no standard system call for atomic filesystem transactions. Modern filesystems are databases, build to store files and query them trough filenames, reiser4 is the first filesystem where search path can be done through plugins, therefore you can index everything you want.
kindly regards daniel
If you're concerned about compression speed, you may want to take a look at LZO. It's got incredibly fast compression, and even faster decompression. I think it was even used on the Mars Rovers.
Reiser4 has a compression plugin coming. We got gzip to work, but it consumes too much cpu, so now we are doing lzo which can compress at disk drive speed. The lzo plugin has a bug, maybe next week....
Hans
(You can email edward@namesys.com for details).
It's a pity no one has ever fully finished the NTFS filesystem module for Linux. I understand that payware solutions are available, but the "EXPERIMENTAL" read-write NTFS module has been around for years and years with nobody finishing it up. Right now you can write to an NTFS volume so long as you don't change much of anything...yeah, uh huh. That's real useful. And if you do accidentally change something, you can totally fsck up the volume, or at the very least you have to do a CHKDSK before anything else can access the volume. That's about as useful as a square bowling ball.
In the end they will lay their freedom at our feet and say to us, Make us your slaves, but feed us. - Fyodor Dostoyevsky
uhhhh... what?
if the filesystem does the compression, the apps (or you) can't see it happen. that's the POINT. your suggestion, above, is ridiculous. If you had a tar.gz file, you could extract it to the FS, but it would actually be equally compressed (cause it's a gzip compressed FS), and then you could play with the files to your heart's content, without worrying about the compression, cause it's transparent. You wouldn't need or want some kinda plugin or something...
Unless the FS wasn't compressed, and you wanted a transparent way to access tar.gz files. That idea would make sense.