The Linux Filesystem Challenge

Joe Barr writes "Mark Stone has thrown down the gauntlet for Linux filesystem developers in his thoughtful essay on Linux.com. The basic premise is that Linux must find a next-generation filesystem to keep pace with Microsoft and Apple, both of whom are promising new filesystems in a year or two. Never mind that Microsoft has been promising its "innovative" native database/filesystem (copying an idea from IBM's hugely successful OS/400) for more than ten years now. Anybody remember Cairo?"

Comments

Don't try to keep up with Microsoft and Apple

[suso]

Instead, try to keep up with the demands and needs of users.

Re:Don't try to keep up with Microsoft and Apple

[Anonymous Coward]

Don't try to keep up with Microsoft and Apple. Instead, try to keep up with the demands and needs of users.

In this case, they're one and the same.

Re:Don't try to keep up with Microsoft and Apple

[jilles]

Actually that involves keeping up with the rest of the field as well. Not every feature MS adds to their OS should be duplicated. But some features are useful and should be considered.

MS has basically announced/demonstrated most of the new features that are in longhorn. Effectively that has given the linux community two years to come up with competing features. Adding database features to a filesystem makes sense, beos has demonstrated that you can do some nifty stuff with it and both apple and MS have anounced to do this.

The linux community however is divided. You can install reiserfs, maybe develop some tools that use some of its more advanced features but that doesn't fundamentally change anything if openoffice, KDE and Gnome and other programs don't coevolve to use the new features.

The same goes for stuff like avalon. While everybody is still talking about how such technology might be used in OSS projects like mozilla, Gnome, MS is well on their way of implementing something that may actually work.

Filesystems with rich metadata were already a good idea ten years ago. The OSS community has talked about them where others have implemented them. Two years of more talking would be fairly consistent. IMHO the OSS community is underperforming in picking up new technology and good ideas.

Re:Don't try to keep up with Microsoft and Apple

[kfg]

Users often demand and think they need all sorts of pointless, worthless, daft shit. Commercial companies, of course, have to cater to this, and the less ethical directly exploit it ( I'll sell you speaker cables that I've meditated over while sitting under a mystical waterfall to infuse them with energy and align their molecules, only $2000 a set. If you can't hear the difference it's because your chakras are blocked, but don't worry, I've developed a homeopathic remedy, only $20 a bottle. Oh, they only work while listening to my taped lecture series though, just $499. Remember to sit on my special magnetic pad at the same time (available to members only)).

How's about this for a better idea, instead of trying to keep with Microsoft try to keep up with sound software engineering principles in designing our file systems?

There may even come a time when the required action to impliment this idea is to do nothing.

KFG

easy answer

[dAzED1]

nfs4, with solid integrations for auth servers (ldap to active directory, etc).

We live in a network-based universe. Local filesystems are already good - whether its just continued development in Reiser, or whatever else.

Nfs4, though - its like afs, only without the sucky stuff. AIX is now including nfs4 in its AIX5.3 release, even! With the Big Dog on board, we should realize there's wisdom in that direction ;)

Filesystems are tools

[tikoloshe]

Filesytems are tools that will suit different purposes. Some are good for databases, some for lots of small files, some for lots of reading, some for writing, some for networks, some for streaming.
So to develop a one handy "swiss army knife" of filesystems may not be the best route. For the most part one knows what a system will be doing and can build in the most appropriate filesystem for the job.

Re:Filesystems are tools

[beee]

A good filesystem should be capable of handling all potential applications (for example, FAT32 has found its way into grandmother's desktop and production web servers). Specializing a FS is a huge mistake, and any highly-specific FS introduced to date has been a huge flop. This is not the best route to travel for Linux.

Re:Filesystems are tools

[kasperd]

A good filesystem should be capable of handling all potential applications
I absolutely agree. And I actually think the current interface to filesystems is good. I don't want any major changes. Because major changes would most likely lead to all new kinds of metadata that no applications know how to deal with. And whenever your files get handled by a program without this knowledge, you are losing metadata which again means new applications that makes use of the metadata get screwed. So most of this inovation will just give us lots of compatibility problems. If anybody really want to inovate, and produce something good, then they should implement a clever implementation of the existing interface, that works well for different cases, that is both small and large files, deep trees, many files per directory, few files per directory. AFAIK reiserfs and XFS are doing quite well.

(for example, FAT32 has found its way into grandmother's desktop and production web servers).
FAT is a horrible example, because it didn't become this widely used because of quality. Minix' FS is simpler than FAT, it have more features, and it is a lot faster for random access to large files. FAT-16 had problems with small files, because on large partitions you were forced to use large clusters, which means lots of disk space wasted (I have seen 30% waste in real life cases). FAT-32 did improve on the problem with small files, because now you could have much larger partitions with 4KB clusters. But since FAT-32 still use linked lists to find data sectors (like previous FAT versions), FAT-32 is worse at handling large files than any previous filesystem. For example seeking to the end of a 625MB file in 4KB clusters requires following 160000 pointers. Most other filesystems use a tree structure, which means you can typically index the entire file with at most 3 or 4 levels of indirection, which means you need to follow 4 or 5 pointers. Would you try to cache the FAT table to speed up the access? Good luck, you would need 4 bytes of FAT table per 4KB cluster on the disk, so for a 160GB disk you would need to use 160MB of your RAM just to cache the FAT. And this doesn't get rid of the CPU time required to traverse the linked list.

Re:Another solution in search of problem

[lightknight]

Right, and how often do you misplace files?

More than three times a week, and that's criminal.

I mean, throwing things about in your home or My Documents directory are fairly standard. How often do you put your (picture) files in a \qw3r3et354t\bchnjc8g45\3j4n45g9u98d directory?

While everyone seems to see WinFS (and associated services) as some sort of search panacea, your ability to retrieve those files is linked to 1.) its metadata and 2.) your ability to recall a search term that appears in the metadata. If your search for "bird" and the metadata specifies "hawk", short of a dictionary search, you still cannot find it. It doesn't matter if the uber search capabilities can span the entire hard drive in 5 secs, and run through multi-dimensional data. You still need a search term, and that search term (in whole or in part) must appear somewhere in the file, be it the filename or metadata.

Essentially, WinFS makes data appear more ordered (assuming you take the time to fill out the fields). Otherwise, it's useless.

not so fast ...

[vlad_petric]

While I agree with the atomicity part, it's all great provided that the code is bug-free. IIRC reiserfs bugfixes where quite frequent in kernel changelog a couple years ago.

Filesystems are so crucial to OS stability, that I'd say it's worth formally-verifying them to a certain extent (i.e. prove that the algorithms/code work, instead of just observing that they work in normal conditions).

P.S. The whole thing - filesystem as a DB - is complete crap. You can't do a bunch of fs operations in a single transaction and have ACID semantics on the transaction as a whole. Sure - searching is great. But database means much more than just a searching interface.

Re:not so fast ...

[AstroDrabb]

ReiserFS 3 had bugs in the early versions just as all software will. That is why reiserFS was not used for productions systems for a while. It will probably be the same with ReiserFS 4. I will use it at home when it first comes out, but not where I don't want to chance data corruption.

Re:not so fast ...

[jbolden]

The backwards compatibility problems are insurmountable

They aren't a problem at all. Every email system can identify file formats it doesn't know how to deal with. Most can get external plugins. The file + attributes can be seen as just a type of file (like say .att). So if you support the .att format you would see a doc plus an icon plus history plus... otherwise you just see a .att file that needs some external app to understand.

Palm

[gveloper]

Doesn't Palm OS have a database/filesystem hybrid too?

Re:Another solution in search of problem

[kfg]

When storing files in a database retrieval is dependant on metadata. That metadata is not derived by magic, it requires human input. You may automagically determine that a file is a jpeg, but classification as a jpeg of a bird is a cognative decision. Maybe you aren't even interested in the bird at all, but in the hemlock limb the bird is sitting on. Unless someone has supplied that metadata you just as lost finding the jpeg of the hemlock branch as you are in finding randomly named jpeg of a bird.

Filenames are metadata and are just as much under user control as database metadata, no more, no less.

KFG

Not gonna play with alpha code

[prisoner-of-enigma]

The SourceForge page says:

• Development Status:: 3 - Alpha

Sorry, I'm not about to trust archived video to alpha code, or even beta code. If there's no release-worthy option on Linux, we have to stick to NTFS on Windows.

Re:Another solution in search of problem

[runderwo]

Finally, what bloat is today, is necesary tomorrow. Imagine an oracle database on hardware from the seventies. Bloated beyond imagination, dog slow. But since the seventies the amount of data stored in a database has grown tremendiously, to the level where we simply need databases like Oracle or SQL Server to store it.

Bloat isn't an absolute metric. Bloat is the ratio of the memory and execution footprint of a program to the useful work it gets done. A program which does the same amount or less useful work than another program, and which is twice the size in core and uses twice as much CPU time as the more efficient program, is referred to as bloated. It would be illogical to refer to a database server such as Oracle simply as bloated, unless it were possible to point out a competing database server which is equally as useful and which has a smaller footprint, either due to careful coding or to better algorithms. In the case of Oracle, this might be true. But just because it doesn't produce useful work on the hardware of 30 years ago doesn't mean it isn't a well engineered piece of software.

A better label to use would be "complex". To respond to your argument that the only obstacles to db-fs are ignorance and blind conservatism, complex software is undesirable. It increases costs in terms of man hours to maintain it, it increases QA overhead, and it increases support calls from users who came to depend on a feature which was included for completeness, but was never audited for correctness or robustness. People don't code complex software unless they are paid to do it (and usually when a manager is making the technical decisions). This is the reason most open source/free software tools seem to follow the Unix philosophy; simple tools which do one task and do it well, but are yet flexible enough to build into more complex systems. A monolithic database filesystem does not appeal to the sort of psyche which produces open source code for that reason: Complexity doesn't make a programmer's job fun. In order to produce large amounts of code at a low cost as in the open source/free software world, the people behind the engineering of the software need to be having fun, and a complex database filesystem is a rather good example of something which is _not_ fun to produce and therefore unappealing to the hacker sort.