State Of The Filesystem

Skeme writes "Have you heard of Plan 9 or Reiser4 but don't know much about them? Are you curious about the improvements free software is making to its filesystems in general? Read my summary of the current developments in the filesystem: namely, what improvements we can expect (a lot), and what Linux and the BSDs can do to improve on the filesystem."

Comments

Transferring Files

[jhunsake]

I've always wondered how these filesystems with metadata handle transferring files between different systems. It would suck to have all your MP3 info in filesystem metadata and then lose it all when you transferred to a system without fs metadata. Anyone have any insight?

Re:Transferring Files

[ThogScully]

I would think that would be prevented by the filesystem implementation. A filesystem's features are only as good as the implementation. So if you have fs metadata and are transferring it to another filesystem, whatever tool you are using to move the files, whether it be at a BASH prompt or drag and drop in Konqueror, should figure some way of properly translating data from one system to another without losing anything. Moreover, the tool should be able to do it without knowing any particulars about the actual types of filesystems in use. It's all in the implentation.
-N

Re:Transferring Files

[sleeper0]

i am skeptical as well, the only reasonable answer seems to be that it would get lost. Especially if you are transfering between platforms. Why would you want to use a metadata system that you can't count on anyone else using? It seems to me that current packaged in file metadata systems are much more practical for transfer, multi-platform use, streaming, etc.

Other things mentioned in the article like the /etc/passwd manipulation just seem like gimmicks. How many files like that would you by hand support before you ended up deciding it was a bad idea to start it in the first place.

If these are the things they are coming up with to add to filesystems now, they must be pretty much figured out & done.

Re:Transferring Files

[warrax_666]

Anyone have any insight?

Are you mad? This is slashdot.

Re:Transferring Files

[Anonymous Coward]

It wouldnt be hard to extend the scp command to send metadata across if there is a compatable (ie. up to date) ssh server running that supports accepting metadata descriptions for files. The remote machine can then do as it pleases with the metadata.

What is needed is a POSIX specification for handling file metadata. Then applications, servers *and* filesystems can be gradually upgraded to include the metadata features in a portsable and consistent way.

--
Blue SSL

Re:Transferring Files

[groomed]

Generally you lose the data, unless you wrap it in another format to encapsulate all the information. This is what Macheads did on Classic MacOS: they .hqx'd or .bin'd their files before transferring them to another system. It's not ideal. The alternative, flat streams-of-bytes, is not ideal either (and not true: even in Unix, files have some metadata that doesn't translate very well).

Hopefully in the future our filesystems and transfer protocols will evolve to have some reasonably broad common ground where metadata is concerned (a development similar to the diminishing need to accomodate DOS 8+3 filenames).

Re:Transferring Files

[axxackall]

Metadata is very application specific and most of filesystem are agnostic about it. Typically it must be handled by another layer on a top of FS.

Often that layer is a DB - database. I suggest you to try ZODB, database in Zope, it's very good to handle files as documents - with many unified metadata about files.

Another good example to study is Subversion, which is revisionining/versioning metadata-management layer on a top of a regular FS.

You may research and find some software implementing a layer (on a top of a regular FS) specially designed to handle MP3 playlists. But again, that would be a layer on a top of FS, not a filesystem by itself.

Re:Transferring Files

[peatbakke]

There are a few of ways to solve the problem, but only one of them ensures compatibility across many different platforms:

The first is the worst: having a separate metadata file accompany each of your files as they move between file systems. This is a logistical nightmare, and requires soooo many tools to be rebuilt that it's not even worth considering.

The second is the "bundle" concept, as popularly demonstrated in OS X. "iTunes.app" is actually a directory, not a big binary file. The directory is split-up into different sub directories, containing multiple binary formats, languages, etc. To transfer iTunes to another computer, just tar/zip the "iTunes.app" directory, and viola. Easy. Of course, the app will only run on platforms that recognize the bundle format, and can execute the binary. This rules out backwards compatability, and requires that all major system vendors agree on the same bundle format -- and that's not gonna happen.

The third way to do it is to keep metadata within a file (like MP3 tags), and then export them into the filesystem space. For example, an MP3 plugin would pull the data out of "my.mp3" and turn it into "my.mp3/title" "my.mp3/genre" etc. The nice thing about this is that it's platform independent, and backwards compatable.

I'd be keen to see option #3 implimented in a modern file system -- do any current FS's have an API that could support it?

Re:Transferring Files

[Surak]

Well, remember Reiser4, for instance, is a TOTALLY DIFFERENT system than the standard POSIX filesystems (Reiser3, Ext2/3, UFS, etc.)

The idea is that data is stored as in a database, and small files can be handled efficiently.

Think of it this way: What *is* a filesystem? Really, it's a relational database. We don't tend to think of it as one, because it isn't implemented as one, but that's what the data it represents is. If you think of each file and directory and associated metadata as a record, with directories being pointer records to other records that show relationships.

Anyways, code needs to be implemented to handle this.

We use metadata NOW. A files permissions, owner/group info, date and time stamp, and even the filesize are metadata. We transfer those between different systems right now. With the correct transport method, this metadata doesn't get lost at all. A tarball can sit on any system no problem. Inside the tarball is all the POSIX metadata. That gets transferred from system to system.

Of course, some OSes (read: Windows) don't handle Unix-style permissions correctly, so these are translated into rough-equivalent ACLs with systems such as Samba.

Re:Transferring Files

[n3rd]

I've always wondered how these filesystems with metadata handle transferring files between different systems

Usually it doesn't. Take for example VxFS (Veritas File System); they support attributes like preallocating space for a file or forcing a file to be contiguous. When you move a file that uses these attributes to another type of file system (VxFS to UFS) you lose them since the target file system doesn't support those attributes.

Another example is ACLs. If they are in some way propriatary they won't transfer and sometimes won't work well together. The situation I ran into was a HP-UX NFS server's ACLs not working correctly with a Solaris client. The server used and enforced the ACLs however the client couldn't view or modify the ACLs on the files that were stored on the server.

If the implementation is truly at the file system level there's nothing you can do. However, as stated below, if there is a layer above the file system that handles metadata then you can more than likely keep the metadata intact.

Re:Transferring Files

[gilesjuk]

That was a right pain in the butt, having resource and data forks etc..

At least with AmigaOS they stored similar things in another file (.info file). Much easier to transfer, but you do run the risk of losing the .info file.

Re:Transferring Files

[ThogScully]

If the other filesystem has no way to store metadata, then perhaps someone will write a sort of add-on for that filesystem that attaches an XML file to something.mp3 that is called something.mp3.meta that will contain the metadata and because that person is running the "patched" filesystem, nothing will be lost.

As for SSH, that's just one of the tools I spoke of - the tools will have to implement the features of the filesystems - it's not up to the filesystems. If a filesystem addon like I specified above becomes common enough, then perhaps it too will become supported in various tools.

Ultimately, the point remains that the tools will support newer filesystem features and do their best to operate as translators from one filesystem to another. If one filesystem really just can't support the data on another, then something will have to be lost, but if that's the case, then perhaps it's time to upgrade the older filesystem.
-N

Re:Transferring Files

[dash2]

True, but as he pointed out for chmod, chown etc., most of these command line user tools could still be implemented, sometimes even as shell scripts.

Obviously, no clever new idea will go anywhere (except on e.g. embedded systems) unless it is compatible with the huge base of existing Unix software, command line tools included.

Re:Transferring Files

[paul.dunne]

> If you think of each file and directory and associated metadata as a
> record, with directories being pointer records to other records that
> show relationships.

Sorry to be snippy, but what you're describing is an *hierarchical*
database. But you're right in your main point: an FS like ext2 is an
example of an hierarchical database.

Re:Transferring Files

[thogard]

Posix file systems aren't required to be hierarchical they just of sort of look that way. Remember hard links? The name space is hierarchical but files and directories can appear in many locations. link (no ln) has been able to create hard links for directories since Unix version 2.

Re:Transferring Files

[greenrd]

If a filesystem introduces new types of metadata, they don't magically get supported by tar.

Yes they do, if they're seen by tar as ordinary files. That was one of the main points of the article, which not many people here seem to have read (as per usual).

Re:Transferring Files

[zatz]

A filesystem is nothing like an relational database. I wish people would stop making this comparison, because it's completely misleading and unhelpful. A filesystem is not a set of user-defined tables, each of which contains an unordered set of rows. Queries and joins are not possible. Constraints and null values are not supported. Files within a directory have an inherent order. Files are variable-length and byte-addressable. Duplicate "rows" are not permitted. The principle relationship modeled is hierarchy... ever heard of a hierarchical database?

Re:Transferring Files

[jafuser]

With the system that was proposed, you would probably need some kind of file packager system (with a rather extensive and flexible library) to assemble a file and it's meta-data into a standard "network" file type before you ship it off to a computer which does not have this kind of filesystem.

Probably something like:

$> mkfile ~/songs/OompaLoopmas

Would take the fs-organized file and meta-data:
~/songs/OompaLoopmas
~/songs/OompaLoo pmas/title
~/songs/OompaLoopmas/artist
~/songs/O ompaLoopmas/genre
~/songs/OompaLoopmas/year
~/so ngs/OompaLoopmas/album
~/songs/OompaLoopmas/track
~/songs/OompaLoopmas/..uid
~/songs/OompaLoopmas /..gid
~/songs/OompaLoopmas/..rwx
~/songs/OompaL oopmas/..type (contains "audio/mpeg")

and make the following basic "networkable" file, with no meta-data other than standard file-system meta-data:

~/songs/OompaLoopmas.mp3
~/songs/OompaLoopmas.m p3/..uid
~/songs/OompaLoopmas.mp3/..gid
~/songs/ OompaLoopmas.mp3/..rwx
~/songs/OompaLoopmas.mp3/. .type (contains "application/octet-stream")

Applications which use network sockets would be aware that before sending a file across the network, it should probably convert it into a network file first.

Perhaps there would be a way to fopen the file with a flag indicating it needs to be run through the library which converts the file into it's original "network form" and feed that stream on the file handle returned from fopen().

It would be much more taxing on the filesystem to change the original file to reflect the meta-data every time something in that meta-data is changed. If one byte of data needs to be added to the front of a long file, that could require the whole file to be re-written. These kinds of changes will probably occur a lot more frequently than the times where a file needs to be made "ready" to go out on the network.

For that reason, I think it would probably be more efficient if the conversion process is made at the time the data is copied to or from the network. A system of libraries to do this conversion would probably be more elegant, and would be much more well integrated with the pupropse of the file system described in the article.

As a bonus, the packager could perhaps also be intelligent enough to do file conversions through some additional modular libraries (if they are available).

$> mkfile -t ogg ~/songs/OompaLoopmas

Obviously you wouldn't be able to convert all files to all types.

$> mkfile -t jpg ~/songs/OompaLoopmas
Error: ~/songs/OompaLoopas contains 'audio/mpeg' and cannot be converted to 'image/jpeg'.

I like most of the ideas proposed in the article, and I hope they get a chance to make it into the mainstream, though I'm sure it will take quite a bit of time before that can happen.

The only thing I'm not quite comfortable about is the mount layers idea, as that seems it could make room for a lot of confusion if it's use is not kept to a minimum.

Re:Transferring Files

[dsarrazin]

Right, but if on the destination FS, files-as-directores aren't supported, you've lost you ability to play any of the files, because they are now stored as directories.

Re:Transferring Files

[spitzak]

Absolutely #3 is the way to go. "metadata" should always be figured out from the contents of the file, the metadata should really be considered a cache of the answers.

The way it would work is a program would look for a piece of metadata. If it was not there it would run a standard program using popen that would then create the metadata and return it, and also set the metadata on the file if it had permission to. This program would probably work something like "file" does now and would use text configuration files to actually determine how to extract interesting information from all known file types. Most likely this actual implementation of looking up metadata would be wrapped in a libc-level function so it just looks like you ask for the metadata.

The obvious advantages:

Files can be transferred by protocols that don't understand metadata.

Files can be stored on filesystems that don't understand metadata.

Metadata is never "lost" as long as the real file data is intact.

Metadata can be stored locally for remote files.

Metadata can be modified without permission to modify the file.

Probably much faster because metadata is not calculated until first needed.

I also think that all files should be treatable as directories, and that the metadata should be presented as being inside this directory (ie the metadata "blah" for file "foo" is foo/blah). But this does not mean it has to be stored this way and does not mean it has to exist when copied.

But when

[infolib]

will ReiserFS be ready for the 2.6 kernel? Just curious.

Why?

[Libor Vanek]

It's really nice. But what does it brings new that we shoudl rewrite 90% of all system tools too use this new features? I find "cp /a/..uid /b/..uid" same as "chmod /a --reference=/b"...

Re:Why?

[cperciva]

You're missing the point. chmod would still exist as a userland program; it is the kernel call which would be removed.

To the user, there would be no change; to the userland programmer, there would be no change; to the C library developer, there would be a change (to implement chmod in terms of the existing filesystem operations); and to the kernel developer there would be a change (mostly in the direction of reduced complexity due to a smaller number of necessary functions).

Re:Why?

[IamTheRealMike]

I think people are distracted by the examples he gave, which make the point clear but are perhaps not representative of what this would be used for in real life.

GConf was a better example. ATM using GConf is, well, not hard, but you have a lot of extra machinery involved, new APIs to learn and so on. Basically all that machinery does is control the backends and give change notification (it does stuff like schema validation as well).

It'd be *much* easier to use GConf if in order to read a value, you didn't have to load up the GConf libs (which in turn depend on CORBA), or parse XML files. At the moment that's really the only way to do it, but in most environments/languages it's far easier to manipulate files and directories than it is to talk to a CORBA server or bind APIs into them.

You also get an increase in efficiency. Parsing XML is kind of cludgy - XML is not a particularly efficient format to store stuff in. It's a good compromise between humans and machines, but both of us have to do lots more work to meet in the middle. The reason it's used, rather than lots of small files, is that otherwise GConf would be too slow. In fact, they are already talking about removing yet more of the files/directories to speed things up, and sticking them all in the same XML file.

Being able to have a configuration system that truly leveraged the filing system would make a lot of stuff easier, more reliable, and faster (because you can take advantage of filing systems that are really really tuned to take advantage of advanced data structures).

It won't really impact the way you do things like set file attributes today. Most of the changes would be under the hood. But used well, everything would become easier for the developers, and so more advanced and slicker for the user.

Re:Why?

[LordBodak]

You're right, XML isn't the most efficient way for gconf to store data. But is rewriting the filesystem really the answer?

plan 9 is awesome

[Sespindola]

There has been talk in the kernel mail list of implementing 9p, the plan 9 distributed filesystem.

Now that is Open Source, I hope it will happen.

Re:plan 9 is awesome

[F2F]

9p is a protocol. the file systems in Plan 9 are venti and fossil

9p in itself is worthy of including in the linux kernel, if only the guys there would do it right (their track record isn't too good with Plan 9 things).

More about 9p dould be found in section 5 of the man pages

very Reiser/plan9 specific...

[grey1]

...and not very general. Interesting for its comments on what's being tried out in R-FS & Plan9 but certainly doesn't manage to be a general summary of what's going on.

How about the changes coming in 2.6 (like xfs support built in)?

The article makes some good points but for me it could have done with rewriting to make it more general, separate the analysis of filesystem implementation problems from technical detail, and included more examples from other file systems.

Re:very Reiser/plan9 specific...

[axxackall]

How about the changes coming in 2.6 (like xfs support built in)?

XFS is a good example of journalling filesystems. But how about filesystems like Coda, AFS and Intermezzo, a new generation of networking (actually - distributed) filesystems allowing disconnected operations?

Good article

[IamTheRealMike]

It doesn't tell you anything you can't already learn by reading up on the articles on Plan 9 and the whitepapers on namesys.com, but it's a well written compressed version for those who perhaps don't want to wade through a description of set theoretic namespaces (whatever they are ;)

The concept of reducing primitives is a good one, and based in sound mathematical theory. As already pointed out though, you need some container format that can handle some of these new ideas, things like very small files, files as directories and so on. This is needed, because when you transfer files through lossy mediums like the internet, or older filing systems, you don't want to lose the structure.

As far as I know, there isn't a container format that can do this. Tar is showing its age already, I wouldn't like to see it hacked yet again. Zip is alright, but you'd need to break compatability to add in all those extra features, and then it wouldn't be zip anymore. It'd also be inefficient.

So, what I propose is rather than reinvent the wheel to solve this problem, we add support for "boxing" to the Linux kernel.

A box is a filing system in a file. We already use them, to some extent - it's been possible to mount ISO images using the loopback filing system for a while. What's needed is to take this to the next level. The first thing is that we need the ability to use files as mount points, not just directories. When files and directories are the same, well, I guess that should be easier.

The .box file format simply contains a short header with some useful metadata, like maybe a checksum, and the filing system it contains (maybe that isn't needed). The fun part is the UI. What you need is the ability to right click on any dirfile (for want of a better term) and choose the "Box it" option. You'd need a better label. What essentially happens then is that the heirarchy below this point is sucked up and transformed into an ISO containing perhaps a "Reiser4-Lite" filing system. You can forgo the journal and other things that are redundant purely for storage.

The user has then converted their file or directory into something that can be transferred across the net, on Windows compatible CDs and so on, without losing the inherant structure of the original.

At the other end, choosing the "Unbox" option mounts the contents of the box using the loopback FS, mounted at the point of the file. To the user, it is seamless, far easier than zips or tarballs.

Of course, there are lots of complications. You have to agree on the format to use inside the box, for one, because the need to have kernel mods and so on makes it more complex than just installing tar.

I think MacOS has something a little bit similar with disk mountable images (.dmg) files, but the MacOS filing system is rather poor, and I don't know how easy it is for users to create them. Also the OS unfortunately applies some magic to them - for instance Safari will automatically extract the contents of the DMG file then destroy it when you download one (but other stuff does not, oops).

Anyway. That's one way to prevent loss of vital structure when transferring across lossy mediums, that I can think of. There are probably others.

Re:Good article

[Surak]

A box is a filing system in a file. We already use them, to some extent - it's been possible to mount ISO images using the loopback filing system for a while. What's needed is to take this to the next level. The first thing is that we need the ability to use files as mount points, not just directories. When files and directories are the same, well, I guess that should be easier.

Actually, although not seamless except from the users point of view, for a more visual representation, think ZIPs and tarballs in Midnight Commander. MC treats the file as a file, and then when you go into the file, you get a directory, in which you can copy, move and delete[*] the files much as in a traditional directory. The beauty is that metadata can be stored, accessed and modified in the directory-as-file version and it could probably contain just about anything you wanted, including a description of the directory structure and files, for instance with a source tree, it could contain all of the info normally contained in README, COPYING, CHANGELOG, etc.

[*] Depends on the features of the underlying file format and its associated program (zip/unzip vs. tar/gzip/bzip2).

Re:Good article

[Anonymous Coward]

Actually, ZIP is extensible. You can add new data blocks to the ZIP file and if you try to unzip it with a decompressor that doesn't understand your extensions, they'll just be ignored. The Be ZIP implementation extends ZIP with support for its native file system attributes meta data, for example.

TAR isn't so much showing its age as dribling into a bib in the old folks home.

Re:Good article

[bigredswitch]

"for instance Safari will automatically extract the contents of the DMG file then destroy it when you download one (but other stuff does not, oops)"

This is a feature of the .dmg format known as an Internet-Enabled Disk Image. See: http://developer.apple.com/ue/files/iedi.html

Re:Good article

[Have Blue]

I think MacOS has something a little bit similar with disk mountable images (.dmg) files, but the MacOS filing system is rather poor, and I don't know how easy it is for users to create them.

It's *very* easy to create a disk image. Drag a folder onto the Disk Copy app and select a destination to store the image. There's no step 3!

Incompatibilities with another system

[panurge]

Windows support for metadata has always sucked, recognised by every Mac user who moved to a PC and discovered that you had to tell the system what a file did by appending a clumsy tla to the end, and passing gently over the inconsistencies of the support for long and short filenames.

If Linux and related systems move to filesystems with really powerful metadata support, presumably the lockin would be much stronger. Moving a directory from Linux to a Windows system may be possible but the programming to do it will become increasingly painful and the risk of data loss will rise. And with mainframe integrity, why would you want to, Mr. customer?

Apart from the CS issues, is this an attempt to use the embrace, extend weapons of Microsoft against it by turning the Linux filesystem into a full mainframe system, effectively squeezing out Windows servers by a convergence between big tin and small boxes? I guess this is pretty pie in the sky but I'd like to think so.

Re:Incompatibilities with another system

[sql*kitten]

Windows support for metadata has always sucked, recognised by every Mac user who moved to a PC and discovered that you had to tell the system what a file did by appending a clumsy tla to the end, and passing gently over the inconsistencies of the support for long and short filenames.

Actually, NTFS support for "metadata" is impressive; you can have 255 streams per file. A stream in NTFS is what Mac users call a fork, but Macs are limited to 2, data and resource. You can happily make every file on NTFS an OLE server too and do away with file extensions altogether, if you want to. Oh, and NTFS has reparse points too - think like a trigger on a database table, but attached to a file. And NTFS has journalled from day 1, whereas Linux filesystems are only just discovering this.

So why are file extensions still in common use? Largely because people who don't know NTFS come out with statements like "windows support for metadata has always sucked" without bothering to read the documentation, so few apps take advantage of this NTFS feature.

Re:Incompatibilities with another system

[poot_rootbeer]

Windows support for metadata has always sucked, recognised by every Mac user who moved to a PC and discovered that you had to tell the system what a file did by appending a clumsy tla to the end

As opposed to Mac OS, where you have to tell the system what a file does by setting an even clumsier 4-character code hidden deep within a file's metadata?

and passing gently over the inconsistencies of the support for long and short filenames.

What inconsistencies? Correctly-written modern applications (say, those written after 1995) support long filenames. Older apps do not, but you can still use files with long names with them by using the equivalent short filename.

The 8.3 compatibility layer is a benefit for those with legacy apps.

Moving a directory from Linux to a Windows system may be possible but the programming to do it will become increasingly painful and the risk of data loss will rise.

And whom would that benefit? Possibly, Linux. But right now, Linux isn't facing a situation where customers are threatening to migrate to Windows. Linux vendors are more likely to lose customers they don't even have yet, because who wants to move to a system they're locked into and can't back out of?

How about fixing the current filesystems?!

[FyRE666]

Really, I think someone should get on with finishing the NTFS filesystem access in the kernel. With people migrating to XP it's really becoming more important that this driver is fixed (how long has it been declared "dangerous" for write use now?!).

I'd really like to know why this driver has taken so long to complete - is there some information that the developers don't have access to? Some technical reason? What?!

Re:How about fixing the current filesystems?!

[IamTheRealMike]

I'd really like to know why this driver has taken so long to complete - is there some information that the developers don't have access to? Some technical reason? What?!

Yes. They don't have access to the NTFS specs. Also, NTFS is a very complex filing system, with many different versions. You don't want to get that wrong. Resizing was a more important goal, and that has been working for many months now.

Of course, it might be included in all distros when completed anyway, due to patents MS hold on the technology.

Re:How about fixing the current filesystems?!

[spaic]

From the Linux-ntfs FAQ

3.8 How was the Linux NTFS Driver written?

Microsoft haven't released any documention about the internals of NTFS, so we
had to reverse engineer the filesystem from scratch. The method was roughly:

1. Look at the volume with a hex editor
2. Perform some operation, e.g. create a file
3. Use the hex editor to look for changes
4. Classify and document the changes
5. Repeat steps 1-4 forever

If this sounds like a lot of work, then you probably understand how hard the
task has been. We now understand pretty much everything about NTFS and we
have documented it for the benefit of others: http://linux-ntfs.sourceforge.net/ntfs/index.html

Actually writing the driver was far simpler than gathering the information.

that should be a function of the os (or a program)

[ecalkin]

there os should have a 'list' of what's supported by each fs. when you copy a file from fsa to fsb the os (or program) should compare feature and let you know (somehow) that something is not (may not) going to work right. if you copy something from the regular ext2 system that is case sensitive to a ms-dos floppy disk, something should try to remind you. or the program checks this and looks for problems.

remember that not all problems can be detected, so are you willing to live with: 'this may not work correctly' messages?

eric

no, no, no

[awx]

if that crock, that bag-on-the-side, that mess is what we have to look forward to, I think i'll switch to BSD.

I mean, acessing owner data by travelling into a directory then backwards out of it again like: vi /directory/..owner is a big ugly crock.

Re:no, no, no

[IamTheRealMike]

Uh, nobody said that'd be the user interface you use. It's simply an implementation detail.

Re:no, no, no

[GammaTau]

if that crock, that bag-on-the-side, that mess is what we have to look forward to, I think i'll switch to BSD.

I mean, acessing owner data by travelling into a directory then backwards out of it again like: vi /directory/..owner is a big ugly crock.

Are you so sure that you would hate it? After reading the PDF, I was thinking of two things:

1. He suggests that files ought to be used for everything
2. The old wisdom: "Everything in Unix is a file"

I don't think it's any more radical than treating network sockets as files. Sure, it might feel a little weird first, but once you'd get used to it, the simplicity would overweigh the clumsiness of existing implementations.

It's also very easy to wrap together a shell script that imitates the existing implementations and put it to /bin/chown or whatever you wish to replace.

Re:no, no, no

[TCM]

But the AC is right. /file/..uid is not ascending out of /file/. The name of the metadata file is simply ..uid. Files can start with dots, not to be confused by the files whose whole name is '.' and '..' respectively. But you know this, right? One might get the idea you don't.

Why pdfs?

[brrrrrrt]

I see it happening more and more that people present their summaries, articles and technical papers on the net as pdfs. This is very inconvenient.

Pdfs are nice for printing and publishing on traditional media, because you can be sure they will be in the correct layout etcetera for the printer. But on the web, where people browse between lightweight, easy html-documents, they're just a nuisance.

Please, if you must publish a pdf, publish an html version next to it.

Questions...

[debilo]

I do not know much about file systems, so I have a few questions.

Once we see the GConf example, other possibilities immediately spring to mind. In almost all multimedia formats--such as MP3, MPEG, and OGG--there is a tagging system for storing things such as the author and title. Instead of storing those attributes in the tag--yet another namespace--store it in the file-as-adirectory. I have a file / music/Millencolin/PennybridgePioneers/RightAboutNo w.ogg. I let RightAboutNow.ogg/title be "Right About Now", /artist be "Millencolin". I could add a file RightAboutNow.ogg/genre and put "punk" in it.

While this is a nice example, I wonder if there really is an advantage to this kind of file system, because it seems like it takes more effort to keep track of all those sub-files instead of keeping all the info in a single file. Anyone can shed some light on this?
Also, what if I format my partitions with different file systems, say Reiserfs 4 and ext3, could there be any imcompatibility issues? Imagine I kept mp3's on both partitions, would my mp3 player know how to handle both formats, since the tag info is dealt with differently on both systems?

Re:Questions...

[roemcke]

What I would like to see, is something like the Hurd translators, that lets you access file content through the FS-API. For instance mp3 tags would be accessed through the files "HappyBirthday.mp3/title" or "HappyBirthday.mp3/artist", but instead of storing the tags in the FS as metadata, the translator store the tags in the mp3-file using the normal mp3 tagformat. This would combine the best feaures of both worlds. Other natural uses of the translator would be to access tar or xml files.

Translators should not be restricted to mappings from flat file to director structure, but should allso allow for mappings like dir->file, dir->dir or file->file

The mapping dir->file would make it possible to implent different access right for different parts of a file.

Security?

[Inode Jones]

Before adopting any of these ideas, one must consider the security implications of doing so.

If we assume that the filesystem is decoupled from the access control layer in the kernel, then one must ensure that any operation that potentially affects security is adequately controlled.

For example, on systems with POSIX_RESTRICTED_CHOWN, the following ought to be illegal:

cp foo/..uid bar/..uid

This can be accomplished by making the UIDs mode 444. Without POSIX_RESTRICTED_CHOWN, the UID is 644. However, we have now moved a systemwide security feature into the filesystem. If multiple filesystems are configured into one kernel, then they ought to be consistent; otherwise the security model will be flawed.

As for things such as allowing access to an environment, doesn't that break encapsulation? It means for a certain filename, the filesystem must grovel through a user-space process to find the environment. Also, if a change in some external environment immediately affects some partially-related processes (e.g. daemons started from that shell), then a whole new raft of security holes will come up based on a process' environment or filesystem layout changing unexpectedly.

Cool ideas, but let's be careful lest we make a steaming pile of Swiss cheese.

databases need transactions?

[ajs]

I'm really getting tired of the ever-creeping assertion that transactions are required for [x]. At first x was ACID-compliant relational databases, and such was true because ACID was defined as such. However, then I started to see assertions that relational databases had to be ACID-compliant (mostly from the anti-MySQL camps who were ignoring the long history of highly valuable, non-ACID relational databases).

Now, in this article, I see the assertion that databases in general require transactions, and thus cannot be supported by a filesystem.

Worse, the logic is self-refuting, as the article previously states that a filesystem is a database, just a limited one. As it happens, POSIX-type filesystems are quite powerful, and let's not kid ourselves into thinking that they have not served us well for 20-30 years! Yes, changes are coming and I'm frankly quite impressed by Hans Reiser's accomplishment in finally coming up with a balanced-tree-based filesystem. Many have tried and failed where he succeeded.

That's because his was a great step forward, not because the old UNIX filesystems weren't also. Let's stop trying to re-define terms so that we can explain why the last 20 years were the dark-ages. They simply were not.

Re:databases need transactions?

[afidel]

Transactions are required for Reliable databases and filesystems. If you don't mind occassional corruption then you can throw out transactions, otherwise you need them and you need to eat the cost (memory, access speed, cpu overhead, whatever). Since PC's are generally faster then most people need at the moment making them more reliable seems like a worthwhile goal.

Re:databases need transactions?

[zatz]

It depends entirely on the sort of updates you need to make. Transactions are a much heavier solution to concurrency than locking or simply storing things in a way which facilitates parallel updates (Maildir instead of mbox, for example). The value of putting all the necessary support in the operating system remains unproven. The main "transactions" required in a filesystem are adequately supported by open with O_EXCL, write with O_APPEND, rename, and mkdir. (Or don't you remember the days when mkdir was a setuid program?)

Re:Yes.

[Anonymous Coward]

Umm... That isn't reiser4.

Reasons to change to another FS

[N_gaAdy]

I agree that we need a revolution in how filesystems work inside an operating system, but it seems that the arguments placed in this paper had alot of holes.

For one thing, the need for changing a filesystem should not really be solely concerned on space or metadata. I think security, speed of data retrieval, and self correcting error engines should be centered on the new systems.

The reason for the speed of data retrieval as being more important than data size is because hardrives are getting much bigger than they are faster. In five years, we may have 20 terabyte drives, but the access speeds will still be horrible.

Security and error correction are obvious points that should be implemented on a systemwide level. When these features are system wide, then management becomes much easier for all system users.

Data, even metadata, belongs in files, not fs

[Fastball]

I agree that metadata in the filesystem is a risky proposition. Just on general principle, I prefer my data inside the file and not left with the filesystem. The MP3 metadata example, to me, is like Windows file extensions on HGH. I remember John Dvorak wrote a piece about Windows file extensions a long while back, and he argued that file types, etc. should be inside the file. A header of sorts. I tended to agree then, and I see filesystem metadata as a bad trend.

Re:Data, even metadata, belongs in files, not fs

[Anonymous Coward]

But there are major advantages to putting metadata in the filesystem...You can do searches across all applications...eg., you can look for all files with "author=Fastball", and find them all whether they're mp3s, jpegs, or emails. You can do this even if the file types don't have any provision for the metadata you want. You can make up any kind of metadata, apply it to any kind of file, and search all your files based on the metadata.

With the appropriate software support (in the filesystem api and browser), this seems like a major advance in usability to me. Certainly people liked the BeOS filesystem. Since Microsoft is claiming to have something like this in the works, it's probably necessary on the Linux side just to keep up...but my guess that ReiserFS will turn out to be cooler than Longhorn.

Re:Data, even metadata, belongs in files, not fs

[hankaholic]

While there are problems to be solved (backup, for instance), there are many benefits as well.

Many detractors of the UNIX security model point to the lack of ACLs and fine-grained security.

The U.S. DoD has contracted Namesys (the ReiserFS guys, led by Hans Reiser) to develop a filesystem upon which security can be build. Reiser's vision is a filesystem which allows users of the filesystem to define what security means in an extensible manner, with plugins.

The future of ReiserFS includes a plugin architecture which makes it easy to implement NT permissions, for instance, without breaking existing programs or requiring new semantics which wouldn't work across, say, NFS.

I'm not sure why the author chose to throw Plan 9 into the mix. Plan 9 has some interesting features, but I don't feel that either Plan 9 or ReiserFS was given sufficient attention to allow the reader to understand just _why_ such things are interesting.

I tried to sum up some interesting Plan 9isms here, but I'd rather not go into it -- reexporting modified views of the filesystems is a complicated thing, and it's hard to justify such complexity in limited space.

As far as ReiserFS goes, the current and future benefits are many.

First, speed is king, and ReiserFS takes the crown. Anyone who has waited more than 10 seconds for "ls /usr/share/doc" on ext2 can attest to the fact that ext2 craps out on large directories. Reiser V3 does quite well, and V4 is rumored to do even better.

Second, space efficiency is nice, and ReiserFS does it better than anything out there. While storage sizes are increasing dramatically, it "feels" wrong to waste a whole block on a trivial file (only a few bytes long, for instance). With ReiserFS, developers don't need to waste time trying to work around the need for efficient small file access -- it's efficient to have many small files, and it's efficient to have many of them in one directory if needed.

I believe it's been said that in the future, ReiserFS may compress the filenames in some way to try to eliminate even that overhead.

ReiserFS V4 implements wandering journals, and support for transactions. What does that mean to the layperson? It means that in the event of a crash, an application which handles important data (for instance, an online purchasing system) can promise that in the event of a crash, a group of related filesystem operations can be guaranteed either to have all completed, or have all failed.

For instance, a purchasing system makes a note to charge the person's credit card, and to ship the items that were ordered. You can tell ReiserFS programmatically to "create a file in /var/orders/ and place this text in it, as well as a file in /var/ccbilling with the customer's credit info in it, and if the system crashes halfway through, throw it all away, because I don't want to charge for products I'm not shipping and I don't want to ship products for which I'm not billing."

If the system crashes, you don't have to work to make sure that you have a one-to-one mapping of orders to sets of billing information -- ReiserFS can guarantee you that either all of it got recorded, or none of it got recorded.

So now you've got a fast, reliable filesystem. I'll even let you ignore the fact that ReiserFS will let you implement per-file compression and encryption plugins. Still not impressed? Wondering what all of this crap about namespaces is about?

The author of this article basically ripped off Hans Reiser's examples from his V4 draft document, such as the /etc/passwd thing. I'm not sure where he got the idea of subusers and I don't much care, as it's not really relevant to the namespace thing.

What is interesting, but not really mentioned, is that file filters can become part of the filesystem.

The specific implementation details in this example are products of my imagination,

Re:Data, even metadata, belongs in files, not fs

[Eraser_]

The one thing I see with plugins like the ms-word plugin is, while cool, seems like it would load the kernel down doing things best left to userspace. How seperate is that ms-word(or anything, acrobat, html, whatever) from the rest of kernel land, where things can go VeryWrong if something fails?

If a buffer over or underflows in the FS plugin is the kernel going to panic, or simply segfault that FS module, core it, and move on? I'm just worried about my system going down because of a mostly-tested plugin.

A cool kernel land example I can see for this is like cat /dev/dc0/ip or /dev/eth0/subnet (whatever your ethernet iface is) because these could be dynamically updated as life goes on for the ethernet interface. (Your server has been up long enough to suffer a complete network overhaul right? :)

Re:Data, even metadata, belongs in files, not fs

[Arandir]

Many detractors of the UNIX security model point to the lack of ACLs and fine-grained security.

You don't throw away the door just because it doesn't have a lock. You simply put a lock on it!

FreeBSD now has ACLs, and it did it without throwing away UFS. It didn't need to replace the "everything is a file" model to do it. It just expanded the available extended attributes. To get ACLs you will certainly need to extend the filesystem, but you don't need to replace it completely.

p.s. FreeBSD also got soft updates without throwing away UFS. In short, you don't need to throw away the old to get the new, something a lot of developers don't seem to understand.

Re:Data, even metadata, belongs in files, not fs

[Webmonger]

As the poster noted, it's possible to run your filters in user space, not kernel space, so buffer overflows and segfaults should be sources of irritation, not sources of vulnerabilities.

Re:Data, even metadata, belongs in files, not fs

[ckaminski]

Yes, but this isn't throwing anything away. It's just using the "files-as-directories" paradigm to add new stuff.

Hence the whole point of implementing this. It will work with everything that uses the OLD style of doing things, while giving us room to grow and expand and enhance.

More important, it extends the "everything-as-a-file" paradigm to meta-data, something that we have never had before. What with procfd and devfs and the like, this is only a good thing.

Ideally, this would be implemented above the filesystems themselves, in the VFS layer, such that *ALL* filesystems could take advantage of the features, and the feature could be controlled via a mount option. Unfortunately, the performance penalty might be too high since the meta-data is so small, and will need to be platform/filesystem agnostic.

I can conceive of a few ways to make it this work, although I'm pretty sure most (if not all) of them will not be high-performance.

Sorry, but

[Morth]

This article seems to just be the author brainstorming or feeling excited about reiserfs. It's hardly a "summary of developments in the filesystem". Now if he was asking about opinions on his article it'd be fine, but he's not, so I'll just discard this as another non-news.

Negativity

[listen]

I'm surprised at the negativity of some of the comments here, moaning that POSIX semantics are perfect and nothing else can possibly be countenanced...

Plan9 namespaces and Reiser4 really do bring a lot more to the table in terms of useful expanded semantics and utility than all the posix filesystems. Posix extended attributes are very limited, and some filesystem implementors seem to be keen to implement them in the most restricted way possible ( eg size limitations in ext3).

The annoying this with Reiserfs is that the VFS will lag it by a few versions, and very very few apps will make any use of its special system call. Sigh. We'll be stuck with databases in a file for a long while yet.

One thing I would like to know about reiserfs is how attributes are attached to directories? If they are just small files in the "directory" bit of a file, what distinguishes them from children of the directory? Or are attributes just banned from dirs? Seems limiting.

On the .. syntax

[IamTheRealMike]

Oh, I should probably mention - if you read the whitepapers available on ReiserFS the "foo/..attr" syntax is just a toy, made up on the spot, syntax to demonstrate the idea of files within files.

Nobody (apart from perhaps this guy) has ever claimed that this syntax will actually ever be used, or needed. There are other possible syntaxes available, and in fact one long term blue sky plan for RFS is to allow many different types of syntax within the same file path, including for instance things that vaguely resemble database queries.

So, don't get hung up on the syntax given in this article.

Re:On the .. syntax

[hansreiser]

No, it is not a toy, it is a style convention.

In regards to attributes on attributes, some (regular unix plugin ) files have relationships to other files that are implemented using stat data plugins, but the stat data files don't have stat data themselves. There are lots of specific plugin implementation details, but it works without infinite recursion problems

plan9 is hardly a "new advance"

[sholden]

We had plan9 machines here 10 years ago...

I don't think any exist anymore, in fact I don't even think the inferno install works anymore.

But anyway, it isn't a "new advance" anymore.

Re:ReiserFS rules.

[lightcycle]

I may be way off base, but I see the need to have a existing non-journalling file system. Someone stop me if I'm wrong, but in my mind things like audio recording, video capturing and the likes would suffer performancewise from being run on journalled file systems

No need for LDAP?

[jackalope]

It seems that the author presumed that the only use of LDAP is to provide passwords for user authentication. While that is a common use of LDAP it is not the only use.

It would seem that having a file system that is LDAP aware could be extremely useful. Imagine if your LDAP tree were reflected as a tree in your file system. You wouldn't need to embed LDAP calls in your application, it would just be data in your file system. So looking up an attribute for the current user, or a user, would be as simple as reading a file that holds the value of the attribute.

Re:No need for LDAP?

[jackalope]

One wouldn't need to copy the LDAP data into a file system. Instead provide a view into the LDAP directory via filesystem calls. In otherwords, the filesystem driver would translate calls to it into LDAP calls to the LDAP repository, whereever it may be. The only persistent data that would need to be stored for the file system would be the how to connect to the LDAP server farm.

In regards to your second question. The standard getxxbyxx() calls are useful for returning username/uid/gid etc. But if the application wants to know something like 'What is the mailstop' of this user? There is no standard call, one must revert to LDAP API calls. So, your statement is correct in regards to system level information, but those services are inadequate for almost all application level user metadata queries.

What I REALLY want

[afidel]

Is for someone to come up with a real unlimited snapshotting filesystem for linux. I don't want to use user mode hacks (as nice as they are rsync style snapshotting isn't reliable enough), or snapshotting that only allows a shadow copy of the entire volume, I want to be able to tell the users that they can just go into ~/.snapshot/time (where time can be hours, days, or weeks in the past) and copy the file they messed up back into their home directory. Basically I want the most usefull feature of netapps without the HUGE markup =) The cost in admin time both in user interaction and reduced need to do tape retrieval and file restores is immense.

Re:What I REALLY want

[IamTheRealMike]

That kind of thing would benefit from this, especially if integrated into the GUI in a nice way.

In effect, it'd be similar to what Linux does with memory - empty disk space is useless! What is the point of that? Might as well use it for something. Storing old versions of files is an ideal use for it, and combined with an LVM means that the more space you have, the further back in time you can go. That's far more flexible and useful than storing stuff in a recycle bin.

Of course, you run into problems straight away if you aren't using lots of small files. Are you really going to store a new copy of that 5meg presentation every time you hit save? Bad idea. You only want to store what has changed.

But - how? Doing it with text is easy enough, we have diff and patch for that. But what about more complex formats?

Ahhh, now we understand why minimizing primitives is useful in the real world. If the internal structure of a file is split into many small files, each representing a facet of the presentation (each slide, each bullet point on the slide, font weight of that bullet point etc) then suddenly it becomes much simpler to write a generic differential analyzer, we can just snapshot the mini files that changed.

As a result, we increase efficiency to a level at which it becomes feasable to keep the undo transaction buffer on disk - just imagine how much hassle we'd save if all apps used this reliably?

There are still plenty of details missing of course - some things still wouldn't work well, in particular large BLOBs with no structure, like say images, or audio files. I guess if you were smart you could leverage the apps transaction buffer, but still..... that would lead to interop problems.

Still, there are many unexplored possibilities that appear when you increase the abilities and even efficiency of low level parts of the OS like this.

It sounded so intelligent until he started playing

[The Monster]

sure, im going to listen to this guy about filesystem implementation when he cant even set up MIME types on his webserver.

or spell: (emphasis mine)

If you just want to give them some default settings,have that line below their home directory's line. Viola! Lock down framework.

I know we hackers like to, uh, fiddle with our computers...

Swiss Cheese

[Anonymous Coward]

Cool ideas, but let's be careful lest we make a steaming pile of Swiss cheese.

Evidently you haven't used ReiserFS. It already does this.

ReiserFS only journals filesystem metadata. Because it uses a B+ tree balanced allocation scheme for file blocks, when the system crashes the last pair of blocks written will often be swapped with respect to their files. For example (this has happened to me and separately to a friend) if you modify out /etc/passwd just before the system crashes, you'll be unable to log in again when the system comes up.

What happened? syslogd wrote a panic message to /var/log/messages as the system went down, and you'll find this message (as well as the rest of its 4k block) in /etc/passwd, while your changed password file may be found at the end of /var/log/messages. This is a feature of ReiserFS, not a bug.

I still miss the raw speed of ReiserFS, to be sure, but EXT3 has kept every last one of my hundred-odd filesystems rock solid for two years now, which is really what you want a journalled FS to do.

ObOnTopicComment: Miner's examples are clumsy and ill-considered. BBN's Dave Mankins put a relational database into the 4.1BSD filesystem back in 1984, and Plan 9 took a more rational approach with its namespace algebra. This is not a new idea, so there's no absolutely no excuse for breathless exposition based merely on coolness factor.

At best, Miner's descriptions obscure any true value Reiser's proposal might have. Organize my MP3 collection with FS metadata and lose it all when I try to move to another FS? What is he thinking? Is he thinking at all?

Sheesh.

Re:Swiss Cheese

[johannesg]

Organize my MP3 collection with FS metadata and lose it all when I try to move to another FS? What is he thinking? Is he thinking at all?

AmigaOS had a single meta-data field in its various filesystems, called the "file comment" or "file note". While it was clearly far less powerful than what is proposed here, it was incredibly useful for lots of purposes.

The problem you describe, losing the meta-data when moving to another filesystem, was non-existant on the Amiga for two reasons: all filesystems implemented filenotes, and LhA (the Amiga-equivalent of tar/gzip) saved and restored filenotes. A similar solution can be used for Linux (tar and gzip are getting very long in the tooth anyway).

I would love to see proper meta-data fields implemented throughout the system, especially if it is pervasive throughout the OS (so I can search for specific values from the desktop, for example).

Finally, I believe BeOS had something similar, and users of that filesystem also tend to be enthousiastic about it. All this doesn't mean the current solution is necessarily the best one, of course - just that the concept is pretty damn useful.

Re:Swiss Cheese

[sql*kitten]

syslogd wrote a panic message to /var/log/messages as the system went down, and you'll find this message (as well as the rest of its 4k block) in /etc/passwd, while your changed password file may be found at the end of /var/log/messages. This is a feature of ReiserFS, not a bug.

Care to explain that? How is corrupting one of the files essential to the normal operation of your system not a bug? The whole point of a modern filesystem is to recover cleanly; even fsck can run unattended (mostly) but what you're saying here is that after an unplanned reboot, you might or might not need a sysadmin to manually intervene.

Seems good for some applications,...

[ddubois]

I feel this very interessting.
For example I like the currents devfs and procfs (although not perfect). Those help me a lot to "debug" new hardware connection. SImple and coherent.

I also imagine some RDB support. Imagine a "select * from account" where account would be asimple directory. Would be nice to use "find" for the where clause :-)

Imagine also implementting OO classes wiht inheritenace using symlinks... And more...

Yes would be very nice!
D.

The logical conclusion of the Tao of Unix

[The Monster]

Reiser's vision of files that are also directories, and putting metadata into files, is just the logical perfection of the fundamental *nix notion that 'everything is a file'. It is this which has given the OS such strength for three decades - before anyone knew about 'object-oriented programming', Unix was behaving in an o-o manner at the macro level, with individual programs doing encapsulation, etc. Having plugins that expose the details as ordinary files, allowing you to access information with your editor of choice, as well as scripting tools such as sed, grep, awk, perl, or Python is what Unix is all about.

The only thing I'm concerned about is backward compatibility - if someone accidentally tries to open a file with a trailing slash, and gets an error because now it's a directory, then it's a Bad Thing.

I just want...

[Anonymous Coward]

A filesystem that goes wrong properly...

Imagine your filesystem is a library...

Imagine you drop a bomb on it...

Books and pages scattered all over the place... Yet you can still work out which page belongs to which book, and where on the shelves they used to sit..

Having been caught short by LVM and reiser before (which just couldn't deal with a 46GB gap in the filsystem where a disk used to be) it seems to me that no-one's made a filesystem that breaks properly...

For me.. speed is not an issue.. nor is CPU usage... I'm quite happy to throw a dedicated box at handling the filesystem... I just want something that is written with the thought in mind... "Our hardware is unreliable.. It will die.. it wll loose bytes... How do we deal with it"... not the usual thing that people do of "Our disk is 100% reliable... how can we most efficiently organise it... Hrm.. now what do we do when we encounter problem X" There should be no fsck... That should be handled by 'mount'

Sure.. there's always RAID... but I reckon I've got about 28MB of data in total that's critically important... the rest is just crap I downloaded from the net and can always get again...

Continuing my anology further...
If your filesystem's a library, then then your physical disks are wings of the library.. (east wing.. west wing.. etc)...
I want a librarian who's job is to work out the best way to organise the data in the different parts of the library... When you're looking for a book, it's often easiest just to ask the librarian.. who magically holds much of that information in his/her head.
The librarian is also responsible for making sure that multiple copies are maintained for popular or important works...

It'd also let me do the thing I've always wanted...

chattr +mirror filename

(i.e. This file is important and must be mirrored on 2 or more disks)

Just my two pennies worth... I'll write it one day if someone else doesn't get there first.

Re:good read, but less relevant

[sultanoslack]

This is said by someone who obviously hasn't done any real world application profiling. It's quite the opposite -- CPU is relatively rarely a limiting factor in desktop applications, dealing with the HDD very often is.

This is very often why adding more memory to a system makes it seem more responsive -- larger disk buffers, less need for disk based virtual memory.

Basically hard disks are very often *the* limitation; CPUs are fast.

Started out good...

[Korgan]

The more I read this, the more it reminded me of the marketing version of how Apple would like us to think of Resource Forks.

Truthfully, there isn't exactly a lot of difference in the concept or the idea. Implementation is vastly different but the idea remains very similar.

Why do I want to accept this sort of idea anymore than I want to accept resource forks? If I copy a file with resource forks from one of my macs to nearly any other OS on the market thats not specifically configured to support them, I lose that information. Why do I want to continue this?

I use HFS+ because I have to. To get all the functionality I want out of my macs, its the only real option I have. But for anything other than system level files that are never likely to be copied to another machine, this is just a waste of time to me.

Next question. Say I do run this file system on my machines. I build up a heap of data and I'm using "files as directories" to store metadata about those files. How do I back it up? Don't even try to tell me "rebuild tar". Haven't we put tar through enough to try and extend its capabilities? I wouldn't touch a file system with these capabilities without a guaranteed way of being able to backup ALL the data. Otherwise its just truly not worth the effort.

Re:Anyone remember Acorn RISC OS 3?

[BinaryCodedDecimal]

Good old Acorn RISC OS already supported the use of directories as files back in the eighties. E.g.: Click on a file to open it, shift click to show a directory of sub-files (recurse at will).

You're thinking of application directories. It could not be done with ordinary files - They had to be a directory.

The only difference was that if a directory had an exclamation mark as its first character, RISC OS's default action would be to execute a file called !Run inside that directory.

Fantastic idea - It meant that you could keep all the libraries and files required for an application in one folder. It also meant that you could move the program and its associated files wherever you liked on the hard disk - There was only ever one icon to move.

If you ever decided to move stuff around like that in Windows, prepare for your programs to stop working.

Some of these ideas are VERY short sighted

[irw]

I wish people with clever ideas to redesign POSIX namespaces would spend ten years in system administration first so they realise what's involved with managing REAL WORKING SYSTEMS.

Some of the ideas might well lead in useful directions, but some (at least as described in the paper) are plain silly. viz:

1) with overlayed mounts:

suppose my home dir is mounted read-write over a read-only system root, and I do not have a "/bin/prog" in my home dir. Consider:

cp /bin/prog /bin/prog

First time, it copies the system /bin/prog into my home fs - Counter-intuitive to the path semantics. If I run this a second time it copies my copy of /bin/prog over itself - Inconsistent.

2) Attributes in the namespace

We have a rather carefully written setuid chown/chgrp/chmod replacement which can be run by users in an "admin" group, and allows devolution of 1st-line support tasks to nominated users. It won't touch files whose uid/gid is 100, so they can only touch non-system files.

If attributes (file uid) is file/..uid and cp is supposed to handle what chown does, the above breaks big-time. We now need a custom cp replacement. Either that or we have to add an ACL for the admin group to every file we want them to manage, which is a great deal of effort, and likely end up inconsistent.

Contrary to the paper, setuid and PARTICULARLY setgid is NOT going to go away in the real world any time soon, as far as files are concerned. Ports less than 1024 are a different matter and I agree with the document.

3) Consider the number of file descriptors involved if /etc/passwd becomes a hierarchy of files. Just logging in one user will involve multiple open()-read()-close() operations. Whilst these might be efficiently implementable at fs-level, it is still very inefficient in user space, or will at least require a dramatic rethink of unix tools.

What's in a name?

[Zog The Undeniable]

I assume Plan9 is an ironic nod to the "worst film ever". When I develop my new filing system, which will only allow numeric characters in filenames, will delete the MFT every time the computer is rebooted, and will require a new directory for each file added to the system - that FAT16 limit of 512 was FAR too generous - I'm going to call it BattlefieldEarthFS.

9p2000 is an open protocol

[DrSkwid]

already one such implementation exists such that FreeBSD can expose its file system to plan9 machines (as you would expect it gets imported into your namespace. Would can be a different place depending on the namespace of the current process. Even (temporarily) "replacing" your local files with versions on the FreeBSD Box, if that's what you want.

Never going to happen

[MarcQuadra]

I used to live like that, then I took my big hard drive, slapped it into a linux box and shared it out with NFS, Samba, and NetAtalk. Now I can access all my files, which automagically get backed-up, from any machine on my LAN. Stop waiting for the 'universal' FS to show up, it'll never happen.

Escape!

[zerocircle]

Start weeding out trailing / in filenames now.

I'm delighted with the prospect of metadata-as-files and files-as-directories (ergo, metadata-as-directories?) -- but here we have another problem to address: Insufficiently escaped data. Human-readable data fields (including filename, if the user can read/write it) should be able to contain any human-readable characters. Filenames should be free to contain normal punctuation; path separators -- again, if the user can read/write paths -- should be selected from outside the normal punctuation set, or else the stuff between the separators should be escaped. Or the user-accessible file and path names should be stored as metadata.

Can't tell you how much frustration I've endured over other people's improperly escaped data. This just looks like one more case.

(Mac OS <=9 used a colon as a path separator, making it the only forbidden character for file/folder names, which could have been avoided so easily: How about a pipe, guys? or (shiver) a backslash? Or, even better, some control character unique to the Mac, akin to the Option-Shift-K Apple logo? Programmers. Sheesh.)

hash-maps instead?

[Bram Stolk]

Maybe we should replace filesystems with
presistent hashmaps that have O(1) lookup?
This way, you can add any attribute to any
object you like. See Python's hashmaps.
So for /etc/passwd access, something like:

passwdhash = disk0["passwords"]
roothash = passwdhash["root"]
rootshell = roothash["shell"]
rootgid = roothash["gid"]

For chown of /bin:
disk0["/bin"]["perms"] = 0755

To check for available attribs, do:
print disk0["/bin/"].keys()

Bram

Maybe I'm just being picky

[Trix]

He lost me as soon as he held up GConf as an example of what was to be accomplished. Have you ever LOOKED at the "xml" files that GConf generates? Ever tried to climb the ~/.gconf (and /etc/gconf) trees? I put GConf (and anything that aspires to be like it) in the same category with the Windows Registry. GConf is, by far, the thing I like least about GNOME (and, on the whole, I like GNOME).

Why do people keep adding needless complexity to fix systems that aren't even broken? If I can't edit my configs with vi, I'd rather use something else.

Metadata in files?

[steveha]

Just on general principle, I prefer my data inside the file and not left with the filesystem. The MP3 metadata example, to me, is like Windows file extensions on HGH.

I'm with you -- I like self-contained file formats.

But I don't think he was proposing that you not use Ogg tags or MP3 tags; he was talking about the filesystem abstracting the tags. If you changed "Stagnation.ogg/album" to the string "Trespass", then the filesystem abstraction layer should update the Ogg "album" tag inside the file to be "Trespass".

The key benefit here is that you would not need some wacky command-line utility program to let you view and change tags on Ogg files. You could just use the shell. In bash:

for ii in *.ogg; do echo "Trespass" > $ii/album; done

Note that this same one-liner would work if you were in a directory with MP3 files, and you changed "*.ogg" to "*.mp3". Currently, you need to run vorbiscomment for your Ogg files, and mp3info for MP3 files. (I just checked, and sure enough, they take different arguments to do the same operations.)

Personally, I'd like to see a standard metadata portable XML format for legacy systems. People talk about copying a file from a rich metadata filesystem and having new files like .attributes created on the target legacy system; I'd be happier if just one big XML file could be created with the same name as the original file.

Suppose you backup server //rich onto server //legacy, and then you want to restore some files from //legacy to //rich. If all the metadata was stored in a big XML file, then when you copy the file from //legacy to //rich you restore all the metadata; you wouldn't accidentally slice off attributes by forgetting to copy one or more rich attributes files.

You could do most of the fancy tricks of the rich metadata filesystem on a legacy filesystem that used the big XML file to store the rich metadata. And as long as the legacy system is just smart enough to look at the main data part of the XML and leave the metadata tags alone, you could still modify the file with sed, awk, perl or whatever, and then copy the big XML file onto your rich metadata filesystem and still not lose any rich metadata.

Note also that the big XML file could be used to deal with existing rich metadata systems, like resource forks from Macintosh filesystems, or multiple data streams from NTFS files.

steveha