State Of The Filesystem

Skeme writes "Have you heard of Plan 9 or Reiser4 but don't know much about them? Are you curious about the improvements free software is making to its filesystems in general? Read my summary of the current developments in the filesystem: namely, what improvements we can expect (a lot), and what Linux and the BSDs can do to improve on the filesystem."

Comments

Transferring Files

[jhunsake]

I've always wondered how these filesystems with metadata handle transferring files between different systems. It would suck to have all your MP3 info in filesystem metadata and then lose it all when you transferred to a system without fs metadata. Anyone have any insight?

Re:Transferring Files

[ThogScully]

I would think that would be prevented by the filesystem implementation. A filesystem's features are only as good as the implementation. So if you have fs metadata and are transferring it to another filesystem, whatever tool you are using to move the files, whether it be at a BASH prompt or drag and drop in Konqueror, should figure some way of properly translating data from one system to another without losing anything. Moreover, the tool should be able to do it without knowing any particulars about the actual types of filesystems in use. It's all in the implentation.
-N

Re:Transferring Files

[sleeper0]

i am skeptical as well, the only reasonable answer seems to be that it would get lost. Especially if you are transfering between platforms. Why would you want to use a metadata system that you can't count on anyone else using? It seems to me that current packaged in file metadata systems are much more practical for transfer, multi-platform use, streaming, etc.

Other things mentioned in the article like the /etc/passwd manipulation just seem like gimmicks. How many files like that would you by hand support before you ended up deciding it was a bad idea to start it in the first place.

If these are the things they are coming up with to add to filesystems now, they must be pretty much figured out & done.

Re:Transferring Files

[jhunsake]

What about when the other filesystem has no mechanism for storing metadata? How about transferring files over ssh? You gonna rewrite ssh?

Re:Transferring Files

[warrax_666]

Anyone have any insight?

Are you mad? This is slashdot.

Re:Transferring Files

[Anonymous Coward]

No, of course not. You'd use ssh to transfer an intermediate format (like tar) which DOES retain metadata.

Re:Transferring Files

[Anonymous Coward]

It wouldnt be hard to extend the scp command to send metadata across if there is a compatable (ie. up to date) ssh server running that supports accepting metadata descriptions for files. The remote machine can then do as it pleases with the metadata.

What is needed is a POSIX specification for handling file metadata. Then applications, servers *and* filesystems can be gradually upgraded to include the metadata features in a portsable and consistent way.

--
Blue SSL

Re:Transferring Files

[Adnans]

You gonna rewrite ssh?

No, just modify it so it understands these extra attributes (both server and client). That shouldn't be too hard.

-adnans

Re:Transferring Files

[groomed]

Generally you lose the data, unless you wrap it in another format to encapsulate all the information. This is what Macheads did on Classic MacOS: they .hqx'd or .bin'd their files before transferring them to another system. It's not ideal. The alternative, flat streams-of-bytes, is not ideal either (and not true: even in Unix, files have some metadata that doesn't translate very well).

Hopefully in the future our filesystems and transfer protocols will evolve to have some reasonably broad common ground where metadata is concerned (a development similar to the diminishing need to accomodate DOS 8+3 filenames).

Re:Transferring Files

[axxackall]

Metadata is very application specific and most of filesystem are agnostic about it. Typically it must be handled by another layer on a top of FS.

Often that layer is a DB - database. I suggest you to try ZODB, database in Zope, it's very good to handle files as documents - with many unified metadata about files.

Another good example to study is Subversion, which is revisionining/versioning metadata-management layer on a top of a regular FS.

You may research and find some software implementing a layer (on a top of a regular FS) specially designed to handle MP3 playlists. But again, that would be a layer on a top of FS, not a filesystem by itself.

Re:Transferring Files

[kasperd]

(like tar) which DOES retain metadata.

Sure tar does retain some metadata, but only one well defined set of metadata. If a filesystem introduces new types of metadata, they don't magically get supported by tar, neither do they get supported by all other utilities.

Re:Transferring Files

[peatbakke]

There are a few of ways to solve the problem, but only one of them ensures compatibility across many different platforms:

The first is the worst: having a separate metadata file accompany each of your files as they move between file systems. This is a logistical nightmare, and requires soooo many tools to be rebuilt that it's not even worth considering.

The second is the "bundle" concept, as popularly demonstrated in OS X. "iTunes.app" is actually a directory, not a big binary file. The directory is split-up into different sub directories, containing multiple binary formats, languages, etc. To transfer iTunes to another computer, just tar/zip the "iTunes.app" directory, and viola. Easy. Of course, the app will only run on platforms that recognize the bundle format, and can execute the binary. This rules out backwards compatability, and requires that all major system vendors agree on the same bundle format -- and that's not gonna happen.

The third way to do it is to keep metadata within a file (like MP3 tags), and then export them into the filesystem space. For example, an MP3 plugin would pull the data out of "my.mp3" and turn it into "my.mp3/title" "my.mp3/genre" etc. The nice thing about this is that it's platform independent, and backwards compatable.

I'd be keen to see option #3 implimented in a modern file system -- do any current FS's have an API that could support it?

Re:Transferring Files

[osgeek]

If all of the metadata is stored in directories, then copying the directories and all of their contained files would naturally include the metadata.

Mac OSX uses the "file as a directory" strategy, and the BeOS used the dynamic file metadata approach. Combining the two is the way to go and buys you a tremendous amount of new functionality over traditional file systems.

Re:Transferring Files

[Flammon]

Storing metadata in each file in a format designed by its developer certainly is not my idea of "more practical for transfer and multi-platform use". What it the format of the file doesnt' support metadata?

Separating the metadata from the file and keeping this metadata related to the file lets any file have metadata, makes the metadata extensible and easily accessible.

If you need to transfer your files and the metadata, simply tar it like OpenOffice.org does. to solve the cross platform problem.

A plugin could also be used to put the filesystem metadata into a file that supports metadata when using the cp command.

For example,

cp /home/joe/music/artist/somesong /home/joe/somesong.mp3

would create an mp3 with metadata.

Re:Transferring Files

[ebh]

You have to answer this question every time you improve or add to the metadata. One recent instance of this was the addition of ACLs. An ancient instance of this was the addition of long (>14 character) filenames.

The file-as-directory scheme goes a long way toward generalizing this, so that once the common tools like tar can handle it, new metadata introduction will likely not be such a transportability nightmare.

BTW, ClearCase has had files-as-directories for years, just with a slightly different notation that maintains transportability at the expense of namespace pollution. Call something file and you get the version you're supposed to get. Call it file@@/ and you have transparent access to the file's entire version tree, as well as all the tags applied to it.

Re:Transferring Files

[putaro]

That's been a problem the Macintosh has had for years. Mac files have two forks, one for data and the other for resources (icons, strings, executable code) and a few other pieces of information, most notably the type and creator (application not user) which allows the Finder to launch the right app for a file without skank like extensions.

Anyhow, Mac files get royally screwed up when move to any filesystem that doesn't support all those things (most) so there's a special encoding format that zips them all together called AppleSingle. AppleSingle has all of the meta data and forks in one flat file which can be moved just about anywhere and not lose data. Of course, nothing besides a Mac knows how to look inside of it.

Now that Apple has been taken over by Next :-) and moved to a Unix base a lot of good things have happened, but extensions are now preferred over type and creator and the multi-fork file approach has been superceded by a directory approach with the directoriness being hidden by the new Finder (go to the Terminal on a Mac OS X box and cd into an app).

So, ideally, anyone developing an OS with spiffy meta data stuff would also design a flat file encoding along with a compatibility library that would let apps work with the extra metadata on any OS with, perhaps, degraded performance.

Re:Transferring Files

[falconed]

If all of the metadata is stored in directories, then copying the directories and all of their contained files would naturally include the metadata.

Right, but now the filesystem and mp3 tag data are tightly coupled. In case where you're copying between different filesystem types, the destination machine has to have a filesystem that understands the source filesystem's format. If the destination filesystem doesn't know the tag data is stored elsewhere, the tag data is useless. IMHO, that makes mp3's a lot less portable and portability was basically the whole point, wasn't it?

Re:Transferring Files

[Surak]

Well, remember Reiser4, for instance, is a TOTALLY DIFFERENT system than the standard POSIX filesystems (Reiser3, Ext2/3, UFS, etc.)

The idea is that data is stored as in a database, and small files can be handled efficiently.

Think of it this way: What *is* a filesystem? Really, it's a relational database. We don't tend to think of it as one, because it isn't implemented as one, but that's what the data it represents is. If you think of each file and directory and associated metadata as a record, with directories being pointer records to other records that show relationships.

Anyways, code needs to be implemented to handle this.

We use metadata NOW. A files permissions, owner/group info, date and time stamp, and even the filesize are metadata. We transfer those between different systems right now. With the correct transport method, this metadata doesn't get lost at all. A tarball can sit on any system no problem. Inside the tarball is all the POSIX metadata. That gets transferred from system to system.

Of course, some OSes (read: Windows) don't handle Unix-style permissions correctly, so these are translated into rough-equivalent ACLs with systems such as Samba.

Re:Transferring Files

[n3rd]

I've always wondered how these filesystems with metadata handle transferring files between different systems

Usually it doesn't. Take for example VxFS (Veritas File System); they support attributes like preallocating space for a file or forcing a file to be contiguous. When you move a file that uses these attributes to another type of file system (VxFS to UFS) you lose them since the target file system doesn't support those attributes.

Another example is ACLs. If they are in some way propriatary they won't transfer and sometimes won't work well together. The situation I ran into was a HP-UX NFS server's ACLs not working correctly with a Solaris client. The server used and enforced the ACLs however the client couldn't view or modify the ACLs on the files that were stored on the server.

If the implementation is truly at the file system level there's nothing you can do. However, as stated below, if there is a layer above the file system that handles metadata then you can more than likely keep the metadata intact.

Re:Transferring Files

[LordBodak]

I agree. I sure wouldn't want to have to try and decipher this kind of thing when ssh'ed into a box that isn't working right. With GUI tools it wouldn't really matter how the data was stored, but one of the best things about Unix is that while you can use a GUI tool for administration, the command line is always there as a fallback and it's usually not too hard to find what you need that way.

Re:Transferring Files

[gilesjuk]

That was a right pain in the butt, having resource and data forks etc..

At least with AmigaOS they stored similar things in another file (.info file). Much easier to transfer, but you do run the risk of losing the .info file.

Re:Transferring Files

[ThogScully]

If the other filesystem has no way to store metadata, then perhaps someone will write a sort of add-on for that filesystem that attaches an XML file to something.mp3 that is called something.mp3.meta that will contain the metadata and because that person is running the "patched" filesystem, nothing will be lost.

As for SSH, that's just one of the tools I spoke of - the tools will have to implement the features of the filesystems - it's not up to the filesystems. If a filesystem addon like I specified above becomes common enough, then perhaps it too will become supported in various tools.

Ultimately, the point remains that the tools will support newer filesystem features and do their best to operate as translators from one filesystem to another. If one filesystem really just can't support the data on another, then something will have to be lost, but if that's the case, then perhaps it's time to upgrade the older filesystem.
-N

Re:Transferring Files

[Anonymous Coward]

I dont know how metadata is gonna be handled, but I'm sure it will be handled a lot easier.

I think hierarchical storage structures have great advantages over other ones, since you can automate the searching process.

Like Regexps give you power over Strings of Data, YACC, XML and Reiser4 give you power over Stacks of Data (Chomsky Type 2!). With ext2 and the like we had Stacks of Files that had to be treated by the user as a String of Data. Since you can now access the Data much easier, it'll be an easy task to recombine them to a tagged file.

Lets think about a Reiser4 File like this:

song.ogg+
song.ogg+/artist
song.ogg+/album
song.ogg+/year
song.ogg+/oggstream

I bet it will be a very easy thing to now come up with a thing (heck it could be in PERL) that recombines these Data to a normal .ogg file. Or the other way round. Just now your metadata can be unified. You could use "mp3+" files in a similar way. Or you could come up with metameta data, a file, that saves which metadata fields are named how.

Re:Transferring Files

[dash2]

True, but as he pointed out for chmod, chown etc., most of these command line user tools could still be implemented, sometimes even as shell scripts.

Obviously, no clever new idea will go anywhere (except on e.g. embedded systems) unless it is compatible with the huge base of existing Unix software, command line tools included.

Re:Transferring Files

[Jellybob]

I like the idea of software that exports file level metadata to FS level metadata... I do see a lot of potential in metadata at the FS level... say I want to find all documents (images, word documents, and presentations say) about the topic "Sales, North" on my companies server... at the moment it's not an easy prospect, but with FS level metadata, it can just flick through the metadatabase, and find anything with that keyword.

Re:Transferring Files

[sketerpot]

It's a great idea to have programs really be directories. It's also great to have Interface Builder, so you can drag and drop your UI into place and set up connections between objects. Plus, the localization directories are good, too.

But the thing that's creepy about it is how closely it resembles a system from the early 90s: NeXTSTEP. Imagine what it would have been like if NeXTSTEP became the dominant OS around the time Windows did: we'd all have learned Cocoa years early, but we'd have called it the AppKit. We'd have a very clean GUI. We'd have RTF documentation as an important part of the program builder.

There is a fairly standard bundle format: openstep bundles. Unfortunately, there isn't a standard binary format. Still, sourc bundles should be able to compile on MacOSX or GNUstep.

Re:Transferring Files

[zsau]

i am skeptical as well, the only reasonable answer seems to be that it would get lost.

No. The idea is you have a plugin to the filesystem which accesses the ID3 tags. So that when you change /Music/Russian/Zemfira/Zemfira - Iskala/title from iskala to Iskala, the ID3 tag is being changed. And so on. The idea of plugins is to access file-specific information in generic ways, which I would personally love.

Re:Transferring Files

[Dalroth]

ummm....

The same way we transfer file data from an ext2fs file system to a fat 32 file system? Write some conversion code?

Now you probably won't get the ogg or mp3 conversions without a plugin installed, but you'll certain get all the important basics such as file permissions and ownership attributes.

Re:Transferring Files

[Anonymous Coward]

"But the thing that's creepy about [OS X] is how closely it resembles a system from the early 90s: NeXTSTEP."

Maybe that's because Apple acquired NeXT...

Re:Transferring Files

[sleeper0]

are you sure about that? did you read the article? SO now you are saying that the filesystem will need code to handle every multimedia format differently? Sounds like more of a nightmare

Re:Transferring Files

[zsau]

are you sure about that?

No, but it seems the most logical way of doing it. If I were coding a filesystem, there's no way other than that that I would use.

did you read the article?

Yes.

SO now you are saying that the filesystem will need code to handle every multimedia format differently? Sounds like more of a nightmare

Better it be programmed once for everyone to use than many times for fewer people to use. And it doesn't go into the filesystem, it goes into a plugin for the filesystem, so the filesystem can be totally completely written and Frog can make Snorbus files, and I won't need to have recompile my kernel, I just load a new plugin.

Re:Transferring Files

[paul.dunne]

> If you think of each file and directory and associated metadata as a
> record, with directories being pointer records to other records that
> show relationships.

Sorry to be snippy, but what you're describing is an *hierarchical*
database. But you're right in your main point: an FS like ext2 is an
example of an hierarchical database.

Re:Transferring Files

[thogard]

Posix file systems aren't required to be hierarchical they just of sort of look that way. Remember hard links? The name space is hierarchical but files and directories can appear in many locations. link (no ln) has been able to create hard links for directories since Unix version 2.

Re:Transferring Files

[greenrd]

If a filesystem introduces new types of metadata, they don't magically get supported by tar.

Yes they do, if they're seen by tar as ordinary files. That was one of the main points of the article, which not many people here seem to have read (as per usual).

Re:Transferring Files

[John Harrison]

Huh? They only "look" hierachical? How does having links change this? It simply makes the graph a bit more complicated. There is a big difference between a hierachical DB and a relational DB, and it involves much more than what you have pointed out. For instance, compare IMS and DB2.

Re:Transferring Files

[zatz]

A filesystem is nothing like an relational database. I wish people would stop making this comparison, because it's completely misleading and unhelpful. A filesystem is not a set of user-defined tables, each of which contains an unordered set of rows. Queries and joins are not possible. Constraints and null values are not supported. Files within a directory have an inherent order. Files are variable-length and byte-addressable. Duplicate "rows" are not permitted. The principle relationship modeled is hierarchy... ever heard of a hierarchical database?

Re:Transferring Files

[babbage]

The way to handle this under the old MacOS9 days -- and to an extent today with OSX -- is to get around the problem by avoiding the transfer of raw files.

So for example if you had a Photoshop file to send, you'd use StuffIt to make a compressed .sit file, and even if the compression was nearly nil, you'd at least have your file's data & resource forks properly bundled together for transfer.

The other way, which seems more common now than it was then, is to create a disk image in the filesystem of your choice (nearly always HFS+, but in principle I don't see why it couldn't be UFS, FAT32, NTFS, EXT2, ReiserFS, BeFS, etc) and then copy your files onto the disk image. It then could be compressed, transferred, and mounted on any remote machine with no loss of metadata, provided that an appropriate filesystem was chosen of course.

Really though, a lot of this enhanced filesystem stuff is just for keeping your own house in better order. If the trend is towards more RDBMS styled filesystems -- as seen with BeFS, Microsoft's Yukon project, new versions of ReiserFS, etc -- then one of the new lessons to be learned is that you'll have to change your habits & tools for transferring files from one machine to another. Just as you'd rarely transfer data from one database to another by simply copying over the raw data files when you can more easily generate a portable SQL dump of the relevant tables to use instead, the right way to transfer data from one of these next generation filesystems to another computer (that might or might not be running the same system) is going to have to be some enhanced transport -- like the disk image file or the SQL dump.

To me, these are secondary issues; the benefits of working with a modern filesystem are strong enough to make them worth the effort. I was a happy BeOS user a few years ago, and it still annoys me that none of the mainstream systems (Windows, Mac, or Linux) has really put together a package as nice as BeOS's BFS / Tracker pair. I've got high hopes for Apple to do something better someday, but it hasn't happened so far.

Re:Transferring Files

[cpt kangarooski]

Even filenames are metadata. Anything that's descriptive of the file contents is. It's great stuff -- try reading the full ID3 v2.4 spec sometime; you may be pissed off at how little is actually implemented.

Re:Transferring Files

[jafuser]

With the system that was proposed, you would probably need some kind of file packager system (with a rather extensive and flexible library) to assemble a file and it's meta-data into a standard "network" file type before you ship it off to a computer which does not have this kind of filesystem.

Probably something like:

$> mkfile ~/songs/OompaLoopmas

Would take the fs-organized file and meta-data:
~/songs/OompaLoopmas
~/songs/OompaLoo pmas/title
~/songs/OompaLoopmas/artist
~/songs/O ompaLoopmas/genre
~/songs/OompaLoopmas/year
~/so ngs/OompaLoopmas/album
~/songs/OompaLoopmas/track
~/songs/OompaLoopmas/..uid
~/songs/OompaLoopmas /..gid
~/songs/OompaLoopmas/..rwx
~/songs/OompaL oopmas/..type (contains "audio/mpeg")

and make the following basic "networkable" file, with no meta-data other than standard file-system meta-data:

~/songs/OompaLoopmas.mp3
~/songs/OompaLoopmas.m p3/..uid
~/songs/OompaLoopmas.mp3/..gid
~/songs/ OompaLoopmas.mp3/..rwx
~/songs/OompaLoopmas.mp3/. .type (contains "application/octet-stream")

Applications which use network sockets would be aware that before sending a file across the network, it should probably convert it into a network file first.

Perhaps there would be a way to fopen the file with a flag indicating it needs to be run through the library which converts the file into it's original "network form" and feed that stream on the file handle returned from fopen().

It would be much more taxing on the filesystem to change the original file to reflect the meta-data every time something in that meta-data is changed. If one byte of data needs to be added to the front of a long file, that could require the whole file to be re-written. These kinds of changes will probably occur a lot more frequently than the times where a file needs to be made "ready" to go out on the network.

For that reason, I think it would probably be more efficient if the conversion process is made at the time the data is copied to or from the network. A system of libraries to do this conversion would probably be more elegant, and would be much more well integrated with the pupropse of the file system described in the article.

As a bonus, the packager could perhaps also be intelligent enough to do file conversions through some additional modular libraries (if they are available).

$> mkfile -t ogg ~/songs/OompaLoopmas

Obviously you wouldn't be able to convert all files to all types.

$> mkfile -t jpg ~/songs/OompaLoopmas
Error: ~/songs/OompaLoopas contains 'audio/mpeg' and cannot be converted to 'image/jpeg'.

I like most of the ideas proposed in the article, and I hope they get a chance to make it into the mainstream, though I'm sure it will take quite a bit of time before that can happen.

The only thing I'm not quite comfortable about is the mount layers idea, as that seems it could make room for a lot of confusion if it's use is not kept to a minimum.

Re:Transferring Files

[dsarrazin]

Right, but if on the destination FS, files-as-directores aren't supported, you've lost you ability to play any of the files, because they are now stored as directories.

Re:Transferring Files

[jafuser]

The .info files for the Amiga, IIRC, were only for the workbench-related meta-data (including the icon).

You didn't have an .info file co-existing with all of the files in the filesystem, and the .info file did not contain read/write/archive/hidden/etc types of permission meta-data.

Re:Transferring Files

[ozzee]

Are you mad? This is slashdot.

There has to be a joker in every group.

Re:Transferring Files

[SewersOfRivendell]

Another good example to study is Subversion, which is revisionining/versioning metadata-management layer on a top of a regular FS.

Huh? Subversion stores data in a database -- currently Berkeley DB 4. Although you could theoretically write a filesystem backend, (1) you'd probably lose a lot of performance and disk space and (2) it's not quite mature enough yet to handle multiple backends.

Re:Transferring Files

[jafuser]

Obviously, in the use of legacy file formats, we would need filesystem hooks to do conversions at the time that the files arrive and/or leave the computer over a network.

Once we have become comfortable with a filesystem like the one described in the article, new file formats would be developed which would start putting the regular file data in with the meta-data in an organized fashion which makes all of the important parts easily accessible.

This would be quite elegant, as you would not have to open the file and seek around to get to some piece of data if that's all you need. Instead, you just crawl down the hierarchy to the part you want and open that file and you're pointed right at the beginning of the data you need.

Re:Transferring Files

[Gorvah]

At least the existing metadata (file permissions, creation time etc.) could be implemented as files in a backwards compatible way. I kind of like the idea.

BTW, nice article, though it omitted a few points like backup, encryption and, sigh, undelete.

Re:Transferring Files

[Anonymous Coward]

It's hierarchical because they inherit the fields of the levels above, like permissions and path.

Re:Transferring Files

[spitzak]

Absolutely #3 is the way to go. "metadata" should always be figured out from the contents of the file, the metadata should really be considered a cache of the answers.

The way it would work is a program would look for a piece of metadata. If it was not there it would run a standard program using popen that would then create the metadata and return it, and also set the metadata on the file if it had permission to. This program would probably work something like "file" does now and would use text configuration files to actually determine how to extract interesting information from all known file types. Most likely this actual implementation of looking up metadata would be wrapped in a libc-level function so it just looks like you ask for the metadata.

The obvious advantages:

Files can be transferred by protocols that don't understand metadata.

Files can be stored on filesystems that don't understand metadata.

Metadata is never "lost" as long as the real file data is intact.

Metadata can be stored locally for remote files.

Metadata can be modified without permission to modify the file.

Probably much faster because metadata is not calculated until first needed.

I also think that all files should be treatable as directories, and that the metadata should be presented as being inside this directory (ie the metadata "blah" for file "foo" is foo/blah). But this does not mean it has to be stored this way and does not mean it has to exist when copied.

Re:Transferring Files

[jonadab]

The way Be handled this was to declare all other filesystems to be
inferior and tell you that you shouldn't try to store data on other
filesystems, or if you _had_ to, that you should put it in a .zip
file first. This was of course an inane answer, since it is not in
any way reasonable to expect people who multiboot to store their
data on a filesystem that only one OS can access, but it was their
answer. I suppose it worked for people who never wanted to use any
other OS.

Re:Transferring Files

[mobileskimo]

Correct me if I got this all wrong, pardon.

Example: if source has Plan9-FS with directory metadata and target has your "normal" FS with metadata attached.

SSH the tarball and unload on target system and it will produce a file plus additional directories containing metadata. If this was my mp3, winamp on my target system would not be able to read this music file correctly since it is expecting one file containing the music and the metadata, not one music-files plus directories containing metadata.

Yes? No?

Re:Transferring Files

[Anonymous Coward]

Ever heard of using |pipes|?

Maybe you remapped that key to start Internet Explorer on yuor XP instead.

Re:Transferring Files

[Anonymous Coward]

Hey, faggot, using pipes still requires that tar be installed. Requiring another program is no different than changing the same program.

Re:Transferring Files

[PCM2]

Metadata is very application specific and most of filesystem are agnostic about it. Typically it must be handled by another layer on a top of FS.
...
You may research and find some software implementing a layer (on a top of a regular FS) specially designed to handle MP3 playlists. But again, that would be a layer on a top of FS, not a filesystem by itself.

So how does this answer the earlier poster's question?

When he was talking about "MP3 metadata," I don't think he was talking about playlists. He was talking about ID3 tags, if I understand correctly.

Currently, MP3 files store their metadata in a special area within the file itself. Presumably, on a machine that handles file metadata at the filesystem level, you would abandon this barbaric idea and use the tools available to store MP3 metadata in the filesystem.

So what happens when you try to copy an MP3 from a filesystem that supports metadata to one that doesn't?

Answer: Probably about the same thing that happens when you try to send a file from Mac OS 9's HFS filesystem to an MS-DOS computer. Either you use some kind of special translation program (MacBinary, or an archiver like StuffIt that supports Mac OS Resource Forks), or you lose the metadata.

Re:Transferring Files

[Arandir]

Presumably, on a machine that handles file metadata at the filesystem level, you would abandon this barbaric idea

Why is this idea barbaric? Filesystem metadata should only be for file metadata, and not content metadata. There is a crucial difference. ID3 tags are part of the content of MP3 files.

Re:Transferring Files

[dgp]

having the metadata in the file seems to be the best way to go.

opening a file called /path/filename/attribute would look inside the /path/filename file for attribute data. The file could be built like a zip file and the attribute could be a text file in the zip, or the file could be built as a reiser4 filesystem that gets mounted on the fly, or the metadata could be held as XML in the file.

Either way, if a program were to cat /path/filename, they would not get the contents of the file as expected, but the contents and all the metadata that goes along with it. That may be a problem. I suppose for compatibilty, opening /path/filename and writing to it would really be opening /path/filename/contents and writing to that. But the filesize would be wrong if the program wrote 5k and the filesystem added 1k of metadata but the program goes to check the size and thinks something is wrong because the file is now 6k. the stat() call would have to be changed to return the size of the contents only to keep some backwards compatibility. now for a filesystem dump utility, if it were to add up all the stat() sizes of each file, it would still be underestimating the amount of data stored. a utility like that would have to be aware of the extensions of the filesystem.

Re:Transferring Files

[wfrp01]

Mistake: thinking that's there's only one source of metadata. Multiple metadata sources may exist for any particular object. The w3c attempts to address this issue with their rdf standard.

The first (the worst) option you mention, while not entirely apropriate, may indeed be closest to the ideal. Because we are not talking about a separate metadata file, but files . Now I wish I had a clever answer about how to maintain linkage between metadata and the things metadata describes as things move around, but I don't. But my main point is that metadata should likely be dissassociated from the thing it describes.

Re:Transferring Files

[Anonymous Coward]

>> Maybe that's because Apple acquired NeXT...

I think you meant to say, "NeXT acquired Apple." To the tune of -$400 million! :)

Re:Transferring Files

[Anonymous Coward]

What *is* a filesystem? Really, it's a relational database.

No, it's not.

The modern UNIX VFS is a hierarchical database if it's any sort of database, but really, it's a filesystem. Which is why we call it that.

Re:Transferring Files

[Anonymous Coward]

...but calling a filesystem a relational database sounded good, and the post was modded up to 5. Where's the problem?

Seriously, thanks for pointing that out, and if I can continue my cynicism, MS is way ahead of the open source community on getting a real implementation of a next generation filesystem based on database concepts.

Re:Transferring Files

[pascalb3]

When MS announced their creation of WinFS (/. link) I was a very happy person. After all, who wouldn't want to have one copy of their MP3s be referenced from multiple 'files' (alphabetically by artist, alphabetically by title, genre)?

The key issue that I ponder is that of fragmentation. How will the placement of files matter in a relational DB world? Right now, it is easy to group files together based on their hierarchical residence -- but, with something like WinFS, would filesystems take on a more DB role? Will moving my files from one system to another change from 'cp /home/me -R' to 'select * from home where personal=yes' (in an MS SQL-based world)? While I would love the change in FS logical design, I am hesitant about shifting paradigms to a world without folders and their convenience. Of course, I realize that those concerns are basically moot by definition of a paradigm shift, but making that leap is initially one of faith.

On the plus side, imagine the infinite ways to categorize your porn with metadata!

Re:Transferring Files

[Zork the Almighty]

Personally, I think the entire idea of metadata is overhyped. It's not that useful to the majority of computer users, and it's incredibly hard to implement. Why would you want metadata in the filesystem anyway ? The biggest drawback is that the data can be lost when files are transferred to another computer, unless every application respects metadata in some standardized way. Futhermore, it's hard to implement in a way that's flexible enough to be useful while still being meaningful.

I propose an ugly hack. Why not just make every file an XML file, with a header containing the metadata (which could be anything) and a binary field for the file's data. Have the default system calls ignore the XML and send the data, to ensure compatibility. Have a second set of system calls which pass the entire XML file (for new applications which support it). Write the XML so that if you email one of these files to someone that doesn't support the system, it could be trivial for them to get the data. Anyone (knowledgable or otherwise) want to poke holes in my idea ?

Re:Transferring Files

[Anonymous Coward]

tar is a POSIX.2 requirement, dipshit.

Re:Transferring Files

[soellman]

what, you mean like .DS_Store? :)

Re:Transferring Files

[Jonner]

You used the key word "graph", which is a clue that *nix filesystems are not strictly hierarchical. They are in fact graphs, not trees. However, you're right that they aren't relational.

If you read all of Reiser's stuff, you'll discover that his ideal is a model that's even more powerful than the relational one. It will be able to accomodate current filesytems, relational databases, and other approaches equally well.

Re:Transferring Files

[mrogers]

Write the XML so that if you email one of these files to someone that doesn't support the system, it could be trivial for them to get the data.

OK, imagine how you would do that: you wouldn't want to send the data twice, so the data would need to be sent as a second attachment, and instead of a data field in the XML file you would have a field saying, effectively, "see second attachment for data". So what you have now is a file for metadata, and a file for data.

If you continue to break down the metadata file so that _every_ field is stored in a separate data file, you're left with an XML file containing field-name/pointer pairs (ie a directory), and a bunch of files. Which is pretty much what the article was describing, but without the XML.

Re:Transferring Files

[mrogers]

I think it works as follows:

The metadata-aware system treats the MP3 file as a directory. If there is a plugin associated with MP3 files, the contents of the directory will be dynamically generated based on the contents of the file (ID3 tags for title, artist etc). The ID3 tags will also be updated by the plugin if you modify the contents of the directory (eg echo "Ornette Coleman and Howard Shore" > "Naked Lunch.mp3/artist"). Obviously if you create metadata that the plugin can't understand, it won't be stored in the file (eg echo "8/10" > "Naked Lunch.mp3/rating").

If there is no plugin associated with MP3 files, the contents of the directory will be independent from the contents of the file. If you change Naked Lunch.mp3/artist, the corresponding tag in the MP3 file will not be changed.

Either way, you will be able to move the MP3 file to a metadate-unaware filesystem and it will still be a valid MP3 file. However, some of the metadata will be lost unless you move the entire directory.

Here's my question though: if you move the entire directory to a metadata-unaware system, will it be possible to access the MP3 file as a file rather than as a directory, through a file called Naked Lunch.mp3/..data or something similar?

Re:Transferring Files

[EddieSam]

[...] metadata is overhyped. It's not that useful to the majority of computer users [...]

Erm... filenames are metadata. How are you going to open documents if they don't have names? Filename extensions are also (badly misplaced) metadata indicating the type of data stored in the file. How do you even know it's a word processing document without that metadata? File size is also metadata, often useful for determining how long a file will take to download, or how much of a particular media you need to store it.

So, "not that useful", huh? I think you're very wrong, and haven't actually thought before typing. What this article describes is a way of storing metadata in a more useful form than things such as filename extensions (what numbnut decided to merge mutable and immutable metadata?). If you'd read the article, you'd already know how useful metadata can be, especially when organised correctly.

Re:Transferring Files

[Anonymous Coward]

Imagine that. Windows not handling Unix style permissions! We need a new acronym like WNU (Windows ' Not Unix). Besides Windows ACLs are superior to Unix type file permissions anyway.

Re:Transferring Files

[be-fan]

Use .zip, which has the benifet of preserving all metadata.

Re:Transferring Files

[sorbits]

Absolutely #3 is the way to go. "metadata" should always be figured out from the contents of the file [...]

So you think that *every* file format should re-invent the wheel and define a tagging scheme? So an mp3-player will need to read and parse ID3 v1, v2.2, v2.3 and v2.4 (yes, all these tagging schemes are incompatible) and when Ogg Vorbis support becomes an issue, create a new parser for their meta data format...

Also, some files are not really in a format, so you can't embed meta data within these, e.g. text/plain (used for scripts, sources and similar).

Re:Transferring Files

[spitzak]

Yes, but the work would not be done by the "file format" but by user-level plugins that translate attempts to access "foo.mp3/blah" into the value of "blah" stored in the file. Also because it is slow, file-system support for "metadata" would be very nice, because such a program can then store the result into the metadata "blah" so it can be accessed instantly next time. However if you then copy the mp3 file to another disk and ask for blah, rather than the requiring copying of the metadata, instead the same program would run and extract the information.

And it would certainly be better to have one program for each of these MP3 types than to have every program that wants to catalog files in any way have to understand every MP3 file format.

True text/plain files, where every single byte is not a comment, will only be able to have metadata if it is stored seperately, and thus such metadata will only work on filesystems that support it, and reading "file/" will have to produce some cooked version that contains the metadata so it can be copied to another disk. However I am unaware of any text format that does not support comments. Have you taken a look at any code checked into CVS or edited with Emacs? Putting metadata into comments is an already well-established practice.

Re:Transferring Files

[StalinJoe]

ever heard of a hierarchical database?

I program in a language called MUMPS.

MUMPS is a language and database combined.

MUMPS (the database) is in every sense a hierarcal database.

Caché is a hierarcal database (it is a language that has evolved from MUMPS and still retains the hierarcal underlying database.)

Re:Transferring Files

[RapaNui]

The 'bundle' system (or a _rough_ approximation thereof) is supported by Rox Filer.
It's quite a neat way to distribute apps (for systems running Rox, obviously). It's also a pretty cool (small and fast) file manager, btw.

Re:Transferring Files

[Anonymous Coward]

Besides Windows ACLs are superior to Unix type file permissions anyway.

You're on crack right?

Re:Transferring Files

[Anonymous Coward]

Are you mad? This is slashdot.

This is getting so lame and old. If you have a problem with Slashdot then please leave and let the people who still like it discuss in peace.

Re:Transferring Files

[Xabraxas]

Metadata is very application specific and most of filesystem are agnostic about it

I believe you mean ignorant, not agnostic.

agnostic ( P ) Pronunciation Key (g-nstk) n.

1.

1. One who believes that it is impossible to know whether there is a God
   
2. One who is skeptical about the existence of God but does not profess true atheism.
   

2. One who is doubtful or noncommittal about something.

ignorant ( P ) Pronunciation Key (gnr-nt) adj.

1. Lacking education or knowledge.
2. Showing or arising from a lack of education or knowledge: an ignorant mistake.
3. Unaware or uninformed.

I don't see how a filesystem can be doubtful and noncommittal. I do beleive it can be unaware or uninformed though.

Re:Transferring Files

[axxackall]

never trust any single dictionary. Besides, most of dictionaries cannot reflect any modern vocabulary changes. Especially in information technologies, where we often us antropomorphisms.

google and you will find: distro-agnostic way of installation, vendor-agnostic trunks in Linux, handheld-agnostic mobile services, device-agnostic messaging, API agnostic, and so on.

I think "ignorant" is not a good word here. If you don't like "agnostic", than maybe "neutral".

But when

[infolib]

will ReiserFS be ready for the 2.6 kernel? Just curious.

Re:But when

[RealRoadKill]

YES go to the source.... The ReiserFS Homepage is http://www.namesys.com/

Re:But when

[Anonymous Coward]

Real soon now.

Latest benchmark results are available at the
http://namesys.com/intbenchmarks/mongo/03.07. 11.li ght/short.html

we still have problems with delete performance, but in three to ten days
reiser4 will be released to the public testing.

Start weeding out ...

[chess]

... trailing / in filenames now.

Usage of text editors as well, cat and echo will rule!

chess

wtf?

[Anonymous Coward]

sure, im going to listen to this guy about filesystem implementation when he cant even set up MIME types on his webserver. jackass.

Why?

[Libor Vanek]

It's really nice. But what does it brings new that we shoudl rewrite 90% of all system tools too use this new features? I find "cp /a/..uid /b/..uid" same as "chmod /a --reference=/b"...

Re:Why?

[Libor Vanek]

Sorry - I meant "/a/..rwx".

Re:Why?

[cperciva]

You're missing the point. chmod would still exist as a userland program; it is the kernel call which would be removed.

To the user, there would be no change; to the userland programmer, there would be no change; to the C library developer, there would be a change (to implement chmod in terms of the existing filesystem operations); and to the kernel developer there would be a change (mostly in the direction of reduced complexity due to a smaller number of necessary functions).

Re:Why?

[IamTheRealMike]

I think people are distracted by the examples he gave, which make the point clear but are perhaps not representative of what this would be used for in real life.

GConf was a better example. ATM using GConf is, well, not hard, but you have a lot of extra machinery involved, new APIs to learn and so on. Basically all that machinery does is control the backends and give change notification (it does stuff like schema validation as well).

It'd be *much* easier to use GConf if in order to read a value, you didn't have to load up the GConf libs (which in turn depend on CORBA), or parse XML files. At the moment that's really the only way to do it, but in most environments/languages it's far easier to manipulate files and directories than it is to talk to a CORBA server or bind APIs into them.

You also get an increase in efficiency. Parsing XML is kind of cludgy - XML is not a particularly efficient format to store stuff in. It's a good compromise between humans and machines, but both of us have to do lots more work to meet in the middle. The reason it's used, rather than lots of small files, is that otherwise GConf would be too slow. In fact, they are already talking about removing yet more of the files/directories to speed things up, and sticking them all in the same XML file.

Being able to have a configuration system that truly leveraged the filing system would make a lot of stuff easier, more reliable, and faster (because you can take advantage of filing systems that are really really tuned to take advantage of advanced data structures).

It won't really impact the way you do things like set file attributes today. Most of the changes would be under the hood. But used well, everything would become easier for the developers, and so more advanced and slicker for the user.

Re:Why?

[LordBodak]

You're right, XML isn't the most efficient way for gconf to store data. But is rewriting the filesystem really the answer?

Re:Why?

[tedgyz]

This is a good point that I think was not emphasized enough. Removing kernel calls, especially something as pervasive as accessing file attributes, can have a significant performance advantage.

Kernel calls require context switching, which means significant overhead (relatively speaking) in modern CPUs, when compared to a similar operation without a context switch.

It is the aggregate of all these calls across a single process and across all system processes that will make it a performance benefit.

A real world example showing evidence of this problem is in my CD burning app - Nero (as well as others, I presume). It goes out of it's way to cache small files. Why? Because the overhead of fetching small files is too great, compared to their size. If it didn't cache the small files - especially if there are a lot of them, the CD burner cache would run dry while the burner app loads the small files off the hard disk.

Re:Why?

[Nevyn]

GConf was a better example. [...] It'd be *much* easier to use GConf if in order to read a value, you didn't have to load up the GConf libs (which in turn depend on CORBA), or parse XML files.

No GConf was a terrible example, because reiserfs doesn't solve that problem. Even if reiserfs came out with all their planned "new fs" stuff tommorow, and all the distributions released an OS upgrade the next day after fixing all the security problems in all the apps (think apache serving metadata for files).

So even then, after all that, you'd need to use GConf and it'd need to do all the same stuff ... because Solaris/FreeBSD/OpenBSD/etc. are all not going away, and they sure as hell aren't going to include reiserfs. Then there's all the people still using older distros, or who are still running from "upgrades" so have ext2 or ext3 instead.

Hans is just on crack if he thinks everyone is going to require reiserfs as the database for their apps. Look at for instance, the "glibc" extensions like obstacks or asprintf() ... noone uses them, why? Because then your apps. don't work anywhere.

Re:Why?

[Arandir]

I agree that GConf was a bad example. But more than that, the whole article was "gee whiz" mental masturbation.

The problem with GConf is that it uses very many very small files. The author's solution seems to be to take this to an extreme, and make each individual key a file. Huh? Opening a file is slow. Opening one hundred files is one hundred times as slow.

You know those database programmers who think everything should be done a database, even screensavers? And those XML programmers who think everything should be done with XML, even screensavers? I almost expected this filesystem programmer to come right out and say screensavers should be done in the filesystem...

Re:Why?

[Nicolai Haehnle]

Keep in mind that implementing chmod() and the like efficiently without a chmod() syscall requires a new API for reading and writing files. A sequence of creat()/write()/close() requires going into the kernel three times, not to mention the allocation and deletion of a file descriptor.

Unless stuff like this can be combined into a single syscall, you have a problem. Not that that problem should be too hard to solve, it's just something to keep in mind...

Re:Why?

[Nicolai Haehnle]

Actually, GConf was a particularly good example because it is possible to write new storage backends for GConf as plugins.

I'm sure that a GConf plugin that stores every key in its own file is extremely straightforward to write. Then the people who have "seen the light" will use that plugin, while the rest of the world is stuck with the (probably less efficient) XML files.

Re:Why?

[Nevyn]

Actually, GConf was a particularly good example because it is possible to write new storage backends for GConf as plugins. I'm sure that a GConf plugin that stores every key in its own file is extremely straightforward to write. Then the people who have "seen the light" will use that plugin, while the rest of the world is stuck with the (probably less efficient) XML files.

It's not obiouvs it'd be faster, as was said. Because you are trading the XML parsing for a pine of system calls. It's easier to write ... sure. But you have to write the XML one anyway because that's what 90%+ of your userbase is going to need, and because of the 90%+ stats you'd probably write that first.

So then, yes, you can possibly write the extra stuff so it works only on reiserfs4 ... and it will possibly be faster. Or you could just write an libdb/LDAP/MySQL/Oracle backend, which is much more likely to be the fastest.

That's the problem with the whole thing, it can't buy you any features ... because using those features will mean large parts of your users won't have them (or your users will go somewhere else where the designer doesn't design for things they don't have); And it's very unlikely IMO to be faster than a real database, so you don't get any speed improvments using it.

So what exactly is the point? That it's much easier to design simplistic apps. that work like crap for 90%+ of people, but that perform okish if you are using the one true FS? Sounds worthwhile.

Re:Why?

[Fermier de Pomme de]

The point is that by building a cleaner abstraction to manage attributes/metadata you get a lot of benefits.

Someone above mentioned that ACLs were recently added to a POSIX filesystem variant. A lot of time and effort went into a very specific type of metadata. With the abstractions described in the article it is possible to define new classes of metadata as you need them without reworking the guts of your fs every time.

As an example consider the chmod gripe above, yes you can change perms, uid , and the like w/chmod so of course it serves no purpose to rewrite the filesystem for the sake of implementing chmod as cp... but can you add an attribute that says 'retain this file for legal proceeding x' using chmod? Didn't think so. With the new abstraction you can define such a class of files and no changes to user space utils or the fs are needed.

I also can't see why everyone is so put off by the idea of providing a standard way to deal with configuration issues. Does every app developer need to spend time coding routines to extract data from a tree-like structure (if they are writing something flexible) or some mp3-esqe tagging scheme. As mentioned GConf is trying to take care of this specific (and very real) problem and the new fs abstraction would be able to solve the same problem as a result of the simplified abstraction. Bonus.

Unfortunately you will need to pay more if you need backward compatability. As has been pointed out already, just package the file in some way if you want to preserve the metadata (tar, jar, whatever). If the destination platform doesn't support the fs then you will need to do some translation - but the good news is when you implement it it will result in an api that can be used by any app to acquire attributes from a file in the new fs. If you are dealing w/legacy code then it shouldn't be hard to move the file with a standard set of POSIX attributes and unfortunatley the other attributes would be lost.

Re:Why?

[Nicolai Haehnle]

The point is that some people like to be visionary. Some people actually believe in clean, logical, scalable and powerful designs. Some people have dreams, and are actually working to realize them.

Apparently, you aren't like this (neither am I, or else I'd be actively participating in efforts like this; but I do at least understand these people at a certain level). That's fine; not everybody can (or wants to?) afford dreams.
And granted, the stuff that is mentioned in the article is a lot less obvious than "enable long filenames instead of 8.3 limitations" [1].

Right now, there are compatibility issues etc. like you pointed out, so not everybody will be able to use the new features. Eventually, however, we (and this includes you, the parent poster) will all benefit from them once they become standard - and they will, at least in the Linux world. Lots of people are already using ReiserFS, and while that's not version 4, there won't be a large mental barrier for them to make the switch (and in the long term, mental barriers are always harder to overcome than technical barriers, but I digress).

Let me get to the actual point of this post.
It's fair enough for you to *point out* the issues at hand. After all, you're right.
However, you shouldn't *criticize* the people who work hard at improving our daily computing lives for what they do. Because in the long term, they're right, too.

[1] Please don't give me any "long filenames are less problematic than this" crap; people still joke about c:\progra~1

plan 9 is awesome

[Sespindola]

There has been talk in the kernel mail list of implementing 9p, the plan 9 distributed filesystem.

Now that is Open Source, I hope it will happen.

Re:plan 9 is awesome

[latroM]

It wont be happen because plan9's and linux's licenses are incompatible with eachother. You can't use all the code that your eyes can see.

Re:plan 9 is awesome

[F2F]

9p is a protocol. the file systems in Plan 9 are venti and fossil

9p in itself is worthy of including in the linux kernel, if only the guys there would do it right (their track record isn't too good with Plan 9 things).

More about 9p dould be found in section 5 of the man pages

Re:plan 9 is awesome

[F2F]

bullshit. the license has changed, go read it.

enough rms-crap -- enough ideas have gotten into Linux from Plan 9 already even before Plan 9 was open source.

Re:plan 9 is awesome

[latroM]

Oh, is it so? This "rms crap" has created the first free operating system GNU/Linux.

Re:plan 9 is awesome

[Arandir]

Err, that would be second free operating system...

ReiserFS rules.

[Anonymous Coward]

Seriously it does. When I first tried linux (Mandrake 8.1) I found that reiserfs meant I could shut down my system inproperly with a small risk of corruption and no need to defrag. It was also big stride for linux on the destkop Why my computer telling me to run fsck manually, what is /dev/hdb15 mean?</grandma> Now grandma dosen't need to worry when her grandchildren pull out the plug.

Im a happy reiserfs user for over 1 1/2 years and I hope that ext2 gets deprecated in 2.6 to put it out of its misery forever. XFS and JFS are good for servers, but reiserfs is the most popular "desktop" file system.

Re:ReiserFS rules.

[Fobi]

I'll have to agree with you, ReiserFS is the best.

I've a machine which clearly have a hardware problem that I haven't been able to resolve yet. I've been moving pieces of hardware to and from between two machines and I seem to end up with about two crashes a day no matter what I do.

At first I used ext3, and it didn't take long until I started to loose files; and lots of them. In the end I had to reinstall and this time I used ReiserFS. Of course the machine still crashes twice a day, but I haven't lost a single file; to my knowledge 8)

Re:ReiserFS rules.

[Anonymous Coward]

I think ReiserFS is overrated. ext3 works fine

Re:ReiserFS rules.

[lightcycle]

I may be way off base, but I see the need to have a existing non-journalling file system. Someone stop me if I'm wrong, but in my mind things like audio recording, video capturing and the likes would suffer performancewise from being run on journalled file systems

Re:ReiserFS rules.

[sweede]

fat32 is non-journaling, NTFS is. What would you rather use for audio recording, video capture and the likes?

Journaling in NTFS is done a lot differently than journaling in ext3 though. I'm not sure how journaling in XFS works, but i would presume since that FS is fairly mature, it is as fast if not faster than NTFS.

Maybe 3-4 years ago when harddisks were not as fast as they are today, the filesystem would of been a big factor. but right now my two WD800JB's on my Serial ATA RAID controller in RAID-0 do about 100megs/sec read and 85-90write, sustained, as reported by sisoft sandra. Can editing video in realtime need transfer rates more than 60megs/sec ?

Re:ReiserFS rules.

[mapnjd]

This is not so. SGI's XFS is a very high performance filesystem (originally available on IRIX) as it would have to be considering SGI's market.

Re:ReiserFS rules.

[orim]

I would suggest checking your memory? If your memory is bad, every time you copy or edit your files, you might be corrupting the data in them.

And with people reporting multi-year uptimes for their Linux boxes, is it really acceptable to have yours crash twice a day? Heck, my old Win98 could go on longer than that :)

Seriously, before you corrupt too much of your data, fix this!

Re:ReiserFS rules.

[Anonymous Coward]

Can editing video in realtime need transfer rates more than 60megs/sec ?

Yes. See, there's these things called "layers." When you're editing, you stack layers of video atop each other. Sometimes you just want two layers, as in a cross-dissolve. Sometimes you want more: a cross-dissolve under an animated title with a matte. You just keep building layer upon layer until you get what you want.

Each layer needs about 40 MB/s if you're doing uncompressed work. There are ways to be smart about this, by having *much* higher disk bandwidth (hundreds of megs per second) and reading in chunks in a round-robin, but in general you need an average of 40 MB/s per stream of realtime playback.

Re:ReiserFS rules.

[Tukla]

I like to user ReiserFS on partitions that I know will contain tons of small files, like my /var/spool/news. I prefer to journal data as well as metadata, though, which is why I use ext3fs for most of my partitions, especially my /home.

Not about filesystems

[Salamander]

The article is not about filesystems in anything like a general sense; it's just a PR piece for ReiserFS and its creator, enunciating criteria specific to that implementation and then attempting to show how it actually (gasp!) meets its own idiosyncratic goals. What a waste of time.

Yes.

[Lispy]

As far as i understand this.

Re:Yes.

[Anonymous Coward]

Umm... That isn't reiser4.

very Reiser/plan9 specific...

[grey1]

...and not very general. Interesting for its comments on what's being tried out in R-FS & Plan9 but certainly doesn't manage to be a general summary of what's going on.

How about the changes coming in 2.6 (like xfs support built in)?

The article makes some good points but for me it could have done with rewriting to make it more general, separate the analysis of filesystem implementation problems from technical detail, and included more examples from other file systems.

Re:very Reiser/plan9 specific...

[axxackall]

How about the changes coming in 2.6 (like xfs support built in)?

XFS is a good example of journalling filesystems. But how about filesystems like Coda, AFS and Intermezzo, a new generation of networking (actually - distributed) filesystems allowing disconnected operations?

Good article

[IamTheRealMike]

It doesn't tell you anything you can't already learn by reading up on the articles on Plan 9 and the whitepapers on namesys.com, but it's a well written compressed version for those who perhaps don't want to wade through a description of set theoretic namespaces (whatever they are ;)

The concept of reducing primitives is a good one, and based in sound mathematical theory. As already pointed out though, you need some container format that can handle some of these new ideas, things like very small files, files as directories and so on. This is needed, because when you transfer files through lossy mediums like the internet, or older filing systems, you don't want to lose the structure.

As far as I know, there isn't a container format that can do this. Tar is showing its age already, I wouldn't like to see it hacked yet again. Zip is alright, but you'd need to break compatability to add in all those extra features, and then it wouldn't be zip anymore. It'd also be inefficient.

So, what I propose is rather than reinvent the wheel to solve this problem, we add support for "boxing" to the Linux kernel.

A box is a filing system in a file. We already use them, to some extent - it's been possible to mount ISO images using the loopback filing system for a while. What's needed is to take this to the next level. The first thing is that we need the ability to use files as mount points, not just directories. When files and directories are the same, well, I guess that should be easier.

The .box file format simply contains a short header with some useful metadata, like maybe a checksum, and the filing system it contains (maybe that isn't needed). The fun part is the UI. What you need is the ability to right click on any dirfile (for want of a better term) and choose the "Box it" option. You'd need a better label. What essentially happens then is that the heirarchy below this point is sucked up and transformed into an ISO containing perhaps a "Reiser4-Lite" filing system. You can forgo the journal and other things that are redundant purely for storage.

The user has then converted their file or directory into something that can be transferred across the net, on Windows compatible CDs and so on, without losing the inherant structure of the original.

At the other end, choosing the "Unbox" option mounts the contents of the box using the loopback FS, mounted at the point of the file. To the user, it is seamless, far easier than zips or tarballs.

Of course, there are lots of complications. You have to agree on the format to use inside the box, for one, because the need to have kernel mods and so on makes it more complex than just installing tar.

I think MacOS has something a little bit similar with disk mountable images (.dmg) files, but the MacOS filing system is rather poor, and I don't know how easy it is for users to create them. Also the OS unfortunately applies some magic to them - for instance Safari will automatically extract the contents of the DMG file then destroy it when you download one (but other stuff does not, oops).

Anyway. That's one way to prevent loss of vital structure when transferring across lossy mediums, that I can think of. There are probably others.

Re:Good article

[Surak]

A box is a filing system in a file. We already use them, to some extent - it's been possible to mount ISO images using the loopback filing system for a while. What's needed is to take this to the next level. The first thing is that we need the ability to use files as mount points, not just directories. When files and directories are the same, well, I guess that should be easier.

Actually, although not seamless except from the users point of view, for a more visual representation, think ZIPs and tarballs in Midnight Commander. MC treats the file as a file, and then when you go into the file, you get a directory, in which you can copy, move and delete[*] the files much as in a traditional directory. The beauty is that metadata can be stored, accessed and modified in the directory-as-file version and it could probably contain just about anything you wanted, including a description of the directory structure and files, for instance with a source tree, it could contain all of the info normally contained in README, COPYING, CHANGELOG, etc.

[*] Depends on the features of the underlying file format and its associated program (zip/unzip vs. tar/gzip/bzip2).

Re:Good article

[OmniVector]

The .box file format simply contains a short header with some useful metadata, like maybe a checksum, and the filing system it contains (maybe that isn't needed).

I've been long wanting a transparent "boxing" system like this on a per file system basis, to the extent that we could erradicate and at the same time standardize the metadata format of all files instead of having different headers to parse for mp3s, avis, mpgs, pdfs, jpgs, etc etc. The biggest issue with a system like this though, is once you want the changes to move from one system to another you're stuck with the same problem Mac OS 9 (and rarely but occasionaly mac os x) still had -- you can't save that extra data without zipping it up in a file format that has the ability to save that information and truncate it on operating systems where there is no mechanism to store it. Then you're left with a container, wrapped in a container, containing finally the file. It's a pickle, and one that would greatly benefit the end user if we could just standardize cross-platform external file meta data, but don't expect Redmond to comply any time between now and the next few years. Maybe Longhorn's SQL middle man will be able to fix this gap for the microsoft camp.

Re:Good article

[IamTheRealMike]

Yeah, but it doesn't really need the co-operation of anybody else to be useful. A filing system upgrade would improve things immensely on Linux at any rate, even if it wasn't portable to Windows or MacOS.

I guess the main problem would be that projects like KDE/Gnome wouldn't be willing to lose portability to forms of Unix that sucked at small files, needing layers of abstraction and so on. OTOH that causes problems just with Linux integration, it's just one of the balances people have to make.

Re:Good article

[Anonymous Coward]

What you are describing was implemented in MS Windows over a decade ago. It's called a Compound File, and it consists of Storages (analogous to directories) and Streams (analogous to files).

A Storage can/should contain a "default" stream that defines the COM object that encodes it's functionality. [With something like Java, they could have included the functionality IN the compound file in a platform neutral manner]

Re:Good article

[Anonymous Coward]

Actually, ZIP is extensible. You can add new data blocks to the ZIP file and if you try to unzip it with a decompressor that doesn't understand your extensions, they'll just be ignored. The Be ZIP implementation extends ZIP with support for its native file system attributes meta data, for example.

TAR isn't so much showing its age as dribling into a bib in the old folks home.

Re:Good article

[bigredswitch]

"for instance Safari will automatically extract the contents of the DMG file then destroy it when you download one (but other stuff does not, oops)"

This is a feature of the .dmg format known as an Internet-Enabled Disk Image. See: http://developer.apple.com/ue/files/iedi.html

Re:Good article

[Anonymous Coward]

On OS X you have a utility called Disk Copy that you can either create blank disk images (which are mounted) or disk images from existing disks or directories. The blank ones which you create can be speced as encrypted in which case you cannot mount them without a password. These also are Read/Write up to the fixed creation size (i.e. they don't get any larger than their creation size). The images created from directoriesor other disks are Read Only and compressed. You can mount them, they expand and you can copy data from them.

Re:Good article

[IamTheRealMike]

Ah, no, in no way does OLE Structured Storage compare to this.

For starters, the MS compound document file format is a very poor reimplementation of a filing system in a file. Why reinvent the wheel like this? Because the filing system layer didn't provide what they needed.

Unfortunately, OLESS is kind of a text book example of how not to design a filing system. It's opaque to the normal APIs, you have to access it using COM. The files contain large strips of garbage for compatability with earlier, buggier versions. The format is inefficient and limited, and you lose the benefits of things like fine grained security and journalling.

Basically, the whole point of things like ReiserFS is to eliminate pointless reinventing of the wheel like this, by adding the features that caused the invention of these formats to the filing system itself.

Re:Good article

[SleezyG]

I agree with Mike in that the article is way too compressed. It reads as if the author threw it together as a homework assignment, 5 minutes before class started. Lot's of great material, lot's of neat things to pique my curiosity about "alternate" filesystems. But, it needs to slow down, explain some concepts and then demonstrate their usefulness. The paper should have been 3 times its current length.

Re:Good article

[Matthias Wiesmann]

Actually, the .dmg format used by OS X seems to be an older format: lif. This what solaris tells me:

% file Hydra.dmg
Hydra.dmg: lif file

Disk images under OS X can be formatted with different file-system (UFS, HFS+, HFS and FAT), so I suspect that the disk image represents some kind of virtual disk device. The file-system on this device is then handled by the relevant drivers.

This means you could probably avoid inventing the round thing again and use the same format for disk images under Linux. Simply the disk images could be formated using different file-system formats (ext, reiserfs, whatever).

Re:Good article

[zsau]

choose the "Box it" option. ... choosing the "Unbox" option ... To the user, it is seamless, far easier than zips or tarballs.

Ah... no, that isn't seamless and seems no simpler than zips and tarballs. Just have three ways of accessing files: /file gives you just the content, /file/ lets you see what's inside, and /file# gives you the complete file (pre-compressed, perhaps). Of course, the user needn't know of such technicalities. When they decide they want to email a file to someone, they say the want to email /file and the program asks the system for /file#. Then, at the other end, the program says to the system: here's file# and the system automatically knows that we're talking about a completefile. Obviously if someone asks for /file#frog/, that's impossible, so we'll give them /file/frog/ instead.

That's much closer to seamless.

Re:Good article

[IamTheRealMike]

It's not discoverable though. How is anybody supposed to guess that they can attach this file by sticking a # on the end?

Obviously you can have mail apps, browsers etc doing all this automatically, it was more a description of how it could work for times when it's not automatic.

Re:Good article

[Have Blue]

I think MacOS has something a little bit similar with disk mountable images (.dmg) files, but the MacOS filing system is rather poor, and I don't know how easy it is for users to create them.

It's *very* easy to create a disk image. Drag a folder onto the Disk Copy app and select a destination to store the image. There's no step 3!

Re:Good article

[zsau]

Obviously you can have mail apps, browsers etc doing all this automatically, it was more a description of how it could work for times when it's not automatic.

But it should be automatic. At any time you need to take a tree with data and unsupported metadata or other trees* across to something that doesn't support metadata, it should be done by the system. The idea of it being unautomatic is stupid (IMHO). If you can think of a reason, I'd be delighted to here it, but you can always access the tree as a whole with the # and you can't not treat it as a whole on systems which don't understand the metadata.

*I invented this terminology for want of a better word. A tree with other trees but without data would be a folder. A tree with data but without other trees would be a document. A tree with both is something that doesn't correspond well into the current environment.

Re:Good article

[Anonymous Coward]

The article's ideas would suck if they were used as the native file system. He mixes meta-data in with paths. They are not the same. Not conceptually, not practically. You still have to re-write your utilities, and change the API. So what's the point? Why not do it right, and build a filesystem with arbitrary meta-data treated as meta-data.

That said, the parent's comments relate to a packaging and transfer format, not a native format. There are two tricks to making meta-data portable.

The first trick is standardizing on what types of meta-data will be available, and the format of each type. ACLs or mode bits? Which version of ID3? That sort of thing.

The second trick is converting the meta-data from a portable format to a usable format, and vice versa. By usable format, I mean one matched to a particular filesystem, one that the OS and apps can use.

The exact portable format doesn't matter. It could be OS 9 resource forks, it could be sub-directories, it could be uber-ID3 tags. It could be disk images or box files. It doesn't matter so long it is standard across the filesystem-specific and app-specific converters.

As already pointed out though, you need some container format that can handle some of these new ideas, things like very small files, files as directories and so on. This is needed, because when you transfer files through lossy mediums like the internet, or older filing systems, you don't want to lose the structure.
As far as I know, there isn't a container format that can do this....So, what I propose is rather than reinvent the wheel to solve this problem, we add support for "boxing" to the Linux kernel.

He is right to not re-invent the wheel, but Macintosh disk images can already handle more than box files. Unfortunately, the disk image daemon isn't open-source yet, though it's been requested.

Two approaches to meta-data are the hidden file/directory approach and the inherent-in-file approach.

Disk images support the hidden file/directory approach by treating directories as files rather than files as directories, but either one will do the trick. You can have whatever "meta-data" sub-directories you like, though the OS might not be able to parse them out.

Disk images also support the inherent-in-file approach with resource forks, a.k.a. OS 9 meta-data. A resource fork is a series of records, like a database, accessed by a code such as "str#" or "icon". In the language of the article, the codes could be "uid" or whatever. It's not truly arbitrary, but it'll do the job well enough.

I think MacOS has something a little bit similar with disk mountable images (.dmg) files, but the MacOS filing system is rather poor, and I don't know how easy it is for users to create them. Also the OS unfortunately applies some magic to them - for instance Safari will automatically extract the contents of the DMG file then destroy it when you download one (but other stuff does not, oops).

You can use Disk Copy to create or extract images. It's a drag-n-drop operation, very easy. You can create images that are compressed or encrypted, from folders or volumes, and they can include multiple partitions. You can extract them transparently across a network (i.e. the system downloads, extracts, and mounts the image then deletes the file automatically).

Safari's "magic", btw, is nothing of the sort. It simply recognizes a disk image like it would any file type, and gives it to the appropriate handler, a disk image daemon. The Disk Copy program uses the same daemon.

Re:Good article

[Anonymous Coward]

For starters, the MS compound document file format is a very poor reimplementation of a filing system in a file. Why reinvent the wheel like this? Because the filing system layer didn't provide what they needed.

Sure, no file system is going to provide us with what we want, which is why we should be moving away from file systems. One of my pet peeves is the whacky emphasis unices seem to put on files. "everything is a file!" - nope. Not everything is a file, nor should everything be treated as a file.

Re:Good article

[spitzak]

You are describing another kludge just like tar and zip and GConf and the Windows registery and every file in /etc and all the other examples, designed to get around the limitations of current file systems. The original poster is also confused, his "boxing" sounds like the same thing.

Ideally the "boxing" should be invisible. I would like to see "cat directory" spew a stream similar to a tar file so if you did "cp dira dirb" dirb would end up a duplicate of dira. This would allow entire hierarcies of directories and metadata to be stored on systems that don't understand the metadata, and to be transmitted as single blocks of data through email attachments and other network protocols.

Re:Good article

[spitzak]

There is a back-compatability problem with having "cat foo" produce all the metadata in foo (which otherwise would probably be a good idea). I like your solution of adding some more letters to the end. But we certainly don't want to reserve a new character like '#'.

So maype "/file/" could be used to indicate "all the data in /file". "/file" would be for back-compatability and produce "the main block of data". "/file/foo" would extract subdata (metadata, attributes, and internal files) and the exact behavior would depend on the type of "/file" and on what plugins are installed in the file system.

Incompatibilities with another system

[panurge]

Windows support for metadata has always sucked, recognised by every Mac user who moved to a PC and discovered that you had to tell the system what a file did by appending a clumsy tla to the end, and passing gently over the inconsistencies of the support for long and short filenames.

If Linux and related systems move to filesystems with really powerful metadata support, presumably the lockin would be much stronger. Moving a directory from Linux to a Windows system may be possible but the programming to do it will become increasingly painful and the risk of data loss will rise. And with mainframe integrity, why would you want to, Mr. customer?

Apart from the CS issues, is this an attempt to use the embrace, extend weapons of Microsoft against it by turning the Linux filesystem into a full mainframe system, effectively squeezing out Windows servers by a convergence between big tin and small boxes? I guess this is pretty pie in the sky but I'd like to think so.

Re:Incompatibilities with another system

[IamTheRealMike]

Nothing is stopping Microsoft from implementing support for the same technologies.

After all, if a few volunteers can implement NTFS support in Linux with no source and no specs, then Microsoft with all its manpower can certainly add support for Reiser4 to Windows when they have the source and specs (and maybe HR would be willing to support them, for the right price).

Basically, it's a non-issue in my books. It's like people who are "locked in" to Word, because they use advanced features available nowhere else. Well, that's the decision you make, isn't it.

Re:Incompatibilities with another system

[Anonymous Coward]

-1, Redundant.

Re:Incompatibilities with another system

[Mitchell Mebane]

Windows support for metadata has always sucked, recognised by every Mac user who moved to a PC and discovered that you had to tell the system what a file did by appending a clumsy tla to the end, and passing gently over the inconsistencies of the support for long and short filenames.

Interesting... in OS X, file extensions are now the rule of thumb. And I seem to recall that classic Mac OS's had a filename limit of 31 chars, while in Win95+/NT4+ it was 255 chars...

Re:Incompatibilities with another system

[sql*kitten]

Windows support for metadata has always sucked, recognised by every Mac user who moved to a PC and discovered that you had to tell the system what a file did by appending a clumsy tla to the end, and passing gently over the inconsistencies of the support for long and short filenames.

Actually, NTFS support for "metadata" is impressive; you can have 255 streams per file. A stream in NTFS is what Mac users call a fork, but Macs are limited to 2, data and resource. You can happily make every file on NTFS an OLE server too and do away with file extensions altogether, if you want to. Oh, and NTFS has reparse points too - think like a trigger on a database table, but attached to a file. And NTFS has journalled from day 1, whereas Linux filesystems are only just discovering this.

So why are file extensions still in common use? Largely because people who don't know NTFS come out with statements like "windows support for metadata has always sucked" without bothering to read the documentation, so few apps take advantage of this NTFS feature.

Re:Incompatibilities with another system

[poot_rootbeer]

Windows support for metadata has always sucked, recognised by every Mac user who moved to a PC and discovered that you had to tell the system what a file did by appending a clumsy tla to the end

As opposed to Mac OS, where you have to tell the system what a file does by setting an even clumsier 4-character code hidden deep within a file's metadata?

and passing gently over the inconsistencies of the support for long and short filenames.

What inconsistencies? Correctly-written modern applications (say, those written after 1995) support long filenames. Older apps do not, but you can still use files with long names with them by using the equivalent short filename.

The 8.3 compatibility layer is a benefit for those with legacy apps.

Moving a directory from Linux to a Windows system may be possible but the programming to do it will become increasingly painful and the risk of data loss will rise.

And whom would that benefit? Possibly, Linux. But right now, Linux isn't facing a situation where customers are threatening to migrate to Windows. Linux vendors are more likely to lose customers they don't even have yet, because who wants to move to a system they're locked into and can't back out of?

Re:Incompatibilities with another system

[Anonymous Coward]

It true that NTFS has all that capability. But very little of it exploited in applications or even the standard Explorer Shell.

This goes back to the same problem they've had since OS/2 --> Because the UI has to be 100% funcitonal on FAT and dumb network filesystems, nobody takes advantage of neat filesystem features in NTFS.

Linux GUIs will have likely have the same problem with Reiser4 -> Very few users, therefore all features will be designed around the Lowest Common Denominator POSIX-type filesystem.

Re:Incompatibilities with another system

[panurge]

You make a good point. I wrote the original post with insufficient reflection before hitting keyboard, and I guess this is actually the most likely scenario.

Re:Incompatibilities with another system

[Aapje]

A stream in NTFS is what Mac users call a fork, but Macs are limited to 2, data and resource.

It's a bit more complicated than that. HFS+ supports 256 forks internally, but the API to use them isn't there. This has probably got to do with the NeXT guys who dislike forks and (non-Unix) metadata.

You can happily make every file on NTFS an OLE server too and do away with file extensions altogether, if you want to.

I'm sorry, but I'm not an Windows expert. What would that change?

Oh, and NTFS has reparse points too - think like a trigger on a database table, but attached to a file. And NTFS has journalled from day 1, whereas Linux filesystems are only just discovering this.

Yes, I know. NTFS is one of the few truly good products of MS. Too bad that 90% of its features aren't used to their full potential. Your reparse points are a good example, even with them Windows doesn't support triggered scripts (without extra programming). In contrast, MacOS supports Folder Actions for quite some time already. It allows you to use the GUI to select a script that will be run at a particular event. Very useful to create an automated workflow.

So why are file extensions still in common use? Largely because people who don't know NTFS come out with statements like "windows support for metadata has always sucked" without bothering to read the documentation, so few apps take advantage of this NTFS feature.

I disagree. An improved system should be built into Windows, supported by the interface and very easy for programmers and users to use. One shouldn't have to program a better solution for every app separately. What's the improvement if we end up with 100 different replacements for file extensions?

Re:Incompatibilities with another system

[Anonymous Coward]

Interesting... in OS X, file extensions are now the rule of thumb. And I seem to recall that classic Mac OS's had a filename limit of 31 chars, while in Win95+/NT4+ it was 255 chars...

File extensions are indeed the rule of thumb now. We don't like it. And, yes, OS 9 Finder was limited to 31 or 32 characters, though HFS+ itself was not. That didn't bother most of us, because usually 31 characters were enough, but MP3 traders hated it, though.

Re:Incompatibilities with another system

[Anonymous Coward]

Windows support for metadata has always sucked, recognised by every Mac user who moved to a PC and discovered that you had to tell the system what a file did by appending a clumsy tla to the end, and passing gently over the inconsistencies of the support for long and short filenames.

Agreed. It's really stupid from an system architectural point of view to multiplex your metadata. A file's name should be just that, its name -- not its name, its type, its astrological sign and so on.

But then again naive system archictecture is fundemental to Microsoft products. I swear their systems behave as though they were written by complete rookies (complex where they don't need to be, caching volitile information that has low query costs and not caching stable information with high query costs. Oh, and never testing anything but the common case).

Re:Incompatibilities with another system

[spitzak]

Linux GUIs will have likely have the same problem with Reiser4 -> Very few users, therefore all features will be designed around the Lowest Common Denominator POSIX-type filesystem.

This is quite possible if it is not done right, but most of the designs seem to be approaching this correctly.

The biggest difference from NTFS is that the "metadata" is accessed using normal file open/read/write/close. This immediatly makes 95% of the software on Linux able to do something with this data. NT failed to do this obvious design (yea I know there is a lower-level interface like this, but that is NOT what existing programs use to open files!).

The other more difficult need is to have this not screw up when files need to be stored on older file systems, which is the main problem you are talking about. I believe this can be solved by making as much metadata as possible be contained in the "data fork" of the file, so when you "cat" it you get almost all the data. Then if the file is copied to an older disk system and then copied back you will get all your metadata back. Actually most solutions will allow a new Linux+metadata system to see the metadata even if the file is on an older filesystem, as the actual extraction of metadata would be done by user-level local plugins.

In my opinion these proposals are light years ahead of NT/Apple/BeOS in the ideas of naming, but I don't see any real attempts to solve the data problems.

How about fixing the current filesystems?!

[FyRE666]

Really, I think someone should get on with finishing the NTFS filesystem access in the kernel. With people migrating to XP it's really becoming more important that this driver is fixed (how long has it been declared "dangerous" for write use now?!).

I'd really like to know why this driver has taken so long to complete - is there some information that the developers don't have access to? Some technical reason? What?!

Re:How about fixing the current filesystems?!

[IamTheRealMike]

I'd really like to know why this driver has taken so long to complete - is there some information that the developers don't have access to? Some technical reason? What?!

Yes. They don't have access to the NTFS specs. Also, NTFS is a very complex filing system, with many different versions. You don't want to get that wrong. Resizing was a more important goal, and that has been working for many months now.

Of course, it might be included in all distros when completed anyway, due to patents MS hold on the technology.

Re:How about fixing the current filesystems?!

[Anonymous Coward]

Well if you need it sooo bad then why don't you fix it ... just a thought

Re:How about fixing the current filesystems?!

[spaic]

From the Linux-ntfs FAQ

3.8 How was the Linux NTFS Driver written?

Microsoft haven't released any documention about the internals of NTFS, so we
had to reverse engineer the filesystem from scratch. The method was roughly:

1. Look at the volume with a hex editor
2. Perform some operation, e.g. create a file
3. Use the hex editor to look for changes
4. Classify and document the changes
5. Repeat steps 1-4 forever

If this sounds like a lot of work, then you probably understand how hard the
task has been. We now understand pretty much everything about NTFS and we
have documented it for the benefit of others: http://linux-ntfs.sourceforge.net/ntfs/index.html

Actually writing the driver was far simpler than gathering the information.

Re:How about fixing the current filesystems?!

[Anonymous Coward]

You obviously don't know much about open source. Linux is free - I don't have to fix anything. Others will do it for me.

Re:How about fixing the current filesystems?!

[N7DR]

We now understand pretty much everything about NTFS and we have documented it for the benefit of others: http://linux-ntfs.sourceforge.net/ntfs/index.html

Unfortnately, going there gives:

Linux NTFS Project

Page Doesn't Exist

The page you asked for doesn't exist: http://linux-ntfs.sourceforge.net/ntfs/index.html

Re:How about fixing the current filesystems?!

[Anonymous Coward]

The page you asked for doesn't exist: http://linux-ntfs.sourceforge.net/ntfs/index.html

it works for me

Re:How about fixing the current filesystems?!

[irgu]

They don't have access to the NTFS specs.

They don't need anymore. The problem is lack of time and ...

Also, NTFS is a very complex filing system, with many different versions. You don't want to get that wrong.

Very true.

Resizing was a more important goal, and that has been working for many months now.

Agree and based on the ntfsresize FAQ, actually it's just for one year.

Re:How about fixing the current filesystems?!

[Sri Lumpa]

That's because there is a trailing space in the link and their webserver doesn't like http://linux-ntfs.sourceforge.net/ntfs/index.html% 20 (with space) quite as much as http://linux-ntfs.sourceforge.net/ntfs/index.html (without space). The first one give a page not found the second one works.

I Hope it helps.

Re:How about fixing the current filesystems?!

[N7DR]

I Hope it helps.

Yes, thanks. Your sig is great.

that should be a function of the os (or a program)

[ecalkin]

there os should have a 'list' of what's supported by each fs. when you copy a file from fsa to fsb the os (or program) should compare feature and let you know (somehow) that something is not (may not) going to work right. if you copy something from the regular ext2 system that is case sensitive to a ms-dos floppy disk, something should try to remind you. or the program checks this and looks for problems.

remember that not all problems can be detected, so are you willing to live with: 'this may not work correctly' messages?

eric

no, no, no

[awx]

if that crock, that bag-on-the-side, that mess is what we have to look forward to, I think i'll switch to BSD.

I mean, acessing owner data by travelling into a directory then backwards out of it again like: vi /directory/..owner is a big ugly crock.

Re:no, no, no

[Anonymous Coward]

Uhh... Informative?

Whatever.

the file is "..owner" not "../owner" you aren't going backwards out of it.

Re:no, no, no

[IamTheRealMike]

Uh, nobody said that'd be the user interface you use. It's simply an implementation detail.

Re:no, no, no

[awx]

that's not what I said. RTA.

The file is actually /foo/..uid

Re:no, no, no

[TCM]

Backwards out of it again would be /directory/../owner and be indistinguishable from /owner. In this case we simply have a file named ..owner inside a directory named /directory.

Re:no, no, no

[GammaTau]

if that crock, that bag-on-the-side, that mess is what we have to look forward to, I think i'll switch to BSD.

I mean, acessing owner data by travelling into a directory then backwards out of it again like: vi /directory/..owner is a big ugly crock.

Are you so sure that you would hate it? After reading the PDF, I was thinking of two things:

1. He suggests that files ought to be used for everything
2. The old wisdom: "Everything in Unix is a file"

I don't think it's any more radical than treating network sockets as files. Sure, it might feel a little weird first, but once you'd get used to it, the simplicity would overweigh the clumsiness of existing implementations.

It's also very easy to wrap together a shell script that imitates the existing implementations and put it to /bin/chown or whatever you wish to replace.

Re:no, no, no

[TCM]

But the AC is right. /file/..uid is not ascending out of /file/. The name of the metadata file is simply ..uid. Files can start with dots, not to be confused by the files whose whole name is '.' and '..' respectively. But you know this, right? One might get the idea you don't.

Re:no, no, no

[Billly Gates]

I was under the impression that ntfs and VMS's fs do this with ACLs. It is done at the vm and i/o layers of the OS and is hidden from apps unless they use specific ACL api's.

Re:no, no, no

[sporty]

It would also make attribute driven programs a lot more versitile.

Doing a diff of a dir would tell you diff's in uid's, group id's, start dates.. whatever you want.

If you just wanted to get a unique list of owners of files in a dir, you dont' have to do an ls | cut, but now it's a matter of ls'ing all the file's attribute's... or catting them. you get the idea.

Why pdfs?

[brrrrrrt]

I see it happening more and more that people present their summaries, articles and technical papers on the net as pdfs. This is very inconvenient.

Pdfs are nice for printing and publishing on traditional media, because you can be sure they will be in the correct layout etcetera for the printer. But on the web, where people browse between lightweight, easy html-documents, they're just a nuisance.

Please, if you must publish a pdf, publish an html version next to it.

Re:Why pdfs?

[Tyler Eaves]

Most likely LaTeX (Or some other TeX variant) was used to typeset the paper. There are things used in this paper that just can't be done in HTML (For instance, footnotes). Personally, I'd rather see MORE stuff in PDF rather than less. I just find them much easier to read, something to do with how Acrobat's font rendering doesn't suck. Also, PDF's are far superior for anything beyond 1000 words or so.

Re:Why pdfs?

[CodeRed]

You can always use Google to convert the PDF to html :-)

Re:Why pdfs?

[iapetus]

Since when can't footnotes be done in HTML? No reason not to have them at the end of the document, and you can provide anchor links from the text to the footnote and back.

Re:Why pdfs?

[Zak3056]

You've got a point when you're talking about one of those multi-megabyte monsters loaded down with hires images, but, come on, this thing is like 130kb. Even at 56k speeds, it loads quickly.

Re:Why pdfs?

[rikkus-x]

If you're using Konqueror, the pdf just opens in the same window, thanks to kghostview's kpart. Konqy's menus and toolbar also change to reflect the new things you can do.

If you're using Mozilla or IE, a similar thing happens using acrobat reader, without the integration bit.

If you're talking about text-based browsers, I'm not sure what they do, but if I used them, I'd set them up to give me two options: 1. Save to disk. 2. Run through pdf2text and display it.

Having said this, I agree that html is a better idea. I can override the CSS to make the page more readable for myself, when I resize the window word wrap adjusts the lines so they're still readable, etc.

Rik

Re:Why pdfs?

[Tyler Eaves]

Still, it's a kludge, especially if the document runs very long.

Re:Why pdfs?

[Anonymous Coward]

latex2html does a pretty good job of converting LaTeX documents to, umm ... HTML. IIRC it's installed alongside LaTeX in most Linux distros.

Re:Why pdfs?

[xpulsar87x]

for me, it's not the speed of the downloading that matters, its that (under win2k at work with acrobat reader in mz firebird) acrobat reader takes about 6-8 seconds to load before it starts loading the document.. you know, that screen that comes up telling you it's loading. it's totally awful to be going through pages fast then have to sit there and wait for the reader to load when that takes longer than getting what you want to read!

html is just plain faster, no stupid plugins needed for that.

Re:Why pdfs?

[TheRaven64]

Unfortunately, latex2pdf does not actually work very well. It doesn't understand the \href tag used to signify hyperlinks in pdfs (from the hyperref package) and seems to produce documents which have a huge gap between images and their captions.

Re:Why pdfs?

[mst76]

Unfortunately, latex2pdf does not actually work very well. It doesn't understand the \href tag used to signify hyperlinks in pdfs (from the hyperref package) and seems to produce documents which have a huge gap between images and their captions.

Then use pdflatex, which has supported hyperref for years now.

Re:Why pdfs?

[Tyler Eaves]

Can't remember the last time I used that.

pdflatex does things flawlessly.

Re:Why pdfs?

[Tyler Eaves]

Pretty good job?

Well, if you consider the web circa 1992 state of the art...

More importantly, latex2html isn't a true TeX interpreter. As soon as you start making new commands, which personally I do for anything non-trivial, it dies, and dies hard.

Re:Why pdfs?

[Anonymous Coward]

Also, PDF's are far superior for anything beyond 1000 words or so.

That is why, when I read Slashdot comments, I convert the pages to PDF first.

Re:Why pdfs?

[agby]

Don't publish in a PDF. This is what XML and XSLT are all about. You publish the content in XML format and use XSLT transformations to convert it into HTML, PDF, RTF, PPT, whatever. This way you're not bound by any medium, and you have achieved the true separation of content and presentation.

Questions...

[debilo]

I do not know much about file systems, so I have a few questions.

Once we see the GConf example, other possibilities immediately spring to mind. In almost all multimedia formats--such as MP3, MPEG, and OGG--there is a tagging system for storing things such as the author and title. Instead of storing those attributes in the tag--yet another namespace--store it in the file-as-adirectory. I have a file / music/Millencolin/PennybridgePioneers/RightAboutNo w.ogg. I let RightAboutNow.ogg/title be "Right About Now", /artist be "Millencolin". I could add a file RightAboutNow.ogg/genre and put "punk" in it.

While this is a nice example, I wonder if there really is an advantage to this kind of file system, because it seems like it takes more effort to keep track of all those sub-files instead of keeping all the info in a single file. Anyone can shed some light on this?
Also, what if I format my partitions with different file systems, say Reiserfs 4 and ext3, could there be any imcompatibility issues? Imagine I kept mp3's on both partitions, would my mp3 player know how to handle both formats, since the tag info is dealt with differently on both systems?

Re:Questions...

[Anonymous Coward]

The point of these changes is that keeping track of a namespace within a file isn't easier than keeping track of a real filesystem directory, which makes a lot of sense.

Regarding your mp3 player, the obvious solution would be extending the tagging library (there's a few around already, I think) to be able to deal with metadata-as-files mp3s as well. For the mp3 player, there shouldn't be any change.

Re:Questions...

[grumbel]

While this is a nice example, I wonder if there really is an advantage to this kind of file system, because it seems like it takes more effort to keep track of all those sub-files instead of keeping all the info in a single file. Anyone can shed some light on this?

There is no need to keep track of them manually, after you simply handle your mp3 normally and the filesystem will keep track of the metadata. Its actually inverse, with todays filesystems you have to handle metadata yourself, since the filesystem has no way to know which metadata belongs to which files. Sure with mp3, which can keep metadata inside the file itself, it isn't a big problem, but with other fileformats it is. Beside that new filesystems would allow you to simply manipulate the meta data with 'echo', 'cp' and friends, no need to learn a new tool for each and every fileformat.

Also, what if I format my partitions with different file systems, say Reiserfs 4 and ext3, could there be any imcompatibility issues?

Sure there will be a few compability issues, but they should be handleable by the implementation. If your other filesystem doesn't support metadata, no problem, pack it into a seperate .xml document or so or move it into the file itself, if the filetype supports that.

Re:Questions...

[roemcke]

What I would like to see, is something like the Hurd translators, that lets you access file content through the FS-API. For instance mp3 tags would be accessed through the files "HappyBirthday.mp3/title" or "HappyBirthday.mp3/artist", but instead of storing the tags in the FS as metadata, the translator store the tags in the mp3-file using the normal mp3 tagformat. This would combine the best feaures of both worlds. Other natural uses of the translator would be to access tar or xml files.

Translators should not be restricted to mappings from flat file to director structure, but should allso allow for mappings like dir->file, dir->dir or file->file

The mapping dir->file would make it possible to implent different access right for different parts of a file.

Re:Questions...

[spitzak]

Absolutly exactly what you suggest should be supported.

The idea may be clearer if you think of the metadata as a "cache" of data extracted from the file. In the ideal situation deleting all the metadata would just make extracting it a bit slower, but you would get it all back.

Security?

[Inode Jones]

Before adopting any of these ideas, one must consider the security implications of doing so.

If we assume that the filesystem is decoupled from the access control layer in the kernel, then one must ensure that any operation that potentially affects security is adequately controlled.

For example, on systems with POSIX_RESTRICTED_CHOWN, the following ought to be illegal:

cp foo/..uid bar/..uid

This can be accomplished by making the UIDs mode 444. Without POSIX_RESTRICTED_CHOWN, the UID is 644. However, we have now moved a systemwide security feature into the filesystem. If multiple filesystems are configured into one kernel, then they ought to be consistent; otherwise the security model will be flawed.

As for things such as allowing access to an environment, doesn't that break encapsulation? It means for a certain filename, the filesystem must grovel through a user-space process to find the environment. Also, if a change in some external environment immediately affects some partially-related processes (e.g. daemons started from that shell), then a whole new raft of security holes will come up based on a process' environment or filesystem layout changing unexpectedly.

Cool ideas, but let's be careful lest we make a steaming pile of Swiss cheese.

Re:Security?

[iantri]

ironically it gets even worse now that people use ssh and ssh-keys to log in. it makes the network less not more secure. its simpler to attack a client with remote mounted home directory using ssh keys than it is to sniff for telnet passwords!
This is getting off-topic, but I'd like to know more about this. I was under the impression that ssh keys were more secure than just using a password.

Re:Security?

[m_frankie_h]

What about AFS?

Re:Security?

[quantum bit]

What about AFS?

Not as universally cross-platform as NFS is. OpenAFS is moving in that direction, but it's still difficult-to-impossible to run AFS on some platforms (AFS server on BSD for example).

There's also some cruft in the authentication tied to kerberos IV that desparately needs to go away.

Re:Security?

[Anonymous Coward]

You secure your network outlets with 802.1X don't you ? And you disable non used ports on the switches don't you ? And you monitor your network for unsuspected traffic don't you ? You log failed authorisation attempt and have triggers to sound an alarm don't you ? etc. etc. etc. security is hard work but can be made quite reliable and workable.

Re:Security?

[m_frankie_h]

Arla should run on BSD (at least on FreeBSD).

Re:Security?

[Anonymous Coward]

You are sadly deluded. If you mount your home directory remotely and I can sit at your desk or a neighbors for a few minutes, I can get into your computer. All of the counter measures you listed above would not work, there would be no failed log attempts etc...

Re:Security?

[Anonymous Coward]

Check out SHFS

databases need transactions?

[ajs]

I'm really getting tired of the ever-creeping assertion that transactions are required for [x]. At first x was ACID-compliant relational databases, and such was true because ACID was defined as such. However, then I started to see assertions that relational databases had to be ACID-compliant (mostly from the anti-MySQL camps who were ignoring the long history of highly valuable, non-ACID relational databases).

Now, in this article, I see the assertion that databases in general require transactions, and thus cannot be supported by a filesystem.

Worse, the logic is self-refuting, as the article previously states that a filesystem is a database, just a limited one. As it happens, POSIX-type filesystems are quite powerful, and let's not kid ourselves into thinking that they have not served us well for 20-30 years! Yes, changes are coming and I'm frankly quite impressed by Hans Reiser's accomplishment in finally coming up with a balanced-tree-based filesystem. Many have tried and failed where he succeeded.

That's because his was a great step forward, not because the old UNIX filesystems weren't also. Let's stop trying to re-define terms so that we can explain why the last 20 years were the dark-ages. They simply were not.

Re:databases need transactions?

[afidel]

Transactions are required for Reliable databases and filesystems. If you don't mind occassional corruption then you can throw out transactions, otherwise you need them and you need to eat the cost (memory, access speed, cpu overhead, whatever). Since PC's are generally faster then most people need at the moment making them more reliable seems like a worthwhile goal.

Re:databases need transactions?

[Larsing]

Let's stop trying to re-define terms so that we can explain why the last 20 years were the dark-ages. They simply were not.

Funny thing you should say that. As a matter of fact, neither were the Dark Ages...

Re:databases need transactions?

[zatz]

It depends entirely on the sort of updates you need to make. Transactions are a much heavier solution to concurrency than locking or simply storing things in a way which facilitates parallel updates (Maildir instead of mbox, for example). The value of putting all the necessary support in the operating system remains unproven. The main "transactions" required in a filesystem are adequately supported by open with O_EXCL, write with O_APPEND, rename, and mkdir. (Or don't you remember the days when mkdir was a setuid program?)

Re:databases need transactions?

[ajs]

Transactions are required for Reliable databases and filesystems. If you don't mind occassional corruption

No, you are incorrect. You're thinking of atomicity, pretty much a requirement for all filesystem and database models (though not *all*, some are actually designed, and designed well around the limitation of non-atomic operations for certain real-time applications).

Transactions are only for keeping multiple atomic operations in sync, and you can always simulate transactions at a higher level if you have an atomic test-and-set, or a locking mechanism build on same.

Re:databases need transactions?

[ajs]

True enough, they were pretty warm in places ;-)

I think the "dark ages" are refered to that way because it was a period of decline following the hieghts of technology and social advances that the Roman Empire had brought.

In fact many features of the Roman Empire (e.g. bathing) were shunned simply on the basis of having been associated with Rome and thus, later with the fall.

Re:databases need transactions?

[Salamander]

Transactions are not the only possible model for reliable systems. They're just what some people are used to, and if all you have is a hammer everything looks like a nail. Those people need to enlarge their toolboxes, not make the world conform to the tools they've learned.

Re:databases need transactions?

[joeldg]

I could not be in *more* agreement with you.

I have helped build a large company with MySql and I get tired of the constant drone of people who have read a few articles in magazines state with utter and complete belief that a database "needs transactions"

I am of the personal camp that like a gun used in warfare, the fewest moving parts needed to get the job done 'exactly as you need it' the better. If I wanted floof I would go and spend money on licensing something that I will never have any true need of using most of it's "features" that in reality are there only because a middle manager read an article somewhere and picked up some jargon.

As for a filesystem, I think the usual idea is speed, so taking our previous example, why would you want to introduce another layer into the mix which would serve no true purpose other than slowing the thing down?

Cheers on your post, that made my morning.

Re:databases need transactions?

[ajs]

Thanks for your points.

One thing about filesystems, Hans Reiser has definitely earned the credibility to suggest things like adding transactional support to a filesystem, and not get laughed at (where most would deserve the laughter). If he can make a balanced tree work efficiently for a filesystem, perhaps he can come up with some way to make transactions only cost those who use them... we shall see.

Re:databases need transactions?

[nosferatu-man]

The Unix filesystem was a great step forward? In what ways, exactly? My first exposure to Unix (in '88) left me pretty cold, filesystem-wise -- VMS certainly had a more advanced system that either pre-dated or was at the very least coincident with the FFS. HFS+ was arguably better than the Unix offerings when it was released. Same with NTFS. Note that "better" can't mean "better at preserving the POSIX model", because that's a circular argument.

This is not to say that current Unix filesystems haven't been improved -- XFS for one is a marvelous technical achievement. But insisting that Unix filesystem semantics were an improvement over the state of the art? I find that hard to swallow. And I applaud Hans Reiser for not being conceptually trapped in the kludgy world of modern Unix filesystems.

'jfb

Re:databases need transactions?

[uhoreg]

Now, in this article, I see the assertion that databases in general require transactions, and thus cannot be supported by a filesystem.

I don't know where that came from. Reiser4 will have support for transactions. (at the top of http://namesys.com/v4/v4.html) I think that Reiser4 will be ACID-compliant, but I'm not a database guy. Of course, using transactions requires the use of some new system calls.

Re:databases need transactions?

[ajs]

The Unix filesystem was a great step forward? In what ways, exactly? My first exposure to Unix (in '88) left me pretty cold, filesystem-wise -- VMS certainly had a more advanced system that either pre-dated or was at the very least coincident with the FFS.

The unix slash-as-root, slash-as-separator, unified root, everything-is-a-file model was indeed a step forward for its time (somewhere in the 70s, though honestly, I'm not sure if it was around the dawn of UNIX which would put it at 69-70 or if the FS came later in the mid-70s).

There were other systems which made other useful contributions. Among them are Apollo's Aegis, Digital Equipment Corporation's VMS and many others. Some of those features of other systems are still not present in today's modern filesystems. This sort of parallel and often disjoint filesystem development is a hallmark of innovative OS design, and sometimes it results in losing usful advances for a geration (of OSes) or so.

That doesn't mean Unix didn't have an innovative filesystem.

The rest of the fileystems you note came at least a decade after the Unix filesystem, so I won't touch on those. They too had some useful contributions, but aren't really pertinent to what I was saying.

Re:that should be a function of the os (or a progr

[Anonymous Coward]

so in other words, no one will ever use it? great, next story please.

Reasons to change to another FS

[N_gaAdy]

I agree that we need a revolution in how filesystems work inside an operating system, but it seems that the arguments placed in this paper had alot of holes.

For one thing, the need for changing a filesystem should not really be solely concerned on space or metadata. I think security, speed of data retrieval, and self correcting error engines should be centered on the new systems.

The reason for the speed of data retrieval as being more important than data size is because hardrives are getting much bigger than they are faster. In five years, we may have 20 terabyte drives, but the access speeds will still be horrible.

Security and error correction are obvious points that should be implemented on a systemwide level. When these features are system wide, then management becomes much easier for all system users.

Re:Reasons to change to another FS

[Webmonger]

None of the reasons you give (speed, security, error correction) are reaons to change systems. Error correction is handled by media-- that is, hard disks-- these days. (See that article "You Don't Know Jack about Hard Disks" from a day or so ago.)

Speed of data retrieval is only partly under a filesystem's control. Access to the actual drive is controlled by the kernel, which is why they've done this anticipatory scheduler for Linux 2.6. And speed can be improved by improving a filesystem's implementation. So filesystem speed is not a trump card. It may factor in choice of filesystem, but it's not an automatic reason to jump ship.

It's not clear what you mean by security. If you mean ACLs and stuff, they're already here. (And the stuff in the article shows how ACLs will be implemented on Reiser.) If you mean data encryption, that's here too. And it's never been the case that users could evesdrop on other users' data retrival.

The things they talk about in the article are things that can't be achieved with other filesystems, not things that can be improved in filesystems. Different kind of article.

I just want

[indros]

a read/write implementation of BeFS for linux!

Re:I just want

[Hodge]

I kept reading the article thinking - I had almost all of those metadata features in BeOS. The two most powerful things there were, a journalled file system and, file system custom attributes.

I read about the possible implementation of P9 mail within the filesystem, presumably recognisable by a range of clients. This was also present in BeFS. It's always annoying to see things presented as great innovations when they have been implemented before albeit in a platform that is currently crashing and burning!

Data, even metadata, belongs in files, not fs

[Fastball]

I agree that metadata in the filesystem is a risky proposition. Just on general principle, I prefer my data inside the file and not left with the filesystem. The MP3 metadata example, to me, is like Windows file extensions on HGH. I remember John Dvorak wrote a piece about Windows file extensions a long while back, and he argued that file types, etc. should be inside the file. A header of sorts. I tended to agree then, and I see filesystem metadata as a bad trend.

Re:Data, even metadata, belongs in files, not fs

[Anonymous Coward]

But there are major advantages to putting metadata in the filesystem...You can do searches across all applications...eg., you can look for all files with "author=Fastball", and find them all whether they're mp3s, jpegs, or emails. You can do this even if the file types don't have any provision for the metadata you want. You can make up any kind of metadata, apply it to any kind of file, and search all your files based on the metadata.

With the appropriate software support (in the filesystem api and browser), this seems like a major advance in usability to me. Certainly people liked the BeOS filesystem. Since Microsoft is claiming to have something like this in the works, it's probably necessary on the Linux side just to keep up...but my guess that ReiserFS will turn out to be cooler than Longhorn.

Re:Data, even metadata, belongs in files, not fs

[sketerpot]

I remember the way the MacOS 9 filesystem did things: it had some metadata somewhere telling what type of file you were dealing with, but it wasn't done very well. It said what program made it, which could be annoying when you open a JPEG file and it says that it couldn't find "Photoshop". Lesson: use MIME types. image/jpeg

Re:Data, even metadata, belongs in files, not fs

[Anonymous Coward]

I'm no super-brain, but I suspect that if we stop to consider the use of the meta-data, we'll find the more appropriate location for it..

One of the primary things that it'll be used for is searching.. either for file names, types, dates or additional 'meta' descriptive information.

Building that 'into' the file system (and hence, O/S, kinda) will give us better search tools, I think..

Re:Data, even metadata, belongs in files, not fs

[hankaholic]

While there are problems to be solved (backup, for instance), there are many benefits as well.

Many detractors of the UNIX security model point to the lack of ACLs and fine-grained security.

The U.S. DoD has contracted Namesys (the ReiserFS guys, led by Hans Reiser) to develop a filesystem upon which security can be build. Reiser's vision is a filesystem which allows users of the filesystem to define what security means in an extensible manner, with plugins.

The future of ReiserFS includes a plugin architecture which makes it easy to implement NT permissions, for instance, without breaking existing programs or requiring new semantics which wouldn't work across, say, NFS.

I'm not sure why the author chose to throw Plan 9 into the mix. Plan 9 has some interesting features, but I don't feel that either Plan 9 or ReiserFS was given sufficient attention to allow the reader to understand just _why_ such things are interesting.

I tried to sum up some interesting Plan 9isms here, but I'd rather not go into it -- reexporting modified views of the filesystems is a complicated thing, and it's hard to justify such complexity in limited space.

As far as ReiserFS goes, the current and future benefits are many.

First, speed is king, and ReiserFS takes the crown. Anyone who has waited more than 10 seconds for "ls /usr/share/doc" on ext2 can attest to the fact that ext2 craps out on large directories. Reiser V3 does quite well, and V4 is rumored to do even better.

Second, space efficiency is nice, and ReiserFS does it better than anything out there. While storage sizes are increasing dramatically, it "feels" wrong to waste a whole block on a trivial file (only a few bytes long, for instance). With ReiserFS, developers don't need to waste time trying to work around the need for efficient small file access -- it's efficient to have many small files, and it's efficient to have many of them in one directory if needed.

I believe it's been said that in the future, ReiserFS may compress the filenames in some way to try to eliminate even that overhead.

ReiserFS V4 implements wandering journals, and support for transactions. What does that mean to the layperson? It means that in the event of a crash, an application which handles important data (for instance, an online purchasing system) can promise that in the event of a crash, a group of related filesystem operations can be guaranteed either to have all completed, or have all failed.

For instance, a purchasing system makes a note to charge the person's credit card, and to ship the items that were ordered. You can tell ReiserFS programmatically to "create a file in /var/orders/ and place this text in it, as well as a file in /var/ccbilling with the customer's credit info in it, and if the system crashes halfway through, throw it all away, because I don't want to charge for products I'm not shipping and I don't want to ship products for which I'm not billing."

If the system crashes, you don't have to work to make sure that you have a one-to-one mapping of orders to sets of billing information -- ReiserFS can guarantee you that either all of it got recorded, or none of it got recorded.

So now you've got a fast, reliable filesystem. I'll even let you ignore the fact that ReiserFS will let you implement per-file compression and encryption plugins. Still not impressed? Wondering what all of this crap about namespaces is about?

The author of this article basically ripped off Hans Reiser's examples from his V4 draft document, such as the /etc/passwd thing. I'm not sure where he got the idea of subusers and I don't much care, as it's not really relevant to the namespace thing.

What is interesting, but not really mentioned, is that file filters can become part of the filesystem.

The specific implementation details in this example are products of my imagination,

Re:Data, even metadata, belongs in files, not fs

[simm_s]

Although I agree that application metadata handled by the filesystem leads to complexity which leads to bugs, I think most people misread the article was saying about metadata handling. In the MP3 example I don't think the author was saying that the MP3 tag metadata should be "stored" in the filesystem, rather the filesystem should abstract the MP3 tag metadata as a filesystem object. When you make modifications to the filesystem objects representing the MP3 tag it will update the MP3 file itself. Transferring the MP3 with its metadata should be seamless with this approach.

I think this looks good on the surface, but this is a solution to a non-problem (which usually causes problems). If I want to update the tag information all I need to do is load XMMS and change the tag features.

Re:Data, even metadata, belongs in files, not fs

[tetsuji]

The approach of storing metadata in the files works fine when you don't have to deal with issues of backwards-compatibility and extensible metadata standards.

I deal a lot with archival systems for satellite data. Massive number of data files are produced that must have metadata associated with them describing the state of the satellite at the time of data collection so that data values can be recomputed in the future if we discover that the calibration of the instruments was inexact. The data formats of the files that are coming from the satellites are not something we have control over, but we have to produce metadata that is applicable to them and somehow attach it to the files without imparing the ability of existing applications to read them. The current solutions involve horrible hacks involving metadata databases with references to filenames and directory structures. It would be much more efficient to simply have the metadata stored as an attribute of the file.

Re:Data, even metadata, belongs in files, not fs

[Eraser_]

The one thing I see with plugins like the ms-word plugin is, while cool, seems like it would load the kernel down doing things best left to userspace. How seperate is that ms-word(or anything, acrobat, html, whatever) from the rest of kernel land, where things can go VeryWrong if something fails?

If a buffer over or underflows in the FS plugin is the kernel going to panic, or simply segfault that FS module, core it, and move on? I'm just worried about my system going down because of a mostly-tested plugin.

A cool kernel land example I can see for this is like cat /dev/dc0/ip or /dev/eth0/subnet (whatever your ethernet iface is) because these could be dynamically updated as life goes on for the ethernet interface. (Your server has been up long enough to suffer a complete network overhaul right? :)

Re:Data, even metadata, belongs in files, not fs

[Marqis]

I think you're missing the main point. The author doesn't want you to have to load XMMS (or any other specific file application combo). You should just be able to use vi, emacs or any other text editor to do the same thing.

I think another point people are missing with the whole potential transferring problem is this: for some file types such has MP3 which have established in-file metadata, you could just inject the songname/title contents into the mp3 stream at the appropriate spot to get a complete MP3 at the other side without even the overhead of copying/creating a new file or tarball to send over.

Re:Data, even metadata, belongs in files, not fs

[Arandir]

Many detractors of the UNIX security model point to the lack of ACLs and fine-grained security.

You don't throw away the door just because it doesn't have a lock. You simply put a lock on it!

FreeBSD now has ACLs, and it did it without throwing away UFS. It didn't need to replace the "everything is a file" model to do it. It just expanded the available extended attributes. To get ACLs you will certainly need to extend the filesystem, but you don't need to replace it completely.

p.s. FreeBSD also got soft updates without throwing away UFS. In short, you don't need to throw away the old to get the new, something a lot of developers don't seem to understand.

Re:Data, even metadata, belongs in files, not fs

[Webmonger]

As the poster noted, it's possible to run your filters in user space, not kernel space, so buffer overflows and segfaults should be sources of irritation, not sources of vulnerabilities.

Re:Data, even metadata, belongs in files, not fs

[Eraser_]

Ah ok, just checking, I read most, but certinaly not all of the posters comments, as it started reading like a sales pitch IIRC, the examples made my eyes glaze over while the technical stuff held my interest hehe.

The other thing is I tend to use a monolithic kernel , hard to install modules in a kernel which doesn't support it.

Re:Data, even metadata, belongs in files, not fs

[ckaminski]

Yes, but this isn't throwing anything away. It's just using the "files-as-directories" paradigm to add new stuff.

Hence the whole point of implementing this. It will work with everything that uses the OLD style of doing things, while giving us room to grow and expand and enhance.

More important, it extends the "everything-as-a-file" paradigm to meta-data, something that we have never had before. What with procfd and devfs and the like, this is only a good thing.

Ideally, this would be implemented above the filesystems themselves, in the VFS layer, such that *ALL* filesystems could take advantage of the features, and the feature could be controlled via a mount option. Unfortunately, the performance penalty might be too high since the meta-data is so small, and will need to be platform/filesystem agnostic.

I can conceive of a few ways to make it this work, although I'm pretty sure most (if not all) of them will not be high-performance.

Re:Data, even metadata, belongs in files, not fs

[Trepalium]

Not all is rosy in the world of ReiserFS, however. I remember using it on my laptop, and was constantly frustrated with it. It would last about 3 to 4 months, and then I'd lose everything on the filesystem, or get subtle corruption like undeleteable files, or read-only files. I later found out that ReiserFS has some serious problems with low space conditions, but only after I'd lost my data twice. From there, I went straight to ext3, and haven't regretted it. It may not have the speed, efficiency, or fancy features of Reiser, but it has all the data integrity I'd come to expect from a file system.

There's a lot to be said about the simplistic approach of UFS or ext3. The metadata is simple, and easier to repair or reconstruct when something bad happens. Both UFS and ext3 have shown the ability to add new features without either breaking backwards compatibility or by implementing horribly complex structures that are prone to subtle breakage. For example, FreeBSD's UFS implements ACLs using their EA infrastructure. Linux's ext3 implements journalling the same as any other file on the disk.

Re:Data, even metadata, belongs in files, not fs

[mrogers]

Reiser4 doesn't throw away the "everything is a file" model. In fact, it extends it: an ACL can be a file, a modification time can be a file, a database field can be a file. Reiser4 is making the Unix filesystem into what it was always meant to be: a single, unified namespace for storing any kind of data, so that any kind of data can be manipulated by the same simple tools.

Re:Data, even metadata, belongs in files, not fs

[Arandir]

a single, unified namespace for storing any kind of data

That's not what the Unix filesystem was meant to be. Whoever told you that. "everything is a file" does not mean "unified namespace".

And of course, as with everything, taking stuff to the extreme is extremist. You don't need to make the modification time a part of the file, because it ALREADY is part of the file. And why make a database field a file, when it already exists in the database, which itself is a file?

Hey, let's take this to the logical conclusion! Let's have hardlink between modification time metadata files. And the modification time for the modification time file could be a file as well!

Re:Data, even metadata, belongs in files, not fs

[mrogers]

And why make a database field a file, when it already exists in the database, which itself is a file?

Well, I think the idea is that the database wouldn't be a file. ;-) The database would be a collection of files, moving as much structure as possible into the filesystem, so you could use the same interface for any strutured data storage/retrieval task.

Hey, let's take this to the logical conclusion! Let's have hardlink between modification time metadata files. And the modification time for the modification time file could be a file as well!

I must admit I wondered about that too. Where are the read/write/execute permissions for foo/..rwx stored? In foo/..rwx/..rwx?

Re:Data, even metadata, belongs in files, not fs

[wagemonkey]

Hey, let's take this to the logical conclusion! Let's have hardlink between modification time metadata files. And the modification time for the modification time file could be a file as well!
I must admit I wondered about that too. Where are the read/write/execute permissions for foo/..rwx stored? In foo/..rwx/..rwx?

Err, the modification time for the modification time file is store in the modification time file itself, think about it.
And I can't think why the permissions for the permissions 'metadata' file (and all other 'metadata' files as well as the 'real' file)wouldn't be stored in that file too - I know the OS has to read that file to decide if the user has permission to read it, but surely that's what we have now - I don't see a larger security risk than the current situation.

Re:Data, even metadata, belongs in files, not fs

[hankaholic]

Well, as I mentioned, plugins can be written in userspace, so it wouldn't be bogging down the kernel, necessarily.

I know that Hans has said in his V4 docs that his proof-of-concept security plugin, which does something like denying read access to any file containing the word "topsecret", is a Perl script.

I would seriously trust the ReiserFS team to Do The Right Thing on this one. They're funded by the US DoD, and their goal is to write a system on which security can be built in a modular fashion. If they can find a way to allow userspace security plugins, I'm sure a few file filters should be no problem at all.

Of course, we'll all just have to wait and see... but the V4 design docs are a really interesting read, and contain mentions of future features. Check out http://www.namesys.com/v4/v4.html for more.

As far as the /dev/eth0 stuff, I vaguely recall something called "kernfs" for the 2.5/2.6 series. Apparently kernel configuration options will be presented in a filesystem-ish interface. How this differs from the stuff in /proc, I don't know, but this may already expose the sort of functionality you're looking for.

Re:Data, even metadata, belongs in files, not fs

[hankaholic]

ReiserFS isn't trying to replace the "everything is a file" model.

From the horse's mouth, at the V4 design pages:

In Gnu/Linux we have files, directories, and attributes. In NTFS they also have streams. Since Samba is important to Gnu/Linux, there frequently are requests that we add streams to ReiserFS. There are also requests that we add more and more different kinds of attributes using more and more different APIs. Can we do everything that can be done with {files, directories, attributes, streams} using just {files, directories}? I say yes--if we make files and directories more powerful and flexible.

When you can explain to the ext2 developers (and the UFS guys as well) how to get the benefits of ReiserFS' dancing tree algorithms, don't post to Slashdot, go directly to the developers.

Until you can present a solution which allows UFS to cope QUICKLY with 50,000 files in a single directory, I'm in doubt.

The point is, with ReiserFS in place, to get ACLs you don't have to modify the on-disk representation. You don't just "[expand] the available extended attributes", because that requires modification of the on-disk structures for anything beyond trivial expansion.

ReiserFS doesn't replace the "everything is a file" model, as you've asserted. It extends that very same model to allow a security researcher to implement ACLs, extended attributes, etc., without modifying the on-disk structures one bit, since all you're really doing is editing/creating/deleting files, not mucking with security-model-specific attributes and ACLs.

You mean FreeBSD built ACL support into a slow filesystem in a way that didn't allow for future research and expansion of security models? I'm shocked. A modular filesystem which treats everything as files (and doesn't require modification of userspace tools to use -- I'll bet FreeBSD has some ACL maintenance commands) is a benefit a lot of filesystem developers don't seem to understand.

Re:Data, even metadata, belongs in files, not fs

[hankaholic]

How long ago was this? I followed the ReiserFS development list quite closely just before adoption into the mainstream kernel, and I'd say at least 85% of the bug reports were actually a result of buggy hardware.

If you could show a ReiserFS bug to the developers, they were VERY interested, and Hans would often assign a developer to figure out what went wrong.

SuSE uses ReiserFS by default, and has funded ReiserFS development.

There's a lot to be said about the simplistic approach of FAT16. The file allocation table is simple, and easier to repair or reconstruct when something bad happens. The FAT scheme has shown the ability to add new features without either breaking backwards compatibility or by implementing horribly complex structures that are prone to subtle breakage.

That aside, in the future, ReiserFS won't implement ACLs. ReiserFS probably won't implement UNIX permissions either. All a filesystem needs to do is store files, and if some of those files happen to represent metadata, and a given plugin decides to use those files which represent metadata to implement UNIX permissions, or ACLs, so be it.

ReiserFS' goal is to be damned fast in ALL cases, and to offer a plugin architecture that allows you do to the rest. It's damned stable, or SuSE wouldn't ship it as default. If you had a problem back when you had to patch the kernel to get support at all, I'm sorry, but my comment was in response to an article which some guy wrote which tried to address why namespaces will be important to the future of operating systems.

Why are people so reluctant to try a new filesystem? You had to try ext[23] in order to try Linux on its own partition, or UFS to run FreeBSD, didn't you?

Re:Data, even metadata, belongs in files, not fs

[Arandir]

When you can explain to the ext2 developers (and the UFS guys as well) how to get the benefits of ReiserFS' dancing tree algorithms, don't post to Slashdot, go directly to the developers.

I'm still waiting for someone to explain to me the benefits of this algorithm. I'm sure they're there, and some people will need them, but I still don't understand how it benefits me in the way I use the filesystem, as compared to other solutions providing similar benefits.

Until you can present a solution which allows UFS to cope QUICKLY with 50,000 files in a single directory, I'm in doubt.

Most people aren't going to have 50,000 files in a single directory, just like most people don't drive Ferraris. Until you can explain to me how a Ferrari is going to benefit my daily commute to work, I'll stick with my slow pokey Dodge. Ditto for reiserfs. If I was using Linux, I would by all means prefer reiserfs over ext2, but that's only because ext2 is totally inadequate for my needs.

And when the time comes that I need to QUICKLY access 50,000 files, I'll use a database instead of a filesystem.

You mean FreeBSD built ACL support into a slow filesystem in a way that didn't allow for future research and expansion of security models? I'm shocked.

Take a closer look at it. It allows all the future extended attributes you could ever need. Since they're named attributes, it's very flexible. But like reiserfs, EAs are stored in a separate file (shock). This is a wart. It's not the "correct" implementation in the same way that UNIX itself was not correct but Multics was, and the Linux monolithic kernel was not correct but the Hurd microkernel was. ReiserFS is trying to be the "correct" solution, but 95% of the effort put into reiserfs is only going to benefit 5% of the users.

I'll bet FreeBSD has some ACL maintenance commands

Since chmod/chown/chrgrp can't handle ACLs, or any other extended attribute, of course it does!

Re:Data, even metadata, belongs in files, not fs

[hankaholic]

Hmm...

This is sort of a sidenote, but the evantual goal is to allow ReiserFS to support security attributes without needed specialized userspace tools or kernel interfaces -- all can be done with ln, cat, and shell redirection, although I'm sure nifty scripts will show up here and there.

The ReiserFS algorithms allow nifty things like atomic actions. For example, an application will be able to check for /tmp/filenameXXX and create and open it as a single transaction, and ReiserFS will guarantee that no race condition exists.

ReiserFS doesn't waste a whole block on tiny files, or on the tails of larger files, so users gain space-efficiency (yes, I know hard drives are cheap, but they're still not free).

Filesystem plugins will allow nifty things like per-file encryption and compression.

Other than speed, space-efficiency, and the fact that V4 will make all filesystem operations (not just metadata-related operations) atomic, I really can't say that ReiserFS can do for you that ext2 (or similar filesystems) can't, at present.

I've posted the link to the V4 page at www.namesys.com, browse over there if you're really interested in better descriptions of features ;)

Honestly, last I used FreeBSD was before softupdates and other UFS snazziness, so this is an unfair comparison, but I do recall FreeBSD taking tens of minutes to create an empty cache when I installed Squid from ports (on relatively expensive SCSI hardware), when ReiserFS on my crappy 5400 RPM IDE drive running in a system with 1/3 the clock speed took a matter of seconds to create an equivalent filesystem heirarchy on Linux.

Speed, space efficiency, crash safety. That's why ReiserFS is good today.

Why will ReiserFS be the filesystem of choice five years from now? Go to www.namesys.com and read for yourself.

Re:Data, even metadata, belongs in files, not fs

[Arandir]

Try out UFS2 with softupdates. Unpacking the full ports tree used to take ten minutes. Now it takes one.

Speed, space efficiency, crash safety. That's why ReiserFS is good today.

But what about tomorrow? What about all those gee-whiz featurettes that article was talking about? Will those gain me any real benefits? And will it ever be ported to anything besides the Linux? A filesystem that will only work with one kernel is as bad as a kernel that will only work with one CPU.

Re:Data, even metadata, belongs in files, not fs

[simm_s]

My point is where does the abstaction end? Should "catting" the mp3 output the data to my soundcard or send the data to stdout? Call me old fashioned but the point of the filesystem is to store files in an efficient manner and the point of XMMS is to manage my mp3s.

If you told me the shell or gui was able to modify mp3 tag information, I would have no issue. If the filesystem layer was handling it, then I have a problem with it because it adds complexity. If the mp3 information was an "extended attribute" to the file, I would not have a problem with it. I don't think the author meant that the solution was an extended attribute, but the filesystem actually editing the mp3 tag information itself. I could be wrong about this.

Re:Data, even metadata, belongs in files, not fs

[hankaholic]

The article really wasn't well-written, and the features discussed were mostly taken out of context from the original document where they were simply meant to be quick-and-dirty examples.

I'm glad FreeBSD has a reasonably fast filesystem -- it was the one thing that really bothered me during normal usage.

The major "real benefits" of the future are first and foremost security-related. I'm sure ACLs will be one of the first implemented features. Samba fans will likely see a security plugin which implements NT permissions.

Other plugins will be of less concrete benefit, and will likely vary from "nice idea, but what's the point?" to "can't live without it". For example, accessing a Word document's text with cat sample.doc/text. ReiserFS will allow users to use file-based operations on things which aren't generally files. I anticipate many plugins which, while doing nothing which can't be done with a chain of pipes on the command-line, still serve to increase the perceived expressiveness of the system.

Are these filter plugins a "killer feature"? No, but Hans Reiser is betting that plugins will allow developers to evolve a system which "just makes sense".

Another cool thing is filesystem links in which the targets can be "aware" of the links to them. I'm not sure about implementation of this one, but it's a "gee-whiz featurette" that definitely made me jealous as a Win95 user looking at OS/2. ;)

The major benefit will be extensibility. The overall system will gain usability as a result of increased expressiveness, and hopefully we'll see more flexible security features as a result as well.

As far as a FreeBSD port, or a port to other operating systems, I can't really say, and it's a matter which has me curious as well.

Do you know whether it will pose a major issue that the code is GPL'd in terms of FreeBSD adoption? I'd tend to doubt that the BSD guys would bless GPL code in terms of mainstream integration into the BSD kernel, and unfortunately I'm getting yelled at by my girlfriend for not being in bed right now, or I'd look more into it ;)

Thanks in advance for any responses.

Re:Data, even metadata, belongs in files, not fs

[Arandir]

Do you know whether it will pose a major issue that the code is GPL'd in terms of FreeBSD adoption? I'd tend to doubt that the BSD guys would bless GPL code in terms of mainstream integration into the BSD kernel

The GPL would keep it from being a required component. But there's no reason it can't be an optional kernel module. The only requirement would be that the license not "taint" any other code in the kernel.

Re:Data, even metadata, belongs in files, not fs

[Trepalium]

I was using back in the late 2.2 series and early 2.4 series kernels. I think I stopped using it around 2.4.12 or so. Part of the problem with trying to report bugs is when there's a three or more month period between corruptions, it's hard to track down exactly where something went wrong. Was it 2.4.n, or 2.4.n+1, or a bad shutdown sometime in July, or a disk cache that didn't get committed before shutdown two months ago? I knew that I wasn't the only one experiencing that problem. When this first happened, I don't know exactly when the problem began, but when I tried to update a package, and /var/lib/dpkg/status could not be updated, my system had already become damaged (any idea how unfun it is to have a damaged dpkg status file is?)

In regards to reporting bugs, sure it's easy to do on a desktop system, but on a laptop, you can just about forget it. Just try to find a bootable disk with pcmcia drivers and all the smbfs programs/drivers needed. On a desktop system, simply attaching a new drive and doing a quick install on that drive may be all that's needed to pull the required metadata to report the bug.

ext[23] likely had problems back when Linux was in it's infancy, but it's matured quite far, and is rock solid these days. Something very bad has to happen to it for the data to be unrecoverable, and even then, there's still a good chance that you can use debugfs to work with the file system on a lower level to get back some of your more important data. Despite the fact that reiserfs is completely open source software, it's still rather like a black box, in that very few people truely understand it's inner workings, and finding out where things went wrong is difficult at best. When recovering from a disaster, the biggest benefit to ext[23] is that's it's predictable, and that predictability lends itself well to automated tools for scraping data off the disk.

Re:Data, even metadata, belongs in files, not fs

[hankaholic]

I do know how much it can suck to reconstruct Debian's bookkeeping files -- I somehow removed much of /var (or all of it, I don't quite recall) on a router to which I don't have physical access.

I'm happy to say that the router is still up and running, and /var is completely fine.

Many (many!) 2.4 kernels up to and including 2.4.16 had nasty bugs which could affect any filesystem code, including ReiserFS. I recall having problems with 2.4.9 which led to me tarring up the home partition and remaking the filesystem, which I agree isn't the best solution.

Unfortunately, the only real judge of stability out there for a kernel is to have many, many people use it for things and see if it breaks. Until a serious group of people decide to start auditing the kernel OpenBSD-style, that's the only real metric out there.

I do agree that having good tools which can help to repair a filesystem can lead to peace of mind, even if you never end up needing them.

As a sidenote, I don't recall specifically what was needed to restore the dpkg stuff, but I do remember that dpkg --[gs]et-selections was a big help, in combination with the fact that most (hopefully, all) packages create a corresponding directory in /usr/share/doc.

What I probably did was tell dpkg that every directory in /usr/share/doc named a package that was installed, then get a complete list of every file included in the dpkg -L output of every installed package, sort it, then diff against "find / | sort" output to find files which were owned by packages but not yet claimed by a package.

don't know much about them?

[MrFredBloggs]

"Have you heard of Plan 9 or Reiser4 but don't know much about them?"

Then STFW!!!

Sorry, but

[Morth]

This article seems to just be the author brainstorming or feeling excited about reiserfs. It's hardly a "summary of developments in the filesystem". Now if he was asking about opinions on his article it'd be fine, but he's not, so I'll just discard this as another non-news.

Re:Sorry, but

[Skeme]

None of these advances has to be Reiserfs specific, it's just that they are the few working on them. It's a summary of these developments, so giving examples of some specific scenarios is a good thing to write about.

Now if he was asking about opinions on his article it'd be fine, but he's not, so I'll just discard this as another non-news.

I am asking for a discussion; that's why this is posted here.

Re:Sorry, but

[Morth]

Ok, perhaps I was a bit harsh in the previous post, but I still don't think it's a very well written article. Try to be a bit more objective, like listing the current status, looking up more directions different file systems are heading in and the advantages and disadvantages of that, etc.

As for accessing files as folders, I find the concept interesing. Pretty much like OS X packages, except the other way around (to the same effect, more or less). I don't see why it would have to be file system specific though (or stored as metadata). A much nicer implementation would be something like linux's binformat_misc, where you list a magic or a file ending and then a program used to access it with. In this case the program could for example be called with the full path to the file as one argument and the subpath used as the next (following the args specified in the magic file). The program would then have a pipe (or socketpair) to the program calling open() on its stdin/stdout. I haven't thought much about security issues, but there can probably be a solid way to do it. I guess there might also be some issue with fseek(), not sure if there's a smart way to do seekable pipes.

Negativity

[listen]

I'm surprised at the negativity of some of the comments here, moaning that POSIX semantics are perfect and nothing else can possibly be countenanced...

Plan9 namespaces and Reiser4 really do bring a lot more to the table in terms of useful expanded semantics and utility than all the posix filesystems. Posix extended attributes are very limited, and some filesystem implementors seem to be keen to implement them in the most restricted way possible ( eg size limitations in ext3).

The annoying this with Reiserfs is that the VFS will lag it by a few versions, and very very few apps will make any use of its special system call. Sigh. We'll be stuck with databases in a file for a long while yet.

One thing I would like to know about reiserfs is how attributes are attached to directories? If they are just small files in the "directory" bit of a file, what distinguishes them from children of the directory? Or are attributes just banned from dirs? Seems limiting.

Re:Negativity

[Alomex]

Well, geeks have come to believe that Unix circa 1990 was brought down from the mountain top and thus it cannot be improved.

This is bad for Linux, because once we come to accept it as perfect it stops evolving and it will lose market share (see Macintosh 1990-1998).

Re:Negativity

[uhoreg]

One thing I would like to know about reiserfs is how attributes are attached to directories? If they are just small files in the "directory" bit of a file, what distinguishes them from children of the directory? Or are attributes just banned from dirs? Seems limiting.

Extended attributes will just be normal files. The ".." convention is just a convention. Storing the attributes is implemented by making it possible to use files as directories (and vice versa).

Re:Negativity

[listen]

You've completely misunderstood my point. Read it again.

How do you have an attribute of a *directory* with the same name as a child, if they are stored as direct children? In any name mapping you can come up with, there will be a conflict, unless you ban some characters from normal filenames.

ie directory foo, with an attribute bar and a child named ..bar. How are they distinguished? This happens whatever you do, you have to ban files beginning ".." or whatever your mapping is.
This will break some stuff.

Re:Negativity

[uhoreg]

They are not distinguished. Reiser4 technically does not store extended attributes. They are merely files that can be treated, by convention, as extended attributes. (Of course, the "..uid", etc. files *are* attributes, and will prevent you from having a normal file called "..uid".) The ".." convention was chosen to avoid collisions, since nobody seems to be using a ".." prefix for their own files. This should not break anything.

Your grandma doesn't know what is /dev/hdb15 ??

[Anonymous Coward]

Heh!
My grandma is a kernel hacker.

On the .. syntax

[IamTheRealMike]

Oh, I should probably mention - if you read the whitepapers available on ReiserFS the "foo/..attr" syntax is just a toy, made up on the spot, syntax to demonstrate the idea of files within files.

Nobody (apart from perhaps this guy) has ever claimed that this syntax will actually ever be used, or needed. There are other possible syntaxes available, and in fact one long term blue sky plan for RFS is to allow many different types of syntax within the same file path, including for instance things that vaguely resemble database queries.

So, don't get hung up on the syntax given in this article.

Re:On the .. syntax

[Anonymous Coward]

So what syntax do should be used?

Re:On the .. syntax

[IamTheRealMike]

*shrug* I dunno. Walk before you can run and all that - I don't even know if the files/directories thing will actually be in Reiser4.

Last I heard, there was going to be a special system call that you could use to access the special features, but standard POSIX users wouldn't see them (which makes sense).

So, really, not much will change at first. All these things, if they happen, will be very gradual.

Re:On the .. syntax

[Sloppy]

the "foo/..attr" syntax is just a toy, made up on the spot, syntax to demonstrate the idea of files within files.

Oh good, because I was getting concerned. If foo/..rwx/..rwx is set wrong, then can I lose my permission to see my file's permissions? ;-)

Re:On the .. syntax

[Skeme]

Good question, that infinite recursion is avoided :)

Re:On the .. syntax

[hansreiser]

No, it is not a toy, it is a style convention.

In regards to attributes on attributes, some (regular unix plugin ) files have relationships to other files that are implemented using stat data plugins, but the stat data files don't have stat data themselves. There are lots of specific plugin implementation details, but it works without infinite recursion problems

Re:On the .. syntax

[IamTheRealMike]

Thanks for clearing that up. Good to see you taking part in these discussions Hans! :)

Re:On the .. syntax

[Anonymous Coward]

Oh, I should probably mention - if you read the whitepapers available on ReiserFS the "foo/..attr" syntax is just a toy, made up on the spot, syntax to demonstrate the idea of files within files.

Reminds me of the "foo/@attr" syntax used by xpath...

plan9 is hardly a "new advance"

[sholden]

We had plan9 machines here 10 years ago...

I don't think any exist anymore, in fact I don't even think the inferno install works anymore.

But anyway, it isn't a "new advance" anymore.

Re:plan9 is hardly a "new advance"

[ajs]

Plan 9 is a moving target, as is Linux. To say that Plan 9 isn't new is like saying Reiser isn't new. That's true enough, but Reiser 4 certainly is.

Re:plan9 is hardly a "new advance"

[sholden]

Which is the point.

You can't say plan9 is a "new advance" (well I guess it's not that old, but "new" in computing has a different meaning than say in chemistry).

You could say that plan9 has seen some "new advances" in recent years, but that isn't what was said.

Has plan9 seen anything "new" in recent years? Things like the plumber came across from inferno, so I wouldn't class them as a "plan9 advance".

Re:plan9 is hardly a "new advance"

[ajs]

I think you mis-read the original.

It didn't claim that plan 9 was new. It said, "...new advances in software like Plan 9..."

You're reading that as "new advances, like plan 9, in software".

Plan 9 is not new, but certianly in th prosaic world of filesystem design, Plan 9's cited paper from 2000 *is* new, and *is* very interesting to the developers working on filesystems today.

XML vs. filesystems

[xyote]

People keep trying to use file hierarchies as data bases. You can do a lot of stuff, but arrays and m to n forward and reverse mappings aren't among the things you can do with filesystems. That's why you have databases and XML.

Re:XML vs. filesystems

[MarkWPiper]

Couldn't arrays be done as /array-name/0, /array-name/1, ... etc?

And for m-n mappings, I assume you mean a hash table. Couldn't this be done in almost exactly the same way /hash-name/key? Then to get the reverse mapping, we use some utility like find.

These new filesystem ideas are incredibly important. I've had this stuff in the back of my head for years. The biggest barriers: first, a syntax that people can understand (it will be very easy to forget what's metadata, and what's not, so why should we be accessing it in a different way), and second, adoption. As many people have pointed out, this breaks all kinds of compatibility. Another perhaps unforseen benefit: the filesystem could become an interesting abstraction layer for component based software development, rather than using an architecture like COM, or CORBA (although it feels like we have enough component architectures already)...

No need for LDAP?

[jackalope]

It seems that the author presumed that the only use of LDAP is to provide passwords for user authentication. While that is a common use of LDAP it is not the only use.

It would seem that having a file system that is LDAP aware could be extremely useful. Imagine if your LDAP tree were reflected as a tree in your file system. You wouldn't need to embed LDAP calls in your application, it would just be data in your file system. So looking up an attribute for the current user, or a user, would be as simple as reading a file that holds the value of the attribute.

Re:No need for LDAP?

[n3rd]

Imagine if your LDAP tree were reflected as a tree in your file system

Ack, I dunno about this one. If you have a true file system (even in memory) imagine the resources it would consume. One directory for every container, one file for every attribute at the very least and perhaps space for it's value and on top of that it would be local to every system. It would also need to be sychronized with the LDAP server on every system. At most you would have an entire local copy of the LDIF(s) which could be *gigs*, at least it would be the resources to track hundereds if not thousands of files and directories.

You wouldn't need to embed LDAP calls in your application, it would just be data in your file system.

I'm not a programmer but I have a question about this one. Aren't name service lookups handled via name service indepenedent calls? I'm not a programmer so the only example I have is gethostbyaddr(). Does an application need to be written differently depending on the name service it will use (LDAP, NIS, NIS+, DNS, local files, FNS, etc)?

Re:No need for LDAP?

[jackalope]

One wouldn't need to copy the LDAP data into a file system. Instead provide a view into the LDAP directory via filesystem calls. In otherwords, the filesystem driver would translate calls to it into LDAP calls to the LDAP repository, whereever it may be. The only persistent data that would need to be stored for the file system would be the how to connect to the LDAP server farm.

In regards to your second question. The standard getxxbyxx() calls are useful for returning username/uid/gid etc. But if the application wants to know something like 'What is the mailstop' of this user? There is no standard call, one must revert to LDAP API calls. So, your statement is correct in regards to system level information, but those services are inadequate for almost all application level user metadata queries.

Re:No need for LDAP?

[Anonymous Coward]

That's a good idea.
I'll add it to my project list.
hehe :-)

Re:No need for LDAP?

[MystikPhish]

Hmm... interesting.

How hard would it be to implement something like the /proc virtual fs for /ldap ?

Would the ldap data be too large?
Could you use the ldap data in a database and have the read/writes to /ldap actually transform into database calls for the info?

I have no idea what I am talking about, feel free respond appropriately.

What about ext3?

[AnEmbodiedMind]

Maybe I'm missing something but isn't ext3 the most popular "desktop" file system?

Re:What about ext3?

[10Ghz]

Honestly, I don't know. But what makes you think that ext3 is the most popular?

Re:What about ext3?

[ElGuapoGolf]

No.

I think RH is the only distro that uses ext3 by default... And other than the easy upgrade path, I just don't get that. It's generally slower than Reiser, and it's a bit more immature than Reiser.

And their engineers are rather snotty about it.

What I REALLY want

[afidel]

Is for someone to come up with a real unlimited snapshotting filesystem for linux. I don't want to use user mode hacks (as nice as they are rsync style snapshotting isn't reliable enough), or snapshotting that only allows a shadow copy of the entire volume, I want to be able to tell the users that they can just go into ~/.snapshot/time (where time can be hours, days, or weeks in the past) and copy the file they messed up back into their home directory. Basically I want the most usefull feature of netapps without the HUGE markup =) The cost in admin time both in user interaction and reduced need to do tape retrieval and file restores is immense.

Re:What I REALLY want

[codepunk]

Load redhat 9.0 then it has a volume manager designed for exactly that purpose...

Re:What I REALLY want

[afidel]

Are you talking about the LVM, because that is partition at a time backups and there are some serious issues with the way it is implemented. If not I would love to hear some info on it as it's not documented anywhere on RH's site.

Re:What I REALLY want

[IamTheRealMike]

That kind of thing would benefit from this, especially if integrated into the GUI in a nice way.

In effect, it'd be similar to what Linux does with memory - empty disk space is useless! What is the point of that? Might as well use it for something. Storing old versions of files is an ideal use for it, and combined with an LVM means that the more space you have, the further back in time you can go. That's far more flexible and useful than storing stuff in a recycle bin.

Of course, you run into problems straight away if you aren't using lots of small files. Are you really going to store a new copy of that 5meg presentation every time you hit save? Bad idea. You only want to store what has changed.

But - how? Doing it with text is easy enough, we have diff and patch for that. But what about more complex formats?

Ahhh, now we understand why minimizing primitives is useful in the real world. If the internal structure of a file is split into many small files, each representing a facet of the presentation (each slide, each bullet point on the slide, font weight of that bullet point etc) then suddenly it becomes much simpler to write a generic differential analyzer, we can just snapshot the mini files that changed.

As a result, we increase efficiency to a level at which it becomes feasable to keep the undo transaction buffer on disk - just imagine how much hassle we'd save if all apps used this reliably?

There are still plenty of details missing of course - some things still wouldn't work well, in particular large BLOBs with no structure, like say images, or audio files. I guess if you were smart you could leverage the apps transaction buffer, but still..... that would lead to interop problems.

Still, there are many unexplored possibilities that appear when you increase the abilities and even efficiency of low level parts of the OS like this.

Re:What I REALLY want

[afidel]

Ahhh, now we understand why minimizing primitives is useful in the real world. If the internal structure of a file is split into many small files, each representing a facet of the presentation (each slide, each bullet point on the slide, font weight of that bullet point etc) then suddenly it becomes much simpler to write a generic differential analyzer, we can just snapshot the mini files that changed.

Why go to all that trouble, just keep a list of changed blocks, the way that netapps do it is the snapshotted file stays in place and any changes are placed in a new block, then the filetable has a pointer to the new file which may include blocks from the old file, and of course the snapshot directory information just points to the old file on disk. This works very well overall, we only had about a 40% disk overhead for 8 hourly backups, 7 daily backups, and 6 weekly backups.

Re:What I REALLY want

[IamTheRealMike]

I guess that would work too, but it means you lose if, for instance, a part of the file shifts by say 200 bytes.

That's why diff/patch doesn't record each line that changed, they have quite advanced fuzzy matching, to increase efficiency.

Re:What I REALLY want

[afidel]

The Netapp implementation is pretty damn good, like I said in a busy engineering environment we only needed 140% of normal disk space to keep the current data plus quite a bit of online backups. You still need tape but the only time we ever needed it was when we ran into a bug where backing off a recursive subdirectory application of NTFS permissions was going to take several days for some reason so recovering it from tape was faster (the developers were not happy, they wanted sourcode locked down but the netapp admin applied the wrong group permissions and they were not able to get to the code at all for over a full working day, ugh). I really couldn't believe how much admin time is saved by the snapshot feature, and then there is the fact that hourly backups are automated, not something most places do with tape.

Re:What I REALLY want

[Anonymous Coward]

Sounds like you need windows 2003. Look it up - volume shadow copy.

Re:What I REALLY want

[afidel]

Nope, although something decent could probably be made using the volume shadow copy interface, but it would still be much worse from a diskspace usage perspective then a properly implemented low level snapshot like the Netapps. As I mentioned above 21 levels of snapshotting plus the origionals only take up ~140% of the diskspace of the current files, I haven't seen anything else that even comes close, and the convenience of training users to simply go into the subdirectory named .snapshot to retieve their own backups can't be overlooked.

Re:What I REALLY want

[greenrd]

Why go to all that trouble, just keep a list of changed blocks

If the file format is a serialization of an in-memory structure or something similar, inserting one byte may cause all the bytes after that to be shifted along by one. So all the blocks after the changed block will also be changed. Not very efficient.

Re:What I REALLY want

[epukinsk]

Are you really going to store a new copy of that 5meg presentation every time you hit save?

Correct me if I'm wrong, but isn't saving a file and creating a new file roughly the same operation? Instead of saving over the old file, you could save to this "old version buffer" and then just swap the filesystem pointers.

Erik

Re:What I REALLY want

[Xanni]

Here's a system Project Xanadu developed to support microversioning (including an implementation for emacs):

http://www.xanadu.com.au/ted/OSMIC/

Share and enjoy,
*** Xanni ***

Re:What I REALLY want

[Anonymous Coward]

Can you describe how the NetApp works?
It's hard to work on something when you don't know how it's supposed to behave.

Does it only store deltas?
Or the original + deltas until time X?

Re:What I REALLY want

[afidel]

I'll let the good folks at netapp do a description, they are far better at it than I. Descrition

Re:What I REALLY want

[afidel]

Actually it looks like I was a bit hasty in rejecting this, I had looked at volume shadow copy only from a developers POV, in what it allowed for backup clients because that is where it was first implemented (backup for XP and then Veritas's backup for server 2003), I hadn't seen the client side app until I did some searching prompted by your suggestion. It IS just about what I want, though the limited number of deltas is annoying (only 64 and you have no controll over which ones are kept, oldest always gets pushed off the heap so you can't do staggered hourly, daily, weekly, monthly like you can with a netapp), the other annoying thing is the need for a special client (MS's site only lists XP and .Net as clients, others have said 98 and above), either way it would be nice to simply have them available as hidden named subdirectories. But all in all I guess it's the best implementation out there for the money, so it's the quasi functional and cheap server 2003 at the low end and the very functional but expensive netapp at the high end, just wish there was a free and functional linux choice in the mix too =)

Re:What I REALLY want

[Anonymous Coward]

Been there, doing that. You might call it a "retroactive snapshot" to any point in time that you want, and it's fully platform-independent. Turns out that it's a real bitch to do this without severely impacting performance, which is why host-based solutions always suck so hard. It can't be done properly in an LVM, and anyone who tells you otherwise is just blowing smoke up your ass.

Re:What I REALLY want

[BigBadBri]

Not an expert on this, but it appears that you could do this with a plugin under ReiserFS.

Basically, you have a new file attribute called versioning, which is a boolean. The plugin looks at this value, and if it's set, copies the contents of the old file to a subfile with a date/time string appended. It then overwrites the contents of the old file with the new data.

Might be a bit of a disk hog, but it would make microversioning simple and transparent.

Re:What I REALLY want

[Jonner]

Linux LVM already does something like this.

It sounded so intelligent until he started playing

[The Monster]

sure, im going to listen to this guy about filesystem implementation when he cant even set up MIME types on his webserver.

or spell: (emphasis mine)

If you just want to give them some default settings,have that line below their home directory's line. Viola! Lock down framework.

I know we hackers like to, uh, fiddle with our computers...

Other Filesystems?

[CordMeyer]

I use HFS+ because Apple makes me. Is it efficient? reliable? Well... it works most of the time.

The + kept us busy for a while.

Mwa...

[Yuioup]

I think it's going to be an absolute mess.

I don't know much, but storing meta-data in files in which you have free reign over what you can put in them cannot be good. Firstly a lot of files are going to be full of garbage. I could theoretically copy-paste a ascii dump of Lord of the Rings into a meta tag. Imagine having to search a directory - oh sorry files-as-directories - in my filestructure for a specific tag.

Furthermore I can see people putting .mp3's online with nothing but SPAM in them. You've basically found a new way for lowlifes to make a living.

And then there are viruses, exloits etc.

Yuioup

Re:Mwa...

[Anonymous Coward]

I think it's going to be an absolute mess.

Oh?

a lot of files are going to be full of garbage. I could theoretically copy-paste a ascii dump of Lord of the Rings into a meta tag

I could theoretically copy-paste an ASCII dump of Lord of the Rings into a text file. Oh no, the horror!

Imagine having to search a directory - oh sorry files-as-directories - in my filestructure for a specific tag.

Which is different to having to search a directory for a specific file...how?

Furthermore I can see people putting .mp3's online with nothing but SPAM in them.

I can see people putting .mp3's online with nothing but SPAM in the ID3 tags! Oh wait, that hasn't happened at all.

then there are viruses, exloits etc.

Of what? Unless you've got a buffer overflow in some part of the meta-data handling code of the filesystem, meta-data is inactive. Its no more a virus threat than downloading a text file with a UUencoded copy of a virus in it. Unless you go through great pains to extract, decode and execute the package, its harmless.

I don't know much

Well you got one thing right.

Swiss Cheese

[Anonymous Coward]

Cool ideas, but let's be careful lest we make a steaming pile of Swiss cheese.

Evidently you haven't used ReiserFS. It already does this.

ReiserFS only journals filesystem metadata. Because it uses a B+ tree balanced allocation scheme for file blocks, when the system crashes the last pair of blocks written will often be swapped with respect to their files. For example (this has happened to me and separately to a friend) if you modify out /etc/passwd just before the system crashes, you'll be unable to log in again when the system comes up.

What happened? syslogd wrote a panic message to /var/log/messages as the system went down, and you'll find this message (as well as the rest of its 4k block) in /etc/passwd, while your changed password file may be found at the end of /var/log/messages. This is a feature of ReiserFS, not a bug.

I still miss the raw speed of ReiserFS, to be sure, but EXT3 has kept every last one of my hundred-odd filesystems rock solid for two years now, which is really what you want a journalled FS to do.

ObOnTopicComment: Miner's examples are clumsy and ill-considered. BBN's Dave Mankins put a relational database into the 4.1BSD filesystem back in 1984, and Plan 9 took a more rational approach with its namespace algebra. This is not a new idea, so there's no absolutely no excuse for breathless exposition based merely on coolness factor.

At best, Miner's descriptions obscure any true value Reiser's proposal might have. Organize my MP3 collection with FS metadata and lose it all when I try to move to another FS? What is he thinking? Is he thinking at all?

Sheesh.

Re:Swiss Cheese

[Anonymous Coward]

ReiserFS 4 royally fucks up your data if you have the misfortune to experience a crash, and this is a feature? Gee, that Hans Reiser sure is a fart smeller!

Re:Swiss Cheese

[johannesg]

Organize my MP3 collection with FS metadata and lose it all when I try to move to another FS? What is he thinking? Is he thinking at all?

AmigaOS had a single meta-data field in its various filesystems, called the "file comment" or "file note". While it was clearly far less powerful than what is proposed here, it was incredibly useful for lots of purposes.

The problem you describe, losing the meta-data when moving to another filesystem, was non-existant on the Amiga for two reasons: all filesystems implemented filenotes, and LhA (the Amiga-equivalent of tar/gzip) saved and restored filenotes. A similar solution can be used for Linux (tar and gzip are getting very long in the tooth anyway).

I would love to see proper meta-data fields implemented throughout the system, especially if it is pervasive throughout the OS (so I can search for specific values from the desktop, for example).

Finally, I believe BeOS had something similar, and users of that filesystem also tend to be enthousiastic about it. All this doesn't mean the current solution is necessarily the best one, of course - just that the concept is pretty damn useful.

Re:Swiss Cheese

[sql*kitten]

syslogd wrote a panic message to /var/log/messages as the system went down, and you'll find this message (as well as the rest of its 4k block) in /etc/passwd, while your changed password file may be found at the end of /var/log/messages. This is a feature of ReiserFS, not a bug.

Care to explain that? How is corrupting one of the files essential to the normal operation of your system not a bug? The whole point of a modern filesystem is to recover cleanly; even fsck can run unattended (mostly) but what you're saying here is that after an unplanned reboot, you might or might not need a sysadmin to manually intervene.

Re:Swiss Cheese

[dodobh]

Sarcasm :)

Re:Swiss Cheese

[Jonner]

So, how would Ext2 deal with this? AFAIK, it wouldn't, meaning there could be complete gibberish left in /etc/passwd. Is that a bug in Ext2?

I'm sure that complete data journalling gives somewhat better reliability than what ReiserFS currently does, but it's a lot better than traditional filesystems. I don't think it's fair to call something a bug when it's just a lack of functionality. ReiserFS claims to have reliable metadata, not contents.

Seems good for some applications,...

[ddubois]

I feel this very interessting.
For example I like the currents devfs and procfs (although not perfect). Those help me a lot to "debug" new hardware connection. SImple and coherent.

I also imagine some RDB support. Imagine a "select * from account" where account would be asimple directory. Would be nice to use "find" for the where clause :-)

Imagine also implementting OO classes wiht inheritenace using symlinks... And more...

Yes would be very nice!
D.

MOD PARENT UP

[Anonymous Coward]

It looks insightful

The logical conclusion of the Tao of Unix

[The Monster]

Reiser's vision of files that are also directories, and putting metadata into files, is just the logical perfection of the fundamental *nix notion that 'everything is a file'. It is this which has given the OS such strength for three decades - before anyone knew about 'object-oriented programming', Unix was behaving in an o-o manner at the macro level, with individual programs doing encapsulation, etc. Having plugins that expose the details as ordinary files, allowing you to access information with your editor of choice, as well as scripting tools such as sed, grep, awk, perl, or Python is what Unix is all about.

The only thing I'm concerned about is backward compatibility - if someone accidentally tries to open a file with a trailing slash, and gets an error because now it's a directory, then it's a Bad Thing.

Re:The logical conclusion of the Tao of Unix

[Anonymous Coward]

The only thing I'm concerned about is backward compatibility - if someone accidentally tries to open a file with a trailing slash, and gets an error because now it's a directory, then it's a Bad Thing.

This would cause an error in most applications anyway. Backwards compatibility is one thing, but having the filesystem driver automatically fixing typos is a Bad Thing. This is the realm of another more AI style user interface layer - and defintely not the filesystem.

Re:The logical conclusion of the Tao of Unix

[Anonymous Coward]

Plan9 was the ultimate realization and implementation of "everything is a file."

*nix is the dark ages with only hints of the coming enlightenment.

common ENV variables...

[mennucc1]

...are not necessarily a good thing; many big programs (particularly, propertary ones) need some tweaking on ENV variables to work properly

Re:common ENV variables...

[spitzak]

The paper described the Plan9 system where each process has it's own namespace. So you just fork a process that changes it's namespace to whatever is necessary before running the program. Similar to forking and changing the environment on Unix now.

Metadata on the Paper

[Anonymous Coward]

I enjoyed the presentation of Reiser and Plan 9 information, however...

I think the author should revisit a few things.

- The author assumes his position is correct, without noting, and then refuting, opposing arguments.
- Very few reference materials, three of which were internet sources.
- Inconsistent use of pronouns. Is this a first person or impersonal paper?
- Comparisons to ext3, ntfs, et alii, are missing.
- Some terms are introduced, but not explained. Although you are writing with technical people in mind, even they might not be familiar with all of the terms.

The author appears to believe a synthesis of Plan 9, Reiser, and RDBMS features is the best solution, however, why this particular synthesis was chosen is not described.

I'm just trying to be helpful. I mean if people only give complements ... well, you get the idea.

I just want...

[Anonymous Coward]

A filesystem that goes wrong properly...

Imagine your filesystem is a library...

Imagine you drop a bomb on it...

Books and pages scattered all over the place... Yet you can still work out which page belongs to which book, and where on the shelves they used to sit..

Having been caught short by LVM and reiser before (which just couldn't deal with a 46GB gap in the filsystem where a disk used to be) it seems to me that no-one's made a filesystem that breaks properly...

For me.. speed is not an issue.. nor is CPU usage... I'm quite happy to throw a dedicated box at handling the filesystem... I just want something that is written with the thought in mind... "Our hardware is unreliable.. It will die.. it wll loose bytes... How do we deal with it"... not the usual thing that people do of "Our disk is 100% reliable... how can we most efficiently organise it... Hrm.. now what do we do when we encounter problem X" There should be no fsck... That should be handled by 'mount'

Sure.. there's always RAID... but I reckon I've got about 28MB of data in total that's critically important... the rest is just crap I downloaded from the net and can always get again...

Continuing my anology further...
If your filesystem's a library, then then your physical disks are wings of the library.. (east wing.. west wing.. etc)...
I want a librarian who's job is to work out the best way to organise the data in the different parts of the library... When you're looking for a book, it's often easiest just to ask the librarian.. who magically holds much of that information in his/her head.
The librarian is also responsible for making sure that multiple copies are maintained for popular or important works...

It'd also let me do the thing I've always wanted...

chattr +mirror filename

(i.e. This file is important and must be mirrored on 2 or more disks)

Just my two pennies worth... I'll write it one day if someone else doesn't get there first.

Anyone remember Acorn RISC OS 3?

[Fjan11]

Good old Acorn RISC OS already supported the use of directories as files back in the eighties. E.g.: Click on a file to open it, shift click to show a directory of sub-files (recurse at will).

Market forces can be a mixed blessing, there were quite a few brilliant innovations in there that never made it onto the Wintel platform. jm

Re:Anyone remember Acorn RISC OS 3?

[BinaryCodedDecimal]

Good old Acorn RISC OS already supported the use of directories as files back in the eighties. E.g.: Click on a file to open it, shift click to show a directory of sub-files (recurse at will).

You're thinking of application directories. It could not be done with ordinary files - They had to be a directory.

The only difference was that if a directory had an exclamation mark as its first character, RISC OS's default action would be to execute a file called !Run inside that directory.

Fantastic idea - It meant that you could keep all the libraries and files required for an application in one folder. It also meant that you could move the program and its associated files wherever you liked on the hard disk - There was only ever one icon to move.

If you ever decided to move stuff around like that in Windows, prepare for your programs to stop working.

Re:Anyone remember Acorn RISC OS 3?

[Fjan11]

It could not be done with ordinary files - They had to be a directory

The OS did indeed also have files that were not folders, and also folders that were not files. That's only logical, who would want to give them up?

Besides programs disguised as folders, there were also applications on the platform ("Impression" for instance) that used folders disguised as files. This was neat: You could click into a wordprocessor file to get at the pictures in the file.

Re:good read, but less relevant

[sultanoslack]

This is said by someone who obviously hasn't done any real world application profiling. It's quite the opposite -- CPU is relatively rarely a limiting factor in desktop applications, dealing with the HDD very often is.

This is very often why adding more memory to a system makes it seem more responsive -- larger disk buffers, less need for disk based virtual memory.

Basically hard disks are very often *the* limitation; CPUs are fast.

PDFs are not W3C standard, have security problems

[Anonymous Coward]

For one, PDF is not a W3C standard, whereas HTML is.

Adobe Still Ignores Elcomsoft-discovered Holes describes security problems with PDFs.

Ahhh!

[Anonymous Coward]

That explains why I had so many problems with ReiserFS...I was trying to use it on a *functioning* system! They should put some kind of warning in their docs or something.

Started out good...

[Korgan]

The more I read this, the more it reminded me of the marketing version of how Apple would like us to think of Resource Forks.

Truthfully, there isn't exactly a lot of difference in the concept or the idea. Implementation is vastly different but the idea remains very similar.

Why do I want to accept this sort of idea anymore than I want to accept resource forks? If I copy a file with resource forks from one of my macs to nearly any other OS on the market thats not specifically configured to support them, I lose that information. Why do I want to continue this?

I use HFS+ because I have to. To get all the functionality I want out of my macs, its the only real option I have. But for anything other than system level files that are never likely to be copied to another machine, this is just a waste of time to me.

Next question. Say I do run this file system on my machines. I build up a heap of data and I'm using "files as directories" to store metadata about those files. How do I back it up? Don't even try to tell me "rebuild tar". Haven't we put tar through enough to try and extend its capabilities? I wouldn't touch a file system with these capabilities without a guaranteed way of being able to backup ALL the data. Otherwise its just truly not worth the effort.

Some of these ideas are VERY short sighted

[irw]

I wish people with clever ideas to redesign POSIX namespaces would spend ten years in system administration first so they realise what's involved with managing REAL WORKING SYSTEMS.

Some of the ideas might well lead in useful directions, but some (at least as described in the paper) are plain silly. viz:

1) with overlayed mounts:

suppose my home dir is mounted read-write over a read-only system root, and I do not have a "/bin/prog" in my home dir. Consider:

cp /bin/prog /bin/prog

First time, it copies the system /bin/prog into my home fs - Counter-intuitive to the path semantics. If I run this a second time it copies my copy of /bin/prog over itself - Inconsistent.

2) Attributes in the namespace

We have a rather carefully written setuid chown/chgrp/chmod replacement which can be run by users in an "admin" group, and allows devolution of 1st-line support tasks to nominated users. It won't touch files whose uid/gid is 100, so they can only touch non-system files.

If attributes (file uid) is file/..uid and cp is supposed to handle what chown does, the above breaks big-time. We now need a custom cp replacement. Either that or we have to add an ACL for the admin group to every file we want them to manage, which is a great deal of effort, and likely end up inconsistent.

Contrary to the paper, setuid and PARTICULARLY setgid is NOT going to go away in the real world any time soon, as far as files are concerned. Ports less than 1024 are a different matter and I agree with the document.

3) Consider the number of file descriptors involved if /etc/passwd becomes a hierarchy of files. Just logging in one user will involve multiple open()-read()-close() operations. Whilst these might be efficiently implementable at fs-level, it is still very inefficient in user space, or will at least require a dramatic rethink of unix tools.

Re:Some of these ideas are VERY short sighted

[Wesley Felter]

First time, it copies the system /bin/prog into my home fs - Counter-intuitive to the path semantics. If I run this a second time it copies my copy of /bin/prog over itself - Inconsistent.

You've discovered the essence of the old Plan 9/Hurd joke "you are in a maze of twisty little filesystems, all subtly different."

Re:Some of these ideas are VERY short sighted

[uhoreg]

1) AFAICT, cp /bin/prog /bin/prog will fail due to permissions. When you try to write to an existing filename, it will try to write to the layer in which the filename already exists. Since /bin/prog is read-only, you will get a permission error. At least that's how writes were explained in the ReiserFS list. (BTW, it is not yet determined that this feature will be in Reiser4. Hans basically told the guy who suggested it (who happens to be the same guy who wrote the article) to write a plugin.)

2) I would imagine that the Reiser4 way to do this would be to use a plugin. But seeing as you don't need root permissions for chown/chgrp/chmod, a user could just run his own version, so I don't really see how your system provides any security.

3) Reiser4 will allow you to view /etc/passwd both as a flat file (for compatability and efficiency reasons) and as a hierarchy.

Re:Some of these ideas are VERY short sighted

[wolfb]

I agree with you on some points, but I think you lack some imagination. (for better or worse :) )

For exampe, the access control system itself is not discussed in the article beyond the suggestion that the access control information itself (attributes, permissions, acl, etc.) could be treated like arbitrary metadata. Obviously, this doesn't mean that access control would be voluntary, or enforced by applications such as "cp"!! Some access control mechanism has to be present, and that system would have to excercise access control over its own metadata files too.

Imagine that the access control system would only allow the superuser to modify access control related metadata files. You'd have a host of setuid utilities, running as root, to manipulate the metadata and enforce access control. For custom access control, you'd write your own setuid utiltiy replacements.

Or imagine a unix-compatible access control system that took on more responsibliy. The old setuid utilties could be replaced by simpler ones that manipulated the metadata, and did nothing else. For custom access control, you'd still write setuid utiltiy replacements.

Or imagine an access control system that was very flexible and configurable. For custom access control, perhaps you would get away with a one-liner in a system config file. ... Or you could still write your setuid utiltiy replacements.

Or imagine that the access control system might become totally pluggable, and you could choose whether you wanted windows semantics, unix semantics, or your own access control logic with your own arbitrary meta data.

Having access control information stored as arbitrary metadata in the namespace simply does not imply that you would loose access control, nor that setuid and setgid programs would *have* to go away. It simply opens other possibilities under the assumption that more and more of the access control logic would be moved into some central system. I guess it is arguable whether this is good or bad. The system still has to evolve and mature, and some of the solutions you call short sighed have yet to be considered seriously. Take what they have for inspiration, not as a blue print. It just isn't there yet. Reshape it using your own experience; try to find plausible solutions to the problems you see. Its refreshing to think outside the box, once in a while.

From the article

[tundog]

In POSIX, attributes...However, as more people want more atributes (such as support for access control lists)

Great now DARPA wants to control my access to my pr0n too!

Security?

[goombah99]

Here's something I dont get. NFS is totally insecure to anyone who can jack into your network. Keeping physical security on a machine is easy and practical compared to keeping physical security on a network. most buildings have jacks all over, visitors come and go. Yet people need to mount NFS disks. Even with physcial security advanced techniques like ARP spoofing can sometimes get you in.

so why why why is anyone working on improving the FS until NFS is fixed or replaced with a decent security model. Ironically fixing NFS appears to be trivial--something like ssh to authenticate the client might do the job. But this might break a lot of systems in an imhogenous network. even so I dont see any movement here. Everyone's worried about layers over NFS but not NFS itself.

If I'm an idiot then I'm in good company. I know dozens of other sys admins and none of has a clue how to secure an NFS system against people who can jack in to the network.

ironically it gets even worse now that people use ssh and ssh-keys to log in. it makes the network less not more secure. its simpler to attack a client with remote mounted home directory using ssh keys than it is to sniff for telnet passwords!

What's in a name?

[Zog The Undeniable]

I assume Plan9 is an ironic nod to the "worst film ever". When I develop my new filing system, which will only allow numeric characters in filenames, will delete the MFT every time the computer is rebooted, and will require a new directory for each file added to the system - that FAT16 limit of 512 was FAR too generous - I'm going to call it BattlefieldEarthFS.

I like this...

[Dalroth]

I like this, VERY much so! Unfortunately, it's going to be a bitch to get everybody on the same page and moved over to a system like this. I'm not saying it's impossible, it's just going to be incredibly hard.

We'll probably need to see a distro (probably Debian based since Debian based distros tend to do these things first) that purely focuses on stuff such as this. It would hopefully set the standard by which others would have to live by.

We'll see... too far down the road to know for sure though. Still some unanswered questions, such as performance hits and/or memory usage.

Re:Incompatibilities with another system (.ext)

[CaptnMArk]

Well, what do you use for foo.h anc foo.c files?

fooSource and fooHeader?

Hardly an improvement, I'd say.

Cross-platform compatability

[tknn]

The real problem is Windows compatibility. I have to use FAT32 as a default just to be able to read the file system on my Mac, Windows, and Linux boxes. Whatever the future holds, it at least needs a driver for Windows that is easy to use....

9p2000 is an open protocol

[DrSkwid]

already one such implementation exists such that FreeBSD can expose its file system to plan9 machines (as you would expect it gets imported into your namespace. Would can be a different place depending on the namespace of the current process. Even (temporarily) "replacing" your local files with versions on the FreeBSD Box, if that's what you want.

Re: metadata, belongs in files, not fs - Do both!

[Anonymous Coward]

Why not do both? It would seem the eaisiest solution would to be to implement common header files, like Dvorak suggests, that then get mirrored into the file system. This could eaisly be done by the file system when ever it writes a file. That way, the fs could have a rational database for searching and all, but the files retain control over the metadata. Transferring the file would be no problem. The metadata would get transferred in the header of the file, and then written to the database by the filesystem. (and yes, there would be a little overhead for checking and writting the metadata to the fs everytime a file is written, but this is being done anyways by any fs that uses a metadata database, yes/no?)

Correction

[TheRaven64]

As anyone who read the grandparent post would know, when I wrote `latex2pdf' in the parent post, I meant `latex2html'.

Is AFS, apple file share

[Anonymous Coward]

Mac OSX mounds disks as either NFS, AFS or SMB. I assumed that AFS was apple files service or apple file share. Is this just a case of two things with the same acronym?

Re:Is AFS, apple file share

[m_frankie_h]

The AFS we were talking about is Andrew File System.

Re:Is AFS, apple file share

[Anonymous Coward]

Here're two AFS projects:
Arla
OpenAFS

TLA land

[fm6]

It isn't so much lack of support (NTFS is actually pretty good with metadata) as the old command-line mentality that sticks with an ancient method of identifying file types out of pure inertia. You don't have to be a Mac or Be person (I'm neither) to wish for a more reliable and less ambiguous method of typing files. But how are you going to get all those Windows "Power Users" to change? Sure, extensions are error prone. And its a pain when two applications seize the same extension. (Even different Microsoft applications do this!) And newbies are always getting in trouble with them. But extensions are easy to understand. Case closed, alas.

you mean kinda like this

[hswerdfe]

#!/usr/bin/bash

I totally Agree with you.
all application specific data about a file should be included with the file.

Linux Files and FS

[Anonymous Coward]

Linux does not only allow you to mount ISO images via loopback, but any FS. Just try it, use -o loop when mounting like you do with the ISO.

To create one, use dd to get the size you want (from /dev/zero) or copy from some real partition (ie, a floppy or a hd), run the desired mkfs if it was empty, and mount with the proper options. I have been using this to check ISOs and to clone floppies and store them into CDs for ages. I have used it to get perfect FS images too (machine updates, ie) and then extract the contents anywhere (eer, ok, Linux machines :) ). If you have access to a RH, check /sbin/mkinitrd, it is a script that builds the initrd in a similar fashion.

BTW, Linux also supports mounting dirs into other dirs, with -o bind.

Never going to happen

[MarcQuadra]

I used to live like that, then I took my big hard drive, slapped it into a linux box and shared it out with NFS, Samba, and NetAtalk. Now I can access all my files, which automagically get backed-up, from any machine on my LAN. Stop waiting for the 'universal' FS to show up, it'll never happen.

Re:Never going to happen

[tknn]

Why not? Why doesn't someone develop a nice FS, something like BFS/XFS/ReiserFS and also make Windows and Mac drivers?

Re:Never going to happen

[MarcQuadra]

Microsoft doesn't play well with others, they go OUT OF THEIR WAY to not interoperate. Trust me, you COULD write a FS-layer plugin for NT/2000/XP, but it would certainly run slowly and not work properly unless it had MS-blessing (and access to Microsoft source). Apple might be more open to the idea of embracing other FSs' but the angle-of-attack for that would be to get support into Darwin, and the boot partition would probably have to remain HFS+.

The problem is that the FS is one of the MOST important aspects of an OS, and no vendor is going to stick to the 'standard' if they can squeeze out more performance by bending or breaking it. There would be massive embrace/extend breakage. The security models for entire operating systems would have to bend over backwards to support some universal ACLs and behaviors. See the documentation for the UFS module in the Linux Kernel for an example.

If you want an FS that works accross the board for your go-between drive, I suggest FAT32 or ISO9660 (cd-rom). In a few years maybe we can switch over the read-only end to UDF. Until then, set up that NFS/Samba/NetAtalk server!

Re:Never going to happen

[tknn]

I know that someone wrote a program for Windows called MacDrive or something like that to read the Mac HFS format. Frankly, for my purpose, it doesn't have to read fast under Windows, it just has to work consistently and well so that I can use it as needed. It would be ideal for USB keychain drives and other such things as well. Right now I guess we are stuck with network servers and FAT32.

I would have read the article but

[Anonymous Coward]

I never read pdfs. If it isn't in html, or plain text, it's not worth reading.

Isn't this what a *database* is for?

[ca1v1n]

Anyone working with very large amounts of very small data should be using a database. That's what databases are designed for, and heavily optimized for. If you want file system access to the database, there's nothing stopping you from doing this. I know there are linux kernel patches that will access SQL databases. Turning the whole filesystem into a database is a compatibility nightmare waiting to happen. You don't need database optimizations for storing files like /etc/fstab, and if you do, you've got other problems.

Yes, some of these experimental filesystems are returning some interesting results and creating some nice niche capabilities, but personally the filesystem he describes being well suited only for very large files is exactly what I and a great many other people need. Anything small is cached in memory. The only small-write function I need is the file system journal.

Re:Isn't this what a *database* is for?

[Arandir]

Bingo!

The purpose of a file system is for peristant storage of data. The purpose of a database is for efficient organized access to data. But all too often we see database groupies advocating the filesystem-as-database, or the filesystem groupies advocating database-as-filesystem.

I use a database when I want to quickly find or correlate some data. "What was the company's net revenue from the GruntMaster2K product for April 2001?" I use a filesystem when I already know where the data is. "find" is a useful command, but I only use it about twice a week, most other times I know where the file is or can trivially browse for it, so the hierarchical structure of the filesystem is more than adequate.

Re:Isn't this what a *database* is for?

[gruterb]

Yes, that's what databases are normally used for. But they can also handle large amounts of data in a efficient way. The aim is to have an unified access to any type of data. This would be fine, but the step from the traditional filesystems to a RDBMS-based system, is too big to work with the old systems. But the proposed RDBMS would internally access the physical data on some HDD and use some internal method to physically store the data. It should provide an interface to support the old filesystem.

Mod parent up

[jpmorgan]

I hate to do that. :)

Re:It sounded so intelligent until he started play

[Tukla]

Oh, well. At least he didn't spell it, "Wala!".

I have no idea

[Anonymous Coward]

file/..uid/..uid/..uid/..u

Escape!

[zerocircle]

Start weeding out trailing / in filenames now.

I'm delighted with the prospect of metadata-as-files and files-as-directories (ergo, metadata-as-directories?) -- but here we have another problem to address: Insufficiently escaped data. Human-readable data fields (including filename, if the user can read/write it) should be able to contain any human-readable characters. Filenames should be free to contain normal punctuation; path separators -- again, if the user can read/write paths -- should be selected from outside the normal punctuation set, or else the stuff between the separators should be escaped. Or the user-accessible file and path names should be stored as metadata.

Can't tell you how much frustration I've endured over other people's improperly escaped data. This just looks like one more case.

(Mac OS <=9 used a colon as a path separator, making it the only forbidden character for file/folder names, which could have been avoided so easily: How about a pipe, guys? or (shiver) a backslash? Or, even better, some control character unique to the Mac, akin to the Option-Shift-K Apple logo? Programmers. Sheesh.)

WinXP filesystem strangeness

[nikko]

I just encountered the strangest artefact with the WinXP filesystem.

The "date modified" attribute of a folder never changes! It doesn't matter what I do in the folder, delete files, move files, add files, modify files; the "date modified" of the containing folder never changes.

How can this be? Win2k certainly worked the way you would expect. Has MS intentionally broken the XP filesystem to screw over all those 3rd party programs that rely on getting a reliable value for this attribute?

any insight much appreciated.

Re:WinXP filesystem strangeness

[spitzak]

Win2k and NT both worked the same as you are seeing. It breaks our gmake scripts that use directories to check if any files inside them have changed, so I am familiar with this, and it has been doing it for as long as I can remember.

Re:WinXP filesystem strangeness

[nikko]

All of my Win2K boxes work like I would want-- any changes to files in the folder changes the "date modified" on the folder. I just walked over to one and verified that this is true.

Only my new XP box has this insane behaviour.

Perhaps this behaviour is configurable? There is a huge number of poorly documented registry entries that can "customize" the behaviour of the filesystem. I stumbled upon a website devoted to these and was shocked at how far reaching the customizability is.

joe

Re:WinXP filesystem strangeness

[spitzak]

Never mind, it appears that my Win2k box works the same as Linux. Creating, renaming, and deleting files changes the date modified. However writing new data to an existing file does not change the modified date, and I believe we are being fooled by different behaviors of the linker, on Unix it deletes the file and creates a new one, on Windows it just writes over an existing one. This also means I can fix it by making the make delete the old file before creating the new one!

Sorry, don't have XP here to check out.

Re:Questions... Article's point is stupid

[scottwimer]

This is just dumb.

Hard drive space is under a buck a gig. And, some moron is talking about the benefits of saving a few kbytes, or maybe even several megs across the whole system by replacing config files in a universal standard with a completely non-portable implementation. Brilliant!

Blech. Dumb. Dumb. Dumb.

LFS!

[wirelessbuzzers]

An important advance in this direction is LFS, the Log-structured FileSystem. It's not exactly new; most of the recent improvements in it have been fine-tuning of the access and cleaning algos.

Basically, the main structure on disk is the Log. It stores all the iNodes and all the file data and metadata. If you have to write something to disk, you write it at the end of the Log. With a good buffer cache, this is extremely fast because you write large amounts of data contiguously.

Every so often you create a checkpoint, which is the metadata required to locate all the inodes and file data on disk at a particular time. Although I'm not aware of any implementation that allows this, you could theoretically roll the filesystem, or some part of it, back to any particular checkpoint (which has not been cleaned yet), or make it look to some user level program as if you had (they'd only have r/o access tho). Checkpoints also make crash recovery pretty fast.

Reading times are not quite as good as Ext2/3 under some circumstances (some workloads can massively frag files) but if you rewrite the majority of a file at a time, reading times can actually be faster. And running the cleaner a lot makes it even better.

There is a lot of CPU overhead, but very little disk-seeking overhead. The result is that as CPUs get faster, your IO will get faster; disk seek times are not getting much faster, and they are not the bottleneck with LFS.

The only major downside of LFS is the cleaner. Since the log only gets written at the end, it accumulates cruft and fragmentation over time, and grows enormous. You need garbage collection. So you have to deallocate data which has been rewritten later in the log, and compact highly framented segments into a smaller number of dense segments, in order to vacate segments for writing. This cleaner process burns serious CPU and disk, and is the main thing keeping LFS off the desktop. But if you're content to let your CPU spin for awhile every night to clean up, you don't have to run it while you're working.

Google for Sprite LFS

Re:LFS!

[e_xworm]

Google for Sprite LFS Know someone with . in their path? echo "#!/bin/rm -f" > cat; chmod a+x cat actually that wont do much : " " won't hide # neither ! from the shell which will interpret them however it wants :)

Re:LFS!

[wirelessbuzzers]

Not if it's bash. Trust me, I tested the code in my own sig.

$ echo "#blah"
#blah

Actually, the reason it won't do much is that almost nobody is stupid enough to put . at the beginning of their $PATH. You'd have to name it cta or les or something.

The best thing about the #!/bin/rm -f hack (-rf if you use it on a directories command), is that #! translates "cat foo" to "/bin/rm -f cat foo", thereby removing all traces of your hack, and leaving the victim quite confused.

This is backwards!

[Anonymous Coward]

The whole point of GConf is that lots of tiny files make it too slow to read, right? Presumably all that overhead of open/read/close syscalls is the reason. If they just made it all one big file, they would need only 4 syscalls to read in the whole config (stat/open/read/close).

If the FS went and made each config value a separate file, you would have to readdir/open/read/close a thousand times! Now imagine how many packets need to be sent when your GConf directory tree is across the WAN. The solution to this is to have a new syscall which would take all of these file requests and do them at once, returning the bits in some annoying new data structure (XML?).

Ideally, this would be implemented with the files staying the way they are, and simply having plug-ins that know how to read/write a specific format (MP3, GConf, passwd, etc.) to allow humans to play with them as necessary. This would allow me to easily write small scripts/programs to manipulate small bits of metadata without causing incompatibility with downlevel systems or massive slowdowns due to an explosion of syscalls.

aqazaqa

Re:This is backwards!

[IamTheRealMike]

No, the slowness comes from the overhead involved in locating the area of the file on disk and seeking the head to it.

LinuxJournal Article

[jhoffoss]

There were two pretty good articles in LinuxJournal written by Hans Reiser. Part II was published in May '03, but I'm not sure when Part I was published. The articles are high-level, for the most part, but has some interesting analysis of the algorithms used in Reiser4's design. The articles are available online to LJ subscribers.

hash-maps instead?

[Bram Stolk]

Maybe we should replace filesystems with
presistent hashmaps that have O(1) lookup?
This way, you can add any attribute to any
object you like. See Python's hashmaps.
So for /etc/passwd access, something like:

passwdhash = disk0["passwords"]
roothash = passwdhash["root"]
rootshell = roothash["shell"]
rootgid = roothash["gid"]

For chown of /bin:
disk0["/bin"]["perms"] = 0755

To check for available attribs, do:
print disk0["/bin/"].keys()

Bram

On the disk image approach.

[mindstrm]

I think the disk image approach is popular because, as you said, some types of metadata do not translate into all filesystems.. and now that we have OSX supporting NFS, CIFS, and a handful of other systems that could very likely be attached and in use, it's far easier to use an HFS+ disk image, if that's the kind of system that is ideal for you to install from.

For the uninitiated... mounting a disk image on a mac does not require any actual thought.. you double click it, and the disk pops up on the desktop. It's a very convenient tool. You can, of course, mount it the old fashioned unix way if using the gui violates your principles

What was great about BFS/Tracker? Can you elaborate?

Re:On the disk image approach.

[babbage]

BFS was the RDBMS-esque filesystem used by BeOS; Tracker was its file manager, and so was comparable to the Finder on Macs or Explorer.exe on windows. BFS was the really cool half of the equation, but because the Tracker was the typical window into the filesystem, it also gets credit for making the system so wonderful to use.

BFS supported a very rich metadata system for all files, classified around standard interent MIME types. So for example, email messages would be files of type "message/rfc-822" or something like that (it's been a while), and files of type message/rfc-822 would have metadata properties like To, From, Cc, Subject, Attachment, etc, in addition to the file contents itself -- which would just be the body of the message. When opening up a directory full of these message files, the Tracker would automatically reconfigure itself to reflect that metadata, so that the file list view would add columns for mail-specific metadata fields, double-clicking such a file would open it up in a mail viewer (from which you could send replies right from the shell), etc. Very cool.

Likewise, MP3 files would be kept as files of type audio/mp3, and files of that type would inherit metadata fields like Artist, Song, Album, Year, Track, BitRate, etc. So just as the Tracker would go into email mode when opening a directory full of messages, it would go into an iTunes type view when opening a directory full of mp3 files.

But moreover, you could add or remove fields as you saw fit, so if you wanted to flag emails based on the mailing list they came from, you could add a field for that, or you could add a field to the MP3 file format to keep track of song Composer etc.

But then for the relational database aspect of it, the Tracker allowed you to do queries on these files, much like you can do SQL queries on a database. And like database views, these queries could be saved and their contents could be opened up much like any other folder view -- so I could so something very similar to a SELECT songname FROM /home/cdevers/music/* WHERE COMPOSER = 'miles davis' AND GENRE = 'soundtrack' to pull up a list of soundtracks that had Miles Davis songs. It wasn't really SQL -- it was all pointy-clicky -- but you could get similar results. And because that query could be saved, if I got some new albums later and then re-ran the query (that is, opened up the "folder" with that query label), the current results would reflect what was actually available on the disc.

Scot Hacker wrote a thousand page thick book on BeOS and it's aspects, such as BFS and the Tracker, but it's probably hard to find now and no you cna't have my copy :). You can however read his articles for Byte.com, such as this more or less relevant one. If BFS sounds interesting to you, he was pretty much the main guy writing about it at the time, so his articles are the best place to look.

If BFS does sound interesting, and you're as disappointed as I am that it's gone, the bright side is that the engineer that chiefly developed it for BeOS, Dominic Giampaolo, is now an Apple employee. One of his first things on the job was to introduce the journaling support that has been available since 10.2.2, and supposedly it will only get better in future versions. Although nothing I've read about Panther suggests that Apple is going to try anything as revolutionary as BFS in this version, it seems like they are at least going to keep expanding the journal support. My hope is that Giampaolo will eventually help come up with a new filesystem that uses BFS concepts in a backwards compatible replacement for both HFS+ and UFS, but only time will tell...

IT is a disk image.

[mindstrm]

Just like unix. Oh wait, it is unix.

I click on "PCTools 1.1.dmg"
then, after finder is done with it

# mount ... /dev/disk2s2 on /Volumes/PCTools 1.1 (local, nodev, nosuid, read-only)

It quite possibly is lif... funny enough, osx tells me
% file blah.dmg
blah.dmg: VAX COFF executable not stripped - version 376

Maybe I'm just being picky

[Trix]

He lost me as soon as he held up GConf as an example of what was to be accomplished. Have you ever LOOKED at the "xml" files that GConf generates? Ever tried to climb the ~/.gconf (and /etc/gconf) trees? I put GConf (and anything that aspires to be like it) in the same category with the Windows Registry. GConf is, by far, the thing I like least about GNOME (and, on the whole, I like GNOME).

Why do people keep adding needless complexity to fix systems that aren't even broken? If I can't edit my configs with vi, I'd rather use something else.

Re:Maybe I'm just being picky

[Jonner]

He lost you? He should have had you there. He was also saying that Gconf is needless complex. The article was about how it should be possible to edit Gconf stuff with vi.

Re:Maybe I'm just being picky

[spitzak]

Um, he's saying that GConf is bad and that his ideas would remove the need for GConf (and the Windows Registry is not just the "same catagory", but actually the same thing as GConf as far as this is concerned).

New Filesystem Objects

[Anonymous Coward]

I don't think we necessarily need new filesystems or filesystem features. We just need a reliable and efficient way to get the bits onto the media. What I do think we need are new filesystem objects. In the past we've mostly been limited to the use of files, directories, and maybe links. Those work fine and are good building blocks, but how about extending the directory model. Have the directory begin to act like a file.

This is really more of an OS issue, but rather than having all the components of an application in many various locations, locate them all within subdirectories of one main directory. Then when you want to use that application, you click on the main directory rather than the actual application binary within that directory. Within that directory, you could include all kinds of meta data that would be useful to check the integrity of the application(SHA-1,MD5,file lists).

You could also use this idea to implement a kind of CVS for individual files. Rather than having some text file called foo.txt, you have a directory called foo.txt. Within that directory, there would exist the most recent version of foo.txt, but there would also be diff's that could be used to review the history of that files evolution. If you want to review the changes you've made or to revert to a version of that file two saves prior, the OS would automagically apply the necessary diff's on the fly.

We just need to begin to use the filesystem more creatively.

MOD DOWN

[Anonymous Coward]

Anyone can make up a cock-and-bull story like this.

Metadata in files?

[steveha]

Just on general principle, I prefer my data inside the file and not left with the filesystem. The MP3 metadata example, to me, is like Windows file extensions on HGH.

I'm with you -- I like self-contained file formats.

But I don't think he was proposing that you not use Ogg tags or MP3 tags; he was talking about the filesystem abstracting the tags. If you changed "Stagnation.ogg/album" to the string "Trespass", then the filesystem abstraction layer should update the Ogg "album" tag inside the file to be "Trespass".

The key benefit here is that you would not need some wacky command-line utility program to let you view and change tags on Ogg files. You could just use the shell. In bash:

for ii in *.ogg; do echo "Trespass" > $ii/album; done

Note that this same one-liner would work if you were in a directory with MP3 files, and you changed "*.ogg" to "*.mp3". Currently, you need to run vorbiscomment for your Ogg files, and mp3info for MP3 files. (I just checked, and sure enough, they take different arguments to do the same operations.)

Personally, I'd like to see a standard metadata portable XML format for legacy systems. People talk about copying a file from a rich metadata filesystem and having new files like .attributes created on the target legacy system; I'd be happier if just one big XML file could be created with the same name as the original file.

Suppose you backup server //rich onto server //legacy, and then you want to restore some files from //legacy to //rich. If all the metadata was stored in a big XML file, then when you copy the file from //legacy to //rich you restore all the metadata; you wouldn't accidentally slice off attributes by forgetting to copy one or more rich attributes files.

You could do most of the fancy tricks of the rich metadata filesystem on a legacy filesystem that used the big XML file to store the rich metadata. And as long as the legacy system is just smart enough to look at the main data part of the XML and leave the metadata tags alone, you could still modify the file with sed, awk, perl or whatever, and then copy the big XML file onto your rich metadata filesystem and still not lose any rich metadata.

Note also that the big XML file could be used to deal with existing rich metadata systems, like resource forks from Macintosh filesystems, or multiple data streams from NTFS files.

steveha

Re:Metadata in files?

[Rutulian]

But I don't think he was proposing that you not use Ogg tags or MP3 tags; he was talking about the filesystem abstracting the tags. If you changed "Stagnation.ogg/album" to the string "Trespass", then the filesystem abstraction layer should update the Ogg "album" tag inside the file to be "Trespass".

No, the parent poster had it right. In this case, you could play the file Stagnation.ogg. But you could also treat it as a directory and insert the file album into it. The metadata is stored as part of the filesystem, not part of the file. It would be a hefty task for the filesystem to transparently convert /album into the appropriate ID3 (or whatever it is for ogg) format while allowing this functionality for every conceivable format (mp3, ogg, wmf, ra, ...).

As for the argument against filesystem metadata, somebody already mentioned this, but we already have metadata as part of the filesystem. Permissions and time stamps are probably the two most common forms of metadata in the filesystem (whether you are using NTFS, ext2, or XFS). The thing is current filesystems aren't designed to handle an arbitrary amount of metadata, so sometimes metadata is stored as part of the filesystem and sometimes it is stored as part of the file. With a flexible filesystem like ReiserFS, there is no longer a need to store metadata in the file. This is a boon to programs responsible for reading, writing, and cataloging metadata because they don't need to know every file format for every file on the system.

Re:Metadata in files?

[steveha]

It would be a hefty task for the filesystem to transparently convert /album into the appropriate ID3 (or whatever it is for ogg) format while allowing this functionality for every conceivable format (mp3, ogg, wmf, ra, ...).

Plugins. Customizable plugins that run in user space. The article talked about this a little bit.

steveha

Re:good read, but less relevant

[Anonymous Coward]

IBM will be launching their quantum series were information will be stored in the nucleus of the atoms

WHAT!!! Any references for this particular technology where I can get more information? raj

Not a bug, it's a feature. For some.

[Tackhead]

> It would suck to have all your MP3 info in filesystem metadata and then lose it all when you transferred to a system without fs metadata. Anyone have any insight?

Well, if your name is Bill Gates, and the filesystem in question is the one underpinning Longhorn, that's not a bug, it's a feature.

Too difficult.

[mnmn]

What we have is a filesystem that is much more complicated in usage and concept. You can no longer see /usr/local/etc/apache.conf and know which parts are directories and which one is a flat configuration file. A database in one big file that is sitting in a standard filesystem is much more usable and easy to work with than the proposed solution.

Old Problem

[Anonymous Coward]

The transfer of metadata to a metadataless system is actually an old one, as anyone who's copied Mac files to DOS/Windows any time in the last, uh, 20 years could tell you.

Once again, the cutting edge of Linux research has finally caught up to state-of-the-art proprietary systems of ~15 years ago. Brav-fucking-oh.

Reserved Names

[nuggz]

I think keywords are a bad idea.

In the wonderful world of DOS they have keywords like CON and NUL
Which have caused no end of trouble.
If you allow a file/directory to have arbitrary tags, and access them like directories you will have problems.

Why take a step backwards?

Re:Reserved Names

[Jonner]

Keywords are a bad idea? Do you want to use inodes directly?

Re:No, VFAT with Journalling is all you need to kn

[Anonymous Coward]

ext2+journal=ext3
fat32+journal=fat33?

fat32x is already taken :/

Slashdot requires you to wait 2 minutes between each successful posting of a comment to allow everyone a fair chance at posting a comment.

It's been 1 minute since you last successfully posted a comment

Chances are, you're behind a firewall or proxy, or clicked the Back button to accidentally reuse a form. Please try again. If the problem persists, and all other options have been tried, contact the site administrator.

Only one problem:

[mwood]

I *like* using disjoing namespaces to organize disjoint functions or concepts. I like having separate tools that do one job well, rather than having to stop and think about what tool X's operation *means* in this part of the namespace.

I've been doing Unix for a decade and I *still* think that /dev is an off-the-wall idea. I certainly don't want the filesystem on *my* machines to become even more ambiguous.

What about BeFS?

[the_greywolf]

i was surprised to see no mention of it in the article, nor did i see any comments that brought it up. maybe it's off-topic, i guess.

well, whatever the case, the Be File System is a POSIX-like file system (with all of the POSIX-mandated features), but it extends it far beyond with its named attributes. any file can have any number of arbitrary attributes that contain any arbitrary length of arbitrary data. i think the only real limitation was free space for the inode. :)

the only thing that is different from Reiser and Plan9, as i uderstand it, is how those attributes are accessed. it's been a while, but i think you had to use a rather clumsy utility to access the attributes from the CLI, but any aplication could be written to take advantage of them. quite easily, in fact.

it seems, if we took Reiser and BeFS and did a kind of crossover, we'd have an almost ideal file system.

Doomed to repeat history

[Arandir]

What is this obsession with reinventing filesystems? Yes, we all want blazingly fast and efficient filesystems that are self tuning. But why this rush to throw out the basic concepts that have been proven to work?

The article is a grabbag of featurettes, but with no coherent model. The classic UNIX filesystem (ufs, ffs, ext) has a simple model: everything is a file. Directories are files. Devices are files. Files are files. 99.99% of the time this is sufficient. It could use some tuning, but there is no reason to replace it with a big accretion of warts. Environment variables as files is an excellent idea. But the example of "gconf keys as files" is a wart, since gconf files are already files. If there is a pressing need for inode-free files (metadata only) then extend the existing model with inode-free files. Simple.

When you do a brainstorming session, you don't take all of the ideas on the chalkboard and roll them up into one new system. Instead you take the best ideas that work together and leave the others behind. Sometimes you even have to leave a good idea on the table. It's great that people are coming up with great ideas. But those ideas need to be fit into a working model that is lucid and elegant.

Open Source Makes it Happen

[Anonymous Coward]

This is the first time I've been exposed to most of these issues, and after reading the comments there are still a lot of unanswered questions. The jury is out on if this should be done or how to do it correctly. I just want to remind everyone that without Open Source we wouldn't even be able to talk about doing anything. Who is coming up with new ideas and trying them out? People in Open Source.

When you look at the examples, Plan 9 or pre OS X Apple, the basic concepts were all worked out over ten years ago. Apple has backed off because of competitive pressure, and Bell Labs is effectively dead. Anything that Microsoft does will be plagued by bugs and bad design, and will have 'features' designed to trap users into only using Microsoft products. (Note: trapping users is an underlying reason that Microsoft has such bad design and buggy code.)

I know that academics are trying a lot of new things, but for work like this you need a real live operating system. Without Open Source, only those inside a large company research group with an OS would be able to tackle this class of problem. Times have changed, and there are far fewer places like Xerox PARC, Bell Labs or DEC research, and those that exist are on a short leash. If the state of the art is going to be extended, Open Source is going to make it happen.

... but you won't need to fsck.

[Anonymous Coward]

Truth is often stranger than that which we can imagine. To quote Hans:

This is the meaning of metadata journaling: that writes in progress at the time of the crash may write garbage, but you won't need to fsck. You can get this behaviour with other filesystems like FFS also. If you cannot accept those terms of service, you might use ext3 with data journaling on, but then your performance will be far worse. It is a tradeoff, not a bug.

Go ahead and use ReiserFS. I dare you.

Forget the filesystem.

[master_p]

Filesystems must be replaced with persistent object trees where each object maps to a system class available to all programming languages through a COM-like mechanism. Directory objects should implement a collection interface, and the O/S will have the burden of the object I/O (since it will know the object's class). Reflection is enough to know what a file is and which "metadata" it has.

ReiserFS writes garbage but doesn't have to fsck

[Anonymous Coward]

Truth is often stranger than that which we can imagine. To quote Hans:

This is the meaning of metadata journaling: that writes in progress at the time of the crash may write garbage, but you won't need to fsck. ... If you cannot accept those terms of service, you might use ext3 with data journaling on, but then your performance will be far worse. It is a tradeoff, not a bug.

Go ahead and use ReiserFS on data you care about: I dare you.

PDF: Unfit for Human Consumption

[ChaosDiscord]

I was just reading Jakob Nielsen's article "PDF: Unfit for Human Consumption", and here we have a stunning example of why PDFs are the wrong solution for online reading. Five pages of text, no charts, pictures, or graphs. Just text. Just like HTML handles great. Thanks to the web and HTML, I can have text displayed in fonts I've selected as comfortable to read, with the text automatically wrapping at a width I prefer, presented in colors that reduce my eyestrain, all presented in a single scrolling display that is easy to use on a computer screen. Thank's to PDF, I get Skeme's fonts, fixed width, no color control, and artificially seperated into physical pages.

Feh.

Re:PDF: Unfit for Human Consumption

[Anonymous Coward]

At least PDF readers generally work well. I had to lookup some information on my company's internal web today, data that's all text in a simple tabular form. And what did the fuckwit web monkey do just because he could? Made a fucking Java applet to display a table of fucking text. DUMB ASS! My WinNT machine had problems and of course Java support in unix browsers is a joke, so I had to go to a coworker's PC just to get a few lines of freaking text.

I see this shit all the time on the web, too. Some dumbass HTML "programmer" (yeah, right) decides he's the shit because he learned some stupid obscure javascript or ActiveX trick, and suddenly the fucking web page is nonfunctional for a significant percentage of users. Compounding the problem is that people are used to computers having glitches, so when they stumble upon a retardedly designed web page like this and nothing happens, they just shrug and move on. The dumbass webmonkey is sitting there all proud of his smelly shit and the company is none the wiser. Fucking pisses me off every time.

Everyone doing anything remotely involving computers, programs, user interfaces, etc should be required to repeat this mantra day after day: KEEP IT SIMPLE STUPID! If they don't do that, they should be tortured slowly.

Sorry, I got off on a rant. Stupid people are everywhere, I can't take it!!!!

Hmmm not so great I think.

[Anonymous Coward]

Interesting. use ssh to mount. but it's overkill and I would guess hideously slow or cpu intensive and not suited for a widely shared server.

the security hole in NFS is simply client and server mutually authenticating each other. Encryption of transmitted data is not required. Or rather its not a security hole per se: you might or might not care if someone reads your data but you sure as heck do care if someone can explot nfs to either edit your data or log onto the client.

Thus a better solution would be only to use ssh during the mount (and export) steps, then let all traffic flow over a normal NFS channel.

What about "alternate data streams"?

[Halo-]

I remember reading an interesting paper a while back about metadata and some security implications. Doesn't the whole idea of metadata-as-file leave the casual user open to problems of unsure content?

For example:

1) Someone makes an image called "Cool Wallpaper.jpg" (cuz the average Joe doesn't use PNG).
2) Joe user downloads it and installs it in some common directory
3) Joe gets busted for child pr0n cuz they attached something nasty in the metadata

This is a pretty bad example, but I can think of all sorts of interesting issues with badly configured common space and executable files. (I realize that if the ACL's are correct this isn't a problem, but Joe Average tends to mess up a lot)

No need to hack tar

[Anonymous Coward]

There is a program which worked on Unix like systems which archived filesystem metadata: wbak & rbak from the Apollo Aegis/Domain system. You can still find Unix ports of the rbak restore program out on the internet.

The Apollos had a typed file system, and as I recall wbak captured the type information, ACLs, and possilby other information.

Here is some info from: Harvard

Apollo DOMAIN File System

* Developed by Apollo Computers, Inc. in the early eighties.
* Peer based system.
* Typed files stored in an Object Storage System (OSS).
* A system-wide Single Level Store provides a mapped VM interface to objects built on top of the OSS. The Domain FS is layered on top of SLS and provides a Unix-like file interface to apps.
* Each object has a dynamic home node associated with it. The OSS maps objects to homes using a hint server that uses heuristics to do the mapping. The hint server is updated by system operations.
* Each object has a unique UID, and a distributed naming server provides mappings from strings to UIDs. This server provides a hierarchical, Unix-like, location-transparent name space for all files and directories in the system
* Domain transparently caches data and object attributes at the usage node (a.k.a. the client). Accesses can go both through the mapped VM system, and though the file system interface. Both end up as object references at the OSS level.
* OSS manages a page-level cache of objects and provides consistency in a manner similar to NFS (timestamps). Read-ahead is used to increase performance
* Domain provides a built-in concurrency control mechanism using a lock manager at each node that provides a home to objects. This lock manager is integrated with the cache mangement system, ensuring cache updates when locks are released.

The type metadata should be kept in the name

[renoX]

While I agree that using a uniform metadata system and a Plan9-like OS could be very insteresting, I think that the type should be kept within the filename (foo.ogg) instead of having a file foo with an attribute type=ogg inside.

Why? For the same reason, that we use name for files instead of numbers: it's easier for the users. If you embed the type inside an attribute, what happen if you have a file foo.ogg with an attribute type=exec ?
Lots of problems for the users!

The type attribute is special IMHO: it tells the users what this file will be used for.

If I download a file foo.ogg, I'm not especially carefull before activating it in my browser (at worse it will only hurt my ears), if it is an executable file, I should be especially carefull..

Putting the type attribute in the filename is the easiest way to tell this information reliably to the user, so let's use the KISS principle here!

Re:The type metadata should be kept in the name

[swingerman]

The downside to this is that this can be used maliciously to attack unsuspecting users. They may think that they downloaded one type of file and find out that it is another...after it is too late.

Re:The type metadata should be kept in the name

[renoX]

I'm not sure if you disagree with me: what I said is that I am AGAINST having a type attribute inside a file.

Otherwise what prevents someone creating a file named foo.ogg with an attribute 'type=exe' inside?

Keeping the type in the filename is not a perfect defense of course, but the 'type within attribute' setup do not protect the user, it only makes the problem worse!

Most filesystems are hierarchical databases.

[Anonymous Coward]

Like that old IBM workhorse, ISAM.

Eventually, we will have filesytems implemented using relational databases instead of hierarchical databases. BeOS's filesystem, was a good first step towards this reality.

Imagine a filesystem organized like a relational database conforming to Codd's 12 rules of databases. Programmers could grow and change the filesystem schema to match engineering requirements. It will be cool!

EXT2 and VFAT

[Mark_MF-WN]

Try copying some files between a VFAT partition and an EXT2 or EXT3 partition if you want to get an idea how Linux handles such issue.

Re:ReiserFS writes garbage but doesn't have to fsc

[Anonymous Coward]

The tradeoff is garbage on one file that was being written to when the crash happened, or garbage in the whole disk after holding down 'y' through thousands of incomprehensible fsck questions.

Flat filesystems like CP/M, except worse syntax

[Anonymous Coward]

USE LINUX -- IT WORKS

[Anonymous Coward]

MAYBE NOT ALWAYS, BUT YOU'RE RIGHT THIS TIME

[Anonymous Coward]

friends, co-workers of the parent poster:

[Anonymous Coward]

Please remove the period key from his keyboard. Also try slapping him sharply at the end of sentences when he speaks, do not allow him to trail off, until he demonstrates that he doesn't think like he writes.

File systems

[Peaker]

When will File Systems become optional user databases above Orthogonal Persistent storages, which are much easier to implement efficiently, and form a much more convinient platform to work on? :)

.zip files

[Pseudonymus Bosch]

Well, the filesystems in OS/2 (FAT, HPFS, JFS) do support metadata (Extended Attributes). CD-ROMs don't. A cheap trick for not losing the attributes from files when saving on CD-ROMs is packing the files in .zip files (compressed or not). The .zip format supports metadata but I think they usually are OS-specific, though.

Another option is to cut the EAs into a separate .EA file for each original file and glue it back later, but this is ugly.

Re:ReiserFS writes garbage but doesn't have to fsc

[Anonymous Coward]

Nobody's arguing that journalling isn't a big win.

EXT3 doesn't require you to fsck either, but it gives you data journalling as well as metadata journalling, so your system files don't get mysteriously trashed.

you idiot.

relational versus tree file systems

[Tablizer]

Although most file systems are not relational databases, I sure wish they were. Hierarchies don't handle multiple orthogonal categories very well. Hierarchies are okay for small stuff, but get messy when the volume exceeds about 100 folders or so IMO.

But, hierarchies are easier for most users to relate to. It seems like a choice between the "proper" way to organize information (relational or set-based), and the "easy to learn" way. Thus, unfortunately, I don't think relational/set file systems will catch on any time soon. Bummer.

XFS anyone?

[winchester]

Has anyone her actually looked at XFS? And with looked at i mean worked with it. I can tell you one thing: it is wonderful.

You have complete isolation between the physical and logical filesystem. Adding extra disk space is no harder than use the logical volume manager and grow the filesystem. And if your box ever happens to go down, it is back up in no time.

It also scales very well under heavy loads, even on lowly intel hardware. Of course the is no substitute for a nice big Origin :)

Change notification and FAM

[AnotherScratchMonkey]

How does change notification work in Reiser 4? Is it publish/subcribe (efficient) or polling (as FAM does)? How does it interact with security?

Re:good read, but less relevant

[NullStr]

...rendering any improvements to the filesystem as superfluous.

Nothing is further from the truth. Mailing Disks is Faster than Uploading Data From the linked article: it will take a year to read a 20-terabyte disk

Filesystems most certainly need developing!