Music Filesystems?

Cutriss asks: "I suspect the odds are fairly good that a high percentage of Slashdot readers have a folder, disk partition, network share, or even a hard drive dedicated to storing Oggs or MP3s. Given that this is the case, wouldn't it be beneficial to develop a filesystem optimized for such a use? For instance, in Windows, Microsoft has special folder layouts for folders declared to have digital music or images in them. The digital music folders don't bother with file permissions or any unnecessary data, but instead display your songs with the appropriate ID3 tag information instead. If a filesystem were developed that intrinsically indexed its files by ID3 tags (or similar mechanisms), wouldn't this be an ideal structure for storing and searching through your album collection? And, more to the point, are there any such efforts under way to develop such a beast?"

Comments

Metadata

[SteveX]

We don't need a special purpose filesystem for this; just one that supports metadata. I know NTFS does; do any of the common Linux filesystems?

Either way you're going to need a custom tool to do searching and whatnot, the custom tool might as well just index the stuff in the ID3 tag and work with an existing filesystem.

- Steve

Re:Metadata

[foobar104]

We don't need a special purpose filesystem for this; just one that supports metadata.

There are a lot of reasons why filesystem-level metadata is harder than it sounds. (XFS supports it, incidentally, though extended attributes. There are command-line tools, and also a C API. Dead simple.)

Filesystem-level metadata is usually stored in the inode itself. If it's not stored literally inside the inode, then it's tightly coupled to the inode. This is good; it makes it fast. Reading a big chunk of metadata (say, 128K) is about as fast as doing a stat(). So that's good.

But most applications do something tricky with their data files. Say, Photoshop for example. When you're working on a file (foo.tiff) in Photoshop and you tell it to "save," Photoshop doesn't truncate and write to the file foo.tiff. Instead it writes to a temporary file. When the write is complete, it unlinks foo.tiff and renames the temporary file "foo.tiff." This is also good, because it makes it very unlikely for a program or system crash to destroy foo.tiff, even if saving the file takes a really long time. A crash or other interruption during the save process leaves foo.tiff just as it was; at worst, it leaves a dangling temporary file laying around.

Here's the catch: when Photoshop saves that new temporary file it gets a new inode. That inode has no metadata in it. All the metadata associated with foo.tiff is in foo.tiff's inode. When the temp file has been written and Photoshop unlinks foo.tiff, that wonderful rich metadata disappears in a tiny puff of blue smoke.

Sure, there are ways to work around this. We could implement a system call like clone() or something that allows an application to get a new, empty file open for writing that has all the filesystem metadata from another file automatically copied to it. Or we could do any number of other clever things. But all of them-- at least to my knowledge-- would involve changing end-user applications. That makes it tough.

ID3 tags are different, because they're stored as part of the MP3 file itself. Copying the file's data, using any method you might choose, will also copy the ID3 tags. But inode-based filesystem-level metadata is something else entirely.

Not a specialized one, no

[melquiades]

I don't see why you'd want a specialized system is a good idea. Instead, we should ask what general principles might make filesystems work better with music. The obvious ones for music files include:

• performance (obviously),
  
• handling of massive storage (pushing the latest n-bit addressing limit,
  
• customizable file metadata (for storing tag information, among other things), and
  
• query facilities for the metadata.
  

That all sounds like BeFS to me. -- which, lo and behold, was discussed on Slashdot just the other day!

What other general features might be good for the specific application of music?

gosh

[pb]

Gee, yeah, who would want file permissions on files, anyhow? Feel free to delete all my MP3's, since they're just music, after all.

Usually this sort of presentation is up to the application, not the filesystem, and Windows is no exception here. The thing presenting the "My Music" folder with ID3 tags and whatnot is the shell (explorer.exe); it's a graphical file browser. There are many of them available for Linux as well.

And personally, I don't really care if my file browser supports this, as long as my MP3 player can read ID3 tags...

Please generalize that idea

[lightspawn]

Music filesystems? Image filesystems? Movie filesystems? Document filesystems?

And please, please, at least store the file's mime type, don't play the extension guessing game.

Web server: let's guess what the file's mime type is based on the extension, and send it.
Browser: Let's guess what this file is, given the extension AND the mime type.
Stupid computers!

metadata

[Mad Marlin]

The problem with metadata information is that it is too inflexible. The ID3 tag is an excellent example. It contains title, artist, album, comment, year, track number, and genre. What about classical music, where you would want a composer field? Items not from a CD don't need album or track number.

A better solution is to take advantage of the flexibility of a modern Unix filesystem, where any charater (excepting '/') can bu used in a file name, including the newline. There is no reason why you couldnt name a file:

Title: Symphony 2
Composer: Bach, Johann Sebastian
Composition Date: a long time ago
Performing Artists: St. Louis Symphony Orchestra
Performance Date: not very long ago
Genre: Classical
.
.
.
Format: .mp3

The Unix filesystem is adequate for the job, metadata is a curse disguised as a blessing.

Metadata directories

[pthisis]

Bear in mind that one of the core driving philosophies that has allowed Unix to survive and evolve is "everything's a file", where a file is a stream of bytes. Attaching metadata directly to a file gets to be a nightmare when you start copying files over the network, dealing with editors that create new files rather than modify in place, backing things up, etc.

NeXT had a great solution to this: instead of just having a file "mysong.mp3", have a directory "mysong" that includes files like data.mp3, title.txt, etc. The file browser presented this to the user in a simple UI (e.g. the directory would have the "mp3" icon in the file browser, double-clicking it would play the song in the mp3 player, displaying the title.txt, etc). But traditional file transfer and backup tools worked just fine, and you could use standard tools (your regular editor, copy program, etc) instead of crufty special purpose ones (regedit, or binhex, or whatever) to work with the metadata.

Think of it this way: you don't want to just have 2-3 defined metadata fields, you would like to be able to add arbitrary new ones in a flexible way. And you want the contents of each to be flexible, without size limits or such. So you want a way of associating names with values--wait, we have that, it's called a filesystem. And you want a way of grouping a bunch of name-value pairs together--wait, we have that, it's called a directory. You'll either end up reimplementing a filesystem, badly, as your metadata, but need special tools to view/edit the metadata or move it around (a la resource forks or the registry), or you'll end up going with a NeXT-like solution and use the filesystem itself (which deals quite nicely with things like this).

Sumner

Re:Metadata directories

[pthisis]

Every key name/value pair--being a file--will take up one cluster, regardless of how much data it actually contains

Not if your filesystem has decent support for small files, like e.g. ReiserFS or some of the recent ext2 devel branches. This is one thing ReiserFS definitely got right (though I'm less sure about the direction they're taking for ReiserFSv4).

As I said, it's a workable approach for grafting extensible metadata onto an existing FS, but it's not how you would design an ideal from-scratch FS.

It's definitely how I would design an ideal from-scratch FS. Metadata in the FS is a really horrible hack that goes against all the namespace unification efforts that OS designers have been making in the last 10-15 years.

Sumner