Music Filesystems?

Cutriss asks: "I suspect the odds are fairly good that a high percentage of Slashdot readers have a folder, disk partition, network share, or even a hard drive dedicated to storing Oggs or MP3s. Given that this is the case, wouldn't it be beneficial to develop a filesystem optimized for such a use? For instance, in Windows, Microsoft has special folder layouts for folders declared to have digital music or images in them. The digital music folders don't bother with file permissions or any unnecessary data, but instead display your songs with the appropriate ID3 tag information instead. If a filesystem were developed that intrinsically indexed its files by ID3 tags (or similar mechanisms), wouldn't this be an ideal structure for storing and searching through your album collection? And, more to the point, are there any such efforts under way to develop such a beast?"

Comments

Metadata

[SteveX]

We don't need a special purpose filesystem for this; just one that supports metadata. I know NTFS does; do any of the common Linux filesystems?

Either way you're going to need a custom tool to do searching and whatnot, the custom tool might as well just index the stuff in the ID3 tag and work with an existing filesystem.

- Steve

Re:Metadata

[LordNimon]

HPFS does as well, although under OS/2 the "extended attributes" are limited to 64K bytes. I don't know if the Linux HPFS driver has the same limitation.

Re:Metadata

[foobar104]

We don't need a special purpose filesystem for this; just one that supports metadata.

There are a lot of reasons why filesystem-level metadata is harder than it sounds. (XFS supports it, incidentally, though extended attributes. There are command-line tools, and also a C API. Dead simple.)

Filesystem-level metadata is usually stored in the inode itself. If it's not stored literally inside the inode, then it's tightly coupled to the inode. This is good; it makes it fast. Reading a big chunk of metadata (say, 128K) is about as fast as doing a stat(). So that's good.

But most applications do something tricky with their data files. Say, Photoshop for example. When you're working on a file (foo.tiff) in Photoshop and you tell it to "save," Photoshop doesn't truncate and write to the file foo.tiff. Instead it writes to a temporary file. When the write is complete, it unlinks foo.tiff and renames the temporary file "foo.tiff." This is also good, because it makes it very unlikely for a program or system crash to destroy foo.tiff, even if saving the file takes a really long time. A crash or other interruption during the save process leaves foo.tiff just as it was; at worst, it leaves a dangling temporary file laying around.

Here's the catch: when Photoshop saves that new temporary file it gets a new inode. That inode has no metadata in it. All the metadata associated with foo.tiff is in foo.tiff's inode. When the temp file has been written and Photoshop unlinks foo.tiff, that wonderful rich metadata disappears in a tiny puff of blue smoke.

Sure, there are ways to work around this. We could implement a system call like clone() or something that allows an application to get a new, empty file open for writing that has all the filesystem metadata from another file automatically copied to it. Or we could do any number of other clever things. But all of them-- at least to my knowledge-- would involve changing end-user applications. That makes it tough.

ID3 tags are different, because they're stored as part of the MP3 file itself. Copying the file's data, using any method you might choose, will also copy the ID3 tags. But inode-based filesystem-level metadata is something else entirely.

Re:metadata

[SIGFPE]

The filename is metadata. What you're proposing is merely a clunky way to exploit the minimal metadata the Unix already supports.

Re:metadata

[President Chimp Toe]

That is as ludicrous as the original question, sir. But i like it!

Instead of designing a new FS, writing drivers and apps, we could use your kludge.
A short script could extract the idv3 from the mp3s in folder A, and copy them accross to partition B with a stupid filename. The stupid filename is therefore all the metadata you could ever want to contain.

Then all you need is a few apps to do whatever you might want with this metadata...... erm, ideas people?

Re:metadata

[mosch]

Okay, so what do you use to list the year of composition? Do you use the original release year, or oh wait... no, that's defined as the original release year of the recording, not the piece itself. What about pieces which are known to be written in one year, first performed in another, first recorded in another, and remastered in another?

ID3v2 is still a standard written by people who don't know how classical music works, and don't really seem to care. Hell, they have a place to list the people in pop music's album credits, but they don't have a way to note that a track may be a single movement of a larger composition, the closest you can come is noting the original track number on the cd, which isn't what you really care about anyway.

Re:Metadata

[foobar104]

My god. I spent ten minutes writing a lengthy reply about how external metadata databases are inherently flawed due to the primary key problem. Then I realized that you're an idiot. Sorry. My mistake.

Re:Metadata

[foobar104]

First of all, New Zealand is beautiful. I am an American, as it happens, but my company has an office in Sydney and I have a business partner in Auckland. I spend about half my time in that general area of the world. Wish I could spend more.

But the fact that you live in a nice place doesn't make you any less of an idiot. I don't mean that in a perjorative sense; I just mean it literally. Your posts and journal entries reveal you to be a person of less-than-average insight. Nothing personal.

Please, by all means, prove me wrong. Share some piece of wisdom that makes us all stop and think. Say something to make me reply, "Yeah, that's a good point." Throw me a bone here. My opinion will change quickly in the face of new information.

I really don't have time for petty personal disputes whenever there is a procedural expert or 'ideas person' around, I'm just trying to get things done & I have no time to waste on trivial matters raised during consultation.

Well, good luck with that. Let us all know how it works out for you.

Not a specialized one, no

[melquiades]

I don't see why you'd want a specialized system is a good idea. Instead, we should ask what general principles might make filesystems work better with music. The obvious ones for music files include:

• performance (obviously),
  
• handling of massive storage (pushing the latest n-bit addressing limit,
  
• customizable file metadata (for storing tag information, among other things), and
  
• query facilities for the metadata.
  

That all sounds like BeFS to me. -- which, lo and behold, was discussed on Slashdot just the other day!

What other general features might be good for the specific application of music?

gosh

[pb]

Gee, yeah, who would want file permissions on files, anyhow? Feel free to delete all my MP3's, since they're just music, after all.

Usually this sort of presentation is up to the application, not the filesystem, and Windows is no exception here. The thing presenting the "My Music" folder with ID3 tags and whatnot is the shell (explorer.exe); it's a graphical file browser. There are many of them available for Linux as well.

And personally, I don't really care if my file browser supports this, as long as my MP3 player can read ID3 tags...

Please generalize that idea

[lightspawn]

Music filesystems? Image filesystems? Movie filesystems? Document filesystems?

And please, please, at least store the file's mime type, don't play the extension guessing game.

Web server: let's guess what the file's mime type is based on the extension, and send it.
Browser: Let's guess what this file is, given the extension AND the mime type.
Stupid computers!

Stupid Windows My Music directory

[jspey]

Another problem that stems from Microsoft assuming it always knows best: You can't change the My Music folder to act like a normal folder. Sometimes when I download music I don't pay attention to the exact filename so I like to sort my music files by date to find the ones I just downloaded. Not an option with the My Music directory. Bleh.

Mr. Spey

Re:Stupid Windows My Music directory

[jspey]

I moved my My Music folder from wherever the default is to a different drive. The next time I clicked on a My Music shortcut WinXP looked for and found the moved My Music folder. Something similiar might work for the My Pictures Directory. Dilbert: You're only hearing what you want to hear!

PHB: Yes, I do look thinner today. It must be the situp I did yesterday.

Mr. Spey

metadata

[Mad Marlin]

The problem with metadata information is that it is too inflexible. The ID3 tag is an excellent example. It contains title, artist, album, comment, year, track number, and genre. What about classical music, where you would want a composer field? Items not from a CD don't need album or track number.

A better solution is to take advantage of the flexibility of a modern Unix filesystem, where any charater (excepting '/') can bu used in a file name, including the newline. There is no reason why you couldnt name a file:

Title: Symphony 2
Composer: Bach, Johann Sebastian
Composition Date: a long time ago
Performing Artists: St. Louis Symphony Orchestra
Performance Date: not very long ago
Genre: Classical
.
.
.
Format: .mp3

The Unix filesystem is adequate for the job, metadata is a curse disguised as a blessing.

Nautilus

[xiox]

Nautilus has a music view, where it shoes the id3 tags as if it was an album sleeve. I'm not sure whether this is searchable though.

Use a database

[janke]

Setting up a specialized filesystem for music storage seems both overkill in terms of implementation and maintenance issues, and underkill in terms of features. What would be more appropriate is a database table schema that stored music and lots of metadata (Artist info, title/album/recording/remix/venue info about a track, genre information, record label, user ratings/comments) about different recordings, storing the recordings themselves as either BLOBS of mp3s/oggs or (probably better) indexes into a normal filesystem containing the mp3s. Write a quick table browser to go through the collection, and construct a bunch of SQL queries to generate playlists by artist or genre.

Not exactly a good idea

[biglig2]

I suspect having a filesystem optimized for one specific data type is possibly a bad idea. Every so often I need to temporarily drop a file into my MP3 partition for some reason.

Also you need to store .txt and .jpg files in a lot of music folders (track listing and cover art)

Anyhow, my main point was this: Can I be the first person to point out that BeFS was/is superb at this?

Metadata directories

[pthisis]

Bear in mind that one of the core driving philosophies that has allowed Unix to survive and evolve is "everything's a file", where a file is a stream of bytes. Attaching metadata directly to a file gets to be a nightmare when you start copying files over the network, dealing with editors that create new files rather than modify in place, backing things up, etc.

NeXT had a great solution to this: instead of just having a file "mysong.mp3", have a directory "mysong" that includes files like data.mp3, title.txt, etc. The file browser presented this to the user in a simple UI (e.g. the directory would have the "mp3" icon in the file browser, double-clicking it would play the song in the mp3 player, displaying the title.txt, etc). But traditional file transfer and backup tools worked just fine, and you could use standard tools (your regular editor, copy program, etc) instead of crufty special purpose ones (regedit, or binhex, or whatever) to work with the metadata.

Think of it this way: you don't want to just have 2-3 defined metadata fields, you would like to be able to add arbitrary new ones in a flexible way. And you want the contents of each to be flexible, without size limits or such. So you want a way of associating names with values--wait, we have that, it's called a filesystem. And you want a way of grouping a bunch of name-value pairs together--wait, we have that, it's called a directory. You'll either end up reimplementing a filesystem, badly, as your metadata, but need special tools to view/edit the metadata or move it around (a la resource forks or the registry), or you'll end up going with a NeXT-like solution and use the filesystem itself (which deals quite nicely with things like this).

Sumner

Re:Metadata directories

[uradu]

The approach you describe is a workable solution for existing file systems, but hardly an ideal solution. Every key name/value pair--being a file--will take up one cluster, regardless of how much data it actually contains. This could easily be 2K or 4K just to store a boolean. Besides, you then still have to worry about doing full text indexing if you want to perform meaningful queries on all these "metadata" files. As I said, it's a workable approach for grafting extensible metadata onto an existing FS, but it's not how you would design an ideal from-scratch FS.

Re:Metadata directories

[pthisis]

Every key name/value pair--being a file--will take up one cluster, regardless of how much data it actually contains

Not if your filesystem has decent support for small files, like e.g. ReiserFS or some of the recent ext2 devel branches. This is one thing ReiserFS definitely got right (though I'm less sure about the direction they're taking for ReiserFSv4).

As I said, it's a workable approach for grafting extensible metadata onto an existing FS, but it's not how you would design an ideal from-scratch FS.

It's definitely how I would design an ideal from-scratch FS. Metadata in the FS is a really horrible hack that goes against all the namespace unification efforts that OS designers have been making in the last 10-15 years.

Sumner

Re:Metadata directories

[uradu]

> It's definitely how I would design an ideal from-scratch FS.

Well, the trend seems to be to make the whole FS a relational database anyway. It will be interesting to watch FS developments over the next five years. Of course, the initial efforts seem to be towards a relational DB FS on top of a legacy FS (seems that's what MS is doing with their SQL IFS), but long-term I guess you would see the FS store the DB straight to the metal.

Re:Metadata directories

[pthisis]

Good call on the single-cluster angle, but the folks at Alesis [alesis.com] already beat you to it.

It's not an issue of getting beat to it, it's a really old idea. Heck, ReiserFS was hardly the first and they've done it for over 5 years. ext2 hasn't, but there is some movement in that direction.

Sumner

Been there, done that with BeOS

[Rude Turnip]

(and got the t-shirt)

BeOS (god rest its soul) uses the Be Filesystem, or BFS. It is a 64-bit journalling and indexing filesystem. It is essentially a filesystem that works much like a database. You can attach any number of arbitrary attributes to any file. Filetyping is handled by MIME filetypes, which are also stored as attributes. The default BeOS installation includes all of the usual attributes for music files and it is very easy to add many more. The BeOS "find file" tool is not unlike a database query tool.

In terms of speed, file searches are done almost instantaneously, even when searching through several gigs of data.

Which version of windows?

[man_ls]

I'm running Win2K Pro SP2 SRP1 and I don't have any special folders for digital media...let alone displaying with ID3 tags!

This sounds like it's a Windows XP thing, and we all know that XP is the devil.

Nautilus

[mobydobius]

For instance, in Windows, Microsoft has special folder layouts for folders declared to have digital music or images in them. The digital music folders don't bother with file permissions or any unnecessary data, but instead display your songs with the appropriate ID3 tag information instead

I don't think this has anything to do with the Windows file system. Its just how explorer presents data in certain folders. Nautilus does the exact same thing, for instance. It displays bitrate, time, artist, and presents a mini-player at the bottom of the folder view. I assume it gets the artist info from ID3 tags.

Re:WORM Advantages

[tps12]

This depends on from where your mp3's come. If you download them, they most likely are obtained a little bit at a time, and probably need to be fixed (ID3 tag inaccuracies are very common) or deleted (so are incomplete or corrupted files) at some point after they are written.

sounds like a job for plan9

[DrSkwid]

with virtual file systems being de rigeur this is exactly the sort of application plan9 would be good at.

But we're stuck with 23 char filenames until the eagerly awaited 9p2000 as I discovered the day I statrted to move my mp3s over, doh!

Store music separately

[EvlG]

Why not store the music separate from the meta data, in big hunks of the disk? Like one poster mentioned, the music doesn't really change once deposited in the filesystem; only the metadata does. So we could store music in arbitrary places on the disk, and then have the metadata categorize it, give it names, etc.

The filesystem would then transparently re-create the MP3 file when copied out to move it across the network, zip file, etc... by attaching the meta-data back onto the music stream as it is ready from disk.

Thoughts?

Uniform attributes

[os2fan]

One could have a standard attribute file that can be automatically updated as files are moved.

One idea would be to do this on a directory by directory basis (like 4dos descriptions), this would overcome the mapping problem. If each file type has its own header, then you could store current and future formats in this.

One can have a utility to go through and look at files by type, eg like a "viewer" dll.

Backing up such a file would allow the support of this feature on cdroms under different operating systems.