Slashdot Mirror
Music Filesystems?
Cutriss asks: "I suspect the odds are fairly good that a high percentage of Slashdot readers have a folder, disk partition, network share, or even a hard drive dedicated to storing Oggs or MP3s. Given that this is the case, wouldn't it be beneficial to develop a filesystem optimized for such a use? For instance, in Windows, Microsoft has special folder layouts for folders declared to have digital music or images in them. The digital music folders don't bother with file permissions or any unnecessary data, but instead display your songs with the appropriate ID3 tag information instead. If a filesystem were developed that intrinsically indexed its files by ID3 tags (or similar mechanisms), wouldn't this be an ideal structure for storing and searching through your album collection? And, more to the point, are there any such efforts under way to develop such a beast?"
We don't need a special purpose filesystem for this; just one that supports metadata. I know NTFS does; do any of the common Linux filesystems?
Either way you're going to need a custom tool to do searching and whatnot, the custom tool might as well just index the stuff in the ID3 tag and work with an existing filesystem.
- Steve
That all sounds like BeFS to me. -- which, lo and behold, was discussed on Slashdot just the other day!
What other general features might be good for the specific application of music?
Gee, yeah, who would want file permissions on files, anyhow? Feel free to delete all my MP3's, since they're just music, after all.
Usually this sort of presentation is up to the application, not the filesystem, and Windows is no exception here. The thing presenting the "My Music" folder with ID3 tags and whatnot is the shell (explorer.exe); it's a graphical file browser. There are many of them available for Linux as well.
And personally, I don't really care if my file browser supports this, as long as my MP3 player can read ID3 tags...
pb Reply or e-mail; don't vaguely moderate.
Music filesystems? Image filesystems? Movie filesystems? Document filesystems?
And please, please, at least store the file's mime type, don't play the extension guessing game.
Web server: let's guess what the file's mime type is based on the extension, and send it.
Browser: Let's guess what this file is, given the extension AND the mime type.
Stupid computers!
I thought that it was "introducing", but what do I know? Cliff may be using a word that I've not been endtroduced to yet!
Another problem that stems from Microsoft assuming it always knows best: You can't change the My Music folder to act like a normal folder. Sometimes when I download music I don't pay attention to the exact filename so I like to sort my music files by date to find the ones I just downloaded. Not an option with the My Music directory. Bleh.
Mr. Spey
Cover your butt. Bernard is watching.
A better solution is to take advantage of the flexibility of a modern Unix filesystem, where any charater (excepting '/') can bu used in a file name, including the newline. There is no reason why you couldnt name a file:
Title: Symphony 2
.
.
. .mp3
Composer: Bach, Johann Sebastian
Composition Date: a long time ago
Performing Artists: St. Louis Symphony Orchestra
Performance Date: not very long ago
Genre: Classical
Format:
The Unix filesystem is adequate for the job, metadata is a curse disguised as a blessing.
Best Slashdot comment ever
Nautilus has a music view, where it shoes the id3 tags as if it was an album sleeve. I'm not sure whether this is searchable though.
Setting up a specialized filesystem for music storage seems both overkill in terms of implementation and maintenance issues, and underkill in terms of features. What would be more appropriate is a database table schema that stored music and lots of metadata (Artist info, title/album/recording/remix/venue info about a track, genre information, record label, user ratings/comments) about different recordings, storing the recordings themselves as either BLOBS of mp3s/oggs or (probably better) indexes into a normal filesystem containing the mp3s. Write a quick table browser to go through the collection, and construct a bunch of SQL queries to generate playlists by artist or genre.
Storing metadata about files in the file system? Could be useful...
And of course it's already been done...
I suspect having a filesystem optimized for one specific data type is possibly a bad idea. Every so often I need to temporarily drop a file into my MP3 partition for some reason.
.txt and .jpg files in a lot of music folders (track listing and cover art)
Also you need to store
Anyhow, my main point was this: Can I be the first person to point out that BeFS was/is superb at this?
~~~~~ BigLig2? You mean there's another one of me?
OK, here's the reference, for either of you that have somehow not yet heard of DJ Shadow.
I think that this problem would be better solved not by a new filesystem, which means implementing a new kernel module for what is essentially a file symantics problem, but by a module to Nautilus, or whatever your favorite filebrowser is, that looks for ID3 tags and displays those.
If you do it at the FS level, you would also have to rewrite things like ls and change the way the stat ioctl works, which would break the posix compliance OS. In short, you'd have to modify a ton of stuff just to get id3 tags to display, not to mention that once you got it all working, you'd have to track the development of all the newest FS features like journaling because people are going to want all that stuff to store their mp3s too.
It would be much easier just to have your filebrowser do it for you, and let the filesystem do its job: store files.
Bear in mind that one of the core driving philosophies that has allowed Unix to survive and evolve is "everything's a file", where a file is a stream of bytes. Attaching metadata directly to a file gets to be a nightmare when you start copying files over the network, dealing with editors that create new files rather than modify in place, backing things up, etc.
NeXT had a great solution to this: instead of just having a file "mysong.mp3", have a directory "mysong" that includes files like data.mp3, title.txt, etc. The file browser presented this to the user in a simple UI (e.g. the directory would have the "mp3" icon in the file browser, double-clicking it would play the song in the mp3 player, displaying the title.txt, etc). But traditional file transfer and backup tools worked just fine, and you could use standard tools (your regular editor, copy program, etc) instead of crufty special purpose ones (regedit, or binhex, or whatever) to work with the metadata.
Think of it this way: you don't want to just have 2-3 defined metadata fields, you would like to be able to add arbitrary new ones in a flexible way. And you want the contents of each to be flexible, without size limits or such. So you want a way of associating names with values--wait, we have that, it's called a filesystem. And you want a way of grouping a bunch of name-value pairs together--wait, we have that, it's called a directory. You'll either end up reimplementing a filesystem, badly, as your metadata, but need special tools to view/edit the metadata or move it around (a la resource forks or the registry), or you'll end up going with a NeXT-like solution and use the filesystem itself (which deals quite nicely with things like this).
Sumner
rage, rage against the dying of the light
Whatever it may be, old fs, new fs, database... how about a feature where you can assign a code or symbol or something that would allow it to find music that has some feature or charateristic as to what your currently listening to.
I like the idea of saying "Play Happy/Sad/Fast/Uplifting songs like <song>"
Yeah, I'm dreaming... so what?
Wiwi
"I trust in my abilities,
but I want more then they offer"
(and got the t-shirt)
BeOS (god rest its soul) uses the Be Filesystem, or BFS. It is a 64-bit journalling and indexing filesystem. It is essentially a filesystem that works much like a database. You can attach any number of arbitrary attributes to any file. Filetyping is handled by MIME filetypes, which are also stored as attributes. The default BeOS installation includes all of the usual attributes for music files and it is very easy to add many more. The BeOS "find file" tool is not unlike a database query tool.
In terms of speed, file searches are done almost instantaneously, even when searching through several gigs of data.
Bill Clinton: Pimp we can believe in. - The Shirt!!!
I'm running Win2K Pro SP2 SRP1 and I don't have any special folders for digital media...let alone displaying with ID3 tags!
This sounds like it's a Windows XP thing, and we all know that XP is the devil.
For instance, in Windows, Microsoft has special folder layouts for folders declared to have digital music or images in them. The digital music folders don't bother with file permissions or any unnecessary data, but instead display your songs with the appropriate ID3 tag information instead
I don't think this has anything to do with the Windows file system. Its just how explorer presents data in certain folders. Nautilus does the exact same thing, for instance. It displays bitrate, time, artist, and presents a mini-player at the bottom of the folder view. I assume it gets the artist info from ID3 tags.
"I like to wear big boy pants."
(Write-Once-Read-Many)
A dedicated music partition would probably have files dumped on to it in large chunks, without much deletion or other write activities being performed on it otherwise.
So, just making sure files aren't fragmented when they're dumped on the partition, and maybe caching things like ID3 tags would be useful.
Is there really a problem to address, though?
I organize them into subdirectories, etc. with a standard naming convention. Nothing new here; everybody does this.
The problem I found is that my playlists would break whenever a renamed or moved a file.
Solution:
NB the relative, not absolute, symlink: this allows me to serve up the lists over NFS, SMB, etc.
Altogether a very easy, powerful and flexible system. If something goes wrong, it's just standard files and directories, so I can fix it without editing inodes or dropping into an SQL shell.
A trick you should also know is that just taking the MD5 of an mp3 does work very well since the ID3 is also part of the mp3 and it might change. So I take the MD5 of the last megabyte of mp3 files (the ID3 tag is near the beginning of the file).
One could easily extend the perl script to use ID3 tags to generate directories and playlists. Oh, and the reason I do it this way is because Konqueror is a very nice visual file manager and it allows creating symlinks easily (albeit absolute ones, but my script fixes that).
Anyway, hope these ideas help. Taking the problem down to the filesystem level introduces all sorts of complexity while the problem can easily be solved by a couple hundred lines of perl (writing perl scripts is much easier than kernel work, trust me). Right tool for the right job and all that. This only took me one night to throw together (this includes organizing the actual playlists, which took the most time).
I also have lots of pr0n, and I'm working on ways to automate pr0n management. Let me know and I can get back to you once I've solved the problem.
with virtual file systems being de rigeur this is exactly the sort of application plan9 would be good at.
But we're stuck with 23 char filenames until the eagerly awaited 9p2000 as I discovered the day I statrted to move my mp3s over, doh!
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
There has to be a purpose for anything. Same goes here. What would the purpose be of doing this whole filesystem snazz. The only reason i can think of would be to datbase the songs perhaps and access them quicker. One could come up with a type of system in which the music is streamed directly from the harddisk and decoded via some OS level function. This would allow click-to-play time to be quicker. As far as database stuff goes, you make a filesystem that could be database oriented. But, i really dont see any speed improvments. But i think it is a good start. It will probly develope into somethign if someone where to attempt it.
Why not store the music separate from the meta data, in big hunks of the disk? Like one poster mentioned, the music doesn't really change once deposited in the filesystem; only the metadata does. So we could store music in arbitrary places on the disk, and then have the metadata categorize it, give it names, etc.
The filesystem would then transparently re-create the MP3 file when copied out to move it across the network, zip file, etc... by attaching the meta-data back onto the music stream as it is ready from disk.
Thoughts?
Didn't I see something on Slashdot once about someone getting a database to mount as a filesystem?? If the MP3's (or Oog's) could be stored in that database.... wouldn't it be the start of something useful???
Then again... what the hell do I know..
Wiwi
"I trust in my abilities,
but I want more then they offer"
I suppose you could have a player store the MP3 in the DATA branch, and the track info (whatever) in the RESOURCE branch. Wouldn't most mp3 players just look at the DATA branch anyways, and ignore the RESOURCE branch? It probably wouldn't screw up noncompliant players. However, I'm not a Mac Guy (tm), so I don't know of the feasibility of writing apps to do this, and modifying mp3s in this way (resedit, anyone?)
What Im thinking of writing is a database system which will sweep through all my mp3's cataloging anything new. I also want to add a little app that I can have running that will allow me to add to the database what I think of each song. As it is I have my xmms set up to dump the name of a song to ~/mp3 and the path/filename of the song to ~/mp3filename. All I will need is to have the app pickup the ~/mp3filename contents and take it along with my comments about the song to the database. I also think a little app, maybe connected to the previous that will generate a playlist for xmms based on the type of songs I want to hear. I typically catagorize my cds this way in my cd cases. I have Christian-ROCK Christian-underground Christian-boring Christian-i bought it at a concert and found out i didnt like it.. ROCK, POP, MIX, soundtracks, soundtracks that didnt end up in the movie itself. I also sort of know the cds i want to listen to by the type of mood they induce, Mellow(like Olive), Racy(Linkin park), Feel like singing with it(like redhot chilli peppers or creed). I want my database apps to be able to pull those files attribuited like that out for times when I feel like whatever. DRACO-
Consider yourself blessed if you are sneezed on by a dragon and only get wet, it could have been a fireball.
You're right, a modern metadata file system [Be or newer, not as old as HFS/HFS+] should support MIME filetyping.
By using MIME, your files will always open in an app that can handle those files regardless of which app made it.
Of course, this is something I've been doing for a few months now as a vital part of the Kaos operating system.
- Kaos games and encryption systems developer
An ObjectOrientedFilesystem's only appropriate place is at the bottom of the ocean.
"God is dead." - Frederik Nietzsche
Whether you store the type of the file using the MIME type in a filesystem-based metadata, using a creator/type pair like the Macintosh, or encode the metadata into the filename (i.e. filename extensions), this information has one thing in common: It's extrinsic to the data. It can get lost or changed to an incorrect value. .vbs viruses because Windows hid the filename extension and they think an e-mail attachment is a .jpg file.
We have countless examples of how hard this sucks. It ranges from an annoyance for Mac users dealing with files that came through a non-Mac system all the way up to lost time and money when users spread
The better way is plain: use an intrinsic property of the data in a file. Put another way: Decide what data a file contains by looking at what data the file contains. That's exactly how the Unix 'file' command works. It matches bit patterns in the file against the "magic" database to determine a file type.
Of course, there are file types not easily identified by simple bit strings. Starting with v3.0, AmigaOS had it right with the DataTypes system. Part of DataTypes was the 'descriptor' which provided the system with three ways to identify types of data: a bit string, a callback function, and a filename pattern. These three can be combined, as well. Thus, it's possible to identify a JPEG image file by looking at the first four bytes, to tell the difference between an MPEG video stream and an MP3 with the callback function, and to fall back to identifying unstructured files, such as a raw dump of PCM sound data, by filename.
Why oh why has no other system adopted this cool system?
One idea would be to do this on a directory by directory basis (like 4dos descriptions), this would overcome the mapping problem. If each file type has its own header, then you could store current and future formats in this.
One can have a utility to go through and look at files by type, eg like a "viewer" dll.
Backing up such a file would allow the support of this feature on cdroms under different operating systems.
OS/2 - because choice is a terrible thing to waste.