Efficiently Reading ID3v2 Tags Over HTTP?

Paul Crowley asks: "Given an HTTP URL for an MP3 file, what's the best way to read its ID3 tags on a GNU/Linux system? It shouldn't be necessary to fetch the whole file: HTTP byteranges should make it possible to fetch only the tiny fraction that's needed, for a big saving in network bandwidth. However, existing ID3v2 libraries are designed to read local files. Extending these libraries for this purpose, or implementing a new one, would be a big job. What's the clean solution - is FUSE the best way, or is there a simpler way that doesn't require root privs? Can I do it using the existing id3lib binary?"

Comments

Perhaps not this simple

[Anonymous Coward]

Why couldn't you save the result of the remote HTTP access to a temporary local file and allow the libraries access to that file?

You'd have to extend the API

[Ayanami Rei]

You'd better be prepared to extend the API with a URL handler...

There's no point adding http:// support without also adding ftp:// URL support. FTP supports range fetching as well.

So you have handlers for http:// URLs, ftp:// URLs, and file:// URLs.

Then you'd have to map all the old (compatibility) file-oriented APIs into the new function handlers for file://. (Or maybe the opposite, map file:// into the old API, leaving the old implementation intact)

HTTP 499

[cryptor3]

It seems like it shouldn't be that hard. You just initiate the HTTP transfer and then cancel it as soon as you have as much data as you need.

I haven't actually done it, but speaking as a server operator, when I look through my server logs, you see some hits that end with status code 499, meaning that the transfer was aborted. So you just have the client software you're writing close the HTTP connection after it locates the end of the ID3 tag. It's probably not 100% efficient, but obviously a lot better than reading the whole MP3 file.

I'm assuming you're doing this in C/C++, but I'll try to do a prototype in perl.

Re:HTTP 499

[eyeball]

From the ID3v2 FAQ:

Q: Where is an ID3v2 tag located in an MP3 file?

It is most likely located at the beginning of the file. Look for the marker "ID3" in the first 3 bytes of the file.

If it's not there, it could be at the end of the file (if the tag is ID3v2.4). Look for the marker "3DI" 10 bytes from the end of the file, or 10 bytes before the beginning of an ID3v1 tag.

That's the problem -- it could be at the end, requiring you to spin through all x bytes (most likely megs) until you get to the end.

Re:HTTP 499

[cryptor3]

That's the problem -- it could be at the end, requiring you to spin through all x bytes (most likely megs) until you get to the end.

Yeah, that could be true, but if it's not within say, the first 100KB, then the smart thing to do is to stop trying to find it and just return an error.

If it's not at the beginning, you could then use byte ranges to try to fast forward to the end and guess that it will be within the last say, 50 KB of the end.

Re:HTTP 499

[pbox]

This is why

1. read first 3 bytes with http bytrange
2. if id3, process tag from byte 0
3. else read last 10 bytes
4. if 3di, process tag from backwards
5. else, see if there is a id3v1 tag at the end
6. if yes, read last 10 bytes before id3v1
7. if 3di, then process backwards

So it is possible. He just needs to read the fricking id3 tag definitions.

ID3v2 Sucks

[DeadSea]

As somebody who has tried to write libraries that read ID3v2 tags, I'd have to say I hate them. The standard is clear and well documented, but the chosen format is horrible. It is very hard to write a parser correctly. It would have been so much better to embed an XML document at the front of the MP3 file. Instead they decided to make each field in a special binary format prepended by a length field.

The number of checks you have to do is phenominal. The biggest worry is buffer overflow where the length given is greater than the actual length of the tag and you read more than is in the file. There are just hundreds of such edge cases. Libraries for ID3v2 are likely to be buggy, crashy, and just no fun.

Re:ID3v2 Sucks

[foofboy]

I hear you on that. I wrote a python module for id3v2 tags. It reminded me of nothing so much as ASN.1/BER/DER.

It does, however, support arbitrary character sets and arbitrary binary formats, though. Not sure there's another way to do it. Vorbis-comments are ASCII only, right?

I look forward to your reply.

Re:ID3v2 Sucks

[Fweeky]

foobar2000 uses APEv2 tags on MP3's by default; the standard's just as flexible (well, as much as anyone wants anyway), but, well, you just need to compare filesizes for their handlers; an ID3v2 reader/writer I saw was ~150k of code -- the APEv2 one was 15k. They're always at the end, but obviously since fb2k is the only player I'm aware of which supports it the appeal may be limited. You can at least mix them with ID3v1, which should be good enough for portables.

And before anyone goes off on one because it's non-standard, I'll point out that MP3 has *no* provision for metadata. ID3v1 and 2's are just as arbitary addons as APEv2; they're just older (and lamer, either in big limitations or extreme overcomplication).

I believe the recommended *standard* way of attaching metadata to an MP3 now is to put it in an MP4 container, which has it's own more sensible format. Again, I'm pretty sure foobar2000 (maybe with some plugin in the Special Installer) can put them in, and I think they should play on anything which knows about MP4. Fully reversable too.

MP3::Info Perl module

[extra88]

Look at the MP3::Info Perl module, you might recognize the author's handle. It reads (and writes) tag info. It's used by the "jukebox" module Apache::MP3 (sample site) to generate pages with track info.

Basically every web jukebox out there does something like this so I'm sure there's plenty of other code available to work from. The mod_perl way is to put SetHandler perl-script then PerlHandler [name of module] in your httpd.conf file so when a URL request falls within that Location or Directory, the perl module handles returning whatever you want it to return.

Vorbis comments UTF-8

[rillian]

Vorbis-comments are ASCII only, right?

No. The field names are ACSII only (actually a printable subset minus '=') but the contents of the fields are specified as UTF-8.

The intention was you could put arbitrary binary data in there too, but there's no general mechanism for marking it as anything else. So any non-UTF-8 use would be application specific.

Amazed that no-one's really tried to attack this

[Paul Crowley]

This is an interesting general problem. I'm sorry that so few people seem to have taken the time to understand it before replying. The general approach seems to be "read the first sentence, assume the poster is an idiot, hit reply".

The problems are these:

1) Reading ID3v2 tags on an MP3 file is a complex business. I have no desire to re-implement the libraries that do that, or even to wade deep into the existing codebases, if I can avoid that. And it should be possible to avoid that.

2) Even knowing the size and location of ID3v2 tags is complex. Contrary to popular belief here, those tags can appear at either the beginning or the end of a file, and can be arbitrary size. I already implemented the "fetch some stuff at the beginning and some stuff at the end and feed that to the library" approach, and it sort-of works, but you have to guess the size of the tag. Guess too big, you fetch lots of data unnecessarily. Guess too small, you get breakage or wrong results. By contrast, the libraries that read ID3v2 tags know exactly where and how much to read to glean the appropriate data, and it should be possible to make use of that.

3) I want to read existing data - changing the format of that data is not an option.

So that's why I was suggesting solutions like "FUSE". With FUSE, when the library does a seek and a read, I can arrange for just the relevant portion of the file to be fetched. I don't have to include any knowledge about ID3 in my application - the library does all the work. But the library doesn't have to worry about HTTP byte ranges - FUSE handles that. And the code will always be correct.

The only trouble is that FUSE requires a kernel patch and root privs. The question is, is there a way to do the same trick without those limitations? Or is there a library for reading ID3v2 tags in an object-oriented language that will let me put an efficient back-end for fetching data on request using HTTP byteranges in place of the file?

The best information I've got out of this is that there's a pure-Python implementation of ID3v2 (most implementations appear to be built on top of the C library). This may be hackable to solve my problem.

Those of you who didn't think reading or thinking was necessary before posting - please don't do the next "Ask Slashdot" post the same discourtesy. Thanks.