Slashdot Mirror
Technical Objections To the Ogg Container Format
E1ven writes "The Ogg container format is being promoted by the Xiph Foundation for use with its Vorbis and Theora codecs. Unfortunately, a number of technical shortcomings in the format render it ill-suited to most, if not all, use cases. This article examines the most severe of these flaws."
I don't see any comment and the website is already down. gg /.
No matter how bad it is, it's still better than AVI. I personally use Matroska, it has all of the ideological benefits (free, non-encumbered, open-source) over stuff like MP4.
Yet Another Tech Blog
(but so much more, including game and movie reviews)
http://yanteb.peasantoid.org
"I would have done it diffferently" does not mean that the format is bad. None of these "flaws" render the format unusable. Maybe it doesn't perform as well as another format, maybe it isn't designed the way you would like, but it's implemented, it's available, and it's in use.
It's not a selling point, it's a starting point. It's a sine qua non. For an application like video on the Web, nothing non-free can even enter the conversation.
There's at least one obvious flaw in his reasoning. He talks about removing the 8-bit version field in the header and replacing that with a 1-bit portion of the flags field to distinguish it from a hypothetical future version. That only works if one assumes there will only *ever* be two versions (v1 and v2). Such a basic failing of analysis is a pretty good indicator that he hasn't thought it all through as completely as he thinks he has.
I'm not an expert in video or audio production, I just dabble in it as a hobby. but one thing I often wonder is, what is the point of these container formats?
I've got a miniDV camera, and a canon point and shoot that thanks to chdk can record good-enough video. Both give me ".AVI" files, even though one is miniDV, while the other is Mjpeg. Mjpeg files don't work in my editor, while miniDV does. but I didn't know this at first, all I knew was that I have a bunch of .AVI files sitting in my hard drive, some work, some don't. I dont care about file extensions, I care about having files that work. I care about codecs. If they were named "filename.minidv" and "filename.mjpg" that information would be useful to me. What good is a container format when only half of the files within that container will play on my system?
I'm not trying to knock the idea of container formats, if they exist, their must be some beneficial reason for them. Could someone please enlighten me on what that reason is?
-I only code in BASIC.-
"In the long run, all file formats become programming languages."
From this I draw a number of conclusions, the first being that when designing a format you need to bring a "language sensibility" to it. If you don't, it's only a question of *when*, not if, your format will become a poorly designed language. OK, "language" may not be the right word. I'd also accept, "byte code" or "executable file", but it's the same idea. JMHO.
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
... of what format *should* be used in its place.
It is all very well claiming one format is not particularly good, but overall rather pointless if you don't argue an alternative.
So the question any .ogg user will have (since they probably chose this slightly obscure format over the more 'normal' .mp3 alternative due to the reputation of being better to listen to from an audiophile POV) is what to use instead? FLAC is fine if you have the space, but sometimes you want to compromise in order to save storage space...
I can't say anything about video, but for audio all my CD collection is converted to Ogg instead of MP3, you can't even spot the difference in quality, thou the filesize is smaller. BTW, my MP3 player supports Ogg playing as well.
Audio quality and compression efficiency are controlled by the codec, not the container format. The article is critiquing the latter.
Eagles may soar, but weasels don't get sucked into jet engines.
His complaints:
On top of this we have the 27-byte page header which, although paling in comparison to the packet size encoding, is still much larger than necessary.
Ok, it's a container format, nobody cares about an extra 27 bytes when you can buy TB of storage for virtually nothing. And if you're complaining because it needs to go in the intertubes, gz compression on the server does a very good job of extracting and compressing plain non-random text like page headers but again, MBits are cheap and unless you're living in the US they are plenty.
The version field could be disposed of, a single-bit marker being adequate to separate this first version from hypothetical future versions. One of the unused positions in the flags field could be used for this purpose
It's kind of important to keep track of versions. If your player can't play the next version or an older version it should be able to detect that so it doesn't try-and-fail. It might also want to suggest what version of the player is required.
A 64-bit granule_position is completely overkill. 32 bits would be more than enough for the vast majority of use cases. In extreme cases, a one-bit flag could be used to signal an extended timestamp field.
That's what they said about our memory too back in the early 90's. 32-bit addressing is enough, nobody will ever have more than 4G of RAM. Again, these open formats tend to be scalable across time because they need to fulfill a certain mission. Look at ZFS, they have 128-bit addressing but nobody (currently) needs that amount of storage.
32-bit elementary stream number? Are they anticipating files with four billion elementary streams? An eight-bit field, if not smaller, would seem more appropriate here.
Why not, how many languages are there around the world? If you need to bring out a media file with subtitles and audio-tracks for each language, braille instructions and who knows what else for open access to certain media, you might want to use more than 256 streams.
The 32-bit page_sequence_number is inexplicable. The intent is to allow detection of page loss due to transmission errors. ISO MPEG-TS uses a 4-bit counter per 188-byte packet for this purpose, and that format is used where packet loss actually happens, unlike any use of Ogg to date.
Well, maybe the makers intended Ogg to be used eventually to replace MPEG (c)(patented) and used across links with much higher transmission errors. Sometimes my MPEG-encoded stream I get from my DTV provider has enough errors to stall and cause artifacts. When NASA wants to use Ogg for a non-repeatable stream from outer space, they should be able to. Again, overhead is a small cost to pay these days.
A mandatory 32-bit checksum is nothing but a waste of space when using a reliable storage/transmission medium. Again, a flag could be used to signal the presence of an optional checksum field
Ah, well, what is reliable these days? Ever used a large array of hard drives? Ever used a freakin' dial-up connection? As the makers of ZFS, Google and a few others recently have shown hard drive and memory reliability is not as good as we take for granted. Silent data corruption is a major cause of data loss these days.
With the changes suggested above, the page header would shrink from 27 bytes to 12 bytes in size.
Whoop-dee-doo, you made it half the size but you sacrificed reliability, error correction and future-proofness.
Latency
You show that the overhead is anywhere from 1% to 7%. That might not be the requirements for latency-sensitive applications but then you would again sacrifice other features. That is always a balance between speed and reliability but for most applications it doesn't really matter if the movie needs to be buffered 5ms longer.
Random access
You've got somewhat of a point there, maybe somebody will find a solution for that. The issues around indexing however is that seeking within a stream is possible. HTTP servers
Custom electronics and digital signage for your business: www.evcircuits.com
Besides it being EBML (a binary and efficient kind of XML), I’ve yet to see a feature that it can’t do. Even a complete 3D TV series with multiple perspectives, languages, subtitles, additional content, hull cover... streamed over the net in one file? No problem.
Also, it’s already the format of choice for HD video and multichannel audio format rips on the net.
A competitor would be nice. Unfortunately, OGG can’t hold a candle to it. But if they manage to catch up, they will be very welcome.
Any sufficiently advanced intelligence is indistinguishable from stupidity.
Quite a bit of the analysis seems to be reasonable on the surface, but something about the way it was presented set me off in a geek-rant that I put in the comments. Since I'm having trouble posting that comment on the site, here it is.
Many of the points sound reasonable, but the argument is strongly undermined by the fact that it offers not a single apples-to-apples comparison between ogg and any other container format in the article. On a section-by-section basis:
Generalities/codec mapping:
Article complains about how there is no global mapping, but does not assert that other containers have one.
Overhead:
The breakdown of where space is wasted is informative and mostly reasonable, but some of them seem to be a reach, such as the checksum being unneeded, and the suggestion of implementing the functionality in optional fields seems like a bad idea to me in general, since it will make the header variable-length, which is something to strongly avoid in my experience. Finally, when the article does "compare" ogg to mp4, it compares some rather hand-wavey numbers for ogg to a different scenario for mp4.
Latency:
The article fires off a bunch of numbers here, but then offers no comparison to the alternatives. In fact it don't even provide an explanation of how other formats avoid this latency in theory, much less in practice, and instead of showing how bad the latency is, it uses it as a platform to show that a naive reaction to the issue will cause bad header overhead.
Random Access:
In this section it lists quite a few worst-case numbers for disk accesses (why isn't it being pre-cached by the filesystem?) and then ends with no comparison to alternatives at all.
Complexity:
Once again it has a bunch of statements of problems the author has with the format, but no comparisons to "good" formats, in addition this section is particularly weak, with statements like, "implementation is annoying", and "ambiguity is bad".
Final Words:
"We have shown" is a rather specific claim to make, which the article has not remotely achieved. This pretty much sums up the whole article, which is titled "Ogg objections", but then tries in the text to bill itself as a rigorous analysis of ogg, which it is not.
If the author had matched the tone of the article to the title, this would be reasonable, but he only hurts his position when he throws around phrases like, "True generality is evidently not to be found with the Ogg format.", "The Ogg format is clearly not a good choice for a low-latency application.", and "We have shown the Ogg format to be a dubious choice in just about every situation.". He has demonstrated NONE of the above claims, and by making them the article has rendered me skeptical of the rest of its claims.
This whole thing is really about bad blood between Xiph and the mplayer folks. Once, long ago, I made disparaging remarks about a particular mplayer developer's extensive collection of ass hats, and they declared war. This stopped being about facts or reason years ago. Here's the last blog thread that got completely hijacked by the anti-Ogg container wingnuts. It's a hell of a read:
http://blog.gingertech.net/2010/02/20/googles-challenges-of-freeing-vp8/
So, rehashing this yet again: The Anti-Ogg bullet points [Not going to bother with complete sentences, because I've wasted too much typing on this recently]:
1) A few of the mplayer/x264 hackers are right pissed that Ogg and Theora are getting all this attention when x264 is so obviously superior. That simply cannot stand. Since only America has patents and there are no computers there anyway, nobody should have to worry about them. Stick it to The Man! (How very ironic, Xiph being considered 'The Man' by folks contributing to an h264 encoder).
2) Xiph should immediately drop Ogg for [insert container here], breaking millions of hardware decoders and hundreds of millions of software decoders:
a) the [patented] mp4/MOV container is one suggestion they actually make seriously. Never mind adding 'willful infringement' to breaking the entire installed software/hardware base, this choice would totally redeem Xiph in their eyes. The benefit: by their own figures, it would reduce container overhead from .7% to .3%.
(Except that number is wrong. I found later that DonDiego screwed up his mp4 overhead figures at the link above; I had simply assumed he got his container numbers right. The mp4 file in his example has almost identical container overhead to the Ogg, a shade under 1%. His demultiplexed mpeg audio and video had framing in them, so it made it appear the mp4 container overhead was much smaller when he subtracted their file sizes.)
b) OK, mp4 is patented and no better, fine, Xiph should have just used Matroska from the beginning. Despite the fact that Ogg and Vorbis predated it by about five years (also mkv's not been able to interleave until just recently, which == no streaming). This is not to say you can't put Theora and Vorbis in Matroska. It's even a good idea! I've come to like MKV. But for streaming, Ogg is still much easier to deal with. Ogg was designed to stream, mkv was not.
c) OK, so, mp4 and Matroska are right out for streaming, Xiph should use Nut, which is the system they designed. Nut came ten years after Ogg was already widespread. And looks almost exactly like Ogg. Which is not to say there aren't things about it I like that improved on the Ogg approach. Eg, the packet length encoding is better. It has a conditional checksum coverage feature I had never considered, etc. At some point we'll make those changes when that wouldn't mean completely abandoning any chance we have at adoption just to save a fraction of a percent and add... no new features.
d) but.. but.. even FLV is better! OK, at this point I can't even entertain the arguement.
3) OK, maybe not adopt another container, but Xiph should immediately improve/change Ogg for, breaking millions of hardware decoders and hundreds of millions of software decoders for a 'better' implementation that won't actually give users any features they don't have now. FOSS need _tools_, not us wasting time overoptimizing something they couldn't care less about.
3) 64 bit timestamp! OMG, waste! Wait, mov/mp4 uses 64 bit stamps... Also, plenty of things in Ogg use a full byte instead of one bit because the container assumes octet alignment. Alignment makes it much faster/easier to deal with (you don't need a bitpacker to read pages, and you don't have to repack packet data to embed it into the page). Remember, all the completely unacc
I sadly have to agree, and I've voiced the same objections for a long time. It really is like he tells it: it's just bad at everything it was intended to achieve. It's a source of bugs, it's horrendously complicated to support, and it's horrendously inefficient at anything but audio (and even then, not so good).
It seems to me, most of what went wrong was trying to support concatenation of Ogg streams. This is a nice idea, but actually quite a rare case. It's also incredibly naive for the specification document to request that Ogg implementation detect this. What, I'm supposed to scan the entire file in case that happens? No. I'll just not be compliant to that, thank you very much.
I even wrote my own Ogg/Vorbis decoder from scratch a while back (and dabble every now and then), and found Ogg to be a never-cooling, never-extinguishing steaming pile of hippo crap left over from consuming a dog. It just made everything so difficult to do. Seeking a stream involves divide-and-conquer - not necessarily a bad thing, but when you have huge streams the number of seeks can be bad. Not to mention if your stream has an endpoint the other side of the Atlantic Ocean. Why oh why did they pick timestamps being at the END of a page and indicating the output byte count produced by the END of that page? That little detail alone probably cost me days of debug.
I almost gave up at one point and went to a container format of my own which would have worked much better. Header: 'CONTAINER v1'. Packet: 'MAGIC', 4 byte Length, 4 byte Output pos. Job done. The sad fact is, that's easier than Ogg, smaller than Ogg (unless you're talking really low bit rate), and does entirely the job of Ogg without the complexity.
I'm probably going to add a Matroska container to my codec just to see how easy they are to produce. The spec looks fantastic, but the devil's always in the details - although seeing the praise on various (engineer) forums, it looks like the way to go.
So, Ogg, please die. We need you to get out of the way.
NUT is another alternative, which is open, simple, and well designed. Along with Matroska, it is also capable of containing Ogg Theora and Vorbis streams, so there is really is no good reason to use the Ogg container anymore. The author of the article is correct--the Ogg container is an awful format.
The main complaints about Matroska are two-fold. One, the EMBL encoding is overly complicated. It requires a considerable amount of code to parse, and also imposes an unnecessary degree of overhead. The second is a much more serious problem: a Matroska file can only contain one timebase. Thus, in order to mux streams with different timebases, approximation is required. To accurately represent the converted timebases, it is necessary to use a much finer granularity, and then you also lose the exact timestamps.
The NUT specification and code is available from svn://svn.mplayerhq.hu/nut, and the (de)muxers are included in MPlayer/FFmpeg, VLC, and probably elsewhere.