Slashdot Mirror
MXF+JPEG-2000+HDD = Future of Video Preservation?
Anonymous Archivist writes "Media Matters, a technical consultancy specializing in archival audio and video material, recently completed a Mellon Foundation funded Digital Video Reformatting Preservation Project for the Dance Heritage Coalition. They conclude that MXF is the recommended container format, JPEG-2000 is the recommended encoding format and HDD is the recommended storage media. It's a very valuable series of experiments and offers a strong indication of where the archival preservation of analogue video is heading."
JPEG standart defines several encoding formats, which include lossless compression as well
" The Material eXchange Format (MXF) is an open file format targeted at the interchange of audio-visual material with associated data and metadata. It has been designed and implemented with the aim of improving file based interoperability between servers, workstations and other content creation devices. These improvements should result in improved workflows and result in more efficient working than is possible with today's mixed and proprietary file formats." -- What is MXF
To make laws that man cannot, and will not obey, serves to bring all law into contempt.
--E.C. Stanton
Why would they go with a compression format that doesn't do inter-frame compression?
It might be nice for editing, but you could get more quality in the same space with something like h264, or even h263 if they have to do this right now (i.e. before h264 is quite ready for prime time).
Recommended Storage Media: Peer to Peer network.
The HDD recommendation doesn't seem to make much sense. The article talks about cost-per-gigabyte, but obviously it is much cheaper to use CDRs or DVDRs. This is video preservation, after all, not storing indefinitely for video /editing/, which would require a more malleable storage medium.
And before someone points out that there are studies showing that the longevity of CDR/DVDR discs is questionable, surely proper storage of discs (and not buying the Best Buy free-after-rebate special) would be sufficient. HDD, after all, is susceptible to head crashes, and being a magnetic medium can be more easily overwritten.
SELECT quote.text AS sig FROM quote NATURAL JOIN attribute WHERE attribute.description = 'witty';
0 rows returned
make their report available on a format other than a '.doc' file. it is known to change a lot and therefore not suitable for long term storage.
OK, let's talk archiveability. Let's talk about a medium that you can leave in a shoebox for a hundred years and read just by shining a light through it. I'm not talking hypothetical here - this technology is proven by the fact that people used it a hundred years ago and it worked. And the technology is even better now, even more stable.
I am of course talking about film. It is very very easy now to write digital images onto film, not very much more difficult than it is to scan film. There's no need to worry about whether the file format will be supported in the future, as I've already said. You don't need to shovel money into vendor's pockets every few years just to copy it to the latest trendiest type of disc. You can build a machine to project film out of junk if you need to, or you can scan it if you want a digital image and when you have a better scanner (e.g. a higher DMax), you can just scan it again.
The dude who wrote this report is just blowing smoke. He's trying to sell snake oil.
Okay, so JPEG 2000 uses wavelets and is therefore quite advanced, but as I have understood, it's still geared for still images (ok, there is probably some form of motion jpeg 2000?).
I would think that most optimal method would be to use something like DIRAC instead (or Ogg Theora). DIRAC uses wavelets and adaptive arithmetic coding, so it should be "on par" with JPEG 2000 - and should also be free of patent encumberance.
JPEG 2000 has one feature that might make it better in "archival" purposes - there is a lossless mode which still achieves higher compression ratios than PNG.
https://bugzilla.mozilla.org/show_bug.cgi?id=36351 (no link for obvious reasons) is the bug report, which has been around since April 2000 but has not progressed much due to licensing issues (copyright ones fixed, patent ones not?).
Ummm what about the sound?!
Were that I say, pancakes?
Avoiding inter-frame compression means that, if you have some small amount of data corruption, you only get one, maybe two corrupted frames of video.
Storing digital information on paper is feasible and lots of research efforts have been put into it.
Storing data on anything magnetic or optical is a bit worrysome. But then, it's not critical data so I guess it doesn't really matter.
I could have told people this as they've replaced video tape, and audio tape for me for the past decade. I find them much more convenient, portable and cross platform. I have SCSI drives from 1994 that will still work in a PC (Linux or Windows) or Mac today. They are easy to backup to and restore from. The HD is about as close to perfection as you can get in a storage medium. At least until you get flash drives that can store 1 terabyte at minimum, and have an infinite number of writes. At least a 100 year lifespan.
-"...bad old ideas look confusingly fresh when they are packaged as technology" - Jaron Lanier (Digital Maoism on Edge.o
When there isn't patent litigation surrounding the format.
...had a guide on capturing analog video, said to be the part of a 3 part series, going over each capturing, cleaning, and compressing. Only part I ever came out - Ars do you read slashdot? - I am waiting on the last guides for some advice on how to preserve these rotting home VHS tapes.
Meanwhile, does anyone else have advice on capturing and cleaning video since we are already talking about compression? What settings are good for capturing and what sort of software exists to clean up VHS and give it the appearance of more clarity? I am using a WinTV card as Ars recommended it.
But survivability isn't the only consideration. Cost is always an issue. (So much for my platinum plates, though your approach isn't exactly cheap either.) You also want to be able to able to access the data in the short term. I worked my way through college operating film projectors. It's is not a convenient medium!
One thing I'd like to know is why archival-quality optical discs weren't considered. (Presumably there's something in the document about this, but it's a poorly structured word file, and finding key facts is more work than I care to expend.) They cost 5 times as much as standard CD-Rs and recordable DVDs, but their manufacters claim the data is good for 300 years. Of course, you need some fairly complicated technology to play them back, but CD and DVD drives are pervasive consumer devices -- they should be around for a very long time.
There will always be multiple backup solutions, but the biggest trend continues to be towards using hard disks for backup. When your data files are enormous (such as with audio/visual data), HDD backup is even more attractive.
MXF is the new, proprietary video compression method jointly sponsored by Microsoft and MTV. The new Most eXtreme Format is the video compression of choice for today's most hard-core, edgy, in-your-face artists with an attitude!
Ashlee Simpson says "When I'm performing for a half-time show of 10,000 screaming fans, I want to make sure that every bit of the live energy is caught perfectly! I give 100% for my fans and want to make sure they get every bit of my performance!"
MXF... in your FACE, Quicktime! This isn't your father's archive-quality lossless video compression algorithm!
(and keep an eye out for Ogg Vorbis 2 - by Mountain Dew!)
There is more to jpeg2000 than a compression scheme offering scaleable quality and resolution within a single losslessly compressed file. There is also the interactive delivery mechanism offered by the JPIP protocol. Now there is something really useful...
In theory, theory and practice are the same. In practice, they're not.
All this is is a method to line some guy's pockets. I'm sure the tape guys are gonna say, use XYZ type of tape. The disk guys are gonna say disk.
What makes this guy think that the interface to the HDD is going to be around in X years?
PC's have only had two dead (non-(e)IDE/ATA) interfaces, the ESDI and the ST506/ST-412 interfaces.
But what if you were trying to find a computer with IPI (1960s mainframe) interface.
The Fed gov't has this problem with trying to find parts for their old 8/9track tape drives..
Here's a good list of all the HDD interfaces over the years: http://www.i-t-s.com/corporate/terms.html
Stick with microfiche, film, that way we don't have to pay some vendor $$$/yr to keep alive a dead technology or pay some other vendor $$$/media to move them from old to new media.
And if you can spare the space, a directory with a wav file and a stack of uncompressed TIFF images is even better. Compression formats are complicated to reverse engineer.
Store .mng + .flac + source code for libmng and libflac, and you don't need to worry about any sort of complicated gnireenigne.
Because when you're archiving digital data, recoverability is paramount. You have to ask yourself, "What if all I had was a piece of this data, say, a hundred gigabytes from the middle of the disk? Could I turn that data into useful information?"
If you're dealing with a run-length-encoded array of packed pixels, the answer is obviously yes. That's among the simplest forms of encoding known. (If you don't RLE the data it's even simpler, but a trade-off between simplicity and storage requirements is okay as long as you maintain a lot of simplicity.) Even if you don't know how the data was encoded, you've got a good chance of figuring it out just by doing some simple analysis on the bytes. But with a complex encoding scheme, it's much more difficult to figure out what you're dealing with just by looking at it.
When talking about archiving, the objective is to be able to recover as much as possible given as little as possible.
For whatever reason (I'm not a video expert) many people prefer intraframe codecs for archival. As you probably guessed, Motion JPEG 2000 just treats each video frame as a still image and compresses it with JPEG 2000.
Dirac will give much better compression that JPEG 2000, but it also introduces the possibility of interframe artifacts.
Because when you're archiving digital data, recoverability is paramount.
No, Viacom is paramount.
"What if all I had was a piece of this data, say, a hundred gigabytes from the middle of the disk? Could I turn that data into useful information?"
As long as your codec is seekable, this works. Motion JPEG is trivially seekable, consisting entirely of keyframes. Toss a redundant copy of the codec on the volume after every GB or so of video data, and recoverability is preserved.
For people concerned with the preservation of "data", they've sure picked an interesting format to write about it in.
--
"Open source is good." - Steve Jobs
"Open source is evil." - Microsoft
indeed, lossless for archival preservation is the
only way, as it fits the basic rule of art restoration
technology -- never apply "improvements" which
cannot be reversibly undone to take advantage
of future science.
ironically then, the lossless format doesn't matter.
however, at least for the instant case of dance video,
the likely input (a myriad of digital tape formats)
is hopelessly neanderthal -- anything having to do with DV,
or MPEG, or even ATSC HDTV already tosses away much
color information. (4:1:1, 4:2:0, and 4:2:2 colorspace is embarrassing
to preserve "losslessly".) ditto for temporal
info, with interlacing being the culprit. even film at
24fps just will not cut it for motion such as dance.
so here's to better camera technology, whether it's
10- or 12-bit 4:4:4 RGB, or something like
carver mead's foveon made swift.