Slashdot Mirror


MXF+JPEG-2000+HDD = Future of Video Preservation?

Anonymous Archivist writes "Media Matters, a technical consultancy specializing in archival audio and video material, recently completed a Mellon Foundation funded Digital Video Reformatting Preservation Project for the Dance Heritage Coalition. They conclude that MXF is the recommended container format, JPEG-2000 is the recommended encoding format and HDD is the recommended storage media. It's a very valuable series of experiments and offers a strong indication of where the archival preservation of analogue video is heading."

25 of 214 comments (clear)

  1. Re:Lossy file formats... by iezhy · · Score: 5, Informative

    JPEG standart defines several encoding formats, which include lossless compression as well

  2. Re:MXF? by Raul654 · · Score: 5, Informative

    " The Material eXchange Format (MXF) is an open file format targeted at the interchange of audio-visual material with associated data and metadata. It has been designed and implemented with the aim of improving file based interoperability between servers, workstations and other content creation devices. These improvements should result in improved workflows and result in more efficient working than is possible with today's mixed and proprietary file formats." -- What is MXF

    --


    To make laws that man cannot, and will not obey, serves to bring all law into contempt.
    --E.C. Stanton
  3. JPEG-2000 by Anonymous Coward · · Score: 3, Interesting

    Why would they go with a compression format that doesn't do inter-frame compression?
    It might be nice for editing, but you could get more quality in the same space with something like h264, or even h263 if they have to do this right now (i.e. before h264 is quite ready for prime time).

  4. Recommended Storage Media by Antonymous+Flower · · Score: 5, Funny

    Recommended Storage Media: Peer to Peer network.

    1. Re:Recommended Storage Media by remahl · · Score: 4, Insightful

      Definitely not!

      Most if not all peer to peer networks require a certain level of interest in an item for it to be retained. Popular items are always easy to find while obscure / old items gradually disappear from the network.

      Try finding a movie that's a few years old. You'll have more trouble finding the original Jurassic Park than Jurassic Park III.

      Peer to peer is not a great way to reliably and systematically preserve cultural heritage.

    2. Re:Recommended Storage Media by Catbeller · · Score: 3, Insightful

      Well, that's more a function of the cost and size of data storage. Give me a Petabyte of soldid state nonvolatile storage, and I'll toss Jurrassic Park I in there for giggles, along with 20's silent films, clips from "Bozo's Circus" on WGN on 1969's Chicago TV, the collected books of mankind, complete 3D terrain maps of Mars and every old time radio recording in existence. Gimme a $200 unit that does this, and I'll preserve anything I can get my hands on!

  5. Why HDD? by Orinthe · · Score: 5, Interesting

    The HDD recommendation doesn't seem to make much sense. The article talks about cost-per-gigabyte, but obviously it is much cheaper to use CDRs or DVDRs. This is video preservation, after all, not storing indefinitely for video /editing/, which would require a more malleable storage medium. And before someone points out that there are studies showing that the longevity of CDR/DVDR discs is questionable, surely proper storage of discs (and not buying the Best Buy free-after-rebate special) would be sufficient. HDD, after all, is susceptible to head crashes, and being a magnetic medium can be more easily overwritten.

    --
    SELECT quote.text AS sig FROM quote NATURAL JOIN attribute WHERE attribute.description = 'witty';
    0 rows returned
    1. Re:Why HDD? by chotchki · · Score: 3, Informative

      If you just mirror it on two hard drives and then put them into storage, they will last for a very long time. HDDs only die when run via wearing out and not just sitting on the shelf.

    2. Re:Why HDD? by drinkypoo · · Score: 4, Insightful

      It would be smarter to use PAR2 (or similar) on a filesystem basis, than to use a RAID filesystem. It's easier to deal with user space programs for reconstructing data.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    3. Re:Why HDD? by FireBug · · Score: 3, Interesting

      I didn't read the article (or the rest of the /. comments), but hard drives make much more sense than any optical storage medium in certain cases.

      Media will always wear out, regardless of what type it is. When you have huge amounts of data to back up, it's much nicer to be able to copy it to the latest greatest storage medium quickly and efficiently. Thousands of CDs/DVDs even with an automated "disc changer" would take a hell of a lot longer to transfer than a bunch of servers with hard drives.

      With a hard drive solution, you can just build a new server with new drives and copy everything over from the old one as fast as the hard drives and network allow. Couple this with RAID and multiple servers in different physical locations and you have a pretty damned resilient data archive. ... and just for fun, here's an (old) example of people using hard drives for large scale backups.

      http://www.tomshardware.com/storage/20030425/index .html

    4. Re:Why HDD? by Jah-Wren+Ryel · · Score: 4, Informative

      If you mirror across two disks and put the into storage, and one develops some minor errors, it is not possible to tell which one has the errors

      Exceptionally incorrect, prepare for smackdown.

      All data on a hard disk is protected by very sophisticated error detection and correction elgorithms. The chance of getting "some minor errors" is effectively nil - either they are corrected by the disc's controller, or the controller returns a "sector unreadble" error - which is what keys any effective mirroring system to go get the data from the second disk. You just don't get bad data from modern hard disks.

      This is why God RAID-5 was invented.

      No, raid-5 was invented to maintain the I in RAID. Mirroring doubles your costs, RAID-5 only increases them by one disk out of the N disks in the parity group, where N is usually but not limited to 4-5 drives.

      --
      When information is power, privacy is freedom.
  6. first step by same_old_story · · Score: 3, Funny

    make their report available on a format other than a '.doc' file. it is known to change a lot and therefore not suitable for long term storage.

  7. Nonsense by sql*kitten · · Score: 5, Insightful

    OK, let's talk archiveability. Let's talk about a medium that you can leave in a shoebox for a hundred years and read just by shining a light through it. I'm not talking hypothetical here - this technology is proven by the fact that people used it a hundred years ago and it worked. And the technology is even better now, even more stable.

    I am of course talking about film. It is very very easy now to write digital images onto film, not very much more difficult than it is to scan film. There's no need to worry about whether the file format will be supported in the future, as I've already said. You don't need to shovel money into vendor's pockets every few years just to copy it to the latest trendiest type of disc. You can build a machine to project film out of junk if you need to, or you can scan it if you want a digital image and when you have a better scanner (e.g. a higher DMax), you can just scan it again.

    The dude who wrote this report is just blowing smoke. He's trying to sell snake oil.

    1. Re:Nonsense by Jeff+DeMaagd · · Score: 3, Informative

      and no you cannot " build a machine to project film out of junk". do you know how much film projectors cost? (hint a good lens alone, is over $5000,00.

      I think it is possible. Such an expensive lense isn't necessary. The best film projectors cost a lot, but I don't see it as that difficult to fabricate a basic one from scratch. It won't be the best but it could actually be watchable on a small scale.

      It might look hard to the monkeys that assembles ATX computers but I think a decent one could be made from scratch as a small senior engineering project for college, and probably could be adjustable with different sprockets and such. A little more complex than just shining a light through it. It may be hard to imagine, but there was a time when people had portable film cameras for home videos. It wasn't fancy and didn't need to be.

      Kodak announcing they'll stop producing film has little to do with anything, IMO. Five years is a lot of time but thus far, the drive to push digital projection is going much slower than people expected. Lucas wanted his Episode III to be exclusively projected in digital video, but it's not going to happen unless he wants to drastically cut the number of screens, I'm thinking a tenth of the screens is not an unrealistic figure.

      Of course, part of that is political and economic, because it saves the film distributors from major costs, but they refuse to pass on the savings to the theater companies that must invest as much as a quarter million dollars just to get started.

  8. OK, so when are we going to have support for it? by bersl2 · · Score: 3, Informative

    https://bugzilla.mozilla.org/show_bug.cgi?id=36351 (no link for obvious reasons) is the bug report, which has been around since April 2000 but has not progressed much due to licensing issues (copyright ones fixed, patent ones not?).

  9. Turn it up! by belg4mit · · Score: 3, Interesting

    Ummm what about the sound?!

    --
    Were that I say, pancakes?
  10. Graceful degredation by Dwonis · · Score: 4, Informative

    Avoiding inter-frame compression means that, if you have some small amount of data corruption, you only get one, maybe two corrupted frames of video.

  11. Re:"it" being JPEG2k by TheoMurpse · · Score: 4, Informative

    When there isn't patent litigation surrounding the format.

  12. Ars Technica... by EMIce · · Score: 3, Interesting

    ...had a guide on capturing analog video, said to be the part of a 3 part series, going over each capturing, cleaning, and compressing. Only part I ever came out - Ars do you read slashdot? - I am waiting on the last guides for some advice on how to preserve these rotting home VHS tapes.

    Meanwhile, does anyone else have advice on capturing and cleaning video since we are already talking about compression? What settings are good for capturing and what sort of software exists to clean up VHS and give it the appearance of more clarity? I am using a WinTV card as Ars recommended it.

    1. Re:Ars Technica... by Noose+For+A+Neck · · Score: 3, Informative

      I wouldn't bother with ArsTechnica. For the definitive guide to capturing analog video and digitally archiving it, you would want to read this guide on Doom9. Plus, they have many other video-related guides on that site and a forum that is second to none in terms of the sheer amount of expertise exhibited by the users there.

      --

      Software piracy is victimless theft.

  13. The move to disk backup continues by RonBurk · · Score: 3, Insightful
    People objecting to the use of hard drives for backup miss several points.

    • Yes, they are (somewhat, not excessively) vulnerable to magnetism. But optical discs are vulnerable to light, fingerprints, chemicals, etc.
    • Optical discs continue to lag far behind in capacity. So, in the land of audio/video backup, the choice is between a single hard disk, or dozens of optical discs. The risk of failure of multiple optical discs is amplified by the increasing number of discs.
    • Bandwidth is another issue. Though hard disk bandwidth lags behind the growth of hard disk storage, optical disc bandwidth lags even further behind.
    • Restore time is another issue. You can line up a bunch of optical disc drives and try to make all your data available at once, but you're probably never going to get the restore speed of solutions like Massive Arrays of Inactive Disks.


    There will always be multiple backup solutions, but the biggest trend continues to be towards using hard disks for backup. When your data files are enormous (such as with audio/visual data), HDD backup is even more attractive.
  14. Re:MXF? by The+Ultimate+Fartkno · · Score: 3, Funny

    MXF is the new, proprietary video compression method jointly sponsored by Microsoft and MTV. The new Most eXtreme Format is the video compression of choice for today's most hard-core, edgy, in-your-face artists with an attitude!

    Ashlee Simpson says "When I'm performing for a half-time show of 10,000 screaming fans, I want to make sure that every bit of the live energy is caught perfectly! I give 100% for my fans and want to make sure they get every bit of my performance!"

    MXF... in your FACE, Quicktime! This isn't your father's archive-quality lossless video compression algorithm!

    (and keep an eye out for Ogg Vorbis 2 - by Mountain Dew!)

  15. Yawn, another technology of the day... by HockeyPuck · · Score: 3, Insightful

    All this is is a method to line some guy's pockets. I'm sure the tape guys are gonna say, use XYZ type of tape. The disk guys are gonna say disk.

    What makes this guy think that the interface to the HDD is going to be around in X years?

    PC's have only had two dead (non-(e)IDE/ATA) interfaces, the ESDI and the ST506/ST-412 interfaces.

    But what if you were trying to find a computer with IPI (1960s mainframe) interface.

    The Fed gov't has this problem with trying to find parts for their old 8/9track tape drives..

    Here's a good list of all the HDD interfaces over the years: http://www.i-t-s.com/corporate/terms.html

    Stick with microfiche, film, that way we don't have to pay some vendor $$$/yr to keep alive a dead technology or pay some other vendor $$$/media to move them from old to new media.

  16. Intraframe vs. interframe by Wesley+Felter · · Score: 4, Interesting

    For whatever reason (I'm not a video expert) many people prefer intraframe codecs for archival. As you probably guessed, Motion JPEG 2000 just treats each video frame as a still image and compresses it with JPEG 2000.

    Dirac will give much better compression that JPEG 2000, but it also introduces the possibility of interframe artifacts.

  17. GIGO and the "born digital" problem by retiarius · · Score: 3, Insightful

    indeed, lossless for archival preservation is the
    only way, as it fits the basic rule of art restoration
    technology -- never apply "improvements" which
    cannot be reversibly undone to take advantage
    of future science.

    ironically then, the lossless format doesn't matter.

    however, at least for the instant case of dance video,
    the likely input (a myriad of digital tape formats)
    is hopelessly neanderthal -- anything having to do with DV,
    or MPEG, or even ATSC HDTV already tosses away much
    color information. (4:1:1, 4:2:0, and 4:2:2 colorspace is embarrassing
    to preserve "losslessly".) ditto for temporal
    info, with interlacing being the culprit. even film at
    24fps just will not cut it for motion such as dance.

    so here's to better camera technology, whether it's
    10- or 12-bit 4:4:4 RGB, or something like
    carver mead's foveon made swift.