Slashdot Mirror
Digital Media Archiving Challenges Hollywood
HarryCaul writes "Movies are moving to digital, but what about long-term archiving of the master source materials? Turns out it's harder for digital media than for contemporary analog. Data is being lost, and studios have to learn to cope. Phil Feiner of the AMPAS sci-tech division says when he worked on studio feature films he 'found missing frames or corrupted data on 40% of the data tapes that came in from digital intermediate houses' How to deal with it? Regular migration from old media to new media. Grover Crisp, says Sony has put in a program of migrating every two to three years. Other studios are following suit, but what about indie features? Will we lose films like we lost the originals of the 20s?"
If they are concerned about digital data being lost: why not introduce redundancy? Make sure that the data is stored at many locations as possible (and also with a high quality). Luckily the Internet already has a solution for this problem: BitTorrent.
"... only wimps use backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it."
I think it'd work well for the MPAA.
Go somewhere random
Wouldn't it be great if in 50 years the only version left of [Insert Future Classic Movie] was a screener with watermarks?
The record industry has gone back to analogue tape. But.. you could store film data in a lazer beam and fire it off into space. Then, once we invent lightspeed transport, we can fly past the lazer beam and then recollect the data.
This will only get worse because they insist on the stupid DRM schemes. If a drive crashes you can usually recover a fair portion of the data, if the drive is heavily encrypted and the crash takes out the key to your cipher, then you are fairly fucked. Sure, it is fine today when everybody and his mother has a HDMI compliant player, but with the amount of key-revocations that will likely be necessary as the scheme is cracked over and over again, sooner or latter the increasing complexity of key-management will cause them to start getting lost. The issue is further complicated by having the "plain-text" all in a central place rather than in everybody's home, a hurricane could easily take out a decade's worth of art that way. Of course none of this will happen because the people who make decisions about where the unencrypted originals are stored have a good understanding of how cryptography works, which is why we have DRM to begin with ...
There are two problems:
Data loss, where the data is actually lost. This is the equivalent of a scratch on a frame of the master negative. The cure is redundancy.
Obsolescence, where the format becomes difficult to read after a period of time. The cure is lossless copying to new formats over time and/or keeping old equipment around.
Another possible cure the the 2nd problem is to convert it to analog in an "easy to digitize" way.
For example, simply "printing" the movie to 3 black-and-white filmstrips, one for each color, is considered archival. These can be rescanned later if needed. For better archiving, use larger film formats.
Preserve each audio track in an archival analog format as well.
Of course this doesn't preserve all the data that a digital filmmaking process has, but you aren't any worse off than you would have been with an analog film.
If you want to, you can preserve each element of each scene separately, in an analog format or a completely-documented digital format but on an archival media, such as a "paper printout" stored on microfilm. I don't think most movie studios will go to this expense.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
"Turns out it's harder for digital media than for contemporary analog"
The negatives of the original 'Wicker Man' movie were either burnt or buried under the M3 motorway. From what I remember, some of the original 'Babylon 5' negatives were eaten by rats. They're gone, nothing will ever bring them back, because they're analogue media which can't be copied without quality loss.
The problem is the whole idea of a 'master copy' of the movie on media that goes obsolete. The benefit of digital data is that it can be copied any number of times without quality loss, so build a big RAID system and stick the movies on there. Over time it will be upgraded but the digital data will remain... the only time you'll put the data on tape will be for backups, though even then you'd probably be better copying it to other RAID servers at remote sites.
This isn't news -- at least not to those of us who deal with data.
The typical procedure is to do a media refresh (ie, copy it) every few years, and to check for damage. There are concepts like LOCKSS (Lots of Copies Keeps Stuff Safe), so those joking about BitTorrent aren't that far off, but it's a little more structured than that.
Dan Cohen gave a talk recently on "Can Today's Scientific Data Be Preserved? The Specter of a 'Digital Dark Age'", which touched on not only the issue of media failure, but also the loss of the knowledge to extract the encoded information. (much like the 'lost languages' that we don't understand now, how do we make sure that future generations have the necessary hardware and software to get the data back out?)
What's disapointing is just how fast the media is failing. Vendors give a 'mean time to failure' estimate that's based on perfect storage, and that they have no real ways of testing (because, well, if you say it's 40 years, are we going to have to wait 40 years before using it?). Even if you're duplicating your tapes, what happens when all of the copies were put on the same potentially bad batch of tapes?
Quite likely, we're going to lose data. And some of it's going to be because we no longer have copies of the data. The rest is going to be lost because there's so much crap being saved that doesn't need to be that we can't find stuff that still has value in the future.
Build it, and they will come^Hplain.
How long until the pirates launch a giant 'mirror' (pun intended)?
to the original Star Wars. Supposedly he has no original copies from which to return the original classic to us (Laserdisc work-arounds notwithstanding)
I doubt his word on this, but if true, he's a bigger fool that Ep 1 made him appear. In any case, its a great case for multiple digital back-ups.
Cost may play a role as well - as important as it is for film history to save as much as possible, how may film makers in the early stages of a career have the money to produce high quality, redundant backups? And then maintain their viability over the years?
Sure strage is cheap - but who can be sure the hardware will be usable in say 50 years? Can a disk last that long without being spun up regularly? Is optical disk / flash memory archival over time? Will the hardware be readable on whatever computer is in use then or will it be like trying to read an 8" CP/M disk today? Of course, then there is the codec issue as well.
Lucas was dealing in analog which make it even more difficult to properly archive copies for posterity.
I'm a consultant - I convert gibberish into cash-flow.
However, what if they sent data that needs to be archived permanently through the first stages of the DVD mastering process, and produced an etched glass master disk. It seems to me that such a disk should last forever as long as it is protected from physical damage.
To avoid damage from creating new DVD stampers from the master if it needs to be read, maybe they could create a special archival reader based on electron microscope or something similar that could read the master disk directly without touching it.
Just back from Tinsel Town after talking to some of the dudes in the article. Still jetlagged, so it feels a bit unreal reading about it in Slashdot. Still, I'll do my best to explain why things are the way they are...
A film is not often made by a single body. If you are shooting to film, then this will get handled by an editorial department. You may have a fast telecine scan for reviewing the material as dailies. Some of these scans may be used as low-resolution proxies for initial grades. Some chosen bits of film may get re-scanned on a slower pin-resolution scanner for inclusion in the final film. Artificial rendered scenes and special effects may be done by specialist houses, then composited in a post-production house. Your film may have 25 4K images per second in the final version, but the data used to generate it is scattered over the place - if you think a good IT department should be backing all this up, then you haven't worked on a film, my friend. As deadlines approach, people may be working stupid hours, and filling up all the available storage. Then the film gets released, and either makes a billion dollars or doesn't. Either way, the tension is off, people take holiday and zonk out. Nobody will be picking over the cutting-room floor or its digital equivalent looking for things that might be useful twenty years from now. By the time people are back from holiday, they don't know or don't care.
Your end product may be big reels of negative film that you send to a film lab to make prints for cinemas. The lab should keep the golden master clean, and make most of the prints from a second copy. This would be a sensible time to make an archival print of the film. The lab can transfer the whole thing to black and white film. Black and white film does not fade, like conventional colour film does, even in the can. You are getting the print lab to do a pretty full backup of the released film when your people have all gone on holiday. These days you need to back up other stuff. The soundtrack is digital. You will have extra data for the releases in different formats (5:4 TV, 16:9 widescreen, IMAX, etcetera). Still, it is a lot better than nothing. But it is not often done.
The other think is to know what to archive. Very little of the newsreel film I had to sit though as a child to get to the cartoon has survived. Key stuff like the Queen's Coronation or the outbreak of WW2 was clearly history, and put on a special shelf, but little of the day to day stuff survives. There is one cache that survived when a cinema closed, and the tins of newsreel went into landfill. The cinema was in Alaska; the landfill was permafrost, and the film was kept in near ideal refrigetrated conditions. Apart from this fluke, it has probably all gone.
There will probably be digital solutions in time. Increasingly, as we have to manage more different sorts of digital data, there is a need to organize and track everything, which ought to mean it is possible to archive all the essential bits that go into any production. Many other people have posted on the problems of knowing what is on (say) a FAT16 Windows 3.1 disk in some 1980's image format. You can keep copying the data to overcome the degradation of the physical medium, but you still have to know what it means. I know of a system for archiving film images, where the people who did the archiving left the company, and one of them took the laptop with them that had the archiving software, so the ability to read the archives went with them. Do you archive the archiving system? Then, do you archive the system that archived that? Yes - basically, that is exactly what people are proposing to do. But it takes a bit of organizing, and we are not there yet.
Film, on the other hand, has visible images. The 35mm format has remained readable for over 100 years. Even where nitrate stock has flowed over time, we still know what shape it ought to have been. Sometimes we can get something back if we want it badly enough.
A simple analogue solution may be to