Slashdot Mirror
Afterlife Will Be Costly For Digital Films
Andy Updegrove writes "For a few years now we've been reading about the urgency of adopting open document formats to preserve written records. Now, a 74-page report from the Academy of Motion Picture Arts and Sciences warns that digital films are as vulnerable to loss as digitized documents, but vastly more expensive to preserve — as much as $208,569 per year. The reasons are the same for video as for documents: magnetic media degrade quickly, and formats continue to be created and abandoned. If this sounds familiar and worrisome, it should. We are rushing pell-mell into a future where we only focus on the exciting benefits of new technologies without considering the qualities of older technologies that are equally important — such as ease of preservation — that may be lost or fatally compromised when we migrate to a new whiz-bang technology." Here's a registration-free link for the NYTimes article cited in Andy's post.
"Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it."
- L. Torvalds
Why is it more expensive to preserve a bunch of bits and bytes than, say, a reel with analog information, printed on some soon-to-be-brittle plastic? I'm very sure the latter will decay in a quicker fashion.
Preservation was a lot easier when the media lasted longer but by far the largest problem is the increase in the amount of data.
What is interesting is that old analog film & tape also degrades, but does so more gracefully. They also get degraded by reading, not just by storage. Archives of old footage etc have largely been converted to digital to allow older signals to be accessed without damaging the originals.
Engineering is the art of compromise.
release the file into the public domain and put it out on bit torrent? You'll get lots of backups made, for free. It will get converted to new formats, and backed up again, for free. Oh, you want future profits? Then quityerbitchin about the archival costs.
"National Security is the chief cause of national insecurity." - Celine's First Law
I cant help but relate some personal experience here. I know its not production quality, or lots of information, but I recently pulled out my Apple IIe from storage. It included the original 5 1/4 floppy disks and drives.
There was also a cardboard box with ~150 floppy disks, some as old as 20+ years. NOT A SINGLE ONE WAS BAD. Yes, "Zork" still works!
Could it possibly be that the quality of media just isn't up to the demands of a longer life of storage anymore? We all know how Cadillac runs that racket, as in sell the crappy car, and make the money off replacement parts. Has media storage gone the same way? As in 'sell the media, but just good enough to work for x years' before being replaced. And with the demands to increase revenue year over year for public companies, perhaps that time-frame has become shorter and shorter over the years to keep the money flowing in.
Or am I just being too cynical? But you know, a world where such works as "Zork" can survive and "Legally Blonde" can not, on their respective media, might not be that bad.
As jonadab once put it:
:-)
> Those who do not study history are doomed to repeat it
Yes, and those who do study history are doomed to watch in frustration
as it is unwittingly repeated by those who do not
Please correct me if I got my facts wrong.
Or, if they don't preserve Chicken Little, will the sky not fall?
Seriously, IF the older films are an authentic art that deserves preservation, the why is most of it scrapped on the cutting room floor? why are all the really old films sitting still on their Nitrate Stock in archives in hollywood slowly turing from film to dust?
AS others point out, released to the Net a movie is saved in various codecs, on various media (hard drive, tape CDR DVDR laserdisc even film FOR FREE just like music and most other data is. Horrible thought that; information in the hands of the people.... unsupervised, heck UN TAXED!
In the 15th century the Church tried desperately to put an end to this new Printing Press because it was putting their scribes out of work. They even excommunicated printers. Now we do the same only we use Lawyers.
I await the next turn of the wheel to see what damn foolishness humans are yet capable of..
- Minutus cantorum, minutus balorum, minutus carborata descendum pantorum.
The reasons are the same for video as for documents: magnetic media degrade quickly,
The myth of bit rot on hard drives is just that- a myth. It's been perpetuated for two decades by the idiot Steve Gibson, selling his own snake oil (Spinrite), and unfortunately, not enough people are calling him on it. I thought it actually did something too, until I read that post from someone who actually knows how modern drives work. As the author points out, there's a track that can only be written at the factory, and if what Gibson claimed were true, ALL drives would be dying left and right after a few years. Funny how I've found drives made almost a decade ago working just fine now...
The problem hasn't changed; it's mostly obsolescence in drive interfaces, and the drives themselves (for tapes.) PATA is common these days, but everything is going towards SATA, for example.
Both DAT and 8mm were in common use as little as 6-7 years ago...but you'd be fairly hard pressed to find a place to but either now save eBay. And...do YOU want to entrust a backup to an ebay drive?
Please help metamoderate.
It's not just the finished product, it's all the footage they keep around for different editions, remastering, deleted scenes, etc. And the source material is often not compressed in a lossy format. Sure, 4000 TB will store a lot of DVDs, but it won't store many movies in raw format. And only a fool wouldn't also have backups.
Be relentless!
The answer is simple, copy it over frequently.
Yeah, from the article, are several silly things are going on here:
Find free books.
If they want to permanently archive digital media, why not just keep the DVD glass masters around? They shouldn't degrade like plastic, and if carefully packaged it seems that they could last for millenia. If a special reader were developed that could optically scan the glass surface without the need for a rot-prone metal layer, then the information could be retrieved without having to risk damaging the master by making a new pressing.
Where he compares salt mine storage of analog media to storage of digital media, and decides to just multiply his made-up $208k figure by 100 years to come up with.. wait for it... $208 million. I guess that's why he went into journalism and not the sciences.
Leaving out the humongous math error, why can't you just store the digital fucking media in the same salt mine? The things that damage analog film are the same things that damage digital media.
Is it any wonder we have the expression "lies, damned lies, and statistics"? This article is all three, with some incompetency thrown in.
It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
They had to hire an MSCE to migrate the data from proprietary Windows Long-Term Archival Backup Media Video format (.wltabmv) to the new, safer (from pirates and such, arrr) long-term Windows Long-Term Protected Archival Backup System Against Pirates And Intellectual Property Theft Format Media Video (.wltpabsapaiptfmv)
And some, god help them, migrated to Apple's Almost Better Than The Competition So You Can Feel Better About Using A Proprietary Format For Only Three Dollars a Pop Codec (.aabttcsycfbauapffotdapc). Those Apple Engineers cost bocoup bucks.
Please stop stalking me, bro.
The standard motion picture format is MJPEG2000. It's not a very efficient format, but it's well defined and going to be around for a long time: there's both a lot of hardware and software that relies on it, and it scales up to high resolutions.
The consumer format wars between Microsoft, Apple, Sony, and other companies have no influence on this.
Once again repeat after me... the benefit of digital is not that it LASTS FOREVER or is EASIER TO PRESERVE. It is that it is EASY TO COPY.
Who gives a rats ass if a given copy of a film will degrade in 10 years. I can make a 100% perfect copy of the thing in minutes. Copy the data every year. Hell copy it 100 times. Copying also makes the obsolescence of formats meaningless.
I still have emails and RTF documents written in 1994. These are 100% perfect copies of the original data. Is that somehow to be interpreted by brain-dead fear-mongers that any day now my data will be "obsolete" since the obviously 15-year old media is almost degraded beyond recognition? Or are people a bit more intelligent and realize I have already copied this from hard drive to disc and back about 30 different times?
First, what kind of film was it that had a tendency to burn? Nitrate-based film?
Second, I just heard that the studio that produced Aerosmith's first album has lost the masters, so they're going to re-record it.
This kind of problem isn't new, and blaming it on electronic media is silly.
Yes, you do have to take steps to ensure the availability of it in the future - but the same is true of analog versions too. If you don't have a good filing system, or your 'vault' is the backseat of a car in southern California, the reels are going to get damaged/destroyed/lost, too.
I was on a railroad photographers' list for a while, and I remember the digital/analog debate came up one time. Someone said, "I'll be laughing when you lose all your files because your hard drive crashed and don't have pictures any more!" Obviously he never considered he could easily lose his negatives/slides, or have them damaged in a flood or fire. Analog media has different risks and storage requirements, but they BOTH require proper storage. (And, frankly, digital has the additional advantage that it can be easily backed up at multiple sites with no loss in quality.)
Technicolor dye transfer (imbibition) prints were much less fugitive. Color separations onto black and white film stock (often termed YCM for yellow, cyan, magenta) are much more robust. Production of these separations (and imbitition relief "matrix" films) was intrinsic to the Technicolor printing process (even if the film was shot in conventional tripack negative, then transferred to Technicolor for printing), and films where these intermediates were saved (or where someone presciently thought to have a set of YCMs made), are much safer for the future than anything kept only on color stock.
In the 70s there were some photo places (especially in Los Angeles) that marketed Eastman Color Negative 5247 movie film (short-end remnants from the movie industry) as a cheaper alternative for 35mm color negative still photography, and printed this onto 5283 color print film (same as movie prints) for 35mm slides.
I recently found a few boxes of these that I had shot back then (and stored under entirely careless, or Arrhenius/Murphy if you prefer, conditions). I am not good at evaluating color negatives by eye, but the positives were faded either to mutated colors or to almost nothing.
Even simple technologies can have amazingly short shelf lives under conditions of disuse. I recently turned on my stereo system after close to 3 years of not being used. The amplifier, CD player, and LP turntable all failed to operate. Part of this might have been due to de-formed electrolytic capacitors; these appear to have more-or-less repaired themselves after a couple of hours with the power turned on. Both the CD player and the turntable suffered additional electromechanical problems that required a combination of manual exercise and cleaning to rectify.
None of these devices have anywhere near the scary sophistication of a modern hard disk drive.
Seeing as I cannot remember what I last set my external firewall password to, imagine the additional challenge of future Hollywood being bitten deeply in the butt by present Hollywood's favored time-bombed destined-to-be-lost-art proprietary DRM technologies, with the keys long since dissipated in Hollywood's perennial miasma of mergers, acquisitions, lawsuits, cocaine, and personal vendettas.
Maybe they are living in 2007, where they are paying a $200,000 a year licensing fee to a patent troll who got a patent for "A business process which preserves digital motion pictures".
In all seriousness, the biggest obstacle to preserving a history of our culture is copyright. If the owner of the copyright doesn't care to preserve the piece of our history that they have their monopoly on, the information will simply deteriorate and there is nothing legally that can be done about it. We can only hope that the evil dirty thieving pirates save our history for future generations.
The DPX format commonly used for digital post production uses about 35 megabytes *per frame*.
My calculator says a 2 hour movie at 24 frames/sec will have about 175,000 frames.
A few more button presses tell me that's a bit north of 6 terabytes of data.
Let's quadruple that to include all the cut scenes and unused footage, to 25 terabytes.
TB drives are available now for $400 or so each. They use under 10 watts idle.
Building a 30 drive RAID would thus cost $12,000, and require perhaps 500 watts if run constantly, including cooling. Let's bump that to $15,000 to pay for controllers and chassis.
Three such arrays (in case of earthquakes, etc... keep 'em at opposite ends of the continent) would cost an initial $45,000, take up perhaps 7u of rack space, and need 50 kWh per day for all three. At 30 cents per kWh, that's 15 bucks a day, or $5500 per year. Let's double that, assuming those 7u cost you $5500 a year.
So... my numbers, triply redundant, come to an initial investment of $60,000 (profit, hey!), and a yearly cost of $20,000 (more profit!).
How the hell they came up with $208k is beyond me. I'm thinking I should start a company that does this for the studios, it's looking quite lucrative.
..in that order.
Yes - You don't need to have 5.25" drive now to read back data that you stored onto an 'old' IDE drive 2 years ago. And that's a bad example because you can still get 5.25" drives. 200 years from now when we're working with crystalline storage methods, we won't have to read back from HDD platters.. just from the holographic storage drives that things were transferred to with the last generation of storage devices.
Will we still have film projectors 200 years from now? Possibly not.
Whocares - because the formats used to store digital film aren't exactly H.264 or whatever fancyschmancy codec the copyright-infringent care about; google 'digital intermediate'. And yes, those formats do tend to change, but they all remain lossless and, again, things can be transferred with each generation.
Will we still know what to do with film 200 years from now? Ahhh.. there's the kicker.. probably, yes.
This is also where the cost comes in - you have to keep upgrading to the latest formats and the latest storage devices to ensure that there will be no 'digital divide', so to speak.
With film, you don't incur this cost. It's lossy in an analog sense, but if somebody looks at a film reel 2,000 years from now - and we assume to still have the same visual system in our watersacks - it will be trivial for them to see, literally, that it is a series of pictures which, in succession, appear to animate. Even if there's no device to play them back then, it would be trivial to build one from scratch using very rudimentary knowledge.
With digital, even if you have the latest format and the latest hardware to read the device it's stored on, it is non-trivial for the layman to read this file and be able to put it back into a picture; in fact, it tends to take people with intricate knowledge of the device and the storage format.
Personally I'm all for doing both, costs be damned, if the material is important enough. That said, do we really need to hold on to all material forevermore? Like a history book, it should be enough to retain the highlights (be they positive or negative), and not cling onto minutiae, as a society. Similarly, like family archives, those who believe something to be well worth the preservation for future generations (either within the family or civilization as a whole), will - or at least should - do so on their own and have history prove them right, or wrong.
For any given resolution, there is a size of film that beats that resolution. Figure on about 100 line-pairs per millimeter of film for still frames, less for movie frames, and you'll be in good shape. If you want a nice margin of error, quadruple the size or resolution of whatever you are copying to in each direction. If your film hits the silver screen at 4,096 pixels in the vertical dimension, this is roughly 2000 line pairs or 20 millimeters of film, well within the bounds of standard movie film. Quadruple that and you've got 80mm. Granted that isn't a standard movie-film size but it's technically doable.
Color gamut may be an issue in edge cases, particularly if the digital color has a higher dynamic range than color film will support. Black and white film used for certain archival purposes has a dynamic range big enough to hold the 12-bits-per-color that digital cinemas use. Color film may be a problem though - either the dymanic range will have to be flattened a bit or the highlights or shadows sacrificed. Printing the film multiple times with different settings should preserve all the color information.
References: Digital Cinema. The HD quality we're talking about here is much higher 'resolution' than even 64-inch film 64-inch film??? Most hollywood film movies are 35 or 70mm, which is less than 3 inches. Also, as far as preserving the 'instructions' to make a film... film is an art. Simply preserving data on camera angles and having some kind of code will not do. Films aren't computer programs - they're art. Being able to reliably copy the Mona Lisa exactly the way it was originally painted will never replace the value of the actual Mona Lisa. If this is true it is true only in a hyper-technical sense. For the average consumer and even the average scholar, an exact brush-stroke-for-brush-stroke reproduction using materials identical to the original materials would be more than adequate.
When you watch a film-print movie you are not watching an original anyways. You are usually watching copy or even an nth-generation copy. Plus, if you watch any print played more than a few dozen times, the copy you are watching has scratches and other damage not intended by the artists.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
> I know that's moded funny, but that might actually be a very good argument for "open sourcing" movies.
I wouldn't call it "open sourcing" exactly, but let's just say that films won't soon go extinct, at least as long as there are people willing to copy them.
Actually, that's how books survived. The only ancient books we have now are the ones people thought were important enough to copy regularly, plus a few random things that survived for a ridiculously long time.
Like Barbra Cartland? Or Penny Dreadfuls? Or the RFC Archive? Or YouTube?
Huge amounts of fundamental culture simply disappears because it is so transparent or ordinary to those it affects. The next generation comes along and they forget about it because of that apparent mediocracy. For example, breast feeding was normal, ordinary, and public in America up through the 1950's. Movie and later Television rule-makers didn't allow showing it unless it was part of some National Geographic type presentation. Today, breast feeding is being re-discovered in a storm of controversy because an entire generation has not only forgotten, but confused the topic with beer commercials.
Then again, how many people want to remember Phillippine Midget Snuff films? And why?
Pacifist paratroopers yell, "Ghandi!" when they jump.