Slashdot Mirror
Afterlife Will Be Costly For Digital Films
Andy Updegrove writes "For a few years now we've been reading about the urgency of adopting open document formats to preserve written records. Now, a 74-page report from the Academy of Motion Picture Arts and Sciences warns that digital films are as vulnerable to loss as digitized documents, but vastly more expensive to preserve — as much as $208,569 per year. The reasons are the same for video as for documents: magnetic media degrade quickly, and formats continue to be created and abandoned. If this sounds familiar and worrisome, it should. We are rushing pell-mell into a future where we only focus on the exciting benefits of new technologies without considering the qualities of older technologies that are equally important — such as ease of preservation — that may be lost or fatally compromised when we migrate to a new whiz-bang technology." Here's a registration-free link for the NYTimes article cited in Andy's post.
Store a copy on celluloid.
"Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it."
- L. Torvalds
Thats 416 1 TB drives - more than enough to backup the digital data for a few feature films several times....
Here's to hoping for a brighter future... for our children.
Why is it more expensive to preserve a bunch of bits and bytes than, say, a reel with analog information, printed on some soon-to-be-brittle plastic? I'm very sure the latter will decay in a quicker fashion.
Assuming that a digital movie can be stored in digital format, on digital media...
Preservation was a lot easier when the media lasted longer but by far the largest problem is the increase in the amount of data.
What is interesting is that old analog film & tape also degrades, but does so more gracefully. They also get degraded by reading, not just by storage. Archives of old footage etc have largely been converted to digital to allow older signals to be accessed without damaging the originals.
Engineering is the art of compromise.
release the file into the public domain and put it out on bit torrent? You'll get lots of backups made, for free. It will get converted to new formats, and backed up again, for free. Oh, you want future profits? Then quityerbitchin about the archival costs.
"National Security is the chief cause of national insecurity." - Celine's First Law
Does this mean no vintage porn in future?
Too bad no mechanism has been created for conveniently distributing around the world hundreds of thousands of copies of a digital film on old-fashioned analogue media, eh.
$208,569
That number is also awfully accurate for an estimate.
"We are rushing pell-mell into a future where we only focus on the exciting benefits of new technologies without considering the qualities of older technologies that are equally important" I want my vinyl records back.
Nobody writes a report like this just for the benefit of the hard drive manufacturers!
Engineering is the art of compromise.
... the same story as this http://en.wikipedia.org/wiki/BBC_Domesday_Project digital document? The answer is simple, copy it over frequently. Granted, certain obscure works will be lost from time to time, if they're skipped over. But so what? I remember once seeing a documentary about some old silent picture, as an example of pictures made on film stock that was rapidly decaying. It was a comedy about newlyweds, pretty much the same story that's been told time and time again.
I cant help but relate some personal experience here. I know its not production quality, or lots of information, but I recently pulled out my Apple IIe from storage. It included the original 5 1/4 floppy disks and drives.
There was also a cardboard box with ~150 floppy disks, some as old as 20+ years. NOT A SINGLE ONE WAS BAD. Yes, "Zork" still works!
Could it possibly be that the quality of media just isn't up to the demands of a longer life of storage anymore? We all know how Cadillac runs that racket, as in sell the crappy car, and make the money off replacement parts. Has media storage gone the same way? As in 'sell the media, but just good enough to work for x years' before being replaced. And with the demands to increase revenue year over year for public companies, perhaps that time-frame has become shorter and shorter over the years to keep the money flowing in.
Or am I just being too cynical? But you know, a world where such works as "Zork" can survive and "Legally Blonde" can not, on their respective media, might not be that bad.
As jonadab once put it:
:-)
> Those who do not study history are doomed to repeat it
Yes, and those who do study history are doomed to watch in frustration
as it is unwittingly repeated by those who do not
Please correct me if I got my facts wrong.
How much can they save by dropping DRM so any backup system will work?
upload those films to FTP server (or torrent one ;) and let the world mirror contents.
Step 1: upload your complete digital master on p2p file sharing networks.
Step 2: wait.
300 years from now archaeologists will look back and see a sudden stop to music, documents, and film. This is what is at stake, our cultural place in history. - SPT
I would think digital would avoid the problems with conventional film where the distribution prints get scratched, faded, and lose segments where the film broke and was spliced back together. Plus the masters are subject to being lost and having the colors degrade in strange ways. Many films have been completely lost and others are only available in an incomplete form. At least with DVDs, a film is unlikely to be lost, even if the DVD version doesn't have the same quality as the master print.
Mea navis aericumbens anguillis abundat
We are talking about backing up originals.
They only put DRM on the films before it gets to the consumer.
... it seems that the first (per movie) is about reasonable, the second is for justifying high prices, quote:" "Right now the best available cost estimate, when we look at our total cost of ownership for storing on disk, which is accessible all the time, is about $1,500 per terabyte per year," says Moore. The cost for archival tape is considerably less, about $500 per terabyte per year, with retrieval time (or latency) in minutes."
CC.
TaijiQuan (Huang, 5 loosenings)
The reasons are the same for video as for documents: magnetic media degrade quickly,
The myth of bit rot on hard drives is just that- a myth. It's been perpetuated for two decades by the idiot Steve Gibson, selling his own snake oil (Spinrite), and unfortunately, not enough people are calling him on it. I thought it actually did something too, until I read that post from someone who actually knows how modern drives work. As the author points out, there's a track that can only be written at the factory, and if what Gibson claimed were true, ALL drives would be dying left and right after a few years. Funny how I've found drives made almost a decade ago working just fine now...
The problem hasn't changed; it's mostly obsolescence in drive interfaces, and the drives themselves (for tapes.) PATA is common these days, but everything is going towards SATA, for example.
Both DAT and 8mm were in common use as little as 6-7 years ago...but you'd be fairly hard pressed to find a place to but either now save eBay. And...do YOU want to entrust a backup to an ebay drive?
Please help metamoderate.
You probably ruined all that media by running on your 20-year-old IIe, with its deteroriating drive. :-P
It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
What the studios could do is agree upon a universally adopted optical medium to store 4 hours of 8K video. Then store it in a humidity-free environment.
The problem isn't as dire as the article may make it out to be. Sure, high-quality originals might be lost, but there're still be millions of copies in DVD, HD-DVD, Blu-Ray or even the digital formats theaters use. The greatest advantage to digital isn't the longevity, it's the redundancy. And as long as you keep the plans for the decoding machines/CODECs hanging around, they will never be unreadable.
Whatever the limitations of the current archival formates have, it's still better than Nitrate film. That stuff would spontaneously combust if not taken care of, and would decay very quickly in non-perfect conditions. The stuff is literally made out of gun cotton (Nitrocellulose.) That's currently the archivist's major concern, is preserving all this volatile film from the 1900s thru the 1930s.
In the near future there will be an easy, hassle-free way to permanently preserve digital film.
Release it to a DRM-free peer to peer file-sharing network (ie, what's commonly known as piracy). Honestly. Back when I had access to a fast on-campus i2 filesharing network, I was always amazed at the breadth and depth of the files being shared. I was also impressed by the redundancy, as even obscure or classic movies could be found on a large number of peers. Furthermore, the speeds were so ridiculously fast - 10 Mbits/s - that I pretty much considered any peer that was connected to the network an extension of my hard drive. On one occasion I chose to download a file that I already had backed up on disc off the network instead of actually copying from the disc, as it was less of a hassle and about as fast. Marveling at how quickly my roommates and I could pull down high bitrate pr0n, I wondered at the possibilities of such a network distributed across the world, the individual personal storage of entertainment files of the individual peers being effectively part of the communal global network.
When the infrastructure exists to support i2-level speeds at the consumer level, the best way to preserve digital film files would be to release the film into public domain and distribute the file via peer-to-peer. Data loss is no longer an issue when your file is distributed to peers throughout the world. Natural disasters and other acts of God are similarly no longer an issue. In the event the Academy suffers data loss, they can just become another peer on the network.
I've got some experience with this...I've been building a multi-terabyte library for over a decade now...the goal is a tablet computer that could teach someone to rebuild civilization.
.TXT, .RTF, .PDF, & .DJVU for text files. I keep all audio in .MP3 format. Any original encoding is OK, but there must be a copy in 16KB/s, 16bits/sample, fixed sample size encoding - in stereo. Given that 99% is talk, that's OK. 8KB/s, 8 bits/sample, mmono are common, but do not play on ALL MP3 players. 16/16/stereo will play on all MP3 players.
.TIFF you can get, and test the heck out of it on 100 random PCs and software packages. NOT THE PCs THAT MOVIE EXECS USE! Grab 100 off eBay and see how many support each of 5 image formats, then tweak the settings in PhotoShop when you save the files to see how to improve the reliability.
Let's assume they want to make certain the data is available, and not in an unusable format...that would mean using a few (very) standard formats. I would have lost my data if I had kept it in the latest, slickest formats. I deal with
For movies, the codecs are a nightmare, so resample every frame into the best
For media, storing it all in one vault is stupid. Google figured out that you need to have 3 coppies in different locations. RAID is for idiots who want to loose data. One disk per file, and no exceptions. Use only standard file systems, and test them on the 100 PCs that we mentioned earlier. I can still plug in an early HDD, and use it in a modern system.
Accessing the data regularly will discover defects before they destroy all your data. Then again, I think Hollywood' main purpose is to prevent people from watching their movies...so why would they ever want to access it?
I'll stop wating my time here...Bye.
Andy Out!
Would it not be cheaper to print the movie using a more traditional archiving method (paper, microfiche, analog tape)? Plenty of digital->analog methods are available, and surely some more could be invented if necessary. Might be a lot of pages, but hey-- paper is cheap and may not degrade as quickly.
Just a thought.
W
-------------------
This is my SIG. There are many like it, but this one is mine.
If they want to permanently archive digital media, why not just keep the DVD glass masters around? They shouldn't degrade like plastic, and if carefully packaged it seems that they could last for millenia. If a special reader were developed that could optically scan the glass surface without the need for a rot-prone metal layer, then the information could be retrieved without having to risk damaging the master by making a new pressing.
I was very surprised to read the following in this article:
"DVDs tend to degrade: according to the report, only half of a collection of disks can be expected to last for 15 years"
For some reason, I thought that both CDs and DVDs each have a shelf life of around 100 years. I think that was one of the benefits of owning a CD over a tape. I remember reading that back in 1992 when I made the decision to buy a CD player and start buying music CDs. Tapes can get de-magnetised, twisted up, caught in tape players etc. But the real reason I started buying CDs is because of the revolutionary "next track" function! I suspect many kids thought the same. Why have to fast-forward when you can immediately get where you want to be on a CD? My apologies for this being a little off-topic, just got a bit nostalgic there.
So film companies want their digital masters to last hundreds of years why? So they can capitalise on them again when our 15 year-old copies have "degraded" and are no-longer watchable? Or will they capitalise on our great-grandchildren in 100 years? If a 15 year shelf-life is indeed true, I feel sorry for all those who went out and purchased tonnes of DVD films. What's been bothering me are these "ultimate collectors' edition" re-packagings that the film companies have been selling. Yes, I love the film Blade Runner, but I swear I'm not going to go out and buy another copy of it. Please just release the original film, the way the director wanted it to be seen. I don't care about owning a six DVD collection of all the different versions of the film. I just want the longest, un-cut and uncensored version available! If they sold it this way, and all us Blade Runner lovers bought a copy, would they need to keep transferring the original to new media every year?
Over 200 000$ a year to take care of one film's digital master seems like a ridiculous amount of money. I'll gladly take on the responsibility of a couple of films for them, and I'll only charge them half what they're paying now. One could build a pretty sweet rig with a nice RAID array for a fraction of the cost and keep a few digital masters there on the hard drives.
Where he compares salt mine storage of analog media to storage of digital media, and decides to just multiply his made-up $208k figure by 100 years to come up with.. wait for it... $208 million. I guess that's why he went into journalism and not the sciences.
Leaving out the humongous math error, why can't you just store the digital fucking media in the same salt mine? The things that damage analog film are the same things that damage digital media.
Is it any wonder we have the expression "lies, damned lies, and statistics"? This article is all three, with some incompetency thrown in.
It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
1/ Draw each frame on a sheet of papyrus, staple the whole thing together on one edge, making a flip book, and hide the whole mess in jars in caves in the desert. Don't forget to include copies of the scripts.
2/ Devise an obscure religion based on your film, spread it to as many people as possible.
3/ Wait.
As nearly as I can tell, the whole concept of recorded history probably ended when we developed means to record reality directly, rather than transcribing it to clay slabs, stone, and paper.
Where's that "-1 RTFA" moderation option when you need it? Oh right, it's called "+1 Insightful" on /.
With all the push by the various arms of media industry to keep finding ways to continue to generate revenue from their products, I'm sure they'll be pushing the envelope with long-term storage solutions. Large capacity storage used to be considered anything greater than 1 GB with technology that was available "way back when" (not that long ago, really). Nowadays, that's a ridiculously small amount of storage that I can (and do) carry around in my shirt pocket.
Computing power used to be awfully expensive, too. Now we've got desktops that are capable of scientific computing sitting around at 99% idle all day. If it weren't for Vista, we wouldn't even be using a tenth of the memory built into them (sorry, had to stick a dig in there somewhere).
My point is that as the market demands new capabilities, technologies emerge that satisfy those needs. As time goes on, the efficiency of these technologies increases while costs decrease. It's just how things work. Today's data retention problems for studios will contribute to tomorrow's advances in long-term storage technology.
I can think of at least a couple of major companies that also have a vested interest in long term archival... Google... cough... Google...
512 MB RAM, 20 GB disk, 200 GB transfer, five datacenters. $19.95/month.
analog also decays. The difference is that it is easier to pull SOMETHING out of it as it decays. The downfall of analog is that it is is MUCH more expensive to protect.
Back in 90/91, I worked for a company that did burning of CDs and Laserdisc (compressed data for the DOD). The CDs cost something like 5 or 10 each, and the laserdiscs were a couple of hundred each. IIRC, These were based on gold, and would last something like 50 or 100 years without losing a single pixel. I would guess that hollywood could easily afford these.
I prefer the "u" in honour as it seems to be missing these days.
The last two digits are all that is important here...
The actual author of that quote is George Santayana, according to this source:
The standard motion picture format is MJPEG2000. It's not a very efficient format, but it's well defined and going to be around for a long time: there's both a lot of hardware and software that relies on it, and it scales up to high resolutions.
The consumer format wars between Microsoft, Apple, Sony, and other companies have no influence on this.
Once again repeat after me... the benefit of digital is not that it LASTS FOREVER or is EASIER TO PRESERVE. It is that it is EASY TO COPY.
Who gives a rats ass if a given copy of a film will degrade in 10 years. I can make a 100% perfect copy of the thing in minutes. Copy the data every year. Hell copy it 100 times. Copying also makes the obsolescence of formats meaningless.
I still have emails and RTF documents written in 1994. These are 100% perfect copies of the original data. Is that somehow to be interpreted by brain-dead fear-mongers that any day now my data will be "obsolete" since the obviously 15-year old media is almost degraded beyond recognition? Or are people a bit more intelligent and realize I have already copied this from hard drive to disc and back about 30 different times?
The way I see it, there's not going to be any official world organization for the preservation of content (in fact, anyone advocating this will be looking to make a quick buck). Rather, it'll be in the hands of each end-user to archive, maintain, and provide worthy content. The different whims and tastes of every single individual will combine into a true representation of what mankind deems valuable, even more so when deciding to upload. This is already happening with BitTorrent trackers (the oink.cd closing comes to mind - disgruntled users migrated to minor clones and took it upon themselves to make the music available again, mostly thanks to their process of selection). Over time, what is really worthy of archival and culturally relevant will still be sought after - especially by newer generations of users. I don't think they'll be talking about movies like Gigli and Glitter in 100 years. Ok, maybe for argument purposes on what's *not* to be done. But I can guarantee you people will still know of Led Zeppelin, Citizen Kane, 1984, and so on. Even a 20-something like myself has benefitted greatly from the people going through the hassle of ripping and providing content from way before when I was born - especially things like reels, lossless soundboard recordings, OCR and so on.
The format issue is somewhat relative - as long as it works on what's commonplace at the time, and value preservation over quality improvement (I mean really, most movies today are fine in the non-HD format, even VHS would be a fine support for the crap Hollywood likes to spew). And if you're talking about content you created (things like family pictures and movies come to mind), I don't think it's worth the hassle of putting it out for the world to download... We're talking about content with a much higher cultural relevance.
Current movies are already printed to film for viewing in theaters, so the problem isn't at a crisis point yet. The problem will come when major film manufacturers quit making movie film.
If the major studios demand it and are willing to pay higher prices for low manufacturing runs, film manufacturers will still make the film. I predict this will happen for the forseeable future.
By the way, nothing but cost says you can't take each element in a digital scene and print it out to its own frame in addition to or instead of printing out the movie frame-by-frame. Also, nothing says you have to use 35 or 70mm format: If your original digital image has more resolution than you can store on 70mm film you can use a larger format.
You can also use microfilm techniques to print technical information such as the descriptions of camera angles and even computer data files and computer code in human-readable, hex, or some other form to film for archiving, along with the computer code for the programs and enough information to build a virtual computer to interpret that code. Sure, it's a lot of information but remember, the goal is to put all of the information in a storage box and be able to retrieve it in 100 years and make use of it.
If they had done this level of preservation with old NASA computer data and data-descriptions we wouldn't have some of the problems we are having today with un-interpretable data.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
digital films are as vulnerable to loss as digitized documents
Maybe that's because digital films are digital documents? How much was the consultant paid who sorted that one out?
And I am skeptical about the DVD rot I keep hearing about. Doesn't a DVD have to undergo so much degradation that a "1" on the disk becomes a "0"? Whereas on analog media, any bit of degradation starts to have an effect on the content of the media. In other words, all the "1" bits on a disk might become ".95", but binary data would still be intact, whereas small color changes would occur on analog media. I realize that even one incorrect bit could damage a compressed piece of data, but why in the world would they archive using a compressed format?
Yakelope Marisco
Simplest way to store these in a non-lossy format would be to use something like, say, a "linear DVD" - which would bear more of a resemblance to a punch card or the metal "cards" the Difference Engine would have used than anything else. Divide each "card" into 13 zones and use microscopic holes (rather than simple changes in brightness) to encode one channel (vR, vG, vB, aFL, aFC, aFR, aCL, aC, aCR, aBL, aBC, aBR, aSW) per zone, one frame per row (for the audio channels, an amount of 32-bit mono audio equivalent to one video frame's duration), at whatever the native (both pixel and time) resolution is.
For those of you wondering how big these holes would be, they'd be roughly 7.5nm in size (for a card sized to match letter-size paper) - larger bits than even a CD. This leads to a capacity of about 13,000 hours per "card" - and the resulting system, due to being a primarily mechanical recording rather than primarily optical, is more resilient to changes over time.
It's obviously not a format that's readily implemented for a home computer, but as for archival purposes, it'd work quite nicely.
For the same reason (as you agree) it'll continue to fade away.
The whole point of both interfaces is to allow external, portable interfaces. I know that any office I go to with a portable USB drive will be able to read it. Worst case scenario, it's over USB 1.1, but it's still usable.
If I bring a firewire drive, I'll get funny looks, and it has a pretty good chance of not working. Even though most recent mid to high end motherboards have it, almost everyone has never used it. There's a very high chance that drivers aren't installed, or that the front panel jack isn't even plugged in. Of course, there's also the very high percentage of laptops without a jack at all, and lower end desktops.
First, what kind of film was it that had a tendency to burn? Nitrate-based film?
Second, I just heard that the studio that produced Aerosmith's first album has lost the masters, so they're going to re-record it.
This kind of problem isn't new, and blaming it on electronic media is silly.
Yes, you do have to take steps to ensure the availability of it in the future - but the same is true of analog versions too. If you don't have a good filing system, or your 'vault' is the backseat of a car in southern California, the reels are going to get damaged/destroyed/lost, too.
I was on a railroad photographers' list for a while, and I remember the digital/analog debate came up one time. Someone said, "I'll be laughing when you lose all your files because your hard drive crashed and don't have pictures any more!" Obviously he never considered he could easily lose his negatives/slides, or have them damaged in a flood or fire. Analog media has different risks and storage requirements, but they BOTH require proper storage. (And, frankly, digital has the additional advantage that it can be easily backed up at multiple sites with no loss in quality.)
1. Build a pyramid 2. Paint copy of movie on walls of catacombs 3. ? 4. Profit
Excuse me, but please get off my Pennisetum Clandestinum, eh!
One of the fundamental qualities of any digital media is the separation of abstract data from the means of it's storage and expression. Analog media is inevitably related to the physical way of expressing or recording it. Copying a vinyl disk is identical to copying the physical shape of the tracts on the disk; referring to analog "music" is identical to referring to the wavelengths we hear.
With digital, data and medium are separate. By itself, data is just an abstract collection of ones and zeroes. While technically data cannot "exist" without a medium, we logically pretend that it can (much like we can refer to the mathematical integers, even though in nature there is no such thing as just "1", only "1 of something"). As long as any physical device is able to transmit and recieve the exact set of 1s and 0s, it doesn't matter what technology that device uses to store or reproduce the data. Of course, any two devices that share the data must use a protocol both understand. But this bears no restriction to the protocol the devices would each use with a 3rd party.
The advantage of digital, therefore, is its liquidity. Data can long outlive any physical medium it is stored on, by simply flowing around different devices, and while physical technology of storage and expression can change from era to era, the data itself remains constant.
On the downside, this opens a new set of challenges - degrees of -logical- as opposed to physical openness, and these include the question of open or closed media and restrictions placed on its use by the logical software that processes it (AKA DRM). While analog media is pretty much restricted to either available as is or not available at all (in the form of encryption for example), digital media can be much more manipulated with regard to what and how is available to access.
Digital media relies on people (or devices) that are actually going to use it. If you dump a recording in a time capsule for generations, there is little advantage between digital and analog. Digital vs. analog is like abstract money vs. barter trade - it means nothing on its own, but it is a logical expression of a particular value (rather than the physical object itself), and works best when it is flowing around, able to represent the value of physical objects that are seemingly incomparable to one another, and requires people to have acceptance of that logical value and common ways of exchanging it for it to have any advantage.
The CB App. What's your 20?
So all my important communications I write by hand in archival black ink on acid-free 100% rag paper. For truly mission-critical stuff, I have some stone tablets.
Nothing stops them from making a celluloid (analog) copy. That way you have at least the same preservation as the old stuff. Are there more options now? Yes. Is that bad?....
Table-ized A.I.
I think the word you were looking for is precise, not accurate.
What about microfilms? I know this is a terribly old solution, but that's what was used in the 1940 time capsule that's scheduled to be opened in 8113. Surely, that's terribly old tech, but anyone can read microfilms.
You just got troll'd!
That's quite possible, thanks.
:-)
English is not my native tongue.
I'm no expert, but AFAIK the problem with CDs is that the data layer is actually on the outside, and is thus subject to anything you can think of: friction, corrosion, different expansion rates potentially promoting separation (that's why you have to be very careful what you use as marker - you're writing directly onto the data layer.
With DVDs the data layer is a bit safer, but there you have the problem that it has to be properly sealed (and that's assuming the sealant itself is not in any way affecting the data layer).
As for the figures, well, I suggest you treat them like statistics. Very decorative, but always in need of full validation..
Insert
I think you underestimate the resources that some enthusiasts are willing to expend. Might be a problem for some films, though.
"National Security is the chief cause of national insecurity." - Celine's First Law
i'm a photographer and this comes up in my day to day life...
digital is easier for storage in the short term, negatives take up far more physical space and the threat of catastrophic physical damage from flood or fire is very real.
but in the long term, negatives have a better chance of survival due to the fact that you can simply hold it up to a light to read the information. If I died today, in 100 years the images stored on my harddrives and DVDs would simply be lost, or at the very least, uneconomical to recover. A properly stored negative will still be around, and while perhaps not in perfect condition, it will still be readable with a very simple device - your eye, and a lightbulb.
-
that won't degrade. punchcards! Just don't make 'em out of paper. How about aluminum or something?
Why is it more expensive to preserve a bunch of bits and bytes than, say, a reel with analog information, printed on some soon-to-be-brittle plastic? I'm very sure the latter will decay in a quicker fashion.
Analog film is extremely fault tolerant. No compression or encryption to go wrong due to a bad bit. No key frames and deltas that can cause errors to perpetuate from one frame to the next for a while. Digital is great in that you can have perfect reproduction, but it is also vulnerable in the sense that it requires perfect reproduction (yes error correction bits can mitigate this to a degree).
Technicolor dye transfer (imbibition) prints were much less fugitive. Color separations onto black and white film stock (often termed YCM for yellow, cyan, magenta) are much more robust. Production of these separations (and imbitition relief "matrix" films) was intrinsic to the Technicolor printing process (even if the film was shot in conventional tripack negative, then transferred to Technicolor for printing), and films where these intermediates were saved (or where someone presciently thought to have a set of YCMs made), are much safer for the future than anything kept only on color stock.
In the 70s there were some photo places (especially in Los Angeles) that marketed Eastman Color Negative 5247 movie film (short-end remnants from the movie industry) as a cheaper alternative for 35mm color negative still photography, and printed this onto 5283 color print film (same as movie prints) for 35mm slides.
I recently found a few boxes of these that I had shot back then (and stored under entirely careless, or Arrhenius/Murphy if you prefer, conditions). I am not good at evaluating color negatives by eye, but the positives were faded either to mutated colors or to almost nothing.
Even simple technologies can have amazingly short shelf lives under conditions of disuse. I recently turned on my stereo system after close to 3 years of not being used. The amplifier, CD player, and LP turntable all failed to operate. Part of this might have been due to de-formed electrolytic capacitors; these appear to have more-or-less repaired themselves after a couple of hours with the power turned on. Both the CD player and the turntable suffered additional electromechanical problems that required a combination of manual exercise and cleaning to rectify.
None of these devices have anywhere near the scary sophistication of a modern hard disk drive.
Seeing as I cannot remember what I last set my external firewall password to, imagine the additional challenge of future Hollywood being bitten deeply in the butt by present Hollywood's favored time-bombed destined-to-be-lost-art proprietary DRM technologies, with the keys long since dissipated in Hollywood's perennial miasma of mergers, acquisitions, lawsuits, cocaine, and personal vendettas.
..in that order.
Yes - You don't need to have 5.25" drive now to read back data that you stored onto an 'old' IDE drive 2 years ago. And that's a bad example because you can still get 5.25" drives. 200 years from now when we're working with crystalline storage methods, we won't have to read back from HDD platters.. just from the holographic storage drives that things were transferred to with the last generation of storage devices.
Will we still have film projectors 200 years from now? Possibly not.
Whocares - because the formats used to store digital film aren't exactly H.264 or whatever fancyschmancy codec the copyright-infringent care about; google 'digital intermediate'. And yes, those formats do tend to change, but they all remain lossless and, again, things can be transferred with each generation.
Will we still know what to do with film 200 years from now? Ahhh.. there's the kicker.. probably, yes.
This is also where the cost comes in - you have to keep upgrading to the latest formats and the latest storage devices to ensure that there will be no 'digital divide', so to speak.
With film, you don't incur this cost. It's lossy in an analog sense, but if somebody looks at a film reel 2,000 years from now - and we assume to still have the same visual system in our watersacks - it will be trivial for them to see, literally, that it is a series of pictures which, in succession, appear to animate. Even if there's no device to play them back then, it would be trivial to build one from scratch using very rudimentary knowledge.
With digital, even if you have the latest format and the latest hardware to read the device it's stored on, it is non-trivial for the layman to read this file and be able to put it back into a picture; in fact, it tends to take people with intricate knowledge of the device and the storage format.
Personally I'm all for doing both, costs be damned, if the material is important enough. That said, do we really need to hold on to all material forevermore? Like a history book, it should be enough to retain the highlights (be they positive or negative), and not cling onto minutiae, as a society. Similarly, like family archives, those who believe something to be well worth the preservation for future generations (either within the family or civilization as a whole), will - or at least should - do so on their own and have history prove them right, or wrong.
"Long tail" is even worse for movies and TV shows. The only way I really see this working is if it was also accompanied by a massive investment by, say, Google.
Don't thank God, thank a doctor!
Quite a lot. I've had to pull apart proprietary formats to extract data from old tapes, and it's a sreious pain that wasted my time, and their time lying about how unique it was. (I'm sorry, but a tar file hidden by 10 blocks of header and chopped into chunks of 1023 blocks is not that hard to re-assemble.)
Split an uncompressed copy across 10 or so archival quality DVDs. Do this a few dozen times - store the copies in various geographically dispersed locations and be done with it. Should last at least 100 years or so. If you are worried about degradation, make new copies every decade or so. Worried we won't have DVD players in 100 years? CD players are almost 30 years old already. When exactly are they going away?
There's an obvious solution to the problem.
Why not just make a copy of the movie on 35mm film?
Movies are just strings of pictures shown one after another (plus sound), there's nothing that prevents a computer from showing each of them and "burning" them into celluloid. After that, you've got a copy that can be preserved in case your digital copy fails (or your only copy, if you want to store then cheaply).
GPG 0x1B479C78
> I know that's moded funny, but that might actually be a very good argument for "open sourcing" movies.
I wouldn't call it "open sourcing" exactly, but let's just say that films won't soon go extinct, at least as long as there are people willing to copy them.
Actually, that's how books survived. The only ancient books we have now are the ones people thought were important enough to copy regularly, plus a few random things that survived for a ridiculously long time.
I'm not convinced we need to keep 90+% of youtube or Friends and similar crap for people to watch 100 years from now.
Engineering is the art of compromise.
Just use AVI... it's not proprietary... :)
-=LaptopZZ=-
http://www.rosettaproject.org/
Whoever builds a technology to read/write the encoded information to/from the Rosetta Disk that includes both video and audio: http://www.rosettaproject.org/about-us/disk/concept will be a rich company.
As somebody who runs an art gallery that shows and sells a lot of video art, this is something I think about and deal with regularly. My current solution is to keep DVD masters and back them up to hard drives. Some are playable DVDs, some are DVD-ROMs with Quicktime files (usually what we use for high-definition videos).
Once everything is a on a hard drive, it's easy to migrate the work to other video formats. If DVD players become difficult to find, we can easily convert the MPEG2 video to MPEG4 or whatever.
The current standard for archival video art is DigiBeta, but it's on its way out -- primarily because it doesn't support high def.
Film is obviously much larger data, but with Moore's law, data volume will soon be a non-issue. There is a limit to how much resolution our eyes can see and films are not likely to start getting longer, so we are probably close to hitting the limit for digital film data size. According to wikipedia, the data rate for Redcode Raw (4096×2304) is 220Mbit/s. At this rate, a 90 minute film is about 150 gigs. In a few years, redundant backups of these should be trivial.
-paul
Hold up a white sign with a big black "0" painted on it. Record a frame of the sign with a movie camera onto archival-grade film stock.
Next, hold up a white sign with a big black "1" painted on it and record a frame.
Repeat several trillion times, alternating between the two signs with varying frequencies, and you can rest assured that your digital films may be safely preserved in their original high-definition quality while still using the traditional film archival techniques.
However, be warned that even this archival method is several orders of magnitude less secure than the tried and true monkey media storage systems that have effectively preserved great literary works that date as far back as the English Renaissance.
Not that this is unique to digital formats. "Missing Presumed Wiped" often recovered recordings so old that only one or two machines still existed that could play them. I shudder to think of the technical complexities of the salvage efforts that were used to recover the Pathe Newsreels from the late 1800s. I've even got photographic negatives from a mining township in Africa from 1909, but you think it would be trivial for me to get reprints? I can't exactly take those down to Walmarts. Given how explosive old film was, I'm not even certain I can keep those negatives without violating fire and safety regulations.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
The answer to the digital storage question has always been the same: "who cares"
not in the flip teenager attitude way meaning that no one does, but in the "those who care will copy it" way. It is a decidedly social, non-technical answer to the issue.
Preserving digital content will happen by those people an orgs who want to keep copies of it, because someone cares enough to use it periodically. Overall, the median rate of use (making a copy) has to be shorter than the median decay rate for the system storing the content. If this is the case, the media survives, if not, it dies.
If no one cares enough to use and copy the digital content, then people will lose it.
I don't see where the "problem" is. We keep that which people are interested in paying to archive because it is accessed regularly, and the rest will be lost.
Although we are probably getting a little too carried away in making everything digital, there is a lot to be said for the long-term storage options of data in an analog form. Even if an item stored in an analog form is destroyed by 50% or more, it's not impossible to recover most of it with fairly reliable accuracy simply due to the amazing ability of the human mind to recognize common patterns and fill in the blanks. Even if the analog were warped out of it's original order, odds are good we could recover it.
On the other hand, digital archival of data, which can offer incredible clarity and potentially 1:1 accuracy in restoration often becomes an all-or-nothing proposition if even a tiny bit of the data is lost or altered. Even with file formats/codecs that offer some form of error correction or redundancy, the final result we may end up seeing could be little more than randomized shifts between a blank screen and a perfect image... all of which are swapped in and out so quickly, we may not see the recoverable parts long enough to identify any usable pattern.
For example, try comparing something like the "scrambled" channels (mostly the porn channels) on cable television back in the early to mid 90s to something like DirecTV during a heavy rain storm. Even though the cable stuff was typically visible warped and uncomfortable to look at, you at least had a good idea of exactly what was going on behind the scrambling, even without the audio channels. But, try watching a DirecTV signal under less than ideal weather conditions, and the best you get is a bounce between a random mosiac and pitch black, combined with severely degraded audio pops here and there. You're luck if you can even get a useful picture of anything on the screen, let alone being able to comprehend what is going on in the show itself.
That said, how difficult would it be to create a micro-film drive (photosensitive analog scanner/burner) that could not only store any document on a computer in an analog form, but do so in a format that could be interpreted entirely by the human eye using a proper magnifying device. For that matter, why not create a hybrid device that would store both an easily visible analog form of a document as a high-resolution thumbnail, along with a digital version using pattern of dots similar to how data would be stored on an optical disc. This way, no matter what device you use to extract the information, you'd always have the means to access the data you need.
8==8 Bones 8==8
Kodak will have the last laugh.
AAD (Analog Analog Digital), ADD (Analog Digital Digital), DDD (Digital Digital Digital), DDA (Digital Digital Analog)
The sad thing is you can't find Infrared film anymore (other than 35mm, and that may be in danger also) , because "everyone" is going digital.
It's a double edged sword for me, as I'm snapping up all these Hasselblads:
http://www.hasselbladusa.com/products/v-system.aspx
- that everyone is discarding, $22,000. systems, tossed like yesterday's salad.
(But, please don't listen to me - they ARE worthless, those lenses too, TRASH, I tell you! You need the latest:
http://www.bhphotovideo.com/c/product/522927-REG/Hasselblad_70380530_H3DII_39_SLR_Digital_Camera.html
There are experiments with holographic data storage systems:
http://www.research.ibm.com/journal/rd/443/ashley.html
http://www.research.ibm.com/journal/rd/443/ashle20.jpg
-as long term storage.
A generation of family / baby pictures will go down the tubes because no one thought about it.
We shoot products on digital (no loss) - but, that's it, the rest, 220 Velvia, 35mm slides and a cool dark place, - yeah, and I LIKE it like that.
~hylas
Like Barbra Cartland? Or Penny Dreadfuls? Or the RFC Archive? Or YouTube?
Huge amounts of fundamental culture simply disappears because it is so transparent or ordinary to those it affects. The next generation comes along and they forget about it because of that apparent mediocracy. For example, breast feeding was normal, ordinary, and public in America up through the 1950's. Movie and later Television rule-makers didn't allow showing it unless it was part of some National Geographic type presentation. Today, breast feeding is being re-discovered in a storm of controversy because an entire generation has not only forgotten, but confused the topic with beer commercials.
Then again, how many people want to remember Phillippine Midget Snuff films? And why?
Pacifist paratroopers yell, "Ghandi!" when they jump.
Film archival should not be a problem. I seem to recall an episode of "Buck Rogers" where Buck was suspected of some crime in the past, so the lawyers pulled out this five hundred year old videotape from a surveillance camera, put it into a wooden box with flashing lights, and it played fine, no problems at all. It was even better because the playback system was able to switch camera perspectives to get the most dramatic angles and stuff.
I don't see the problem: Take all the original footage, and record it losslessly (a few times - so that they can be stored on multiple continents) to a few blue-ray DVD-ROMS (not chemical based RW), and keep the masters.
Sure, it's expensive. Boohoo for the MPAA members that spend $200 million making a movie, charge theaters so much that they have to pass it on to me in the form of $7 popcorn, and can't spend (a one-time cost of) a couple dozen grand preserving their creation.
50 years from now you won't care that you can't play your Crossroads DVD.
For that kind of money, just print it onto a film for the long-term archive? Hell, maybe they could encode the digital data on film as patterns of white and black squares?
Karma: pi (Mostly due to circular reasoning in posts).
These clashing objectives are really getting out of hand.
The only reason there is a problem is because studios are so terrified of extra copies siphoning off their revenue. Wholesale disappearance of material they are no longer interested in doesn't seem to bother them.
Then they claim it costs $200,000 per item to store them? That would mean... *negative* revenues against sales of zero.
Is it reasonable to say that modern hard drives can last some twelve years? So, every ten years you simply copy over your materials onto the newest spiffy drive.
Someone talked about favoring popular materials... absolutely not. If there were *no penalties* involved, everyone would carve out a niche because the storage effort could be semi coordinated. Then you could find the French edition of Casablanca if you wanted to, because *someone* would have it.
My first Journal Entry ever, in 8 years! http://slashdot.org/journal/365947/aphelion-scifi-fantasy-horror-poetry-webzine
Those figures can't be right. To keep old film media, you need a huge warehouse, with temperature and humidity control. Too hot and humid, fungus growth. Too cold, it brittle. And guess how much to run such warehouse? Building the warehouse, install temperature control equipment, maintenance, electricity bills,etc, it can easily cost millions. I bet it is an accountant that help make up the report. Many accountant refuse to change the mindset of inventory pile up and warehouse cost. Isn't it obvious, since transfer "new media" involved cost, but film media wont' incur much of it. Well, not before you took into account of the cost of running the super warehouse for old films. OTH, reducing the films reel to digital forms to DVDRAM/Blueray/HDDVD, Terabytes disk platter can shrink the medium and reduce warehouse size, and save millions on long terms maintenance bills.
> Yes, and those who do study history are doomed to watch in frustration as it is unwittingly repeated by those who do not :-)
I've also seen it said that "Those who foresee misfortune suffer it twice."
Perhaps we should just archive anything and everything on analog media. Chances are we'll always have a way of converting analog media to whatever happens to be the going digital format of the day.
I think you meant language, not tongue :-)
1. The very best data-retention method known involves the "Curie Point" of a magnetic material. When lava fresh out of a volcano cools below the Curie Point, the Earth's weak magnetic field is sufficient to align magnetic particles. Geophysicists can "read" 100-million-year-old lava, and determine the orientation of the magnetic grains. This data-retention property was combined with a special optical effect (the direction of the magnetic grain affects the way light reflects), and "magneto-optical" data storage was born. They pretty much guarantee 40+ years of reliability; the disks haven't been around long enough for anyone to know just how long they actually can retain data. Unfortunately, the technology is being phased out, because a 3-1/2-inch disk only holds 2.3GB, not enough. The 5-1/4-inch disks are still around; they hold about as much as a DVD (on each side of the disk). A few of these MIGHT be enough to hold all the data for a digital production.
2. I think there's a rumor about hard-disk-drive technology planning on incorporating a combination of a tiny magnetic read-write head with a laser beam. The laser heats the disk surface to near the Curie Point when writing, the goal is extreme data density where neighboring bits CANNOT magnetically affect each other at room temperature --but a side-effect will be long-long-term data storage (at least until the drive mechanically fails).
3. We might imagine a special variation on the previous theme. Start with stainless steel, a substance with a long long lifespan; most of them are non-magnetic, but a few are magnetic. We want a variety that has magnetic properties but does not magnetize easily (that's what the Curie Point is for; just below that temperature, it will magnetize much more easily than at room temperature). Let's make a long thin ribbon of this stainless steel, and wind it on a spool, very much like a more ordinary magnetic tape. Yes, I'm aware that "wire recorders" existed in the 1940s, but they used easily-magnetize-able wire. Basically, this notion is to separate the storage media from the read/write technology, so that if that hardware fails, the data-media can simply be moved to replacement hardware.
4. Ordinary "flash" memory has a drawback in that it can only be re-written a limited number of times, but it is available in multi-GB sizes, and certain varieties of flash memory just happens to have a pretty long data-retention ability (40 years or more). Some of the new non-volatile memories coming down the R&D pipeline, such as MRAM, may offer even longer data retention ability. It seems to me that if a digital production was stored in such chips as these, simply READING the data should be fairly easy, modest hardware. All that remains to be standardized is the storage format.
See, bit torrent is a backup medium :)
http://www.rense.com/general79/wdx1.htm
I see your point, but in many cases, books and films may have historical value that isn't apparent right now. Very old movies that we'd hardly enjoy now still sometimes interesting because they contain propaganda or ideas that became influential.
I have no interest in watching the "Saw" movies, for example, and would not personally choose to preserve them. But future historians may learn something important about our time period by seeing that people were entertained by such cruelty and suffering.
the British Library is a copyright repository and they turned down quite a few now famous authors and books as they felt they were too populist. Now, I think, they don't turn down anything - just in case.
so, archiving things isn't just a problem for niche market things - quiet a lot of art is lost because it's not thought to be worthwhile at the time. Consider famous cave paintings - a few primitive daubs, now of immense value and interest.
As long as the movie is encoded into open format, the point of passing codecs is moot. It doesn't matter if it goes out of style, there will always be software that will read the open source codec - if there isn't, it doesn't take much to code it. As for drive failure, all you need to do is to keep copying, or making backups of, the file to new drives or media. This article is alarmist at best.
in girum imus nocte et consumimur igni
Because torrents go stale and die.
Increasingly, the edit decision list isn't given to a negative cutter; the final assembly is sent directly to a digital output device which generates film. I don't know if projection prints are individually struck like this or duplicated optically: I suspect it depends on the volume. The first film to do this starting from live-action shooting was O Brother Where Art Thou in 2000. It's obviously universal for projects shot on DV and then distributed on film, where there aren't negatives as such. This whole discussion will be dead within ten years, because anyone who has watched digital projection will be increasing dissatisfied with optical prints anyway. The only optical print I've ever seen that comes close to modern digital projectors is the dye sublimation print of Apocalypse Now done a few years ago. I used to go to the Stanford Cinema in Palo Alto and see clean prints of B&W classics: seeing the BFI's digital restoration of The Dam Busters a few months ago knocked those into a cocked hat.
Let's quadruple that to include all the cut scenes and unused footage, to 25 terabytes.
;)
Careful now - the historical archives, AFAIK (not much) don't store that much cutting-room-floor footage, so you've just imposed stronger requirements on digital than analog, and increased the cost.
That said, cutting room floor footage can sometimes be as high as 200x the final product. So if you want to go that route, your estimate is way low.
Again, we're making wild guesses without understanding requirements, so we know we're still on Slashdot.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
For more redundancy you could break up the size of the RAID into smaller RAID6 arrays and stripe them. Or use striped mirrors.
Hey, you better be using ZFS for this job!
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
They MUST be making some fairly bad assumptions to come up with their numbers.
I'll assume that no compression will be used at all (since this is for archival use) and that it takes 100 DVDs to store a full uncompressed archival master. For safety, we'll make 5 copies. That's a media cost of about $120. Now, store the media and a few computers capable of reading it for 10 years in several safe places. After 10 years, copy it to whatever storage is well documented and well known at that time. Assuming 5 computers w/ DVD drive and 5 copies of the film, decent pay for someone to make the copies, and that those computers will be dedicated to just that one film then rounding UP, we come to around $10,000 for 10 years. That leaves us with about $100,000 for 100 years.
Of course, in real life, it will cost less because we want to archive more than just one film and so don't have to dedicate 5 brand new computers to each film. Also, the trend is going to be for each 10 year generation to get cheaper than the previous one since capacity is always increasing. By the end of 100 years, all of the films may be archived on a single holographic cube of some plastic like substance (of course, with 4 backup copies) and still looking exactly like they do right now.
All of this assumes the only option is to use high end consumer grade hardware. Surely it's possible to come up with a DVD like format with much greater durability. Perhaps burning tiny holes into a sheet of stainless steel and encasing it in resin. It's not necessary that an archival format be capable of real-time recording or playback.
It's not as if the old analog film has fared all that well anyway. A fair number of them on nitrocelulose stock spontaniously combusted years ago. Others turned into a mass of toxic and potentially explosive goo and were recovered and restored at great cost (and are NOT exactly like the original).
On more than one occasion, archived video and film has been lost but was recovered from somebody's personal copy. The more widely distributed something is and the less DRM it has on it, the more likely it is to be preserved. Should a DRM mechanism that can't be cracked in a few months ever go into use, it will likely become the leading cause of lost media a few decades later.
Cineon was the standard for a long time when it came to archiving or exchaning the final frames, but EXR has pretty much taken over. The problem really isn't archiving the final frames, its the fact you need to archive every element used to make those frames, and those files could be rla, exr, cin, jpg. This could be 1PB or more because people get really paranoid and just archive everything online when the shot is finaled.
As soon as you leave the walled garden, you have to use grown up tools.
It's ubiquity. USB is everywhere, firewire is not.
If the system you need to connect to doesn't have a firewire interface? A lot more that 10% slower, I'd imagine.
This article is a farce. There will always be a market in content. Cost of storage will drop, archives and redundancies will become more prevalent. This article could have been written twenty years ago and instead of video being the focus, it could have been written about documents, encyclopedias, what have you. After video, what next? Preserving a digital model of the neural cortex of brilliant minds like Einstein's? Absolutely, it will happen (assuming the technology enables it). People generally have a tendency to save, make backups, etc. to document our history. This will not change.
I neither use nor care one bit about AV tools.
I guarantee the people you reference have their cheques signed by people using grown up computers.
You asked for my personal experience with AV equipment. That's when it "became about me".
You'll find you're incorrect.
That's how we choose standards, for the most part.
If you only intend to plug the drive into your own systems, there's scant reason for them to be external. You might as well throw them inside the case. If you're moving around too much data to be practical over a network (or over long distances) and you control both endpoints, eSATA is probably better.
If you need portability, you need an interface that you can count on being everywhere. That's not firewire.
I use external drives because I have data that I don't need or want (backups) to keep on my main HDD, as well as for transporting data. iMacs don't have room for extra drives -- most people don't need them. Same goes for laptops, too, before you try to get me on that one.
If you need portability, you need an interface that you can count on being everywhere. That's not firewire.
Damn right. But not everyone is looking for those things; some people have other requirements. Otherwise, you could make the same argument for eSATA.
At the moment, comparing USB to FireWire is still like comparing apples to oranges. They do have a large crossover in terms of what they do (hence our discussion), but the USB 2 spec, and AFAIK, the USB 3 spec, still hasn't made FireWire redundant. Maybe it will in the future -- who knows. If so, then perhaps FireWire will fade away.
I think you've granted that for portable media, USB is king, which was my original point.
For non portable (or minimally portable) external drives, there are other superior interfaces to firewire. If multiple people need to access the data, NAS on a gigabit lan does the job better. If large amounts of data need to be moved around a few discrete locations, eSATA is superior.
USB alone may not have made firewire redundant, but it is redundant none the less. I fail to see any real use cases were there isn't a better solution.
I know very well what it is.
For all reasonable applications, one of NAS, USB or eSATA is superior to firewire.
You're running back into the portability argument. What are you going to do with that camera when you're on vacation and need to use some netcafe with windows 98 boxen?
Why would you ever need to stream live footage from a camcorder? I think you've stepped way outside reasonable use cases.
I've never seen a mac netcafe, and I seriously doubt they exist in any real numbers. Macs are still very much a niche product purchased largely by those with more money than sense.
Lots of people pass around things like cameras expecting them to "just work". That doesn't happen with firewire.
This discussion was actually about portable media.
This isn't an "anti-Mac" thing. I don't really care one way or another about Apple or it's products. Is it "anti-Mac" to ignore them?
Portability is a prerequisite to ease of use and reliability. It's pretty hard to use an interface that doesn't exist on the systems you want to use.
I don't imagine you'll find many netcafes that won't allow you to burn cds/dvds of data you bring in. You'll have to have a way to get that data on their systems though, and that isn't firewire.
USB host really isn't that tough. My mp3 player can act as one, and will snarf photos over a standard camera USB interface (mtp I assume, but I've never tried it).
Firewire does have many advantages over USB. That isn't in question. Most of them are rather
irrelevant, but not all of them. It probably will hang on in esoteric or high end niches, lots of stupid and irrelevant standards do (hello, minidisc).
Here's the ancestor.
Would you really want to find a firewire only external drive 15 years from now?