Slashdot Mirror
'Hobbit' Creates Big Data Challenge
CowboyRobot writes "In the past five years there has been an 8x increase in the amount of content being generated per every two-hour cinematic piece. Although 3D is not new, modern 3D technologies add from 100% to 200% more data per frame. In 2009, Avatar was one of the first movies to generate about a petabyte of information. The Hobbit: An Unexpected Journey was shot in a new digital format called High Frame Rate 3-D, which displays the movie at 48 frames per second, twice the standard 24-fps rate that's been in place for more than 80 years."
But with digital storage transcending some other limitations of conventional projection techniques, it's not just framerate that directors are now able to play with more easily; it's the length of movies themselves, which stats suggest just keep getting longer.
I read TFA, and nowhere does it say how big The Hobbit was.. only that Avatar was about a Petabyte. Why isn't this stated anywhere? It's very frustrating, and also makes the article less useful, since its entire premise is that "The Hobbit creates big data challenge" with no specificity.
Am I the only one who longs for the return of an intermission? If only for a little relief rather than ducking out for 3-4 minutes and missing that one important little line of dialogue on which the whole thing pins?
So there I was, scribbling down some notes off the PC screen by hand, when I reached for the keyboard and Ctrl-S'd.
It was so classy. I'm sure it would help with the theater owners concession sales as well.
There may be no "I" in team, but there's also no "F" in way.
:poor guys that pirated avatar.. they had to download over a petabyte:
Huh? Most of the stuff you find on the various pirate channels is compressed down - at the most, you'll get a raw Blu-Ray rip. You can still get an xvid (avi usually under 2GB) version of just about anything that is available in any other format.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
That's only relevant for the first copy.
Beyond that, content will be heavily re-compressed. It doesn't matter if it's Apple pushing the bits or The Pirate Bay.
A Pirate and a Puritan look the same on a balance sheet.
The Petabyte figure is almost certainly for all of the working copies of the movies while it was being produced. Nobody is sending a Petabyte to every theater in the country, and much less to every home. Once the movie is finished a final copy is compressed and sent to theaters and the disc authoring house. The disc authors have to further compress the image to make it fit on the Blu-Ray or DVD. Your average pirate is going to compress the movie even further because full Blu-Ray rips are still rather unwieldy for most broadband connections and personal storage solutions.
I read the internet for the articles.
which stats suggest just keep getting longer"
And in the Hobbit's case, longer, and longer, and...... just waay too long. LOTR movies had 1000 pages of book to fill them with interesting content. Hobbit, not so much. In many of the scenes you can almost feel the director guy just out of camera view making that "stretch" motion with his hands.
Tolkien writes that Hobbits are between two and four feet (0.61–1.22 m) tall, the average height being three feet six inches (1.07 m).
I'm ok with advances in technology and the new challenges it creates. What I'm not OK with is a director deciding to make the source material "better" by changing the narrative. Jackson completely gutted Tolkien's Hobbit, rearranged the important events, and has replaced a light-hearted adventure story with the dark themes from LotR. Mr. Peter Jackson, why do you hate the work of JRR Tolkien?
The Admin and the Engineer
I guess they mean it is easy to get carried away during the recording phase because "film" is very cheap now. This presents a storage challenge. I imagine keeping 1 PB in redundant, geographically diverse storage would get spendy fast.
I suspect it would be more cost-effective to convert the raw bits to some kind of common standard and then just save the raw footage and the finished copy. Saving all the editing stuff won't help in the long run anyway - chances are the editing programs 20 years on won't know what to do with your old working files.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
there is a reason for them to be long(>2 hours).
I'm a HUGE Tolkien fan, and went to the LOTR Extended Version Trilogy Marathon recently before seeing The Hobbit.
I was surprised at how well the longer versions of the films held up, after not watching them for around five years.
However, The Hobbit film was a let down on several levels, most of which I won't go into here. My main complaint? You do not need three films to tell the story. PJ has thrown in everything but the kitchen sink into The Hobbit, and it drags. Even the uber-videogame-esque "escape from the Goblins" scene drags... Too much of a good thing can ruin a film.
I would also say the same thing about the last Batman film. Too long and drawn out. Scenes that should be edited or removed alltogether. Thats why they call it the Directors Cut!
It makes me wonder if there aren't people involved in the film such as producers or editors who tell guys like PJ or Nolan, "hey bro, you might want to trim things down, just a smidge... You know, just to kind of keep the flow of the film going"
We play the game with the bravery of being out of range
At the London preview screening Peter Jackson said that because 48fps + 3d is 4x the frames it's taken longer to render and the last scene with the coins was only finished a couple days before the premiere. He did mention the complexity in moving coins though
I know people who are still downloading it, you insensitive clod!
BS the size of the versions released to consumers will remain limited by the capacity of the media it is on and in any case the pirates can always recompress.
Afaict what this sort of thing is really about is flexibility. Want if they want to zoom in on something? or run something in slow motion? even remove something from a scene? it's much much cheaper if they can reprocess the existing data than if they have to re-shoot the scene. Compression artifacts that are invisible to the human eye during normal playback of a peice of video can cause big issues once someone starts messing with the footage. Also interframe compression turns even basic temporal editing into a lossy process.
So the solution is ideally to avoid compression at all during production and if they can't avoid it to stick to intraframe compression only.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
the biggest problem actually should be how to present 48fps content in the home screen...
Why should this be a problem? It's close to 20 years since I bought my first double-rate TV[*], and today, they tend to be quad-rate (200 or 240 Hz depending on where you live).
48 fps in stereo won't need more than 96 Hz progressive.
If you think of Blu-Ray discs, they're not locked to 30/60 (or 25/50) either, like older generations of video.
[*]: A Grundig, which could do 24-frame movies in 48 Hz non-interlaced or 96 Hz interlaced or video in 50p/100i PAL or 30p/60i NTSC. Of course, there wasn't a lot of video sources that delivered more than 25p/50i.
How does this have anything to do with Big Data? Storing large amount of data isn't the important part, it is being able to analyze that data. You do not analyze a movie's data file. You just load and display the movie, which can easy be stored in one large continous file. A Big Data problem would be Netflix trying to determine what kinds of movies to recommend, not storing and then displaying a long movie to users.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
I mean they did the 1000 page Battlefield Earth in under 2 hours in film and that turned out great...
"Battlefield Earth" + "turned out great" in the same sentence?... Travolta, is that you?
Nitpick: LOTR was 6 books, and not a trilogy. It's just that publishers often published it as three volumes.
The standard pirate sizes ate 700MB for a recompressed DVD-rip and 4.4GB for a recompressed blu-ray rip. These sizes are used because they are just small enough to fit onto a CD-R and DVD-R respectively. It's not universal though. Especially long or difficult films might go up to 8GB in HD, and there has been a recent trend towards smaller files where quality wouldn't be compromised rather than just assuming media-size for everything.
Even for 48fps 3D, it would require less than 2TB/hour of uncompressed 4:2:2 video (at 1920x1080), so although nobody is shipping a petabyte around, it's possible that the uncompressed data is being shipped around.
Except that they're probably storing it with some kind of RGBA (32-bit) uncompressed standard, which brings you to ~2.6TB/hour. And then if you decide to shoot it in 4K (4096x2160) that brings you to 11.2TB/hour or a bit over 30TB for the raw version of The Hobbit (48fps, 3-D). Now add in all of the rough cuts, editing revisions, unused footage, CGI, and everything else and you could see it *very* easily getting up over a petabyte. That's just for the studio though. What goes out the door, even in its rawest form, wouldn't get anywhere close to that.
As an aside, even ridiculously oversampled audio, running at 192k, 96-bit, 8 channels, and uncompressed is only going to run you ~62GB / hour.
The Hobbit is projected at 192 images/second. Each frame is shown four times, left, right, left right.
The Hobbit, scarcely 300 paper pages of children's story in print, leads to a big data problem. Now there's excessive bloat if I ever saw any.
Learn from the mistakes of others. There isn't enough time to make them all yourself.
Private trackers that I have access to offer full-frame blu-rays which are between 15 and 45 GB in size (or close to that). Using metropolitan P2P connections, I can download those monsters in 1-2 hours, depending on size. Movies which I download and manage to watch entirely deserve me buying a cinema ticket which I don't use. I buy the cinema ticket online, I don't go, everyone's happy. Rare are the movies I watch more than once, and those I usually buy as a hard-copy anyway.
...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
I just decided to settle for the compressed version. Much smaller file size and I don't think it degraded the quality appreciably.
Knowledge Brings Fear