'Hobbit' Creates Big Data Challenge
CowboyRobot writes "In the past five years there has been an 8x increase in the amount of content being generated per every two-hour cinematic piece. Although 3D is not new, modern 3D technologies add from 100% to 200% more data per frame. In 2009, Avatar was one of the first movies to generate about a petabyte of information. The Hobbit: An Unexpected Journey was shot in a new digital format called High Frame Rate 3-D, which displays the movie at 48 frames per second, twice the standard 24-fps rate that's been in place for more than 80 years."
But with digital storage transcending some other limitations of conventional projection techniques, it's not just framerate that directors are now able to play with more easily; it's the length of movies themselves, which stats suggest just keep getting longer.
I read TFA, and nowhere does it say how big The Hobbit was.. only that Avatar was about a Petabyte. Why isn't this stated anywhere? It's very frustrating, and also makes the article less useful, since its entire premise is that "The Hobbit creates big data challenge" with no specificity.
Am I the only one who longs for the return of an intermission? If only for a little relief rather than ducking out for 3-4 minutes and missing that one important little line of dialogue on which the whole thing pins?
So there I was, scribbling down some notes off the PC screen by hand, when I reached for the keyboard and Ctrl-S'd.
It was so classy. I'm sure it would help with the theater owners concession sales as well.
There may be no "I" in team, but there's also no "F" in way.
:poor guys that pirated avatar.. they had to download over a petabyte:
This story is barely good marketing fluff. There is no informational value here beyond, more frame rate = more data. Post processing is hard.
Sometimes I wonder why journalism is dying, then I read crap like this. I want my 5 minutes back.
Huh? Most of the stuff you find on the various pirate channels is compressed down - at the most, you'll get a raw Blu-Ray rip. You can still get an xvid (avi usually under 2GB) version of just about anything that is available in any other format.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
That's only relevant for the first copy.
Beyond that, content will be heavily re-compressed. It doesn't matter if it's Apple pushing the bits or The Pirate Bay.
A Pirate and a Puritan look the same on a balance sheet.
...so your 3rd eye can maintain perspective.
But with digital storage transcending some other limitations of conventional projection techniques, it's not just framerate that directors are now able to play with more easily; it's the length of movies themselves, which stats suggest just keep getting longer.
Digital storage transcends limitations? And the limitations were due to projection techniques? So, storage is better at doing something than projection is? Yeah, projection techniques were bad at storing my files. Storage transcends that limitation and can store my files.
Here's a better version:
Movies are also getting longer.
It has nothing to do with the costumer side of equation, the file size reported is for the production of the picture, all shots in RAW format and the stereoscopic and 48fps factors that doubles the content each.
For your average movie rip, extra frame compression (not used in production) should take care of most of the extra bits, so don't worry... the biggest problem actually should be how to present 48fps content in the home screen... BD and TV compatibility, audio... damn, even at theaters there's some many problems with the footage showed before the actual movie, stereo incorrectly set in the trailers, audio going crazy...
Time is money. I think you're more likely to get in seat catheters before you intermissions.
The Petabyte figure is almost certainly for all of the working copies of the movies while it was being produced. Nobody is sending a Petabyte to every theater in the country, and much less to every home. Once the movie is finished a final copy is compressed and sent to theaters and the disc authoring house. The disc authors have to further compress the image to make it fit on the Blu-Ray or DVD. Your average pirate is going to compress the movie even further because full Blu-Ray rips are still rather unwieldy for most broadband connections and personal storage solutions.
I read the internet for the articles.
which stats suggest just keep getting longer"
And in the Hobbit's case, longer, and longer, and...... just waay too long. LOTR movies had 1000 pages of book to fill them with interesting content. Hobbit, not so much. In many of the scenes you can almost feel the director guy just out of camera view making that "stretch" motion with his hands.
Tolkien writes that Hobbits are between two and four feet (0.61–1.22 m) tall, the average height being three feet six inches (1.07 m).
I'm ok with advances in technology and the new challenges it creates. What I'm not OK with is a director deciding to make the source material "better" by changing the narrative. Jackson completely gutted Tolkien's Hobbit, rearranged the important events, and has replaced a light-hearted adventure story with the dark themes from LotR. Mr. Peter Jackson, why do you hate the work of JRR Tolkien?
The Admin and the Engineer
From 100% to 200% more data - so what? That's not a big step compared to the steps we had in the past like going from 1.44MB 3,5" floppies to 650MB/700MB CDs. Or from those CDs to 4.4GB DVDs.
Sounds like a job for handy, dandy DVD Shrink!
I don't respond to AC's.
Directors have never understood this, which is why most of the time they aren't responsible for the final edits. Length is one of the primary filtering factors for me to decide what movie to see. Anything substantially over 2 hours had better be a damned good movie.
there is a reason for them to be long(>2 hours).
I'm a HUGE Tolkien fan, and went to the LOTR Extended Version Trilogy Marathon recently before seeing The Hobbit.
I was surprised at how well the longer versions of the films held up, after not watching them for around five years.
However, The Hobbit film was a let down on several levels, most of which I won't go into here. My main complaint? You do not need three films to tell the story. PJ has thrown in everything but the kitchen sink into The Hobbit, and it drags. Even the uber-videogame-esque "escape from the Goblins" scene drags... Too much of a good thing can ruin a film.
I would also say the same thing about the last Batman film. Too long and drawn out. Scenes that should be edited or removed alltogether. Thats why they call it the Directors Cut!
It makes me wonder if there aren't people involved in the film such as producers or editors who tell guys like PJ or Nolan, "hey bro, you might want to trim things down, just a smidge... You know, just to kind of keep the flow of the film going"
We play the game with the bravery of being out of range
Why wouldnt someone simply re-encode the film?
Not everything is a conspiracy; increasing filesize in a format where excess data can easily be stripped out is a pretty terrible way to fight piracy.
The main benefit for going digital was to cut production costs down, you don't have to drag dailies around or courier anything beyond a few hard drives when you go digital (heck you can torrent between locations), but now you have to store data electronically which shifts the cost to running servers or some other backup.
The effective resolution of film is infinite, with good optics on your digital scanner you could go down to any resolution you want for easy of digital editing. Store the film in a vault and forget about it for 90 years.
The Hobbit was also shot (and maybe shown?) at 4K resolution.
That's another bump in the data size.
nope caps proved in canada that it wont dampen piracy it always finds a way
At the London preview screening Peter Jackson said that because 48fps + 3d is 4x the frames it's taken longer to render and the last scene with the coins was only finished a couple days before the premiere. He did mention the complexity in moving coins though
You've probably made one of the top 10 most idiotic comments on Slashdot. And that's saying a lot--it's like winning the Special Olympics. Among retards, you stand out!
I have all the intermissions I want because these days I watch movies / TV shows / videos exclusively on my computer (Netflix, Hulu, YouTube, (borrowed) DVDs, etc.) I do this partially as a "Fuck You" to the MPAA, but mainly because the "home experience" is by far more preferable to me than the "theater experience".
If you think enterprise storage subsystem can have usable capacity of 1024GB (1PB). So if you go beyond, just buy 2 of those (DS8700 comes to mind, but there are others by EMC and Hitachi)
I know people who are still downloading it, you insensitive clod!
It's interesting to note that for typical footage, as frame rate goes to infinity, so should the compression ratio.
BS the size of the versions released to consumers will remain limited by the capacity of the media it is on and in any case the pirates can always recompress.
Afaict what this sort of thing is really about is flexibility. Want if they want to zoom in on something? or run something in slow motion? even remove something from a scene? it's much much cheaper if they can reprocess the existing data than if they have to re-shoot the scene. Compression artifacts that are invisible to the human eye during normal playback of a peice of video can cause big issues once someone starts messing with the footage. Also interframe compression turns even basic temporal editing into a lossy process.
So the solution is ideally to avoid compression at all during production and if they can't avoid it to stick to intraframe compression only.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
100 to 200 percent more data for "modern" 3D technologies. Stereoscopy adds exactly 100 percent more data, as it has since the late 1800s. There is no new 3D. There wasn't for Avatar, there wasn't for Captain EO, and there wasn't for any of the other marketing-oriented attempts to act like the red and blue glasses were the "old" 3D while the polarized lenses are the "new". Didn't you kids have ViewMaster? Sheesh.
Cloudiot: A person who does not see offsite storage as a way to lose control over access to his or her own data.
I wonder why this is news... I guess it's a good time to invest in ENTERPRISE level storage!!
The Hobbit was shot in 5k resolution, and in "true 3D" with two cameras on every mount. And shot at 48fps (5k x 48 fps x 2 cameras) is MASSIVE film stock. Add multiple shoots for setup, testing lights, and then the actual acting. Not to mention all the digital elements that have to be stocked at high resolution as well.
Yup that's a LOT of data. But then so was Star Wars. I suppose with those really long copyrights it means THEY have to keep all that digital treasure safe.. Because they only sell the stripped out Home copies. The Hobbit was really shot for theaters 10years from now when 4k becomes commonplace. I used Star Wars because while Licas was counting his Billions, the actual MOVIE and resources on film spent 20 years rotting in a damp basement. Because every scrap of these movies has to be copyrighted for 95+ years they don't want any bits leaking out... But saving EVERYTHING it's really hard... Even if they make a billion dollars from it.
I still don't see what the "news" is. I mean Banks are so big they fill LTO 5 tapes with transaction data several times a DAY (not accounts, just the individual records of card swipes and such in real time) . I guess banks don't have to keep that 95 years though...
How much storage would they need if they dropped all of the "short people walking" bits except for one? Once again, lossy compression schemes come to the rescue!
No matter "big" the movies data is, no matter how many fps they film it at, no matter how many 3d effects they add in it still sucked and so do the majority of the movies that rely on special effects. Digital special effects make film makers lazy. In the past decade how many big budget special effects crazy movies have been made that were classics of film vs how many classic movies in the past decade didnt have computer effects? Quentin tarantino alone has made more quality movies in the past 10 years than big budget special effects films.
So bottom line is make it fancy as you want but that alone doesnt make it a good movie.
the biggest problem actually should be how to present 48fps content in the home screen...
Why should this be a problem? It's close to 20 years since I bought my first double-rate TV[*], and today, they tend to be quad-rate (200 or 240 Hz depending on where you live).
48 fps in stereo won't need more than 96 Hz progressive.
If you think of Blu-Ray discs, they're not locked to 30/60 (or 25/50) either, like older generations of video.
[*]: A Grundig, which could do 24-frame movies in 48 Hz non-interlaced or 96 Hz interlaced or video in 50p/100i PAL or 30p/60i NTSC. Of course, there wasn't a lot of video sources that delivered more than 25p/50i.
How does this have anything to do with Big Data? Storing large amount of data isn't the important part, it is being able to analyze that data. You do not analyze a movie's data file. You just load and display the movie, which can easy be stored in one large continous file. A Big Data problem would be Netflix trying to determine what kinds of movies to recommend, not storing and then displaying a long movie to users.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
I would think, additionally, that with the petabyte figure comes all the additional footage and audio as well as revision histories that never make it to the consumer.
Tolkien's work is love-it-or-hate-it and unfortunately I fall squarely on the "hate-it" side. I guess it's good to know that we can enjoy hours of tedium at a higher-than-normal frame rate, though.
I mean they did the 1000 page Battlefield Earth in under 2 hours in film and that turned out great...
Look at even LOTR, special editions probably make it a 12 hour film for the same page count.
I remember watching Ghandi in the theaters in 1982, and it had an intermission because it was too large for a single reel, so there was 10 minutes while they changed the reel. Spartacus, and Ben Hur were also over 3 hours. They are just listing the longest upcoming movies expected, with no actual analysis of percent movies over 2 hours and 3 hours per year. It's multiple anecdotes pieced together, ignoring all data contradicting the hypothesis.
Call me when the average time of a Disnet movie is 150 minutes or higher. Disney fires directors who refuse to keep them at 90 minutes (90 minutes is the time for one extra show a day, 2-3 shows more than the Hobbit). Makes for better results in the box office.
Learn to love Alaska
Comment removed based on user account deletion
The effective resolution of film is infinite, with good optics on your digital scanner you could go down to any resolution you want for easy of digital editing.
Nope.
Comment removed based on user account deletion
Everybody wants to go to digital distribution, but I refuse to accept compromise for the experience. I want uncompressed sound and stunning visual clarity in my movies, not some overly compressed barely HD content with stereo sound split to 5.1 false channels.
Everybody wants to move to the cloud but I live in a G8 country where my bandwidth is throttled and still stuck at 20th century download speeds and upload speeds that are barely better then dial-up.
So yes, the next big challenge for big data is to deliver on the promise of offering high quality, high bandwidth cloud solutions that don't actually suck.
I haven't thought of anything clever to put here, but then again most of you haven't either.
And what do we use this giant-screen, high frame rate, 3D movie to film? Megan Fox french kissing Amanda Seyfried?
No. A bunch of fat midgets and other tiny freaks, and two old guys, one with a layer of bird poop down his temple.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
You think that's big kiddo? Think about all the shit the brain sucking spy devices like google and facebook gather all the time. 24/7 365 Think about what science experiments produce, like nuclear research looking at subatomic particles. Think about storing all network traffic data globally, internet and phone calls.
Now. that's big.
You activate the app before the movie starts, it vibrates in your pocket st the start of "pee friendly" zones.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
I noted with interest a few job titles in the credits that I hadn't noticed before. In particular, there were 4 "Data Wranglers" for The Hobbit.
If you think of Blu-Ray discs, they're not locked to 30/60 (or 25/50) either, like older generations of video.
Actually, they are, although 24fps is also an option. But, no other frame rate is legal, so no hardware player will support anything but the required rates, which means no one will produce content at any other frame rates for Blu-Ray.
I remeber Avatar with Ang, Prince Zuko and Katara. Good movie. M. Night Shyamalan is awsome!
Nobody is sending a Petabyte to every theater in the country, and much less to every home. Once the movie is finished a final copy is compressed and sent to theaters and the disc authoring house.
Even for 48fps 3D, it would require less than 2TB/hour of uncompressed 4:2:2 video (at 1920x1080), so although nobody is shipping a petabyte around, it's possible that the uncompressed data is being shipped around.
I can think of a couple points, both of which have nothing to do with the TV:
* 1080p 3D over HDMI is limited to 24 and 30fps, due to bandwidth limits of HDMI 1.4.
* There's no current support for anything other than 24, 30, and 60fps at 1080p over HDMI 1.4.
So you could:
* Work on a new HDMI spec to handle 48/60fps 3D content, which means folks will go through yet another upgrade cycle in order to watch this stuff.
* Telecine 2D 48fps to 60fps and play back at 1080p60, and run the 3D version at 24fps.
The TV itself isn't the problem, but rather the lack of foresight/adaptability of HDMI to date, and that this means hardware upgrades most of the time when a new spec comes out to address the lack of foresight.
The standard pirate sizes ate 700MB for a recompressed DVD-rip and 4.4GB for a recompressed blu-ray rip. These sizes are used because they are just small enough to fit onto a CD-R and DVD-R respectively. It's not universal though. Especially long or difficult films might go up to 8GB in HD, and there has been a recent trend towards smaller files where quality wouldn't be compromised rather than just assuming media-size for everything.
Blu-ray players don't have the hardware to decode it though, and HDMI doesn't support it either. The screen does, but that's all. You *might* be able to get away with a firmware flash, but for most people it'll mean buying new hardware. Again.
the longer movies that are coming out. I was getting really fed up of more and more films creeping under the 90 minute mark. For kids films I understand keeping them shorter as most kids won't stay interested for the duration of a 2 to 3 hour film. As an adult though, I appreciate the extra character development and depth that can be provided in a longer film (not that I would ever use Avatar as an example of a film where the extended time was well used).
Even for 48fps 3D, it would require less than 2TB/hour of uncompressed 4:2:2 video (at 1920x1080), so although nobody is shipping a petabyte around, it's possible that the uncompressed data is being shipped around.
Except that they're probably storing it with some kind of RGBA (32-bit) uncompressed standard, which brings you to ~2.6TB/hour. And then if you decide to shoot it in 4K (4096x2160) that brings you to 11.2TB/hour or a bit over 30TB for the raw version of The Hobbit (48fps, 3-D). Now add in all of the rough cuts, editing revisions, unused footage, CGI, and everything else and you could see it *very* easily getting up over a petabyte. That's just for the studio though. What goes out the door, even in its rawest form, wouldn't get anywhere close to that.
As an aside, even ridiculously oversampled audio, running at 192k, 96-bit, 8 channels, and uncompressed is only going to run you ~62GB / hour.
The Hobbit is projected at 192 images/second. Each frame is shown four times, left, right, left right.
You ship around 4K, not 1080p. Theatre projectors play 4K directly, you work on the movie internally (SFX, editing) at least at 4K, and I'm not sure what the disc authoring guys use but they might use the 4K source directly too. 1080p is just what the customers get.
Lutheran churches in the US seem to operate as much on coffee as they do on faith. However, there are great differences depending on the nationality of those who started the congregation. Go to a German Lutheran church and the coffee is a little on the weak side. Go to a Norwegian Lutheran church and your eyes will pop open after the first sip.
The Hobbit, scarcely 300 paper pages of children's story in print, leads to a big data problem. Now there's excessive bloat if I ever saw any.
Learn from the mistakes of others. There isn't enough time to make them all yourself.
Private trackers that I have access to offer full-frame blu-rays which are between 15 and 45 GB in size (or close to that). Using metropolitan P2P connections, I can download those monsters in 1-2 hours, depending on size. Movies which I download and manage to watch entirely deserve me buying a cinema ticket which I don't use. I buy the cinema ticket online, I don't go, everyone's happy. Rare are the movies I watch more than once, and those I usually buy as a hard-copy anyway.
...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
Have they already gone blue from waiting?
Of course movies are longer! Movies began with double features and a cartoon and newsreel. They did away with double features and cartoons (sniff) when exhibitors wanted to churn the theater every 90 minutes like they were flipping burgers. When we spend a quarter of a hundred dollars, (sounds bigger that way) and then pay the same amount on drinks and snacks, we really want more than 90 minutes minus the 8 minutes of credits.
Longer films actually provide more product, better story-telling opportunities (not always used well), and a better value for our time and money. The Hobbit seemed too short for real Tolkien fans. Avatar's length was appropriate for the story. Maybe we all want better stories? If all one wants of a movie is to pass the time in an airliner, one really is not interested in the film. Perhaps one only wants a TV game show. IMHO.
I just decided to settle for the compressed version. Much smaller file size and I don't think it degraded the quality appreciably.
Knowledge Brings Fear
Sorry, the torrent I downloaded was a better version than yours. I'm not sure about compression differences, but my version has all of the extras included.
I speak England very best
I start rolling my eyes when I start fidgeting in the cheap ass chair cause my ass is going to sleep and I am tired of holding a bag of popcorn. Sure its a three hour movie, but you get there a little early cause of lines and you dont want to be that douche that walks in during the opening scene bumping everyone cause they want the middle seats for them and their 4 buddies, then you sit there for 20 min worth of previews.
So that 3 hour movie usually turns into 3:45, and you all know that most of that extra hour is just because they feel the need to drag along on less important stuff as if it were the main focus.
In the great white Yukon of Canada, downloading it would cost you $7,864,320.00 in bandwidth charges.
Yeah. I have to pay $7.50/gig.
may be to convince the studio to keep the third installment, given that their best case audience is pretty much limited to those who saw and were happy with the first two installments. The dragon will presumably be slayed in the second; will there be enough to bring the non-died in the wool LOTR buffs back for the third?
Please refer to http://en.wikipedia.org/wiki/Humor...
The most interesting aspect about 48 FPS is that the Bluray spec doesn't cover 1080p48, so we'd need a new standard and new devices for home cinema use if 48 FPS will take off (as it should).
I don't think it was meant to be a joke. If it were just the first sentence, it could be. But the second sentence makes an accusation.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
That audio stream only represents a single track though, a movie like this is going to have multiple tracks (music, effects, actor A dialog, actor A redubs, actor B...).
I read the internet for the articles.