'Hobbit' Creates Big Data Challenge

← Back to Stories (view on slashdot.org)

'Hobbit' Creates Big Data Challenge

Posted by timothy on Thursday January 3, 2013 @05:15AM from the such-a-small-creature-makes-such-a-big-difference dept.

CowboyRobot writes "In the past five years there has been an 8x increase in the amount of content being generated per every two-hour cinematic piece. Although 3D is not new, modern 3D technologies add from 100% to 200% more data per frame. In 2009, Avatar was one of the first movies to generate about a petabyte of information. The Hobbit: An Unexpected Journey was shot in a new digital format called High Frame Rate 3-D, which displays the movie at 48 frames per second, twice the standard 24-fps rate that's been in place for more than 80 years." But with digital storage transcending some other limitations of conventional projection techniques, it's not just framerate that directors are now able to play with more easily; it's the length of movies themselves, which stats suggest just keep getting longer.

24 of 245 comments (clear)

Min score:

Reason:

Sort:

How big was the hobbit? by iONiUM · 2013-01-03 05:19 · Score: 5, Interesting

I read TFA, and nowhere does it say how big The Hobbit was.. only that Avatar was about a Petabyte. Why isn't this stated anywhere? It's very frustrating, and also makes the article less useful, since its entire premise is that "The Hobbit creates big data challenge" with no specificity.
1. Re:How big was the hobbit? by Anonymous Coward · 2013-01-03 05:30 · Score: 5, Funny
  
  According to the torrent sites; 2.32 GB though they do use the lossy video camera conversion...
2. Re:How big was the hobbit? by bughunter · 2013-01-03 06:43 · Score: 3, Informative
  
  Well, I can't find an official number, but we can estimate using data from here and here:
  From the first link, which says the max data rate is 250Mbps, and doubling that to account for HFR, we have a 500Mbps data rate. Multiply that by the 169 minute running time and you get
  500e6 bit/sec x 1/8,589,934,592 GB/bit x 169 min x 3600 sec/min = 35,400 GB
  (assuming the limit on precision is the running time at three significant digits).
  Divide that by 1024 GB/TB and you have about 34.6 TB. Not impossible to set up, and probably far less expensive than the projector... but that's for the non-IMAX version, which probably explains why I could only find three theaters with the HFR IMAX version near my house in Pasadena CA.
  I also expect that some theaters will not operate at the maximum data rate but use some other, more lossy compression. It's probably safe to assume a lot of theaters are showing distributed versions that are about 10 or 20 TB large.
  
  --
  I can see the fnords!
3. Re:How big was the hobbit? by mill3d · 2013-01-03 06:56 · Score: 4, Informative
  
  Actually, it's all the data required as input to make the *final* frames. We're talking many layers of video at 32 bit per channel (128bit images), VFX cache data which can be GBs per second of footage, thousands of textures that are also GBs in size, point clouds... All of that is meant to retain a maximum amount of flexibility before finalizing the footage. Read up on the REYES pipeline for detailed info.
  Disclaimer: Film and animation professional and professor.
  
  --
  Nothing is enough for whom enough is too little - Confucius
4. Re:How big was the hobbit? by log0n · 2013-01-03 11:46 · Score: 3
  
  Having the additional resolution helps preserve quality during the making process. For example, most music is recorded at least at 24bit 48-96khz per track (dozens to hundreds depending on the music), even tho the destination will usually be 16bit 44-48khz. The extra fidelity makes all the eventual filtering, dithering, resampling, correction, compression, etc ultimately that much higher in quality.
  Disclaimer: worked in TV and music production.
Comment on Movie length by Dartz-IRL · 2013-01-03 05:19 · Score: 4, Interesting

Am I the only one who longs for the return of an intermission? If only for a little relief rather than ducking out for 3-4 minutes and missing that one important little line of dialogue on which the whole thing pins?

--
So there I was, scribbling down some notes off the PC screen by hand, when I reached for the keyboard and Ctrl-S'd.
1. Re:Comment on Movie length by interkin3tic · 2013-01-03 05:27 · Score: 5, Funny
  
  Until they do bring it back, there is an app for that
2. Re:Comment on Movie length by cervesaebraciator · 2013-01-03 05:28 · Score: 3, Insightful
  
  This would be helpful. Statistics may show that movies are getting longer, but my experience shows also that minute-for-minute they feel longer. At least they do when they're something like Michael Bay movies with their interminably long CGI-gasms (I mention Michael Bay, but most directors seem to be doing action sequences in his style; as much as I like the Hobbit, the best comment I saw about it was [paraphrased] 'I kept waiting for Peter Jackson to put down his X-Box controller and get on with the movie). An intermission would give me just enough time to think seriously about the horrible decision I've made and how hours of my life would be better spent by going home for a beer and a book.
3. Re:Comment on Movie length by vurian · 2013-01-03 06:21 · Score: 3, Funny
  
  I had a free refill, once, in Europe. In Linkoping airport, where I was allowed to fill my coffee cup again from the can. I was so surprised... Never happened anywhere else. And given the quality of the coffee, I didn't bother in Linkoping airport either.
4. Re:Comment on Movie length by Immerman · 2013-01-03 07:12 · Score: 3, Funny
  
  Yeah, but you wouldn't believe the compression algorithms necessary to let a phone replace the plastic bag..
  
  --
  --- Most topics have many sides worth arguing, allow me to take one opposite you.
5. Re:Comment on Movie length by afidel · 2013-01-03 08:09 · Score: 3, Insightful
  
  People who use phones in the theater should be beaten with rubber hoses and then forced to watch Barney.
  
  --
  There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Bring back the intermission. by Naatach · 2013-01-03 05:20 · Score: 5, Insightful

It was so classy. I'm sure it would help with the theater owners concession sales as well.

--
There may be no "I" in team, but there's also no "F" in way.
Re:Smart play by the studios by hpacheco · 2013-01-03 05:22 · Score: 4, Funny

:poor guys that pirated avatar.. they had to download over a petabyte:
Re:Smart play by the studios by MightyYar · 2013-01-03 05:24 · Score: 3, Informative

Huh? Most of the stuff you find on the various pirate channels is compressed down - at the most, you'll get a raw Blu-Ray rip. You can still get an xvid (avi usually under 2GB) version of just about anything that is available in any other format.

--
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
Re:Smart play by the studios by jedidiah · 2013-01-03 05:24 · Score: 3, Interesting

That's only relevant for the first copy.
Beyond that, content will be heavily re-compressed. It doesn't matter if it's Apple pushing the bits or The Pirate Bay.

--
A Pirate and a Puritan look the same on a balance sheet.
Re:Smart play by the studios by jandrese · 2013-01-03 05:28 · Score: 4, Informative

The Petabyte figure is almost certainly for all of the working copies of the movies while it was being produced. Nobody is sending a Petabyte to every theater in the country, and much less to every home. Once the movie is finished a final copy is compressed and sent to theaters and the disc authoring house. The disc authors have to further compress the image to make it fit on the Blu-Ray or DVD. Your average pirate is going to compress the movie even further because full Blu-Ray rips are still rather unwieldy for most broadband connections and personal storage solutions.

--

I read the internet for the articles.
it's the length of movies themselves by Spy+Handler · 2013-01-03 05:31 · Score: 3, Interesting

which stats suggest just keep getting longer"
And in the Hobbit's case, longer, and longer, and...... just waay too long. LOTR movies had 1000 pages of book to fill them with interesting content. Hobbit, not so much. In many of the scenes you can almost feel the director guy just out of camera view making that "stretch" motion with his hands.
1. Re:it's the length of movies themselves by Sir_Eptishous · 2013-01-03 05:57 · Score: 3, Informative
  
  This.
  
  How many times do we need to see Goblins getting knocked off wooden plank bridges by dwarves with a pole?
  Not enough it seems.
  
  --
  We play the game with the bravery of being out of range
2. Re:it's the length of movies themselves by afgam28 · 2013-01-03 08:30 · Score: 3, Funny
  
  Granted, the special effects were well done, but "a decent story"? All I remember was this:
  WARNING: SPOILER ALERT!!!
  Some dwarves, a hobbit and a wizard go on an "adventure". They get into trouble, and then the wizard saves them. Then they get into trouble again, and the wizard saves them again. Then they get into trouble again, and the wizard summons some really big birds to save them again.
  I still don't understand why they didn't just take the birds from the start, and all the way to the end. It would've saved a lot of trouble, not to mention hard disk space.
Wikipedia to the rescue by Anonymous Coward · 2013-01-03 05:31 · Score: 5, Funny

Tolkien writes that Hobbits are between two and four feet (0.61–1.22 m) tall, the average height being three feet six inches (1.07 m).
Re:Smart play by the studios by Anonymous Coward · 2013-01-03 06:08 · Score: 5, Funny

I know people who are still downloading it, you insensitive clod!
Not a Big Data Problem by ranton · 2013-01-03 06:41 · Score: 5, Insightful

How does this have anything to do with Big Data? Storing large amount of data isn't the important part, it is being able to analyze that data. You do not analyze a movie's data file. You just load and display the movie, which can easy be stored in one large continous file. A Big Data problem would be Netflix trying to determine what kinds of movies to recommend, not storing and then displaying a long movie to users.

--
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
Re:Battlefield Earth by 7bit · 2013-01-03 07:34 · Score: 3, Insightful

I mean they did the 1000 page Battlefield Earth in under 2 hours in film and that turned out great...
"Battlefield Earth" + "turned out great" in the same sentence?... Travolta, is that you?
Re:Smart play by the studios by SuricouRaven · 2013-01-03 09:04 · Score: 4, Informative

The standard pirate sizes ate 700MB for a recompressed DVD-rip and 4.4GB for a recompressed blu-ray rip. These sizes are used because they are just small enough to fit onto a CD-R and DVD-R respectively. It's not universal though. Especially long or difficult films might go up to 8GB in HD, and there has been a recent trend towards smaller files where quality wouldn't be compromised rather than just assuming media-size for everything.