Slashdot Mirror
Lord of the Terabytes
Dracos writes "TheOneRing.net has an article from OnLine (a New Zealand film mag), in which Jon LaBrie, CTO of WETA Digital, discusses the hardware and software WETA is using to produce the sfx in Peter Jackson's Lord of the Rings trilogy."
When awaiting any movie, we have to remember: It's Just A Movie. I think a lot of Star Wars fans forgot that little detail, and so when Episode I sucked, they got royally peeved. They felt robbed, when they were victims of their own overexcited expectations...
I still like the Dune movie. I've read that novel two or three times, so I can fill in the parts that were skipped, ignore the ones that don't belong, and make sense of the obscure references and characters. And I can remember - It's Just A Movie. Thus, I can enjoy it - even when it comes on TV and suffers from prologues and commercials...
I can see the fnords!
Undoubtedly by the time the third film hits the screen, the computer animation "state of the art" will have progressed 1 or 2 generations beyond what was used to create the first. How can the director resist "improving" the effects as hardware becomes cheaper and better?
Even if he keeps the same hardware, another year or two of render time could make for significantly more advanced fx.
Dude, home systems with IDE drives this is not.
100 Terabytes of RAID storage, probably fibre chanel...
Not cheap, but ultra cool and fast as well
-I just work here... how am I supposed to know?
Linux has been in the mainstreem for a bit.
I know many studios have been using it for various tasks, including rendering.
I can think of several studios that are using it to render (including us).
The price performance ratio to SGI boxes is large enough that even SGI houses (such as Weta) are looking at and using Linux. Notice though, that they are still using SGI boxes, just these are of the intel flavor.
-I just work here... how am I supposed to know?
But think of all the CPU time required to repeatedly compress/decompress all that stuff!
I suspect it may cost more for the extra CPU power for the compression/decompression, than it would for the extra storage to store it all uncompressed.
Quidquid latine dictum sit, altum viditur.
He went into more detail then the artical, including the AI system they are using to make the battle scenes. They apparently have differnt behaviours for the different creatures, And they give them intelligence levels, stamina, stuff like that. Also they might fight differently for each creature they fight with, limp if they are hurt, etc.
I'm not sure if there's anything on the net about it, that can explain it better than what i can. But it seems pretty interesting stuff. Maybe if i ask them nicely, they'll let me play round with there system for a while.
I dont know so much that they had longer attention spans, but maybe that there was less to do back then. A visit to the globe theater in it's time, was basically a day long affair. It wasnt like movies are today when you can slip one in when you have a couple of hours to kill while your downloading the latest release of your favorite distro :p
mmmmmmm Shiner Bock
I guess it depends on the size of the raid system.... some of those get sort of heavy...;)
If it is like our environment, only 5-20 megs are being pushed arround at any one time... and I can almost guarantee that servers are connected with 1000bT fibre.
-I just work here... how am I supposed to know?
Bzzzzt! Sorry, wrong answer, but thanks for trying.
Although you are correct when you say 'wide lvd scsi arrays do not quite make it then'.
There is much more to enterprise class storage arrays than buzzword compliancy. While lvd scsi wipes the floor with IDE drives, FC-AL arrays wipe the floor with standard scsi.
Once you get into the multi TB range, there are issues involving disk read/write throughput and just plain the overhead with controlling that many discs.
As an example, an EMC FC4500 array has fully redundant CPUs, cache(4GB), power supplies, and controllers. If all of that fails, it has an onboard UPS which writes the contents of the disk cache to a small scsi disk so that all pending writes don't get flushed.
I'm not sure if sgi makes their own arrays, or re-badges like HP does (EMC), but a _small_ EMC fibre attached drive cabinet with ~500GB of disc costs in the range of $125,000.
In the immortal words of Socrates, who said; 'I drank what?'
100Tera..BYTES?! Seriously people. This is OVERKILL. With the digital theatres only having about 3000x2000 (thereabouts) resolution, 100TB of digital video is way overkill. Even in film-based cinematography, that would be overkill. 24 frames a second, 35mm of film per frame isn't that much. Not 100TB worth anyway.
Personally, I'm getting tired of this exponential blowup in storage capacity. 640K ought to be enough for anybody.
This is still news for nerds (I hope), and I can't think of a more popular story than Lord of the Rings (with the possible exceptions of Star Wars and Star Trek.)
-Ted
>Am I the only person who finds themselves wildly swinging back and forth between sort of fevered anticpation of this series and totally cynical distrust?
I was swinging back and forth, but now I am stuck on Cynical Distrust. I have heard about some of the changes that will be made and IMHO they are bad ideas.
- There are supposed to be a load of Elves fighting at Helm's Deep. Of course, this didn't happen.
- What's her name, Liv Tyler (Arwen?) will be traveling with the Company. Altering the fundamental makeup of the story this way is spooky. They may as well throw in a freaking Space Monkey like Lost in Space did.
The films will clearly be a different version of the story, and I am extremely skeptical that the end result will be quality. I think there are some books that really can't make the transition to film well. LOTR is one. Neuromancer is another. Sometimes the imagination is the best palette.
All that said -- I think that this will be a cool trilogy in its own way -- it will just disappoint the Tolkien purist in me. I am expecting it to be a lot like Dune. In other words, it won't be true to the story, but it will be a lot of good eye candy that fits the world the story is set in, and I will enjoy it on that level.
(Dune sucked as a film, but the portrayal of the Harkonnen, Atreides, Spacing Guild, Mentats etc. was great. OK, except the Sardukar, they screwed that up...)
Of course, those of you who actually read the article know that they only have 4TB LIVE at any one time, and they are ramping that storage up to 10. 100TB is the sum of the tape storage, which is easy to believe as they probably want to "measure twice cut once" and keep everything they create.
They couldve at least used some form of OCR. Way too slow to load. Using images to display text---great use of bandwidth. It keeps loading and loading...
Good god, I'd want a couple of Octane boxes for my house if I gave an interview like that...
-b
If I wanted a sig I would have filled in that stupid box.
"the three films being shot simultaneously"
duh.
Only the State obtains its revenue by coercion. - Murray Rothbard
I stuck a copy of the image on my server at http://house.ofdoom.com/~hunger f3/ mirror/698.jpg
--
your insinuation the Ghost in the Shell is not insanely great earns you a whipping.
It's very good, I have a copy on laserdisc. I also have all Armitage III issues, both editions of Blade Runner, and the original PKD novel, "Do Androids Dream of Electric Sheep." They're all essentially exploring the same plotline, I just think the Director's Cut of Blade Runner does it best visually and in its storytelling.
Also, I like all the different movies listed, for very different reasons. They also have parts about them that suck.
[
I remember my first encounter with Lord of The Rings was in grade 5... they played this DARK, HORRIBLE-LOOKING MOVIE on a Video Disk (some of you may remember this little techno oddity...).
I was just a kid, but I was supremely unimpressed by the animation and modeling in this movie. The colours ranged from dark brown to light brown to reddish-brown to green-brown, and the characters all looked like they were traced badly (rotoscope style).
I look forward to someone doing it right. It's taken long enough!
I recommend also getting a copy of his movie 'bad taste'. Hilarious low budget stuff.
Love the sheep. baaa. booom. Why do I think of McDonalds?
PS. Dead Alive was renamed from 'Brain Dead'.
Out Zombies all other Zombie movies. Superb.
6 hours X 60 miniutes per hour x 60 seconds per miniute X 24 frames per second = 518400 frames
3000 pixels X 2000 pixels X 3 Bytes per pixel (ignoring very possible alpha channel) = 18000000 Bytes per frame (18 MB - ignoring the 1,000,00 doesn't equal 1 MB conversion)
18000000 Bytes X 518400 Frames = 9,331,200,000,000 Bytes
Add all the extra takes, the takes done for special effects, the 3D models, the textures, the stuff done for compositing, the database to keep track of all these resources, my math errors and all the stuff I'm forgetting. 100 TB is a reasonable spec for storage.
Now of course that is uncompressed. But seriously, are you going to be compressing images that are getting processed many times? You might be able to encode the final cut, but you can't be encoding, decoding and then re-encoding stuff every time you want to process something.
Bleh!
Lords of the Terabytes
Figuring out how much data the The Lord of the Rings project would create was the starting point for Jon Labrie. "The three films are being shot simultaneously," he says, "and we're creating the effects for all three films in a single block."
"We expect to create and manage 100 Terabytes of data before this show is over. It's my job to plan for that growth and to meet the infrastructure challenges associated with it."
"There are 200 people in the Weta production facility, every one of them involved heavily in the creation of a lot of digital data. I think of it as a kind of blizzard or storm of data flying through the facility," Labrie says. "What we're about is the ability to move large amount of information around the facility all day, every day, and we rely on SGI to help us do that. Ninety percent of our equipment is SGI."
Weta uses two SGI Origin 2000s as its primary file servers. One Origin 2000 is for near-line and offline tape-based storage. The other is for central online disk storage.
"Given the 100 Terabytes, we're demanding a lot from those file servers. We currently have four Terabytes of data on hard drives, which will eventually grow to 10. For nearline / offline storage we're using DMF, SGI's hierarchial storage management system. It's greatly simplified our management of the thousands of tapes needed to store the bulk of the data."
Weta's primary rendering resource is based on SGI I200 Linux servers. "Rendering hinges on the ability to efficiently use processor cycles," Labrie says." We have 32 dedicated processors today, and expect to extend that to well over 200 by the time we're finished."
"We also have 90 Octane Irix workstations for the principal artists, and another 25 or so Linux-based workstations that we're using primarily for paint, rotoscoping, and compositing. Tese systems also contribute to the rendering pool when available."
The Linux workstations run Nothing Real's Shake, which is the primary compositing application at Weta. An eight-processor Onyx II also runs Inferno, Discrete Logic's high-speed, single seat compositing system.
"We have a large number of seats running Maya, Alias/Wavefront's modelling, rendering and animation system. Maya is the core 3D application for the facility," Labrie says.
"We also have a number of other applications for niche requirements: Houdini for effect and particle animation; 3D Equalizer and Softimage for camera match-moving. There's a sprinkling of other things, and a lot of proprietary technologies that we've been working on specifically for The Lord of the Rings.
"There are unique graphical applications that PEter Jackson has asked us to create. We've been in research and development for three years, planning and working on standalone proprietary systems or extensions to off-the-shelf applications. We're moving into the actual production of shots now."
"We've been writing custom extensions to Maya for the past two years to improve the look and performance of our computer-generated characters."
"Over the past four years we've used SGI wrokstations to custom-build a new crowd animation system, called Massive. We're using Massive for battle animation scenes with hundreds of thousands of fighting, screaming, and dying Orcs, Elves and all the other magical and fantastical creates that appear in The Lord of the Rings."
"For those sorts of graphical challenges, we prefer to work in the Irix/Unix world. The graphics available to us on the SGI platforms make our jobs easier. Of the 140 special-effects artists that will work on this project, nine out of 10 will work on SGI workstations."
"There are both classic and unique IT challenges," says Labrie. "Classic challenges are the kinds of IT issues any company would expect to face: What sort of networking technology you use, how do you manage backups and disaster recover? What kind of workstation is typical for a user? This typically breaks down to 'what's the fastest machine I can use to achieve what I need in a reasonable time for a reasonable cost'?"
"The unique challenges are things that are specific to digital visual effects for film. Every director wants to deliver a unique viewing experience, to show people things they've never seen before. We break down those expectations and requirements into specific etchnical challenges and get to work on them. The results could be as mundane as a new way of describing the internal skeleton of a creature or as fantastic as Massive."
The three movies are The Fellowship of the Ring (to be release worldwide on 14 December 2001), The Two Towers (December 2002), and The Return of the King (December 2003). The trilogy shoot should wrap up early next year.
Those Weta machines were busy crunching Seti packets. There's at least 8 of them in the top 1000 for New Zealand, a damn sight faster than any other machine down here....
--
Consultancy: If you're not part of the solution, there's money to be made in prolonging the problem
Incidentally, Branaugh's version was the last major movie to be shot in 70mm. Man, what incredible picture quality. It was like you could just get up out of your chair and walk right into the screen. It's really too bad 70mm has fallen out of favor. I'd love to see Lord of the Rings in that format!
Free Hans!
6 hours * 5 channels * 3600 seconds/hour * 44000 samples/second *4 bytes/sample=19,008,000,000 bytes. Maybe they should invest in 100.19 TB worth of disk space.
and you scratch mine.
When you are stretched over a barrel like these guys, you want to (A) have a real good relationship with your vendor and (B) make sure your project is a high profile project for your vendor.
If SGI and LotR is inextricably linked in the mind of every geek in the world, SGI will do whatever it takes to make sure that their name is not associated with disastrous cost and schedule overruns.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
Plus side: Peter Jackson is pretty cool. Dead Alive is an incredibly funny, incredibly gross movie. Heavenly Creatures was good. Some of the casting sounds good. The One Ring preview was good.
Minus Side: Is it even conceivable that they won't fsck it up? I mean, how can it compare with the depth, the atmosphere, the brilliance of what is unquestionably the greatest fantasy series of all time?
Anyone else?
-Dan
I have written a truly remarkable operating system which this sig is too small to contain.
for those who couldn't be arsed reading the article. The New Zealanders are employing revolutionary "SHEEP"(tm) technology. The sets are entirely made of sheep and gaffer tape. Support cast ... sheep and the protagonist is Bilbo Baaaaaaaggins. Check it out for yourself if you don't believe me.
Baaaaaaaa.
When a maxstor 60 gig drive is all of $200, a terabyte is less than $5000 and 6 dual ide controllers. 200 times this is indeed a megabuck, but a megabuck ain't what it used to be. I run over a terabyte of storage here in my own house, so sorry, impressed I am not.
:: Ghost in the Shell : Blade Runner
oops.
You were doing great until you took one of the best movies ever made and tried to compare it to that movie where harrison ford tries to act like a robot, but succeeds only in imitating himself. I'm not sure how it ends, i fell alseep, but hopefully mr. ford dies. with luck he dies folded up like a paper crane, but beggars can't be choosers.
I think a more accurate analogy would have been:
Cool World : Ghost in the Shell
But then that's just me, and I don't snort as much coke as george bush, so i could be wrong.
--
What happens when you outlaw guns
As a note, they are filming all three at once, not completeing them at once. Once the filming is finished, the first movie Fellowship will go into post production. A bulk of the efforts will go towards getting that movie in the can.
So in other words, technical advances in the next year plus will be able to be used in the post production of the second and third movies.
I must be the one guy who didn't like (or didn't "get"?) Foundation. Some parts of it were a bit cute (I really liked the end of the 2nd book), but overall, I just couldn't suspend the disbelief. I guess that's what happens when someone writes a book in the early 50s, and it gets read by someone in the 80s who has been exposed to that whole butterfly-causing-hurricane nonsense; I thought the whole psychohistory thing was ridiculous. The science was too "soft" IMHO.
OTOH, I gotta admit that psychohistory is a nice fantasy. I suppose that understanding the world so well as to make high-level predictions hundreds of years in advance, is one heck of a holy grail.
What's funny is I seem to recall reading essays by Asimov where he asserted that most major historical events were the result of technological advances. (e.g. Longbow in 1066?) But in Foundation, the psychohistorians couldn't predict tech advances -- but it didn't matter anyway because their predictions were based on mass psychology, not tech. It's as if Asimov didn't agree with the premise of his own novels. Maybe he just changed his mind. People with long lives can do that. :-)
---
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
4 hours?! Wow. Something sounds funny. If it's truly faithful, then does that mean that it also took 4 hours to perform at the Globe in front of Elizabethans? People must have had serious attention spans back then.
I think I would pass; Hamlet just ain't that good. I would sure love to own a piece of the action in the beer consessions stand, though.
---
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
But which will be better... Lord of the Rings or Dungeons and Dragons?
Standard analogy format reads (A : B :: a : b), or 'A' is to 'B', as 'a' is to 'b'.
My guess, just from trailers (and I'm sure people will disagree)...
[