Slashdot Mirror
Weta Prepares to Render LOTR: ROTK
Dee Arsmith writes "Peter Jackson's special-effects company Weta Digital has just taken delivery of 588 IBM blade servers, each with two 2.8 gigahertz Intel Xeon processors. Seven racks of IBM blade servers have been added to Weta's existing 15-rack server cluster to make up the largest Intel-based high- performance computer site in the world with more than 2000 linked processors. The cluster will be used to render the frames drawn by the animators to complete the final installment of The Lord of the Rings trilogy, The Return of the King."
If you look at top500.org, you see that the current top Intel-based cluster is #5, the one with 2304 procs in LLNL.
The article says their cluster has 'more than 2000 processors'. So presumably they mean 'more than 2304'?
but its not a Beowulf cluster...
Rendering and beowulf do not play nice together... its a distributed system... with queueing... much more like Sun's Grid, I am sure.
-Tim
-I just work here... how am I supposed to know?
See for instance http://www.wired.com/news/digiwood/0,1412,56778,0
Q: What platforms does Massive run on? A: Massive runs under Linux and Irix.
Many interesting details at http://www.massivesoftware.com/
ok, my "render math" isn't the greatest, but I can NOT imagine that the system he had before was THAT bad? What do you really gain by adding that much MORE horsepower? Is that the difference between a frame being rendered in 45sec vs 50sec? I understand that every little bit counts, but a LOT of these movies was done live action. Unless that little Gollum thing is in every scene, why does he need more? (ok, I know, I always want faster, better too....I'm just saying)
Well, when you're talking about a 2.5 minute CGI shot, you have 24 frames/second (minimum) X 60 seconds/minute X 2.5 minutes = 3600 frames to render. 3600 frames X 5 minutes/frame savings = 18000 minutes or 300 hours in total saved by reducing a frame render from 50 minutes to 45.
That's just in 2.5 minutes of on-screen CGI, too - when the lions share of the film requires complex digital effects, the rest is easy to justify to the bean counters. In fact, I'd be suprised if they don't end up with even more horsepower by the time ROTK is in theatres - saving that much time provides big returns on investment.
Soko
"Depression is merely anger without enthusiasm." - Anonymous
You can select using a button on the front of each blade which of the 14 blades in the BladeCenter chassis has ownership of the 'MediaTray'.
Of course this switching can also be done remotely over Ethernet using the management interface (which also provides power, reset, remote video and much, much more).
From the OS viewpoint the Floppy and CD-ROM drive are USB devices, so switching the MediaTray to another blade server actually causes a USB disconnect/connect.
If you want to see what they look like: http://ibm.com/servers/eserver/bladecenter/
Well if anything you can't fault VFX with the story. But yes each prequel had over 2000 VFX shots. You have to read the article though, last film Weta did about 800 VFX shots and for this they are doing upwards of 1200. As their technology matures (Massive, muscle dynamics, subsurface scattering) you can even throw more things at to the VFX.
It certainly is a big setup, they are adding 1,176 new processors to what they already had (which was stated in an article some time ago). Probably ILM and Imageworks have a bit more though. The article says that they have the largets Intel deployment, but places like ILM and Imageworks, besides their Intel/Linux machine still have quite bit of SGI hardware around. An article on the SGI websitye a couple years back stated ILM had an 800 CPU Origin 2000 machine, and around 500 O2s. Since then a lot of the TDs, animators and compositors have gotten Dell Linux workstations and several of them keep the 2 machines side by side (the O2 and the Dell). ILM and Pixar also recently added to their renderfarm via RackSaver:
Pixar switches from Sun to Intel
Racksaver testimonials
AMD debuts server processor, readies 'Barton'
SGI Powers 5 Summer films
It certainly is nice that New Line is paying for this though. I'm sure other studios are envious ;-).
In terms of number of processors, ASCI Red at Sandia has had > 9000 Intel pentium pro (and them pentium II Xeon) procesors since the late 1990s.
It's still # 15 on the top 500 list
Bah, last post was apparently in HTML format and managed to make a single unreadable block of text. That will teach me not to preview...Anyway, reposted in plain text:
A long, long ways.
Computer games can run at 60+ frames per second because they are barely doing any work when compared to top of the line rendering engines.
Raytracing, dozens of texture passes, multiple realistic lightsources; and these are just for a two dimensional surface. Making realistic looking skin requires multiple translucent layers to simulate the complicated appearance of skin.
Also, there is the size factor. Video games generally run at 1024x768 to 1600x1200. Movie quality shots are rendered at many times that resolution, which greatly increases the number of pixels that have to be rendered. Gollum may only be 800 pixels tall on your monitor, but he's probably rendered at least ten times as large; we'll say 10,000x 10,000 for calcualtion simplification.
That's 10E7 pixels, so to display it at 24 frames per second you would need to be pushing 24E8 pixels a second. 24,000,000,000.
Even if every pixel only took a single cycle (which it might, with the right hardware pipeline in the future), you would need 240 terahertz of power (plus overhead) to display it in real time, along with enough RAM to hold the model and texture data for everything that's going to be onscreen within the next minute or so.
Considering that they have around 2000 x 2.0 X 2 = 8 terahertz available to them, and it still takes ages to render each frame of the complicated battle scenes, I'd say we are going to hit the limit of Moore's law before we could reasonably get hte power to render cinematic scenes in real time. Perhaps with quantum processing we will be able to within the next 20 years or so.
It mightn't be in the "home video production" realm but Shake, now brought to you by your favourite fruit company, can distribute rendering tasks. iMovie can't be far behind...
When they came for the communists, I said "He's next door. Take him away. Goddam commies."
I would just like to know why 588 computers?
IBM Blade Center that holds the blade server is 7U. Each Blade Center holds 14 blade servers. IBM's racks are 42U.
42U Rack / 7U Blade Center = 6 Blade Centers/rack
14 servers X 6 Blade Centers = 84 servers/rack
7 Racks X 84 servers = 588 Servers
Also, there is the size factor. Video games generally run at 1024x768 to 1600x1200. Movie quality shots are rendered at many times that resolution, which greatly increases the number of pixels that have to be rendered. Gollum may only be 800 pixels tall on your monitor, but he's probably rendered at least ten times as large; we'll say 10,000x 10,000 for calcualtion simplification.
This is actually not true. Film resolution is around 2048x1556 and everything is rendered the size that it is needed. For the most part, the difference in rendering speed is because hardware is very fast and very efficient, and takes lots of shortcuts. There aren't many textures, they aren't very high resolution, there isn't any raytracing, there are very few lights, no global illumination, no hair rendering, no volumetric rendering, not nearly as many polygons, no particles or cloth simulations, very few deformations, and lighting calculation is done on vertexes and then interpolated instead of on every pixel (this will change with Doom 3 and Half Life 2 which is the real reason they look so much better). Renderman also subdivides everything down to one polygon per pixel to get perfectly smooth sufaces and good displacement. There is also the issue of motion blur, depth of field, and rendering of composites, which also takes a very long time. Anti-aliasing in every step is crucial for any kind of non-realtime CG, but it not as important for games, and that by itself makes a huge huge difference.
This Wiki Feeds You TV and Anime - vidwiki.org
Combustion (a compositing package similar to shake) already does this. Most 3D rendering applications support network rendering now as well. Weta is rendering to an individual file for each frame. Because of the nature of multithreaded processing and the inherent problems of multiple computers writing to the same file simultaneously, encoding to mpeg and to dvd is still time consuming, and typically only involve one machine. Even the pro-level applications (cleaner for encoding to MPEG, and Apple's DVD Studio Pro or Sonic ReelDVD for authoring DVDs) still only use one machine. Rendering 3D scenes is a completely different process than encoding video.
I'm out of my mind right now, but feel free to leave a message.....
588 blades
x 2CPUs each
== 1176 physical CPU's
x 2cpus/cpu (hyperthreading on the xenons)
== 2352 hyperthreaded cpu's
x 2.8GHz
== 6585.6GHz
~6.6THz
well... thats a just a bit of rendering power, wonder whats gona happen once they are done with them. Which also makes me wonder, what happended to that somewhat famous renderfarm for toystory? Seems whenever a movie requiring horsepower like this comes out, they just buy new equipment since the stuff used on the last movie is probably obsolete already... ohwell
Tm
Support TBI Research: http://www.raisinhope.org