Weta Prepares to Render LOTR: ROTK

Dee Arsmith writes "Peter Jackson's special-effects company Weta Digital has just taken delivery of 588 IBM blade servers, each with two 2.8 gigahertz Intel Xeon processors. Seven racks of IBM blade servers have been added to Weta's existing 15-rack server cluster to make up the largest Intel-based high- performance computer site in the world with more than 2000 linked processors. The cluster will be used to render the frames drawn by the animators to complete the final installment of The Lord of the Rings trilogy, The Return of the King."

Comments

Really the top?

[cly]

If you look at top500.org, you see that the current top Intel-based cluster is #5, the one with 2304 procs in LLNL.

The article says their cluster has 'more than 2000 processors'. So presumably they mean 'more than 2304'?

Re:Really the top?

[Copid]

Indeed. The LLNL clusters use Quadrics interconnects. They're phenomenally fast, and they're what really separate loose clusters and useful general purpose supercomputers. However, Weta doesn't need too much cross talk between the nodes (like a physicist doing fluid dynamics calculations might). Rendering separate frames is basically a perfectly parallel operation. Send some geometry data and a single machine can render the frame without needing to send anything to any other nodes. Big advantage.

There are a lot of big machines out there that are loosely connected like this one is. I suspect that's why they don't end up listed on the Top 500 site. They're not nearly as useful for the types of calculations done by most of the scientific computing sites out there as a really expensive cluster with a bitchin' interconnect.

What does this "massive" cluster run?

[Mostly a lurker]

Regelous created Massive, the special-effects program behind the colossal battles in The Lord of the Rings film trilogy.
See for instance http://www.wired.com/news/digiwood/0,1412,56778,00 .html

Q: What platforms does Massive run on? A: Massive runs under Linux and Irix.
Many interesting details at http://www.massivesoftware.com/

Re:How fast is fast?

[Soko]

ok, my "render math" isn't the greatest, but I can NOT imagine that the system he had before was THAT bad? What do you really gain by adding that much MORE horsepower? Is that the difference between a frame being rendered in 45sec vs 50sec? I understand that every little bit counts, but a LOT of these movies was done live action. Unless that little Gollum thing is in every scene, why does he need more? (ok, I know, I always want faster, better too....I'm just saying)

Well, when you're talking about a 2.5 minute CGI shot, you have 24 frames/second (minimum) X 60 seconds/minute X 2.5 minutes = 3600 frames to render. 3600 frames X 5 minutes/frame savings = 18000 minutes or 300 hours in total saved by reducing a frame render from 50 minutes to 45.

That's just in 2.5 minutes of on-screen CGI, too - when the lions share of the film requires complex digital effects, the rest is easy to justify to the bean counters. In fact, I'd be suprised if they don't end up with even more horsepower by the time ROTK is in theatres - saving that much time provides big returns on investment.

Soko

Re:More the ILM?

[malducin]

Well if anything you can't fault VFX with the story. But yes each prequel had over 2000 VFX shots. You have to read the article though, last film Weta did about 800 VFX shots and for this they are doing upwards of 1200. As their technology matures (Massive, muscle dynamics, subsurface scattering) you can even throw more things at to the VFX.

It certainly is a big setup, they are adding 1,176 new processors to what they already had (which was stated in an article some time ago). Probably ILM and Imageworks have a bit more though. The article says that they have the largets Intel deployment, but places like ILM and Imageworks, besides their Intel/Linux machine still have quite bit of SGI hardware around. An article on the SGI websitye a couple years back stated ILM had an 800 CPU Origin 2000 machine, and around 500 O2s. Since then a lot of the TDs, animators and compositors have gotten Dell Linux workstations and several of them keep the 2 machines side by side (the O2 and the Dell). ILM and Pixar also recently added to their renderfarm via RackSaver:

Pixar switches from Sun to Intel
Racksaver testimonials
AMD debuts server processor, readies 'Barton'
SGI Powers 5 Summer films

It certainly is nice that New Line is paying for this though. I'm sure other studios are envious ;-).

Re:Question

[zokrath]

Bah, last post was apparently in HTML format and managed to make a single unreadable block of text. That will teach me not to preview...Anyway, reposted in plain text:

A long, long ways.

Computer games can run at 60+ frames per second because they are barely doing any work when compared to top of the line rendering engines.

Raytracing, dozens of texture passes, multiple realistic lightsources; and these are just for a two dimensional surface. Making realistic looking skin requires multiple translucent layers to simulate the complicated appearance of skin.

Also, there is the size factor. Video games generally run at 1024x768 to 1600x1200. Movie quality shots are rendered at many times that resolution, which greatly increases the number of pixels that have to be rendered. Gollum may only be 800 pixels tall on your monitor, but he's probably rendered at least ten times as large; we'll say 10,000x 10,000 for calcualtion simplification.

That's 10E7 pixels, so to display it at 24 frames per second you would need to be pushing 24E8 pixels a second. 24,000,000,000.

Even if every pixel only took a single cycle (which it might, with the right hardware pipeline in the future), you would need 240 terahertz of power (plus overhead) to display it in real time, along with enough RAM to hold the model and texture data for everything that's going to be onscreen within the next minute or so.

Considering that they have around 2000 x 2.0 X 2 = 8 terahertz available to them, and it still takes ages to render each frame of the complicated battle scenes, I'd say we are going to hit the limit of Moore's law before we could reasonably get hte power to render cinematic scenes in real time. Perhaps with quantum processing we will be able to within the next 20 years or so.

Re:588???

[Tihstae]

I would just like to know why 588 computers?

IBM Blade Center that holds the blade server is 7U. Each Blade Center holds 14 blade servers. IBM's racks are 42U.

42U Rack / 7U Blade Center = 6 Blade Centers/rack
14 servers X 6 Blade Centers = 84 servers/rack
7 Racks X 84 servers = 588 Servers

Re:Question

[donglekey]

Also, there is the size factor. Video games generally run at 1024x768 to 1600x1200. Movie quality shots are rendered at many times that resolution, which greatly increases the number of pixels that have to be rendered. Gollum may only be 800 pixels tall on your monitor, but he's probably rendered at least ten times as large; we'll say 10,000x 10,000 for calcualtion simplification.

This is actually not true. Film resolution is around 2048x1556 and everything is rendered the size that it is needed. For the most part, the difference in rendering speed is because hardware is very fast and very efficient, and takes lots of shortcuts. There aren't many textures, they aren't very high resolution, there isn't any raytracing, there are very few lights, no global illumination, no hair rendering, no volumetric rendering, not nearly as many polygons, no particles or cloth simulations, very few deformations, and lighting calculation is done on vertexes and then interpolated instead of on every pixel (this will change with Doom 3 and Half Life 2 which is the real reason they look so much better). Renderman also subdivides everything down to one polygon per pixel to get perfectly smooth sufaces and good displacement. There is also the issue of motion blur, depth of field, and rendering of composites, which also takes a very long time. Anti-aliasing in every step is crucial for any kind of non-realtime CG, but it not as important for games, and that by itself makes a huge huge difference.

Re:Clusters for home video production?

[Dynedain]

Combustion (a compositing package similar to shake) already does this. Most 3D rendering applications support network rendering now as well. Weta is rendering to an individual file for each frame. Because of the nature of multithreaded processing and the inherent problems of multiple computers writing to the same file simultaneously, encoding to mpeg and to dvd is still time consuming, and typically only involve one machine. Even the pro-level applications (cleaner for encoding to MPEG, and Apple's DVD Studio Pro or Sonic ReelDVD for authoring DVDs) still only use one machine. Rendering 3D scenes is a completely different process than encoding video.