Slashdot Mirror
Linux at Industrial Light and Magic
Nicholas DePetrillo writes "Linux Journal has a big story about how LINUX is being used in hollywood, specificly at Industrial Light and Magic with some GREAT screenshots
and a very descriptive article with configuration details." Word has it that their rendering farms have gotten even bigger since this article was published.
GFX: Industrial Light & Magic
Posted on Monday, July 01, 2002 by Robin Rowe
Discussing the move to Linux on ILM's renderfarm, with speed and stability comes responsibility.
Star Wars, Episode II: Attack of the Clones, released in May 2002, is Industrial Light & Magic's (ILM) first movie produced after converting its workstations and renderfarm to Linux last year. Located north of San Francisco in San Rafael, California, ILM was founded in 1975 to produce the visual effects for Star Wars. Although owned by George Lucas, ILM produces visual effects for more than Lucasfilm productions, such as the Star Wars and Indiana Jones films. Many other studios seeking that bit of something extra on the cutting edge of special effects use ILM. ILM has received 14 Academy Awards, including ones for its work on Forrest Gump, Jurassic Park, Terminator 2, Who Framed Roger Rabbit? and E.T.
-PHOTO OMITTED-
``Linux is increasing the quality of our work, not the quantity'', says Andy Hendrickson, director of research and development. Large amounts of processing power enable more user control. He explains,
We often go into a show knowing what we want but are forced to scale back realism with shortcuts because of a lack of processing power. Using Linux we can add more realism. We direct effects. It isn't enough to have a cloud that is an NOAA-accurate model. Artistic staff directs the effects with, ``Make that cloud more fluffy''. Or, if we simulate an entire ocean, as in Perfect Storm, ``Make that wave larger''.
ILM made a bold move to undertake their Linux conversion in the midst of a major movie production, switching while work was underway on Episode II. ``We thought converting to Linux would be a lot harder than it was'', says Hendrickson. ``Linux is so like what we had before. We pushed forward deployment in November 2001 and will finish conversion after Episode II.'' During the changeover, ILM is supporting existing SGI IRIX machines and Linux PCs to avoid overwhelming users with too much change.
Sequence supervisor Robert Weaver is a technical director on Episode II. Weaver's desk has a Linux PC on the left side and an SGI O2 on the right. Because the Linux desktop is configured to look like the SGI O2, it isn't immediately apparent which screen is which, until Weaver demonstrates the difference in speed. He says,
The old system is so slow that the clones firing lasers appear to be throwing javelins. We've seen about a five times speed improvement with Linux, which is appreciated! I'd say Linux is one of the most successful efforts we've had. I can't say enough good things about it. It is intuitive, incredibly stable, and we can get stuff fixed at a moment's notice. Our renderer hasn't been ported yet but will be by the summer. That's all that's keeping me on the O2.
All ILM 3-D particle simulations are done in Alias|Wavefront Maya. ``We have, I'd say, 90% of our Maya users on Linux'', says Weaver. ``It seems incredibly stable on Linux. I haven't had Maya crash on me in months. I'm evaluating that the correct cycles have been put in. I do that in wireframe mode.'' To extend the functionality of Maya, Weaver writes plugins. ``Maya makes writing plugins fairly easy. I add stuff to the shelf.'' The shelf is a set of plugin tabs visible across the top in Maya. The ocean in Perfect Storm is an example of the effects ILM achieves with Maya plugins.
``Our compositing software, CompTime, has been ported to Linux'', notes Weaver. ILM created its own compositor with a plugin architecture for doing motion picture editing rather than choosing a commercial package. Weaver writes compositor plugins, too. ``The compositor plugins are in Python'', he notes. ``We're a big Python shop...and MEL.'' MEL is the Maya scripting language.
Maya is considered by ILM a tool best for TDs (technical directors); animators at ILM use SOFTIMAGE. The conversion to Linux triggered a company-wide upgrade from version 3.8 of SOFTIMAGE (on IRIX) to the 4.0 version that recently became available for Linux.
In the years since the first Star Wars trilogy, animation software has become capable of greater facial expression. ILM created their own caricature facial animation application that reads and writes SOFTIMAGE scenes directly, not as a plugin. Senior Digital Model Supervisor Geoff Campbell used this software to set up facial expressions for animation in Episode II.
-PHOTO OMITTED-
``There are 11 muscles in the face that are key to giving a performance'', says Campbell. ``I can stretch a face in SOFTIMAGE as much as I want. When I like what I've done, I'll save it as a new shape. At my desk I have a little camera and a mirror I use to view my real facial expressions. You invest a little bit of yourself in each character.'' Campbell says an important detail in a character's performance is ``eye darts'', the little telling looks that performers give each other when interacting. In Episode II Yoda had eye darts even with his eyes closed. ``Linda Bell developed an animation of the eyes while sleeping, that is, the eyes moving in REM sleep under closed eyelids.''
``I wanted Yoda to look better than the puppet, to have the lip movements better match the words'', says Campbell. She explains that
The interesting thing with Yoda is George didn't want us to go light years ahead of the puppet's limitations. When Yoda is speaking in Episode II, and he has a lot of lines, he uses phonemes in a very simplistic way. A lot of the shapes we are trying to mimic are the puppet shapes that Frank Oz created moving his hand in rubber.
The later CG Yoda matches the character in other movies in the series, but he has more exact lip phonemes, with the lips curling to make an ``M'' or ``B'' sound, than the puppet could create.
-PHOTO OMITTED-
The hair on Yoda is another character feature manipulated with ILM's facial animation software. Because moving individual hairs would be too cumbersome, there are single control hairs that influence the hairs around them. To style Yoda's hair interactively, speed is important. When running ILM's facial animation software on the SGI O2, it took seconds to redraw the screen after each change, and the delay made work difficult. ``With Linux we manipulate high-res models in real time in a way we couldn't with our SGI system'', says Campbell.
ILM still builds some physical models but mostly for backgrounds or for organic-looking things that can be created easier than with CG. Although ILM doesn't construct many spaceship models anymore, their computerized motion control cameras are still shooting background plates nonstop.
R&D Principal Engineer Phil Peterson reports that ILM is about 80% finished with its Linux software conversion. He says, ``A team of three people ported over a million lines of code to Linux.''
``The biggest issue we had in porting was the compiler and other tools'', says Peterson. ``Newer C++ code is fairly dependent on STL.'' The gcc 2.96 compiler included with Red Hat didn't support the C++ Standard Template Library (STL), so ILM uses gcc 3.01 instead. Their multiplatform build environment is customized based on Python cooperating with GNU make.
ILM had to accommodate some CPU differences, such as floating-point implementation and number precision. ``In some cases we hand-optimized in-line assembly to get the most out of Linux'', says Peterson. One issue is how to track memory access per thread in Linux, which handles thread IDs differently from IRIX. Another annoyance is that a floating-point exception isn't allowed to throw a C++ exception (because FP exceptions are asynchronous).
In integrating legacy Motif applications, ILM had to overcome some issues with widget differences. ``We were using Motif mostly'', says Peterson, ``but use FLTK in our latest applications''. ILM made their SGI-based apps look similar on Linux, including the fonts and colors. ILM software projects may incorporate 80 or 90 libraries. For sound ILM uses OSS, but Peterson says they may switch to ALSA. SGI provides the dMedia libraries, but on Linux ILM had to create some of their own media libraries to fill in missing functionality. To play back movies, which at 2k by 1k are more than 27 times larger than typical 320 × 240 PC video, ILM created their own QuickTime-compatible library used in their flipbook player.
-PHOTO OMITTED-
``With Linux the increase in speed is what everyone is noticing'', Hendrickson says. He says the speed increase is not just 20% or 30%, but five times faster. We have an incredible appetite here for computation. It is the limiting factor. If an artist can get more iterations, the result will be that much better. We have lots of Jedi knights, lots of robes to model. Without the speeds coming out of hardware, we would not be able to do it. Before we had to hand-animate approximations. Episode II uses digital stunt double work, blending seamlessly between the live and the CGI performer. We are using CGI to replace rubber prosthetics on actors. We're doing more simulation, more rigid body dynamics. As droids get sliced and diced with light sabers, pieces fly off in realistic ways.
George Lucas, who used 400 shots in the original Star Wars, used 2,000 in Episode II. Creating that required three visual effects supervisors, as if doing three shows. ``Expect a jump in what we're able to do after Episode II'', says Hendrickson. Thanks, in part, to Linux.
The Computers of ILM
Resources
Robin Rowe (Robin.Rowe@MovieEditor.com) is a partner in the motion picture technology company MovieEditor.com. He has led video R&D at a Fortune 500 IT company, taught C++ at two universities and was an NBC-TV technical director. He leads two users' groups: LinuxMovies.org and OpenSourceProgrammers.org.
ILM says they have rarely seen artists get excited by hardware, but artists fought to get the new Linux workstations--Dell single-CPU P4s with NVIDIA Quadra 2 Pro graphics cards. The question became, ``Where's my Linux box?''
Production Engineering Manager Ken Beyer says
``We've changed over quite a bit of our plant here to Linux--half of our desktops and about 30% of our 2,000 CPU renderfarm is now Linux'', says ILM Director of Research and Development Andy Hendrickson. ``We've got 700-plus O2 machines'', adds Beyer. ``But it isn't affordable to replace those with Octanes.'' SGI is recognized for producing high-end workstations and servers but has abandoned competing with commodity PC hardware. SGI seems to be rebounding in the military market but less so in entertainment.``Our renderfarm towers carry the Deathstar logo'', points out Beyer. A render tower is a stack of 1RU 2-CPU units connected together with inexpensive 100Base-TX. He says:
An unexpected snag arose during the upgrade: all the PC fans had to be replaced because they were defective.Systems R&D Group Manager Mike Kiernan reports a few problems with Linux:
But he adds that ``Linux needs work on NFS big time.'' ILM is comfortable with multiple platforms. Its 1,400 employees use a variety of operating systems. The art department has Macs, with the rotoscopers and painters transitioning to OS X. Hendrickson sees OS X as a possible player. ``What attracts us is the BSD-like Darwin core and network compatibility.'' ILM has few Windows boxes, besides those on business side. ``There's no advantage to a Windows conversion for us'', says Hendrickson. ``We're a UNIX shop and probably always will be.''R&D Principal Engineer Phil Peterson says ILM chose the Red Hat distro because it seemed easier to go with what's popular. ``At ILM the 2.4.9 kernel is deployed, and 2.4.17 or 2.4.18 is in test. We tweak the kernel--things like shared memory size, number of file descriptors, default stack size--nothing dramatic.'' Open Motif 2.1 did a good job maintaining the look-and-feel of IRIX, so ILM didn't try LessTif. ILM workstations include limited installations of GNOME and KDE. ``No special effort was spent to strip machines down'', says Peterson. ``We just left out unused portions of the full install. We're pretty vanilla.''
An unusual aspect of the ILM Linux workstation configuration is the replacement of the MESA libs with the SGI open source, OpenGL implementation. ``MESA is behind compared to the SGI version in aspects such as libGLU'', explains Peterson. Other studios haven't experienced the best stability using Maya on Linux with NVIDIA drivers. It seems that may be due to MESA and not Maya, NVIDIA or Linux, as previously thought. ILM has replaced the MESA libraries with a combination of NVIDIA's core OpenGL and libraries from the SGI open-source sample implementation.
``Chances are you will not find solutions in any documentation'', notes Peterson.
Hendrickson concurs that Linux support can be a problem. He says, ``As we get into Linux we're not finding one company to hand-hold. IBM and HP aren't there, yet. But, before Linux it was out of our control and out of control. [Now] we own our Linux problems.''Is it possible for Linux to be too fast? ``Due to the speed of Linux, for the first time in my life, 15 years in the business, I'm starting to feel some RSI [repetitive strain injury]'', says Technical Director Robert Weaver. ``Usually you are working the machine, but Linux is so fast it can overwork you.'' Weaver has to remember to take breaks because with Linux he doesn't get any breaks waiting for the machine anymore.
The Computers of ILM
ILM says they have rarely seen artists get excited by hardware, but artists fought to get the new Linux workstations--Dell single-CPU P4s with NVIDIA Quadra 2 Pro graphics cards. The question became, ``Where's my Linux box?''
Production Engineering Manager Ken Beyer says
More than 350 Linux boxes were deployed during Episode II. Animators and modelers got their workstations first, then compositors. The first group had flat panels because animators lack the desk space for monitors. There were problems with monitor calibration under Red Hat 7.1. We used flat panels to get Linux out there. Last to get workstations were TDs. They push the envelope of what they ask for. An issue was how quickly we could get things ported for them.
``We've changed over quite a bit of our plant here to Linux--half of our desktops and about 30% of our 2,000 CPU renderfarm is now Linux'', says ILM Director of Research and Development Andy Hendrickson. ``We've got 700-plus O2 machines'', adds Beyer. ``But it isn't affordable to replace those with Octanes.'' SGI is recognized for producing high-end workstations and servers but has abandoned competing with commodity PC hardware. SGI seems to be rebounding in the military market but less so in entertainment.
``Our renderfarm towers carry the Deathstar logo'', points out Beyer. A render tower is a stack of 1RU 2-CPU units connected together with inexpensive 100Base-TX. He says:
These are 1RU, 2-CPU P4 units. If we lose a unit it is more convenient now that it is just two CPUs rather than four or eight with SGI 2800. For Episode II we had to double available capacity and power. It's 512 processors. We use dual 225 kVA UPS systems, and have three AC systems that rotate. Power goes out often in the San Rafael area. We can run on UPS for 15 minutes then [on a] diesel generator.
An unexpected snag arose during the upgrade: all the PC fans had to be replaced because they were defective.
Systems R&D Group Manager Mike Kiernan reports a few problems with Linux:
Sometimes when I arrive in the morning a quarter of the Linux cluster is locked up. Fortunately, it doesn't happen too often. VM problems in the 2.4 kernel appear to be at the root of our kernel lockup problem. Recent improvements in the 2.4 kernel may resolve that. Things look promising.
But he adds that ``Linux needs work on NFS big time.''
We won't be going to Linux for our NFS servers. I wish we could replace NFS, but none of the document management systems is flexible enough. And the ones that are flexible have a rather high integration cost. When AFS is distributed natively for all the client platforms we need to support, perhaps we'll consider it.
ILM is comfortable with multiple platforms. Its 1,400 employees use a variety of operating systems. The art department has Macs, with the rotoscopers and painters transitioning to OS X. Hendrickson sees OS X as a possible player. ``What attracts us is the BSD-like Darwin core and network compatibility.'' ILM has few Windows boxes, besides those on business side. ``There's no advantage to a Windows conversion for us'', says Hendrickson. ``We're a UNIX shop and probably always will be.''
R&D Principal Engineer Phil Peterson says ILM chose the Red Hat distro because it seemed easier to go with what's popular. ``At ILM the 2.4.9 kernel is deployed, and 2.4.17 or 2.4.18 is in test. We tweak the kernel--things like shared memory size, number of file descriptors, default stack size--nothing dramatic.'' Open Motif 2.1 did a good job maintaining the look-and-feel of IRIX, so ILM didn't try LessTif. ILM workstations include limited installations of GNOME and KDE. ``No special effort was spent to strip machines down'', says Peterson. ``We just left out unused portions of the full install. We're pretty vanilla.''
An unusual aspect of the ILM Linux workstation configuration is the replacement of the MESA libs with the SGI open source, OpenGL implementation. ``MESA is behind compared to the SGI version in aspects such as libGLU'', explains Peterson. Other studios haven't experienced the best stability using Maya on Linux with NVIDIA drivers. It seems that may be due to MESA and not Maya, NVIDIA or Linux, as previously thought. ILM has replaced the MESA libraries with a combination of NVIDIA's core OpenGL and libraries from the SGI open-source sample implementation.
``Chances are you will not find solutions in any documentation'', notes Peterson.
We don't have a support line to call. We fix things and extend. It introduces a layer of maintenance we're not used to. We had to use open-source drivers with tablets. With calibrating monitors, the work is ongoing. Still, we've had an easy road. Our artists are technically savvy, able to endure pain. Having the best testers in the world around the corner from you provides quick feedback.
Hendrickson concurs that Linux support can be a problem. He says, ``As we get into Linux we're not finding one company to hand-hold. IBM and HP aren't there, yet. But, before Linux it was out of our control and out of control. [Now] we own our Linux problems.''
Is it possible for Linux to be too fast? ``Due to the speed of Linux, for the first time in my life, 15 years in the business, I'm starting to feel some RSI [repetitive strain injury]'', says Technical Director Robert Weaver. ``Usually you are working the machine, but Linux is so fast it can overwork you.'' Weaver has to remember to take breaks because with Linux he doesn't get any breaks waiting for the machine anymore.
....and fairness to call it GNU/Star Wars then. Or there might be some dirty deal going on, like we get to see a Master Stallmann in GNU/Episode3. Who knows...
I feel so sig.