Slashdot Mirror
Animation and SFX with Linux
Zurk writes "Here's an article with the inside scoop on how animation studios and special effects shops actually deploy Linux in house. Also mentions how the Linux systems are replacing SGI systems at a rapid clip and
some regular user comments on working with linux for graphics work."
Many large animation and special effects studios, including ILM, Disney, Digital Domain, and Dreamworks are all moving towards Linux as a replacement for their aging SGI machines. Only large studios cn adopt linux, because they need to have the manpower to develop their own applications, and the time to train the artists to use them. It takes a lot of resources to do this sort of thing in-house.
If decent visual effects applications were available on Linux (I do NOT mean POV-RAY or GIMP!) then perhaps the Linux user base among "digital content" professionals would increase. So far, Linux developers have shown practially zero interest in developing applications which are truly useful to artists. Where, for instance can I find a linux app that plays TIFF sequences at 24fps?
Unfortunately, Dreamworks is operating in a self contained Linux bubble. For most end users interested in visual effects or animation, Windows or Mac machines are far better alternatives. I'm not trolling, here, just pointing out a very disheartening fact. It is frustrating to see the Linux community celebrate a studio like Dreamworks, but when it comes to developing code, they just don't give a s*** about the sort of work Dreamworks is doing!
It mentioned them releasing the graphics compositing code and the wacon driver improvements they did.
It doesn't make sense for them to release their actual tools; these tools are only useful to people animating movies; such people are the competition. But it makes sense for them to release anything useful in other industries, because it is likely that it will get used and improved by someone who isn't competing with DreamWorks. Free software in business works because you only have to stay ahead of the competition, not all of the other people who might use the software.
It is also quite likely that they are not using any GPL code, so they can even sell their programs. They are using LGPL libraries plenty, I'm sure, so if they modify those for their own use they need to release those modifications.
Just because it runs on Linux does not mean it has to use the GPL, no matter how much Bill Gates wishes otherwise...
There's still a lot of algorithms that you can't easily map to hardware (in other words, stuff that's much easier to program in software only). There's also the problem about the ammount of memory on the graphics card being much lower than the system's memory. Even if you can rasterize textured triangles much faster in hardware than in software, you still want to apply lots of texture layers to your objects, you want to work with larger and much more detailed textures or in general you need more realism than what current hardware-based algorithms can deliver.
But yes, you are right, hardware accelerated render farms will be something common in the future. The massively parallel pieces of silicon that current GPUs are is something you can't ignore. I have done some work on hardware accelerated distributed rendering, but alas, the website is not up yet. For interactive applications is not as good as I had hoped (instead of increasing the speed I can increase the problem size with a little performance gain on the side) but for applications where you want your stuff rendered faster (100x or 1000x faster), it's definitely a go.
The OpenGL case is really sweet: you can use GLX PBuffers (I understand WGL supports something similar too, as will DX-something, if not the current one) to render off-screen. In practical terms that means the size of your image is limited by the size of the available memory on the card. If you don't have GLX 1.3 support (ATI can I have some docs please, I want to understand how the card performs memory management in order to write such an extension for the Radeon under XFree86) you can still use tiling and get the same result (slower maybe, but anything faster than pure software is a gain here)
Read the article, it answers precisely this question. First consider what the task at hand is: rendering animations for the silver screen, that means hardware accelerated OpenGL rendering doesn't cut it, it's highly probable that you want to do software rendering. You have shadows, volumetric effects, particle systems, whatever, the point is you want to render this at a very high resolution and with a very high level of detail. For that kind of task, hardware rendering, either SGI or PC based, doesn't cut it or doesn't make sense (why would you want to render this interactively? noone is interacting with the scene). Morale: you want faster machines with lots of memory. Compare a farm of SGIs with a farm of PCs with the same raw processing power and the same ammount of memory. If you ignore the memory bandwidth issue, PCs beat SGIs hands down (if you doubt this try to figure out what SGI is doing right now and why they are getting into the IA64 market)
Then there's the "it's Unix" factor. If you are a Unix house (and have been for a lot of time -- DreamWorks is) you don't want to forget about all your in house tools and move to Windows to profit from the better price/performance ratio of PCs. You want to keep using Unix. Linux, for practical purposes, is that. Why not other PC Unices? (Solaris or the BSDs) Linux has better support, both in terms of hardware and software. It's easier to get people skillfull in Linux than the other alternatives, and the number is increasing, which means it's probably a bit cheaper, too.
Regarding the compiler issue someone else mentioned, there are several alternatives, like the Portland Group compilers, to name one. If it is an issue, the studio can invest in this. I'm sure they have evaluated the cost/benefit of this. If they say GCC cuts it for them...
And last, the above post is an obvious troll... moderators, are you awake?
"Linux Has one of the best OpenGL implementations available, namely Mesa."
Oh please! Mese is meant to be a knock-off in-software OpenGL implementation. Sure, it may work properly, but you don't use Mesa for hardware accelerated OpenGL. Also, the nVidia drivers aren't all that great when it comes to things like "gasp" OpenGL-in-a-Window. (ok, it works, but not that well)
OpenGL in X kinda just works perfectly if you're using a real UNIX machine (yes, I've got an SGI and a Sun). On a Linux box, it's hack 'n pray. Remember, for professional work it's the "quality" of the implementation/output just as much as the speed. (and most pro-grade PeeCee 3D cards are still NT-only, it seems)
I can't believe I am responding to a troll, but anyway... It's sunday morning and I am bored...
It is extremely arrogant of software engineers to assume that the entire US industry revolves around us - because it simply does NOT.
I would be tempted to dissagree with the fact that there is no "linux jobs", because that is simply not true. RedHat may not be hiring additional engineers at this time, but they are employing plenty including many of the big names of linux kernel development. Also were I work, we've started using linux for right about all of our server work. Sysadmins now administer linux boxes, some new programmers were hired, but most are the old programmers that used to program on other platforms (VAXes, Digital Unix). What's wrong with that? Absolutely nothing.
Are you under the impression that now all the sudden that linux is here MORE positions should exist? Why is that?
nop@bandwagon:~$ ls -l
-rwxr-xr-x  1  rootroot   887712Mar2517:35 
nop@bandwagon:~$size 
text data bss dec hex filename
869332 13232 15132 897696 db2a0
That's not to say that an 800k libc on x86 isn't big. It gets even bigger on more RISCy platforms like MIPS. The Agenda people are sticking with a patched glibc-1.0.3 until they can decide how to rationally compile out features.
In my opinion (which is not so humble after a lot of embedded Linux hacking), Linux is defined as not just a kernel, but also by libc. I can live with all kinds of wacky new kernel features as long as the C library uses and hides them from me. But changes, even bug fixes, to libc can break code in all kinds of unexpected places. Remember when Netscape needed a very specific libc version in order to cope with netscape's, uh, issues?
The people who work on glibc deserve a lot more respect and visibility.
Disney Feature Animation's main concern is to get the look they want for the lowest price or highest productivity.
Let's take CAPS for example. Disney asked Pixar to write CAPS to replace their ink and paint department because traditional cel ink and paint was a huge resource drain. Traditional ink and paint is a manager's worst nightmare: a noncreative department full of people doing a boring and repetitive job, but where the cost of fixing a mistake was huge (because the entire cel (or group of cels) would have to be re-copied and re-painted). To make things worse, mistakes happened often because it the job was tricky as well as boring. For example: using more overlaid cels would often save work (less to animate, less to paint etc if only a small part of the image was moving), but painting them was harder because cels have an intrinsic colour which meant that you had to paint overlaid cels with slightly lighter colours to compensate. Painters often forgot which cel depth they were working at and made costly mistakes as a result.
CAPS meant that this entire department could be axed. Moreover, it added some more flexibility. The final sequence could be tweaked at will, say, if the director wanted that shadow a bit darker. In the end, it made what they were already doing cheaper and easier.
The bottom line is that Disney Feature Animation won't touch a computer unless it a) makes it easier to do something they're already doing without losing quality, or b) allows them to do something for artistic reasons that they otherwise would not have been able to do.
sub f{($f)=@_;print"$f(q{$f});";}f(q{sub f{($f)=@_;print"$f(q{$f});";}f});
Think for a moment how you'd implement that. It's not as amazing as you might think. (Bear in mind that I have no idea how the CAPS system specifically works, so I'm guessing here.) I'd get the CAPS system to extract the animator's drawn triangle as a separate layer, then fill in Belle's cheeks in the appropriate model colour without that triangle. Then blur the blush (introducing alpha), do some careful stuff to make sure the blush doesn't escape Belle's face, then overlay.
Remember that CAPS is more than just an inker and painter. It's a full compositing system, complete with virtual multiplane camera. Quite a nice system, actually.
Yes, Dreamworks Feature Animation is much better at this. Take a look at the town backgrounds during the chase sequence at the start of The Road to El Dorado if you want to see it done seamlessly.
sub f{($f)=@_;print"$f(q{$f});";}f(q{sub f{($f)=@_;print"$f(q{$f});";}f});
You're right. I wasn't very careful with what I said. What I meant to say is that the job of cel inking and painting, along with the high cost of error, high incidence of error and high boredom, was no longer necessary.
Obviously I did not mean to say that the CAPS software works without operators. :-) It is also true that they are called "ink and paint" even though they don't touch physical ink or physical paint.
For the record, I've never seen CAPS in operation or read any specs. I'm going by the way that similar tools (e.g. Animo) work.
sub f{($f)=@_;print"$f(q{$f});";}f(q{sub f{($f)=@_;print"$f(q{$f});";}f});
I seem to remember that some years ago SGI said that they wouldn't continue developing IRIX and that they would move away from MIPS technology.
.NET".
Not surpriningly most managers who heard this backed away from a high price investment in SGI-machines. Why wouldn't they? Who in their right mind would spend a whole lot of money on a product which the company doesn't even believe in itself?
Also they lost a few good people from MIPS-tech when they claimed to move away from that architecture.
SGI is just picking up speed again with their new processors, 500MHz I believe are the latest.
The manager who said all these things "No MIPS, no IRIX" went over to become manager for Microsoft. I wonder if he'll do the same there? "No Windows, no x86, no
I'd also hate to see SGI go, but I wouldn't count them out just yet. They have a good technical solution and could make it, with enough good press. They are, after all, _outstanding_ when it comes to high-performance-computing.
You are not entitled to your opinion. You are entitled to your informed opinion. -- Harlan Ellison
(OK: so, it's functionally equivalent to a chimney in terms of heat output, but that's not what it is.
--
Free Software: Like love, it grows best when given away.
Despite all this, they got Maya right. Maya represents something that almost never happens in either business or programming. SGI bought Alias and Wavefront. which were separate companies selling incompatible animation systems. They then had them develop a new system with the best of both systems. This actually worked, and Maya is a good system, although hard to use.
Think about this for a moment. SGI actually got synergy from a merger. That is incredibly rare.
During this period, softimage|3D, the previous leader, was sold off by Microsoft and bought by Alias, many people left, and the new softimage|XSI is years late.
This is exactly what ruined The Hunchback Of Notre Dame for me; you could tell exactly what was computer generated (everything but the main characters) and what wasn't (the main characters.) Or, at least, everything but the main characters was high res, high detail, well designed, and fairly realistic looking, which the characters were standard 2d 'four color' cel drawings.
Vintage computer games and RPG books available. Email me if you're interested.
A renderfarm is just a whack of machines which accept work items from a master scheduler. You submit a job to the schedule, and a while later, an image file winds up in a directory somewhere. Doesn't even need to be fanicer than an NFS share called 'input,' an NFS share called 'output' and a script running on each machine that grabs and removes the next item from input, renders it, and saves the result to output.
Vintage computer games and RPG books available. Email me if you're interested.
Vintage computer games and RPG books available. Email me if you're interested.
Yes, but usually the background still maintains a distinctly cartoony look. Not so in the HBND. The dichotomy is too jarring, and breaks you out of the world the film is trying to create.
Vintage computer games and RPG books available. Email me if you're interested.
What in the wide wide world of sports does Alpha or FreeBSD support have to do with AMD? Mind you, I don't personally care what they run as a processor, I was just curious as the reasons for going with an Intel solution. I would think that Athlons using DDR would really shine doing this kind of rendering work. Pretty much every benchmark I've seen for the Athlon has it screaming through intensive floating point operations. Especially on Linux.
The line must be drawn here. This far. No further.
Comment removed based on user account deletion
Comment removed based on user account deletion
What's really impressive with the SGI platform is not the actual hardware, but the compilers. Sure, they do have great OpenGL accelerators, but it is the compilers that makes the difference. And ofcourse IRIX ;), the UNIX choice of all sysadmins.
Probable impossibilities are to be preferred to improbable possibilities.
Aristotele
It makes a good marketing point.
"People who spend the Big Bucks choose Linux"
Now if we could only get MS bashing into the movies ... realistic computer scenes with blue screens and the rest.
;-)
Check out the Vinny the Vampire comic strip
"It is a greater offense to steal men's labor, than their clothes"
I have to wonder how much Disney is using CGI assistance for their feature animation (we're not talking Pixar, natch ^_^ ).
We know about their CAPS (Computer Aided Production System) paintbox system that was used originally in the movie The Rescuers Down Under, the Deep Canvas system used to create backgrounds that was first seen in the movie Tarzan, and some animation sequences originally rendered on computer (remember the wildebeest stampede in The Lion King?).
Has Disney reached the point (at least since the early 1990's) they are trying to do animated features that have most of the feel of traditional cel animation but with heavy computer assistance?
It will be interesting to see how much of their computer systems that run CAPS and Deep Canvas use Linux as their operating systems.
W are replacing the O200s with Linux/PC boxes, not more IRIX boxes. We do use O2000 for part of our file serving, but we don't have any need for any expensive desktop graphics box. We don't really need anything better than O2 performance 3D graphics. We are much more dependent on memory and CPU. We do require good quality 2D performance, though, which isn't usually a priority for the 3D card vendors, unfortunately.
It would be great if we could get a nice cheap box that had decent 3D speed and was able to display color callibrated, film resolution, 24bpp images at 24 fps. It would be even better if we could display 36bpp, but that won't happen for quite some time.
Daniel WexlerI work in the R&D department at PDI/DreamWorks. My website has some real renderfarm statistics and some specific Shrek statistics. This article was fairly accurate, which is great. You can believe the hype. The entire production industry is behind Linux and is pushing the hardware and software vendors to firm up their offerings. We just had an industry wide meeting on Linux, and the movement toward Linux and away from Windows is clear and strong. There is even talk that some of the high end studios may work together to release some OpenSource tools, but right now that's still pie-in-the-sky. However, the fact that these studios, which previously guarded their trade secrets jealously, are even talking about this possiblity is exciting.
Also, PDI/DreamWorks is trying to release some code under OpenSource. I just released my frame buffer library under GPL. It is a small, but important, gesture, as it represents the first source code that we have ever released. We hope to have more soon.
Someone mentioned TIFF playback, which, these days is really easy, since it the latest rounds of cards are able to support 30fps playback using standard OpenGL calls. We have had our internal flipbook and quicktime programs working for over a year now. I'm sure the public tools will soon cover this gap. Also, audio is starting to work well. Our tools have nice sync'ed audio playback, which was one of the last things we got in place. We are now placing Linux workstations on animator desktops as opposed to batch use on the renderfarm. It has been a long road, but we are finally there.
Daniel Wexler...rendering animations for the silver screen, that means hardware accelerated OpenGL rendering doesn't cut it, it's highly probable that you want to do software rendering.
This is rapidly becoming a bit of a grey area; I personally reckon some kind of hybrid approach is going to become increasingly popular in the near future.
Modern graphics cards are designed specifically to rasterise gigantic numbers of tris very quickly; they have extremely well-tuned pipelines in place to do this, and using CPUs for this when there's such great custom hardware in place is kind of wasteful. GPUs have been on a Moore's-Law-on-steroids for a while now, so this is going to be increasingly the case.
I understand that some production renderers work by first generating very fat, 2D renders (eg. where each pixel contains a Z, UVs, material ids, a normal and so on) and then applying materials and effects as a 2.5D postprocess. Pass1 of this approach is absolutely ripe for hardware acceleration, in fact there's no real reason this can't be done right now.
So the question is, how long before the cheap renderfarm is joined by the cheap rasterisation farm, stuffed with nothing but GeForce2s, Celerons and RAM? Anyone?
With todays processors linux based rendering farms are really hard to beat. The two industry standard rendering programs have already been ported to linux, Mental Ray and the famous Photorealistic Renderman. On the artists side Softimage|3D 3.9 is already running on Linux. Porting XSI will be more tricky since it was developed for Wintel (it will create a fake registry in the IRIX version, you get the idea). Side FX has some products running on Linux. Maya 4.0 is also ported to Linux.
Add to this that most of those shops have developed lots of in-house tools that run on, surprise, IRIX, so porting to Linux seems a better choice than trying some acrobatic effort to make them work on Redmond's.
You want composition software? No problem, you can use NothingReal's impressive Shake
IMHO you may call those pplications which are truly useful to artists. Sure, they're damn expensive, but this is a niche market so that's not surprising.
It's pretty clear that this application of Linux is limited to inside the Dreamworks "bubble". It's the OS for custom software written for a machine with a specific purpose -- far different from home/office desktops.
So how to break out of the bubble? Look at how it broke into the bubble in the first place. You see, it's still all about the killer app. In this case, the animation tools are the killer app, because they run faster under linux and the workstations are cheaper, both of which translate into more productivity, which is critical in the production of a movie. Nobody cares if the machine can't do any other task (e-mail, browsing, spreadsheets, etc). The killer app is the one thing that makes it go.
This presents an opportunity for linux. Why are people so loyal to Macs? For some it's the Steve Jobs cult of personality, but for most Mac people it is becuase it is the finest tool for their job. They are willing to put up with expensive hardware, less of a software selection, Steve Jobs ego, and all the other downsides for one simple reason -- the core thing they do, their bread and butter, the killer app for them, is best on a Mac. Graphic design and desktop publishing are good examples of this.
So how does this present an opportunity for Linux? Make a workstation of some kind, for whatever market. Let's pick desktop video. As a videographer myself, I had at one time a HD bay and two hard drives on my computer. One was for desktop video, one was for programming, databases, games, e-mail, browsing, word processing, etc. The deskop video config had nothing else -- I kept it as clean as possible for stability and performance. And this is not uncommon. My solution was a poor-man's fix for not buying a separate PC. I think most videographers would prefer a wholly separate workstation dedicated to just video. So if a bunch of Linux guys got together and created the killer desktop video solution, at a price comparable (ideally better) than a windows one, a lot of people would buy them (relatively speaking).
This would be a good model for Linux development. Don't develop just an app. Develop an environment. To the end user, the environment is not the OS, but the true working space. Create a linux environment for digital video editing. Or CAD. Or whatever else people buy a system to do almost exclusively. Just like Dreamworks created a custom environment exclusively for animation. In these markets, software is a big part of the cost, usually more than the hardware itself. Trust me, these people will see the benefits of free software.
No, it doesn't have the thrill of taking on the giant Microsoft in the home or general purpose office desktop. But remember a few things: 1) linux can at least get to a critical mass where it sustains itself without needing to dramatically increase the number of users, and 2) "normal" (non-techie) people would be getting exposure to linux, and 3) what kind of system are these people going to recommend to friends? I have personally kept at least a half dozen people from joining or staying with AOL by telling them their alternatives. Over time, word-of-mouth works.
Just make the environment good. Make it rock solid -- for people in these type of workstation markets, reliability is huge, and linux has a big advantage. Make it a killer -- if it's desktop video, for instance, make it something that when I see it, I *gotta* have it, because it will work better and faster than any other desktop video workstation at twice it's price.
-----------------------
-----------------------
Stay in school, kids! Peace out, Dubya
I thought that said "Animation and SEX with Linux" for a second!
---
------
Sig