Renderfarm Setup Tips?

CarlosOlivaG4 asks: "We're in the process of acquiring and setting up a renderfarm, and I'm hoping the Slashdot community might light us up a little here. We'll use 6 to 8 nodes first, but would like to be able to expand it in the future." There was an earlier version of this question, but it dealt more with the hardware of the farm's nodes, rather than the network and software infrastructure on which these nodes would be based.

"In the hardware side, we still haven't made a choice between using AMD's Opteron or Apple's Xserve G5 (they have some very nice and price convenient cluster nodes which seem to be ideal for this kind of job), with Linux. As for the networking between them, is Gigaethernet enough or should we be going for Fiber? The software used to manage the render queues is another important point as well: I've been looking into Rush, and even though it's a commercial package, it works on all of the platforms we currently use (W2k/XP, Irix, OS X and Linux). But then there is also Dr. Queue, which is open source and is supported on at least the *NIX members of the aforementioned OS's. Other options include RenderPal and Pixar's RenderMan, but I would prefer an F/OSS alternative. Finally, it's worth noting that we'll be using the renderfarm for Maya and Adobe AfterEffects."

Comments

Cinelerra

[selfabuse]

You might want to check out Cinelerra. It has pretty good support for renderfarms. I built one out of scrap 300mhz machines, and it only took a weekend.

I'm a Machead, but...

[Jonas the Bold]

Don't use macs. First off, you have less choice in renderers, and second, the hardware is more expensive. Rendering is grunt work. Buy cheap systems that you can upgrade more often, and run linux or something.

Macs are very nice hardware, but you really don't need that for rendering. For workstations they make sense, but for rendering you really want to have a lot of fast computers rather than nice computers.

Our experiences

[Thagg]

We had a renderfarm for "The Chronicles of Riddick" of 40 boxes. Each box was a dual-proc Opteron.

We evaluated several render-queue management systems, and decided on Rush. The most persuasive arguments for using Rush were the very good experience we have heard from other users, and the simplicity of extending it to manage a variety of different tasks. I have to add Hammerhead to the list of happy customers. It did everything we could have hoped for. In particular, it was able to handle the inevitable crashing of machines pretty well.

While it's true that Rush is a proprietary, gotta-pay-for-it system; a robust render queue management system pays for itself very quickly in the ability to make your renferfarm productive. Perhaps a render queue manager is overkill when you have just 6 or 8 systems, but once you get up to 30 or 40 it is essential.

Our experience is all under Linux, but if you're going to be running After Effects that means that you're not going to be running Linux -- so there's not too much more I can help you with there. We did find that the dual Opterons worked much more efficiently than dual Xeons in multiprocessor rendering -- don't know about the Xserves, though. We were running mostly Maya, RenderMan, Shake, and our own in-house tools on the farm.

This farm is unfortunately powered down now that Riddick is done -- if you need some dual opterons, let me know at thad@hammerhead.com

High Performance Computing Perspective

[Anonymous Coward]

I manage a large Linux cluster that is used for a wide variety of tasks, including running Renderman. Our nodes are dual Xeon 2.4 GHz with 2 GB RAM and they are more than sufficient for the design group's rendering needs (primarily Maya). Our scheduler (OpenPBS) lets them request up to 64 processors at a time, which, I'm told by their IT guy, works well with their Renderman maitre de.

There is a fiber interface (Myrinet) to each node used by the MPI crowd, but our rendering group doesn't use it; they seem content with the performance of Ethernet over copper. Your needs may be different, of course, but latency isn't really an issue for rendering, and copper should provide all the bandwidth you need.

I'm not knowledgable regarding all the software packages you list there, but I'm wondering if any of them would really take advantage of a 64-bit kernel (either on Opteron or G5-PPC970). Of course you can put a PPC version of Linux on the Xserve, but not without sacrificing nearly all Apple-provided management. If you expand the cluster to a large number of nodes, or even if you keep a small number of nodes but place it in a remote location, Xserve running Linux would be painful to manage (no remote power-off/-on, remote console problems). Xserve is shiny and has the requisite blue LED's, but and AMD or Intel box (from the right vendor) would be much easier to manage remotely.

Re:Small cluster

[Anonymous Coward]

Secondly, from your questions it is obvious you're no rendering farm guru so why did the task of planning/researching the configuration get assigned to you? You should ... find a small and focused SI who will propose you a better solution than you can come up on your own

This attitude bothers me, and not for the first time here on Slashdot. How the hell do you think those experts got to be experts? Do you think they just *poofed* into being with all their knowledge and skills already existent? No, at some point, they started with little or no knowledge of the subject and gradually accumulated enough knowledge and experience to become experts!

Sheeesh! If everybody listened to this advice they never would do anything new or different for fear of coming up with some sub-optimal solution.

Depends on your workstations

[BBStriker]

Hey

I've set up and administrated a number of farms over the years (doing it as I type. its.. what I do). One thing you really want to do, certainly with Maya's renderer, is to try to use the same OS and platform on your farm as you use on your user workstations. There can be subtle or even obvious differences in the render output between OS's, and since you'll have enough issues to deal with you'll want to keep cross-platform incompatabilities out of the mix. Please, trust me on this. Had to deal with Maya Irix/Win2k/Linux differences in the past.

As for queueing software, give Condor a look-see. Free and functional. I reverse-engineered a Perl version of it before they made their source available, and my version has been run quite successfully at several animation studios and an effects house over the years. It's a well architected system for distributed computing.

Feel free to contact me if you've got any other render system or management questions. I'm always interested in seeing how other studios approach the challenge.

Deadline Render Queue (beta)

[bhouston]

At Frantic Films we have over the past year developed our own network rendering solution: Deadline. Our solution has just recently entered a beta testing period thus if people are so inclined one can have a look at the current product (screenshots) and possibly download a trial version (download page). We used Deadline on a number of recent feature films including Scooby Doo 2 and Paycheck.

We did this because we primarily use Discreet's 3dsmax (with Brazil and V-Ray) and Eyeon's Digital Fusion. We have found that most existing render farm solutions do not support these two packages very well -- thus we decided to develop our own custom solution. We also support After Effects, Alias|Maya, AIR and other RenderMan compliant rendering packages.

Of interest to the general Slashdot crowd may be that this Deadline Render Management Solution is based on the open source (BSD License) Exocortex C# library originally released with this C# 3D Engine. Deadline is built with C# in the hopes that using Mono we will be able to start supporting Linux with minimal extra effort.

I'll be reading all the posts on this Slashdot thread but I would also appreciate any direct feedback on our current beta product. We also found solutions such as Rush and Smedge to be less than user friendly in many respects. Thus we have tried as best as we could to increase a 3D package that is not well supported by most render farm management solutions -- except for Discreet's Backburner (which we found not that that scalable.)

Some Tips

[mnemonic_]

These are just some tips I've heard in my 4 years of experience.

Whichever processors you go with, make sure the entire farm uses the same type. Otherwise peculiar rendering differences might occur, in things like particles, hair/fur and fluids.

I suggest going with the Opterons just for the PC compatibility. While the CG industry is becoming more diverse hardware-wise, it is still dominated by PC's and to a much lesser extent SGI boxes (5 years ago it was all SGI). Using PC's keeps your options open. Perhaps someday you will find 3ds max and its included distributed rendreing software more suitable for a task, and that can only be used with PC's. Same goes with the Mental Ray and Brazil renderers and the Combustion compositing software. Macs just have not been widely used in the 3d graphics industry, and so the vast majority of 3d content creation software is PC and SGI only (Maya Unlimited is only available on PC and SGI, while a lower end wersion is on Mac). And VirtualPC cannot be used to emulate 3d hardware acceleration (and it shouldn't be used for anything processor intensive anyways), though this only applies to the hardware rendered viewports in the apps. Having only Macs would be risky, and could limit your capabilities significantly.

Pixar's PRMan (Photorealistic RenderMan) is a full blown renderer, not just something to help distribute render jobs. It is generally considered the best in the industry, though MentalRay and Brazil have gained significant followings. For a cheap but effective render queueing system, check out Smegde. Smedge was used by Manex Visual Effects for handling some of the effects shots in the Matrix trilogy. If you're running the Linux version of Maya (x86 only) it is not too difficult to distribute the render tasks yourself using shell scripts and the command-line renderer.

GB Ethernet should be fine, the bottleneck will be in the actual image processing not data transfer rates. 100Mb ethenet might even get the job done, thught I'd use GB for the added speed when copying large files. YMMV of course.

Overall I'd try to create a very flexible system, one that will definitely support the newest CG software down the road and one that ensures compatibility with everything, for those always short deadlines. Goodl luck with your rendering.