Renderfarm Setup Tips?

CarlosOlivaG4 asks: "We're in the process of acquiring and setting up a renderfarm, and I'm hoping the Slashdot community might light us up a little here. We'll use 6 to 8 nodes first, but would like to be able to expand it in the future." There was an earlier version of this question, but it dealt more with the hardware of the farm's nodes, rather than the network and software infrastructure on which these nodes would be based.

"In the hardware side, we still haven't made a choice between using AMD's Opteron or Apple's Xserve G5 (they have some very nice and price convenient cluster nodes which seem to be ideal for this kind of job), with Linux. As for the networking between them, is Gigaethernet enough or should we be going for Fiber? The software used to manage the render queues is another important point as well: I've been looking into Rush, and even though it's a commercial package, it works on all of the platforms we currently use (W2k/XP, Irix, OS X and Linux). But then there is also Dr. Queue, which is open source and is supported on at least the *NIX members of the aforementioned OS's. Other options include RenderPal and Pixar's RenderMan, but I would prefer an F/OSS alternative. Finally, it's worth noting that we'll be using the renderfarm for Maya and Adobe AfterEffects."

Comments

I'm a Machead, but...

[Jonas the Bold]

Don't use macs. First off, you have less choice in renderers, and second, the hardware is more expensive. Rendering is grunt work. Buy cheap systems that you can upgrade more often, and run linux or something.

Macs are very nice hardware, but you really don't need that for rendering. For workstations they make sense, but for rendering you really want to have a lot of fast computers rather than nice computers.

Re:I'm a Machead, but...

[MidnightBrewer]

Depends on which renderer you're using. Except for LightWave, RenderMan, Mental Ray, Animation Master, Cinema 4D, and even Blender, to name a few, it's true, you'll have problems getting support for OSX...oh, wait, that's quite a few right there, isn't it. Nevermind.

The question with building a headless Intel render farm is going to be licensing. Great, you can build cheap machines, but are you going to have to buy extra licenses just because you chose to go cross-platform? It's not just the main render engine you have to worry about, but also the 3rd-party shaders that you inevitably buy for doing specific things the default package doesn't cover.

If you do in-house programming instead, are you going to have your programmers write an Intel version of the shader as well as a Mac version? Good luck with that! Talk about unnecessary headaches. I think duplicate effort counts as an unjustifiable waste of money.

Finally, it's important to make sure that you don't run into limitations in the software, specifically procedural shaders, that cause the images to render differently on different platforms. It's annoying to get your frames back and they don't look just like the tests; sometimes, that can render the results unacceptable.

Why run Linux?

[puppetluva]

Why would you run Linux on the Apple boxes? Wouldn't OSX be just as good?

Software?

[m0rph3us0]

What render queue products are supported by your pieces of software? Why don't you try a few of them?
I'm sure a demo can be arranged.
I wouldn't go blindly marching in the direction of FOSS especially in something that is valuable enough to setup a renderfarm for.

Most importantly, find out what the people who will be using the software like and dislike about each package. And what works for them. If it saves you $30 per hour times 5 people software and hardware cost become insignifigant after one work week.

The biggest renderfarm in the world is useless if your people can't use it. Always remember that software is only good in its ability to meet the goals of the organization it supports.

Go with the G5's - your work is the important item

[Selecter]

Personally I'd go with the G5 Xserve with a few diskless cluster nodes tossed in.

Thats only if you desire maximum ease of use with minimum setup and running hassles. The same ease of use the regular G5's have is built into all their server stuff too. I'm sure the linux dudes will have something to say about that.....

I would take a really hard look at the ready made bio-information cluster they have all setup, and just load yer software as needed and off you go. But that's me. Some people seem to like futzing with computers.....After 20+ years doing that at work, I just wanna do what I wanna do when I wanna do it. Apple makes that easy.

I get paid to deal with headaches, I'm not gonna deal with them at home too.

Why ask slashdot?

[Anonymous Coward]

The first thing that you should do it find someone else to ask other than slashdot. Seriously, there are an extremely small number of readers who would actually know that answer to that, and none of them will post. You're just going to get a bunch of opinions from people who have no idea what they are talking about, like:
"I have a mac and I have editied video recorded on my digital video recorder. Let me tell you everything you need to know..."
"Beowulf cluster! Beowolf cluster! I have no idea what that means, but I'll say it anyways, because it seems appropriate in this circumstance!"
If you have real money invested on this, get serious.

Re:Cinelerra

[dhartman]

I've done a bit of work with Cinelerra and have been very pleased with the results. With three Athlon XP 1900 ish machines with no more than 512MB ram I was able to render at 30fps with full DVD resolution (720x480) output.

The render nodes only run cli tools and do not require local storage. A bootable Knoppix cd could be made to create a temporary render node. Imagine using 10 computers in your office to render video in the off hours and reboot into Windoze in the morning with the user not knowing.

Sun Grid Engine, at least for Maya

[FueledByRamen]

I don't know what the setup for After Effects and such looks like, but I managed to build a pretty good Sun Grid Engine system for distributing Maya batch renders. SGE is free, works on Linux or Solaris, x86 or SPARC (I obviously used the x86 linux binaries), and seems to be very well designed. I set up a pretty solid system for netbooting the clients, running them diskless (or with local swap drives), adding new clients on the fly (all scripted), and it all worked flawlessly. You could submit a batch job and it would distribute it per-frame as an array job to all the different nodes. Or you could just run SETI on all of them...

It's since been taken down in favor of running Alfred (because I no longer use Maya's builtin renderer, we've moved on to MTOR and PRMan), but I still have all of the files and scripts for it. If anyone's interested, I'd be happy to share: sabretooth@gmail.com

Re:Small cluster

[Anonymous Coward]

Secondly, from your questions it is obvious you're no rendering farm guru so why did the task of planning/researching the configuration get assigned to you? You should ... find a small and focused SI who will propose you a better solution than you can come up on your own

This attitude bothers me, and not for the first time here on Slashdot. How the hell do you think those experts got to be experts? Do you think they just *poofed* into being with all their knowledge and skills already existent? No, at some point, they started with little or no knowledge of the subject and gradually accumulated enough knowledge and experience to become experts!

Sheeesh! If everybody listened to this advice they never would do anything new or different for fear of coming up with some sub-optimal solution.

Darwin

[Dav3K]

But more to the parent's point, OSX could be run Darwin-only, AKA BSD-style command-line interface. The bloat can be stripped out of OSX just as easily as it can for linux. Additionally, the user would have the comfort in knowing that his render farm is using the same OS as the workstations used to control it.

As other respondants have suggested, I guess it would come down to which OS supported the entire collection of desired applications for the job.

Some Tips

[mnemonic_]

These are just some tips I've heard in my 4 years of experience.

Whichever processors you go with, make sure the entire farm uses the same type. Otherwise peculiar rendering differences might occur, in things like particles, hair/fur and fluids.

I suggest going with the Opterons just for the PC compatibility. While the CG industry is becoming more diverse hardware-wise, it is still dominated by PC's and to a much lesser extent SGI boxes (5 years ago it was all SGI). Using PC's keeps your options open. Perhaps someday you will find 3ds max and its included distributed rendreing software more suitable for a task, and that can only be used with PC's. Same goes with the Mental Ray and Brazil renderers and the Combustion compositing software. Macs just have not been widely used in the 3d graphics industry, and so the vast majority of 3d content creation software is PC and SGI only (Maya Unlimited is only available on PC and SGI, while a lower end wersion is on Mac). And VirtualPC cannot be used to emulate 3d hardware acceleration (and it shouldn't be used for anything processor intensive anyways), though this only applies to the hardware rendered viewports in the apps. Having only Macs would be risky, and could limit your capabilities significantly.

Pixar's PRMan (Photorealistic RenderMan) is a full blown renderer, not just something to help distribute render jobs. It is generally considered the best in the industry, though MentalRay and Brazil have gained significant followings. For a cheap but effective render queueing system, check out Smegde. Smedge was used by Manex Visual Effects for handling some of the effects shots in the Matrix trilogy. If you're running the Linux version of Maya (x86 only) it is not too difficult to distribute the render tasks yourself using shell scripts and the command-line renderer.

GB Ethernet should be fine, the bottleneck will be in the actual image processing not data transfer rates. 100Mb ethenet might even get the job done, thught I'd use GB for the added speed when copying large files. YMMV of course.

Overall I'd try to create a very flexible system, one that will definitely support the newest CG software down the road and one that ensures compatibility with everything, for those always short deadlines. Goodl luck with your rendering.

Re:Small cluster

[Welsh Dwarf]

This is something I posted on an earlier discussion. The fact of the matter is that once you've got to a certain level in a subject, you forget what it's like to be starting out, or even that you 'started out', and hence loose all consideration for people new to your field.

Not a dig, just a remark on human nature