Renderfarm Setup Tips?

CarlosOlivaG4 asks: "We're in the process of acquiring and setting up a renderfarm, and I'm hoping the Slashdot community might light us up a little here. We'll use 6 to 8 nodes first, but would like to be able to expand it in the future." There was an earlier version of this question, but it dealt more with the hardware of the farm's nodes, rather than the network and software infrastructure on which these nodes would be based.

"In the hardware side, we still haven't made a choice between using AMD's Opteron or Apple's Xserve G5 (they have some very nice and price convenient cluster nodes which seem to be ideal for this kind of job), with Linux. As for the networking between them, is Gigaethernet enough or should we be going for Fiber? The software used to manage the render queues is another important point as well: I've been looking into Rush, and even though it's a commercial package, it works on all of the platforms we currently use (W2k/XP, Irix, OS X and Linux). But then there is also Dr. Queue, which is open source and is supported on at least the *NIX members of the aforementioned OS's. Other options include RenderPal and Pixar's RenderMan, but I would prefer an F/OSS alternative. Finally, it's worth noting that we'll be using the renderfarm for Maya and Adobe AfterEffects."

Comments

Cinelerra

[selfabuse]

You might want to check out Cinelerra. It has pretty good support for renderfarms. I built one out of scrap 300mhz machines, and it only took a weekend.

node deployment: g4u!

[hubertf]

Check out g4u for deploying your render machines - it's a image based disk cloning tool that uses DHCP and FTP and which doesn't care what you run on your clients. (g4u itself is based on NetBSD, but that doesn't matter for the application).

I've used g4u to setup a ~50 node video rendering cluster, see my webpage on the Regensburg Marathon Cluster.

Enjoy!

- Hubert

Our experiences

[Thagg]

We had a renderfarm for "The Chronicles of Riddick" of 40 boxes. Each box was a dual-proc Opteron.

We evaluated several render-queue management systems, and decided on Rush. The most persuasive arguments for using Rush were the very good experience we have heard from other users, and the simplicity of extending it to manage a variety of different tasks. I have to add Hammerhead to the list of happy customers. It did everything we could have hoped for. In particular, it was able to handle the inevitable crashing of machines pretty well.

While it's true that Rush is a proprietary, gotta-pay-for-it system; a robust render queue management system pays for itself very quickly in the ability to make your renferfarm productive. Perhaps a render queue manager is overkill when you have just 6 or 8 systems, but once you get up to 30 or 40 it is essential.

Our experience is all under Linux, but if you're going to be running After Effects that means that you're not going to be running Linux -- so there's not too much more I can help you with there. We did find that the dual Opterons worked much more efficiently than dual Xeons in multiprocessor rendering -- don't know about the Xserves, though. We were running mostly Maya, RenderMan, Shake, and our own in-house tools on the farm.

This farm is unfortunately powered down now that Riddick is done -- if you need some dual opterons, let me know at thad@hammerhead.com

Re:Our experiences

[Thagg]

Good questions

1.) The words 'dual' and 'Opteron' both surprised us. We were kind of under the impression that maybe single proc machines would be better for a render farm. We were really curious why dual was chosen over single. Did the extra cost end up being worth it?

The computers are relatively cheap -- it's render licenses (especially for RenderMan) that are expensive. With the newest version of RenderMan, Pixar has deigned to let us use the two processors of a dual-processor machine with a single license. This lowers the cost of rendering by about 60%, if the machine rendered twice as fast with dual processors. In fact, for RenderMan, the Opterons were indeed almost twice as fast, where the Xeons were only about 50% faster.

Our other big rendering application was Shake, and it also allowed the use of two processors with one license.


2.) You mentioned that Opteron was more efficient than Xeons. I just had to ask: Was the particular software you were using particularly tuned to Opteron (i.e. 64-bit?) or was the 32-bit side of it just pleasant to work with? Any more insight you can share with me about the use of Opteron would be most helpful.

Yes and no. The Opterons are the first AMD machines to implement the SSE2 instructions, which are heavily used by RenderMan. Also, the HyperChannel communication between processors on the Opteron is light-years beyond the communication between Athlons and Xeons. On the other hand, there is absolutely no advantage in the 64-bitness of the Opterons -- we were running a 32-bit Linux (RedHat 9), and we weren't using more than 4 GB memory on any of the boxes.

3.) Did you guys end up buying a bunch of machines from a place like IBM or something, or was it more like "we bought the components and assembled ourselves..?" If it's the former, how'd you like the service?

We hired a beige-box manufacturer. We specced it out to various places, and PC Mall built them for the best price. If I had to do it over again, I'm not sure that I wouldn't go with IBM -- while they cost a lot more, I expect that they'd build more solid systems.

4.) Any regrets or things you'd do differently next time around?

We bought minitower machines instead of the more trendy, space- and power-efficient 1U or blade machines. We did that so that we could potentially use the new Gelato renderer from NVidia -- that software uses the current NVidia high-performance graphics cards as an external array processor, giving significantly better render performance.

As we didn't end up using Gelato, that was perhaps a mistake. We ended up power and HVAC constrained in the end -- as happens with almost every renderfarm I've heard of.

5.) Why are you getting rid of the machines used for Riddick? Or did I read that wrong?

No, you read it right. Hammerhead is a small company, typically working on just one show at a time. We don't see a use for the machines for another nine months or so, as we begin development of the next project -- and it just isn't right to leave all that compute horespower idle.

Thad Beier
Hammerhead Productions

Re:Why run Linux?

[Anonymous Coward]

Why would you run Linux on the Apple boxes?

Because of its unnecessary flashiness? OS X is notoriously bloated. For the command line junkies among us, Linux fits the bill.

Depends on your workstations

[BBStriker]

Hey

I've set up and administrated a number of farms over the years (doing it as I type. its.. what I do). One thing you really want to do, certainly with Maya's renderer, is to try to use the same OS and platform on your farm as you use on your user workstations. There can be subtle or even obvious differences in the render output between OS's, and since you'll have enough issues to deal with you'll want to keep cross-platform incompatabilities out of the mix. Please, trust me on this. Had to deal with Maya Irix/Win2k/Linux differences in the past.

As for queueing software, give Condor a look-see. Free and functional. I reverse-engineered a Perl version of it before they made their source available, and my version has been run quite successfully at several animation studios and an effects house over the years. It's a well architected system for distributed computing.

Feel free to contact me if you've got any other render system or management questions. I'm always interested in seeing how other studios approach the challenge.