Building a Render Farm?

Dark Bard asks: "What is the best configuration for a rendering machine. Given the variety of chips and components what are the best options balancing cost verses speed. I've been running Lightwave and plan to shift soon to Maya. AMD chips seem to be rated as a bit faster for rendering and they cost less, as well. Given the number of types of RAM available, what are the advantages of each verses cost? Should ECC be considered? What about motherboards? Integrated video would be ideal but it would have to be adequate to run the software. Is there any advantage to running the new 64 bit processors? Should you consider dual chips? What about operating system? Lightwave won't run on Linux but Maya will. How well do the major operating systems compare when used for this purpose?"

Comments

now or later

[aminorex]

If Maya came in an amd64 native linux binary,
it would kick some serious booty, but assuming
that you need your pipeline filling up NOW
rather than later, go for 4-way Athlon MPs
with all the ECC DDR you can cram into them.
More than 4-way, and you're paying premium
prices for the chipsets. Less than 4-way,
and you're wasting cycles that could be
amortized over wait states. ECC is de rigeur
just because you don't want to be chasing
down defective dimms all the live long day.
Similarly, I can't see any reason for paying
the 10% performance tax of Windows 2000 Pro
(the best of the Windows lot for SMP use)
or suffering its 2-way limitations, unless
theres some Lightwave feature that is
critical to your operation.

wtf?!-Shag the error monster.

[Anonymous Coward]

"Rendering is adding light to an image that is going to be displayed for a small part of one second."

Rendering is math. Think error accumulation.

My ideas

[FueledByRamen]

I just built up a workstation on which to run Maya 5, and have been using the hell out of it for a week or two, so here's what I have to say about Maya render stations (which are basically workstations minus the Quadro or FireGL card):

Motherboard: Because Maya's renderer is SMP-enabled, you'll probably want to get dual-processor boxes. I suggest AMD machines based on cost - they're a hell of a lot cheaper than their Intel counterparts. A good motherboard is the Asus A7M266-D (760 MPX chipset) - it supports upto 3.5 GB of RAM, and has been rock-solid stable under Linux for me, but it doesn't have onboard video or networking. A good board with onboard video/LAN/SCSI/etc is the Tyan S2466 dual-Athlon board. Keep in mind, though, that these things suck a LOT of power; a good (think Antec) 400 watt (or better) PS is a MUST, or you're going to fry it. I had a Tyan S2460 (2466 minus SCSI, NIC, and onboard video) that fried an off-brand 400 watt power supply because it was sucking so much juice. Don't worry about the specs of the onboard video, because Maya's batch renderer doesn't even bother setting up an OpenGL context; it's completely software rendering from the command line.

Processors: Currently, the Athlon MP 2600+ is at a good price/performance point (approx $150 ea, and the next one up is $200 ea). I'd load every box with a pair of those. If you're looking to save some money and don't mind furiously voiding the warranty on each and every CPU you buy (like me), you can grab some conductive paint, a paintbrush, and a bit of tape, and convert Athlon XP processors into MP-capable processors simply by connecting one of the bridges on the top of the CPU. I did that to a pair of XP2000+ processors, and it worked great; they're still churning away together just fine after almost a year. The price difference is about $70 - if it's worth the savings to you, go ahead and try it. If the chip won't run in SMP mode (rare), you can always stick it into a cheap motherboard and make a desktop workstation for your favorite manager.

Memory: Maya is HEAVY on RAM usage. If you're not planning on having a disk in every machine with plenty of swap space (for those larger scenes), I'd go with at least 1.5 GB (and probably 2 GB) of registered DDR RAM. You should be able to get away with only putting in 1 GB per machine if you have swap space. If you can afford it, spend the extra for ECC memory, because it's nice not to have to worry about memory errors. I did a quick test render (Maya software, 640x480, draft quality) of a 350,000-poly object I'm currently working on, and its average memory usage was right around 600 MB (peak arena size 1186.25 MB, as reported by the renderer). Don't underestimate the amount of RAM that you'll need! My workstation has 1.5 GB of RAM, and it still hits swap.

If you're not booting over the network, I'd throw a fast 18 GB drive in every machine. Make a swap partition of at least 2 GB, and install the OS (I prefer Debian stable, use whatever you want) on the rest of it. You shouldn't have to go overboard on disk; a 7200 RPM IDE drive will be more than adequate.

Networking: Depending on how your scene data is distributed (central fileserver? every node gets a copy beforehand?), I'd go with 100mb switched or full GigE. It basically depends on whether or not you're willing to pay the premium for the faster interconnect. If you put GigE cards in all of the nodes (which doesn't cost much more than putting a good 100mbit card in anyway - I recommend the Netgear GA302 and GA604 gigabit cards), you can always replace the switch if you find things are too slow.

Power and cooling - you'll need plenty of both. One Athlon puts of quite a bit of heat, but pack two into a small box and put 20 of them in a rack (assuming 2U), and you're talking about serious overheating possibilities. You're going to need one hell of an air-conditioning system. If you've already got a datacenter set up, then y

Re:Dear Slashdot... do my work for me please...

[foooo]

I agree in many configurations good harddrives would be important, but this person isn't talking about a single machine rendering op. He wants some sort of render "farm" implying multiple machines. In which case my first reccomendation (to save money) is to ditch as much cluster "client" storage hardware as possible, after all the cluster clients don't exist to store, they exist to render. Rendering images to a ram drive might be a good thing, or get a small fast hard drive for the images and OS, remembering that the primary storage for any sizable render farm is usually a Fibre Channel RAID array. In best case scenarios all of the render clients have direct fibre channel to the RAID box, making local storage an extra expense.

If you're doing a home cooked solution, I would suspect that local hard drives would become more important. And on a single machine rendering setup you'd want a hella fast raid array.

And obviously the more RAM the better, but budget may constrain this. Chopping images into small pieces might make even 256MB workable even for today's high res requirements for images. Somewhat akin to dividing up large scientific calculations.

My question to you... does Maya provide tools for managing render farms? I'm pretty sure the tools I suggested all offered "render farm" capabillity. Although I admit I have no clue what "extras" are included in Maya.

Cheers!

~foooo

Re:Dear Slashdot... do my work for me please...

[tolldog]

Maya has a real cheesey remote render tool. Its not really worth the time.

I built a 500 node/1000 proc render farm. Maya is a beefy application. We had scene files that were easiy 500mb. You can not chop up a scene file like that, it all needs to be loaded so that lighting can be done correctly.

Rendering to ram doesn't make much sense. You need all of that memory for the OS and the application.

A local hard drive makes a world of difference. I have seen night and day differences between rendering off of scene files locally and off the network (connected to an SGI raid file server). Few NFS servers can keep up with many boxes. And slow reads and writes are even worse. It is much better to burst the read and writes by doing whole files at a time. The server will thank you.

I can't recomend Platform's LSF enough, it allowed
a small shop to scale from 40 desktops to 500 render boxes. All it does is job scheduling, so you need to put some intelligence into it. With one or two developers, you can come up with a snazy system that is pretty reliable.

I am also impressed with Qube! from pipelinefx. I have only demoed the software. It has the logic in it that LSF does not. So it all depends on how out of the box you want to go.

I have found that most render solutions cater to studios with 3 artists and 5 render boxes and don't scale well outside of that. If one wants to work on a larger scale, spending the time looking into the slightly more pricey options is well worth it.

-Tim