Slashdot Mirror
Building a Render Farm?
Dark Bard asks: "What is the best configuration for a rendering machine. Given the variety of chips and components what are the best options balancing cost verses speed. I've been running Lightwave and plan to shift soon to Maya. AMD chips seem to be rated as a bit faster for rendering and they cost less, as well. Given the number of types of RAM available, what are the advantages of each verses cost? Should ECC be considered? What about motherboards? Integrated video would be ideal but it would have to be adequate to run the software. Is there any advantage to running the new 64 bit processors? Should you consider dual chips? What about operating system? Lightwave won't run on Linux but Maya will. How well do the major operating systems compare when used for this purpose?"
Even if 64 bit binaries are available you will probably get greater performance at a lower cost by using "cheap as chips"(sic) chips in SMP configurations.
Future proofing is another issue however. Many clustering technologies rely upon a common denominator. For instance with OpenMOSIX running on varied hardware, your code must be compiled for the lowest common denominator. So if you have 20 P4's and one P2, you will only be able to run software compiled for P2 on the cluster (at least without errors).
YMMV - It's been a while. :P
Q.
Insert Signature Here
Have you looked at RenderDrive? The company that my wife works bought one of these recently. The guy that uses it loves it. It does the job much quicker.
It's a general pupose computer that has special hardware that is used to do the rendering. The OS is linux. In order to get it on the newtork you setup a floppy with your config file. There's a plug-in for your system that is used to do the rendering on he computer.
I am a render farmer... and this is what I have gathered.
If you are doing single frames, get good systems.
If you are doing animation, get many systems.
You need a scheduler or batch queueing system.
I suggest building one off of LSF like I have in the past and it is what I use now -or- get a nice prebuilt one like what pipelinefx has. This is extremely important. Without a good job scheduler, your cpu utilization will be less than optimal and it will cause you to get more machines than needed.
I have worked both with inhouse and 3rd party renderers. Maya doesn't do that bad of job, and the license for rendering is free. Big plus.
Inhouse is great too, but if you don't have a good sized company, you probably won't have one and probably wouldn't be asking this question.
As for the machine. Most products are only speced for RedHat and for only certain releases. This is important when suport becomes an issue, and it will, when things don't work for a shot. I suggest a dual proc Pentium 4. I do not know if AMD's chips have been blessed by the vendors or not. I know when last I looked it was being investigated. Go for a dual system, even if you don't have a mutli-threaded renderer. Also max out the system ram at 4gb.
As for hardware vendors, I have used VA-Linux, HP, Compaq and SGI linux boxes as well as white-boxed. Hardware support is a big deal when you are a small shop. Go with a name brand unless you want to buy a ton of hot spares.
Storage is a big deal too. You obviously need a place to put the images, but you also need a server beefy enough to serve up a couple hundred mb to several boxes at the same time. When you get 500+ boxes, it gets to be an even bigger deal.
Other administration software i found usefull...
SystemImager and gsh ( a distributed shell ) they allowed me to manage the 500 boxes as well as develop code and debug renders... I can not imagine what it would have been like without it.
-Tim
-I just work here... how am I supposed to know?
A few corrections...
Hard drives are important. You never ever want to render to NFS. Bad things happen. Trust me.
Also, its best if the files are local that its rendering. So having a 20gb scsi drive speeds things up... cache the scene file local, render local, copy the result back.
And 256 mb ram... hmm.. try 4bg of ram. I have hit the 2gb limit for renders many times, and if you want to have two renders going at the same time, you need that extra memory.
Software wise, as for 3rd party tools, renderman is great, but dang expensive. Maya is a good product. SoftImage is good as well, but Lightwave, pov-ray and others are not really in the camp of commercial products for animation.
100mb to the rack switch, dual gb from rack to big switches is plenty. No need for 1000mb. Most servers can't serve up data that fast. Not for hundreds of boxes.
And trust me, nobody has "that whole render farm thing figured out". Not even the big guys... we learn stuff every day.
-Tim
-I just work here... how am I supposed to know?
Nope nope nope:
I wouldn't consider anything less than 512MB of RAM. Get as much RAM as you can afford, but spread the wealth; each machine on the renderfarm should have the same amount.
As others have pointed out, being able to cache the files locally is important. The individual rendering machines' disks need not be enormous; an 18GB SCSI disk would suffice. Each render node should have a 100Mbit ethernet card, and all should connect to a gigabit switch, which then connects to your file server and rendering job server.
As for actual software, Blender is the last thing he wants if he's trying to upgrade from Lightwave. Maya's animation, modelling, and dynamics are much better than Lightwave, but the stock Maya software renderer is worse.
When it comes to rendering software, there are a few options. When it comes to RenderMan-compliant renderers, there's BMRT, Entropy, 3Delight, RenderDotC, and, of course, Pixar's own PhotoRealistic RenderMan which, as of version 11, supports raytracing and global illumination. The problem with these is that Maya's built-in RenderMan exporter sucks, so you'll need a 3rd party one (Liquid is a good, open-source one that was written by a Weta employee and used on the Lord of the Rings trilogy).
However, Exluna, the company that made Entropy and owned the rights to BMRT, is no more. They were sued by Pixar on some bogus patent issue, and ended up getting bought by nVidia, who promptly discontinued development of Entropy. BMRT is no longer officially available.
3Delight seems to be fairly fast, but has limited raytracing abilities and no global illumination. It's available as a free download from 3delight.com, but only for private and limited-commercial use.
However, Maya versions 4.5 and up ship with mentalRay, the same ray-tracing global illumination renderer that comes with SoftImage, and unless I'm mistaken Alias has been kind enough to allow unlimited render nodes. This means you can buy one copy of Maya for doing your modelling and animation, and install Maya's software renderer or mentalRay on your renderfarm.
Also, unless you plan on using Maya's hardware renderer on the renderfarm (not really all that necessary; it's fast enough that you can do it on your workstation), not a single machine in your render farm needs a video card. Once they're set up and configured, there's no need for you to access them directly. Maya can do rendering without a GUI, and RenderMan renderers don't have GUIs.
Just wanted to drop off a tip that has boosted my productivity tremendously. Multi-Pass rendering. My company got me a copy of After Effects, Adobe's compositing package. With it, I learned how to break up a scene into smaller elements and put them back together in After Effects. This has allowed me to do all kinds of things to save on rendering time. For example, I recently rendered an animation of a machine that has a fast moving piece on it. Motion blur is an expensive feature of a scene to render. However, it can be a waste of render time if only one element of the scene really really needs it. I was able to render the static elements of the scene sans motion blur, and render only the moving bits with the motion blur, thus saving a great deal of render time that could be dedicated to other things.
I just wanted to throw this bit of advice. If your animators don't have a compositing package, it would be a worthwhile expenditure, even if it costs a machine or two from the render farm. I can't speak for other compositing packages, but I can tell you that After Effects is a damn cool, useful app for rendering. Me personally, I'd rather have 1 machine in my render farm with After Effects, than 2 machines in it without AE. If that gives you an idea.
"Derp de derp."
It's essentially a buffer that boosts the clock signal so that the edges of the clock appear more sharply defined to the memory (the clock gets weak when it's directly driving a bunch of memory modules). The buffer tries to isolate the motherboard chipset from the pile of chips on a large DIMM, which is why lots of motherboards require registered DIMMs in order to use lots of memory. That buffer also delays data transfer by one clock cycle.