Slashdot Mirror
Building a Render Farm?
Dark Bard asks: "What is the best configuration for a rendering machine. Given the variety of chips and components what are the best options balancing cost verses speed. I've been running Lightwave and plan to shift soon to Maya. AMD chips seem to be rated as a bit faster for rendering and they cost less, as well. Given the number of types of RAM available, what are the advantages of each verses cost? Should ECC be considered? What about motherboards? Integrated video would be ideal but it would have to be adequate to run the software. Is there any advantage to running the new 64 bit processors? Should you consider dual chips? What about operating system? Lightwave won't run on Linux but Maya will. How well do the major operating systems compare when used for this purpose?"
Oh, wait. Never mind.
If Maya came in an amd64 native linux binary,
it would kick some serious booty, but assuming
that you need your pipeline filling up NOW
rather than later, go for 4-way Athlon MPs
with all the ECC DDR you can cram into them.
More than 4-way, and you're paying premium
prices for the chipsets. Less than 4-way,
and you're wasting cycles that could be
amortized over wait states. ECC is de rigeur
just because you don't want to be chasing
down defective dimms all the live long day.
Similarly, I can't see any reason for paying
the 10% performance tax of Windows 2000 Pro
(the best of the Windows lot for SMP use)
or suffering its 2-way limitations, unless
theres some Lightwave feature that is
critical to your operation.
-I like my women like I like my tea: green-
At times like this it's hard to resist kharma whoring... so I won't.
r ont.comD _Modellin g.html
Dear Slashdot,
Please do my research for me. I can't possibly use google and look up the 309830983 gazillion studies on cluster/render farm configurations.
I won't/can't give you any specific information related to budget, my personal experience with hardware, the team to be working on the project (it could be just me... or 500 people) and the importance of data integrity (if I lose a week of work it might end my life... or I might not care).
-clueless reader
Dear Clueless,
Well, if your budget is a billion dollars why not just buy Pixar? They have the whole render farm thing figured out.
If your budget is any thing less than that try looking for open source, GNU, freeware, shareware, free for non-commercial use renderfarmy stuff and run it on AMD based Linux boxes.
Hard drives aren't so important on the individual machines. Get a decent raid on a server machine that will support the number of computers in the render farm. Use 100-1000 mbit ethernet (unless you want to spend a fortune). Get at least 256 mb of ram in each box rendering is memory intensive.
If noise is a concern, build a seperate room or building even.. cause the more computers you have the more it's going to start sounding like a server room and less like a place hospitible to human life.
Take a look at learning Blender instead (or in addition to) Maya. Learning to use special rendering software like BMRT would also be cool. BMRT, which is free for non-commercial use, churns out very good images but is very slow. Renderman is the most popular and the most expensive. It's fast too but there are some things it can't do such as global illumination, which BMRT can do. (I'm not sure about the new release of Renderman, though.) There's also Entropy, a fast BMRT, but not free.
All software I've mentioned works for Linux.
For more resources, check out these links:
http://www.blender3d.com
http://www.aliaswavef
http://www.linux.org/apps/all/Graphics/3
http://www.linuxmovies.org
By the way it took me 5 minutes to find this on google and I know jack shit about animation... my only limited experience was fooling around with good 'ol POV-RAY.
For god sakes! If your'e going to ask a bunch of very smart people questions that demand detailed answers... provide detail on your questions.
It makes me if moderators have any standards at all for ask slashdot items. (Why do I ask when I already know the answer!!)
~foooo
Even if 64 bit binaries are available you will probably get greater performance at a lower cost by using "cheap as chips"(sic) chips in SMP configurations.
Future proofing is another issue however. Many clustering technologies rely upon a common denominator. For instance with OpenMOSIX running on varied hardware, your code must be compiled for the lowest common denominator. So if you have 20 P4's and one P2, you will only be able to run software compiled for P2 on the cluster (at least without errors).
YMMV - It's been a while. :P
Q.
Insert Signature Here
Should ECC be considered?
If you're in charge of building a rendering farm, your company is in trouble. If you haven't even figured out that the RAM MUST be ECC, you shouldn't be even aloud near the farm.
Get paid to code OSS
Have you looked at RenderDrive? The company that my wife works bought one of these recently. The guy that uses it loves it. It does the job much quicker.
It's a general pupose computer that has special hardware that is used to do the rendering. The OS is linux. In order to get it on the newtork you setup a floppy with your config file. There's a plug-in for your system that is used to do the rendering on he computer.
I just built up a workstation on which to run Maya 5, and have been using the hell out of it for a week or two, so here's what I have to say about Maya render stations (which are basically workstations minus the Quadro or FireGL card):
Motherboard: Because Maya's renderer is SMP-enabled, you'll probably want to get dual-processor boxes. I suggest AMD machines based on cost - they're a hell of a lot cheaper than their Intel counterparts. A good motherboard is the Asus A7M266-D (760 MPX chipset) - it supports upto 3.5 GB of RAM, and has been rock-solid stable under Linux for me, but it doesn't have onboard video or networking. A good board with onboard video/LAN/SCSI/etc is the Tyan S2466 dual-Athlon board. Keep in mind, though, that these things suck a LOT of power; a good (think Antec) 400 watt (or better) PS is a MUST, or you're going to fry it. I had a Tyan S2460 (2466 minus SCSI, NIC, and onboard video) that fried an off-brand 400 watt power supply because it was sucking so much juice. Don't worry about the specs of the onboard video, because Maya's batch renderer doesn't even bother setting up an OpenGL context; it's completely software rendering from the command line.
Processors: Currently, the Athlon MP 2600+ is at a good price/performance point (approx $150 ea, and the next one up is $200 ea). I'd load every box with a pair of those. If you're looking to save some money and don't mind furiously voiding the warranty on each and every CPU you buy (like me), you can grab some conductive paint, a paintbrush, and a bit of tape, and convert Athlon XP processors into MP-capable processors simply by connecting one of the bridges on the top of the CPU. I did that to a pair of XP2000+ processors, and it worked great; they're still churning away together just fine after almost a year. The price difference is about $70 - if it's worth the savings to you, go ahead and try it. If the chip won't run in SMP mode (rare), you can always stick it into a cheap motherboard and make a desktop workstation for your favorite manager.
Memory: Maya is HEAVY on RAM usage. If you're not planning on having a disk in every machine with plenty of swap space (for those larger scenes), I'd go with at least 1.5 GB (and probably 2 GB) of registered DDR RAM. You should be able to get away with only putting in 1 GB per machine if you have swap space. If you can afford it, spend the extra for ECC memory, because it's nice not to have to worry about memory errors. I did a quick test render (Maya software, 640x480, draft quality) of a 350,000-poly object I'm currently working on, and its average memory usage was right around 600 MB (peak arena size 1186.25 MB, as reported by the renderer). Don't underestimate the amount of RAM that you'll need! My workstation has 1.5 GB of RAM, and it still hits swap.
If you're not booting over the network, I'd throw a fast 18 GB drive in every machine. Make a swap partition of at least 2 GB, and install the OS (I prefer Debian stable, use whatever you want) on the rest of it. You shouldn't have to go overboard on disk; a 7200 RPM IDE drive will be more than adequate.
Networking: Depending on how your scene data is distributed (central fileserver? every node gets a copy beforehand?), I'd go with 100mb switched or full GigE. It basically depends on whether or not you're willing to pay the premium for the faster interconnect. If you put GigE cards in all of the nodes (which doesn't cost much more than putting a good 100mbit card in anyway - I recommend the Netgear GA302 and GA604 gigabit cards), you can always replace the switch if you find things are too slow.
Power and cooling - you'll need plenty of both. One Athlon puts of quite a bit of heat, but pack two into a small box and put 20 of them in a rack (assuming 2U), and you're talking about serious overheating possibilities. You're going to need one hell of an air-conditioning system. If you've already got a datacenter set up, then y
Every cloud has a silver lining (except for the mushroom shaped ones, which have a lining of Iridium & Strontium 90)
I am a render farmer... and this is what I have gathered.
If you are doing single frames, get good systems.
If you are doing animation, get many systems.
You need a scheduler or batch queueing system.
I suggest building one off of LSF like I have in the past and it is what I use now -or- get a nice prebuilt one like what pipelinefx has. This is extremely important. Without a good job scheduler, your cpu utilization will be less than optimal and it will cause you to get more machines than needed.
I have worked both with inhouse and 3rd party renderers. Maya doesn't do that bad of job, and the license for rendering is free. Big plus.
Inhouse is great too, but if you don't have a good sized company, you probably won't have one and probably wouldn't be asking this question.
As for the machine. Most products are only speced for RedHat and for only certain releases. This is important when suport becomes an issue, and it will, when things don't work for a shot. I suggest a dual proc Pentium 4. I do not know if AMD's chips have been blessed by the vendors or not. I know when last I looked it was being investigated. Go for a dual system, even if you don't have a mutli-threaded renderer. Also max out the system ram at 4gb.
As for hardware vendors, I have used VA-Linux, HP, Compaq and SGI linux boxes as well as white-boxed. Hardware support is a big deal when you are a small shop. Go with a name brand unless you want to buy a ton of hot spares.
Storage is a big deal too. You obviously need a place to put the images, but you also need a server beefy enough to serve up a couple hundred mb to several boxes at the same time. When you get 500+ boxes, it gets to be an even bigger deal.
Other administration software i found usefull...
SystemImager and gsh ( a distributed shell ) they allowed me to manage the 500 boxes as well as develop code and debug renders... I can not imagine what it would have been like without it.
-Tim
-I just work here... how am I supposed to know?
Just wanted to drop off a tip that has boosted my productivity tremendously. Multi-Pass rendering. My company got me a copy of After Effects, Adobe's compositing package. With it, I learned how to break up a scene into smaller elements and put them back together in After Effects. This has allowed me to do all kinds of things to save on rendering time. For example, I recently rendered an animation of a machine that has a fast moving piece on it. Motion blur is an expensive feature of a scene to render. However, it can be a waste of render time if only one element of the scene really really needs it. I was able to render the static elements of the scene sans motion blur, and render only the moving bits with the motion blur, thus saving a great deal of render time that could be dedicated to other things.
I just wanted to throw this bit of advice. If your animators don't have a compositing package, it would be a worthwhile expenditure, even if it costs a machine or two from the render farm. I can't speak for other compositing packages, but I can tell you that After Effects is a damn cool, useful app for rendering. Me personally, I'd rather have 1 machine in my render farm with After Effects, than 2 machines in it without AE. If that gives you an idea.
"Derp de derp."
It's essentially a buffer that boosts the clock signal so that the edges of the clock appear more sharply defined to the memory (the clock gets weak when it's directly driving a bunch of memory modules). The buffer tries to isolate the motherboard chipset from the pile of chips on a large DIMM, which is why lots of motherboards require registered DIMMs in order to use lots of memory. That buffer also delays data transfer by one clock cycle.