Slashdot Mirror
Building a Render Farm?
Dark Bard asks: "What is the best configuration for a rendering machine. Given the variety of chips and components what are the best options balancing cost verses speed. I've been running Lightwave and plan to shift soon to Maya. AMD chips seem to be rated as a bit faster for rendering and they cost less, as well. Given the number of types of RAM available, what are the advantages of each verses cost? Should ECC be considered? What about motherboards? Integrated video would be ideal but it would have to be adequate to run the software. Is there any advantage to running the new 64 bit processors? Should you consider dual chips? What about operating system? Lightwave won't run on Linux but Maya will. How well do the major operating systems compare when used for this purpose?"
Oh, wait. Never mind.
If Maya came in an amd64 native linux binary,
it would kick some serious booty, but assuming
that you need your pipeline filling up NOW
rather than later, go for 4-way Athlon MPs
with all the ECC DDR you can cram into them.
More than 4-way, and you're paying premium
prices for the chipsets. Less than 4-way,
and you're wasting cycles that could be
amortized over wait states. ECC is de rigeur
just because you don't want to be chasing
down defective dimms all the live long day.
Similarly, I can't see any reason for paying
the 10% performance tax of Windows 2000 Pro
(the best of the Windows lot for SMP use)
or suffering its 2-way limitations, unless
theres some Lightwave feature that is
critical to your operation.
-I like my women like I like my tea: green-
At times like this it's hard to resist kharma whoring... so I won't.
r ont.comD _Modellin g.html
Dear Slashdot,
Please do my research for me. I can't possibly use google and look up the 309830983 gazillion studies on cluster/render farm configurations.
I won't/can't give you any specific information related to budget, my personal experience with hardware, the team to be working on the project (it could be just me... or 500 people) and the importance of data integrity (if I lose a week of work it might end my life... or I might not care).
-clueless reader
Dear Clueless,
Well, if your budget is a billion dollars why not just buy Pixar? They have the whole render farm thing figured out.
If your budget is any thing less than that try looking for open source, GNU, freeware, shareware, free for non-commercial use renderfarmy stuff and run it on AMD based Linux boxes.
Hard drives aren't so important on the individual machines. Get a decent raid on a server machine that will support the number of computers in the render farm. Use 100-1000 mbit ethernet (unless you want to spend a fortune). Get at least 256 mb of ram in each box rendering is memory intensive.
If noise is a concern, build a seperate room or building even.. cause the more computers you have the more it's going to start sounding like a server room and less like a place hospitible to human life.
Take a look at learning Blender instead (or in addition to) Maya. Learning to use special rendering software like BMRT would also be cool. BMRT, which is free for non-commercial use, churns out very good images but is very slow. Renderman is the most popular and the most expensive. It's fast too but there are some things it can't do such as global illumination, which BMRT can do. (I'm not sure about the new release of Renderman, though.) There's also Entropy, a fast BMRT, but not free.
All software I've mentioned works for Linux.
For more resources, check out these links:
http://www.blender3d.com
http://www.aliaswavef
http://www.linux.org/apps/all/Graphics/3
http://www.linuxmovies.org
By the way it took me 5 minutes to find this on google and I know jack shit about animation... my only limited experience was fooling around with good 'ol POV-RAY.
For god sakes! If your'e going to ask a bunch of very smart people questions that demand detailed answers... provide detail on your questions.
It makes me if moderators have any standards at all for ask slashdot items. (Why do I ask when I already know the answer!!)
~foooo
Even if 64 bit binaries are available you will probably get greater performance at a lower cost by using "cheap as chips"(sic) chips in SMP configurations.
Future proofing is another issue however. Many clustering technologies rely upon a common denominator. For instance with OpenMOSIX running on varied hardware, your code must be compiled for the lowest common denominator. So if you have 20 P4's and one P2, you will only be able to run software compiled for P2 on the cluster (at least without errors).
YMMV - It's been a while. :P
Q.
Insert Signature Here
Should ECC be considered?
If you're in charge of building a rendering farm, your company is in trouble. If you haven't even figured out that the RAM MUST be ECC, you shouldn't be even aloud near the farm.
Get paid to code OSS
"Rendering is adding light to an image that is going to be displayed for a small part of one second."
Rendering is math. Think error accumulation.
Have you looked at RenderDrive? The company that my wife works bought one of these recently. The guy that uses it loves it. It does the job much quicker.
It's a general pupose computer that has special hardware that is used to do the rendering. The OS is linux. In order to get it on the newtork you setup a floppy with your config file. There's a plug-in for your system that is used to do the rendering on he computer.
Go and use full scale professional tools, like Houdini (http://www.sidefx.com/), that offer free linux versions as well (small watermark on the output), and export to Renderman and use existing Renderman cluster stuff (BlueMoon Render Tools IIRC).
ICQ# : 30269588
"I used to be an idealist, but I got mugged by reality."
I just built up a workstation on which to run Maya 5, and have been using the hell out of it for a week or two, so here's what I have to say about Maya render stations (which are basically workstations minus the Quadro or FireGL card):
Motherboard: Because Maya's renderer is SMP-enabled, you'll probably want to get dual-processor boxes. I suggest AMD machines based on cost - they're a hell of a lot cheaper than their Intel counterparts. A good motherboard is the Asus A7M266-D (760 MPX chipset) - it supports upto 3.5 GB of RAM, and has been rock-solid stable under Linux for me, but it doesn't have onboard video or networking. A good board with onboard video/LAN/SCSI/etc is the Tyan S2466 dual-Athlon board. Keep in mind, though, that these things suck a LOT of power; a good (think Antec) 400 watt (or better) PS is a MUST, or you're going to fry it. I had a Tyan S2460 (2466 minus SCSI, NIC, and onboard video) that fried an off-brand 400 watt power supply because it was sucking so much juice. Don't worry about the specs of the onboard video, because Maya's batch renderer doesn't even bother setting up an OpenGL context; it's completely software rendering from the command line.
Processors: Currently, the Athlon MP 2600+ is at a good price/performance point (approx $150 ea, and the next one up is $200 ea). I'd load every box with a pair of those. If you're looking to save some money and don't mind furiously voiding the warranty on each and every CPU you buy (like me), you can grab some conductive paint, a paintbrush, and a bit of tape, and convert Athlon XP processors into MP-capable processors simply by connecting one of the bridges on the top of the CPU. I did that to a pair of XP2000+ processors, and it worked great; they're still churning away together just fine after almost a year. The price difference is about $70 - if it's worth the savings to you, go ahead and try it. If the chip won't run in SMP mode (rare), you can always stick it into a cheap motherboard and make a desktop workstation for your favorite manager.
Memory: Maya is HEAVY on RAM usage. If you're not planning on having a disk in every machine with plenty of swap space (for those larger scenes), I'd go with at least 1.5 GB (and probably 2 GB) of registered DDR RAM. You should be able to get away with only putting in 1 GB per machine if you have swap space. If you can afford it, spend the extra for ECC memory, because it's nice not to have to worry about memory errors. I did a quick test render (Maya software, 640x480, draft quality) of a 350,000-poly object I'm currently working on, and its average memory usage was right around 600 MB (peak arena size 1186.25 MB, as reported by the renderer). Don't underestimate the amount of RAM that you'll need! My workstation has 1.5 GB of RAM, and it still hits swap.
If you're not booting over the network, I'd throw a fast 18 GB drive in every machine. Make a swap partition of at least 2 GB, and install the OS (I prefer Debian stable, use whatever you want) on the rest of it. You shouldn't have to go overboard on disk; a 7200 RPM IDE drive will be more than adequate.
Networking: Depending on how your scene data is distributed (central fileserver? every node gets a copy beforehand?), I'd go with 100mb switched or full GigE. It basically depends on whether or not you're willing to pay the premium for the faster interconnect. If you put GigE cards in all of the nodes (which doesn't cost much more than putting a good 100mbit card in anyway - I recommend the Netgear GA302 and GA604 gigabit cards), you can always replace the switch if you find things are too slow.
Power and cooling - you'll need plenty of both. One Athlon puts of quite a bit of heat, but pack two into a small box and put 20 of them in a rack (assuming 2U), and you're talking about serious overheating possibilities. You're going to need one hell of an air-conditioning system. If you've already got a datacenter set up, then y
Every cloud has a silver lining (except for the mushroom shaped ones, which have a lining of Iridium & Strontium 90)
I am a render farmer... and this is what I have gathered.
If you are doing single frames, get good systems.
If you are doing animation, get many systems.
You need a scheduler or batch queueing system.
I suggest building one off of LSF like I have in the past and it is what I use now -or- get a nice prebuilt one like what pipelinefx has. This is extremely important. Without a good job scheduler, your cpu utilization will be less than optimal and it will cause you to get more machines than needed.
I have worked both with inhouse and 3rd party renderers. Maya doesn't do that bad of job, and the license for rendering is free. Big plus.
Inhouse is great too, but if you don't have a good sized company, you probably won't have one and probably wouldn't be asking this question.
As for the machine. Most products are only speced for RedHat and for only certain releases. This is important when suport becomes an issue, and it will, when things don't work for a shot. I suggest a dual proc Pentium 4. I do not know if AMD's chips have been blessed by the vendors or not. I know when last I looked it was being investigated. Go for a dual system, even if you don't have a mutli-threaded renderer. Also max out the system ram at 4gb.
As for hardware vendors, I have used VA-Linux, HP, Compaq and SGI linux boxes as well as white-boxed. Hardware support is a big deal when you are a small shop. Go with a name brand unless you want to buy a ton of hot spares.
Storage is a big deal too. You obviously need a place to put the images, but you also need a server beefy enough to serve up a couple hundred mb to several boxes at the same time. When you get 500+ boxes, it gets to be an even bigger deal.
Other administration software i found usefull...
SystemImager and gsh ( a distributed shell ) they allowed me to manage the 500 boxes as well as develop code and debug renders... I can not imagine what it would have been like without it.
-Tim
-I just work here... how am I supposed to know?
First, if you could buy a 4 way Athlon MP it would not be as fast as a 4 way amd64 based system becauge the Athlon MPs do not have nearly as good a frontside bus as the amd64. Just get the amd64 system and run it in 32 bit mode. It will still be faster, for most things, than any other x86 system on the planet.
Maya render farms are notoriously hard to manage on Intel/AMD/windoze/linux hardware. You should check out Apple's XServes and their new QMaster software, it can manage huge render queues for Maya and Shake. Nobody uses just Maya on a render farm, you need compositing software too, like Shake. You can't beat Qmastered Xserves for a render farm. And Shake comes with unlimited licenses for OS X (but not for linux). The cost savings in licenses alone will astonish you. Even their XServe RAID products are less costly than other competitive RAIDs of similar spec.
And I can report from personal experience that Maya rendering speed on dualG4s is top rate. We should see dualG5 Xserves in January, nothing will come close to that config in speed.
Just last week I set up my first Lightwave render farm. Unfortunately all the systems in the rendering farm were over 5 years old. The main problem I ran into, beyond our artist having placed all his files outside his content directory, and the farm being slower than some of our newer PC's, was that the systems were very short on memory (96 to 128 mb each), so we'll be upgrading them soon.
For a rendering cluster, I'd look for the systems that provide the best price / floating point performance ratio, and max out the ram. A brief pass through Tom's Hardware seems to suggest that Intel gets better lightwave performance, though I can't say for certain.
And NewTek released "ScreamerNet for Linux" as an update to 7.5 sometime in the past year, but I haven't tried it and obviously not every plugin will work. If you're stuck with an Microsoft OS, I think screamernet will run on any version of Windows.
Be aware that I've never used lightwave so I can't possibly be an expert on it.
This might be interesting to some readers: I just read that DrQueue, free render queue management software, now has more integration with Blender. It also seems to have good integration with Maya, as well.
Any other good batch process managers out there, useful for rendering, compiling, or other heavy work? (I've heard of Condor which looks great except for a few serious restrictions on I/O)
reed
VOS/Interreality project: www.interreality.org
The best price/power processor is the G5. Right now it is only available in a tower with non-ECC RAM. The good part is the RAM they use is actually very good quality and from the testing we've done there has been very few errors. Hopefully we'll see G5 xserves show up at macworld in january. The best part is apple has one of the best storage solutions available with the xserve raid. About middle of the pack as far as cost, but with top-end performance and integrity. There is no need to bother trying linux on a G5 box, OSX gets great rendering performance for both shake and maya. BTW, qmaster works on standard G5 towers, not just xserves.
Ohhhh, look at all the pretty shiny things.
Just wanted to drop off a tip that has boosted my productivity tremendously. Multi-Pass rendering. My company got me a copy of After Effects, Adobe's compositing package. With it, I learned how to break up a scene into smaller elements and put them back together in After Effects. This has allowed me to do all kinds of things to save on rendering time. For example, I recently rendered an animation of a machine that has a fast moving piece on it. Motion blur is an expensive feature of a scene to render. However, it can be a waste of render time if only one element of the scene really really needs it. I was able to render the static elements of the scene sans motion blur, and render only the moving bits with the motion blur, thus saving a great deal of render time that could be dedicated to other things.
I just wanted to throw this bit of advice. If your animators don't have a compositing package, it would be a worthwhile expenditure, even if it costs a machine or two from the render farm. I can't speak for other compositing packages, but I can tell you that After Effects is a damn cool, useful app for rendering. Me personally, I'd rather have 1 machine in my render farm with After Effects, than 2 machines in it without AE. If that gives you an idea.
"Derp de derp."
it makes no sense to pay top dollar for state of the art hardware and then spend a year learning to use it. go get yourself 3 or 4 cheap pc's (i've seen piii 700mhz boxes for around $100) and experiment with them.
On that note, what exactly is registered ram? Yes, I buy it for my dual athlon system because the specs on the mobo tell me to, but I'm wondering exactly what it means when it says "registered".
Don't blame me, I voted for Kodos
It's essentially a buffer that boosts the clock signal so that the edges of the clock appear more sharply defined to the memory (the clock gets weak when it's directly driving a bunch of memory modules). The buffer tries to isolate the motherboard chipset from the pile of chips on a large DIMM, which is why lots of motherboards require registered DIMMs in order to use lots of memory. That buffer also delays data transfer by one clock cycle.
you've got to pin down exactly what you're trying to do. for instance, rendering is embarassingly parallel, so if you have lots of frames, don't even think about parallelism within a frame. rendering is not particularly memory-intensive, so you might well get away without ECC, since the expected failure rate for non-ECC memory depends primarily on how much you have, and how hard you use it. if you're talking more than a handful of boxes, you definitely want to make them diskless/netboot. going SMP is probably a bad choice since it drives up the price (particularly if you're going for lean, diskless nodes.)
gigabit networking is almost certainly the right choice, since it's practically free. if you wind up using a lot of IO (complex models, simple frames, big images), then gigabit/diskless might end up being inadequate. but adding a disk per node will drastically hurt reliability for any large cluster! faster networks (myri/quadrics/IB) are a huge marginal cost unless your nodes are fairly high-end.
don't worry about things like LSF or Mosix - it's trivial enough to set up a farm with static jobs for rendering. LSF/PBS/etc are vast overkill, and Mosix is pointless since you don't care about migration and dynamic load-bal is pointless.
If you want expert advice from experienced people, go lurk in the appropriate mailing lists or Usenet groups. Once you get up to speed, you can ask specific questions and get help.
If you "Ask Slashdot" a hopelessly vague question, you deserve a little gentle sarcasm.
BTW, sarcastic guy Googled himself a rough plan of how he'd do it, and is getting lots of corrections and suggestions. Why? Because he's got some actual details to talk about!!
We've done quite a bit of experimenting with renderfarm hardware. Don't know quite as much about the software aspects of it, as I'm heavily invested in 3dsMax, knowledgewise, so that's what we've been sticking with. The best solution we found as far as bang for buck was getting whatever AMD chip was fastest at the $100 or less price point, and adding motherboard, powersupply, and memory, 512 meg per unit. no cases, we just have all the motherboards in a custom built case (an old server rack that we gutted and tricked out with extra fans..). No hard drives, everything boots of the network. No video, only the actual workstation needs video. All the mother boards have integrated 100 ethernet, so no network cards. No floppies, mice, keyboards, sound, speakers, all that stuff is pretty much not needed for renderfarm nodes. The $100 price point for the AMD chips is not the result of extensive research, it just seems that after buying them for the last few years that seems to be the best point as far as horsepower for dollars. We're still running nt4, as those are the licenses we had.. tried XPpro on a couple computers, and it was way slow. Max has network rendering stuff that comes with that has worked great for us, I don't know about Maya or the other packages. We're going to stick with this setup for now. If a huge project comes down the pipe, we plan on doing some research into an all linux setup, with some kind of open source rendering software that will integrate well with a decent 3d package. Last but not least, consider NOT buying a render farm. When we had some huge projects and a tight deadline, we made deals with some of our security and consulting customers, to use their entire network of PC's as a render farm. 700 average desktops at a state office end up doing a pretty good job as a render farm.
No, a render farm, is were bacon grease is made(1).
(1) You don't want to know were baby oil comes from.
why lots of motherboards require registered DIMMs in order to use lots of memory
I've 512MB registered RAM in my dual Athlon, yet I got a blue screen today while typing in Notepad.
Maybe I should clean out the sliced-up mouse guts from the fan or stop using W2K.
Windows has noticed a mouse movement and needs to be restarted. Yes / No.