Build Your Own Render Farm

Another installment of Tom's Hardware's how-to articles has a look at what it might take to build your own render farm. The article looks at everything from top-to-bottom roll-your-owns to buying things pre-built and the pricing insanity that goes along with it. "If you are working as a freelance artist in the above-mentioned media, toying with the idea, or doing so as a hobbyist, then building even a small farm will greatly increase your productivity compared to working on a single workstation. Studios can even use this piece as a reference for building new render farms, as we're going to address scaling, power, and cooling issues. If you're looking at buying a new machine and are thinking of spending big bucks to get a bleeding-edge system, you might want to step back and consider whether it would be more effective to buy the latest and greatest workstation or to spend less by investing in a few additional systems to be used as dedicated render nodes."

Comments

Thanks for this

[TheModelEskimo]

The article touches on general bits of info that might have been time consuming to find. I live in a small town where commercials for clients like the local chamber of commerce are often put together in iMovie, and delivered in a rush. Recently I was approached by a local art director and was asked about moving from 3D stills (which I do occasionally) to 3D animation to be composited into commercial work (probably for bigger clients than the chamber...). I've determined that I can afford about 2-3 minutes of render time per frame before deadlines really start to get pushed out. So rendering infrastructure is very important.

My studio is unique in that I work with open source software, Blender, Lux, etc. And my clients dig it because many of them are into sustainability and see my philosophy as being similar to theirs. I've looked at outsourcing the animation projects to commercial renderfarms, but when you start to "Better Know a Linux Network," you move beyond "get it done" and start to take interest in your own little LAN. Next to my video compositing and 3D graphics books I have a big ol' fat Pro Linux System Administration book, and it's handy, and I like it that way.

The article points out that I can save $140 per node by not needing to buy Windows XP Pro 64 bit edition. This is actually great for me since I typically use the money I save on software to buy more hardware.

BTW, what's up with Slashdot javascript? I'm going to have to build a freaking /. renderfarm pretty soon, and I'll be sending my receipts to CmdrTaco.

Re:Thanks for this

[dr00g911]

All fair points, but I must say that the Mental Ray workflow that's so prevalent among pro-sumer/small studio CG (now that Autodesk owns most everything and bundles MR) is terribly hard on memory usage, displacement or no, 32-bit float or no, physically accurate shading/lighting or no. Renderman is far far more efficient, however due to the licensing costs, not many of the little shops are using it.

The article suggests buying a crapload of boxes with 4GB RAM mainboards, and my argument is that if you find yourself in need of building a render farm of more than a box or two, you're doing yourself a huge disservice by following that advice.

I can tell you first hand that it's a nightmare best avoided to assemble a bunch of assets into layout only to find that you're throwing memory exceptions right & left on deadline.

I stand by that advice: if you're actually to the point of investing a fair bit of cash for some render boxes, spend just a couple more bucks on the mainboard and RAM so you don't build yourself into a corner. I'm talking about a $50 difference (including ram and mainboard) per machine to safeguard against blowing deadlines.

Sure, there are lots and lots of workarounds for memory/detail/physics/computation issues, but on deadline you don't always have the luxury of using those cheats... and sometimes you've just got to brute-force your way through a sequence. Dynamics and rendering in Maya with Shave & a Haircut come to mind. Incredibly powerful, but single-threaded. You'd be better off rendering hair and contact shadow passes on a single P4 than an i7 unless you're using Renderman in this case.

I think you misunderstood me on the crap GPU issue: there are some apps that literally won't launch at all without a certain level of hardware acceleration under Windows. Most pro-level apps have a CLI-only render interface that's commonly invoked by farm management software -- but not all do. The point I was making is: if you're buying a ton of anything, make sure it's going to do what you want it to do ahead of time. It would suck to get a bunch of motherboards in 1U cases that can't even take an AGP/PCI-E video card, and that was preventing you from using the software that you built the farm for.

The main message was really: these guys give you a low-end, once-size-fits-all recipe for building a 16U farm, basically, and at that level of game, I think their advice is pretty poor hardware-wise. You'd be insane to fork over the cash for that much kit and stick 4GB mainboards in there. Imagine someone who had a need for that kind of horsepower but were limiting themselves to the low end of the capability spectrum on such a major purchase when the price difference in the grand scheme of things to future-proof is so small. Not that there's such a thing as future-proofing, but if you're trying to render anything using mental ray with a 4GB system, I can guarantee you that you'll hit a memory wall after playing with ZBrush or mia* physically accurate materials for a few weeks. Or real global illumination etc etc. Particularly under Win32. Sure there are cheats for everything, but they take time too, and sometimes you just need to hit render and know that 1200 frames will be done by Monday AM without spending a huge amount of time tuning cheats.

If you're a small shop, in the vast majority of cases an Autodesk product (or XSI) with Mental Ray bundled, and it's an engine that is not at all comfortable with a 4GB RAM limit.

Part Time Render Farms are cool

[mdarksbane]

I really loved the system they have set up at ACCAD at Ohio State. They had some clustering software running on all of the workstations that could take it over when it wasn't in use. So you had a very nice computer lab and a render farm all rolled into one. And as a user you could set how much you wanted to share while you were working - so if you were just web browsing, the second core could be churning away on someone's render, but if you were using Maya yourself you could have it all to yourself.

I really wish I remember what the software was, and I'm sure this is a common arrangement at these sorts of facilities, but I remember being impressed at the execution of it.

Playstation 2?

[Uncle Ira]

PS2s are cheap now, and I know they've had linux running on them for some time. Has anyone managed to get something like ClusterKnoppix running on PS2 hardware? A renderfarm of slim PS2s sitting on a bookshelf would be kind of neat looking.

Separate machines are the way to go

[Anonymous Coward]

I did this two years ago with four cheap Dell Inspirons ($299 each, with free shipping). They're thin, easy to stack, and consume less power combined than my desktop. No discrete graphics, smallest possible HDD; all they need is processors (dual-core) and RAM. I run a stripped-down Ubuntu on them, and use some Python scripts to distribute Blender render jobs to them over the network, assembling the final frames on a file server.

Separate machines make an enormous difference. Even though rendering is relatively amenable to parallelization, a quad core machine isn't nearly as fast as two dual-core machines with the same specs. Even today, you would have to spend an awful lot of money to get a single machine that renders animations as fast as my two-year-old cluster of four.

I could even have built my own machines, and saved a few tens of dollars per machine, but the price was already pretty reasonable.

Re:A classic quote

[Anonymous Coward]

You really haven't looked into the 3D animation industry yet have you?

Here are the main competitors out there for 3D suites:

Softimage XSI - Windows, UNIX
Maya - Windows, UNIX
3DS Max - Windows only, but who cares?
Lightwave - Windows, Unix

Even with 3DS Max being windows only, all of the renderers you want to use with it have native UNIX versions too. Do you want to know why the 3D industry seem to like UNIX so much? Shear speed:

http://linux.slashdot.org/article.pl?sid=05/07/27/1551250&tid=126
http://www.linuxjournal.com/article/6011
http://www.linuxjournal.com/article/4803

I think I've got the three main players there for you. Wow, am I the only person to comment on this topic who is a 3D artist?

Re:4 Gig of ram is the max for 2 procs? WTF?

[BitZtream]

Really? Tell that to all the apps that are PAE aware, MSSQL server for instance.

Its the same as using the old segmented memory model from a practical perspective, although the OSes today use a completely different API for accessing the other memory.

http://en.wikipedia.org/wiki/Physical_Address_Extension

DrDgaf

[Theodore]

Didn't read, don't give a fuck.
Building your own cluster can be done by any retard.

I've been looking into building one for myself, mainly for Blender and LuxRender.
Now, if there were CUDA/OpenCL versions for the above programs, the Zotac atom/nvidia-ion boards might be nice, expensive, but nice and low powered (or add PCI geforce 9500's, which would also work with my following idea (why the fuck won't they put a PCI-E/16 on these boards?))...
I've been looking into mini-itx mobos (off of Newegg, that mainly shows me zotac geforce 7 series),
and intel Wolfdale procs that the reviews say overclock to al least around 3.5G...
Add 4G of ram (or whatever the board will take), a gigabit switch, set up PXE or a command line only linux distro off of flash,,,
DING!!! FRIES ARE DOEN!!!

Building your own render farm is easy and cheap if you have half a clue as to what you're doing.
Oh,,, knowing where your circuit breakers are when setting it up will help too.