Renderfarm Setup Tips?

CarlosOlivaG4 asks: "We're in the process of acquiring and setting up a renderfarm, and I'm hoping the Slashdot community might light us up a little here. We'll use 6 to 8 nodes first, but would like to be able to expand it in the future." There was an earlier version of this question, but it dealt more with the hardware of the farm's nodes, rather than the network and software infrastructure on which these nodes would be based.

"In the hardware side, we still haven't made a choice between using AMD's Opteron or Apple's Xserve G5 (they have some very nice and price convenient cluster nodes which seem to be ideal for this kind of job), with Linux. As for the networking between them, is Gigaethernet enough or should we be going for Fiber? The software used to manage the render queues is another important point as well: I've been looking into Rush, and even though it's a commercial package, it works on all of the platforms we currently use (W2k/XP, Irix, OS X and Linux). But then there is also Dr. Queue, which is open source and is supported on at least the *NIX members of the aforementioned OS's. Other options include RenderPal and Pixar's RenderMan, but I would prefer an F/OSS alternative. Finally, it's worth noting that we'll be using the renderfarm for Maya and Adobe AfterEffects."

Comments

Cinelerra

[selfabuse]

You might want to check out Cinelerra. It has pretty good support for renderfarms. I built one out of scrap 300mhz machines, and it only took a weekend.

Re:Cinelerra

[selfabuse]

And here is the link I forgot.

Re:Cinelerra

[dhartman]

I've done a bit of work with Cinelerra and have been very pleased with the results. With three Athlon XP 1900 ish machines with no more than 512MB ram I was able to render at 30fps with full DVD resolution (720x480) output.

The render nodes only run cli tools and do not require local storage. A bootable Knoppix cd could be made to create a temporary render node. Imagine using 10 computers in your office to render video in the off hours and reboot into Windoze in the morning with the user not knowing.

Simple

[hanssprudel]

(a) Wait for the next new processor technology to hit Slashdot.

(b) Build a Beowulf cluster of those.

node deployment: g4u!

[hubertf]

Check out g4u for deploying your render machines - it's a image based disk cloning tool that uses DHCP and FTP and which doesn't care what you run on your clients. (g4u itself is based on NetBSD, but that doesn't matter for the application).

I've used g4u to setup a ~50 node video rendering cluster, see my webpage on the Regensburg Marathon Cluster.

Enjoy!

- Hubert

I'm a Machead, but...

[Jonas the Bold]

Don't use macs. First off, you have less choice in renderers, and second, the hardware is more expensive. Rendering is grunt work. Buy cheap systems that you can upgrade more often, and run linux or something.

Macs are very nice hardware, but you really don't need that for rendering. For workstations they make sense, but for rendering you really want to have a lot of fast computers rather than nice computers.

Re:I'm a Machead, but...

[cbreaker]

Hehe, but to me, fast IS nice. =)

I'm not a Machead, I'm an x86head. Always will be.

Re:I'm a Machead, but...

[Chanc_Gorkon]

Um, why? Others have used Macs and this is a ideal thing for a Xserve and it obvously was not a problem for the person in this question. Both platforms he has presented are both new and are both goign to run him a lot of money. Plus Apple has a Cluster Node config that will work well for him (only one hard disk....in this case you'd use some other storage or a Xserve Raid. In fact, I bet Pixar may use G5's soon if they don't have them soon. Xserves are fast and if you make yuor cluster out of them, you will get your renders much quicker.

Re:I'm a Machead, but...

[mcdesign]

But if you are using Shake the OS X version is $2999.00 or $2000 dollars cheaper than the Linux version. The OS X version also comes with unlimited render only nodes for free. Each Linux render node costs $1499.

So for say 10 computers:
Cluster node version of Xserver 10 @ $2,999.00 = $29,990*
Shake 1 @ $2,999 = $2,999
Total = $ 32 989

Now the Linux version will cost:
Shake 1 @ $4,999 = $4,999
Render nodes 9 @ = $13 491
Total costs software = $18 490

This leaves you with $14 499 to buy 10 x86 boxes or $1449.90 each. Those G5's don't seem so expensive after all.

* Yes I know it will need more RAM but so will the Linux boxes.

Re:I'm a Machead, but...

[MidnightBrewer]

Depends on which renderer you're using. Except for LightWave, RenderMan, Mental Ray, Animation Master, Cinema 4D, and even Blender, to name a few, it's true, you'll have problems getting support for OSX...oh, wait, that's quite a few right there, isn't it. Nevermind.

The question with building a headless Intel render farm is going to be licensing. Great, you can build cheap machines, but are you going to have to buy extra licenses just because you chose to go cross-platform? It's not just the main render engine you have to worry about, but also the 3rd-party shaders that you inevitably buy for doing specific things the default package doesn't cover.

If you do in-house programming instead, are you going to have your programmers write an Intel version of the shader as well as a Mac version? Good luck with that! Talk about unnecessary headaches. I think duplicate effort counts as an unjustifiable waste of money.

Finally, it's important to make sure that you don't run into limitations in the software, specifically procedural shaders, that cause the images to render differently on different platforms. It's annoying to get your frames back and they don't look just like the tests; sometimes, that can render the results unacceptable.

Why run Linux?

[puppetluva]

Why would you run Linux on the Apple boxes? Wouldn't OSX be just as good?

Re:Why run Linux?

[Anonymous Coward]

Why would you run Linux on the Apple boxes?

Because of its unnecessary flashiness? OS X is notoriously bloated. For the command line junkies among us, Linux fits the bill.

Re:Why run Linux?

[GFLPraxis]

Actually, I find Mac OS X to run faster on my 1 GHz G4 than Mandrake Linux 10.0 on my 2.6 ghz Pentium 4. That flashiness doesn't slow it down that much, you see...the Mac makes use of the 3d graphics card (which usually does almost nothing when you're not playing a 3d game) to control all the window effects. As a result, the processor is hardly taxed.

Software?

[m0rph3us0]

What render queue products are supported by your pieces of software? Why don't you try a few of them?
I'm sure a demo can be arranged.
I wouldn't go blindly marching in the direction of FOSS especially in something that is valuable enough to setup a renderfarm for.

Most importantly, find out what the people who will be using the software like and dislike about each package. And what works for them. If it saves you $30 per hour times 5 people software and hardware cost become insignifigant after one work week.

The biggest renderfarm in the world is useless if your people can't use it. Always remember that software is only good in its ability to meet the goals of the organization it supports.

Our experiences

[Thagg]

We had a renderfarm for "The Chronicles of Riddick" of 40 boxes. Each box was a dual-proc Opteron.

We evaluated several render-queue management systems, and decided on Rush. The most persuasive arguments for using Rush were the very good experience we have heard from other users, and the simplicity of extending it to manage a variety of different tasks. I have to add Hammerhead to the list of happy customers. It did everything we could have hoped for. In particular, it was able to handle the inevitable crashing of machines pretty well.

While it's true that Rush is a proprietary, gotta-pay-for-it system; a robust render queue management system pays for itself very quickly in the ability to make your renferfarm productive. Perhaps a render queue manager is overkill when you have just 6 or 8 systems, but once you get up to 30 or 40 it is essential.

Our experience is all under Linux, but if you're going to be running After Effects that means that you're not going to be running Linux -- so there's not too much more I can help you with there. We did find that the dual Opterons worked much more efficiently than dual Xeons in multiprocessor rendering -- don't know about the Xserves, though. We were running mostly Maya, RenderMan, Shake, and our own in-house tools on the farm.

This farm is unfortunately powered down now that Riddick is done -- if you need some dual opterons, let me know at thad@hammerhead.com

Re:Our experiences

[NanoGator]

Welp, your post sparked some debate here at the office. I'm at a small studio making a full length animated movie. One aspect of it we've been chewing on is what to get in terms of render farm down the road. I just had a few questions for you, if you don't mind:

1.) The words 'dual' and 'Opteron' both surprised us. We were kind of under the impression that maybe single proc machines would be better for a render farm. We were really curious why dual was chosen over single. Did the extra cost end up being worth it?

2.) You mentioned that Opteron was more efficient than Xeons. I just had to ask: Was the particular software you were using particularly tuned to Opteron (i.e. 64-bit?) or was the 32-bit side of it just pleasant to work with? Any more insight you can share with me about the use of Opteron would be most helpful.

3.) Did you guys end up buying a bunch of machines from a place like IBM or something, or was it more like "we bought the components and assembled ourselves..?" If it's the former, how'd you like the service?

4.) Any regrets or things you'd do differently next time around?

5.) Why are you getting rid of the machines used for Riddick? Or did I read that wrong?

Appreciated,

NanoG

Re:Our experiences

[Anonymous Coward]

The Maya renderer is not SSE optimized, so the extra fpu in the Opteron and Athlons gives them an edge. The Opterons are about equal to P4s in SSE enabled apps, the Athlons lag behind. Mac hardware is overpriced, especially if you are going to buy more in the future. And find some benchmarks - I don't think many apps use Altivec, but many do use SSE/SSE2.

www.zoorender.com has a limited/single benchmark for Maya and Mental Ray on a variety of hardware. Pay attention to their comments about how the Maya renderer has better throughput on a dual-proc box when you run 2 processes vs 1 process with 2 cpus. Our queue did that by default.

Rendering is all about CPU power and dual procs scale almost linearly. It's also nice to have half the servers to manage and also the flexibility of running 1 or 2 processes on a dual proc box with 2GB+ RAM. But, dual-proc hardware is more expensive - the cheapest option is always consumer whitebox hardware. Also, under windows, there is a ~1.6GB memory limit for a single process. But XP service pack 2 may expand this (for certain hardware? I don't know details.)

Eventually you will have I/O bottlenecks, and this will become your main headache. 6-8 procs should work for normal samba/nfs NAS. I don't know AE, but it may render much faster than 3D stuff - in which case you will probably use attached disk.

I can also vouch for RUSH's internals, I signed a NDA and got to see the code/architecture. It is amazingly scalable. We were considering buying it to replace our homegrown queue that managed 350 procs. Instead, we went out of business.

But, with any queue you buy, there will always be scripting 'glue' and troubleshooting that needs to be done.(usually: can't find the files or I/O bottlenecks on the files, and tuning rendering/raytrace parameters) Bottom line: If I were you, I'd find some AE benchmarks on different hardware, then I'd get dual-proc Opterons (for flexibility and memory bandwidth) with 2GB RAM to start, and run linux - but I think AE will force you to run XP(?)

Re:Our experiences

[Thagg]

Good questions

1.) The words 'dual' and 'Opteron' both surprised us. We were kind of under the impression that maybe single proc machines would be better for a render farm. We were really curious why dual was chosen over single. Did the extra cost end up being worth it?

The computers are relatively cheap -- it's render licenses (especially for RenderMan) that are expensive. With the newest version of RenderMan, Pixar has deigned to let us use the two processors of a dual-processor machine with a single license. This lowers the cost of rendering by about 60%, if the machine rendered twice as fast with dual processors. In fact, for RenderMan, the Opterons were indeed almost twice as fast, where the Xeons were only about 50% faster.

Our other big rendering application was Shake, and it also allowed the use of two processors with one license.


2.) You mentioned that Opteron was more efficient than Xeons. I just had to ask: Was the particular software you were using particularly tuned to Opteron (i.e. 64-bit?) or was the 32-bit side of it just pleasant to work with? Any more insight you can share with me about the use of Opteron would be most helpful.

Yes and no. The Opterons are the first AMD machines to implement the SSE2 instructions, which are heavily used by RenderMan. Also, the HyperChannel communication between processors on the Opteron is light-years beyond the communication between Athlons and Xeons. On the other hand, there is absolutely no advantage in the 64-bitness of the Opterons -- we were running a 32-bit Linux (RedHat 9), and we weren't using more than 4 GB memory on any of the boxes.

3.) Did you guys end up buying a bunch of machines from a place like IBM or something, or was it more like "we bought the components and assembled ourselves..?" If it's the former, how'd you like the service?

We hired a beige-box manufacturer. We specced it out to various places, and PC Mall built them for the best price. If I had to do it over again, I'm not sure that I wouldn't go with IBM -- while they cost a lot more, I expect that they'd build more solid systems.

4.) Any regrets or things you'd do differently next time around?

We bought minitower machines instead of the more trendy, space- and power-efficient 1U or blade machines. We did that so that we could potentially use the new Gelato renderer from NVidia -- that software uses the current NVidia high-performance graphics cards as an external array processor, giving significantly better render performance.

As we didn't end up using Gelato, that was perhaps a mistake. We ended up power and HVAC constrained in the end -- as happens with almost every renderfarm I've heard of.

5.) Why are you getting rid of the machines used for Riddick? Or did I read that wrong?

No, you read it right. Hammerhead is a small company, typically working on just one show at a time. We don't see a use for the machines for another nine months or so, as we begin development of the next project -- and it just isn't right to leave all that compute horespower idle.

Thad Beier
Hammerhead Productions

Re:Our experiences

[forkazoo]

From previous posts, I seem to recall that you use Lightwave, right? Regarding 2), head over to blanos.com, and check out his benchmarks. I just looked at 2p P4 Xeons vs 2p Opterons. Looks like Opterons wipe the floor from the specific scenes I was looking at, but I didn't see any P4 Xeon 2p results for the radiosity test scene. I'd like to see that, as LW is supposed to use SSE pretty heavily in radiosity, so the clock speed of the P4 would be a potential bonus.

The big difference between Pentia and Opterons is that Pentia use a shared front side bus, while the Opties use Hypertransport, whith a memory controller on each CPU, so the CPU's aren't fighting each other to get at data to chew on as much.

Re:Our experiences

[NanoGator]

"From previous posts, I seem to recall that you use Lightwave, right?"

Yes, that is correct.

" 2), head over to blanos.com, and check out his benchmarks."

Good idea!

"but I didn't see any P4 Xeon 2p results for the radiosity test scene"

Heh I just did that this morning. I ran the skull radiosity test with 8 threads. It was on a Dual P4 Xeon 2.4ghz 533 bus and Hyperthreading enabled. 119s. (I'll try to remember to log that at Blanos, time permitting...) That was running LW8, not sure if that make a difference.

"The big difference between Pentia and Opterons is that Pentia use a shared front side bus, while the Opties use Hypertransport, whith a memory controller on each CPU, so the CPU's aren't fighting each other to get at data to chew on as much."

Ugh. I hate trying to balance all this stuff because it's not so clear what exactly LW needs. For example, it takes so long to render, that it's not all that clear that moving 300 megs of textures through the memory bus is a huge bottleneck. (Good thing we got blanos!)

Thanks for the reply

Re:Our experiences

[Thagg]

Are you able to tell us which productions were these machines involved in rendering?

These particular machines were just used for The Chronicles of Riddick. Computer technology advances so fast, has lowered in price so quickly, and movie post-production schedules are so long (six to nine months, typically) that we typically don't use any particular machines for more than a couple of films.

Also, in the interest of understanding how much it costs to set up a significant render farm, how much does this sort of thing cost? Is it all in the PCs, or would the backplane infrastructure cost surprise us a lot?

In fact the dominant cost, at least for us, is not the render boxes themselves. The network is a significant expense, as is the data server system. An even larger expense, though, are the licenses for the rendering software. Top-of-the-line rendering systems like RenderMan (for 3D) and Shake (for 2D) cost thousands of dollars per node. And then there are significant infrastructure costs in just electrical wiring and cooling.

At least in the 10-to-50 server range, I would say that the costs are pretty linear. As you get bigger than that, you can start to see some economies of scale.

At some point, it becomes profitable to start developing in-house software tools instead of buying licenses. Digital Domain's Nuke system was originally developed as a renderer for Flame, for example, so that the expensive Flame machines could be used for the interactive work and the batch rendering could happen on commodity hardware. For Riddick, we developed our own smoke-rendering system rather than use RenderMan, to free up render licenses for other parts of the movie.

I'm afraid that an explict cost-per-node breakdown would get into details that we keep confidential, but this should give you an overview of our situation.

Thad Beier
Hammerhead Productions

p.s we don't do Videos, we make Films.

Re:Our experiences

[dcmeserve]

An even larger expense, though, are the licenses for the rendering software.

What did you think of the freeware options, e.g. Aqsis?

Re:Our experiences

[Thagg]

Well, I should have mentioned that are largest line item, by far, is the animators that drive all of this software. At least on a project of the scale of Riddick, it was best to use software with which our animators would be most productive. For rendering, that means RenderMan. There are many people who have strong experience in writing shaders and doing lighting in RenderMan -- and RenderMan is practically bullet-proof after decades now of work.

We did try a couple of other rendering alternatives. We hoped to be able to use the new Cg shading language to do hardware rendering for some of the shots. We hired an absolute wizard named Hal Bertram to help us with that, but in the end decided that the hardware (at the time) was still not good enough to do final rendering. Interestingly, with the new NV40 chips incorporating Vertex Shader 3.0 and Pixel Shader 3.0 capabilities, almost all of those limitations have been overcome, and we'll be examining hardware rendering anew for our future projects. It seem inevitable to me that in the near future, we will be using hardware rendering for most of our shots -- the quality of what can be rendered with hardware is improving at a breathtaking pace.

We also looked at some other non-free and free (although not open-source) software renderers, some even claiming to be compatible with RenderMan. While they are truly amazing efforts by small teams, we very quickly ran into devastating problems with each of them. I really hate to mention who they are, because they are trying so hard -- but they're not there yet.

One renderer that I'm extremely excited about is Splutterfish's Brazil. It renders some spectacular images mind-bogglingly fast. In particular, it does what has come to be known as "ambient occlusion" so fast that I just can't believe it. Unfortunately, it only runs on the Operating System That Will Not Be Named.

So, in the end, we spent the money for RenderMan to make best use of the really expensive resource, our animators. Having a familiar environment, and one where everything worked every time with no "gotcha's" was worth it. [OK, we did run into some RenderMan surprises...but nothing too serious].

Thad Beier
Hammerhead Productions
thad@hammerhead.com

Go with the G5's - your work is the important item

[Selecter]

Personally I'd go with the G5 Xserve with a few diskless cluster nodes tossed in.

Thats only if you desire maximum ease of use with minimum setup and running hassles. The same ease of use the regular G5's have is built into all their server stuff too. I'm sure the linux dudes will have something to say about that.....

I would take a really hard look at the ready made bio-information cluster they have all setup, and just load yer software as needed and off you go. But that's me. Some people seem to like futzing with computers.....After 20+ years doing that at work, I just wanna do what I wanna do when I wanna do it. Apple makes that easy.

I get paid to deal with headaches, I'm not gonna deal with them at home too.

Oops.

[Mr. Bad Example]

My subconscious desire to get out of the computer field as a career must be surfacing--I read this as "reindeer farm". Then reality set back in and I almost made a lame "LapLANder" joke before tasering myself.

XGRID

[artlu]

Recently, I have been working a lot with Apple's xgrid. We have been linking about 4 G5s/G4s together and getting impressive results. I don't understand your hardware situation, but if you are a Mac guy try it out.

GroupShares.com

Re:XGRID

[Johnny Mnemonic]

If you're considering Apple G5s, either in workstations or in Xserves, take a look at Apple's mailing list for help and resources. Folks there have been working about clustering and Xgrid on the Mac for a while now.

Do it properly

[smharr4]

If you're going to do this, do it properly. Get systems with massive amounts of I/O that will cope with all the data you're trying to throw at them. For this kind of work, you need only buy from one vendor: SGI.

Don't bother with Intel/Linux, with dodgy hardware and the frequently-changing Linux code. Pay the money, get decent hardware with a support contract and a steady, stable, tried and trusted OS.

Apple *may* be an appropriate choice, now that Pixar have ported RenderMan to OS X, but I don't like the idea of my arrays running at 7200 rpm's. Get SGI, get fibre channel, and (possibly) get gigabit ethernet.

It'll all pay off - it won't be cheap, but in the long run, the results will be worth the money and the wait.

Re:Do it properly

[Donny Smith]

>Get systems with massive amounts of I/O that will cope with all the data you're trying to throw at them.

Render nodes get input via simple render scripts, output frames get written to the file server one by one every X seconds as they get rendred. Textures are shared but it's never "massive" and never "thrown at them" (compute nodes).
The I/O loading is concentrated on the file server.

> Don't bother with Intel/Linux, with dodgy hardware and the frequently-changing Linux code.

So HP, IBM and other Intel/Linux servers that rendered all those movies are "dodgy hardware"?

>Get SGI, get fibre channel, and (possibly) get gigabit ethernet.

I don't think 8 nodes can write and read data quickly enough to saturate a gigabit link to NFS server since while frames are being rendered NFS' I/O is very low.
With that number of nodes perhaps they could muddle thru without external storage (maybe even internal SCSI would do). GbE is more important (and cheaper) than FC storage so I'd say GbE is a must.

n1 n2 n3 n4 n5 n6 n7 n8

[gigabit lan]

nfs1
(optionally with direct-attach FC disk array)

pvm is the way

[Goeland86]

I think that what you're looking for is a renderfarm for computer graphics rendering, right? in that case, you should be looking at PVM or OpenMOSIX or even MPI. In either case, since you're going to have more data crunching than actual data transfer, I think that even T100 would be enough. gigabit will be nice, but fiber is not worth it. Drqueue is nice... if you can get it to work, and I didn't. We used pvmpovray for many things, and I think that might be worth a look. pvmpovray exists for gentoo with an ebuild script, which would make the installation and configuration the minimum pain for it. But that option requires a conversion from maya to povray files. Also, I don't know what the pricing is going to be like, but if it were up to me, I'd take the Opterons, because I believe they are faster, although I'm not positive on that, and because I know they're well supported under linux, and again, I think that's a more personal choice to make, but the impression I got from AMD is that you always get the most for your buck. The Opterons also let you find replacement parts or upgrades a little easier than the G5 if you burn a CPU or motherboard. That's just my $.02 worth of advice.

My $0.02..

[midifarm]

I just want to let you know, I'm not a F500 company with jillions of dollars to spend, hence my two cents! I've been running After Effects and FCP using a host of various Macs for offline rendering. It's like a museum of Macs that are able to run OSX and I've been able to do it rather efficiently. I'd love to delve into Maya and perhaps Renderman eventually. I must tell ya that Pixar is completely converting to G5's and OSX for Renderman. I imagine they're getting the family discount, but it says a lot when the world's leading animation studio is going in that direction. I find that I have no problems with the computers. Granted I'd love a fridge rack of XServes for offline rendering, but for the time being my motley cluster is working just fine.

Peace

Re:My $0.02..

[Selecter]

Damn, Motley Cluster is a hell of a name for a band....

Re:My $0.02..

[Thagg]

I could be wrong, but I thought that Pixar was going to stick with Intel-type machines for the renderfarm, although they are apparently moving to OS X boxes for their animator workstations.

I'm porting all of our animation code from Linux to OS X as well -- more or less as an exercise in code portability -- and it's going pretty well. OS X 10.3 is dramatically more standards- (or at least Linux-) compliant than the earlier OS X versions were. Almost every program I have compiles with virtually no changes. The OpenGL and Glut implementations seem to work perfectly. The only surprise is how far behind the times the Mac graphics cards arel, but few of our programs really need to render 5 million polygons a second.

Thad Beier

Light us up

[Reckless Visionary]

I'm hoping the Slashdot community might light us up a little here.

Light us up the bomb!

I had a related question

[RelliK]

... but it was rejected. How do you deal with terabytes of data (50+ TB), all in a single directory tree, all must be accessible to every node? This is larger than what you can store on a single filer. Also, for performance reasons, the data must be separated across multiple filers. Currently we use lots of symlinks to tie it all together into a single logical directory tree, but that's a really ugly solution. There's got to be a better way. Right? Anyone?

Re:I had a related question

[zulux]

Currently we use lots of symlinks to tie it all together into a single logical directory tree, but that's a really ugly solution.

I don't think a bunch of symlinks is ugly at all. If it works well - who cares?

Are you haveing trouble with any of your simlinked directory structures - in my little FreeBSD world, I've never noticed any problem at all except for a few utilities that are aware of simlinks and will allow you to chose to traverse them or not - like rsync.

Re:I had a related question

[Anonymous Coward]

Take a look at Veritas Storage Foundation Cluster File System. You can have your 50+ Terabytes on a SAN fabric, and each server on the fabric can have the 50+ terabyte LUN mapped to them. The Cluster File System manages all of the concurrent access to the filesystem from each node so things don't get clobbered.

You'll get your performance through the SAN by utilizing high performance FCAL disks and multiple HBAs to your servers. You can have the load balanced across the HBAs to give you the bandwidth that you require.

Re:I had a related question

[sith]

I agree that a SAN could be the way to go. Moving your storage to fibre instead of trying to throw it on the same network link as your render management and all make sense.

All the SAN solutions out there are rather a bit wonky. Not saying they won't work, they're just all wonky. If Apple's xSan actually performs in the real world the way the claim (and the way it was working at NAB) then it could be the be-all-end-all. It'll be crossplatform and all, with file-level locking. Whether it really works or not...

Re:I had a related question

[dolphin-brother]

That's a difficult question to answer without knowing something of your setup. How are the spindles organized--SAN, individual file servers NFS cross-mounting, or what? Which OSes are you running? Also, how much money are you able to spend to resolve this problem?

If you could rebuild everything from the ground up (and had tons of money to throw at it), you'd most likely want to build a system based on a very expensive vendor solution.

Assuming that you can't do that, your best bet would be to go with some sort of parallel filesystem, the likes of Lustre, GFS, Ibrix, GPFS or CxFS. The architectures of these vary, but the basic principle they share is performance scalability based on increasing the number of data paths to the disk. So if you have, say, 100 nodes on a high-speed network, you take 10 of them and attach them to your SAN. The parallel filesystem spans the entire SAN and so requests from the nodes can reach any bit on the SAN from any of the ten paths. If you need more performance, you add more paths: controllers, HBAs and storage nodes. I know GPFS scales linearly in performance based on the number of paths to the data, and I believe the others scale well also.

I haven't hit 50 TB on disk (I have on tape, but your post suggests that tape wouldn't give you the performance you need), but I have set up several 4-8 TB GPFS filesystems that could easily grow to 50 TB if I had the spindles.

Good luck finding a solution; symlink-based solutions on a convnentional *NIX filesystem are a nightmare; I sympathize.

Sun Grid Engine, at least for Maya

[FueledByRamen]

I don't know what the setup for After Effects and such looks like, but I managed to build a pretty good Sun Grid Engine system for distributing Maya batch renders. SGE is free, works on Linux or Solaris, x86 or SPARC (I obviously used the x86 linux binaries), and seems to be very well designed. I set up a pretty solid system for netbooting the clients, running them diskless (or with local swap drives), adding new clients on the fly (all scripted), and it all worked flawlessly. You could submit a batch job and it would distribute it per-frame as an array job to all the different nodes. Or you could just run SETI on all of them...

It's since been taken down in favor of running Alfred (because I no longer use Maya's builtin renderer, we've moved on to MTOR and PRMan), but I still have all of the files and scripts for it. If anyone's interested, I'd be happy to share: sabretooth@gmail.com

High Performance Computing Perspective

[Anonymous Coward]

I manage a large Linux cluster that is used for a wide variety of tasks, including running Renderman. Our nodes are dual Xeon 2.4 GHz with 2 GB RAM and they are more than sufficient for the design group's rendering needs (primarily Maya). Our scheduler (OpenPBS) lets them request up to 64 processors at a time, which, I'm told by their IT guy, works well with their Renderman maitre de.

There is a fiber interface (Myrinet) to each node used by the MPI crowd, but our rendering group doesn't use it; they seem content with the performance of Ethernet over copper. Your needs may be different, of course, but latency isn't really an issue for rendering, and copper should provide all the bandwidth you need.

I'm not knowledgable regarding all the software packages you list there, but I'm wondering if any of them would really take advantage of a 64-bit kernel (either on Opteron or G5-PPC970). Of course you can put a PPC version of Linux on the Xserve, but not without sacrificing nearly all Apple-provided management. If you expand the cluster to a large number of nodes, or even if you keep a small number of nodes but place it in a remote location, Xserve running Linux would be painful to manage (no remote power-off/-on, remote console problems). Xserve is shiny and has the requisite blue LED's, but and AMD or Intel box (from the right vendor) would be much easier to manage remotely.

PCI-Express

[Viking Coder]

My question is - when is someone going to make a blade with PCI-Express and enough room for the latest batch of cards from NVidia and ATI?

Seriously - this is going to become a huge issue, as more rendering is pushed out to the stream processor that is the GPU.

Render managers

[tikoloshe]

We have a 50 node dual 3Ghz render farm, 20 on w2k and 30 on linux 2.4.20 (ok Redhat9) We are currently using Muster www.vvertex.com, but are not that happy with it. It was reasonably priced, and the developers are very helpful, but its not quite there yet. I have been seriously considering building my own using a SQL database (currently Postgresql, but may swich to MySql) and perl. A render manager is really just a database with a bit of network sockets and scripts to run occasionally. A simple concept that is probably going to come back and bite me :)

dynebolic

[Anonymous Coward]

There is a distro called dynebolic which is specifically for multimedia. It features Veejay, Cinelera, MJPEG tools etc. It is supposed to run off the CD. It is supposed to use every computer hooked to the network that is running dynebolic in the manner of a render farm. That is a very attractive idea. Go around the house, put in dynebolic CDs and go nuts.

I can't say that it is great because I haven't been able to do much of the above. Maybe you will have better luck.

Re:Small cluster

[Anonymous Coward]

Secondly, from your questions it is obvious you're no rendering farm guru so why did the task of planning/researching the configuration get assigned to you? You should ... find a small and focused SI who will propose you a better solution than you can come up on your own

This attitude bothers me, and not for the first time here on Slashdot. How the hell do you think those experts got to be experts? Do you think they just *poofed* into being with all their knowledge and skills already existent? No, at some point, they started with little or no knowledge of the subject and gradually accumulated enough knowledge and experience to become experts!

Sheeesh! If everybody listened to this advice they never would do anything new or different for fear of coming up with some sub-optimal solution.

Depends on your workstations

[BBStriker]

Hey

I've set up and administrated a number of farms over the years (doing it as I type. its.. what I do). One thing you really want to do, certainly with Maya's renderer, is to try to use the same OS and platform on your farm as you use on your user workstations. There can be subtle or even obvious differences in the render output between OS's, and since you'll have enough issues to deal with you'll want to keep cross-platform incompatabilities out of the mix. Please, trust me on this. Had to deal with Maya Irix/Win2k/Linux differences in the past.

As for queueing software, give Condor a look-see. Free and functional. I reverse-engineered a Perl version of it before they made their source available, and my version has been run quite successfully at several animation studios and an effects house over the years. It's a well architected system for distributed computing.

Feel free to contact me if you've got any other render system or management questions. I'm always interested in seeing how other studios approach the challenge.

What kind of rendering?

[beegle]

This is a fun idea in the abstract, but if you're looking for concrete advice, you need to give us some concrete data. Most importantly, -what- are you rendering?

If you're at a university and you're doing some sort of bioinformatics visualization, use whatever the researchers are most comfortable with. The odds are good that this is whatever the CS department was teaching on 5 years ago. Probably Suns or Windows machines. Slave... errr, grad student labor is cheap, so use an OSS scheduling and job management system if you can.

At most other places, a similar rule applies: use whatever the users are most comfortable with. If you're using Mac workstations and software, then it may make sense to go with a G5 rendering farm. If you're using Windows... well, okay. Windows render farms still suck, but at least buy PCs to leave your options open. Unless you're a really large organization (that is, the sort that doesn't have to resort to Ask Slashdot for research), you probably want to use products that come with support contracts. $20k/year is a pretty good deal when compared to keeping a full-time support person for the same task.

Deadline Render Queue (beta)

[bhouston]

At Frantic Films we have over the past year developed our own network rendering solution: Deadline. Our solution has just recently entered a beta testing period thus if people are so inclined one can have a look at the current product (screenshots) and possibly download a trial version (download page). We used Deadline on a number of recent feature films including Scooby Doo 2 and Paycheck.

We did this because we primarily use Discreet's 3dsmax (with Brazil and V-Ray) and Eyeon's Digital Fusion. We have found that most existing render farm solutions do not support these two packages very well -- thus we decided to develop our own custom solution. We also support After Effects, Alias|Maya, AIR and other RenderMan compliant rendering packages.

Of interest to the general Slashdot crowd may be that this Deadline Render Management Solution is based on the open source (BSD License) Exocortex C# library originally released with this C# 3D Engine. Deadline is built with C# in the hopes that using Mono we will be able to start supporting Linux with minimal extra effort.

I'll be reading all the posts on this Slashdot thread but I would also appreciate any direct feedback on our current beta product. We also found solutions such as Rush and Smedge to be less than user friendly in many respects. Thus we have tried as best as we could to increase a 3D package that is not well supported by most render farm management solutions -- except for Discreet's Backburner (which we found not that that scalable.)

Re:Deadline Render Queue (beta)

[brobison]

We have almost requirements here at Blur Studio. I tried the Deadline beta, and it seems like a definate improvement over Backburner =) AS you mentioned, it's not very scalable. How many machines do you guys have on your farm? I wasn't very confident in Deadline being able to handle a large number of hosts, with it's XML polling architecture.

We have our own in-house solution as well: Assburner. Assburner is GPLed and we hope to do a public release in the next month or two. Along with our production tracking software: Resin.

Re:Deadline Render Queue (beta)

[bhouston]

Cool stuff. We will be posting a new beta on the site later today -- with a couple enhancements. I just loaded up Deadline Monitor here and it shows that we have 98 machines current rendering, 3 idle, 41 offline (user workstations) and 5 machines disabled. Overnight during really heavy production loads (i.e. back in February with Scooby Doo 2) we would have all possibly machines rendering except for the 5 or so disabled ones (which are file servers or underpowered machines.) Thus from a pragmatic standpoint it scales at least to 150 machines without any trouble. How big is the blur render farm?

Darwin

[Dav3K]

But more to the parent's point, OSX could be run Darwin-only, AKA BSD-style command-line interface. The bloat can be stripped out of OSX just as easily as it can for linux. Additionally, the user would have the comfort in knowing that his render farm is using the same OS as the workstations used to control it.

As other respondants have suggested, I guess it would come down to which OS supported the entire collection of desired applications for the job.

For true 64 bitness, launch every Linux!

[LightStruk]

Apple has not yet released a true 64-bit version of OS X, while Gentoo released a PPC64 version a few weeks ago. If you're going to buy 64 bits of CPU, you might as well get 64 bits of OS too.

Some Tips

[mnemonic_]

These are just some tips I've heard in my 4 years of experience.

Whichever processors you go with, make sure the entire farm uses the same type. Otherwise peculiar rendering differences might occur, in things like particles, hair/fur and fluids.

I suggest going with the Opterons just for the PC compatibility. While the CG industry is becoming more diverse hardware-wise, it is still dominated by PC's and to a much lesser extent SGI boxes (5 years ago it was all SGI). Using PC's keeps your options open. Perhaps someday you will find 3ds max and its included distributed rendreing software more suitable for a task, and that can only be used with PC's. Same goes with the Mental Ray and Brazil renderers and the Combustion compositing software. Macs just have not been widely used in the 3d graphics industry, and so the vast majority of 3d content creation software is PC and SGI only (Maya Unlimited is only available on PC and SGI, while a lower end wersion is on Mac). And VirtualPC cannot be used to emulate 3d hardware acceleration (and it shouldn't be used for anything processor intensive anyways), though this only applies to the hardware rendered viewports in the apps. Having only Macs would be risky, and could limit your capabilities significantly.

Pixar's PRMan (Photorealistic RenderMan) is a full blown renderer, not just something to help distribute render jobs. It is generally considered the best in the industry, though MentalRay and Brazil have gained significant followings. For a cheap but effective render queueing system, check out Smegde. Smedge was used by Manex Visual Effects for handling some of the effects shots in the Matrix trilogy. If you're running the Linux version of Maya (x86 only) it is not too difficult to distribute the render tasks yourself using shell scripts and the command-line renderer.

GB Ethernet should be fine, the bottleneck will be in the actual image processing not data transfer rates. 100Mb ethenet might even get the job done, thught I'd use GB for the added speed when copying large files. YMMV of course.

Overall I'd try to create a very flexible system, one that will definitely support the newest CG software down the road and one that ensures compatibility with everything, for those always short deadlines. Goodl luck with your rendering.

Re:Small cluster

[Welsh Dwarf]

This is something I posted on an earlier discussion. The fact of the matter is that once you've got to a certain level in a subject, you forget what it's like to be starting out, or even that you 'started out', and hence loose all consideration for people new to your field.

Not a dig, just a remark on human nature

nVidia Gelato

[Guspaz]

You might want to look into nVidia Gelato. It's a 3D renderer that uses Quadro FX cards as secondary FPUs, supposedly doubling or more the speed of rendering. They claim it's two to six times faster than the leading renderer. There's a demo, so you can verify those claims for your uses.

It runs under Linux, and "will function with whatever [render farm] management system you currently use.".

To reiterate, it's a SOFTWARE renderer, that is hardware accelerated by using the video card as a co-processor.

Re:using google

[CromeDome]

You're right. It probably is. However, Slashdot is supposed to be a community, right? So what's wrong for asking the community for help? That is one of the things community is about.

I don't see the problem with asking here. You can actually get a lot more insight from a lot of different backgrounds in one place. Yeah, you have to weed out some of the gems, but moderation helps some with that.

You elitist pigs are starting to bug me. We've all needed help from time to time, yourself included I'm sure. Don't knock others for asking for help.

CromeDome

Re:gigabit ethernet likely overkill

[NovySan]

3D rendering is proc intensive. 2D composite work (Shake, Digital Fusion, Nuke, etc) is NETWORK I/O intensive. You must have gigE at least from the main background plate server and shared element server. The pipe that drops off the final product is usually sending (HD proxies for internal review) and receiving (final output frames) so it's best to have that gigE too.

Advice gleaned from years of bitter experience

[gorodish]

Having built two generations of renderfarms, and now working on a third, I'd suggest building it as cheap as possible. You will want to upgrade every 2 years or so, so make sure that you won't feel bad disposing of the old farm when it's time for the new.

Regarding networking: you have to look carefully at the way the farm will be used. If you are doing any kind of compositing (which requires high I/O rates), you'll benefit from gigabit ethernet. You'll also benefit from gigabit if you have exceptionally short render times (less than 30 minutes per frame), since in this case I/O is a significant fraction of each frame's render cycle. But the longer your per-frame render time, the less necessary gigabit is. We've always used 100base and it still serves us well. Fiber is expensive and provides nothing you'll need that copper can't provide.

The individual machines should have identical configurations and be interchangable. Your goal is to not care when an individual machine dies. In light of this, there should be no local storage of data. You can save money on support if you buy spares instead of service contracts. Warranties also work, but the big manufacturers give their worst service to warranty-only customers.

Don't wire anything but ethernet to the machines. KVM wiring is expensive and unnecessary. Each machine should run unattended until it dies; when it does, you can wheel over a monitor and keyboard to diagnose it.

Opterons are fast, compatible, cool, lower-power and cheap. Xserves are nice, but we've found that Darwin doesn't integrate well into a pure Unix environment. You'd also be locking yourself into a single manufacturer.

Linux is cheap and effective, and easier to configure correctly as a server OS than as a desktop OS. There is so much commercial software available for it now that there is little reason to consider Windows or a commercial Unix. We haven't found Linux support from the big manufacturers to be all that great; if you use Linux, assume that you will have to solve most problems on your own.

Post Production House

[acidream]

I work at a small post production facility in Hollywood. We have a render farm of 6 dual Xeon Win2000 Boxx rack machines as well as five dual Xeon Win2000 Boxx workstations that render in their spare time. We run Maya and After Effects and we use Smedge to handle the distributed rendering for both maya and AE and Mental Ray. We also have another Dual Xeon box running Server 2003 with a eight drive raid setup with two main partitions, one is raid 1 for the maya scenes and the AE projects and the other is raid 0 for the rendered frames and comps. We started with just two rack machines, but we add one or two every couple months when the budget permits. Our renders are sent to Smedge via a script that we run from an Interix csh. It greatly simplifies the process of sending out renders. Each project we work on has a script with the name of the project, or the shot, and when we're ready to render we run the script which parses the text file with all the parameters for the render such as frame range, render quality and so on. Some people don't care for Smedge to much, but it gets the job done, and works for most all 3d applications as well as most compositing apps.