Slashdot Mirror
Renderfarm Setup Tips?
"In the hardware side, we still haven't made a choice between using AMD's Opteron or Apple's Xserve G5 (they have some very nice and price convenient cluster nodes which seem to be ideal for this kind of job), with Linux. As for the networking between them, is Gigaethernet enough or should we be going for Fiber? The software used to manage the render queues is another important point as well: I've been looking into Rush, and even though it's a commercial package, it works on all of the platforms we currently use (W2k/XP, Irix, OS X and Linux). But then there is also Dr. Queue, which is open source and is supported on at least the *NIX members of the aforementioned OS's. Other options include RenderPal and Pixar's RenderMan, but I would prefer an F/OSS alternative. Finally, it's worth noting that we'll be using the renderfarm for Maya and Adobe AfterEffects."
You might want to check out Cinelerra. It has pretty good support for renderfarms. I built one out of scrap 300mhz machines, and it only took a weekend.
(a) Wait for the next new processor technology to hit Slashdot.
(b) Build a Beowulf cluster of those.
(c) Profit!
Quidquid latine dictum sit, altum sonatur.
Check out g4u for deploying your render machines - it's a image based disk cloning tool that uses DHCP and FTP and which doesn't care what you run on your clients. (g4u itself is based on NetBSD, but that doesn't matter for the application).
I've used g4u to setup a ~50 node video rendering cluster, see my webpage on the Regensburg Marathon Cluster.
Enjoy!
- Hubert
Don't use macs. First off, you have less choice in renderers, and second, the hardware is more expensive. Rendering is grunt work. Buy cheap systems that you can upgrade more often, and run linux or something.
Macs are very nice hardware, but you really don't need that for rendering. For workstations they make sense, but for rendering you really want to have a lot of fast computers rather than nice computers.
Everything seemed to be going so nice
'till the end of all beings punched right through the ice
Why would you run Linux on the Apple boxes? Wouldn't OSX be just as good?
What render queue products are supported by your pieces of software? Why don't you try a few of them?
I'm sure a demo can be arranged.
I wouldn't go blindly marching in the direction of FOSS especially in something that is valuable enough to setup a renderfarm for.
Most importantly, find out what the people who will be using the software like and dislike about each package. And what works for them. If it saves you $30 per hour times 5 people software and hardware cost become insignifigant after one work week.
The biggest renderfarm in the world is useless if your people can't use it. Always remember that software is only good in its ability to meet the goals of the organization it supports.
We had a renderfarm for "The Chronicles of Riddick" of 40 boxes. Each box was a dual-proc Opteron.
We evaluated several render-queue management systems, and decided on Rush. The most persuasive arguments for using Rush were the very good experience we have heard from other users, and the simplicity of extending it to manage a variety of different tasks. I have to add Hammerhead to the list of happy customers. It did everything we could have hoped for. In particular, it was able to handle the inevitable crashing of machines pretty well.
While it's true that Rush is a proprietary, gotta-pay-for-it system; a robust render queue management system pays for itself very quickly in the ability to make your renferfarm productive. Perhaps a render queue manager is overkill when you have just 6 or 8 systems, but once you get up to 30 or 40 it is essential.
Our experience is all under Linux, but if you're going to be running After Effects that means that you're not going to be running Linux -- so there's not too much more I can help you with there. We did find that the dual Opterons worked much more efficiently than dual Xeons in multiprocessor rendering -- don't know about the Xserves, though. We were running mostly Maya, RenderMan, Shake, and our own in-house tools on the farm.
This farm is unfortunately powered down now that Riddick is done -- if you need some dual opterons, let me know at thad@hammerhead.com
I love Mondays. On a Monday, anything is possible.
Thats only if you desire maximum ease of use with minimum setup and running hassles. The same ease of use the regular G5's have is built into all their server stuff too. I'm sure the linux dudes will have something to say about that.....
I would take a really hard look at the ready made bio-information cluster they have all setup, and just load yer software as needed and off you go. But that's me. Some people seem to like futzing with computers.....After 20+ years doing that at work, I just wanna do what I wanna do when I wanna do it. Apple makes that easy.
I get paid to deal with headaches, I'm not gonna deal with them at home too.
My subconscious desire to get out of the computer field as a career must be surfacing--I read this as "reindeer farm". Then reality set back in and I almost made a lame "LapLANder" joke before tasering myself.
Recently, I have been working a lot with Apple's xgrid. We have been linking about 4 G5s/G4s together and getting impressive results. I don't understand your hardware situation, but if you are a Mac guy try it out.
GroupShares.com
-------
artlu.net
If you're going to do this, do it properly. Get systems with massive amounts of I/O that will cope with all the data you're trying to throw at them. For this kind of work, you need only buy from one vendor: SGI.
Don't bother with Intel/Linux, with dodgy hardware and the frequently-changing Linux code. Pay the money, get decent hardware with a support contract and a steady, stable, tried and trusted OS.
Apple *may* be an appropriate choice, now that Pixar have ported RenderMan to OS X, but I don't like the idea of my arrays running at 7200 rpm's. Get SGI, get fibre channel, and (possibly) get gigabit ethernet.
It'll all pay off - it won't be cheap, but in the long run, the results will be worth the money and the wait.
I think that what you're looking for is a renderfarm for computer graphics rendering, right? in that case, you should be looking at PVM or OpenMOSIX or even MPI. In either case, since you're going to have more data crunching than actual data transfer, I think that even T100 would be enough. gigabit will be nice, but fiber is not worth it. Drqueue is nice... if you can get it to work, and I didn't. We used pvmpovray for many things, and I think that might be worth a look. pvmpovray exists for gentoo with an ebuild script, which would make the installation and configuration the minimum pain for it. But that option requires a conversion from maya to povray files. Also, I don't know what the pricing is going to be like, but if it were up to me, I'd take the Opterons, because I believe they are faster, although I'm not positive on that, and because I know they're well supported under linux, and again, I think that's a more personal choice to make, but the impression I got from AMD is that you always get the most for your buck. The Opterons also let you find replacement parts or upgrades a little easier than the G5 if you burn a CPU or motherboard. That's just my $.02 worth of advice.
---- I am certain of only one thing : I know nothing else.
Peace
Light us up the bomb!
I think I'll stop here.
... but it was rejected. How do you deal with terabytes of data (50+ TB), all in a single directory tree, all must be accessible to every node? This is larger than what you can store on a single filer. Also, for performance reasons, the data must be separated across multiple filers. Currently we use lots of symlinks to tie it all together into a single logical directory tree, but that's a really ugly solution. There's got to be a better way. Right? Anyone?
___
If you think big enough, you'll never have to do it.
I don't know what the setup for After Effects and such looks like, but I managed to build a pretty good Sun Grid Engine system for distributing Maya batch renders. SGE is free, works on Linux or Solaris, x86 or SPARC (I obviously used the x86 linux binaries), and seems to be very well designed. I set up a pretty solid system for netbooting the clients, running them diskless (or with local swap drives), adding new clients on the fly (all scripted), and it all worked flawlessly. You could submit a batch job and it would distribute it per-frame as an array job to all the different nodes. Or you could just run SETI on all of them...
It's since been taken down in favor of running Alfred (because I no longer use Maya's builtin renderer, we've moved on to MTOR and PRMan), but I still have all of the files and scripts for it. If anyone's interested, I'd be happy to share: sabretooth@gmail.com
Every cloud has a silver lining (except for the mushroom shaped ones, which have a lining of Iridium & Strontium 90)
There is a fiber interface (Myrinet) to each node used by the MPI crowd, but our rendering group doesn't use it; they seem content with the performance of Ethernet over copper. Your needs may be different, of course, but latency isn't really an issue for rendering, and copper should provide all the bandwidth you need.
I'm not knowledgable regarding all the software packages you list there, but I'm wondering if any of them would really take advantage of a 64-bit kernel (either on Opteron or G5-PPC970). Of course you can put a PPC version of Linux on the Xserve, but not without sacrificing nearly all Apple-provided management. If you expand the cluster to a large number of nodes, or even if you keep a small number of nodes but place it in a remote location, Xserve running Linux would be painful to manage (no remote power-off/-on, remote console problems). Xserve is shiny and has the requisite blue LED's, but and AMD or Intel box (from the right vendor) would be much easier to manage remotely.
Unless you have plenty of cash to blow. In my experience with clusters the network is rarely your biggest bottleneck if you're using GigE. Rendering should be parallelized enough that it won't be saturating the network.
For one, consider creating a standard image for all of your machines and have them install using pxeboot or etherboot. If any of the machines get hosed, you can reboot them and force them to reinstall their OS in 5 minutes.
It seems to me that fiber is a waste of money. You implement gigE using copper. I would think that most of the data transfer is going to be the scene data and then an image transfered back.
The company i used to work for was developing a global illumination raytracer and we created a program that tiled the output image; however, it still needed the entire scene so it could calculate all of the lighting and rays. I don't know what rendering software exists nowdays so I can't give any advice there.
(I think this image was created using our renderer using environmental lighting because i vaguely remember this scene (while we were fixing bugs for them) from this company before we went under in 2002)
My question is - when is someone going to make a blade with PCI-Express and enough room for the latest batch of cards from NVidia and ATI?
Seriously - this is going to become a huge issue, as more rendering is pushed out to the stream processor that is the GPU.
Education is the silver bullet.
We have a 50 node dual 3Ghz render farm, 20 on w2k and 30 on linux 2.4.20 (ok Redhat9) We are currently using Muster www.vvertex.com, but are not that happy with it. It was reasonably priced, and the developers are very helpful, but its not quite there yet. I have been seriously considering building my own using a SQL database (currently Postgresql, but may swich to MySql) and perl. A render manager is really just a database with a bit of network sockets and scripts to run occasionally. A simple concept that is probably going to come back and bite me :)
--
There is a distro called dynebolic which is specifically for multimedia. It features Veejay, Cinelera, MJPEG tools etc. It is supposed to run off the CD. It is supposed to use every computer hooked to the network that is running dynebolic in the manner of a render farm. That is a very attractive idea. Go around the house, put in dynebolic CDs and go nuts.
I can't say that it is great because I haven't been able to do much of the above. Maybe you will have better luck.
Just catch a plane down to New Zealand, kidnap Peter Jackson's iPod, and tell him he ain't gettin' it back until he signs over Weta's setup to you as sole proprietor. Sheesh - all this talk of hardware, software, fiber/GbE, and the like is waaay too complicated.
I also reply below your current threshold.
I would suggest you look into a product specific rendering cluster. For example, Alias Maya, Newtek Lightwave, etc... I have a render farm with all of the render clients on the same machines. So whether the project requires a specific application I can use the same computers to render.
.02.
My
SGI is history in this biz... There is no way to cost effectively use those boxes anymore. I work at DD and we use a linux/win2k renderfarm. Only thing SGIs are good for is Flame... Also, since this guy is running After Effects, he's either Win or Mac.
You know, some of us aren't here because of the oh-so-important /. karma. Many are here to keep themselves informed and educated. How's about giving the guy the benefit of the doubt and go render yerself a chill-pill?
Take note: I'm logged in and not afraid of the big bad flamebait mod.
The cheapest and easiest way to set up a renderfarm is to use someone else's computers!
We used to run Lightwave for work-related projects, but we only had a few machines allocated to the media lab. So after hours, we'd sneak around the cube farms and pop a client boot disk in every machine we could find.
If ever anyone ever missed a point, it's you.
Interesting post, but doesn't belong here. He's talking about redering images, not meat.
Secondly, from your questions it is obvious you're no rendering farm guru so why did the task of planning/researching the configuration get assigned to you? You should ... find a small and focused SI who will propose you a better solution than you can come up on your own
This attitude bothers me, and not for the first time here on Slashdot. How the hell do you think those experts got to be experts? Do you think they just *poofed* into being with all their knowledge and skills already existent? No, at some point, they started with little or no knowledge of the subject and gradually accumulated enough knowledge and experience to become experts!
Sheeesh! If everybody listened to this advice they never would do anything new or different for fear of coming up with some sub-optimal solution.
Who wants to be the first to kick YOU in the balls for not understanding that they only made special effects for Riddick? Food on the table, you know.
Blame the director if you really feel that bad about spending $10 in the theater.
So, get someone with mod points to mod me down. I don't lose sleep at night over my slashdot karma.
Hey
I've set up and administrated a number of farms over the years (doing it as I type. its.. what I do). One thing you really want to do, certainly with Maya's renderer, is to try to use the same OS and platform on your farm as you use on your user workstations. There can be subtle or even obvious differences in the render output between OS's, and since you'll have enough issues to deal with you'll want to keep cross-platform incompatabilities out of the mix. Please, trust me on this. Had to deal with Maya Irix/Win2k/Linux differences in the past.
As for queueing software, give Condor a look-see. Free and functional. I reverse-engineered a Perl version of it before they made their source available, and my version has been run quite successfully at several animation studios and an effects house over the years. It's a well architected system for distributed computing.
Feel free to contact me if you've got any other render system or management questions. I'm always interested in seeing how other studios approach the challenge.
rendering is processor intensive (usually) and not io intensive so you will probably do just fine with plain old 100 mbit ethernet. take the money you save on networking and spend it on more nodes.
Try a cluster file system.
"Filers" create "hot spots" whereby often-accessed directories/files create IO bottlenecks.
I think you can use this CFS to create a directory tree with over 200TB of data (/home/lun1, /home/lun2, .../home/lun255). You can't "tie" them together like with LVM but you do get huge throughput as opposed to a single-host bottleneck with a volume manager and/or clustered NAS filers with the hot spot problem.
Everything you have mentioned is dependent on what render engine you are using.
First, figure that out, then go to the forums for that engine and ask your question there.
I just built an 8-node renderfarm for using Vray, which means 3D Studio, which means Win2k and it was a piece of cake.
6-8 nodes is tiny. Figure your use, then ask on the right forum. Many, many people have done this already.
I'm out of my mind right now, but feel free to leave a message.....
This is a fun idea in the abstract, but if you're looking for concrete advice, you need to give us some concrete data. Most importantly, -what- are you rendering?
If you're at a university and you're doing some sort of bioinformatics visualization, use whatever the researchers are most comfortable with. The odds are good that this is whatever the CS department was teaching on 5 years ago. Probably Suns or Windows machines. Slave... errr, grad student labor is cheap, so use an OSS scheduling and job management system if you can.
At most other places, a similar rule applies: use whatever the users are most comfortable with. If you're using Mac workstations and software, then it may make sense to go with a G5 rendering farm. If you're using Windows... well, okay. Windows render farms still suck, but at least buy PCs to leave your options open. Unless you're a really large organization (that is, the sort that doesn't have to resort to Ask Slashdot for research), you probably want to use products that come with support contracts. $20k/year is a pretty good deal when compared to keeping a full-time support person for the same task.
--
Here is a fresh After Effects grid rendering tool just released last month for both mac & pc. Havent tried it yet, as I kindof laugh at comp render times when compared to 3d rendering. http://www.gridironxfactor.com/index_home.asp And this is the render manager we use, thats worked like a dream since its first install. We use it for Lightwave, but there is the option for Maya as well (even though mayas native renderer is 8P)available for PC,MacOSX,Linux http://www.liquiddreamsolutions.com/
We did this because we primarily use Discreet's 3dsmax (with Brazil and V-Ray) and Eyeon's Digital Fusion. We have found that most existing render farm solutions do not support these two packages very well -- thus we decided to develop our own custom solution. We also support After Effects, Alias|Maya, AIR and other RenderMan compliant rendering packages.
Of interest to the general Slashdot crowd may be that this Deadline Render Management Solution is based on the open source (BSD License) Exocortex C# library originally released with this C# 3D Engine. Deadline is built with C# in the hopes that using Mono we will be able to start supporting Linux with minimal extra effort.
I'll be reading all the posts on this Slashdot thread but I would also appreciate any direct feedback on our current beta product. We also found solutions such as Rush and Smedge to be less than user friendly in many respects. Thus we have tried as best as we could to increase a 3D package that is not well supported by most render farm management solutions -- except for Discreet's Backburner (which we found not that that scalable.)
IRUSH
BOXX nodes
ok I only have experance with a custom T&L rendering platform but the grid engine worked very well I have used it at a couple of hardware design places and it's CPU blancing was very nice indeed
it supports
Apple Mac OS/X
Compaq Tru64 Unix 5.0, 5.1
Hewlett Packard HP-UX 11.x
IBM AIX 4.3, 5.1
Linux x86, kernel 2.4, glibc >= 2.2
Linux AMD64 (Opteron), kernel 2.4, glibc >= 2.2
Silicon Graphics IRIX 6.5
Sun Microsystems Solaris (Sparc) 7 and higher 32-bit
Sun Microsystems Solaris (Sparc) 7 and higher 64-bit
Sun Microsystems Solaris (x86) 8 and higher
see http://gridengine.sunsource.net/
I would look at what software you want to run BEFORE buying hardware
regards
John jones
First off, the G5 XServe's are very fast machines and the cluster node is, of course, designed for just this sort of thing in the minimal amount of space. You should be aware that they are LOUD as hell though... louder than the G4 XServes, which sound like a plane taking off. That said, plan on hiding them away from your workspace because even one of them in the same room with you will drive you nuts. This is no different than a 1U server from Dell/Whoever though.
You'll only really need to buy a single non-cluster-node XServe to make your system work, and even that isn't needed if you have a desktop Mac to admin them from. Make sure you invest in either Apple's Remote Desktop application, or you install a VNC server or something on each of your cluster nodes. You'll want to get access to the desktop, and with those machines sans-video card, a remote desktop application is obviously the only way. For your type of application, sometimes the command prompt can only get you so far. :)
That said, if you do go with the Apple solution, there is not much reason to run Linux on them. Most open source apps you can build for Linux will build on OSX without issue, and keeping OSX on the machines gives you the ability to run Adobe After Effects on the machines and use the After Effects built-in cluster rendering. It's not that I dislike Linux, but it's just not needed here, especially when OSX offers you the addition of a whole host of commercial applications over the usual open source ones. Also, the cluster nodes include OSX Server (5 User). The 5 user licenses are for the filing sharing functionality, so they don't apply to what you're doing and for all intents and purposes, it works and acts largely like the OSX you know and use everyday, so the OS is included in the price of the hardware and you're not paying an extra invoice for it.
As for interconnecting them, the XServe machines all feature dual gigabit ethernet, which you'll want to obviously connect to a gigabit ethernet switch.
That said, you're left with the option of storage. The best bet for a smaller scale cluster (6-12 CPU's, as you mentioned) would be just to dedicate a single non-cluster-node XServe with all 4 disks installed and striped into a single volume. Connect the second gigabit ethernet port of your Xserve cluster machines to a second switch and then use NFS on OS X server to share to your cluster nodes. This will provide plenty of performance over a seperate gigabit ethernet connection, though your mileage may vary depending on what you want to do.
Obviously, for more machines, you'll do well to look into more advanced storage. XSan can do just this once they release it, which I imagine it will be when you're ready to scale up that far.
Hope that helps, at least as far as Apple solutions go.bash-3.00$ uname -a
SunOS panda 5.10 Generic sun4u sparc SUNW,Ultra-2
But more to the parent's point, OSX could be run Darwin-only, AKA BSD-style command-line interface. The bloat can be stripped out of OSX just as easily as it can for linux. Additionally, the user would have the comfort in knowing that his render farm is using the same OS as the workstations used to control it.
As other respondants have suggested, I guess it would come down to which OS supported the entire collection of desired applications for the job.
Apple has not yet released a true 64-bit version of OS X, while Gentoo released a PPC64 version a few weeks ago. If you're going to buy 64 bits of CPU, you might as well get 64 bits of OS too.
but I do work in a 3D department at a television production studio. I can't say Rush is the best, but we use it and the animators are generally happy with it. We have about 10 render only boxes and another 9 or 10 workstations that Rush let's you "online" when you are not using them (lunch, nights, weekends, long meetings...).
Stupid sexy Flanders.
These are just some tips I've heard in my 4 years of experience.
Whichever processors you go with, make sure the entire farm uses the same type. Otherwise peculiar rendering differences might occur, in things like particles, hair/fur and fluids.
I suggest going with the Opterons just for the PC compatibility. While the CG industry is becoming more diverse hardware-wise, it is still dominated by PC's and to a much lesser extent SGI boxes (5 years ago it was all SGI). Using PC's keeps your options open. Perhaps someday you will find 3ds max and its included distributed rendreing software more suitable for a task, and that can only be used with PC's. Same goes with the Mental Ray and Brazil renderers and the Combustion compositing software. Macs just have not been widely used in the 3d graphics industry, and so the vast majority of 3d content creation software is PC and SGI only (Maya Unlimited is only available on PC and SGI, while a lower end wersion is on Mac). And VirtualPC cannot be used to emulate 3d hardware acceleration (and it shouldn't be used for anything processor intensive anyways), though this only applies to the hardware rendered viewports in the apps. Having only Macs would be risky, and could limit your capabilities significantly.
Pixar's PRMan (Photorealistic RenderMan) is a full blown renderer, not just something to help distribute render jobs. It is generally considered the best in the industry, though MentalRay and Brazil have gained significant followings. For a cheap but effective render queueing system, check out Smegde. Smedge was used by Manex Visual Effects for handling some of the effects shots in the Matrix trilogy. If you're running the Linux version of Maya (x86 only) it is not too difficult to distribute the render tasks yourself using shell scripts and the command-line renderer.
GB Ethernet should be fine, the bottleneck will be in the actual image processing not data transfer rates. 100Mb ethenet might even get the job done, thught I'd use GB for the added speed when copying large files. YMMV of course.
Overall I'd try to create a very flexible system, one that will definitely support the newest CG software down the road and one that ensures compatibility with everything, for those always short deadlines. Goodl luck with your rendering.
This is something I posted on an earlier discussion. The fact of the matter is that once you've got to a certain level in a subject, you forget what it's like to be starting out, or even that you 'started out', and hence loose all consideration for people new to your field.
Not a dig, just a remark on human nature
Ask 8 slackers a question, get 10 awnsers (a citation, but I can't remember from who)
Good stuff, much easier to set up and maintain than that SAN/NAS crap.
http://www.isilon.com/
And no, I don't work there.
Well quit mixing your media here. What do you mean GbE is enough, it runs over both copper and fiber. Now there are other layer 2 protocols that run over both copper and fiber (all though different cables) for example - what SAN are you going to use ? iSCSI, FC, SRP, NFS ?
What networking are you going to use ? GbE, TokenRing, FDDI, ???
What HPC/Interconnect are you going to use Infiniband, Myrinet, VI, ???
Some of these are the same networks, some of these are not, they all run at various speeds with various types of cabling. Now go ask your question again
PS GbE is probably good for networking, probably not for your SAN
I have mod points and I am not afraid to use them
All your Linux are belong to us!
You might want to look into nVidia Gelato. It's a 3D renderer that uses Quadro FX cards as secondary FPUs, supposedly doubling or more the speed of rendering. They claim it's two to six times faster than the leading renderer. There's a demo, so you can verify those claims for your uses.
It runs under Linux, and "will function with whatever [render farm] management system you currently use.".
To reiterate, it's a SOFTWARE renderer, that is hardware accelerated by using the video card as a co-processor.
You're right. It probably is. However, Slashdot is supposed to be a community, right? So what's wrong for asking the community for help? That is one of the things community is about.
I don't see the problem with asking here. You can actually get a lot more insight from a lot of different backgrounds in one place. Yeah, you have to weed out some of the gems, but moderation helps some with that.
You elitist pigs are starting to bug me. We've all needed help from time to time, yourself included I'm sure. Don't knock others for asking for help.
CromeDome
Gee, what is it like to be perfect and born with all of the knowledge of the Universe? What? You weren't born with that knowledge? Then where did you get it?
I do my video editing work on a old G4 using FCP 3.0 (I know, I'm behind.) Occasionally I get stuck with 8 hour renders that generate 6 gig or better render files.
Is there any cheap solution that will let me spread this work out? I don't use AE at all. My shop is incredibly cheap so it would have to be a free or nearly free solution. I know that's asking a lot. It just isn't worth it to get more macs and put FCP express on them. That's the only thing I can think of.
Any tips would be greatly appreciated.
Why do I have this? I don't smoke.
imp is a POV-RAY based distibuted renderfarm for Win or Lin.
I work at a small post production facility in Hollywood. We have a render farm of 6 dual Xeon Win2000 Boxx rack machines as well as five dual Xeon Win2000 Boxx workstations that render in their spare time. We run Maya and After Effects and we use Smedge to handle the distributed rendering for both maya and AE and Mental Ray. We also have another Dual Xeon box running Server 2003 with a eight drive raid setup with two main partitions, one is raid 1 for the maya scenes and the AE projects and the other is raid 0 for the rendered frames and comps. We started with just two rack machines, but we add one or two every couple months when the budget permits. Our renders are sent to Smedge via a script that we run from an Interix csh. It greatly simplifies the process of sending out renders. Each project we work on has a script with the name of the project, or the shot, and when we're ready to render we run the script which parses the text file with all the parameters for the render such as frame range, render quality and so on. Some people don't care for Smedge to much, but it gets the job done, and works for most all 3d applications as well as most compositing apps.
Experts became experts by READING the appropriate documentation.
/.'ers opinion.
Building a renderfarm has been done before and is a well-documented process. Choosing the software that you're going to be using should be based on the work you're going to perform -- not some random
Conformity is the jailer of freedom and enemy of growth. -JFK
Personally I think openmosix and a IBM PPC bladecenter woulbe be the ideal option, but as I recall openmosix hasn't been ported to PPC yet. In the short term, I think the Opterons will have better linux support, but IBM certainly has the resources to change that if they want to.
Consider with the bladeservers, you can fit 14 SMP blades into a 7U bladecenter, double the density of using Apple's offering. IBM's PPC blades are slightly cheaper than Apple's xserve too. Not to mention that the PPC blades cost about as much as the Xeon ones, and they actually come w/ both procs.
I don't even want to imagine the power bill for 28 P4 Xeons @~200w each!
We're replacing our a**-slow "Platform LSF" commercial cpu farm management software with GridWare / Sun Grid Engine. Its great! Grid Engine performs operations on jobs in an instant just as if you were running them locally (*unlike LSF which is too slow at common tasks like job submission or job killing to be useful*). its also free, with support from sun available if you feel like wasting money.
I know little about it myself but i think apple's Xgrid technology could be worth a look.
You are obviously a troll. Despite the fact that the "director" may have made decisions different than you would have made, you don't have a clue as to how complex it is to pull off visual effects like the ones in this movie, especially from a small house. I go to see certain movies such as this not with the expectation that they will blow me away, but as an animator, to experience new effects and visuals. Be more open minded and less childish.
If only I had Mod Points
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
I didn't miss the point. Gibson's rendering farm is for a series of high-quality video clips from "footage" that appears on the internet. It's the prisoners' job to hand-render each frame. Sorry if I failed to make that clear, but I was trying to avoid divulging too much information about the plot.
I'm happy to admit that I haven't the first clue what goes into the rendering process, but Gibson gave the impression that it is a highly labour-intensive process.
...Whether my Maker is prepared for the great ordeal of meeting me is another matter.
Churchill
The internet. I never bought a book or attended a workshop or anything like that.
I do not claim to be an expert, but I would do my own research and then ask one-two very specific questions on a specialized forum.
The way the question is asked makes it clear the poster hasn't done his homework and expects others to do it for him.
Okay maybe I was too harsh in the way I said it, but really, this question belongs to a newsgroup/forum for beginners.
I believe most Slashdotters prefer to see a "how I created a high performance render farm" rather than "what is a high performance render farm" kind of article.
Existing answers to this kind of question abound on the Internet, if you do a simple search you'll find the 1st result ("Need help for building and setup a render farm" at highend2d.com) to be exactly the same like this question: search Google for:
render cluster maya nodes small
Well, someone from Hamerhead (who worked on the FX for the new Riddick movie) posted about the renderfarm they used and his experiences. Sounds like the original question got some answers....
If everyone listened to his advice, they'd at least have a good picture of the real requirements of the project they were undertaking, which would leave them in a much better position to learn while doing than just plunging in without a clear definition of problem.
And besides that, a smart person could learn alot from doing the requirements gathering and then working closely with an SI to define and implement a solution.
Except maybe for the guy that responded well above you that worked for Hammerhead.
You know, the guys that did the graphics for The Chronicles of Riddick?
Yeah. Open mouth, insert foot. You got it.
For a "Render Farm" manager Here.
Just thought this was an appropriate discussion. Also, maybe a "slashjobs" would be an interesting addition to this site.
Stupid sexy Flanders.
1) Did you notice how many people who had actual experience (including someone who works at Hammerhead Productions) chimed in to give advice. Therebye proving you dead wrong.
2) Did you notice that in almost every single Ask Slashdot there's one punk who asks "Why ask here? You're just wasting your time." If these people really hate Ask Slashdot, they should stop reading it.
That's gotta fit into your schema somewhere
Regarding networking: you have to look carefully at the way the farm will be used. If you are doing any kind of compositing (which requires high I/O rates), you'll benefit from gigabit ethernet. You'll also benefit from gigabit if you have exceptionally short render times (less than 30 minutes per frame), since in this case I/O is a significant fraction of each frame's render cycle. But the longer your per-frame render time, the less necessary gigabit is. We've always used 100base and it still serves us well. Fiber is expensive and provides nothing you'll need that copper can't provide.
The individual machines should have identical configurations and be interchangable. Your goal is to not care when an individual machine dies. In light of this, there should be no local storage of data. You can save money on support if you buy spares instead of service contracts. Warranties also work, but the big manufacturers give their worst service to warranty-only customers.
Don't wire anything but ethernet to the machines. KVM wiring is expensive and unnecessary. Each machine should run unattended until it dies; when it does, you can wheel over a monitor and keyboard to diagnose it.
Opterons are fast, compatible, cool, lower-power and cheap. Xserves are nice, but we've found that Darwin doesn't integrate well into a pure Unix environment. You'd also be locking yourself into a single manufacturer.
Linux is cheap and effective, and easier to configure correctly as a server OS than as a desktop OS. There is so much commercial software available for it now that there is little reason to consider Windows or a commercial Unix. We haven't found Linux support from the big manufacturers to be all that great; if you use Linux, assume that you will have to solve most problems on your own.
I work at a small post production facility in Hollywood. We have a render farm of 6 dual Xeon Win2000 Boxx rack machines as well as five dual Xeon Win2000 Boxx workstations that render in their spare time. We run Maya and After Effects and we use Smedge to handle the distributed rendering for both maya and AE and Mental Ray. We also have another Dual Xeon box running Server 2003 with a eight drive raid setup with two main partitions, one is raid 1 for the maya scenes and the AE projects and the other is raid 0 for the rendered frames and comps. We started with just two rack machines, but we add one or two every couple months when the budget permits. Our renders are sent to Smedge via a script that we run from an Interix csh. It greatly simplifies the process of sending out renders. Each project we work on has a script with the name of the project, or the shot, and when we're ready to render we run the script which parses the text file with all the parameters for the render such as frame range, render quality and so on. Some people don't care for Smedge to much, but it gets the job done, and works for most all 3d applications as well as most compositing apps.
FILE SERVER
For a large farm, the most important thing is a strong file server that can handle the load you're going to throw at it.
Netapp is a great choice; many production companies, small and large have switched to netapp when all other PC solutions failed (W2K servers, OSX servers, Linux servers), and their problems went away. Netapp is pricey, but you can find used equipment, or rental solutions.
IMHO, linux/windows is great for the nodes, but not as file servers. Linux especially has had its NFS services beaten into submission when you get a 30 host (60 cpu) farm beating on the one linux server over NFS. Whereas an SGI Origin rack mount or Netapp server can handle that and more (100 - 200 hosts).
A problem with Windows servers is that they license connections. If you try to run a Windows XP or 2K system as a file server, you'll quickly find the OS won't let you share the drive to more than 10 machines. You have to run 2K or 2003 Server to server more than that, and you have to make sure you have enough client licenses to serve the network as you grow it. Linux servers (and most other servers, including SGI) have no such connection limit on NFS.
LOAD BALANCING
When you start getting above 100 hosts, consider using a switch that can handle load balancing, so that when the farm is rendering hard, the workstations still get high priority bandwidth to the disk server, otherwise loading a tiny text file from a workstation can take minutes when the farm is being heavily utilized.
With Cisco switches, the terminology is QOS (Quality Of Service), made popular by VoIP services which deal with high volumes of network traffic, which happens to be a similar problem to file system traffic on a render farms.
There are usually two bottle necks; the network at the file server, and the file server itself. Rarely are the workstations/nodes a bottle neck.
SUBNETTING
Sometimes it pays to have multiple network segments all hanging off the same file server, where the server has separate network cards for each segment. This way you distribute the network traffic, rather than concentrating it all at one point (the server's network card).
One way is to put all the workstations on one network, the farm on another. Or split a large farm into separate subnets. You may be able to then use the server's own config to balance the network packets so it gives higher precedence to the workstations, and lower to the farm, so users can access files quickly, no matter how busy the farm is.
AIR CONDITIONING
Especially in hot regions like Los Angeles, it's very important to have a suitable A/C system to keep the machine room cool. Often people have to scramble to upgrade their A/C system when their farm starts heating up, and find out there's no physical way to upgrade their A/C due to the limitations of their machine room, or adequate exposure to the outside for ventilation.
You sysadmins probably want to have a pager system that alerts you when the machine room starts heating up, or a script that shuts down the farm if the temperature exceeds some maximum.
Watching the temp of the cpus is a good way to detect dead or dying CPU fans, the major cause of outages on nodes.
USE AN EXPANDABLE SWITCHBOX
Many of the Belkin switch boxes are chainable, so that you can have literally hundreds of machines accessable from one monitor/keyboard combo.
Get a good expandable switch box so that you can see the physical console of each node, in case the machine(s) blue screen, panic, etc. so you can see what the problem is with a box, so that you can solve it, rather than just rebooting the box regularly. Often problems are obviously advertised on the console, but a cheap farm setup where there's no easy way to see the console of each box (ie. "lets just use remote desktops!") looses the ability to see what's wrong with a machine when it dies. Get to know how to read a BSOD, so you can determine if it's a ram problem, or a driver problem.
COPPER
As a professional After Effects designer, I'll just remind you that a 2D compositing program such as AE will have different demands to a 3D raytracer such as Maya. AE will render in seconds per frame, so the overall render times will come down to network bandwidth and storage efficiency. There are some specific, unique situations where After Effects will render faster on a single machine than on a network, because the transfer time for all the render assets may be longer than the render itself. Depending on the complexity of your renders and the age of your machines, I'd say it would be unusual for AE to spend for than about 10 seconds on any single frame, and some simple compositions will render at about 1 second per frame or faster. So a network set up specifically for AE will rely heavily on throughput. Maya renders will be much slower, several minutes to several hours per frame, so network traffic is not going to have an immediate effect on render speeds in the same way. But the assets required by Maya can be much bigger - you might end up with several hundred meg. of texture files. Again, it's possible with highly detailed renders that it can take more time to send data over the network than it does to render it. Pixar have had this problem and had to recode the parts of their software dealing with cacheing etc etc. Personally I'd go for the Mac solution, their stuff just works.
Putting syrup in coffee is some form of blasphemy.
Imagine a Beowulf cluster of renderfarms...a 2D cluster!
Yo, I and a cohort developed a business plan around a render farm business just before the dotbomb (thus, why we didn't actually go after funding). We have a substantial amount of research done, although the stuff is three years old now. If you want any of this info, just let me know. Email me at ryan(at)suydams.com Just a quick note, a massive array of disposable nodes, connected by simple ethernet (GigE) is plenty for most rendering jobs. You gain little by having a massively parallel system image, but the cost goes up tremendously. As the rendering batch jobs are naturally threaded (i.e., each frame of the animation can be rendered as it's own work-unit), you will find just a simple LAN will be plenty sufficient for your needs. Finally, check out www.boxxtech.com as they have some awesome rendernodes already built (on Xeon or Opteron platforms).
On the flip side Linux enjoys having engineering programs like Fluent and Nastran, which OSX doesn't have support for yet. If the programs you need only ran on Irix, Linux or Windows then that is hardware platform you would need to acquire.
I have to say I really liked the movie for the first 20-30 minutes. The SFX and environment was great.
After that, the plot took an odd twist and I just never caught up with it.
If you're using 3dsmax, you should look into the pdf by Gary Davis. I haven't read the whole thing, but he gives excellent insights to developing one, including the great advantages and common pitfalls of them. Google for it. He emailed it to me, but I think he has it back up on his server.
Tahya al-Moqawama al-Iraqiya!
Death to American dogs, and death to all the invaders in our lands. We will not stop fighting until ten million of you have paid the price for shedding the blood of our brothers!!
Tahya al-Moqawama al-Iraqiya!
Tahya al-Moqawama al-Iraqiya!