Slashdot Mirror
Renderfarm Setup Tips?
"In the hardware side, we still haven't made a choice between using AMD's Opteron or Apple's Xserve G5 (they have some very nice and price convenient cluster nodes which seem to be ideal for this kind of job), with Linux. As for the networking between them, is Gigaethernet enough or should we be going for Fiber? The software used to manage the render queues is another important point as well: I've been looking into Rush, and even though it's a commercial package, it works on all of the platforms we currently use (W2k/XP, Irix, OS X and Linux). But then there is also Dr. Queue, which is open source and is supported on at least the *NIX members of the aforementioned OS's. Other options include RenderPal and Pixar's RenderMan, but I would prefer an F/OSS alternative. Finally, it's worth noting that we'll be using the renderfarm for Maya and Adobe AfterEffects."
You might want to check out Cinelerra. It has pretty good support for renderfarms. I built one out of scrap 300mhz machines, and it only took a weekend.
(a) Wait for the next new processor technology to hit Slashdot.
(b) Build a Beowulf cluster of those.
(c) Profit!
Quidquid latine dictum sit, altum sonatur.
Check out g4u for deploying your render machines - it's a image based disk cloning tool that uses DHCP and FTP and which doesn't care what you run on your clients. (g4u itself is based on NetBSD, but that doesn't matter for the application).
I've used g4u to setup a ~50 node video rendering cluster, see my webpage on the Regensburg Marathon Cluster.
Enjoy!
- Hubert
Don't use macs. First off, you have less choice in renderers, and second, the hardware is more expensive. Rendering is grunt work. Buy cheap systems that you can upgrade more often, and run linux or something.
Macs are very nice hardware, but you really don't need that for rendering. For workstations they make sense, but for rendering you really want to have a lot of fast computers rather than nice computers.
Everything seemed to be going so nice
'till the end of all beings punched right through the ice
Why would you run Linux on the Apple boxes? Wouldn't OSX be just as good?
What render queue products are supported by your pieces of software? Why don't you try a few of them?
I'm sure a demo can be arranged.
I wouldn't go blindly marching in the direction of FOSS especially in something that is valuable enough to setup a renderfarm for.
Most importantly, find out what the people who will be using the software like and dislike about each package. And what works for them. If it saves you $30 per hour times 5 people software and hardware cost become insignifigant after one work week.
The biggest renderfarm in the world is useless if your people can't use it. Always remember that software is only good in its ability to meet the goals of the organization it supports.
We had a renderfarm for "The Chronicles of Riddick" of 40 boxes. Each box was a dual-proc Opteron.
We evaluated several render-queue management systems, and decided on Rush. The most persuasive arguments for using Rush were the very good experience we have heard from other users, and the simplicity of extending it to manage a variety of different tasks. I have to add Hammerhead to the list of happy customers. It did everything we could have hoped for. In particular, it was able to handle the inevitable crashing of machines pretty well.
While it's true that Rush is a proprietary, gotta-pay-for-it system; a robust render queue management system pays for itself very quickly in the ability to make your renferfarm productive. Perhaps a render queue manager is overkill when you have just 6 or 8 systems, but once you get up to 30 or 40 it is essential.
Our experience is all under Linux, but if you're going to be running After Effects that means that you're not going to be running Linux -- so there's not too much more I can help you with there. We did find that the dual Opterons worked much more efficiently than dual Xeons in multiprocessor rendering -- don't know about the Xserves, though. We were running mostly Maya, RenderMan, Shake, and our own in-house tools on the farm.
This farm is unfortunately powered down now that Riddick is done -- if you need some dual opterons, let me know at thad@hammerhead.com
I love Mondays. On a Monday, anything is possible.
Thats only if you desire maximum ease of use with minimum setup and running hassles. The same ease of use the regular G5's have is built into all their server stuff too. I'm sure the linux dudes will have something to say about that.....
I would take a really hard look at the ready made bio-information cluster they have all setup, and just load yer software as needed and off you go. But that's me. Some people seem to like futzing with computers.....After 20+ years doing that at work, I just wanna do what I wanna do when I wanna do it. Apple makes that easy.
I get paid to deal with headaches, I'm not gonna deal with them at home too.
My subconscious desire to get out of the computer field as a career must be surfacing--I read this as "reindeer farm". Then reality set back in and I almost made a lame "LapLANder" joke before tasering myself.
Recently, I have been working a lot with Apple's xgrid. We have been linking about 4 G5s/G4s together and getting impressive results. I don't understand your hardware situation, but if you are a Mac guy try it out.
GroupShares.com
-------
artlu.net
If you're going to do this, do it properly. Get systems with massive amounts of I/O that will cope with all the data you're trying to throw at them. For this kind of work, you need only buy from one vendor: SGI.
Don't bother with Intel/Linux, with dodgy hardware and the frequently-changing Linux code. Pay the money, get decent hardware with a support contract and a steady, stable, tried and trusted OS.
Apple *may* be an appropriate choice, now that Pixar have ported RenderMan to OS X, but I don't like the idea of my arrays running at 7200 rpm's. Get SGI, get fibre channel, and (possibly) get gigabit ethernet.
It'll all pay off - it won't be cheap, but in the long run, the results will be worth the money and the wait.
The first thing that you should do it find someone else to ask other than slashdot. Seriously, there are an extremely small number of readers who would actually know that answer to that, and none of them will post. You're just going to get a bunch of opinions from people who have no idea what they are talking about, like:
"I have a mac and I have editied video recorded on my digital video recorder. Let me tell you everything you need to know..."
"Beowulf cluster! Beowolf cluster! I have no idea what that means, but I'll say it anyways, because it seems appropriate in this circumstance!"
If you have real money invested on this, get serious.
I think that what you're looking for is a renderfarm for computer graphics rendering, right? in that case, you should be looking at PVM or OpenMOSIX or even MPI. In either case, since you're going to have more data crunching than actual data transfer, I think that even T100 would be enough. gigabit will be nice, but fiber is not worth it. Drqueue is nice... if you can get it to work, and I didn't. We used pvmpovray for many things, and I think that might be worth a look. pvmpovray exists for gentoo with an ebuild script, which would make the installation and configuration the minimum pain for it. But that option requires a conversion from maya to povray files. Also, I don't know what the pricing is going to be like, but if it were up to me, I'd take the Opterons, because I believe they are faster, although I'm not positive on that, and because I know they're well supported under linux, and again, I think that's a more personal choice to make, but the impression I got from AMD is that you always get the most for your buck. The Opterons also let you find replacement parts or upgrades a little easier than the G5 if you burn a CPU or motherboard. That's just my $.02 worth of advice.
---- I am certain of only one thing : I know nothing else.
Peace
First, that is a very small cluster with embarrassingly parallel compute tasks (rendering of individual frames), so you should be able to find the answers on your own using Google.
Secondly, from your questions it is obvious you're no rendering farm guru so why did the task of planning/researching the configuration get assigned to you? You should rather work with the users to collect and write down the goals (what is the purpose of this cluster, how do users use it, what are their expectations, how it's going to be managed, etc.) and find a small and focused SI who will propose you a better solution than you can come up on your own even with help of Slashdotters.
It's impossible to give a good answer to your questions since you've provided very little info, but in addition to the render nodes, one NFS node for file server and a single gigabit network will do.
Don't skimp on the software and don't use F/OSS just because you "prefer" it (Why do you prefer it? It doesn't sound like you're going to dive into the source and improve it or something.)
Instead of focusing on the coolness, focus on creating a user-friendly and maintenance friendly system. Do geeky stuff at home.
Light us up the bomb!
I think I'll stop here.
... but it was rejected. How do you deal with terabytes of data (50+ TB), all in a single directory tree, all must be accessible to every node? This is larger than what you can store on a single filer. Also, for performance reasons, the data must be separated across multiple filers. Currently we use lots of symlinks to tie it all together into a single logical directory tree, but that's a really ugly solution. There's got to be a better way. Right? Anyone?
___
If you think big enough, you'll never have to do it.
I don't know what the setup for After Effects and such looks like, but I managed to build a pretty good Sun Grid Engine system for distributing Maya batch renders. SGE is free, works on Linux or Solaris, x86 or SPARC (I obviously used the x86 linux binaries), and seems to be very well designed. I set up a pretty solid system for netbooting the clients, running them diskless (or with local swap drives), adding new clients on the fly (all scripted), and it all worked flawlessly. You could submit a batch job and it would distribute it per-frame as an array job to all the different nodes. Or you could just run SETI on all of them...
It's since been taken down in favor of running Alfred (because I no longer use Maya's builtin renderer, we've moved on to MTOR and PRMan), but I still have all of the files and scripts for it. If anyone's interested, I'd be happy to share: sabretooth@gmail.com
Every cloud has a silver lining (except for the mushroom shaped ones, which have a lining of Iridium & Strontium 90)
There is a fiber interface (Myrinet) to each node used by the MPI crowd, but our rendering group doesn't use it; they seem content with the performance of Ethernet over copper. Your needs may be different, of course, but latency isn't really an issue for rendering, and copper should provide all the bandwidth you need.
I'm not knowledgable regarding all the software packages you list there, but I'm wondering if any of them would really take advantage of a 64-bit kernel (either on Opteron or G5-PPC970). Of course you can put a PPC version of Linux on the Xserve, but not without sacrificing nearly all Apple-provided management. If you expand the cluster to a large number of nodes, or even if you keep a small number of nodes but place it in a remote location, Xserve running Linux would be painful to manage (no remote power-off/-on, remote console problems). Xserve is shiny and has the requisite blue LED's, but and AMD or Intel box (from the right vendor) would be much easier to manage remotely.
Unless you have plenty of cash to blow. In my experience with clusters the network is rarely your biggest bottleneck if you're using GigE. Rendering should be parallelized enough that it won't be saturating the network.
For one, consider creating a standard image for all of your machines and have them install using pxeboot or etherboot. If any of the machines get hosed, you can reboot them and force them to reinstall their OS in 5 minutes.
It seems to me that fiber is a waste of money. You implement gigE using copper. I would think that most of the data transfer is going to be the scene data and then an image transfered back.
The company i used to work for was developing a global illumination raytracer and we created a program that tiled the output image; however, it still needed the entire scene so it could calculate all of the lighting and rays. I don't know what rendering software exists nowdays so I can't give any advice there.
(I think this image was created using our renderer using environmental lighting because i vaguely remember this scene (while we were fixing bugs for them) from this company before we went under in 2002)
My question is - when is someone going to make a blade with PCI-Express and enough room for the latest batch of cards from NVidia and ATI?
Seriously - this is going to become a huge issue, as more rendering is pushed out to the stream processor that is the GPU.
Education is the silver bullet.
We have a 50 node dual 3Ghz render farm, 20 on w2k and 30 on linux 2.4.20 (ok Redhat9) We are currently using Muster www.vvertex.com, but are not that happy with it. It was reasonably priced, and the developers are very helpful, but its not quite there yet. I have been seriously considering building my own using a SQL database (currently Postgresql, but may swich to MySql) and perl. A render manager is really just a database with a bit of network sockets and scripts to run occasionally. A simple concept that is probably going to come back and bite me :)
--
This is a bit off-topic, but William Gibson put forth an interesting idea about rednering farms in his most recent novel, Pattern Recognition. [This is something of a spoiler, so if you are planning on reading the book, stop reading.]
At one point in the book, the main character discovers a huge rendering farm in Russia that is staffed entirely by white-collar criminals who are "doing time". Since Russian prisons are a well-known hotebed of communicable diseases, this "Dream Factory" (as it is called) is referred to as the only prison in Russia that prisonners are trying to get inside. They are forced to work all day, but in exchange they can live comfortably without fear of the infection (or some other grisly fate) that would befall them in an ordinary prison.
Personally, I don't find this idea so far-fetched, especially in places like Russia or even the USA where the prison systems are under tremendous strain and there is a huge amount of untapped labour that would likely be happy to work in exchange for a guarantee of safety and sanitary conditions.
...Whether my Maker is prepared for the great ordeal of meeting me is another matter.
Churchill
Sure! You have a choice of match, lighter, flamethrower, or handy-dandy C4 block. =)
There is a distro called dynebolic which is specifically for multimedia. It features Veejay, Cinelera, MJPEG tools etc. It is supposed to run off the CD. It is supposed to use every computer hooked to the network that is running dynebolic in the manner of a render farm. That is a very attractive idea. Go around the house, put in dynebolic CDs and go nuts.
I can't say that it is great because I haven't been able to do much of the above. Maybe you will have better luck.
Just catch a plane down to New Zealand, kidnap Peter Jackson's iPod, and tell him he ain't gettin' it back until he signs over Weta's setup to you as sole proprietor. Sheesh - all this talk of hardware, software, fiber/GbE, and the like is waaay too complicated.
I also reply below your current threshold.
I would suggest you look into a product specific rendering cluster. For example, Alias Maya, Newtek Lightwave, etc... I have a render farm with all of the render clients on the same machines. So whether the project requires a specific application I can use the same computers to render.
.02.
My
Who else can get two +5 posting from a single idea simply by failing to include a link?!
Bow before the kramam whore master!!
The journey is better then the end.
SGI is history in this biz... There is no way to cost effectively use those boxes anymore. I work at DD and we use a linux/win2k renderfarm. Only thing SGIs are good for is Flame... Also, since this guy is running After Effects, he's either Win or Mac.
You know, some of us aren't here because of the oh-so-important /. karma. Many are here to keep themselves informed and educated. How's about giving the guy the benefit of the doubt and go render yerself a chill-pill?
Take note: I'm logged in and not afraid of the big bad flamebait mod.
to kick you in the balls for making that movie...
The cheapest and easiest way to set up a renderfarm is to use someone else's computers!
We used to run Lightwave for work-related projects, but we only had a few machines allocated to the media lab. So after hours, we'd sneak around the cube farms and pop a client boot disk in every machine we could find.
http://www.gridironxfactor.com
So, get someone with mod points to mod me down. I don't lose sleep at night over my slashdot karma.
It's quite common. I'd even say it's valid. The +5 cap is the reason why.
Hey
I've set up and administrated a number of farms over the years (doing it as I type. its.. what I do). One thing you really want to do, certainly with Maya's renderer, is to try to use the same OS and platform on your farm as you use on your user workstations. There can be subtle or even obvious differences in the render output between OS's, and since you'll have enough issues to deal with you'll want to keep cross-platform incompatabilities out of the mix. Please, trust me on this. Had to deal with Maya Irix/Win2k/Linux differences in the past.
As for queueing software, give Condor a look-see. Free and functional. I reverse-engineered a Perl version of it before they made their source available, and my version has been run quite successfully at several animation studios and an effects house over the years. It's a well architected system for distributed computing.
Feel free to contact me if you've got any other render system or management questions. I'm always interested in seeing how other studios approach the challenge.
"We're in the process of acquiring and setting up a renderfarm, and I'm hoping the Slashdot community might light us up a little here."
Sorry Slashdot is NOT a power utility!
rendering is processor intensive (usually) and not io intensive so you will probably do just fine with plain old 100 mbit ethernet. take the money you save on networking and spend it on more nodes.
Try a cluster file system.
"Filers" create "hot spots" whereby often-accessed directories/files create IO bottlenecks.
I think you can use this CFS to create a directory tree with over 200TB of data (/home/lun1, /home/lun2, .../home/lun255). You can't "tie" them together like with LVM but you do get huge throughput as opposed to a single-host bottleneck with a volume manager and/or clustered NAS filers with the hot spot problem.
Everything you have mentioned is dependent on what render engine you are using.
First, figure that out, then go to the forums for that engine and ask your question there.
I just built an 8-node renderfarm for using Vray, which means 3D Studio, which means Win2k and it was a piece of cake.
6-8 nodes is tiny. Figure your use, then ask on the right forum. Many, many people have done this already.
I'm out of my mind right now, but feel free to leave a message.....
This is a fun idea in the abstract, but if you're looking for concrete advice, you need to give us some concrete data. Most importantly, -what- are you rendering?
If you're at a university and you're doing some sort of bioinformatics visualization, use whatever the researchers are most comfortable with. The odds are good that this is whatever the CS department was teaching on 5 years ago. Probably Suns or Windows machines. Slave... errr, grad student labor is cheap, so use an OSS scheduling and job management system if you can.
At most other places, a similar rule applies: use whatever the users are most comfortable with. If you're using Mac workstations and software, then it may make sense to go with a G5 rendering farm. If you're using Windows... well, okay. Windows render farms still suck, but at least buy PCs to leave your options open. Unless you're a really large organization (that is, the sort that doesn't have to resort to Ask Slashdot for research), you probably want to use products that come with support contracts. $20k/year is a pretty good deal when compared to keeping a full-time support person for the same task.
--
Here is a fresh After Effects grid rendering tool just released last month for both mac & pc. Havent tried it yet, as I kindof laugh at comp render times when compared to 3d rendering. http://www.gridironxfactor.com/index_home.asp And this is the render manager we use, thats worked like a dream since its first install. We use it for Lightwave, but there is the option for Maya as well (even though mayas native renderer is 8P)available for PC,MacOSX,Linux http://www.liquiddreamsolutions.com/
We did this because we primarily use Discreet's 3dsmax (with Brazil and V-Ray) and Eyeon's Digital Fusion. We have found that most existing render farm solutions do not support these two packages very well -- thus we decided to develop our own custom solution. We also support After Effects, Alias|Maya, AIR and other RenderMan compliant rendering packages.
Of interest to the general Slashdot crowd may be that this Deadline Render Management Solution is based on the open source (BSD License) Exocortex C# library originally released with this C# 3D Engine. Deadline is built with C# in the hopes that using Mono we will be able to start supporting Linux with minimal extra effort.
I'll be reading all the posts on this Slashdot thread but I would also appreciate any direct feedback on our current beta product. We also found solutions such as Rush and Smedge to be less than user friendly in many respects. Thus we have tried as best as we could to increase a 3D package that is not well supported by most render farm management solutions -- except for Discreet's Backburner (which we found not that that scalable.)
IRUSH
BOXX nodes
Make sure that you have software (or compilers) available that will make use of the vector processor on the G5. If you don't (and it sounds like the stuff form Pixar and such should), it would probably be more cost effective to go with some other system running linux.
The benefit to this is that your consolidating your storage while removing any data/disk traffic off your network. SAN's offer much better fault tolerance and its WAY easier to administer.
Backups become much easier to handle when your data is on one spot and when it comes time to expand, your options are virtually unlimited (depending on your initial solution).
Apple's G5 server is a good packaged solution, but will limit you when it comes to any type of large expansion plans. Compaq servers work well and they also offer an excellent SAN based solution.....and no, I dont work for either Compaq or Apple!
No sig here...
OS X Server can run without a GUI. Imagine that.
ok I only have experance with a custom T&L rendering platform but the grid engine worked very well I have used it at a couple of hardware design places and it's CPU blancing was very nice indeed
it supports
Apple Mac OS/X
Compaq Tru64 Unix 5.0, 5.1
Hewlett Packard HP-UX 11.x
IBM AIX 4.3, 5.1
Linux x86, kernel 2.4, glibc >= 2.2
Linux AMD64 (Opteron), kernel 2.4, glibc >= 2.2
Silicon Graphics IRIX 6.5
Sun Microsystems Solaris (Sparc) 7 and higher 32-bit
Sun Microsystems Solaris (Sparc) 7 and higher 64-bit
Sun Microsystems Solaris (x86) 8 and higher
see http://gridengine.sunsource.net/
I would look at what software you want to run BEFORE buying hardware
regards
John jones
First off, the G5 XServe's are very fast machines and the cluster node is, of course, designed for just this sort of thing in the minimal amount of space. You should be aware that they are LOUD as hell though... louder than the G4 XServes, which sound like a plane taking off. That said, plan on hiding them away from your workspace because even one of them in the same room with you will drive you nuts. This is no different than a 1U server from Dell/Whoever though.
You'll only really need to buy a single non-cluster-node XServe to make your system work, and even that isn't needed if you have a desktop Mac to admin them from. Make sure you invest in either Apple's Remote Desktop application, or you install a VNC server or something on each of your cluster nodes. You'll want to get access to the desktop, and with those machines sans-video card, a remote desktop application is obviously the only way. For your type of application, sometimes the command prompt can only get you so far. :)
That said, if you do go with the Apple solution, there is not much reason to run Linux on them. Most open source apps you can build for Linux will build on OSX without issue, and keeping OSX on the machines gives you the ability to run Adobe After Effects on the machines and use the After Effects built-in cluster rendering. It's not that I dislike Linux, but it's just not needed here, especially when OSX offers you the addition of a whole host of commercial applications over the usual open source ones. Also, the cluster nodes include OSX Server (5 User). The 5 user licenses are for the filing sharing functionality, so they don't apply to what you're doing and for all intents and purposes, it works and acts largely like the OSX you know and use everyday, so the OS is included in the price of the hardware and you're not paying an extra invoice for it.
As for interconnecting them, the XServe machines all feature dual gigabit ethernet, which you'll want to obviously connect to a gigabit ethernet switch.
That said, you're left with the option of storage. The best bet for a smaller scale cluster (6-12 CPU's, as you mentioned) would be just to dedicate a single non-cluster-node XServe with all 4 disks installed and striped into a single volume. Connect the second gigabit ethernet port of your Xserve cluster machines to a second switch and then use NFS on OS X server to share to your cluster nodes. This will provide plenty of performance over a seperate gigabit ethernet connection, though your mileage may vary depending on what you want to do.
Obviously, for more machines, you'll do well to look into more advanced storage. XSan can do just this once they release it, which I imagine it will be when you're ready to scale up that far.
Hope that helps, at least as far as Apple solutions go.bash-3.00$ uname -a
SunOS panda 5.10 Generic sun4u sparc SUNW,Ultra-2
But more to the parent's point, OSX could be run Darwin-only, AKA BSD-style command-line interface. The bloat can be stripped out of OSX just as easily as it can for linux. Additionally, the user would have the comfort in knowing that his render farm is using the same OS as the workstations used to control it.
As other respondants have suggested, I guess it would come down to which OS supported the entire collection of desired applications for the job.
Apple has not yet released a true 64-bit version of OS X, while Gentoo released a PPC64 version a few weeks ago. If you're going to buy 64 bits of CPU, you might as well get 64 bits of OS too.
We'll use 6 to 8 nodes first
:)
Sounds more like a 'gentlemans renderfarm' than an actual working one
On the cheap we have created a small 12 node cluster using PlumpOS for the drone nodes and Gentoo patched for OpenMosix as the master node(s). On the hardware side we're using a mix of Linux compatible PC's (P4's @ 2.4-3.0 GHz) all connected via Gbit NIC's and a Gbit switch. Took about 8 hours to setup. The system is very stable and fast.
but I do work in a 3D department at a television production studio. I can't say Rush is the best, but we use it and the animators are generally happy with it. We have about 10 render only boxes and another 9 or 10 workstations that Rush let's you "online" when you are not using them (lunch, nights, weekends, long meetings...).
Stupid sexy Flanders.
Mod parent (selfabuse) up. It was an on-topic defense to the grandparent's accusations, after this guy made two very informative and interesting points. Don't punish him for being helpful!
These are just some tips I've heard in my 4 years of experience.
Whichever processors you go with, make sure the entire farm uses the same type. Otherwise peculiar rendering differences might occur, in things like particles, hair/fur and fluids.
I suggest going with the Opterons just for the PC compatibility. While the CG industry is becoming more diverse hardware-wise, it is still dominated by PC's and to a much lesser extent SGI boxes (5 years ago it was all SGI). Using PC's keeps your options open. Perhaps someday you will find 3ds max and its included distributed rendreing software more suitable for a task, and that can only be used with PC's. Same goes with the Mental Ray and Brazil renderers and the Combustion compositing software. Macs just have not been widely used in the 3d graphics industry, and so the vast majority of 3d content creation software is PC and SGI only (Maya Unlimited is only available on PC and SGI, while a lower end wersion is on Mac). And VirtualPC cannot be used to emulate 3d hardware acceleration (and it shouldn't be used for anything processor intensive anyways), though this only applies to the hardware rendered viewports in the apps. Having only Macs would be risky, and could limit your capabilities significantly.
Pixar's PRMan (Photorealistic RenderMan) is a full blown renderer, not just something to help distribute render jobs. It is generally considered the best in the industry, though MentalRay and Brazil have gained significant followings. For a cheap but effective render queueing system, check out Smegde. Smedge was used by Manex Visual Effects for handling some of the effects shots in the Matrix trilogy. If you're running the Linux version of Maya (x86 only) it is not too difficult to distribute the render tasks yourself using shell scripts and the command-line renderer.
GB Ethernet should be fine, the bottleneck will be in the actual image processing not data transfer rates. 100Mb ethenet might even get the job done, thught I'd use GB for the added speed when copying large files. YMMV of course.
Overall I'd try to create a very flexible system, one that will definitely support the newest CG software down the road and one that ensures compatibility with everything, for those always short deadlines. Goodl luck with your rendering.
This attitude bothers me, and not for the first time here on Slashdot. How the hell do you think those experts got to be experts? Do you think they just *poofed* into being with all their knowledge and skills already existent?
By RTFM'ing and Googling.
It's probably faster to search the web than submit to Slashdot, hope its accepted, and then sort through the replies.
Be careful how you configure your network or you may end up with something like this.
Test 1 2 3 4
Good stuff, much easier to set up and maintain than that SAN/NAS crap.
http://www.isilon.com/
And no, I don't work there.
DyneBolic
How are you ever going to get Karma?
Why is the parent modded flamebait? He is absolutely right. Renderfarms are a highly specialized use of computers. You don't consult a general pratitioner group on brain surgery. Why consult a general technology group on renderfarming? The best you can hope for is that someone who knows a little something about it can point you to a good resource somewhere (which you probably could have found through Google anyway). The worst is that some idiot with no clue will give you bad advice and screw your setup.
Well quit mixing your media here. What do you mean GbE is enough, it runs over both copper and fiber. Now there are other layer 2 protocols that run over both copper and fiber (all though different cables) for example - what SAN are you going to use ? iSCSI, FC, SRP, NFS ?
What networking are you going to use ? GbE, TokenRing, FDDI, ???
What HPC/Interconnect are you going to use Infiniband, Myrinet, VI, ???
Some of these are the same networks, some of these are not, they all run at various speeds with various types of cabling. Now go ask your question again
PS GbE is probably good for networking, probably not for your SAN
I have mod points and I am not afraid to use them
All your Linux are belong to us!
You could also hire competent unemployed systems people to help out -- like myself!
Zhrodague.net - I do projects and stuff too.
You might want to look into nVidia Gelato. It's a 3D renderer that uses Quadro FX cards as secondary FPUs, supposedly doubling or more the speed of rendering. They claim it's two to six times faster than the leading renderer. There's a demo, so you can verify those claims for your uses.
It runs under Linux, and "will function with whatever [render farm] management system you currently use.".
To reiterate, it's a SOFTWARE renderer, that is hardware accelerated by using the video card as a co-processor.
I do my video editing work on a old G4 using FCP 3.0 (I know, I'm behind.) Occasionally I get stuck with 8 hour renders that generate 6 gig or better render files.
Is there any cheap solution that will let me spread this work out? I don't use AE at all. My shop is incredibly cheap so it would have to be a free or nearly free solution. I know that's asking a lot. It just isn't worth it to get more macs and put FCP express on them. That's the only thing I can think of.
Any tips would be greatly appreciated.
Why do I have this? I don't smoke.
use hypertransport, or infiband. dont build some POS cluster. Build something that has decent hardware, and have fun doing it!
This is surely an extreme situation that requires extreme action. Nothing less than the budget-friendly P4EE.
imp is a POV-RAY based distibuted renderfarm for Win or Lin.
Firstly, I would recommend PC hardware over MAC, you can get your hands on really cheap used / off lease PC's. A pIII 500 will run you somewhere in the range of $150 - $200 (Canadian). I have been using an assorted collection computers, both of the cheap variety and free computers scrounged from friends and family, to render my 3D animated series (which I'd plug but I dont need a slashdotting ;).
r view.asp">plugin for after effects 6 that lets you use all your CPU's together in real time while you are working in after effects to generate preview frames while you work, and to render.
Your render farm computers don't need keyboards, mice, sound cards, cd rom's or monitors because you can remotely admin each using UltraVNC.
3 pentium III 500's are basicly equivalent to 1 p4 1.4 ghz. It all depends on how many computers you want to have on at one time sucking back the electricity and heating up the house.
3D Max has a very usable network rendering manager, I'm not familiar with solutions for MAYA or Renderman renderers. There is a a href="http://www.gridironxfactor.com/products/ove
I work at a small post production facility in Hollywood. We have a render farm of 6 dual Xeon Win2000 Boxx rack machines as well as five dual Xeon Win2000 Boxx workstations that render in their spare time. We run Maya and After Effects and we use Smedge to handle the distributed rendering for both maya and AE and Mental Ray. We also have another Dual Xeon box running Server 2003 with a eight drive raid setup with two main partitions, one is raid 1 for the maya scenes and the AE projects and the other is raid 0 for the rendered frames and comps. We started with just two rack machines, but we add one or two every couple months when the budget permits. Our renders are sent to Smedge via a script that we run from an Interix csh. It greatly simplifies the process of sending out renders. Each project we work on has a script with the name of the project, or the shot, and when we're ready to render we run the script which parses the text file with all the parameters for the render such as frame range, render quality and so on. Some people don't care for Smedge to much, but it gets the job done, and works for most all 3d applications as well as most compositing apps.
Personally I think openmosix and a IBM PPC bladecenter woulbe be the ideal option, but as I recall openmosix hasn't been ported to PPC yet. In the short term, I think the Opterons will have better linux support, but IBM certainly has the resources to change that if they want to.
Consider with the bladeservers, you can fit 14 SMP blades into a 7U bladecenter, double the density of using Apple's offering. IBM's PPC blades are slightly cheaper than Apple's xserve too. Not to mention that the PPC blades cost about as much as the Xeon ones, and they actually come w/ both procs.
I don't even want to imagine the power bill for 28 P4 Xeons @~200w each!
We're replacing our a**-slow "Platform LSF" commercial cpu farm management software with GridWare / Sun Grid Engine. Its great! Grid Engine performs operations on jobs in an instant just as if you were running them locally (*unlike LSF which is too slow at common tasks like job submission or job killing to be useful*). its also free, with support from sun available if you feel like wasting money.
1) determine what kind of problem you want to solve with your render farm
2) determine what software will solve this problem
3) determine what hardware supports the software
4) hire a competent admin to make it work
I know little about it myself but i think apple's Xgrid technology could be worth a look.
Get a couple of Skids of X-Boxes, good bang for the buck and who could resist saying the next big thing was rendered on 100 X-boxes via Linux/OpenMosix .... sigh....
I know the original poster said Maya, so there happens to be an inexpensive renderfarm software called Butterflynet from liquid dream solutions (liquiddreamsolutions.com). It supports both Maya and Lightwave; I use it here at Long Island University quite effectively for rendering student projects on unlimited nodes. It's fairly stable, though your mileage may vary. I think its also available for the MAC platform as well.
It handles crashed nodes fairly well, it doesn't handle crashed machines too well. It is able to run in the backgroun on low priority, so that someone can be using their machine with little slowdown in their user experience.
For a "Render Farm" manager Here.
Just thought this was an appropriate discussion. Also, maybe a "slashjobs" would be an interesting addition to this site.
Stupid sexy Flanders.
Regarding networking: you have to look carefully at the way the farm will be used. If you are doing any kind of compositing (which requires high I/O rates), you'll benefit from gigabit ethernet. You'll also benefit from gigabit if you have exceptionally short render times (less than 30 minutes per frame), since in this case I/O is a significant fraction of each frame's render cycle. But the longer your per-frame render time, the less necessary gigabit is. We've always used 100base and it still serves us well. Fiber is expensive and provides nothing you'll need that copper can't provide.
The individual machines should have identical configurations and be interchangable. Your goal is to not care when an individual machine dies. In light of this, there should be no local storage of data. You can save money on support if you buy spares instead of service contracts. Warranties also work, but the big manufacturers give their worst service to warranty-only customers.
Don't wire anything but ethernet to the machines. KVM wiring is expensive and unnecessary. Each machine should run unattended until it dies; when it does, you can wheel over a monitor and keyboard to diagnose it.
Opterons are fast, compatible, cool, lower-power and cheap. Xserves are nice, but we've found that Darwin doesn't integrate well into a pure Unix environment. You'd also be locking yourself into a single manufacturer.
Linux is cheap and effective, and easier to configure correctly as a server OS than as a desktop OS. There is so much commercial software available for it now that there is little reason to consider Windows or a commercial Unix. We haven't found Linux support from the big manufacturers to be all that great; if you use Linux, assume that you will have to solve most problems on your own.
I work at a small post production facility in Hollywood. We have a render farm of 6 dual Xeon Win2000 Boxx rack machines as well as five dual Xeon Win2000 Boxx workstations that render in their spare time. We run Maya and After Effects and we use Smedge to handle the distributed rendering for both maya and AE and Mental Ray. We also have another Dual Xeon box running Server 2003 with a eight drive raid setup with two main partitions, one is raid 1 for the maya scenes and the AE projects and the other is raid 0 for the rendered frames and comps. We started with just two rack machines, but we add one or two every couple months when the budget permits. Our renders are sent to Smedge via a script that we run from an Interix csh. It greatly simplifies the process of sending out renders. Each project we work on has a script with the name of the project, or the shot, and when we're ready to render we run the script which parses the text file with all the parameters for the render such as frame range, render quality and so on. Some people don't care for Smedge to much, but it gets the job done, and works for most all 3d applications as well as most compositing apps.
FILE SERVER
For a large farm, the most important thing is a strong file server that can handle the load you're going to throw at it.
Netapp is a great choice; many production companies, small and large have switched to netapp when all other PC solutions failed (W2K servers, OSX servers, Linux servers), and their problems went away. Netapp is pricey, but you can find used equipment, or rental solutions.
IMHO, linux/windows is great for the nodes, but not as file servers. Linux especially has had its NFS services beaten into submission when you get a 30 host (60 cpu) farm beating on the one linux server over NFS. Whereas an SGI Origin rack mount or Netapp server can handle that and more (100 - 200 hosts).
A problem with Windows servers is that they license connections. If you try to run a Windows XP or 2K system as a file server, you'll quickly find the OS won't let you share the drive to more than 10 machines. You have to run 2K or 2003 Server to server more than that, and you have to make sure you have enough client licenses to serve the network as you grow it. Linux servers (and most other servers, including SGI) have no such connection limit on NFS.
LOAD BALANCING
When you start getting above 100 hosts, consider using a switch that can handle load balancing, so that when the farm is rendering hard, the workstations still get high priority bandwidth to the disk server, otherwise loading a tiny text file from a workstation can take minutes when the farm is being heavily utilized.
With Cisco switches, the terminology is QOS (Quality Of Service), made popular by VoIP services which deal with high volumes of network traffic, which happens to be a similar problem to file system traffic on a render farms.
There are usually two bottle necks; the network at the file server, and the file server itself. Rarely are the workstations/nodes a bottle neck.
SUBNETTING
Sometimes it pays to have multiple network segments all hanging off the same file server, where the server has separate network cards for each segment. This way you distribute the network traffic, rather than concentrating it all at one point (the server's network card).
One way is to put all the workstations on one network, the farm on another. Or split a large farm into separate subnets. You may be able to then use the server's own config to balance the network packets so it gives higher precedence to the workstations, and lower to the farm, so users can access files quickly, no matter how busy the farm is.
AIR CONDITIONING
Especially in hot regions like Los Angeles, it's very important to have a suitable A/C system to keep the machine room cool. Often people have to scramble to upgrade their A/C system when their farm starts heating up, and find out there's no physical way to upgrade their A/C due to the limitations of their machine room, or adequate exposure to the outside for ventilation.
You sysadmins probably want to have a pager system that alerts you when the machine room starts heating up, or a script that shuts down the farm if the temperature exceeds some maximum.
Watching the temp of the cpus is a good way to detect dead or dying CPU fans, the major cause of outages on nodes.
USE AN EXPANDABLE SWITCHBOX
Many of the Belkin switch boxes are chainable, so that you can have literally hundreds of machines accessable from one monitor/keyboard combo.
Get a good expandable switch box so that you can see the physical console of each node, in case the machine(s) blue screen, panic, etc. so you can see what the problem is with a box, so that you can solve it, rather than just rebooting the box regularly. Often problems are obviously advertised on the console, but a cheap farm setup where there's no easy way to see the console of each box (ie. "lets just use remote desktops!") looses the ability to see what's wrong with a machine when it dies. Get to know how to read a BSOD, so you can determine if it's a ram problem, or a driver problem.
COPPER
With Linux you will never get freedom from the swarms of flies that follow you.
Or the Open Sores that will infest every inch of your dirty hairy communist body.
No, you must march in the direction of FOSS.
Your battle cry can be "GNU GNU LONG LIVE GNU!"
How could you possibly sleep at night if you dont use GNU/Linux? Just think of how you are improving the lives of the poor people in India by using GNU/Linux.
AND THIS IS THE YEAR OF LINUX ON THE DESKTOP!!!
"Welp, your post sparked some debate here at the office"
And it would have sparked debate here too, but we always seem to get a message that our subnet has been banned.
Here is the location of some source code for establishing a network renderer:
reppu-0.1pre1.tar.gz - [size 12556 bytes]
Here is the page that goes with it:
Network Rendering Program
Although it was originally written for use with Blender, I know, for a fact, it has been modified to use with other 3D packages, with some tweaks in the code.
I run a render farm, but don't want to get into it because of how things are moderated here; however, it consists of SGI, Mac, Dec Alpha, and Linux boxen and works well.
I use most 3D packages with the exception of SI...due to cost... and have run a modified REPPU with all of them.
REPPU was originally written by Upi Tamminen.
And Linux is the r0x0r! Everything should be done in Linux!!!
You know what? We will never use Linux here! Why? Because every time some IT nerd infested with flies comes into my office suggesting Linux I laugh at him! Its quite funny. I pretend to listen to the cost savings that Open Sores will give us, let him mumble and fidget for half an hour, then in the end tell him that we are a Microsoft only shop and we dont use software stolen from the likes of SCO. He then mumbles something about how Linux is not stolen, it is open source, the community supports it blah blah blah. I laugh at him and tell him to have a good day.
I don't know if this will help out but for what it is worth.....
/24 networks connected to a Foundry switch, as is the server. The server is connected to the main array via 4 2G FC, 2 active, 2 passive. I want to install a FC card in one node per farm network and have the remaining nodes in that network request files from it instead of the sun server. Since it has a direct connection the the array (84 disks in RAID 5), my hopes are my NFS issues will be a thing of the past. My only concern is the bus in the Sun box. I may look at new hardware, I like what they have done with the Opteron Architecture from a main board perspective. The can move massive amounts of data around. Its all about IO..
I take care of a 400 cpu render farm at work and found the biggest problem was NFS timeouts due to a very busy server. My point being the conduit between the farm and remote file system should have lots of thought invested. We run gigabit between the render nodes (linux) and the server, (Solaris on a quad 480) attached to the core switch via a pair of bonded gig links. Depending on the job and size of textures, et al., the server would have issues handling the load/requests.
Once the current main project in production is finished, I have plans to change things. Currently the farm set up has 6
For queuing we use custom software, prior to that we used a product called "Muster". It worked well enough, but it would not do path substitution on referenced or included files.
My take on dual CPU setups, Maya, MR, RM, and most other current offerings have some level of multi threading. At one time I sent 2 jobs to each machine with each cpu working on one, now I send one job and let both CPUs work on it. We didn't notice a huge difference in render times, but it did help with the NFS issues a bit, and that is what we were after at the time.
Should you go to fibre? No not unless you have distance issues. You are actually talking about the media the data travels over. The difference is light vrs electricity. The biggest advantage to using fibre is the distance you can go with out the use of a repeater of some sort. Gigabit is gigabit be it over air, glass, copper or any other media. Always use switches if you can.
If I was setting up my first render farm, I'd use the cheapest current hardware I could find (within reason). You are after CPU more than anything. Good memory, but it doesn't have to be the best, and of course a gigabit port. Reason I say this is with cheap hardware it doesn't hurt the pocket book as much, and with the savings you can throw a heck of a lot more CPUs at the solution. Also doing a 'forklift' upgrade will not hurt as much. Keep a few spares on a shelf, this should result in a quicker turn around time than a RMA. And the money you would save on the support contracts could be used toward the spares. I like Apple, I think their SAN solution is very "neat". But I would not use them for a render farm solution unless there was lots of money for the project and you *know* it will support all the render engines you plan to deploy. Running linux on an Xserv is a bit of a silly idea if the thinking is to run something like 'maya for linux' as it is not compiled for powerPC. OSX has more render client support than linux on powerPC.
Good luck with things,
greg
As a professional After Effects designer, I'll just remind you that a 2D compositing program such as AE will have different demands to a 3D raytracer such as Maya. AE will render in seconds per frame, so the overall render times will come down to network bandwidth and storage efficiency. There are some specific, unique situations where After Effects will render faster on a single machine than on a network, because the transfer time for all the render assets may be longer than the render itself. Depending on the complexity of your renders and the age of your machines, I'd say it would be unusual for AE to spend for than about 10 seconds on any single frame, and some simple compositions will render at about 1 second per frame or faster. So a network set up specifically for AE will rely heavily on throughput. Maya renders will be much slower, several minutes to several hours per frame, so network traffic is not going to have an immediate effect on render speeds in the same way. But the assets required by Maya can be much bigger - you might end up with several hundred meg. of texture files. Again, it's possible with highly detailed renders that it can take more time to send data over the network than it does to render it. Pixar have had this problem and had to recode the parts of their software dealing with cacheing etc etc. Personally I'd go for the Mac solution, their stuff just works.
Putting syrup in coffee is some form of blasphemy.
the best render manager ive used is Rush...
its runs on every platform, and support just about every app your ever likely to use..
as for going for G5s, if your planning on rendering AE material then forget about running Linux on them.
if you planning on using them for Shake then savings made on render licenses can make them the cheaper overall option..
Imagine a Beowulf cluster of renderfarms...a 2D cluster!
Yo, I and a cohort developed a business plan around a render farm business just before the dotbomb (thus, why we didn't actually go after funding). We have a substantial amount of research done, although the stuff is three years old now. If you want any of this info, just let me know. Email me at ryan(at)suydams.com Just a quick note, a massive array of disposable nodes, connected by simple ethernet (GigE) is plenty for most rendering jobs. You gain little by having a massively parallel system image, but the cost goes up tremendously. As the rendering batch jobs are naturally threaded (i.e., each frame of the animation can be rendered as it's own work-unit), you will find just a simple LAN will be plenty sufficient for your needs. Finally, check out www.boxxtech.com as they have some awesome rendernodes already built (on Xeon or Opteron platforms).
On the flip side Linux enjoys having engineering programs like Fluent and Nastran, which OSX doesn't have support for yet. If the programs you need only ran on Irix, Linux or Windows then that is hardware platform you would need to acquire.
A possible concern with the apple solution is continued support. Adobe is not going to be putting too much effort into supporting OSX with Steve Jobs developing competing applications. If you were going to run Apple's new compositing app, the G5s would be the way to go. While Linux cannot run AE, the hardware can, and you could boot the proper OS for whatever application needs rendering.
I have to say I really liked the movie for the first 20-30 minutes. The SFX and environment was great.
After that, the plot took an odd twist and I just never caught up with it.
At the risk of being blatantly self-promotional, check out qube from www.pipelinefx.com - it's based on the system we used for the Final Fantasy movie, and is now being sold as a commercial product to film, TV, and game companies. Distributed processing will form the backbone of your pipeline, and should be carefully considered and evaluated. Take a look at all the different systems out there, and think about your growth and future needs - you want a system that can handle anything you think to throw at it.
If you're using 3dsmax, you should look into the pdf by Gary Davis. I haven't read the whole thing, but he gives excellent insights to developing one, including the great advantages and common pitfalls of them. Google for it. He emailed it to me, but I think he has it back up on his server.
Tahya al-Moqawama al-Iraqiya!
Death to American dogs, and death to all the invaders in our lands. We will not stop fighting until ten million of you have paid the price for shedding the blood of our brothers!!
Tahya al-Moqawama al-Iraqiya!
Tahya al-Moqawama al-Iraqiya!