I don't think that there is curently a 3D video that has been done purely on Linux systems yet.
I know that our video's will be about as close as we can get. We animate and model on Octantes. I would say 75+ % of the frames are rendered on linux and the rest on our SGI boxes. We use only SGI's to create video to import on our NT AVID system.
I applaud all of the work that SGI has done with the linux communtiy and I wish that we could get some more SGI boxes in for quick, single frame renders.
I still think that linux is the best choice for a render farm, though. Small, fast, cheap. I like to treat my boxes as appliances. Maya files in, frames out.
I agree with all that you say, and I thik that SGI boxes are still needed, but less and less.
I doubt that this will be used for rendering or animating or anything in the traditional sense.
I see the system more used as an interface to displaying precreated models interacting in an environment. Similar to 3D video games.
What you should be able to do with this system is load in some textured models, lights and the like, like there are in any game, and have it "hardware render" the frames as you watch. Like any 3D game. The main difference is that now you would have 16x's the power of the PS2 so you could handle more objects, more poly's or maybe larger textures, but that is doubtful.
The output could be a digital signal that is then read by a system that can display it real time, if needs be, or to store it for later viewing.
I am unsure why they are going this route but it seems like a no brainer if you have everything you want mostly pre-defined. You can use a package like A|W's Maya and create everything you need, throw it in the PS2 and play with the objects.
I don't see this being used in a tradition 3d animation sense. The textures and models need to be of a higher quality than what I see the PS2 being able to provide. The motion needs to be key framed so the animator can go back and tweak movement to fit the mood or the sound track.
You could use this to produce "good enough for web" videos or live content but I see that as being the limit.
ALthough A|W announced the Maya Linux port and the renderer is already there, this would be great for a render farm.
I got to use a 4 node Origin 2100 for a bit until it replaced the 2 node Origin we had for a file server. It was the fastest renderer I had. Great for single frames and good for multiple frames. If a company needs large frame renders or single frames done fast, a NUMA style machine is needed. Having the large capacity of memory alos alows several smaller render jobs at a time.
A machine like this would be a dream for me.
I don't think that the cost/performance metric would pay off for the type of rendering that we do but I can see other places that would benifit from it.
I was talking about the engine, not the process. I am responsible for runing it on multiple machines. Each machine needs 500+mb ram because each process is seperate. I was talking more along the lines of a true distributed render, where all the machines work with shared memory (via ccNuma or something like that) as opposed to several boxes tied together by some software (such as LSF, which we use). The render process is a tile renderer instead of a scan line renderer. I seen scan line renderers be ported and work well but never a tile base one. It is amazing on multiple procs on the same machine, though, so it can be threaded. The memory useage is what prevents it from being used well over a standard farm setup. Our solution is to break shots up in to frame slices, each slice of 5+ frames per box. That allows shots to be done in parallel and is decent at load balancing. I just adjust the number of frames in a slice based on how long the shot takes.
I agree with GL being GL, but do you think that they coded the whole thing in such a low level? I imagine that they used some wrapers and libraries instead of native GL calls. So... first you have to get the inbetween working. And... the GL support on the hardware is not at the same level, not yet. But then again, I could be as lost as anyone on this.
You couldn't be more wrong. We have 50+ SGI seats of Maya and like 5 NT seats of Maya. We have 40 Linux boxes for the render farm. We are looking at the move to linux to replace the SGI boxes in the future (well... before we put NT boxes down). With an app like Maya, the people don't need to know the OS, only the app. The TD and IT groups need to know and support the OS so a more UNIX solution, the better.
Dunno about how easy the port is. We are talking about a port from an IRIX box with GL extensions and optimizations. That is much harder than porting standard POSIX code. Maya is a large package and it has its own scripting language. The UI is writen in this language. Having all of this work together as well as the hardware and software render portions of the Maya core is not a small undertaking.
UNIX (IRIX) != Linux (well... not without a bunch of work).
Maya is already ported for linux with the render portion. So the render farm aspect is already there. Advances in linux clustering won't help much though. The way the renderer is writen, it won't scale well across multiple machines, only multiple procs on the same machine.
I come from a studio of 50+ octanes and 40+ VA Linux boxes and we use Maya exclusively. This announcement has been expected, but it has been a long time in the waiting. I figure that A|W should have done this earlier, but I would rather wait for a stable product than get an earlier release with bugs. SGI has had a box to support this for at least a month, yet no good products to use on it. This will be a welcome addition to our studio as I am sure to studios everywhere.
When we looked at buying linux machines we looked at three options: 1) Building them 2) Going with our NT machine supplier or 3) Going with a Linux shop.
We picked 3 and went with VA. Although we could have saved money with option 1 and possibly saved with option 2 we could not put a price tag on knowing that VA *knew* Linux.
I have had two experiences all ready with VA support that was better than what we could have done or what we would get from an NT/linux shop.
The Linux shops will stay arround because of service. Machines will need serviced. Support calls will be made. Who do you want to pay to answer those calls? I will always choose a Linux place.
It looks like the boxes have served most of what they are good for, recognition. With ~16 single proc 450's with 32mb of ram... what sort of problems can you solve. Sure, sure, they offer parallel tasking. But 32mb? How many large tasks that need parallel tasking can be done on a several single proc boxes with that little memory... and with PVM? I have used PVM before on a similar set up of systems (ok... they were 300's and I only had like 4 of them) but I was able to sort numbers and compute digits of PI... but that was about it.
Now... on the other hand, buying real systems... like 42 dual 600's with 1gb ram. Those can rip through problems... well... rendering. And... Beowulf isn't always the soultion. Sometimes shared resources aren't good... can you imagine a network supporting a shared memory of up over 500mb's for a process?
Beowulf has its place, pvm has its place... but in a lot of places, it is good for research. I hope that they were able to acomplish this.
I know that these boxes look expensive when you compare them to a home brew system. They are more expensive than a home brew system. But that home brew system has only you behind it. Only you developing tweaks for your specific layout. Only you to service it and customize it. To some of you, that sounds fine, but to the people who need these in a high availabiltiy environment (such as... hmm... a CG Studio) this is an extremely important difference. I have been disapointed with prices that I have seen from SGI before for the current Linux offerings and for the cost of current MIPS boxen. This is the nature of the buisness though... they are going to be expensive. What blows my mind is that a company that develops for buisnesses and was (and might still be considered) the leader in high end 3d is creating a product that is priced for the above average home user... and is good enough to use in the studio, where do we have the right to complain? The company backing makes up for the price difference. SGI is still a good company with support and R&D. This is the extra that you pay for. Not every solution is going to be cheap or free (or even open) but those that have bits and pieces that follow those goals should be applauded.
Re:Now we just need some software to run in Linux.
on
SGI's New Linux Boxes
·
· Score: 1
AS others have said... and I will back up, Maya is out for Linux. We have several nodes in our Linux Render farm, all using Maya's renderer. They are still in development (my code, not A|W) and the seem extremely promising.
I would have to disagree with your comment about rendering. We have SGI boxes, but when going to a pure render farm, we chose the INTEL/Linux combo. The price vs. time for frame ratio was pretty sweet. Single proc Intel can hold its own against a single proc Octane. And in a world where memory is everything, the Linux boxes destroy the Octanes. Going with SGI's for proc rendering is not a good way to go. Well... unless you had several procs working on one frame. But if you really need a frame done that fast...
I wonder how news like this jives with the new SGI Linux boxes that are to arrive. They are supposed to run GL apps as fast or faster than a windows box. The kicker... it is running a "standard" Linux and it has an NVIDIA card. None of this seems to sound like it works with NVIDIA's current game plan. Sort of like the NVIDIA SDK that was supposed to come out... oh, lets see... over a year ago. I have yet to see anything on it. They said that documentation was holding them up. I know that I hope that they get something going soon, or my video will have to be upgraded again, and we all know how much a techie hates to upgrade...;)
Having just purchased a linux render farm, I can really appreciate this article. We went through the process of determining what the best solution was for our system and for the software that we use for rendering (A|W's Maya) and for load balancing (Platorm's LSF) was to expand to linux boxes and use the same software.
We explored Beowulf, but after talking to those that are in the know, Maya's tile renderer is not well suited to a Beowulf system. I looked at other solutions as well, but due to shared memory and the network bottle neck, nothing could take what we saw as a distributed system and turn it in to a parallel system.
By using a load balancing cluster, we are given the opertunity to render multiple frames at the same time, giving us a speed advanteage. This uses more overall memory than a massively parallel beowulf cluster, but it keeps the speed gain of a parallel system the same. The overhead exists for scene file loading becuase that is done on every machine, but it takes minutes when rendering takes hours. A fair trade. The distributed system needs horsepower and memory more than network speed or file system speed. It is true that an increase in those will speed up the process, but the money is better spent in CPU and mem concerns. Our systems are all dual 600 mhz with a gig of ram per box. It may seem extreme but from our SGI render benchmarking, the scenes that we render can take over 500-600 mb of system memory.
Is it worth the cost?
We are taking our current render system of SGI boxes, which currently are used as desktops durring the day and render boxes at night and adding full time render boxes as well. The cost comparison of a linux render box can be seen in the hardware price alone. We are using these linux boxes to keep par with boxes that cost at least 3x's as much. The only disadvantage is that the linux boxes can not be rolled out to desktop systems when new hires arive, where as the SGI boxes can. This is due to Maya's modeler being SGI/NT only and our support of Maya on the SGI only.
All in all, in our situation, a linux cluster is a God send, allowing me to have more horsepower and to allow the company to save money.
Does this mean that if I took screen captures of the pages for King and then printed them up, all using Microsoft technology that Microsoft would be in contempt of the new Copyright laws?
I know of the isues that you raise. I assume, though, that the OS is going to be out of the way. From what I have read about the virtual machines is that you can assign an amount of memory and processors to each VM. If there is a good multi-threaded linux renderer, and it can support 4+ procs well, then the OS overhead is lower than it is for a single proc machine. The more procs per job, the faster the render. The biggest issue is where the fall off is for multiple procs. Assuming that one of the virtual machines is a file server, then the data can be sent to the render processes at a greater speed than over a fiber conection. Every little speed up makes a difference. Also, how the render jobs are divided up can make a big difference on what performance gains that can be made. The way that we divide shots on our render farm is good for quicker shot speed and lower frame speed. This works well for several machines with single or dual proc configs. With a mainframe render farm, and 10+ procs a VM, then the frame speed can be increased as well. Memory requirements lessen because of shared memory and turn arround time can stay about the same. The cost for similar horsepower from an alpha farm or an SGI Origin farm would be much higher, I assume.
Belief and believe. Hmm... almost too sticky of difference here. What I think that can explain his paradox is that he has reasons for what he believes and others may have beliefs that have not been reasoned out, just adopted.
The difference may seem slim, but it is there. I know for myself that I have beliefs that have been reasoned out... and others that I hold on to that I haven't really thought through. I think that he is attacking the later and suporting the former.
I find all of this a big suprise. Because SGI is banking on both Linux and NVIDIA, you would think that SGI would ask them to play nice. I have a feeling that maybe things will change in the late 2nd or early 3rd quarter when SGI starts rolling out its new machines with NVIDIA cards. Until that day, though, I guess we will all wonder where they stand.
Slashdot Mirror
How can the shared memory work accross the NUMA interface when the byte order differs. I find this hard to believe.
I can see them working together in a networked cluster, but that is how it works now, so this can't be what you are talking about.
I would like to see more information about this.
I would think that if they were developing this technology they would cary it over to the Octane and the Workstation lines.
I don't think that there is curently a 3D video that has been done purely on Linux systems yet.
I know that our video's will be about as close as we can get. We animate and model on Octantes. I would say 75+ % of the frames are rendered on linux and the rest on our SGI boxes. We use only SGI's to create video to import on our NT AVID system.
I applaud all of the work that SGI has done with the linux communtiy and I wish that we could get some more SGI boxes in for quick, single frame renders.
I still think that linux is the best choice for a render farm, though. Small, fast, cheap. I like to treat my boxes as appliances. Maya files in, frames out.
I agree with all that you say, and I thik that SGI boxes are still needed, but less and less.
I doubt that this will be used for rendering or animating or anything in the traditional sense.
I see the system more used as an interface to displaying precreated models interacting in an environment. Similar to 3D video games.
What you should be able to do with this system is load in some textured models, lights and the like, like there are in any game, and have it "hardware render" the frames as you watch. Like any 3D game. The main difference is that now you would have 16x's the power of the PS2 so you could handle more objects, more poly's or maybe larger textures, but that is doubtful.
The output could be a digital signal that is then read by a system that can display it real time, if needs be, or to store it for later viewing.
I am unsure why they are going this route but it seems like a no brainer if you have everything you want mostly pre-defined. You can use a package like A|W's Maya and create everything you need, throw it in the PS2 and play with the objects.
I don't see this being used in a tradition 3d animation sense. The textures and models need to be of a higher quality than what I see the PS2 being able to provide. The motion needs to be key framed so the animator can go back and tweak movement to fit the mood or the sound track.
You could use this to produce "good enough for web" videos or live content but I see that as being the limit.
ALthough A|W announced the Maya Linux port and the renderer is already there, this would be great for a render farm.
I got to use a 4 node Origin 2100 for a bit until it replaced the 2 node Origin we had for a file server.
It was the fastest renderer I had. Great for single frames and good for multiple frames.
If a company needs large frame renders or single frames done fast, a NUMA style machine is needed.
Having the large capacity of memory alos alows several smaller render jobs at a time.
A machine like this would be a dream for me.
I don't think that the cost/performance metric would pay off for the type of rendering that we do but I can see other places that would benifit from it.
3D animation is not going away any time soon.
And I disagree about it being cold and soulless...
Watch a veggie video if you disagree.
It is full of warmth and fun.
Also, many other videos benifit from the 3D for f/x, such as Stuart Little. 3D is a strong and growing industry.
But as mant have said, you must have a strong story. If the story is weak, people will grow tired of the neato graphics.
I was talking about the engine, not the process.
I am responsible for runing it on multiple machines. Each machine needs 500+mb ram because each process is seperate. I was talking more along the lines of a true distributed render, where all the machines work with shared memory (via ccNuma or something like that) as opposed to several boxes tied together by some software (such as LSF, which we use).
The render process is a tile renderer instead of a scan line renderer. I seen scan line renderers be ported and work well but never a tile base one.
It is amazing on multiple procs on the same machine, though, so it can be threaded. The memory useage is what prevents it from being used well over a standard farm setup.
Our solution is to break shots up in to frame slices, each slice of 5+ frames per box.
That allows shots to be done in parallel and is decent at load balancing. I just adjust the number of frames in a slice based on how long the shot takes.
Hey... can you drop me a line... I am interested in talking with others strugling with the multiplatform hell and Maya.
I agree with GL being GL, but do you think that they coded the whole thing in such a low level?
I imagine that they used some wrapers and libraries instead of native GL calls. So... first you have to get the inbetween working. And... the GL support on the hardware is not at the same level, not yet.
But then again, I could be as lost as anyone on this.
You couldn't be more wrong.
We have 50+ SGI seats of Maya and like 5 NT seats of Maya. We have 40 Linux boxes for the render farm. We are looking at the move to linux to replace the SGI boxes in the future (well... before we put NT boxes down). With an app like Maya, the people don't need to know the OS, only the app. The TD and IT groups need to know and support the OS so a more UNIX solution, the better.
Dunno about how easy the port is.
We are talking about a port from an IRIX box with GL extensions and optimizations. That is much harder than porting standard POSIX code.
Maya is a large package and it has its own scripting language. The UI is writen in this language. Having all of this work together as well as the hardware and software render portions of the Maya core is not a small undertaking.
UNIX (IRIX) != Linux (well... not without a bunch of work).
Maya is already ported for linux with the render portion. So the render farm aspect is already there.
Advances in linux clustering won't help much though. The way the renderer is writen, it won't scale well across multiple machines, only multiple procs on the same machine.
I come from a studio of 50+ octanes and 40+ VA Linux boxes and we use Maya exclusively. This announcement has been expected, but it has been a long time in the waiting. I figure that A|W should have done this earlier, but I would rather wait for a stable product than get an earlier release with bugs.
SGI has had a box to support this for at least a month, yet no good products to use on it. This will be a welcome addition to our studio as I am sure to studios everywhere.
When we looked at buying linux machines we looked at three options: 1) Building them 2) Going with our NT machine supplier or 3) Going with a Linux shop.
We picked 3 and went with VA. Although we could have saved money with option 1 and possibly saved with option 2 we could not put a price tag on knowing that VA *knew* Linux.
I have had two experiences all ready with VA support that was better than what we could have done or what we would get from an NT/linux shop.
The Linux shops will stay arround because of service. Machines will need serviced. Support calls will be made. Who do you want to pay to answer those calls? I will always choose a Linux place.
It looks like the boxes have served most of what they are good for, recognition.
With ~16 single proc 450's with 32mb of ram... what sort of problems can you solve. Sure, sure, they offer parallel tasking. But 32mb? How many large tasks that need parallel tasking can be done on a several single proc boxes with that little memory... and with PVM?
I have used PVM before on a similar set up of systems (ok... they were 300's and I only had like 4 of them) but I was able to sort numbers and compute digits of PI... but that was about it.
Now... on the other hand, buying real systems... like 42 dual 600's with 1gb ram. Those can rip through problems... well... rendering. And... Beowulf isn't always the soultion. Sometimes shared resources aren't good... can you imagine a network supporting a shared memory of up over 500mb's for a process?
Beowulf has its place, pvm has its place... but in a lot of places, it is good for research. I hope that they were able to acomplish this.
I know that these boxes look expensive when you compare them to a home brew system. They are more expensive than a home brew system. But that home brew system has only you behind it. Only you developing tweaks for your specific layout. Only you to service it and customize it.
To some of you, that sounds fine, but to the people who need these in a high availabiltiy environment (such as... hmm... a CG Studio) this is an extremely important difference. I have been disapointed with prices that I have seen from SGI before for the current Linux offerings and for the cost of current MIPS boxen. This is the nature of the buisness though... they are going to be expensive.
What blows my mind is that a company that develops for buisnesses and was (and might still be considered) the leader in high end 3d is creating a product that is priced for the above average home user... and is good enough to use in the studio, where do we have the right to complain?
The company backing makes up for the price difference. SGI is still a good company with support and R&D. This is the extra that you pay for.
Not every solution is going to be cheap or free (or even open) but those that have bits and pieces that follow those goals should be applauded.
AS others have said... and I will back up, Maya is out for Linux. We have several nodes in our Linux Render farm, all using Maya's renderer.
They are still in development (my code, not A|W) and the seem extremely promising.
I would have to disagree with your comment about rendering. We have SGI boxes, but when going to a pure render farm, we chose the INTEL/Linux combo.
The price vs. time for frame ratio was pretty sweet. Single proc Intel can hold its own against a single proc Octane.
And in a world where memory is everything, the Linux boxes destroy the Octanes.
Going with SGI's for proc rendering is not a good way to go. Well... unless you had several procs working on one frame. But if you really need a frame done that fast...
I wonder how news like this jives with the new SGI Linux boxes that are to arrive. They are supposed to run GL apps as fast or faster than a windows box. ;)
The kicker... it is running a "standard" Linux and it has an NVIDIA card. None of this seems to sound like it works with NVIDIA's current game plan.
Sort of like the NVIDIA SDK that was supposed to come out... oh, lets see... over a year ago. I have yet to see anything on it. They said that documentation was holding them up.
I know that I hope that they get something going soon, or my video will have to be upgraded again, and we all know how much a techie hates to upgrade...
Having just purchased a linux render farm, I can really appreciate this article. We went through the process of determining what the best solution was for our system and for the software that we use for rendering (A|W's Maya) and for load balancing (Platorm's LSF) was to expand to linux boxes and use the same software.
We explored Beowulf, but after talking to those that are in the know, Maya's tile renderer is not well suited to a Beowulf system.
I looked at other solutions as well, but due to shared memory and the network bottle neck, nothing could take what we saw as a distributed system and turn it in to a parallel system.
By using a load balancing cluster, we are given the opertunity to render multiple frames at the same time, giving us a speed advanteage. This uses more overall memory than a massively parallel beowulf cluster, but it keeps the speed gain of a parallel system the same. The overhead exists for scene file loading becuase that is done on every machine, but it takes minutes when rendering takes hours. A fair trade.
The distributed system needs horsepower and memory more than network speed or file system speed. It is true that an increase in those will speed up the process, but the money is better spent in CPU and mem concerns. Our systems are all dual 600 mhz with a gig of ram per box. It may seem extreme but from our SGI render benchmarking, the scenes that we render can take over 500-600 mb of system memory.
Is it worth the cost?
We are taking our current render system of SGI boxes, which currently are used as desktops durring the day and render boxes at night and adding full time render boxes as well. The cost comparison of a linux render box can be seen in the hardware price alone. We are using these linux boxes to keep par with boxes that cost at least 3x's as much.
The only disadvantage is that the linux boxes can not be rolled out to desktop systems when new hires arive, where as the SGI boxes can. This is due to Maya's modeler being SGI/NT only and our support of Maya on the SGI only.
All in all, in our situation, a linux cluster is a God send, allowing me to have more horsepower and to allow the company to save money.
Does this mean that if I took screen captures of the pages for King and then printed them up, all using Microsoft technology that Microsoft would be in contempt of the new Copyright laws?
I know of the isues that you raise. I assume, though, that the OS is going to be out of the way.
From what I have read about the virtual machines is that you can assign an amount of memory and processors to each VM.
If there is a good multi-threaded linux renderer,
and it can support 4+ procs well, then the OS
overhead is lower than it is for a single proc machine. The more procs per job, the faster the render. The biggest issue is where the fall off is for multiple procs.
Assuming that one of the virtual machines is a file server, then the data can be sent to the render processes at a greater speed than over a fiber conection. Every little speed up makes a difference.
Also, how the render jobs are divided up can make a big difference on what performance gains that can be made. The way that we divide shots on our render farm is good for quicker shot speed and lower frame speed. This works well for several machines with single or dual proc configs.
With a mainframe render farm, and 10+ procs a VM, then the frame speed can be increased as well. Memory requirements lessen because of shared memory and turn arround time can stay about the same.
The cost for similar horsepower from an alpha farm or an SGI Origin farm would be much higher, I assume.
This would work well for a render farm if...
;)
There was terabytes of ram. You would want a couple hundred meg of ram for each render job.
One thing that would be better is to assign 4 procs to a machine, thus reducing the total amount of memory to be 10,000 machines X 250-500 MB.
I am sure that we could get that much memory for the system...
I don't think that most render engine would port well to 100 procs... but... if it was reprogramed with that in mind... it might do really well.
Too bad I don't have the time or the resources.
Belief and believe. Hmm... almost too sticky of difference here. What I think that can explain his paradox is that he has reasons for what he believes and others may have beliefs that have not been reasoned out, just adopted.
The difference may seem slim, but it is there. I know for myself that I have beliefs that have been reasoned out... and others that I hold on to that I haven't really thought through. I think that he is attacking the later and suporting the former.
I find all of this a big suprise.
Because SGI is banking on both Linux and NVIDIA, you would think that SGI would ask them to play nice.
I have a feeling that maybe things will change in the late 2nd or early 3rd quarter when SGI starts rolling out its new machines with NVIDIA cards.
Until that day, though, I guess we will all wonder where they stand.