Slashdot Mirror
Pixar Eclipses Sun with Linux/Intel
lieutenant writes "Pixar Animation Studios is replacing servers from Sun in its render farm with eight new blade servers from Rackspace. In all, the blade system contains 1,024 Intel 2.8GHz Xeon processors, and it runs the open-source Linux operating system. Pixar has ported its Renderman software to run on Linux." I'd love to see their electric bill ;)
With that type of processing power, they should be able to calculate to infinity...and beyond.
Perhaps if Sun spent more time getting their processors faster at good cost they wouldn't be losing this kind of ground. Sun took way too long to come out with their UltraSparc III processor and now clustering technology is at the point where it's much cheaper to string together a bunch of commodity PCs than purchase a high end Sun box.
Not 1024 CPUs in one box. Each CPU sits on a "blade" card and acts like a seperate system. It's a bug cluster.
"I'd love to see their electric bill "
Dude, they render stuff... would you not prefer to see that...
For around $25,000 you too can make Pixar quality movies
...plus the several million $ for the creative talent!
If you have a task that can be easily partitioned off (oh like each individual frame would be an easy break for this) you can send each task to a different machine allowing you to parellelize the task.
This is a poor mans version of NUMA (Non Uniform Memory Access) created and popularized by Sequent (now a division of IBM) where rather than have a single pool of addressable memory, you have multiple pools of memory, some with very fast access, some with slower.
What I am wondering is what do they do for the cluster cross connect. In large scale cluster environments, this tends to be a significant bottleneck. In large scale clusters you start seeing things like HIPPI, VIA, and soon to be Infiniband... wonder what this is stocked up with
I have mod points and I am not afraid to use them
As far as I know Rackspace is a managed hosting company. Rackable Systems makes servers - Yahoo and Google both use them. Anyone know if the article has it wrong, and Pixar is actually using Rackable machines?
My god, I thought they had trouble scaling Linux that far. Seriously. How the hell do you do that when "stock" linux doesnt like 8 CPUs?
Because it's not a single system image. Rendering movies is easy to parallelize because you don't need to have once scene rendered before you can render the next; all the information you need is in the model file.
For around $25,000 you too can make Pixar quality movies
1) Download renderman from https://renderman.pixar.com/
2) Learn how to use it.
3) ???
4) Profit!
Eureka! The missing link is "...plus the several million $ for the creative talent!"!
isn't
720 hours @ 75,776W = 54,558,720kWh.
actually 54,558,720Wh (watt-hours, not kilowatt hours), which is 54,558kWh
making it not 3.2million, but only $3200 a month?
I teach MCSE courses down in Chatsworth, recently we got a lot of Engineers from boeing coming over for Windows XP classes. Why? They're dumping all their Sparc Stations and moving to XP on cheap Intel hardware. Its faster, and 2/3s of the applications they need run it already. The last third they were working on.
The IT people I talked to were surprisingly happy with XP so far. These were all Unix only kind of people actually.
The other thing they were doing were looking into dumping their Crays in favor of LINUX clusters. The comments were along the lines of how much faster and cheaper it was to put together a cluster of a 100 cheap Intel boxes than getting a new Cray. That, and they were all already familiar with the unix style interface. On top of it all, the GUI interface (I think they were running Gnome) was so much nicer than CDE on Solaris.
So Sun it getting it from both sides- Cheap Wintel boxes and Cheap Linux boxes. No wonder they finally relented and released Solaris 9 on Intel.
They have half-depth 1U boxes. That's right, two servers in 1U, back to back.
Includes space between the two for cabling and cooling.
They specialize in delivering easy to manage (physically) racks of highly commoditized systems.
(I work with them in a reseller relationship)
Imagine a 71U rack(minus 1U for a switch), with 142 boxes, all dual proc. 248 procs in a rack!
Man, I wish they'd put the right link in there.
Striving to achieve a lower state of conciousness
> > For around $25,000 you too can make Pixar quality movies
>
And if that's too expensive you can forgo the creative talent and make Star Wars prequels!
Sheesh, evil *and* a jerk. -- Jade
A lot of people are going to be saying "just one example of how Sun is dying", but coming from a place that runs several hundred Sun machines (and being a Sun fanboy), I can understand why they made this switch. For shere processing power on-the-cheap, the x86 world has had a lead on Sun and other big UNIX vendors for a few years. Having a decent OS (linux) to run on those machines, makes it even easier to switch.
It's about using the right tool for the job, and now that x86/linux/bsd has matured to a point where it can be used for some professional applications, it only makes sense to see things like this happen.
Sun is going to be around for a long time. As many other people have pointed out, they're just retreating somewhat to more a of niche market, where they are the right tool for the job.
The executives at my company are very interested in linux, because of the outrageous leap in processing power per dollar, and the reductions in CPU-based licensing costs for software like Oracle is staggering. The concern, though, is stability.
Sun Fire and Enterprise servers are really expensive, but they stay up all the time. Swapping a failed processor or NIC or memory stick without halting the box is really important on a mission-critical server. Likewise, a well built Sun box never panics, and if it ever does, Sun will insist that their engineers look at the crash dump to figure out what went wrong.
I think Linux has won the performance battle, but what about the stability battle? You need to win both to win the war.
My god, I thought they had trouble scaling Linux that far. Seriously. How the hell do you do that when "stock" linux doesnt like 8 CPUs?
I often see this misconception about multiprocessor machines. Some machines have a true tightly coupled multiprocessor architectures with a shared memory space, like big iron machines from SGI, Sun, and HP. These can be used to run a multithreaded process to speed up time-to-solution for a task. The speed-up is subject to the usual Amdahl's Law restrictions. The blade server machines, like Pixar is using, are 'tightly bolted' multiprocessors which share mechanical components and power supplies, but they effectively look like separate computers. Possibly some of the blades have shared multiprocessors, but no more than a 2-4 cpus per blade. Separate instances of the OS run on each blade.
For easy to partition tasks like computer graphic rendering, each frame render task can be run single threaded, and there can be many tasks running at the same time. The time-to-solution for a single rendered frame is not reduced by parallelization, but the overall throughput is increased by multiple tasks.
Nine women cannot make a baby in one month, but nine women can make nine babies in nine months.
I recently attended a talk by Google's chief engineer. They have approximately 15,000 x86 machines running Linux at seven data centers in the United States.
Weird failures occur so often, such as disks returning garbage without the controller informing the OS, that Google does a checksum on _every_ data structure in their user-level software. He also talked about how Linux is good enough for them, but it doesn't perform well with respects to I/O under heavy load. He says they like Linux because they have the source-code and that they minimize excessive I/O loads on their machines. Nobody asked why they don't use FreeBSD but I suspect its because Linux has better hardware support and Google builds their own machines with numerous different components based on the latest technology.
The only natural base is e. Man arbitrarily likes whole numbers, nature like real numbers, and e is everywhere.
Therefore, ln (1024) = 6.931471806... which is not an even number.
I suggest therefore that an even number of processors for the render farm is either
e^6 = 403.4287935 or
e^7 = 1,096.633158.
Of course, Intel is wedded to the whole numbers of processors thing, which utterly thwarts mathematical logic and correctness. Their site also runs on IIS, so what other foolishness can you expect? Heathens.
Fire and Meat. Yummy.
... I guess we know which one you are.
Pixar is on the right track. I do ASIC verification, mainly on Sun boxes (fastest USparc IIIs, multi-proccessor, 14GBs memory, etc). Lately, I have been running the exact same jobs on an LSF enabled Linux farm of Intel boxes.
The improvement is 3-4 times speedup ie 8 hour Sun jobs take 2 hours on Intels.
For the price of one dual proccesor Sun workstation, you can get ten Intel boxes running linux.
Not only is the speedup great, I need less licences to run the CAD software (doing multiple regression jobs). Since a license seat per CAD tool can run from 30K to 200K each plus 10% a year maintence fee, the savings are huge.
Changing over to linux was trivial. I like and have used Suns for years and Suns were a major player in this industry. But I firmly believe that this paradigm is going to be a SUN KILLER!
No, cooling (AKA air conditioning) just moves heat from one location to another. This takes a lot less energy than the actual heat load being moved.
This is why heat pumps are a lot more efficient than resistance heating. Heat pumps move ambient heat outside (yes, there's ambient heat; even on a cold day) into the building, which requires less energy than producing the heat energy directly.
The parent post is somewhat misleading and more than a little spotty, but it got modded up, so I feel I should clarify.
... we hate you because of your strives for world domination, but then you go and support linux ... bastards we just love to hate you.
That Sun had tried renderman (or whatever they call it) to run on 32 bit processors and it was a horrible disaster. Something about how it seemed more feasible and cost efficient to use Sun until the days in which the competiting 64 bit processors became cheaper.
Renderman is a standard for going exporting frames to a renderer. Pixar's implementation is called Photorealistic renderman. Sun is not involved in this at all. It has run on x86 procs, as well as Linux for quite a while now. Renderers are relativly easy to port, especially from different Unixes. I am not sure if there are speed advantages to 64 bit computers, or if it is just accuracy and memory like always, which is still a big advantage for a renderer. ( can anyone clarify?) I have a PRman rendered image on my desktop right now on my 450 Mhz PIII. The above quote is pretty much completly false.
Doesn't dreamworks use this type of technology already?
The technology is just running off the shelf software and hardware. Different parts of dreamworks do use Linux heavily.
Damned MPAA members
This is horribly misinformed. I don't have the energy to go into the whole issue here but suffice to say that this is wildly misplaced frustration. First of all, Pixar is not a member of the MPAA. They have a deal with Disney, which is. That attidude would be fitting and understandable with Disney for various reasons, but making Pixar your enemy is just wrong (except when they sued Larry Gritz personally to hold off competition to Renderman). The same goes for Visual Effects companies. ILM, Imageworks, Digital Domain, PDI, Pixar, Rythm and Hues, Weta, etc. are the best thing that's happening to Linux right now. They are so far removed from the wrongdoings of the MPAA its like me blaming someone for crime when their friends dad is part of the NRA. They are doing only good for Linux, and they are not hyprocrites. They do have deals with studios that are intern part of the MPAA. Not everything is perfect, and these issues are not something that they as companies are, should be, or will be concerned about. They are also starting to contribute to Linux, and I am confident more will come as Linux matures in their pipeline. Building up anger towards Visual Effects companies perpetuates the sterotype of free software advocates being zealots without understanding the whole issue.
This Wiki Feeds You TV and Anime - vidwiki.org
1982 - born in Nebula - incorporated with 4 employees
1984 - protostar - NFS is introduced
1995 - main sequence begins - Java Released
1996 - red giant - Using Java technology, NASA engineers develop an interactive application allowing anyone on the Internet to be a "virtual participant" in the space administration's groundbreaking mission to Mars.
SUPERNOVA - Sun battles MS over Java and Windows
Blackhole - TODAY!
References:
http://www.sun.com/aboutsun/coinfo/history.html
Its all about the distinction between shared and distributed memory architectures. Different applications benefit from different types of parralelism which the above architectures provide. If to solve the problem independent chunks of code can be run that require no communication at run time then clearly a blade type solution (distrbiuted memory) is viable, but if the calculations are co dependent on each other and require communication of interrim results then the overhead of communication can quickly become the critical path and shared memory parallelism becomes a better solution. It also depends on the level of parralelilsm built into the implementation of the algorithms inside pixars redering program itself.
Its one damn thing before another. (Dick Bird 1999)
The most important linux skill is how to use internet for help, not any unix skills. For a newbie, it is a hit or miss affair. He grabs a modern desktop oriented linux, installs it in 30 or less minutes, if all of his hardware are supported and all programs newbie wants are already installed, good news, we have a new linux fan. Chances are, that won't happen.
If something goes wrong, it is best option for linux fans that newbie just forgets the idea, right then. Most probably he now has a functional system but with a non-functional usb mouse, cd burner or a sub-optimal refresh rate. He will want to fix and use the system. It is just the mouse, or the printer, or excel documents. He almost succeeded in this linux thing!
Wrong. He still misses the crucial skill.
He will try to fix it and fail, seek help and fail again, try to skim docs and fail, learn where to seek help and fail, read documents and seek help at the correct place with the correct attitude and if he has some luck, succeed at last. Now we have a brand new whiner instead fo a fan. Worse, he half knows what he is talking about.
Eveyone whines about windows all the time too, but it is not the same thing. We don't want scared potential new users. In case of windows, user already knows how much of that whining is about a real problem, that is not the case with linux.
Solution is aiming higher. Linux has to be considerably easier to use and install than windows because non-techie users just have a lot of experience with windows. Even if the fix isn't optimal, there is always a fix a phonecall to someone you know away. Linux doesn't have nearly the same installed base so is denied the luxury. Linux still requires a crucial skill; it shouldn't.
In some areas (considering desktop) linux already is better than windows and in others, it is not too far behind. But it has to better on all fronts. Till than, linux is not ready. You can argue that had market shares of linux and windows magically flipped, we would be saying windows is not ready. Probably you would be right, too. But market share (or rather, user base) has not magically flipped and that is not irrelevant.
I know, I should have read the grandparent.
Gentlemen, you can't fight in here, this is the War Room!
I have heard from several places that Intel's PR flacks have been flogging this story mercilessly, so it's not too surprising to see it show up in Slashdot. Twice.
:)
To get the inaccuracy out of the way -- RenderMan has been running on Linux for several years now, and I would be surprised if Linux wasn't the dominant platform for RenderMan for quite some time, outside of Pixar of course.
I am really surprised, though, that at this point in time they'd go from 64-bit to 32-bit machines, especially as 64-bit PC-like machines are just becoming available. Why not go with Itanium or the new Hammer? Each of Pixar's movies to date have been gloriously more complex and hard-to-render than the last one -- and while I know that they go to fairly extreme lengths to keep the memory footprint down I would think that they'd be bumping up against the 4GB limit already. If not now, then quite soon.
Perhaps this is just a stopgap to get Nemo finished, even 1024 servers is a fairly small cost. Certainly it would be compared to the RenderMan licenses
Every RenderMan user except for Pixar has to look to get the maximum rendering power per CPU, as the licenses are $5,000 and up, while the CPUs are far far cheaper than that. I suppose Pixar's figure of merit is rendering power per dollar or rendering power per BTU (for cooling limited situations), or even render power per ft^2. Still, the 32-bit machines are a baffling choice to me.
thad
ps. My company has a render garden (too small to be a render farm) of a dozen or so Athlons.
I love Mondays. On a Monday, anything is possible.
Sorry Guys... This article looks to be a bit off base!
/. I though you guys did better about checking this kind of thing out! Just because it's on c-net doesn't mean it's accurate. Well kudos to who ever really got this job.
-- Not an Official RS response --
I work for Rackspace Managed Hosting. The company the link "Rackspace" references in the C-Net article. This kind of cluster is not consistent with our business. We are most focused on web-centric managed hosting vrs colocation. A rendering cluster is something that, from my experience, we've never done. Also We don't carry Blade servers. C-Mon
Matthew Montgomery
Rackspace Managed Hosting.
Nine women cannot make a baby in one month, but nine women can make nine babies in nine months.
.NET allow for nine women to make a baby in one month?
Won't Microsoft's soon-to-be-released BabyMaker
I thought I saw a press release about it a while back but can't seem to find a link now.
I just eclipsed Windows with Linux on my home system.
I just eclipsed my old toothbrush with a new one.
I just eclipsed the shit in my ass-crack with toilet paper.
Now, don't I sound FUCKING STUPID? Yes, I do.
This sig is part of your complete breakfast.