Slashdot Mirror
Pixar Eclipses Sun with Linux/Intel
lieutenant writes "Pixar Animation Studios is replacing servers from Sun in its render farm with eight new blade servers from Rackspace. In all, the blade system contains 1,024 Intel 2.8GHz Xeon processors, and it runs the open-source Linux operating system. Pixar has ported its Renderman software to run on Linux." I'd love to see their electric bill ;)
How fast they can now render over the old Sun servers?
(imaging a Beowulf cluster of THESE!)
1024 xeon's? jeeze, my electric is $120/mo with one amd and one intel running half the time.
For around $25,000 you too can make Pixar quality movies (+ the cost of those servers). https://renderman.pixar.com/
I am actually scared, to imagine a beowulf cluster of these.
What signature defines me as a person?
My god, I thought they had trouble scaling Linux that far. Seriously. How the hell do you do that when "stock" linux doesnt like 8 CPUs?
Tequila: It's not just for breakfast anymore!
With that type of processing power, they should be able to calculate to infinity...and beyond.
Didn't I follow the same link from the earlier Rendezvous with Rama story?
"E pur si muove!" - attributed to Galileo Galilei, 1564-1642
Perhaps if Sun spent more time getting their processors faster at good cost they wouldn't be losing this kind of ground. Sun took way too long to come out with their UltraSparc III processor and now clustering technology is at the point where it's much cheaper to string together a bunch of commodity PCs than purchase a high end Sun box.
Not 1024 CPUs in one box. Each CPU sits on a "blade" card and acts like a seperate system. It's a bug cluster.
1024 is 2^10. Computers operate in binary, and 1024 is an "even" number when you consider binary.
I assumed that Apple created their Xserve rack-mounted servers for exactly this purpose: not just for animation studios, but for Pixar in particular (since Steve Jobs runs both companies and does things like selling Pixar DVDs to Apple to give away in promotions, thereby increasing the number of DVDs sold at launch, getting his movies in front of more people, and of course providing more incentive to buy whatever it is he's bundling the DVDs with).
I guess the density of the blade servers is higher than the Mac servers, but it would have been a big boost to the Xserve's credibility if Pixar had chosen to use a ton of them. Perhaps Apple will make a new server (Xblade?) that's more suited to this use. It wouldn't surprise me...
"I'd love to see their electric bill "
Dude, they render stuff... would you not prefer to see that...
I could have sworn that the software couldn't run at all in 64 bit. I'm just wondering if they didn't take a step down when they converted 64-bit optimized code to run on regular high cache 32-bit pentiums.
Great for linux and anyone who has half a brain knows that you can make a very nice system from the Intel Xeon chips and Linux. But Sparcs aren't x86's and they certainly don't run the same. I've been running a server off of a pII 400 mhz Xeon with 2 megs cache on it for nearly 4 years now. It's never failed me yet and I have no intentions of upgrading anytime soon, but then again I'm not rendering anything in 3 deminsions either.
Doesn't dreamworks use this type of technology already?
Damned MPAA members ... we hate you because of your strives for world domination, but then you go and support linux ... bastards we just love to hate you.
Lastly I'm really surprised that Pixar didn't go for a server farm of OS X boxen, just goes to show ya, right tool for the job. Maybe they'll throw darwin on their at least.
Ignore the "p2p is theft" trolls, they're just uninformed
As far as I know Rackspace is a managed hosting company. Rackable Systems makes servers - Yahoo and Google both use them. Anyone know if the article has it wrong, and Pixar is actually using Rackable machines?
given the G4's that sit inside the box are easily out-performed by Intel/AMD these days.
G4s still have one of the best vector units in town (far better than MMX/SSE, see Ars Technica for more details), and the kind of stuff Pixar is going to be using them for (ray tracing I assume) is perfectly suited to opperation on an AltiVec unit. I wouldn't be surprised if, with properly optimised code, a G4 couldn't replace 2-3 Xeons in this particular application. (No, I'm not saying that a 1GHz G4 Mac is faster than a 3THz P7 before I get any flames)
I am TheRaven on Soylent News
1,024 Intel 2.8GHz Xeon processors... I'd love to see their electric bill
Well, ignoring the power requirements of RAM, bus controllers, network adapters, hard disks which are probably used for boot only...
Intel rates these things for 74.0W thermal dissipation, which is a pretty good measure of the electrical power consumed... since, unless something is badly wrong, your Xeon chip will not dissipate energy as light or sound.
74W x 1,024 = 75,776W continuous.
Assume they're on 24/7. Assume a cost of $0.06 per kWh, including distribution, debt retirement, Ontario's capped electric rates, etc.
There are 30 days in the average month. There are 24 hours in the average day [grin]. Therefore, there are 720 hours per month.
720 hours @ 75,776W = 54,558,720kWh.
Just a little over $3.2 million per month.
I'd imagine it's less than that; their electric rate is probably somewhat less based on their consumption. But consider that the depreciation on that hardware is probably a greater monthly expense than the electricity to power it...
I'm glad Linux is ready for Pixar, because Linux sure ain't ready for the desktop.
Fire and Meat. Yummy.
I must be lost here, but most of these renderfarms I've seen that use Sun products is for network storage solutions, though they're even losing the marketshare these days. I think what people are starting to realize is that just because you paid a whole lot for it, doesn't mean you got "The Best".
Supercomputers of 5 years ago can be built today with computers being thrown away and setup into a computing cluster. Obviously the good old days of 40 trillion dollar super computers paid for by the goernment aren't the super computers of today.
Ignore the "p2p is theft" trolls, they're just uninformed
I'm actually a little surprised they use general purpose CPUs for this kind of task. I'd have thought that a load of custom DSPs might be faster, and probably cheaper - How about 1 DSP per pixel (About 10 million?). I'm sure that would really zip along, if they could sort out the memory access issues inherent in this kind of application. Ray tracing is perfect for parallel execution, since each pixel really is independent of each other pixel, and each frame is likewise independent.
I am TheRaven on Soylent News
I teach MCSE courses down in Chatsworth, recently we got a lot of Engineers from boeing coming over for Windows XP classes. Why? They're dumping all their Sparc Stations and moving to XP on cheap Intel hardware. Its faster, and 2/3s of the applications they need run it already. The last third they were working on.
The IT people I talked to were surprisingly happy with XP so far. These were all Unix only kind of people actually.
The other thing they were doing were looking into dumping their Crays in favor of LINUX clusters. The comments were along the lines of how much faster and cheaper it was to put together a cluster of a 100 cheap Intel boxes than getting a new Cray. That, and they were all already familiar with the unix style interface. On top of it all, the GUI interface (I think they were running Gnome) was so much nicer than CDE on Solaris.
So Sun it getting it from both sides- Cheap Wintel boxes and Cheap Linux boxes. No wonder they finally relented and released Solaris 9 on Intel.
They have half-depth 1U boxes. That's right, two servers in 1U, back to back.
Includes space between the two for cabling and cooling.
They specialize in delivering easy to manage (physically) racks of highly commoditized systems.
(I work with them in a reseller relationship)
Imagine a 71U rack(minus 1U for a switch), with 142 boxes, all dual proc. 248 procs in a rack!
Man, I wish they'd put the right link in there.
Striving to achieve a lower state of conciousness
A lot of people are going to be saying "just one example of how Sun is dying", but coming from a place that runs several hundred Sun machines (and being a Sun fanboy), I can understand why they made this switch. For shere processing power on-the-cheap, the x86 world has had a lead on Sun and other big UNIX vendors for a few years. Having a decent OS (linux) to run on those machines, makes it even easier to switch.
It's about using the right tool for the job, and now that x86/linux/bsd has matured to a point where it can be used for some professional applications, it only makes sense to see things like this happen.
Sun is going to be around for a long time. As many other people have pointed out, they're just retreating somewhat to more a of niche market, where they are the right tool for the job.
The executives at my company are very interested in linux, because of the outrageous leap in processing power per dollar, and the reductions in CPU-based licensing costs for software like Oracle is staggering. The concern, though, is stability.
Sun Fire and Enterprise servers are really expensive, but they stay up all the time. Swapping a failed processor or NIC or memory stick without halting the box is really important on a mission-critical server. Likewise, a well built Sun box never panics, and if it ever does, Sun will insist that their engineers look at the crash dump to figure out what went wrong.
I think Linux has won the performance battle, but what about the stability battle? You need to win both to win the war.
I recently attended a talk by Google's chief engineer. They have approximately 15,000 x86 machines running Linux at seven data centers in the United States.
Weird failures occur so often, such as disks returning garbage without the controller informing the OS, that Google does a checksum on _every_ data structure in their user-level software. He also talked about how Linux is good enough for them, but it doesn't perform well with respects to I/O under heavy load. He says they like Linux because they have the source-code and that they minimize excessive I/O loads on their machines. Nobody asked why they don't use FreeBSD but I suspect its because Linux has better hardware support and Google builds their own machines with numerous different components based on the latest technology.
Tux already had a role in TS2 as Wheezy.
Sun uniprocessor performance has been very uncompetitive for quite some time now. I bet they would have switched a long time ago if it was not for the difficulty of porting software from Solaris to Linux. Plus human inertia ...
The worst problem for Sun is once they loose customers to Linux, there is no turning back.
They still hold well in 64-bit area, however, once commodity hardware such as x86-64 gets there, this battle will also be over.
This is the main reason why the company is likely to go down the drain.
The only natural base is e. Man arbitrarily likes whole numbers, nature like real numbers, and e is everywhere.
Therefore, ln (1024) = 6.931471806... which is not an even number.
I suggest therefore that an even number of processors for the render farm is either
e^6 = 403.4287935 or
e^7 = 1,096.633158.
Of course, Intel is wedded to the whole numbers of processors thing, which utterly thwarts mathematical logic and correctness. Their site also runs on IIS, so what other foolishness can you expect? Heathens.
Fire and Meat. Yummy.
... I guess we know which one you are.
This cluster is so powerful, when they try to render anything with it, all they get is "42" on the console.
Pixar is on the right track. I do ASIC verification, mainly on Sun boxes (fastest USparc IIIs, multi-proccessor, 14GBs memory, etc). Lately, I have been running the exact same jobs on an LSF enabled Linux farm of Intel boxes.
The improvement is 3-4 times speedup ie 8 hour Sun jobs take 2 hours on Intels.
For the price of one dual proccesor Sun workstation, you can get ten Intel boxes running linux.
Not only is the speedup great, I need less licences to run the CAD software (doing multiple regression jobs). Since a license seat per CAD tool can run from 30K to 200K each plus 10% a year maintence fee, the savings are huge.
Changing over to linux was trivial. I like and have used Suns for years and Suns were a major player in this industry. But I firmly believe that this paradigm is going to be a SUN KILLER!
Anyone get the feeling that Sun's "brightest hours" are behind them? As others have mentioned, they're getting hit from the Windows XP side, as well as Linux. If Solaris dwindles as a result of this, and becomes a niche/high-end item, what does this say for HP, SGI, and the rest of Unixen?
I've been thinking in terms what are/will be the Big Three:
Linux, Mac OS X, and that other thing.. uh, Windows XP. I wouldn't bet on traditional Unixen as a growth area, by any means. Won't be long for some companies to become "Unix-free and Windows-free" zones...
You know, I really don't know what the logic is of arguing that. The people who are using Linux on their desktops now know Linux well enough to completely disregard that. I suppose you will scare newbies away until someone gives them a knoppix CD to play with, but MS spends BILLIONS already for that your little rant is insignificant in comparison.
Maybe Linux is more than ready for the desktop, it just isn't ready for your narrow view of what a desktop should be. And it is not that I really care that you are not satsified, but bitching to a bunch of volunteers seems a bit insane, because I don't think they really care that your are not satisfied, either.
Regardless, Linux isn't going away anytime soon (at least not in my lifetime), so why don't you create a project devoted to "making it ready for the desktop according to my definitions" instead of wasting your life away making complaints about the fruits of a VOLUNTEER EFFORT.
Do you complain about the Salvation Army or Goodwill Industries, as well?
Sdelat' Ameriku velikoy Snova!
1982 - born in Nebula - incorporated with 4 employees
1984 - protostar - NFS is introduced
1995 - main sequence begins - Java Released
1996 - red giant - Using Java technology, NASA engineers develop an interactive application allowing anyone on the Internet to be a "virtual participant" in the space administration's groundbreaking mission to Mars.
SUPERNOVA - Sun battles MS over Java and Windows
Blackhole - TODAY!
References:
http://www.sun.com/aboutsun/coinfo/history.html
For some reason, I just can't seem to resist nit picking here. BSD has been mature enough for professional use for quite a long time now.
In fact, I seem to remember a time (pre-Solaris) when Sun systems ran a form of BSD.Salvation Army sure ain't ready for the battlefield.
According to the article: Intel, Sun and AMD submitted bids, Intel won. Apple did not submit a bid. If you don't bid on a contract, don't expect to win it.
Pixar is looking for the most processing power money can buy. Everyone knows except for a few specific cases Apple hardware is slower then offerings containing Intel or AMD processors. What Apple is good for is pretty interfaces and easy of use, both of which are pretty useless in a renderfarm.
Since Jobs is CEO at both Pixar and Apple, we can sure of the fact Apple knew Pixar was shopping for a new server. Jobs being CEO at both would raise conflict of interest charges, if Pixar went with Apple hardware.
Xserver is also targeted at smaller markets, specifically ones that don't have an army of support workers.
Is renderman open source yet?
Renderman is a specification, not a product. There are various open-source efforts to implement the renderman specification, but they all seem to be dormant at the moment. See here.
Have you got your LWN subscription yet?
I rather suspect that their Intel blade system is cramming in more than 2 CPUs per rack unit on average. Apple may yet try a 4 CPU per U configuration, but the current Xserve ain't it. FWIW, 2 x 1.25Ghz G4s would put up a FAIRLY good showing against a single P4 2.8... but 1024 Xserves would take up 1024 rack Units - /48 would give over 21 full racks - a lot of space!
That was classic intercourse!
Funny, I've administered IRIX, HP/UX, Solaris, SunOS over the past 13 years....and after administering (and doing systems & application programming) for Linux for the past 4 years, I haven't found it to be less reliable than the others. In fact, I note that MORE patches for security and reliability are required for the proprietary OS's than Linux (those "monster patch" CD's for Solaris are HUGE, man!). Also, I see many of the huge proprietary vendors of operating systems are now SELLING Linux, as it erodes the market for their closed source OS offerings.
I would predict that Linux will destroy AIX, IRIX and HP/UX.....and Sun's planned weak CPU offerings for the next 3-4 years will make running Solaris very unattractive.
Its all about the distinction between shared and distributed memory architectures. Different applications benefit from different types of parralelism which the above architectures provide. If to solve the problem independent chunks of code can be run that require no communication at run time then clearly a blade type solution (distrbiuted memory) is viable, but if the calculations are co dependent on each other and require communication of interrim results then the overhead of communication can quickly become the critical path and shared memory parallelism becomes a better solution. It also depends on the level of parralelilsm built into the implementation of the algorithms inside pixars redering program itself.
Its one damn thing before another. (Dick Bird 1999)
Why the hell 1024 procesors?? Why not 1000??
1024 nodes makes a perfect 10 dimension hypercube. Hypercubes can have major advantages for speeding communications within sub-cubes, which can speed certain types of parallelized applications. Also with this architecture you can avoid a central switch system.
However, you would have to buy 10 ethernet cards per machine, which would be hard to pull off with blades, and I can't think of a way off the top of my head why a hypercube would help with frame rendering, It might be a data server locality thing... but either way, they have their reasons.
ooops! Here you go: http://tipatat.com/artworks/eclipse.jpg
Ignore my comment about 10 ethernet cards per machine... you could avoid that and still build a hypercube.
I have heard from several places that Intel's PR flacks have been flogging this story mercilessly, so it's not too surprising to see it show up in Slashdot. Twice.
:)
To get the inaccuracy out of the way -- RenderMan has been running on Linux for several years now, and I would be surprised if Linux wasn't the dominant platform for RenderMan for quite some time, outside of Pixar of course.
I am really surprised, though, that at this point in time they'd go from 64-bit to 32-bit machines, especially as 64-bit PC-like machines are just becoming available. Why not go with Itanium or the new Hammer? Each of Pixar's movies to date have been gloriously more complex and hard-to-render than the last one -- and while I know that they go to fairly extreme lengths to keep the memory footprint down I would think that they'd be bumping up against the 4GB limit already. If not now, then quite soon.
Perhaps this is just a stopgap to get Nemo finished, even 1024 servers is a fairly small cost. Certainly it would be compared to the RenderMan licenses
Every RenderMan user except for Pixar has to look to get the maximum rendering power per CPU, as the licenses are $5,000 and up, while the CPUs are far far cheaper than that. I suppose Pixar's figure of merit is rendering power per dollar or rendering power per BTU (for cooling limited situations), or even render power per ft^2. Still, the 32-bit machines are a baffling choice to me.
thad
ps. My company has a render garden (too small to be a render farm) of a dozen or so Athlons.
I love Mondays. On a Monday, anything is possible.
Actually there is one hosted at Sourceforge that is very active, called Aqsis. There were a couple of other projects like gman that never took off, or were just University projects. Aqsis is making good progress:
Aqsis
There are a few other implementations that also run on Linux like AIR, The aforementioned RenderDotC (which I believe Cinesite used), and 3Delight. Hopefully a product like Liquid (from a guy that worked at Weta), which is a Maya to RIB translator (kinda like MTOR) will also take off which could help in making a more powerful combo.
They write their own rendering software, and ported it to Linux for this switch. I'm sure they could have done a PPC port instead, if that's what was needed.
It's hard to be religious when certain people are never incinerated by bolts of lightning.
I just eclipsed Windows with Linux on my home system.
I just eclipsed my old toothbrush with a new one.
I just eclipsed the shit in my ass-crack with toilet paper.
Now, don't I sound FUCKING STUPID? Yes, I do.
This sig is part of your complete breakfast.
but when are they going to spend some money and teach their animators how to model a human that doesn't look like a puppet?
They use Linux instead of Solaris because the platform is cheaper, thus increasing their profits. Their parent company is opposed to the DVD aspects of Linux because they believe it will reduce their profits. There's no irony there, just a consistent focus on making money.
If you look closely you'll find that no major company supports GPL'd software out of principle, they all do it to make more money.
Aye, there's the rub.
Given that they picked Intel chips over Athlons, and given that they must have carefully compared chip performance on their particular application (i.e. Renderman), that says to me that Renderman is mostly memory-bandwith limited, rather than heavy-math-calculation limited.
I know a little bit about how that program works: it's not a ray-tracer. It does some basic 3d calcs to trace lines from camera to objects, and to subdivide polygons. But to determine the actual color of each pixel, it's mostly a matter of one or a number of texture-map, shadow-map, and other lookups. Each is addressing a small part of a big range of memory -- probably breaks through the cache incessantly. I'd bet that many of the geometric calculations are memory-limited too, due to the absolutely humongous number of objects they put into a given scene (e.g. blue fur).
From what I understand, DSP's are good for 2D image processing -- because the algorithms are fairly standard, and require a lot of signal-processing-like math. For 3D, perhaps the matrix calcs involved in the coordinate transforms could be done in a dsp, but as I described, that's probably not a big enough piece of the puzzle to make it worth it.
To go a little further: dedicated hardware was actually the original goal of Pixar, even before it was split off as its own company. But they noticed that hardware advances were so fast that their designs were getting obsolete fast. (The name "renderman" actually came from a quip by one of their engineers, commenting on how they'd soon be able to design a machine that could fit in a pocket-sized device that you could carry around like a Walkman.) Anyways, they eventually discarded the custom hardware, because their software-only "practice" version was getting quite acceptable performance levels all by itself -- on general-purpose processors.
"Orthodoxy is unconsciousness" - Orwell