Slashdot Mirror
Pixar Eclipses Sun with Linux/Intel
lieutenant writes "Pixar Animation Studios is replacing servers from Sun in its render farm with eight new blade servers from Rackspace. In all, the blade system contains 1,024 Intel 2.8GHz Xeon processors, and it runs the open-source Linux operating system. Pixar has ported its Renderman software to run on Linux." I'd love to see their electric bill ;)
1024 xeon's? jeeze, my electric is $120/mo with one amd and one intel running half the time.
For around $25,000 you too can make Pixar quality movies (+ the cost of those servers). https://renderman.pixar.com/
My god, I thought they had trouble scaling Linux that far. Seriously. How the hell do you do that when "stock" linux doesnt like 8 CPUs?
Tequila: It's not just for breakfast anymore!
With that type of processing power, they should be able to calculate to infinity...and beyond.
Perhaps if Sun spent more time getting their processors faster at good cost they wouldn't be losing this kind of ground. Sun took way too long to come out with their UltraSparc III processor and now clustering technology is at the point where it's much cheaper to string together a bunch of commodity PCs than purchase a high end Sun box.
Not 1024 CPUs in one box. Each CPU sits on a "blade" card and acts like a seperate system. It's a bug cluster.
"I'd love to see their electric bill "
Dude, they render stuff... would you not prefer to see that...
I could have sworn that the software couldn't run at all in 64 bit. I'm just wondering if they didn't take a step down when they converted 64-bit optimized code to run on regular high cache 32-bit pentiums.
Great for linux and anyone who has half a brain knows that you can make a very nice system from the Intel Xeon chips and Linux. But Sparcs aren't x86's and they certainly don't run the same. I've been running a server off of a pII 400 mhz Xeon with 2 megs cache on it for nearly 4 years now. It's never failed me yet and I have no intentions of upgrading anytime soon, but then again I'm not rendering anything in 3 deminsions either.
Doesn't dreamworks use this type of technology already?
Damned MPAA members ... we hate you because of your strives for world domination, but then you go and support linux ... bastards we just love to hate you.
Lastly I'm really surprised that Pixar didn't go for a server farm of OS X boxen, just goes to show ya, right tool for the job. Maybe they'll throw darwin on their at least.
Ignore the "p2p is theft" trolls, they're just uninformed
As far as I know Rackspace is a managed hosting company. Rackable Systems makes servers - Yahoo and Google both use them. Anyone know if the article has it wrong, and Pixar is actually using Rackable machines?
1,024 Intel 2.8GHz Xeon processors... I'd love to see their electric bill
Well, ignoring the power requirements of RAM, bus controllers, network adapters, hard disks which are probably used for boot only...
Intel rates these things for 74.0W thermal dissipation, which is a pretty good measure of the electrical power consumed... since, unless something is badly wrong, your Xeon chip will not dissipate energy as light or sound.
74W x 1,024 = 75,776W continuous.
Assume they're on 24/7. Assume a cost of $0.06 per kWh, including distribution, debt retirement, Ontario's capped electric rates, etc.
There are 30 days in the average month. There are 24 hours in the average day [grin]. Therefore, there are 720 hours per month.
720 hours @ 75,776W = 54,558,720kWh.
Just a little over $3.2 million per month.
I'd imagine it's less than that; their electric rate is probably somewhat less based on their consumption. But consider that the depreciation on that hardware is probably a greater monthly expense than the electricity to power it...
I'm glad Linux is ready for Pixar, because Linux sure ain't ready for the desktop.
Fire and Meat. Yummy.
I'm actually a little surprised they use general purpose CPUs for this kind of task. I'd have thought that a load of custom DSPs might be faster, and probably cheaper - How about 1 DSP per pixel (About 10 million?). I'm sure that would really zip along, if they could sort out the memory access issues inherent in this kind of application. Ray tracing is perfect for parallel execution, since each pixel really is independent of each other pixel, and each frame is likewise independent.
I am TheRaven on Soylent News
I teach MCSE courses down in Chatsworth, recently we got a lot of Engineers from boeing coming over for Windows XP classes. Why? They're dumping all their Sparc Stations and moving to XP on cheap Intel hardware. Its faster, and 2/3s of the applications they need run it already. The last third they were working on.
The IT people I talked to were surprisingly happy with XP so far. These were all Unix only kind of people actually.
The other thing they were doing were looking into dumping their Crays in favor of LINUX clusters. The comments were along the lines of how much faster and cheaper it was to put together a cluster of a 100 cheap Intel boxes than getting a new Cray. That, and they were all already familiar with the unix style interface. On top of it all, the GUI interface (I think they were running Gnome) was so much nicer than CDE on Solaris.
So Sun it getting it from both sides- Cheap Wintel boxes and Cheap Linux boxes. No wonder they finally relented and released Solaris 9 on Intel.
They have half-depth 1U boxes. That's right, two servers in 1U, back to back.
Includes space between the two for cabling and cooling.
They specialize in delivering easy to manage (physically) racks of highly commoditized systems.
(I work with them in a reseller relationship)
Imagine a 71U rack(minus 1U for a switch), with 142 boxes, all dual proc. 248 procs in a rack!
Man, I wish they'd put the right link in there.
Striving to achieve a lower state of conciousness
A lot of people are going to be saying "just one example of how Sun is dying", but coming from a place that runs several hundred Sun machines (and being a Sun fanboy), I can understand why they made this switch. For shere processing power on-the-cheap, the x86 world has had a lead on Sun and other big UNIX vendors for a few years. Having a decent OS (linux) to run on those machines, makes it even easier to switch.
It's about using the right tool for the job, and now that x86/linux/bsd has matured to a point where it can be used for some professional applications, it only makes sense to see things like this happen.
Sun is going to be around for a long time. As many other people have pointed out, they're just retreating somewhat to more a of niche market, where they are the right tool for the job.
The executives at my company are very interested in linux, because of the outrageous leap in processing power per dollar, and the reductions in CPU-based licensing costs for software like Oracle is staggering. The concern, though, is stability.
Sun Fire and Enterprise servers are really expensive, but they stay up all the time. Swapping a failed processor or NIC or memory stick without halting the box is really important on a mission-critical server. Likewise, a well built Sun box never panics, and if it ever does, Sun will insist that their engineers look at the crash dump to figure out what went wrong.
I think Linux has won the performance battle, but what about the stability battle? You need to win both to win the war.
I recently attended a talk by Google's chief engineer. They have approximately 15,000 x86 machines running Linux at seven data centers in the United States.
Weird failures occur so often, such as disks returning garbage without the controller informing the OS, that Google does a checksum on _every_ data structure in their user-level software. He also talked about how Linux is good enough for them, but it doesn't perform well with respects to I/O under heavy load. He says they like Linux because they have the source-code and that they minimize excessive I/O loads on their machines. Nobody asked why they don't use FreeBSD but I suspect its because Linux has better hardware support and Google builds their own machines with numerous different components based on the latest technology.
Sun uniprocessor performance has been very uncompetitive for quite some time now. I bet they would have switched a long time ago if it was not for the difficulty of porting software from Solaris to Linux. Plus human inertia ...
The worst problem for Sun is once they loose customers to Linux, there is no turning back.
They still hold well in 64-bit area, however, once commodity hardware such as x86-64 gets there, this battle will also be over.
This is the main reason why the company is likely to go down the drain.
The only natural base is e. Man arbitrarily likes whole numbers, nature like real numbers, and e is everywhere.
Therefore, ln (1024) = 6.931471806... which is not an even number.
I suggest therefore that an even number of processors for the render farm is either
e^6 = 403.4287935 or
e^7 = 1,096.633158.
Of course, Intel is wedded to the whole numbers of processors thing, which utterly thwarts mathematical logic and correctness. Their site also runs on IIS, so what other foolishness can you expect? Heathens.
Fire and Meat. Yummy.
... I guess we know which one you are.
Pixar is on the right track. I do ASIC verification, mainly on Sun boxes (fastest USparc IIIs, multi-proccessor, 14GBs memory, etc). Lately, I have been running the exact same jobs on an LSF enabled Linux farm of Intel boxes.
The improvement is 3-4 times speedup ie 8 hour Sun jobs take 2 hours on Intels.
For the price of one dual proccesor Sun workstation, you can get ten Intel boxes running linux.
Not only is the speedup great, I need less licences to run the CAD software (doing multiple regression jobs). Since a license seat per CAD tool can run from 30K to 200K each plus 10% a year maintence fee, the savings are huge.
Changing over to linux was trivial. I like and have used Suns for years and Suns were a major player in this industry. But I firmly believe that this paradigm is going to be a SUN KILLER!
1982 - born in Nebula - incorporated with 4 employees
1984 - protostar - NFS is introduced
1995 - main sequence begins - Java Released
1996 - red giant - Using Java technology, NASA engineers develop an interactive application allowing anyone on the Internet to be a "virtual participant" in the space administration's groundbreaking mission to Mars.
SUPERNOVA - Sun battles MS over Java and Windows
Blackhole - TODAY!
References:
http://www.sun.com/aboutsun/coinfo/history.html
Salvation Army sure ain't ready for the battlefield.
According to the article: Intel, Sun and AMD submitted bids, Intel won. Apple did not submit a bid. If you don't bid on a contract, don't expect to win it.
Pixar is looking for the most processing power money can buy. Everyone knows except for a few specific cases Apple hardware is slower then offerings containing Intel or AMD processors. What Apple is good for is pretty interfaces and easy of use, both of which are pretty useless in a renderfarm.
Since Jobs is CEO at both Pixar and Apple, we can sure of the fact Apple knew Pixar was shopping for a new server. Jobs being CEO at both would raise conflict of interest charges, if Pixar went with Apple hardware.
Xserver is also targeted at smaller markets, specifically ones that don't have an army of support workers.
I agree. I think whats going to end up happening long term is Windows will take and keep the desktop (I just don't see it happening with Linux, this coming from someone whose used it as his only OS at home for ten years), Linux in the datacenter, and OS-X in the same niche role its in now, with the caveat that I think it'll start pulling away the tiny percentage of people who want to run Unix on their desktop.
Ten years running Linux, and tomorrow morning I'm dropping the bills on one of those spiffy gigahertz 17" iMacs. I want Unix, and I want more functional stability than Linux has ever given me (not OS stability, but stability in terms of what programs I can use to do what, what works with what else, etc... )
I rather suspect that their Intel blade system is cramming in more than 2 CPUs per rack unit on average. Apple may yet try a 4 CPU per U configuration, but the current Xserve ain't it. FWIW, 2 x 1.25Ghz G4s would put up a FAIRLY good showing against a single P4 2.8... but 1024 Xserves would take up 1024 rack Units - /48 would give over 21 full racks - a lot of space!
That was classic intercourse!
You know, I really don't know what the logic is of arguing that. The people who are using Linux on their desktops now know Linux well enough to completely disregard that. I suppose you will scare newbies away until someone gives them a knoppix CD to play with,
I use Linux on my desktop. It's great. It's beautiful. But it's *still not ready* for the desktop - as in, it's still not ready to compete with Windows - because it's still more comparable with Windows 3.1 than it is with Windows XP.
Maybe Linux is more than ready for the desktop, it just isn't ready for your narrow view of what a desktop should be. And it is not that I really care that you are not satsified, but bitching to a bunch of volunteers seems a bit insane, because I don't think they really care that your are not satisfied, either.Maybe my viewpoint is narrow. Or maybe I'm as big a power user as you can get without actually *thinking* in C.
Note that I administer my own domain on a server farm of Linux and OpenBSD machines which live in my bedroom.
Primarily, my main desktop is an e-mail drone. If Evolution actually worked (ie. didn't take 8 minutes to exit on my machine), then it would be fine. But without a spellchecker competitive to prevalent software, Linux/KDE or Linux/Gnome doesn't even make a good e-mail drone. The spellchecker is so 1995. I want an underlining spell checker.
Does that give me a narrow viewpoint, because I expect features which I could take for granted among the apps of more estabished operating systems? Apparently.
Your lack of a realistic viewpoint and your immediate dismissal of my page as FUD is symptomatic of what is wrong with the Linux/OS community, and why I'm starting to believe that Linux will never be able to get its shit together enough to be more of a fringe group like Apple users.
Try using Windows 2000 or XP sometime. Look at it from a user's perspective - you know, the sort of idiot who opens e-mail virii and who makes the *bulk* of the computer-using public. From that perspective, Windows is great. It does everything reasonably well, whether you're a newbie or expert. Linux doesn't do that yet, and therefore isn't as good a desktop solution as Windows.
I'm waiting for the day someone can prove me wrong, but until you get some actual real-world experience with what end-users want from their operating systems, you'll still just be a whiny 14-year-old living in Mommy and Daddy's basement.
Fire and Meat. Yummy.
Its all about the distinction between shared and distributed memory architectures. Different applications benefit from different types of parralelism which the above architectures provide. If to solve the problem independent chunks of code can be run that require no communication at run time then clearly a blade type solution (distrbiuted memory) is viable, but if the calculations are co dependent on each other and require communication of interrim results then the overhead of communication can quickly become the critical path and shared memory parallelism becomes a better solution. It also depends on the level of parralelilsm built into the implementation of the algorithms inside pixars redering program itself.
Its one damn thing before another. (Dick Bird 1999)
Why the hell 1024 procesors?? Why not 1000??
1024 nodes makes a perfect 10 dimension hypercube. Hypercubes can have major advantages for speeding communications within sub-cubes, which can speed certain types of parallelized applications. Also with this architecture you can avoid a central switch system.
However, you would have to buy 10 ethernet cards per machine, which would be hard to pull off with blades, and I can't think of a way off the top of my head why a hypercube would help with frame rendering, It might be a data server locality thing... but either way, they have their reasons.
ooops! Here you go: http://tipatat.com/artworks/eclipse.jpg
The most important linux skill is how to use internet for help, not any unix skills. For a newbie, it is a hit or miss affair. He grabs a modern desktop oriented linux, installs it in 30 or less minutes, if all of his hardware are supported and all programs newbie wants are already installed, good news, we have a new linux fan. Chances are, that won't happen.
If something goes wrong, it is best option for linux fans that newbie just forgets the idea, right then. Most probably he now has a functional system but with a non-functional usb mouse, cd burner or a sub-optimal refresh rate. He will want to fix and use the system. It is just the mouse, or the printer, or excel documents. He almost succeeded in this linux thing!
Wrong. He still misses the crucial skill.
He will try to fix it and fail, seek help and fail again, try to skim docs and fail, learn where to seek help and fail, read documents and seek help at the correct place with the correct attitude and if he has some luck, succeed at last. Now we have a brand new whiner instead fo a fan. Worse, he half knows what he is talking about.
Eveyone whines about windows all the time too, but it is not the same thing. We don't want scared potential new users. In case of windows, user already knows how much of that whining is about a real problem, that is not the case with linux.
Solution is aiming higher. Linux has to be considerably easier to use and install than windows because non-techie users just have a lot of experience with windows. Even if the fix isn't optimal, there is always a fix a phonecall to someone you know away. Linux doesn't have nearly the same installed base so is denied the luxury. Linux still requires a crucial skill; it shouldn't.
In some areas (considering desktop) linux already is better than windows and in others, it is not too far behind. But it has to better on all fronts. Till than, linux is not ready. You can argue that had market shares of linux and windows magically flipped, we would be saying windows is not ready. Probably you would be right, too. But market share (or rather, user base) has not magically flipped and that is not irrelevant.
I know, I should have read the grandparent.
Gentlemen, you can't fight in here, this is the War Room!
I have heard from several places that Intel's PR flacks have been flogging this story mercilessly, so it's not too surprising to see it show up in Slashdot. Twice.
:)
To get the inaccuracy out of the way -- RenderMan has been running on Linux for several years now, and I would be surprised if Linux wasn't the dominant platform for RenderMan for quite some time, outside of Pixar of course.
I am really surprised, though, that at this point in time they'd go from 64-bit to 32-bit machines, especially as 64-bit PC-like machines are just becoming available. Why not go with Itanium or the new Hammer? Each of Pixar's movies to date have been gloriously more complex and hard-to-render than the last one -- and while I know that they go to fairly extreme lengths to keep the memory footprint down I would think that they'd be bumping up against the 4GB limit already. If not now, then quite soon.
Perhaps this is just a stopgap to get Nemo finished, even 1024 servers is a fairly small cost. Certainly it would be compared to the RenderMan licenses
Every RenderMan user except for Pixar has to look to get the maximum rendering power per CPU, as the licenses are $5,000 and up, while the CPUs are far far cheaper than that. I suppose Pixar's figure of merit is rendering power per dollar or rendering power per BTU (for cooling limited situations), or even render power per ft^2. Still, the 32-bit machines are a baffling choice to me.
thad
ps. My company has a render garden (too small to be a render farm) of a dozen or so Athlons.
I love Mondays. On a Monday, anything is possible.
Sorry Guys... This article looks to be a bit off base!
/. I though you guys did better about checking this kind of thing out! Just because it's on c-net doesn't mean it's accurate. Well kudos to who ever really got this job.
-- Not an Official RS response --
I work for Rackspace Managed Hosting. The company the link "Rackspace" references in the C-Net article. This kind of cluster is not consistent with our business. We are most focused on web-centric managed hosting vrs colocation. A rendering cluster is something that, from my experience, we've never done. Also We don't carry Blade servers. C-Mon
Matthew Montgomery
Rackspace Managed Hosting.
I just eclipsed Windows with Linux on my home system.
I just eclipsed my old toothbrush with a new one.
I just eclipsed the shit in my ass-crack with toilet paper.
Now, don't I sound FUCKING STUPID? Yes, I do.
This sig is part of your complete breakfast.