Slashdot Mirror
One Data Center To Rule Them All
1sockchuck writes "Weta Digital, the New Zealand studio that created the visual effects for the 'Lord of the Rings' movie trilogy, has launched a new "extreme density" data center to provide the computing horsepower to power its digital renderings. Weta is running four clusters that are each equipped with 156 of HP's new 2-in-1 blade servers, and use liquid cooling to manage the heat loads. The Weta render farms currently hold spots 219 through 222 on the current Top 500 list of the world's fastest supercomputers."
Imagine a Beowulf cluster of these...
a balrog cluster of them!
How do they release the heat in the hot water?
Two Towers?
I'll show you heat loads.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
I can't mention who I work for for obvious reasons, but we did some experimenting with "extreme density" computing some time ago as part of a black ops project for the government. We achieved densities previously unheard of by man.
Unfortunately, we got greedy. We increased the density so far that the entire facility ended up collapsing into a black hole, wiping out much of the state of North Dakota. We were able to contain the damage, and we've managed to keep it a secret by replacing the state with a hologram projection, but eventually someone is going to go there and figure out that something is amiss.
I'm of a mixed mindset when it comes to water-cooled datacenters.
On one hand, you've got the makings of a biblical scale disaster with all that water and electricity mixing.
On the other hand, you can't argue with it's effectiveness.
I'll stick to non-catastrophic issues when my
air conditioner breaks down.
Check out my sysadmin blog!
That will be your last mistake. Ever.
I think it's great that they are using liquid coolants for their system. Whenever I see a traditional server farm, I just can't help but think that air conditioners are so inefficient for the task of cooling computers. Not only do you have to cool the air, you also have to blow it around. The floors in some data centers are raised just to allow better airflow. And if you think about it, only the insides of the computers have to be cooled, not the entire freaking room. I hope this ushers in a new age of more power-efficient computing.
I also think it's pretty funny that a supercomputer is used to make movies.
A NYC lawyer blogs. http://www.chuangblog.com/
Note the vague phonetic similarity between "North Dakota" and "Mordor."
Did this project happen to hand out t-shirts... or, hm, something less obvious like... rings?
If their rendering farm has enough network bandwidth to be on the top 500 list, then they're not doing it right.
I wonder how much they have to pay for external bandwidth. I always thought that "super data centers" are used to help split the job between multiple special effects studios, so, say, group in London can work on part of the shot and still have all data in the same place.
Except in New Zealand there are no "unlimited" plans, and there are severe bandwidth caps in place.
Hyperom.com
"Weta Digital, the New Zealand studio that created the visual effects for the "Lord of the Rings" movie trilogy, has launched a new "extreme density" data center to provide the computing horsepower to power its digital renderings. Weta is running four clusters that are each equipped with 156 of HP's new 2-in-1 blade servers, and use liquid cooling to manage the heat loads. The Weta render farms currently hold spots 219 through 222 on the current Top 500 list of the world's fastest supercomputers."
I think people should keep the above in mind next time they talk about GPUs being a replacement.
...will still pale in comparison to ILM. LOTR had some of the worst compositing I had ever seen. The fact that ROTK won an Oscar for Visual Effects only further illustrates the irrelevance of the politics behind the Oscars.
Just curious, but what does ILM run?
I'm from 2053 and my watch would hold today's 1st to 16th spots . Only 16 cores, I know it's lame but then again I only paid 25 EuroYens. Damn cheap piece of crap made in the United States of Canaxico, you know how it is.
I also think it's pretty funny that a supercomputer is used to make movies.
It was pretty funny forty years ago, too.
The LHCb experiment has a large processor farm for their online data analysis, all water cooled. Apparently it makes the computer scientists very nervous. OTOH, the main computing centre just uses air cooling, so we've got a real mix of technology.
xterm -n 8
Amazing! So what was North Dakota like before it became the vast, desolate wasteland devoid of any trace of humanity it is now?
but have you considered the following argument: shut up.
Does it run chrome?
I was working on a show which had a bunch of SGI Boxes similar to this. Two servers (2 dual core processors per server) in a 1U case. Only problem with it was that if something went wrong with one of two servers in the box you had to send the whole box back to SGI to get fixed.
Now you would think that normally it wouldn't be a problem if 8 CPUs out of 200 go down on a farm but the way they usually set up the farm (in my experience) is by assigning some processors to each department using a priority structure. So lets say the simulation department had sent some stuff to the farm. 20 min later the stuff finishes. Now lighting can get those CPUs at a low priority until the sim guys send another job to the farm. Problem is if your have a small section of the farm and one of those machines in that small section broke (which happens a lot when the farm starts to get flooded near the end of a show) your stuck not being able to do work for a while until IT gives you priority on some other machines (which at the end of a production can take a ling time).
What I would love to see is the farm software seperating the job by CPU cycles so that when I come in early and there nobody's around I could get the full processing power of the farm to my self.
The Weta render farms currently hold spots 219 through 222 on the current Top 500 list of the world's fastest supercomputers.
That's a lot of orcs.
I drank what? -- Socrates
After I virtualized way back in 2003 I discovered ... that if I built my own "sky" each "cloud" I deployed in its own virtual machine was very simple and easy to understand and maintain.
Just like Unix itself, which takes small working units and connects them together to make a complex whole, the "sky" is complex but each "cloud" is simple.
And makes it easy to deploy and move "clouds" to super-centers like the New Zealand data-center as it makes sense (speed, location, power, price) to do so.
Today Fedora Core + VMware's GNU solution (aka Vmware Server 1.x, formerly VMware GSX). Both are free. Virtual machines can run anything. No connectivity to the host, even put the firewall in a v.m. Runs fine on a dual Xeon P3 / 1.2GB ram and easy disk management using LVM. A backup (duplicate machine) gives me twelve clouds in my current sky playground.
Richard showed up a couple years ago at the Wairarapa Railway Modellers annual BBQ with a 7 1/2" scale locomotive that was built at Weta Workshops for a film. In the photo located at http://www.certsoft.com/NZ2/richard_taylor.jpg Richard is the one in the middle.
North Dakota is 2-dimensional, you insensitive clod!
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
to rule them all... (sorry, had to do it)
Impetuous! Homeric!
I can't seem to find the video I'm looking for but I'm a fan of the new Sun Data Center cooling design. Works on the idea of cooling each cab individually and not all the rest of the rubbish air floating around. I know its been done before but this time its done right. Info somewhere here