Slimming Down a Supercomputer

1sockchuck writes "Happy Feet animator Dr. D Studios has packed a large amount of supercomputing power into a smaller package in its new render farm in Sydney, Australia. The digital production shop has consolidated the 150 blade chassis used in the 2007 dancing penguin feature into just 24 chassis, entirely housed in a hot-aisle containment pod. The Dr. D render farm has moved from its previous home at Equinix to the E3 Pegasus data center in Sydney. ITNews has a video and photos of the E3 facility."

Comments

Priceless

[syousef]

Cost of real estate in prime metropolitan area - $15 million
Cost of state of the art server rocks - $30 million
Cost of flying in a cooler the size of a small bus on a 747 - $2 million
Cost of seeing data center employee's face when they realise they're on call 24/7 for no extra cash - Priceless.

About relocating supercomputing power to Australia

[hallux.sinister]

You all do realize that electrons spin backwards there, right?

What about the rest of it ?

[slincolne]

Physical space is the least interesting point of this article. Other things would be:

What racks are they using (at least 42RU in height) ?

How do they get power into these (4 chassis, each with 6 x 15A power inlets) ?

Are they using rack top switches, or is there more equipment?

Are they using liquid cooled doors - if so whose ?

I once tried to get answers from HP on how to power their equipment at this density - they diddn't have a clue. It's worth remembering that each of these chassis has six power supplies, each rated at up to 2.2KW. Even allowing for a 2N configuration, that's a massive amount of power, and a lot of cables.

Re:What about the rest of it ?

[imevil]

TFA says they use 48RU, and each cabinet uses 14.4 kW (60A) which in my opinion is not that impressive: you just need 3 phases at 20A, 240V.

As for cooling, you can easily get away with no water-cooling if your hot aisle confinement is well done. From the pics it is just Dell's 1U servers, and if you fill one 48U rack with those you do get to 14.4kW. But not all racks are for number-crunching, you have racks for storage, control and network, and those make less than 8kW.

The problem is not powering those things, but more cooling. With a good hot-aisle or cold-aisle confinement you can go up to 15kW/rack, but depending on the air volume, you're quickly screwed up if the cooling fails.

Re:What about the rest of it ?

[dbIII]

Dell servers? Australia is the land Dell forgot so that's the last thing you want. From Australia you end up talking to Dell people from three different continents to get even the smallest problem solved, and the timezone difference as a barrier to communication spins things out to weeks that should be solved in a couple of days. Plus there is far better gear from whitebox suppliers using SuperMicro boards so why use Dell in the first place? Dell can't do two boards with 8 cores on each in 1U and I've got some of those a couple of years old now.
As for power, with 3 phase, plenty of spare capacity and a good electrician to add more outlets it's not a problem.

Re:Tried to check out the E3 Networks site

[deniable]

It's targeted at managers with money. Need I say more?

Re:About relocating supercomputing power to Austra

[M8e]

Not only that they are also upside down.