How Google Cools Its 1 Million Servers

1sockchuck writes "As Google showed the world its data centers this week, it disclosed one of its best-kept secrets: how it cools its custom servers in high-density racks. All the magic happens in enclosed hot aisles, including supercomputer-style steel tubing that transports water — sometimes within inches of the servers. How many of those servers are there? Google has deployed at least 1 million servers, according to Wired, which got a look inside the company's North Carolina data center. The disclosures accompany a gallery of striking photos by architecture photographer Connie Zhou, who discusses the experience and her approach to the unique assignment."

Comments

The same way as everybody else.

[Anonymous Coward]

Take the heat you produce, and dump it somewhere else.

Be nice if it could be my house! I want to avoid turning the heat on till Thanksgiving if possible, but it's getting a bit tough.

Re:The same way as everybody else.

[TheLink]

Take the heat you produce, and dump it somewhere else.

Sure but there are different ways of doing it.

Google says they have the cold air come up from their raised floor.

Facebook does it differently- the cold air drops down:
http://opencompute.org/2012/08/09/water-efficiency-at-facebooks-prineville-data-center/

I'm no data center engineer but the Facebook way makes more sense to me.

Re:The same way as everybody else.

[sjwt]

I'm no thermodynamic engineer, but it seems to me facebook's way would lose out on the fact that the hot air would mingle with the cold air on the way down and warm up a little

Re:The same way as everybody else.

[Anonymous Coward]

Not particularly, either to the different ways, or the Facebook way.

Heat naturally flows up, cool air dropping down would fight that ventilation effect.

Re:The same way as everybody else.

[Gumbercules!!]

No, that would be the Apple way... In their case we're just holding the data centre wrong.

I always thought

[Anonymous Coward]

they use the chilling effect from all those DMCA notices they receive.

Google faked some of the pictures

[Anonymous Coward]

Google faked at least one picture. Take a look at this picture.

The left-hand side is exact copy of the right-hand side. Take a look at the details: The halos from the lights and the texts in the white labels.

Re:Google faked some of the pictures

[nedlohs]

The grill near the first triangular shadow at the bottom, is far more lit on the right. The triangular shadow is also different. However, those labels do seem far too similar...

Re:Google faked some of the pictures

[swillden]

Google faked at least one picture. Take a look at this picture.

The left-hand side is exact copy of the right-hand side. Take a look at the details: The halos from the lights and the texts in the white labels.

If you read the link with the interview with the photographer you'll find that she's into heavy post-production editing. Arguably, *all* of the images are "faked" to some extent. She takes many shots of each scene and layers them together selectively to get the effect she wants. She clones out stuff she doesn't want (e.g. she mentions removing an exit sign) and clones in stuff she feels is needed to make the image symmetric, and therefore more beautiful. She doesn't worry about barrel, pincushion and perspective distortion in the original shots and does heavy correction of the final images to straighten the lines and make the angles pleasing to the eye. She shot almost all of the images with long exposures in a darkened room, which makes the relatively small LEDs appear to glow intensely and makes their cast light powerful enough to be very visible when in reality it's not very visible at all.

In short, she's interested in beauty more than in fidelity, and does whatever it takes to achieve it. Personally, I think her results are fantastic.

Oblig: Series of tubes

[stevegee58]

So Ted Stevens (may he rest in peace) was right! It really *is* a series of tubes!

Immersion Would Be Better For the Environment

[theshowmecanuck]

I'm really starting to like the idea that they should submerge all their servers in circulating liquid and cool them that way. It could absorb a lot more heat and it would probably make it a lot easier to reclaim the energy from the waste heat. Air has a really low specific heat. It's why they make fluffy down filled jackets; to trap the air which acts as a good thermal insulator (and getting it wet in the winter will kill you if you don't get somewhere warm because liquids conduct/transfer heat better due to greater specific heat).

Re:Immersion Would Be Better For the Environment

[DigiShaman]

As been discussed before, the problem with immersion is the serviceability of the hardware by human hands. Even mineral oil can pose a safety issue in which a technician slips and breaks his/her neck on the floor. Not to mention it's just messy all around. Over time, liquids of any form work their way (via surface tension) in all sorts of places you don't want them being.

Re:Immersion Would Be Better For the Environment

[joib]

Current generation google datacenters already have a PUE around 1.1, so whatever they do by tweaking the cooling they cannot reduce the total energy consumption by more than 10 %. Of course, at their scale 10% is still a lot of energy, but the question is how much they could actually reduce that by going to immersion cooling. So far the anecdotal answer seems to be "not enough", since otherwise they would surely already have done it.

Re:Immersion Would Be Better For the Environment

[Victotronics]

Nonsense. My employer has a rack or two hanging in mineral oil, and the drives were first sealed in epoxy. Check out Green Revolution Cooling.

Re:Immersion Would Be Better For the Environment

[theshowmecanuck]

I'm thinking not only of the efficiency but of something similar to cogeneration. Maybe it would be better to locate centres furthur north so they could help heat a small town in winter with the exccess heat. IIRC in Winnipeg for a long time, "waste steam" from a smaller electrical generation station near the core was used to heat a whole section of office buildings in the downtown (this was from the 1920s to maybe the 80s... but the idea is still valid I think). Or would it be possible to use the excess heat to run boilers to generate electricity for other areas of the facility? I'd bet that the heated air generated now couldn't be used for either idea as easily as using liquid.

Re:Immersion Would Be Better For the Environment

[zenith1111]

In that situation you could have catwalks above the liquid and use some kind of mechanism that lifted the rack with the equipment that required service to the height of the operator.

As for the slippery floor hazard, I've recently been in a vegetable oil recycling plant and they use a floor coating that resembles sand paper so people don't slip on greasy areas.

Re:They should use the waste heat

[K. S. Kyosuke]

You do understand that the minuscule temperature differential makes this extremely inefficient, right?

Re:Nah

[K. S. Kyosuke]

No, they have an OCD photographer:

I’m obsessed with everything being symmetrical for all my work, so I cloned over the left servers to the right side. It just bothered me that there would be a hole when usually servers would be there. I wanted it to look beautiful, and symmetry is beautiful to me.

Re:Hot aisle containment

[joib]

Google runs their datacenters at quite high temperatures, the cold side is around 25 C, hot side 50 C. I suppose it would be a pretty unpleasant working environment if the main space of the server rooms would be at 50 C rather than 25 C.

Re:Hot aisle containment

[Jstlook]

Their cold air is essentially room temperature, as they're using 80 degrees (presumably F) for that side. So really they've just contained the servers, sucked all the heat out, cooled it down to room temperature, and dumped it back into the room. It's far more efficient because they're not using the servers to heat a whole room / building, then air condition each room for human usage.

Re:STEEL TUBES, full of WATER?!

[zenith1111]

No one said they invented anything. I imagine that, like me, there are a lot of readers that don't have the chance to see one of these modern data centers everyday. I found the original photos very interesting.

Zhou manipulated her pics

[vikingpower]

A Dutch newspaper, which this week published several of Zhous photos, found out - after a thread on Reddit began mentioning possible photoshopping - that, indeed, Zhou manipulated her pictures: http://www.nrc.nl/nieuws/2012/10/20/google-publiceert-prachtige-fotos-van-datacentra-maar-zijn-ze-echt-nee/ is the link ( story, of course, in Dutch )

When you look at the pictures with a critical eye, you see it quite quickly: on half of the servers, the LEDs are on the wrong side, they are simply mirrored. Zhou declared she is "crazy about symmetry". As one commentor on Reddit put it: "I knew it! For a long time, Google has been trying to make us believe that they have a lot of servers. Well, this proves that they only have very many servers" Google quite quickly admitted to the news, but did not see a reason to take the Zhou series of pictures offline.