Managing Servers In the Frigid Cold

1sockchuck writes "Some data centers are kept as chilly as meat lockers. But IT operations in colder regions face challenges in managing conditions — hence Facebook's to use environmentally controlled trucks to make deliveries to its new data center in Sweden, which is located on the edge of the Arctic Circle. The problem is the temperature change in transporting gear. 'A rapid rate of change (in temperature) can create condensation on the electronics, and that's no good,' said Facebook's Frank Frankovsky."

Comments

From Minnesota here

[onyxruby]

This isn't anything new, anytime you take something from the extreme cold and bring it inside you risk condensation. This is usually dealt with by simply letting something sit at room temperature for several hours before powering it on.

In the middle of January if you take a freezing cold delivery and power it on right away and fry your new (XXXXXX) you deserve to void your warranty. There is no excuse for stupidity. Why is this on slashdot as news?

Re:From Minnesota here

[Pieroxy]

There is no excuse for stupidity. Why is this on slashdot as news?

Because most slashdotters aren't from extremely cold areas. What seems like obvious to you isn't even considered by most, just because most aren't subject to the same conditions as you are.

Let me ask you another question in return. I think I deserve an answer, since I answered yours: Why do you think everybody should know what you think is obvious?

Re:From Minnesota here

[vlm]

Living slightly to the east, yet just as cold in winter, the strategy is to leave the gear sealed in the box while you prep the racks and wiring and gather tools. Its really not that complicated.
You don't have to wait until the gear reaches room temperature, merely gets above the interior air dew point, which I assure you is very low in the winter.

Re:From Minnesota here

[prakslash]

Why is this on slashdot as news?

So you can get a taste of the luxuries you can afford after a 100 billion dollar IPO. Why wait a few hours before powering up you equipment when you can transfer it using expensive, climate-controlled trucks. At Facebook, even lifeless plastic and metal rides in style on the gravy train.

Re:From Minnesota here

[Joce640k]

Have they not heard of an invention called "plastic bags"?

I hear they do wonders against condensation.

Re:From Minnesota here

[gl4ss]

well, it's not particularly interesting article I'm afraid, the interesting part is that they mix incoming fresh air with already circulated heated air instead of having isolated heat exchanger arrangement, other than that it seems like a fairly traditional datacenter - no ssd devices dipped in epoxy sitting outside in ice or crazy stuff like that.

"woot heated trucks".. well duh, not everything likes to be frozen. ever had partially frozen milk on school lunch? it sucks and we walked to school. both ways. or bicycled(on ice). or used kicksleds(if there wasn't sand on the route).

Re:From Minnesota here

[vlm]

Oh I thought of another one. The problem has nothing to do with temperature. The problem is when the indoor/outdoor/dew points intersect which happens all the time, not just when its cold.

One excruciatingly humid summer day I was hauling around a protocol analyzer worth about as much as my car, and it cold soaked in front of the car air conditioner duct cooling itself to 40 degrees or whatever the AC output is, then it was dripping condensed water as I carried it into the customer premises, an un-airconditioned factory floor. So I'm sitting there doing nothing and explaining to the customer how I have to do nothing, until the test set dries off because its too cold (customer VP looks out window at blue sky 110 degree day). Yes that was an unpleasant meeting.

Re:From Minnesota here

[LordLimecat]

Because they dont understand common sense solutions to simple problems, and would rather throw money at the problem?

Re:From Minnesota here

[oodaloop]

In the Marine Corps, I had some cold weather training before a deployment to Norway. We were instructed to leave our rifles outside of our tents. Otherwise, they would accumulate condensation inside the barrels, which would then freeze when you walk outside. Lots of fun stuff like that.

Re:From Minnesota here

[hawguy]

RUNNING machines in plastic bags? I'm not sure you understood the problem domain.

You don't understand the solution.

Put the equipment into plastic bags before loading it on the truck (if it's new equipment, it's probably already wrapped in plastic.

Then when you unload it in the warm datacenter, the moisture condenses on the outside of the bag instead of inside your server.

Once the server is up to room temperature, take it out of the bag, rack it and plug it in, and you're good to go. (Note that in warm humid states, you can have the opposite problem - the cold server is taken from the 65 degree datacenter out to the 95 degree and humid outside air and moisture condenses on it. The plastic bag works here too.)

You don't need to go to Sweden to experience cold temperatures - many datacenters throughout the USA experience temperatures cold enough to cause condensation problems for at least part of the year. The plastic also helps protect equipment that's exposed to moisture that condenses in clouds and falls to the ground (i.e. rain) as it's transferred from the truck to the facility. A problem that Facebook will discover once they open their first datacenter in a rainforest and perhaps they can invent some self deploying canopy that shields the equipment from this mysterious moisture from the sky since they don't seem to like the low-tech plastic bag solution.

Re:From Minnesota here

[petermgreen]

Even if you do have the ability to artificially raise or lower the humidity that doesn't mean it makes sense to keep it at the same level all year round.

Most electronics is specified for quite a wide range of relative humidity. Usually 5% to 95% or so.

In winter you want low relative humidity to reduce the risk of condensation on stuff brought in from outside (yeah you try to seal stuff and let it warm up before unwrapping but mistakes and emergencies happen). It's also cheap to achieve low relative humidity due to the low outside air temperature (for a given absoloute humidity relative humidity goes down as temperature goes up).

In summer humidity doesn't matter so much since stuff brought in from outside will be warm. It's also likely to be more expensive to achieve low relative humidity since it involves active dehumidification (which is achieved by cooling the air to the point where the water condenses out)

Re:From Minnesota here

[DerekLyons]

This isn't anything new, anytime you take something from the extreme cold and bring it inside you risk condensation. This is usually dealt with by simply letting something sit at room temperature for several hours before powering it on.

True. But what you're forgetting (generously assuming you knew it in the first place) is that condensation isn't the only issue. Servers are made of a variety of materials - all of which expand and contract with temperature at different rates. Extreme cold can actually physically damage equipment by pulling pins from sockets, cracking PCB traces, etc... etc... That damage (obviously) can't be fixed just by letting it sit in at room temperature for several hours.
 
Just a bit of trivia since Mars landers and rovers are in the news today... it was the lenses in the PanCams that set the lower temperature bound for the Spirit and Opportunity rovers - the lens mounts shrunk faster than the lenses, and eventually would shrink enough to crack the lenses.

Re:From Minnesota here

[afidel]

Low RH is bad because you get static buildup. Sure we've got anti-static wax on the floor and all the cages are grounded, but I still don't want to risk frying a computer because I couldn't keep RH in the right range. Also low RH is easy to achieve since CRAC units due it by their nature =) Even in the middle of summer you have to run humidifiers.

Re:From Minnesota here

[Pf0tzenpfritz]

That's not the point. Every time you move something from a cold place in to a warmer one (higher humidity in the air implicit, since higher temperature means higher point of saturation) condensation occurs, as the air near the cold item cools down and "drops" dew on the cold surface. If the latter is intransparent, like server rails, backplanes or transformator cores/coils the condensed water will collect there. Basically, condensation always occurs where temperature difference exist and it always happens at the coldest surface in the room. (Hence all the trouble with moisture and poorly insulated walls in colder regions.) Now a truckload of servers is basically one large thermal buffer. Move it from arctic cold -supposed the machinery had time to adapt to outside temperatures- to room temperature and you will find a lot of water condensing. We're talking about tons of material -with a lot of surface- that will take hours to warm up.

Re:From Minnesota here

[AK Marc]

Early in my career I worked for Polaris and used to arrange deliveries of computers to places in the Arctic circle. We took a number of precautions keep the equipment from getting destroyed by the extreme cold. We never kept things in a heated container though and I was shipping computers to places like Nome Alaska. We never shipped anything in a heated truck though.

Nome isn't in the Arctic circle. Call us when you go someplace interesting, like going on a Polaris from Faribanks to Anaktuvuk Pass with a server strapped on the back (no, I haven't. Alaska Air has multiple destinations above the arctic circle, and I've been to all of them, on a plane, I try to not go outside when it's cold).

My favorite was discussing wind chill with regards to electronic equipment, or finding anything rated to work below -40, as most goes to -40, so anything rated for colder requires non-standard boards, caps, chips, etc., and there's just so little need that nobody does it. Best I got for common things was "buy enough and we'll put in the warranty that cold temperatures will not void the warranty, so if it fails, we'll replace it, but we won't guarantee it will work." You'd be surprised how many things just won't power up colder than -60, I can only assume issues with the power supplies not liking the cold, but I'm not going to spend my time at -60 troubleshooting the exact reason. Interestingly, some will work once booted, if you can boot it, so you take it inside, warm it up, then run it outside and power it up as fast as possible. Oh, and it wouldn't reboot, so if it did have an issue, it'd be off until May or so.

Try -68 F Room Temperature

[Isbjorn]

This is nothing--years ago I deployed PCs at Alaskan oilfield installations. Extreme cold makes everything brittle, kept having problems with things like cracked motherboards, just from setting the PC on a desk.

Re:Try -68 F Room Temperature

[vlm]

Cold drive bearings don't want to spin up / SMART fail from drive motor overcurrent.
Happens to cooling fans too. Fan can't spin so equipment overheats.
I've never knowingly had a voice coil bearing seize up, which is interesting because its probably the lowest power actuator in the system yet probably the highest precision / smallest tolerances.

In Canada, Nokia DSLAM's Overheat!

[N1zaam]

A few years back while doing Tier 2 level support for a major Canadian telco, I started seeing overheating alarms from some Nokia DSLAM's. The odd thing was that it was -40C outside at the time. It turns out the fan's on these DSLAM's froze solid and the devices thought they where overheating and throwing alams left right and centre. We had to put a tarp over them with a heater during the winter to make sure they kept going.

Re:In Canada, Nokia DSLAM's Overheat!

[mooingyak]

The odd thing was that it was -40C

Random tidbit: at -40, you can just say -40. Kelvin doesn't go negative, and -40F == -40C. Unless you're using some other scale, but that should cover most of the cases.

Re:In Canada, Nokia DSLAM's Overheat!

[j_sp_r]

Not including units should be a capital offence. don't encourage it.

Re:Heh

[Antipater]

And it was uphill both ways to get there, right?

Wild climate in N. Sweden.

[Alkonaut]

Luleå has to have one of the most extreme temperature ranges anywhere. Summer temp is quite consistently 15-20C with occasional peaks of 30C and winter temp is zero to 40C below. So the range is nearly 90C (130F)! This of course seasonal variation and not "rapid change" so data centers should not be affected by this. The fastest changes there are probably in winter when the temperature in rare cases can go from -40 (and zero humidity) to zero (and damp) in a day or two. That kind of change, especially the other way round, could mean trouble (condensation in air in/outlets etc.)

In fact, if google just wanted cold/dry climate, there has to be better locations. Northern sweden is mild, and has quite warm summers. Arctic inland climate further from the gulf stream atlantic would be more logical. Border between Russia and Finland for example. But there are probably logistical reasons (huge cargo airport, good port, good roads, railroads, lots of good technical people, ridiculous backbone connection) that placed the datacenter there.

Humidity more than Temperature

[Roger W Moore]

Every time you move something from a cold place in to a warmer one (higher humidity in the air implicit, since higher temperature means higher point of saturation)

Actually that is not implicit. Up here in the frozen wastes of central Alberta in the winter the indoor humidity drops to incredibly low values of 10-20% because there is no moisture in the outside air because it is at -40C and even then has low humidity. This means that condensation is never really a problem - you might get a bit of it but it very quickly evaporates because of the incredibly low humidity inside. In fact the humidity gets so low that our data centre has a humidifier to bump it up to the safe operating range of machines.

Conversely in the UK where there is no extreme cold weather (yes I know the beeb goes nuts if London drops below -5C but sorry, that doesn't count!) but lots of humidity. As a kid I used to have far more problems with my glasses fogging up when I came inside during the winter that I do in Canada.