Cooler Servers or Cooler Rooms?

mstansberry writes "Analysts, experts and engineers rumble over which is more important in curbing server heat issues; cooler rooms or cooler servers. And who will be the first vendor to bring water back into the data center?"

Comments

Aquafina...

[vmcto]

Will probably be the first vendor to bring water into the datacenter... I believe I've seen evidence in some datacenters already.

Re:Aquafina...

[_Sharp'r_]

Well, since my last company had most of their servers in a data center room where we had two different floods, I'd say I have a pretty good idea which hosting company will be the first to bring water into the data center ...

The first problem was snow that piled up outside, combined with clogged drains, that led to melting snow coming in through the wall where some pipes entered/exited. Since their layout was power in the floor and networking in the ladder racks, it's actually pretty amazing that a large portion of the power plugs and switches still worked, even while being submerged in 6 inches of water.

So about a year after they had taken care of that issue, a water pipe for a bathroom on the floor above burst, and of course the water came down right in our room in the hosting center. It wasn't so bad until the flourescent lights in the ceiling filled up and started to over flow. We were able to limit the damage by throwing tarps over the tops of all the racks (there goes your cooling effect, though), but we still lost about 100K worth of server and switching equipment.

So yeah, water in the data center? It's been done.

well I've always wondered this

[Saven Marek]

I've always wondered this. why have duplication of a function in a server across every single server box when it could all be done in the environment. For example all servers get electricity from the server room and all servers get network from the server room so why not all servers get cooling from 10F cooling in the server room.

It makes sense!

Re:well I've always wondered this

[T-Ranger]

What I have never understood is why servers virtually always have AC power supplies. Yes, you can get NEBS(?) compliant servers that take DC, but this isnt really a general option, but a distinct model line compleatly.

UPSs take AC, turn it to DC, charge their batteries. A sepearate system takes DC from batteries, inverts it and sends out AC. (Good UPSs, anyway. Otherwise they are "battery backups", not uninteruptable) Computer power supplies take AC and distribute DC inside the case. WTF?

Why doesn't APC start selling ATX power supplies? Directly swap out AC powersupplies, have them plug into the DC providing UPS and/or per-rack (or even per-room) powersupplies.

Electrical codes are a BS excuse. Even if you needed verdor specific racks, a DC providing rack is, so far as the fire marshal should be concerned, just a very large blade enclosure, which are clearly acceptable.

I cant beleive that Im the first one to ever come up with this idea. So there must be some problem with it.... Some EE want to explain why this wouldnt work?

Re:well I've always wondered this

[david.given]

Some EE want to explain why this wouldnt work?

I'm not an EE, but it's something I've always wondered about. I don't have a datacentre, but I do have far too many computers: why does my machine room contain about fifteen wall warts, all producing slightly different DC voltages and plugged in to their various appliances via fifteen different non-standard connectors? Why not just have one low-voltage standard and have all these things plug into that?

One possible reason is that (IIRC) power losses vary according to current, not voltage. By increasing the voltage, you can push the same amount of energy down a wire using a smaller current, which limits losses. This is why power lines use very high voltages.

This means that if you produce regulated 5V at one side of your datacentre, by the time it's reached the other side it's not 5V any more. But it should be easy to get round this by producing 6V and having DC regulators; they're very small and extremely efficient these days.

However, I suspect that the main reason why this kind of thing isn't done is inertia. There's so much infrastructure in place for dealing with high-voltage AC supplies that you wouldn't get off the ground using DC.

Re:well I've always wondered this

[drinkypoo]

When you use DC power in a data center you run a high voltage (24V, 48V, depending on equipment) and then it's regulated down to all the usual voltages. However, it's a lot easier to transmit AC power over distances. It's not exceptionally lossy to convert AC to DC or vice versa any more.

Re:well I've always wondered this

[windex]

We use -48VDC, and it's a pain in the ass to find power supplies for modern hardware.

Whenever we need something outside of normal ATX, we wind up paying custom development fees.

No one makes DC to DC power supplies that are worth a damn, and the few vendors who do sell them (Sun, IBM, etc) charge an arm and a leg above and beyond what we pay to have them custom engineered.

Re:well I've always wondered this

[Detritus]

Due to the insane current and voltage regulation requirements of today's motherboards, the power supply for the CPU and associated chips has to be physically close and tightly integrated into the motherboard. You can't just pipe in regulated DC voltages from an external power supply directly to the chips on the motherboard. In your typical PC, the power supply (the metal box) provides bulk regulated DC power. Some stuff can run directly from the power supply. Components with demanding power requirements, like the CPU, are powered by dc-to-dc converters on the motherboard. These take DC power from the power supply, convert it to high-frequency AC, and back to regulated DC.

The general rule is that stricter requirements for power supply performance can only be met by decreasing the physical distance between the power supply and the load. The trend towards lower supply voltages and higher currents makes the problem worse.

AC power wiring is cheap and well understood. It doesn't require huge buss bars or custom components. It is the most economical way to distribute electrical energy.

Once you reach the box level, you want to convert the AC to low-voltage DC. Confining the high-voltage AC to the power supply means that the rest of the box doesn't have to deal with the electrical safety issues associated with high-voltage AC. The wiring between the power supply and load is short enough to provide decent quality DC power at a reasonable cost. Those components that require higher quality power can use the DC power from the power supply as the energy source for local dc-to-dc converters.

You could feed the box with -48 VDC like the telephone company does with its hardware. You would still end up with about the same amount of hardware inside the box to provide all of the regulated DC voltages needed to make it work. Cost would increase because of the lower production volumes associated with non-standard power supplies.

In the end, it boils down to economics. DC power distribution costs more money and it doesn't meet the performance requirements of modern hardware. The days of racks full of relays, powered directly from battery banks, are long gone.

Cray still has water cooling!

[green pizza]

Unlike most companies that are considering going back to water cooling, Cray has always used water cooling for their big iron. In fact, the only air cooled Crays are the lower end or smaller configured systems.

All hail the Cray X1E !

Re:Cray still has water cooling!

[1zenerdiode]

I thought Cray's originally used Fluorinert(tm), which is definitely *not* water... *spark* *spark* *fizzle*

All hail non-conductive fluorochemicals!

If you have cooler servers...

[havaloc]

...you won't need as much cooling in the room. Easy enough. This will save a ton of money in the long run, not to mention the environment and all that.

inevitability breeds contempt

[icebrrrg]

"Roger Schmidt, chief thermodynamics engineer at IBM, [recently] admitted that, while everyone knows servers are one day going to be water-cooled, no one wants to be first, believing that if their competitors still claim they are fine with air cooling, the guy who goes to water cooling will rapidly drop back in sales until others admit it is necessary."

you know, some times the market actually rewards innovation. tough to believe, i know, and this isn't innovation, it's common sense, but mfg's are afraid of this? come on, people, the technocenti have been doing this for their home servers for a long, long time, let's bring it into the corporate world.

Re:Why not both?

[Anonymous Coward]

Cost.

The question isn't whether it's good to keep both cool. The question is, which makes more financial sense? Cooling the whole room? Spending the money to purchase servers with a low heat-to-computation ratio?

Probably a combination. But to say that people should equip every computer with VIA ultra-low-power chips _and_ freeze the server room is silly.

Re:uh, "rumble"

[Master_T]

Yeah man.... I got stuck in one of them cool room or cooling system "rumbles". They are hardcore. I lost a buddy in one those... pen wound. I still have a scar from when a Cool Roomer stabbed me with a ISA Modem.

Water cooling, pah!

[hazee]

Water cooling? Pah! Why not take a leaf out of Seymour Cray's book - build a sodding great swimming pool, fill it with non-conductive freon, then just lob the whole computer in.

Also has the added benefit that you can see at a glance which processors are working the hardest by looking to see which are producing the most bubbles.

Wonder if you could introduce fish into the tank and make a feature of it? If you could find any freon-breathing fish, that is...

No way, not in my shop

[doc_traig]

The sign on the door clearly states, "No Food or Drink". Of course, shirts are still optional.

Cooler servers...

[Aphrika]

From experience of aircon failing/breaking.

At least if a server fails it's one unit that'll either get hot or shutdown which means a high degree of business continuity.

When your aircon goes down you're in a whole world or hurt. Ours went in a powercut, yet the servers stayed on because of the UPSes - hence the room temperature rose and the alarms went off. Nothing damaged, but it made us realise that it's important to have both, otherwise your redundancy and failover plans expand into the world of facilities and operations, rather than staying within the IT realm.

Reminds me of an amusing anecdote

[Raul654]

About 4 years ago, I was touring the US National Supercomputing Center in San Diego. One of the supercomputers had a clear plexiglass side where you could see inside, and it had running water and even a waterfall. Mind you, this 'water' was running directly over the electronic components. So the guy doing the tour said that it wasn't really water, but a chemical compound similiar to water, but very nonconductive. He tells us that it costs $10,000 per barrel, and that he always gets questions about what happens if you drink it. "Well, we're not sure what happens if you drink it, but we figure one of two things will happen. It could be toxic, and you drink it and die. Or, it could be nontoxic, and when our finicial guys found out you were drinking their $10,000-a-barrell water, they'll kill you."

Hmmm...

[biglig2]

PHB: Dear god, that server is actually red hot!

SA: Yes, but notice that the room is lovely and cool.

PHB: That's all right then. By the way, what's delaying that upgrade to Windows 2003?

SA: Every time we put the CD in the drive it melts. We think it's going to be fixed in the next service pack.

The A/C company brought our water

[Roadkills-R-Us]

Bob was changing backup tapes when something caught his eye at his feet. Looking through te holes in the cooling tile in the raised floor, something was moving, like a bundle of shiny snakes. Looking closer, we had 1/2" of water down there!

We spent several hours with a tiny shop vac (we need a bigger one!) emptying the water and being thankful Bob had seen it before it got high enough to get into the power conduits.

An A/C unit drain pan had a clogged drain, so the sump pump couldn't carry the water away. Whoever had the units installed had purchased water alarms, but *they had never been hooked up*. Now *that* was a brilliant move.

We now have water alarms down there.

Meanwhile, the room stays about 70 degrees, and the servers stay comfy, as do we. I like it that way,

Re:Why not neither? Remove the power supplies.

[Iphtashu Fitz]

Converting to DC can help a lot in big datacenters if you have a lot of hardware. UPS's run exclusively off DC. (remember, they're basically just chains of car batteries daisy-chained together) The datacenters lose power & generate heat in the conversion from AC to DC and back to AC. They're always happy to avoid that second step if possible. And if you happen to have hardware located in a datacenter where telcos have equipment you're likely to find a huge DC infrastructure already in place since a lot of telco equipment runs on DC.

Personally I think BOTH the power & the cooling needs to be addressed. I've worked in datacenters where cabinets are filled with 30+ 1U servers. Not only is it a royal pain in the ass to deal with all the power cabling for 30 individual servers but the heat blowing out the back of those cabinets is enough to melt the polar ice caps...

I've also worked on blade servers like IBM's BladeCenter. Their next generation of blades will require even more power than it currently does. Trying to convince a datacenter to run 4 208 volt feeds to handle just a pair of BladeCenters (28 blades) is like pulling teeth. They can't comprehend that much power in such a small footprint. A rack full of BladeCenters could easily require 8 208 volt feeds, whereas a rack full of 1U's may only need 3 or 4 110 volt feeds.

Water-chilled Cabinets Already In Use for Blades

[miller60]

Liebert and Sanmina have been selling blade server cabinets that use chilled water for at least three years. Vendors and data center operators have been wrestling with the heat loads generated by blade servers since 2001, and the dilemma of how to cool high-density "hot spots" has caused many tech companies to wait on buying blades to replace their larger servers. That's changing now, driven by the need to save costs with more efficient use of data center space.

The industry has taken a two-pronged approach. Equipment vendors have been developing cabinets with built-in cooling, while design consultants try to reconfigure raised-floor data center space to circulate air more efficiently. The problem usually isn't cooling the air, but directing the cooled air through the cabinet properly.

There was an excellent discussion of this problem last year at Data Center World in Las Vegas. As enterprises finally start to consolidate servers and adopt blade serves (which were overhyped for years), many are finding their data centers simply aren't designed to handle the "hot spots" created by cabinets chock full of blades. Facilities with lower ceilings are particularly difficult to reconfigure. The additional cooling demand usually means higher raised floors, which leaves less space to properly recirculate the air above the cabinets. Some data center engineers refer to this as "irreversibility" - data center design problems that can't be corrected in the physical space available. This was less of an issue a few years back, when there was tons of decent quality data center space available for a song from bankrupt telcos and colo companies. But companies who built their own data centers just before blades became the rage are finding this a problem.